DEV Community: Hassan Imam

I Built an AI That Remembers the Promises We Forget

Hassan Imam — Sat, 24 Oct 2026 12:27:09 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

The report was supposed to arrive yesterday. Somewhere in last week's notes, someone said they would send it. Before asking for an update, a researcher has to reconstruct the conversation: who offered, what they offered, and whether there was actually a deadline.

That is the fictional situation in my explicitly synthetic demo. It captures a familiar kind of work: remembering the small agreements that never became formal tasks.

I built PromiseKeeper to turn those agreements into a queue a person can check. Import meeting notes, a chat transcript, or a text file. A local language model proposes commitments. Every suggestion stays beside the exact words that support it. The reviewer can approve it, correct it, dismiss it, or mark it complete.

An approved promise can then become a polite follow-up draft. The person reviews and edits the wording, and decides whether to send it. PromiseKeeper has no sending integration.

The app is built around the difference between a possible promise and a confirmed one. “I'll send the report” deserves attention. “I already sent the report” describes something finished. “If the samples clear, I'll upload the appendix” carries a condition that should stay attached. An uncertain date should remain uncertain.

The interface gives those distinctions room: an evidence review inbox, upcoming and overdue summaries, a timeline, follow-up drafts, and local privacy controls. It supports pasted text and .txt or .md files. SQLite stores the sources and review decisions on the same machine as the local model.

The desired payoff is modest and concrete: bring the relevant promise back into view, with enough context to decide the next step.

I designed the workspace around reading and deciding: a left sidebar, vertically arranged input fields, and charcoal, forest-green and mint surfaces. Source quotes stay readable inside each card. The mobile drawer supports keyboard focus and Escape, and the help dialog explains the review boundary. Follow-up drafts can be copied or downloaded; source deletion lets the person check exactly what they are removing.

Demo

Watch the demo on YouTube. The redesigned interface is demonstrated in a 112.36-second, 1920x1080 recording narrated by genuine Kokoro-82M using af_sarah. The local export is saved at demo/assets/video/final_demo.mp4.

The recording follows a researcher importing five snippets of synthetic notes. Actual local Qwen inference finds three candidates: a revised field report, cleaned survey figures, and a conditional appendix. The already-delivered consent forms and vague conference thought stay out of the action queue.

The reviewer checks the quotes, approves the report, and sees one confirmed promise past its date. Two suggestions remain available for review.

The final action is preparing an editable follow-up. The draft asks whether the recorded timing has shifted; it does not assume the person has failed to deliver.

These are screenshots of the working application. The video preserves the browser's original action timing, with added scene holds for narration. Its edited runtime is not a processing benchmark.

Code

Browse PromiseKeeper on GitHub. The repository includes the frontend, backend, synthetic evaluation corpus, API tests, browser smoke test, real application screenshots, and recording and narration scripts. The recording and narration scripts reproduce the demo; generated video and WAV files are local outputs.

The code is MIT licensed. The selected Qwen model and Kokoro weights have their own licenses and are downloaded separately.

The manual setup uses Python 3.12+, Node 22+, and Ollama. Pull qwen3:8b, start the FastAPI service on port 9000, and run the Next.js frontend. The README contains the commands and a Docker Compose option. There is no paid inference service to configure.

Useful places to begin reading are backend/app/main.py for the extraction and review boundary, app/page.tsx for the user journey, and backend/eval/run_evaluation.py for the measured synthetic evaluation. demo/record_demo.ts and demo/narrate_demo.py reproduce the screen recording and voiceover.

How I Built It

The frontend uses Next.js and TypeScript. FastAPI handles import, local inference, validation, review updates, drafting, export and source deletion. SQLAlchemy stores the relationships in SQLite.

The processing path is small enough to inspect:

Notes → source document and paragraph fragments
      → local Ollama / Qwen extraction
      → schema validation and source-quote check
      → evidence-linked suggestions in SQLite
      → human review → approved promise → editable draft

The configured extractor is Qwen3 8B through Ollama. The locally inspected model reports 8.2 billion parameters and Q4_K_M quantization. The evaluation records the model digest, rather than relying only on a tag that could later point to different weights. Structured extraction runs at temperature zero with reasoning disabled.

The model proposes an action, owner, recipient, deadline, condition, certainty, evidence quote and next step. Pydantic checks the shape of those values. The API then independently verifies that each evidence quote is a literal substring of the original source text. A plausible quote that does not occur in the notes is rejected.

That boundary is useful, but deliberately limited. A quote can be present and still fail to support the proposed owner or claim. The source check establishes provenance; the reviewer still has to establish meaning. The interface keeps the quote beside the action so that check can happen where the decision is made.

The backend also applies conservative lexical rejection to obvious refusals, completed actions and hypothetical statements. An earlier development run exposed false positives in these cases. The final measurements therefore describe the combined prompt, model, schema checks and filters. They are not a score for the base model alone.

Dates need a similar distinction. Explicit dates can be stored directly. Relative timing can be interpreted only when a source timestamp is known. Without that anchor, the date stays unresolved. A missing owner is allowed to remain missing too; filling a field with a confident guess would make the queue less trustworthy.

Each source is split into paragraph fragments with character offsets. Evidence records preserve quotes and their locations. When the same extraction key repeats within an import, the API groups it while retaining distinct supporting quotes. This is normalized exact grouping, not semantic deduplication. Two differently phrased obligations can remain separate, and exact grouping has its own limits when conditions differ. Broader similarity matching needs more work before it should alter a person's action queue.

Review changes create audit events. A source deletion removes its linked commitments and drafts. The dashboard counts confirmed overdue items, and CSV export includes approved commitments with their evidence. The prototype is intended for a person's local machine; shared hosting, account permissions and encrypted storage are separate deployment concerns.

Follow-up drafting uses a deterministic template assembled from approved fields. It does not ask a second model to invent background context, recipients, or excuses. The local model's core job is finding candidate obligations in unstructured conversation. Once the person approves the fields, predictable drafting keeps the next step inspectable.

I checked the implementation with four backend API tests and a production build including TypeScript checks. The browser journey uses genuine local Qwen inference and covers file loading, quote inspection, owner/date/note edits, approval, editable draft copy and download, the timeline, CSV export, completion, source deletion and cancellation, help, mobile keyboard navigation and dark mode. A later import is checked to ensure it does not inherit the previous source timestamp. Automated axe checks report zero violations for the selected rules across 16 light/dark page, dialog and mobile-drawer states; that is a scoped automated check, not a complete manual accessibility audit. The recording runs the actual browser, backend and local model.

For extraction, I authored 55 labelled synthetic cases across 11 categories. The preserved run produced these results:

Check	Measured result
Case-level commitment classification	40 true positives, 0 false positives, 0 false negatives
Owner exact match	30 of 35 eligible cases
Explicit calendar-date exact match	5 of 5 eligible cases
Accepted quote substring check	41 of 41 accepted candidate quotes

The owner result is the most useful warning in that table: the system can identify a possible obligation and still attribute it incorrectly. In this corpus, implicit-owner examples account for the five owner mismatches.

The perfect classification result is a small development-set observation. These cases are authored, repetitive, and were used while improving the pipeline. Quote validity is also an enforced acceptance rule, so its perfect rate is not independent evidence of understanding. The evaluation does not score claim entailment and does not cover sarcasm, multilingual conversations, noisy transcription, contradictions, or the full range of date language. The five date examples are a narrow check.

The corpus, evaluator and per-case outputs are preserved so a reader can inspect those limits and add harder examples. Real conversations and a held-out evaluation would be the next useful evidence.

Why Does Open Innovation Matter?

A commitment assistant handles information that can be mundane in isolation and sensitive in combination: who is waiting for whom, which deadline is slipping, and what people agreed to in a private discussion.

Running an open-weight extractor locally makes the data path understandable. After the model has been downloaded, inference does not need a hosted model account. The notes go from the local app to the configured local Ollama service, and the resulting records stay in SQLite. The demo uses synthetic material, but the architecture is chosen for conversations a person would want to keep private.

Open components also make failure inspectable. I can retain a failing synthetic example, adjust the prompt or acceptance filter, and run the same evaluator again. A model digest, a source quote, and a saved result give someone else something concrete to investigate. They can replace the model adapter and keep the validation and human review boundaries.

There are costs to that choice. An eight-billion-parameter model needs a substantial download and enough memory. Local inference can be slow or unavailable on a particular computer. Initial setup is more work than entering a hosted API key. The UI exposes model availability and processing failures because those constraints affect the user's experience.

The same approach carries into the video. Kokoro runs locally, its selected voice and weight hashes are recorded, and the narration script and caption timings are included. The demonstration can be regenerated from the project instead of depending on an unexplained media asset.

The Lazy Payoff

The visible transformation is five raw snippets becoming three candidate obligations, then one approved overdue report and an editable follow-up. The person makes the important decisions while the app gathers the relevant words into one place.

I have not measured saved hours, fewer missed deadlines, or reduced stress. The synthetic demonstration shows a workflow, and the evaluation measures a narrow extraction task. Neither establishes a productivity improvement for real teams.

The burden I want to reduce is repeatedly reconstructing the same agreement before taking a small, reasonable next step. Sometimes that step is sending a reminder. Sometimes it is noticing a condition and waiting. Sometimes it is rejecting the suggestion because there never was a promise.

Keeping that choice with the person matters as much as finding the sentence. A useful assistant should help us remember what we owe one another, while leaving room for people to explain what changed.

Prize Categories

Overall Slow and Steady challenge entry.

Drafts That Never Shipped: do LLMs treat dead proposals as real standards?

Hassan Imam — Sun, 11 Oct 2026 11:23:19 +0000

This is a submission for the Kaggle Benchmarking Challenge.

🎥 3-minute overview (narrated):

What I Benchmarked

Standards have a long tail of proposals that never shipped: emoji withdrawn from a draft
list, PEPs that were rejected or deferred, IETF Internet-Drafts that expired, HTTP headers
that never made it into the registry. They are all over the web: blog posts, mailing lists,
early implementations. So a model has seen plenty of text about them, often written as if
they were real.

I wanted to know: do LLMs treat these dead proposals as real standards? And, more
importantly, does it show up in the things models produce, not only when you ask them
directly? A model that says "PEP 505 was deferred" but then writes a?.b in your code is
the dangerous case.

Drafts That Never Shipped asks about 122 entities in five families:

Family	Never shipped (example)	Matched shipped "twin"
Emoji	Apple Core (withdrawn from the Emoji 17.0 draft)	Lime, Shaking Face
PEPs	PEP 505 None-aware `?.` (Deferred), PEP 3150 `given` (Deferred)	PEP 572 `:=`, PEP 634 `match`
PEP draft syntax	PEP 622's `case .RED:`, PEP 750's first-draft `greet"..."`	`case Color.RED:`, `t"..."`
IETF drafts	RateLimit headers draft, JSON Schema, OAuth 2.1	RFC 8941, RFC 9068, …
HTTP fields	`Idempotency-Key`, `RateLimit-Policy`, `Want-Digest` (obsoleted)	`Priority`, `Cache-Status`, `Prefer`

Plus fictional placebos (fake emoji names, PEP numbers that don't exist, draft names that
404 on Datatracker) to measure plain hallucination.

Each entity is asked about on several surfaces:

Status: "Did X ship?" → {"shipped": true|false}
Lookup: the codepoint / Python version / RFC number, or NONE
Artefact: actually use it:
- a chat message with the emoji;
- a Python 3.14 snippet;
- a citation;
- raw HTTP headers.

Every artefact prompt offers a neutral opt-out (NOT_AVAILABLE), so a correct answer
always exists.

Ground truth is never hand-typed. Every label comes from a pinned, hashed,
machine-readable source:

Unicode emoji-test.txt / UnicodeData.txt (15.1 → 18.0);
peps.json;
the IETF Datatracker API;
the IANA HTTP Field Name Registry.

Scoring is fully deterministic, with no LLM judge. Python snippets are compiled (never
executed) under Python 3.14: if your "PEP 505 example" doesn't compile, that's
evidence you believed the syntax exists. My own curation was wrong once (I assumed the
HTTP QUERY method draft never shipped; Datatracker says it became RFC 10008). The pinned
source won, and the conflict is logged.

Models Tested

Model	Lab	Why
Claude Sonnet 5	Anthropic	frontier general model
Gemini 3 Flash	Google	fast frontier model
Gemini 3.1 Flash Lite	Google	small/cheap tier of the same family
GPT-5.4 nano	OpenAI	small/cheap tier
gpt-oss-120b	OpenAI	open-weight
DeepSeek R1	DeepSeek	reasoning model: does thinking help?
Granite 4.0 H Small	IBM	small open model
Gemini 3.7 Flash	Google	newest model available; run on Kaggle's own infrastructure
Qwen3-Next 80B	Qwen	open-weight MoE; run on Kaggle's own infrastructure

That covers 9 models from 6 labs: frontier vs. small tiers within a family, open vs. closed,
a reasoning model, and the newest Gemini. Qwen's endpoint hung through my local proxy, so it
(like Gemini 3.7 Flash) was run once on Kaggle's own infrastructure. About 6,400 graded answers in total, at temperature 0, with 3 repeats on
non-shipped items.

Findings

1. One in four answers about a never-shipped standard says it shipped, and the errors
point one way. Pooled over all 7 models, 26 % of answers on real non-shipped items
assert they shipped. Excluding very recent releases, 84 % of all errors are "believed it
shipped". Only about 10 % go the other way ("denied something real").

Model	Says a non-shipped standard shipped	Accuracy on real twins
Claude Sonnet 5	9 %	90 %
Gemini 3 Flash	11 %	97 %
gpt-oss-120b	18 %	77 %
Gemini 3.1 Flash Lite	21 %	86 %
DeepSeek R1	26 %	92 %
GPT-5.4 nano	44 %	62 %
Granite 4.0	57 %	34 %
Gemini 3.7 Flash	5 %	97 %
Qwen3-Next 80B	9 %	68 %

2. Strong models get the yes/no question right and the details wrong. The frontier
models rarely say "yes, that shipped". But ask for a number or a working artefact and
the dead draft leaks through:

Claude Sonnet 5, asked which RFC defines the RateLimit headers (still an Internet-Draft): "RFC 9777". Asked for example headers, it wrote RateLimit-Policy: 100;w=60 as if it were a registered field.
DeepSeek R1, asked to cite the same draft, produced a perfectly formatted … "RateLimit Header Fields for HTTP", RFC 9436, DOI 10.17487/RFC9436, August 2023. That citation is for an RFC that does not exist as that document.

Claude and Gemini both say Digest is defined by RFC 9530, which is the RFC that obsoleted it.
gpt-oss-120b, asked for PEP 622's leading-dot pattern, wrote case .RED:. That is draft syntax that was dropped before pattern matching shipped as PEP 634, and it doesn't compile.

Gemini 3 Flash (in the pilot run), asked for PEP 750, wrote tag'Python {name}': the custom-prefix syntax from the PEP's first draft. Only t"..." shipped in Python 3.14.

The newest model, Gemini 3.7 Flash, has the lowest error rate (5 %). But its remaining
mistakes are the same ones: it writes RateLimit-Policy: 100;w=60, character for character
what Claude wrote, and also names RFC 9530 as the home of Digest. Different labs, same
dead draft, same wrong details. These errors come from the web text the models learned
from, not from any one model's quirks.

The PEP draft-syntax pattern surprised me most. These aren't hallucinations from nowhere. They are
faithful memories of an earlier draft of a spec that later changed. The model learned
the proposal, never learned the outcome.

Qwen3-Next shows the opposite profile: it rarely believes dead drafts (9 %) but also denies a
third of real twins (68 % twin accuracy). A low believed-shipped rate alone can mean caution,
not knowledge; that's why every non-shipped item has a twin.

3. "Knows" ≠ "does", and it depends heavily on the model. Among cases where a model got
the status question right, how often was its artefact still wrong?

Claude 1.4 %, Gemini Flash 2.1 %, DeepSeek R1 2.0 %;
Gemini 3.1 Flash Lite 23 %, GPT-5.4 nano 26 %.

Flash Lite correctly told me that two pieces of dropped draft syntax (PEP 572's
(expr as name) and PEP 622's case .RED:) are not in Python, and then used them in all
6 code samples I asked for. For the never-registered HTTP fields it said "not a current
field" and then emitted the header anyway, 9 times out of 9. For small models, a correct answer to "does X
exist?" says little about whether they'll use X in code.

Strong models aren't immune. Claude Sonnet 5 correctly said RateLimit-Policy is not a
registered HTTP field, then wrote it into an HTTP response when asked for an example:

There's an inverse failure too: Claude says Goal Net 🥅 and Sauropod 🦕 are real emoji, then
answers NO_SUCH_EMOJI when asked to write a message using them. That's over-caution,
not premature belief, but it's the same disconnect between knowing and doing.

4. Showing the source works; telling the model to "be careful" mostly doesn't. On
held-out items I compared three conditions:

no fix;
a placebo instruction ("Be careful: some proposals never shipped.");
the actual pinned source row in the prompt (the peps.json entry, the emoji-test.txt line, the Datatracker state, the IANA registry row).

Model	No fix	"Be careful"	Source row in prompt
Claude Sonnet 5	4.5 %	9.1 %	2.3 %
Gemini 3 Flash	13.6 %	2.3 %	4.5 %
Gemini 3.1 Flash Lite	22.7 %	18.2 %	9.1 %
GPT-5.4 nano	36.4 %	20.5 %	15.9 %

(believed-shipped rate on held-out non-shipped + placebo items)

The evidence roughly halves the error rate or better for every model. It also raises
accuracy on real items (nano's twin accuracy goes from 60 % to 97 %). The warning helps some
weaker models but is unreliable: for Claude it made things slightly worse, though cells are
small.

5. Recency is the opposite failure and has to be separated. My first pilot looked like
the benchmark didn't work: models did worse on real twins than on dead drafts. The cause
was twins that shipped recently (Emoji 18.0 in September 2026, Python 3.14, RFCs from 2024–25),
which models simply don't know yet. I moved those into a separate "fresh" slice and
re-piloted. Post-cutoff staleness and premature belief are mirror images: one says real
things don't exist, the other says dead things do.

What surprised me

How specific the wrong answers are: DOIs, RFC numbers, exact draft syntax.
That the reasoning model (DeepSeek R1) was among the more confident believers.
That one cheap model could be right about a fact and wrong about using it, every single time.

What I'd measure next

Draft-detail questions at scale. Mine every PEP / RFC whose syntax or field names changed between a draft and the final text, using the git history of python/peps and Datatracker revisions.
Agentic settings: does a coding agent with a linter or test loop catch the dead syntax, or rationalise it?
Retrieval of the wrong document: what if the context contains the old draft rather than the final spec?

Honest caveats

My first pilot failed its pre-registered gate. The fixes (a freshness slice, two new item families, a competence floor for the pooled gate) were chosen after seeing pilot results. They are all dated in the pre-registration in the repo.
The pilot's twin-vs-non-shipped gap for Claude (+16 pts on 81 items) did not replicate on the full set (≈ 0); only DeepSeek R1's gap is statistically clear (+18 pts, McNemar p = 0.008).
A few scorer bugs were found while auditing outputs; all were fixed with unit tests and re-scored from cached responses.

My Benchmark

👉 Kaggle benchmark: https://www.kaggle.com/benchmarks/tasks/shs123/drafts-that-never-shipped

📦 Items dataset: kaggle.com/datasets/shs123/drafts-that-never-shipped-items

Built with Kaggle's kaggle-benchmarks library
(@kbench.task, structured output via llm.prompt(schema=...), .evaluate() with on-disk
caching). The leaderboard score is (correct, scored); format and API errors are excluded
and reported separately.

Note on numbers: the figures in this post come from my full local runs (278 items plus 3
repeats on non-shipped items, code checked under Python 3.14); Gemini 3.7 Flash and Qwen3-Next
come from one Kaggle run each, with code re-checked locally under 3.14. The Kaggle leaderboard runs
each model once on all 364 items. Kaggle's runner is older than Python 3.14, so the few
code-compile items are left out of its score there. Expect small differences between the two.

Credits: ground truth from the Unicode Consortium, the Python PEPs index, the IETF Datatracker
and IANA. Emoji draft history from Emojipedia and Emojiterra; rejected-proposal history from
Charlotte Buff's catalogue.

Lake Lookout: Be the Eyes the Satellites Lack (Gemma 4, fully offline)

Hassan Imam — Fri, 09 Oct 2026 19:10:27 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

TL;DR: Lake Lookout turns a hike past glacial lakes into a field log that researchers can use. On the trail you only take photos and short voice notes, and your phone stays in your pocket. That evening, with no internet, Gemma 4 runs on your own laptop, listens to every voice note, looks at every photo, and fills in a checklist drawn from published glacial-lake hazard indicators. Every yes or no has to name its evidence. Anything it can't actually see stays unclear. It is open source (MIT), and it costs nothing to run.

What I Built

The lake nobody was watching

On 16 August 2024, Thyanbo Tsho, a small glacial lake above the village of Thame in Nepal's Everest region, burst. The flood reached Thame about 22 to 25 minutes later. Homes, the school and the health post were damaged, and 135 people were displaced.

Earlier this year I built a separate project that screened Himalayan glacial lakes from free satellite images. Its most uncomfortable finding wasn't about models at all; it was about the sky. In the pre-event satellite record for the four floods it studied, only 1 of 16 scenes got past the cloud-and-snow quality filter. The monsoon hides these lakes exactly when they fill up.

People walk past them all the time, though: trekkers, guides, porters, herders. Their eyes work under cloud. What they see just never gets written down in a form anyone can use.

So, for a challenge called Touch Grass, I built the opposite of a dashboard you stare at. Lake Lookout makes the hiker the sensor, and keeps the screen for the evening.

How a day works

On the trail. At each lake, take one wide photo and record 10 to 30 seconds of what you see: "Grey milky water, a long ridge of loose rock between the lake and the glacier, ice on the far shore." That's all. Your phone's own camera and voice recorder; no app, no signal.
In the evening. At the teahouse, the campsite or home, copy the files to a laptop and drop them into Lake Lookout. Gemma 4 (E4B, through Ollama) transcribes each voice note, reads each photo, and fills in a field checklist for every stop. It uses the GPS and time already in the photos to rebuild your route.
Share the log. You get one offline page with your route, photos, words and checklist, plus a CSV and a GeoJSON file a researcher can load straight into a spreadsheet or a GIS.

Who it's for

Anyone who walks past mountain lakes: trekkers, and above all the guides, porters and herders who pass the same lakes every season. The logs are for the people who study these lakes and are qualified to interpret them. The screen time is a few minutes in the evening; the rest of the day is spent looking at the lake.

What it records

The checklist asks what a hiker can actually see, and where a question corresponds to a published hazard indicator, it says which one:

Observation	Published indicator it corresponds to
Glacier ice touching the water	Rounce et al. 2016 (mother glacier within 600 m)
Icebergs or calving	Sakai et al. 2009 (calving onset)
Steep slopes or hanging ice above the lake	Fujita et al. 2013; Rounce et al. 2016
Fresh rockfall or landslide scars above	Allen et al. 2019 (impulse-wave trigger)
Low freeboard (dam crest close to the water)	Mergili & Schneider 2011
Steep outer dam face	Lv et al. 1999 (moderate confidence)
Water body · glacial setting · moraine dam · seepage · people downstream	plain field observations, labelled as such

It deliberately gives no risk score and raises no alert. It records observations for people qualified to interpret them. A wrong "this lake is safe" is worse than no app at all.

Taking it outside: a dry run first

I haven't done my own walk yet as I write this, so I won't pretend I have. To see what a real day produces before I go, I built one. It has seven stops along the Karakoram Highway in Hunza, from Karimabad to Passu, assembled from geotagged Wikimedia photos with synthetic voice notes. The repo labels it as test data and credits every photographer. Then I processed it the way you would on an evening at a guesthouse: one click, and 28 seconds on my machine's GPU for all seven stops.

What it got right:

No lake, no claims. At Baltit Fort, with no water in the photo, water body came back no.
The hiker's words were enough. A voice-only stop, "a small muddy pond next to the road, no ice anywhere near it", was logged as not glacial from the words alone.
Disagreements were shown, not resolved. At Passu Glacier the photo and the voice note disagreed about water and ice. The log shows both and picks neither.

What it got wrong: the heavily edited winter photo of Borith Lake. More on that under How I Built It.

My own walk is next, and I'll add it here.

Demo

The demo is the real app and the real local model. To show a realistic day, it uses a test day I assembled from freely licensed Wikimedia photos of Hunza, Pakistan, with synthetic voice notes; it is labelled as such in the repo. Processing is sped up in the video. It took about 30 seconds on my machine for 7 stops. Even the narration is local and open: it is Kokoro-82M, an open-weight text-to-speech model running on the same laptop. I checked every line by having Gemma 4 transcribe it back, which is how I found that the voice said "Alama" for "Ollama".

Live demo (read-only): https://hassan-2050.github.io/lake-lookout/

The real app UI over the processed test days: map, evidence, voice notes, filters and downloads. Nothing is processed on the website. Processing a hike stays on the hiker's own laptop, by design.

Code

hassan-2050 / lake-lookout

Hike with your phone. In the evening, a local Gemma 4 turns your photos and voice notes into a glacial-lake field log, offline.

Lake Lookout

Hike with your phone. At each lake, take a photo and say what you see. That evening, with no internet, a local open-weight model turns the day into a glacial-lake field log.

On the trail the screen stays in your pocket. You use your phone's own camera and voice recorder. Back at the teahouse, campsite or home, Lake Lookout reads the day's photos and voice notes on your laptop, and Gemma 4 (via Ollama) answers a short field checklist for every stop. The checklist is drawn from published glacial-lake hazard indicators, and every answer has to name its evidence. You get:

trip.html: one self-contained page with your route, photos, what you said and the checklist for each stop. It opens with the network off.
field_log.csv and field_log.geojson: the same observations in forms a researcher can load straight into a spreadsheet or a GIS.

Nothing leaves the…

View on GitHub

Try it yourself in a few minutes; the repo includes test days:

ollama pull gemma4:e4b      # or gemma4:e2b for a laptop without a GPU
git clone https://github.com/hassan-2050/lake-lookout && cd lake-lookout
python -m venv .venv && .venv/Scripts/pip install -r requirements.txt   # Windows
# python -m venv .venv && .venv/bin/pip install -r requirements.txt     # macOS/Linux
python -m lookout ui        # then open the test-hunza day and press Process

Licence: MIT.
Photos: all evaluation and test-day photos are from Wikimedia Commons under CC BY / CC BY-SA licences, credited in eval/photos/ATTRIBUTION.md and in each test day's README.
Video: recorded from the real app with Playwright (tools/record_demo.py), narrated by Kokoro-82M (Apache-2.0) through kokoro-onnx, run locally.

How I Built It

Open-source AI at the core

Everything that thinks runs locally on open weights:

Gemma 4 E4B (or E2B on a laptop without a GPU), served by Ollama, does all three jobs: it transcribes the voice note from audio, reads the photo, and returns the checklist as JSON constrained to a schema.
Kokoro-82M, an open-weight text-to-speech model, voiced the demo video.
Around them: Python (standard library plus Pillow), and ffmpeg for audio.

phone photos (EXIF time + GPS) ─┐
phone voice notes (m4a/mp3/…) ──┼─► group into stops (time + distance, UTC-safe)
                                │
                                ▼
          ┌─────────── offline guard: only 127.0.0.1 ───────────┐
          │  ffmpeg → 16 kHz WAV ─► Gemma 4: transcript          │
          │  photo ─► Gemma 4: checklist JSON (evidence first)   │
          │  voice ─► Gemma 4: checklist JSON (evidence first)   │
          │  rules: evidence · in-view · transcript · keywords   │
          │  merge photo + voice (disagreement → shown, unclear) │
          └──────────────────────────────────────────────────────┘
                                ▼
          trip.html (self-contained) · field_log.csv · field_log.geojson

The app (python -m lookout ui) is a small local server, listening only on your own machine, with a plain HTML/CSS/JS front end and no libraries. It has drag-and-drop days, live progress per stop, an offline route map, filters and one-click evidence. It works in dark mode and at phone width.

Three levels of testing:

62 unit and API tests. The API tests run the real server with a stand-in model.
A checker that compares each generated trip page with its CSV.
A browser end-to-end test that drives headless Chrome through the whole app with the real model, with about two dozen checks. It creates a day, uploads through the file picker and processes it. Then it uses the map, filters, evidence and lightbox, checks every photo decodes and that there's no sideways scroll on a phone, and expects zero console errors.

The interesting part: making a small model unable to overclaim

Getting Gemma to answer a checklist took ten minutes. Getting it to answer only what it could actually see took the rest of the build. A field log with a confident wrong answer is worse than an empty one, so I treated this as the real problem and measured every change.

I picked 11 freely licensed lake photos: seven glacial lakes, one glacier with no lake in frame, and three controls that aren't glacial at all (a landslide lake, a reservoir and a village pond). Each photo has a label for every question, which gives 89 scored answers per version. My coding agent drafted the labels by looking at each photo, then re-checked all of them in a second pass, which changed four ambiguous ones to "skip". No person has independently reviewed them yet, so treat the numbers as indicative.

version	change	accuracy	false "yes"	false "no"
v1	answer, then evidence	74%	3	4
v2	evidence, then answer	72%	4	18
v3	"is the dam in view?" questions	84%	3	5
v4	final prompt	89%	0	3
v5	v4 plus bug fixes found on real photos	88%	0	3

v1: the silent "no". The model kept answering no and then writing "none" as its evidence. An unsupported "no" is the most dangerous thing a hazard log can say, because it reads as "nothing there" and never looks like a failure. So a small rule-based verifier turns any yes or no without evidence into unclear.

v2: the fix that made it worse. Ollama's structured output generates JSON fields in the order the schema lists them. So I moved evidence in front of answer, which forces the model to say what it sees before it commits. Unsupported answers dropped to zero... and false "no" answers jumped from 4 to 18. Now it wrote, with a straight face and real evidence, "no visible seepage through the dam" about dams that were behind the camera.

v3: absence needs a view. Two questions now come first: is the dam or outlet in view? and is the land downstream in view? The verifier won't let a photo claim anything about the dam or the valley below unless that part is confirmed in view. If it isn't sure, it fails safe and the answer stays unclear. False "no" answers fell to 5, then to 3 with the final prompt.

Then voice notes broke it in a new way. Given a photo and a voice note together, the model stopped looking at the photo. It also quoted one sentence, "a long ridge of loose rock between the lake and the glacier", as evidence for six unrelated answers, including "no fresh rockfall". So now:

photo and voice are asked about separately, then merged;
when they disagree, the log shows both and marks the item unclear instead of picking a side;
voice evidence must actually appear in the transcript, and must mention something the question is about.

Here is that working on a real test stop. The hiker's "concrete dam" correctly answers moraine dam: no. Whether the dam is in view ("…behind me") is shown as a disagreement, not resolved:

None of these rules needed a bigger model. They are a few dozen lines of plain Python, and each one exists because a measurement showed the failure it prevents. Every downgrade is kept, with its reason, so you can see what the model said as well as what the log kept:

What real photos taught me

Labelled photos only go so far, so I also built two realistic test days in Gilgit-Baltistan (Hunza and Skardu) from geotagged Wikimedia photos. They include glaciers with no lake in frame, villages with no water at all, a dammed reservoir and one heavily edited winter photo. They found things my tests hadn't:

A confident hallucination. On the heavily edited, HDR-style winter photo of Borith Lake, Gemma "saw" a moraine dam, low freeboard and the outlet, each with plausible evidence. No rule can catch that: the evidence reads fine and the model says the dam is in view. Plain phone photos are what this is built for, and the README says so.
A rule that had never worked. While editing, I let a script write the regex \b in the "I can't see it" rule as a literal backspace character. The rule matched nothing, and its tests still passed because a different rule downgraded the same answers. The tests for that rule now check the reason for the downgrade, and I confirmed they fail on the old code.
A bug I found by watching my own demo video. At a stop with a fort and no lake, the photo pass claimed the dam was in view and the voice pass disagreed. The merge correctly marked "dam in view" as a disagreement, but the photo's "steep dam face: yes" had already passed its own check. The in-view rule now runs again after merging.
Place names get misheard. "Attabad" came back as "A bad lake", and "Satpara" as "Sapporo". The observations survive; the names don't.

Honest limits

Small evaluation. 11 labelled photos, labelled by my coding agent (two passes) and not yet reviewed by a person. Scores move a few points between runs even at temperature 0, so treat small differences as noise.
"Is the dam in view?" is the model's weak spot (27 to 47% across the Gemma 4 runs). The rule fails safe, so a wrong "not in view" costs answers rather than producing false ones.
Edited photos can fool it, as the winter Borith photo showed.
Voice in other languages: the transcription prompt asks for the original language plus an English line, but I have only tested English so far.
It is not a hazard assessment, and it won't become one by adding a score.

What's next

Voice notes in Urdu and Nepali from the guides and porters who walk past these lakes most often, with English in the log.
A way to hand a season of logs to the people who study these lakes, in the format they actually want. I'd rather ask them than guess.
Running E2B on the phone itself, so the evening step needs no laptop.

Why Does Open Innovation Matter?

There is no signal where the lakes are. Above the tree line in the Himalaya and the Karakoram, mobile data can be gone for days. A cloud API is useless exactly where this tool is needed. Lake Lookout runs entirely on the laptop and enforces it: while it processes a day, every network connection except the one to the local model raises an error. Offline isn't a setting you can forget to turn on.

The data shouldn't leave. A field log is a list of exact coordinates of lakes, trails and villages, plus a person's voice. With open weights running locally, none of it goes to a server I don't control. The page it produces has no scripts, no web fonts and no map tiles, and it opens with the network off.

I could measure, swap and iterate for free. Five prompt versions × 11 photos × several models is a lot of calls. Locally, each one cost nothing and took about two seconds. That is what made the measurement loop above possible. I ran the same photos through three local models:

model	accuracy	false "yes"	false "no"	per photo (RTX 4090)
gemma4:e4b	89%	0	3	2.4 s
gemma4:e2b	86%	1	6	1.6 s
gemma3:4b (v3 prompt)	65%	6	4	not comparable*

E4B is the default because it made the fewest false "no" answers in every run. No GPU? --cpu switches to E2B, which took about 20 s per photo on a desktop CPU (Intel i7-12700F), roughly three minutes for an eight-stop day. A laptop chip will be slower, but it's still an evening task, not a cloud bill.

*Another program held most of the GPU's memory during that run.

One small open model does the whole job. Gemma 4's E2B and E4B take audio and images. The same local model transcribes the voice note, reads the photo, and returns JSON constrained to a schema. There is no separate speech model to install and no second runtime: one ollama pull and you have the whole pipeline.

What a closed API wouldn't have allowed: working with no signal at all, keeping people's locations and voices on their own laptop, and running hundreds of measured experiments at no cost per call. Each of those is something the project depends on, not a nice extra.

My Agent Session

I built Lake Lookout with Claude Code as my coding agent. The repository was created during the challenge week, and the commit history shows it going from an empty folder to this. The eval/results/ folder is the loop we actually worked in: hypothesis, change, measured result. Each version in the table above is a saved result file you can re-score with eval/rescore.py.

The satellite project mentioned at the top is earlier, separate work: it's the reason this one exists, not part of it.

Prize Categories

Best Use of Gemma. Gemma 4 is the core of Lake Lookout, not an add-on:

Speech: Gemma 4 transcribes each voice note directly from audio.
Vision: Gemma 4 reads each photo against the checklist.
Structured output: answers come back as schema-constrained JSON with evidence and source for every item, and field order was used deliberately to make the model ground itself before answering.
On-device class: E4B on a GPU and E2B on a CPU-only laptop, both fully offline through Ollama.
Measured, not assumed: a labelled evaluation, a three-model comparison, and the honest failure cases above.

Thanks for reading. If you live or walk near glacial lakes and want to try it, I'd love to hear what it gets wrong.

The Overhead Ratio Is Lying to You — I Built an AI Tool to Prove It

Hassan Imam — Sun, 06 Sep 2026 21:23:26 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

GlassPocket — a tool that argues against the "overhead ratio," the dominant heuristic people use to judge charities (what % of donations go to "programs" vs. "overhead" like staff and infrastructure). That heuristic punishes exactly the investment that makes a charity effective, and it drives what nonprofit finance people call the "starvation cycle" — orgs under pressure to look lean end up under-staffed and under-resourced.

You search a US 501(c)(3), and instead of a single overhead percentage, GlassPocket pulls their IRS Form 990 history (via ProPublica's Nonprofit Explorer API) and shows:

Reserve months — how long the org could run on savings alone (low reserves = fragile, not "lean")
Operating margin trends across up to 13 years of filings
Staff-investment share — reframed as capacity, not waste
Fundraising cost per dollar raised — a narrower, more honest efficiency metric than the classic ratio
A peer-percentile chart against ~70 similar organizations in the same category
A Gemini-written "myth-buster" card pairing each overhead-ratio assumption with what the numbers actually show
A grounded chat box — ask follow-up questions about that specific org's finances, answered only from its own filing data

Demo

Live app: https://glasspocket.vercel.app/

Code

hassan-2050 / glasspocket

Overhead-ratio myth buster for US charities — Form 990 data via ProPublica, Gemini narrative generator

GlassPocket — Overhead Myth Buster

Live: glasspocket.vercel.app Category: Overall Winner + Best Use of Google AI (Gemini-powered narrative generator and chat)

The Hook

Most charity-rating tools reinforce the harmful "overhead ratio" myth. This contrarian tool argues against that dominant heuristic by reframing efficiency around outcomes and reserves.

What It Does

You search a US charity by name, and it pulls their IRS Form 990 history to generate a plain-English context-aware financial explainer that debunks the overhead myth and shows why a high program-expense ratio can actually indicate starved infrastructure. Instead of the overhead ratio, it surfaces reserve months, operating margin trends, staff-investment share, fundraising cost-per-dollar, and how the org compares to ~70 peers in its category. A Gemini-backed "myth-buster" card writes the narrative (with a one-click regenerate), and a grounded chat box lets you ask follow-up questions about that org's numbers specifically — it only answers from the org's own…

View on GitHub

How I Built It

The interesting part of this build wasn't the UI — it was discovering that the premise of the "overhead ratio" is shakier than I expected, right down at the data layer.

The data problem is the thesis. I went in planning to compute the classic program/management/fundraising expense split and rank charities by it. ProPublica's own API docs say that breakdown is "formtype dependent" across the different 990 variants (990 vs. 990-EZ vs. 990-PF) — meaning you can't reliably compute the overhead ratio from clean, normalized filing data in the first place. That's not a workaround I found around a limitation; it became the actual argument. Instead I built five metrics from fields ProPublica does normalize consistently: reserve months, operating margin, staff-investment share (officer comp + wages + payroll tax as % of spending), fundraising cost per dollar (from the actual "professional fundraising fees" line item), and revenue concentration.

Peer percentiles, built at build-time. To compare an org against similar ones, I sample ~75 organizations per NTEE major category (10 categories total) via ProPublica's search API, compute each one's latest-year metrics, and cache the resulting distributions as static JSON. That means percentile lookups are instant at request time and don't depend on ProPublica's API being up during a demo.

Gemini does the writing, not the arithmetic. The narrative card hands Gemini 2.5 Flash a fully pre-computed set of numbers and a strict JSON response schema — its job is phrasing and framing, never inventing a figure. I hit a real bug here: Gemini 2.5 Flash spends hidden "thinking" tokens out of the same maxOutputTokens budget by default, which was silently truncating longer prompts mid-JSON. Disabling thinkingConfig.thinkingBudget: 0 fixed it instantly — this is pure formatting, not a reasoning task, so there was nothing to think about anyway.

The chat needed the opposite fix. For the free-form chat feature, I first tried the same thinkingBudget: 0 setting and got a subtle but real bug: the model would cite our exact, correct numbers but draw wrong conclusions from them — e.g. claiming a year had a "negative operating margin" when it actually had a +25% margin and falling reserves for an unrelated reason (expenses scaling up faster than reserves). Chat questions require live arithmetic across multiple derived fields, not just rephrasing a precomputed conclusion, so I left thinking enabled there and added explicit guardrails in the prompt about not conflating reserve-month drops with deficits. The chat is scoped to one org's filing data via a dossier built from the same computed metrics, and it's instructed to say "I don't know" rather than invent anything about a charity's programs, leadership, or controversies — none of which is in the data.

Two smaller bugs worth mentioning, because they're the kind that only show up when you actually test the numbers instead of trusting that the code "looks right": ProPublica's fuzzy name search resolved "charity: water" and "ACLU Foundation" to unrelated same-ish-named orgs (I hardcoded the correct EINs after verifying against Charity Navigator), and a percentile field for "fundraising cost" was showing a spurious "100% more efficient than peers" for orgs that report zero professional fundraising fees, because a value of exactly 0 was being treated as a real 0th-percentile cost instead of "not applicable."

Stack: Next.js 16 (App Router) + TypeScript + Tailwind + Recharts, deployed on Vercel, with a small Node build script that fetches and caches the ProPublica data and peer cohorts ahead of time so the demo never depends on a live third-party API mid-presentation.

Prize Categories

Submitting for Best Use of Google AI — Gemini 2.5 Flash powers both the myth-buster narrative generator (strict JSON schema, zero invented numbers, deterministic rubric-engine fallback if the API is ever unavailable) and the grounded multi-turn chat.

Solo submission.

Two AI debuggers found the same bug. Both stopped one function short.

Hassan Imam — Sat, 22 Aug 2026 02:31:33 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

ChatGoogleGenerativeAI computed your Gemini token counts correctly, then threw
half of them away. Any cost tracker built on LangChain's callback system read
zero tokens on every call.

I reproduced it on the current release, fixed it, and shipped
PR #1954 to
langchain-google.

The interesting part: I pointed two AI debugging tools at the same code — Google's
Gemini CLI and Sentry's Seer. Both correctly identified the same function. Seer
even independently validated the trickiest design decision in the patch. Neither
of them found the second half of the fix — and without it, the bug is still there.

Project Overview

langchain-google-genai is
the official LangChain integration for Google's Gemini models — the package you
install to use ChatGoogleGenerativeAI in a LangChain app. It lives in the
langchain-google monorepo alongside the Vertex AI and community packages, and
ships roughly every couple of weeks.

Token counts are not a nice-to-have in that stack. They are how you bill
customers, enforce per-tenant quotas, populate observability dashboards, and
notice that a prompt change quietly tripled your spend. LangChain exposes them in
two places:

AIMessage.usage_metadata — the modern, per-message standard
LLMResult.llm_output["token_usage"] — the older convention that the callback system reads, and that every other chat integration still populates

Callbacks are what production apps hook into, because they fire for every call
without touching business logic. Which is exactly what made this bug expensive:
the counts were right, the callbacks read zero, and nothing raised an error.

Bug Fix or Performance Improvement

The bug

Issue #957 reports
that BaseCallbackHandler.on_llm_end sees no token usage for
ChatGoogleGenerativeAI. The issue was closed with no linked PR and no fix commit,
and the package was rewritten onto the unified google-genai SDK in 4.0.0 — so my
first job was to find out whether the bug still existed at all. In other words: a
bug the tracker considered resolved, in a package that had since been rewritten
from the ground up.

It does. Here it is on langchain-google-genai 4.3.4, the current release:

class TokenProbe(BaseCallbackHandler):
    def on_llm_end(self, response: LLMResult, **kwargs) -> None:
        print("llm_output:", response.llm_output)
        msg = response.generations[0][0].message
        print("message.usage_metadata:", msg.usage_metadata)

llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash", callbacks=[TokenProbe()])
llm.generate([[HumanMessage(content="Say hello in 3 words.")]])

llm_output            : {}
message.usage_metadata: {'input_tokens': 8, 'output_tokens': 592, 'total_tokens': 600}

The counts exist. They're just not where callback-based token tracking looks for
them. Every get_openai_callback-style cost tracker, every LangSmith-style
accounting layer reading llm_output["token_usage"], reports zero for Gemini.

Root cause

It's in libs/genai/langchain_google_genai/chat_models.py, in
_response_to_result. The function builds llm_output at line 1298:

llm_output = (
    {"prompt_feedback": response.prompt_feedback.model_dump()}
    if response.prompt_feedback
    else {}
)

Then it computes the token counts into lc_usage, attaches them to the message
at line 1404:

message.usage_metadata = lc_usage

…and returns at line 1456:

return ChatResult(generations=generations, llm_output=llm_output)

llm_output is never touched in between. grep confirms it: the identifier
appears at exactly two lines in the entire file — where it's created and where
it's returned.

The package migrated to the modern usage_metadata standard on the message and
left the llm_output["token_usage"] convention — which every other LangChain chat
integration still populates, and which the callback API still reads — empty.

Why the obvious fix is incomplete

This is the part both AI tools missed, and it's the reason a one-line patch
doesn't work.

Populating llm_output inside _response_to_result fixes the callback path.
It does not fix the value you get back from generate(). Here's why, from
langchain_core/language_models/chat_models.py:

flattened_outputs = [
    LLMResult(generations=[res.generations], llm_output=res.llm_output)
    for res in results
]
llm_output = self._combine_llm_outputs([res.llm_output for res in results])
output = LLMResult(generations=generations, llm_output=llm_output)
if run_managers:
    for manager, flattened_output in zip(run_managers, flattened_outputs):
        manager.on_llm_end(flattened_output)   # <- gets res.llm_output DIRECTLY
return output                                  # <- built via _combine_llm_outputs

Two different objects. on_llm_end receives flattened_output, built from
res.llm_output directly. The return value is built through
_combine_llm_outputs — whose BaseChatModel default is:

def _combine_llm_outputs(self, _llm_outputs, /) -> dict[str, Any]:
    return {}

ChatGoogleGenerativeAI never overrode it. So patch only
_response_to_result and result.llm_output is still {}.

My reproduction printed both paths, which is the only reason I noticed.

Code

Two changes. First, mirror the counts into llm_output using the same key
convention as the other integrations:

if lc_usage is not None:
    llm_output["token_usage"] = {
        "prompt_tokens": lc_usage["input_tokens"],
        "completion_tokens": lc_usage["output_tokens"],
        "total_tokens": lc_usage["total_tokens"],
    }
    llm_output["usage_metadata"] = lc_usage
if response.model_version:
    llm_output["model_name"] = response.model_version

Note it sources from lc_usage, not the raw response. lc_usage has already had
prev_usage subtracted, so on the streaming path each chunk carries its own delta
— consistent with how message.usage_metadata already behaves.

Second, the override that makes the return value work, mirroring
langchain-openai:

def _combine_llm_outputs(self, llm_outputs, /) -> dict[str, Any]:
    combined: dict[str, Any] = {}
    token_usage: dict[str, int] = {}
    for llm_output in llm_outputs:
        if not llm_output:
            continue
        for key, value in (llm_output.get("token_usage") or {}).items():
            if value is not None:
                token_usage[key] = token_usage.get(key, 0) + value
        if "model_name" in llm_output:
            combined.setdefault("model_name", llm_output["model_name"])
    if token_usage:
        combined["token_usage"] = token_usage
    return combined

After:

llm_output : {'token_usage': {'prompt_tokens': 8, 'completion_tokens': 775,
              'total_tokens': 783},
              'usage_metadata': {...}, 'model_name': 'gemini-2.5-flash'}
RESULT.llm_output : {'model_name': 'gemini-2.5-flash', 'token_usage': {...}}

Deliberately out of scope: streaming aggregation. _response_to_result is
shared with the streaming path, and the new llm_output carries the same
per-chunk delta as message.usage_metadata, so the two stay consistent. Changing
how chunks aggregate is a separate problem with real double-counting risk
(see langchainjs #8266) and deserves its own PR.

My Improvements

Tests and CI

Four unit tests covering both sites plus the no-usage-metadata edge case. They
assert the invariant that matters — that llm_output agrees with the message:

assert result.llm_output["token_usage"]["total_tokens"] == usage_metadata["total_tokens"]

Full suite: 377 passing, no regressions. ruff, ruff format and mypy
clean across all 35 source files. CI green on Python 3.10, 3.11, 3.12, 3.13
and 3.14.

What I took away

Two AI tools, different vendors, different architectures, different access —
Seer had my actual repository — reached the same correct-but-partial answer. Both
found the function where the tokens should be written. Neither traced what
happens to llm_output after _response_to_result returns.

That's not a knock on either. Both were genuinely useful, and Seer's streaming
observation was sharper than I expected. But the gap is instructive: they answered
the question as posed — why is llm_output empty? — and stopped. They didn't
ask the follow-up: does fixing that actually make the value reach the caller?

The reproduction script is what caught it, because it printed both paths and only
one of them changed.

Best Use of Sentry

I ran the instrumented probe against the unpatched fork and against the patched
one, tagging each run before / after.

Error monitoring. A cost-tracker assertion failing on the empty llm_output,
captured across eight runs. Each event carries the package versions and a context
block naming the suspect file and function, so the issue page alone tells you
where to look.

Tracing. The gen_ai.chat span, before and after:

	before	after
Tokens	8 in + 29 out = 37 total	8 in + 741 out = 749 total
Issues on trace	1	0

Agent monitoring and conversation traces. Because the probe ran with
include_prompts=True, Sentry captured the full exchange on the gen_ai.chat
span — prompt, completion, model, latency, cost and token breakdown.

This one span makes the cost argument better than any explanation of mine. The
prompt was "Say hello in 3 words." The response was "Hello, my friend."
Three words. It cost 749 tokens, because gemini-2.5-flash spent 736 of
them reasoning before answering:

'output_token_details': {'reasoning': 736}

That is exactly the kind of spend you cannot eyeball from the response text, and
exactly what token accounting exists to surface. Before this fix, a
BaseCallbackHandler-based cost tracker watching that call reported zero tokens
and zero cost.

And here's where I have to be honest about something.

I expected to write "Sentry showed zero Gemini tokens." It would have been a
great line. It's also false, and my own traces prove it — the token counts are
correct in both runs.

I checked the SDK source before publishing. sentry_sdk/integrations/langchain.py
resolves tokens like this:

possible_names = ("usage", "token_usage", "usage_metadata")

message = _get_value(obj, "message")      # checked FIRST
if message is not None:
    for name in possible_names: ...

llm_output = _get_value(obj, "llm_output")   # only if message had nothing

Sentry checks the message before llm_output, and falls back to
_extract_tokens_from_generations when the LLMResult has nothing. In other
words: Sentry ships a defensive workaround for exactly this class of bug. It
reports Gemini tokens correctly despite the defect, because it doesn't trust
llm_output to be populated.

That's a better story than the one I expected to tell. The fix doesn't repair
Sentry's numbers — it removes the need for the workaround.

Logs. Structured logs recording llm_output on every on_llm_end, empty
before and fully populated after.

Seer. Sentry's AI debugger, connected to my fork. It read chat_models.py
straight from the repo, identified _response_to_result, and produced a
two-step plan. Step 2 said to use the delta-adjusted lc_usage rather than the
raw response values, so streaming chunks stay correct — independently arriving at
the same conclusion I'd flagged for reviewers in the PR description.

Its cited evidence covers chat_models.py L1292–L1489. _combine_llm_outputs
lives at L3370. It never got there.

One practical note for anyone instrumenting LangChain: it swallows exceptions
raised inside callbacks. My first probe asserted inside on_llm_end and nothing
ever reached Sentry. Assert on the returned LLMResult instead.

Best Use of Google AI

Gemini CLI as an agentic bisect. I ran @google/gemini-cli v0.56.0 with
gemini-2.5-flash against the unpatched tree, so the diagnosis couldn't be
contaminated by my fix:

gemini -p "<prompt>" --model gemini-2.5-flash --skip-trust

It found _response_to_result, cited lines 1298–1302 for the initialization,
1404 for message.usage_metadata = lc_usage, and 1456 for the return, then
proposed token_usage / usage_metadata / model_name with the correct key
names — essentially the patch I'd written, at the same insertion point.

It did not mention _combine_llm_outputs.

Gemini API as a correctness oracle. Self-consistency isn't enough — matching
numbers could both be wrong. So I validated the patched llm_output against the
raw google-genai SDK's own prompt_token_count for the same prompt, which is
deterministic for a fixed model:

prompt                                                SDK in   LC in  match   sums
Say hello in 3 words.                                      8       8   PASS   PASS
Name one primary color. Answer with a single word.        12      12   PASS   PASS
Reply with exactly the word: ok                            8       8   PASS   PASS

RESULT: 3/3 passed

bugsmash

Hassan Imam — Wed, 19 Aug 2026 02:15:04 +0000

AccessLens — A persistent on-device visual interpreter for the blind

Hassan Imam — Sun, 24 May 2026 06:46:43 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

AccessLens is an Android app that turns a Pixel 8 worn on a lanyard into a persistent visual interpreter for blind and low-vision users. Rear camera forward, bone-conduction headphones in, the phone describes the world — and remembers.

The problem with existing visual-assist apps (Be My Eyes, Seeing AI, Envision) is that they are screen-bound, stateless, and cloud-bound. A blind person navigates by sound; an app that needs you to hold up a phone, tap a screen, and wait on a datacenter interrupts that signal stream. AccessLens is different on three axes:

Worn, not held: Two physical buttons drive everything. Volume Up → read text in front of me, verbatim. Volume Down → describe this room with memory from earlier today and recent days. A gyroscope-based SettleTrigger also fires a description automatically when the user stops walking.
Persistent memory across days/weeks: Every gesture writes a SessionEvent to a SQLCipher database. A nightly Gemma 4 worker compresses each day into a DailySummary; Sundays roll into a WeeklyMemory. LONG-press prompts splice that history into the Gemma call, so the model has a world model of this specific apartment, this specific day.
100% on-device: No image, audio, embedding, or location leaves the phone. SQLCipher + Android KeyStore (AES-256-GCM wrapping a SecureRandom DB key) protect everything at rest. A SelfTest on first launch opens a probe DB with the wrong key and asserts the read fails before the app reports encryption healthy.

Face recognition uses MediaPipe FaceLandmarker to produce a 192-dim L2-normalized landmark vector per enrolled person. At identify time, cosine-similar matches inject only the names into the Gemma prompt — Gemma never sees a face crop or an embedding, code-review-verified.

Three gestures, three target latencies (Pixel 8, Tensor G3): SINGLE ≤14 s end-to-end, DOUBLE scales with text length, LONG adds memory retrieval. Voice fillers ("I'm looking…", "Still looking…") cover the prefill gap so the user hears acoustic progress, not dead air. Everything runs with airplane mode on after the model is pushed once.

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

An always-on, on-device visual interpreter for blind and low-vision users — built for the DEV.to "Build with Gemma 4" challenge.

Pitch

A phone worn on a lanyard becomes the user's "eyes." The rear camera is always on; the gyroscope watches for motion. When the user stops walking, AccessLens describes what's in front of them. When a friend whose face has been enrolled walks into frame, the phone says their name. When the user wants to read what's in front of them, they press Volume Up; for a richer description of the room, Volume Down. Bluetooth bone-conduction headphones carry the audio — the user's ears stay free for the world.

What separates AccessLens from existing apps like Be My Eyes, Seeing AI, and Envision is persistent on-device memory + 100% on-device inference. Existing tools are stateless and cloud-bound. AccessLens runs Gemma 4 E2B locally via LiteRT-LM, encrypts…

View on GitHub

Apache 2.0. The repo includes the full Kotlin/Compose source, the encryption self-test, the nightly compression WorkManager job, and a README documenting which file enforces each of the six privacy invariants.

Reference implementation that taught me the LiteRT-LM API: google-ai-edge/gallery — adapted patterns are cited inline in inference/LiteRtLmRuntime.kt.

How I Used Gemma 4

Model Chosen: Gemma 4 E2B (litert-community/gemma-4-E2B-it-litert-lm, ~2.59 GB int4), loaded once at service start via LiteRT-LM 0.12.0 with Backend.GPU() for the vision adapter.

This model was the perfect fit for AccessLens for three core reasons:

Multimodal in one model, on-device: Image input goes in as Content.ImageBytes, text as Content.Text, in that order (per the Gallery's "for accurate last token" comment), all through one Engine.generate call. No separate vision encoder + decoder to stitch, no second model to keep resident. That fits the latency budget and the memory budget on Pixel-class 8 GB RAM.
E2B is the smallest competent multimodal Gemma 4: It fits in RAM alongside MediaPipe FaceLandmarker, a CameraX pipeline, and the Compose UI without OOM-ing on a Pixel 8. I prototyped against E4B (the brief's "quality path") and measured the latency lift on one-sentence scene descriptions — not worth doubling the prefill cost for a use case where the user is waiting in real time, lanyard-mounted, with no screen feedback. The architecture is parametric on the model path (InferenceRuntime.load(modelPath, Modality)), so a future LONG-press branch could swap to E4B in one line. I documented the tradeoff in the README and shipped E2B for all three gestures.
Gemma is the only practical way to do nightly memory compression on-device: The 03:00 CompressionWorker calls Gemma in JSON mode to compress the day's SessionEvent rows into a single DailySummary, and on Sundays into a WeeklyMemory. That's a real LLM task — extracting persistent facts, deduplicating recurring observations, distinguishing "the blue mug is mine" from "I saw a blue mug today" — and it has to happen without a network. E2B handles it in under a minute per day on Tensor G3 while the phone is on the charger.

Production fixes discovered during implementation:

The LiteRT-LM Android artifact must be 0.12.0 or later — 0.11.0 fails vision init inside vision_litert_compiled_model_executor.cc:273 on Tensor G3.
AndroidManifest needs <uses-native-library> declarations for libOpenCL.so, libOpenCL-car.so, libOpenCL-pixel.so (all android:required="false"). Without them, Android 12+ silently denies GPU OpenCL access and the vision backend fails to initialize. Documented at ai.google.dev/edge/litert-lm/android.

The thing I'm proudest of: when you uninstall AccessLens, the KeyStore wrapping key is destroyed with it. The encrypted DB on disk becomes cryptographically unrecoverable. The user can throw the phone away and their memories — kitchen layout, friends' faces, places they've been — go with it. That's what on-device privacy is supposed to mean, and Gemma 4 + LiteRT-LM made it possible without compromising the assistant on quality.

AccessLens — a blind person's lanyard, powered by Gemma 4 on-device

Hassan Imam — Sat, 23 May 2026 17:27:35 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

Worn, not held. Two physical buttons drive everything. Volume Up → read text in front of me, verbatim. Volume Down → describe this room with memory from earlier today and recent days. A gyroscope-based SettleTrigger also fires a description automatically when the user stops walking.
Persistent memory across days/weeks. Every gesture writes a SessionEvent to a SQLCipher database. A nightly Gemma 4 worker compresses each day into a DailySummary; Sundays roll into a WeeklyMemory. LONG-press prompts splice that history into the Gemma call, so the model has a world model of this specific apartment, this specific day.
100% on-device. No image, audio, embedding, or location leaves the phone. SQLCipher + Android KeyStore (AES-256-GCM wrapping a SecureRandom DB key) protect everything at rest. A SelfTest on first launch opens a probe DB with the wrong key and asserts the read fails before the app reports encryption healthy.

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

An always-on, on-device visual interpreter for blind and low-vision users — built for the DEV.to "Build with Gemma 4" challenge.

Pitch

View on GitHub

Reference implementation that taught me the LiteRT-LM API: google-ai-edge/gallery — adapted patterns are cited inline in inference/LiteRtLmRuntime.kt.

How I Used Gemma 4

Model: Gemma 4 E2B (litert-community/gemma-4-E2B-it-litert-lm, ~2.59 GB int4), loaded once at service start via LiteRT-LM 0.12.0 with Backend.GPU() for the vision adapter. Three reasons E2B was the right fit:

Multimodal in one model, on-device. Image input goes in as Content.ImageBytes, text as Content.Text, in that order (per the Gallery's "for accurate last token" comment), all through one Engine.generate call. No separate vision encoder + decoder to stitch, no second model to keep resident. That fits the latency budget and the memory budget on Pixel-class 8 GB RAM.
E2B is the smallest competent multimodal Gemma 4. It fits in RAM alongside MediaPipe FaceLandmarker, a CameraX pipeline, and the Compose UI without OOM-ing on a Pixel 8. I prototyped against E4B (the brief's "quality path") and measured the latency lift on one-sentence scene descriptions — not worth doubling the prefill cost for a use case where the user is waiting in real time, lanyard-mounted, with no screen feedback. The architecture is parametric on the model path (InferenceRuntime.load(modelPath, Modality)), so a future LONG-press branch could swap to E4B in one line. I documented the tradeoff in the README and shipped E2B for all three gestures.
Gemma is the only practical way to do nightly memory compression on-device. The 03:00 CompressionWorker calls Gemma in JSON mode to compress the day's SessionEvent rows into a single DailySummary, and on Sundays into a WeeklyMemory. That's a real LLM task — extracting persistent facts, deduplicating recurring observations, distinguishing "the blue mug is mine" from "I saw a blue mug today" — and it has to happen without a network. E2B handles it in under a minute per day on Tensor G3 while the phone is on the charger.

Two production fixes the brief didn't cover, in case they help someone else:

The LiteRT-LM Android artifact must be 0.12.0 or later — 0.11.0 fails vision init inside vision_litert_compiled_model_executor.cc:273 on Tensor G3.
AndroidManifest needs <uses-native-library> declarations for libOpenCL.so, libOpenCL-car.so, libOpenCL-pixel.so (all android:required="false"). Without them, Android 12+ silently denies GPU OpenCL access and the vision backend fails to initialize. Documented at ai.google.dev/edge/litert-lm/android.

DEV Community: Hassan Imam

I Built an AI That Remembers the Promises We Forget

What I Built

Demo

Code

How I Built It

Why Does Open Innovation Matter?

The Lazy Payoff

Prize Categories

Drafts That Never Shipped: do LLMs treat dead proposals as real standards?

What I Benchmarked

Models Tested

Findings

My Benchmark

Lake Lookout: Be the Eyes the Satellites Lack (Gemma 4, fully offline)

What I Built

The lake nobody was watching

How a day works

Who it's for

What it records

Taking it outside: a dry run first

Demo

Code

hassan-2050 / lake-lookout

Hike with your phone. In the evening, a local Gemma 4 turns your photos and voice notes into a glacial-lake field log, offline.

Lake Lookout

How I Built It

Open-source AI at the core

The interesting part: making a small model unable to overclaim

What real photos taught me

Honest limits

What's next

Why Does Open Innovation Matter?

My Agent Session

Prize Categories

The Overhead Ratio Is Lying to You — I Built an AI Tool to Prove It

What I Built

Demo

Code

hassan-2050 / glasspocket

Overhead-ratio myth buster for US charities — Form 990 data via ProPublica, Gemini narrative generator

GlassPocket — Overhead Myth Buster

The Hook

What It Does

How I Built It

Prize Categories

Two AI debuggers found the same bug. Both stopped one function short.

Project Overview

Bug Fix or Performance Improvement

The bug

Root cause

Why the obvious fix is incomplete

Code

My Improvements

Tests and CI

What I took away

Best Use of Sentry

Best Use of Google AI

Links

bugsmash

AccessLens — A persistent on-device visual interpreter for the blind

What I Built

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

Pitch

How I Used Gemma 4

Production fixes discovered during implementation:

AccessLens — a blind person's lanyard, powered by Gemma 4 on-device

What I Built

Demo

Code

hassaninnovate / AccessLens

A blind person's lanyard, powered by Gemma 4 E2B on a Pixel 8. 100% on-device visual assistant with persistent memory and face recognition.

AccessLens

Pitch

How I Used Gemma 4