This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
What I Built
"Every evening I walk past a road on my campus that has been dug up for weeks, with red poles and tape across it. Everyone walks around it; nobody reports it."
Everyone in an Indian city knows the footpath problems on their street. Almost nobody reports them, because reporting is tedious: stop, take a photo, open a portal, pick a category, type a description, pin a location, repeat. So complaints, when they happen at all, sound like "the roads near my house are bad", and nothing gets fixed.
Kerbside turns a walk into a complaint report. You walk your street with your phone in your pocket, saying what you see out loud:
"I can see some red poles here for a barricade."
"The road is not fully built, some stones here and there."
"Huge rain tree here, nice shade."
Your phone records a voice memo and a GPS track with free apps. Back home, Kerbside runs entirely on your laptop, with open models only, and turns them into:
- a map of every problem (and every good thing) at the spot you mentioned it,
- a one-page report for a ward office or a complaint portal, most severe first, each item with a map link and the resident's own words,
- a spreadsheet for portals and groups that want a table.
A person reviews everything before it goes anywhere. Kerbside saves you the hour of typing; it doesn't file complaints on its own.
How it touches grass: most touch-grass apps get you outside. Kerbside makes your walk useful to everyone who walks that street after you. The only input it accepts is a walk, so there is no way to use it from your desk, and the screen comes after you've been outside, for a few minutes of review.
Who it's for: residents, resident welfare associations, students and NGOs who do walkability audits. Not everyone needs to install it: residents only record on their phones, and one organiser with a laptop processes the walks and shares the map link.
Demo
Live map of my walk: https://thebhavye.github.io/kerbside/demo/map.html (open it on your phone)
This is my real 12-minute evening walk on campus on 11 October: a voice memo and a GPS track from my iPhone, processed on my laptop, then reviewed by me.
Before review: Gemma's 11 draft findings, with my words under each one.
After review: 6 real problems left.
A finding from the generated report:
Road dug up for construction and barricaded with tape, on the road by Just Bake and Enzo. · moderate (2/3) · 18:30 · map
Resident's words: "I can see some red poles here for a barricade. So it looks like some construction is going on. Okay, there is a full construction going on. They have put some tapes like barricades."
On the phone map, tap a circle and then Copy complaint text to paste a ready-made complaint into WhatsApp or a portal.
Code
thebhavye
/
kerbside
Walk your street, talk about what's broken, get a map. Runs fully on your laptop with Whisper and Gemma.
Kerbside
Walk your street, talk about what's broken, get a map. Everything runs on your own laptop.
Live demo map · demo report (a real 12-minute evening walk on a college campus: voice memo and GPS from an iPhone, processed by Whisper and Gemma on a laptop, then reviewed by the walker. Kerbside drafted 11 findings; review removed 5 mishearings and corrected the rest.)
You walk your street with your iPhone in your pocket, recording a voice memo and a GPS track. Kerbside transcribes what you said with faster-whisper, uses Gemma running in Ollama to pick out the street problems you mentioned ("footpath broken here", "drain blocked", "streetlight not working") and the good things ("huge rain tree, nice shade"), places each one on your walked route, and gives you:
-
map.html: a phone-friendly map of everything you found, with a place search box -
report.md: a one-page report ready…
MIT licensed. Python, no cloud services, no API keys.
How I Built It
voice memo (.m4a) ─┐
├─► 1. work out when the recording started
GPS track (.gpx) ──┘ 2. faster-whisper: speech → timestamped sentences
3. each sentence → a point on the GPS track
4. Gemma 3 (4B) in Ollama: sentences → findings
5. map + report + GeoJSON + CSV
6. a person reviews and edits, then shares
Everything runs on a Windows laptop with an Intel i5, 16 GB of RAM and no GPU, either from the command line or from a small local web app (python kerbside.py --app) where you drop in the two files, watch progress, and review findings on a map. The app only listens on 127.0.0.1, so nothing leaves the machine.
Speech: faster-whisper on CPU
faster-whisper runs the small model in int8 on the CPU and skips the silence between observations. An 8.5-minute recording transcribes in about 30 seconds.
One bug taught me something. With silence skipping on, Whisper placed one sentence 66 seconds before it was spoken, because after a long pause a segment's start can snap back to the end of the previous speech. At walking pace that's about 80 metres of error. Taking each sentence's time from its first word (word_timestamps=True) fixed it.
Each sentence's time is then matched to the GPS track. If the nearest GPS fix is more than 30 seconds away, the finding is flagged "location approximate" and drawn with a dashed outline, so the reviewer checks it first.
Gemma: from "list the findings" to "label every line"
This is where most of the work went. Gemma 3 4B runs in Ollama with structured outputs: the request includes a JSON schema, so the model can only answer with valid categories and severities.
My first version asked Gemma for a list of findings from each chunk of 12 sentences. On a held-out set it lumped three unrelated lines into one finding and skipped others, and two identical runs gave different answers. A small model given a list to fill in will happily skip lines.
So I changed the question. Gemma now labels every single line: a problem category, a good thing, "same as previous" (a follow-up like "someone could trip on this") or "not an observation" ("ok, turning left"). The code merges follow-ups into one finding. Three details that each fixed a real failure:
- Numbering lines 1..n in every chunk. Gemma kept renumbering lines 13–15 as 1–3, and the code threw away correct answers.
- Summary before severity in the schema. Ollama generates fields in order, so the model describes the problem before rating it.
- A severity floor in code. Gemma kept rating an open drain as "moderate". Open drains, missing covers, manholes and holes are now always "severe". A fixed rule is more predictable than more prompt tuning.
What my first real walk taught me
All of that was tested on scripted walks. Then I did a real one, and it broke in a way my tests never did.
Real people don't only name problems. They narrate: "there is a lot of greenery on this road", "on the left I have the back side of a shop, where you can buy chips", "I see a tennis court". On my 12-minute walk, Gemma produced 47 findings, and about 40 of them were scenery.
Adding more "don't count this" rules to the prompt barely helped. What worked was making Gemma commit to a simple decision first: for every line it now says what kind of line it is (a problem, praise, a description of the surroundings, or chatter) before it labels it, and the code drops any category attached to a description or chatter line. Two more rules moved from the prompt into plain code: a line right after a finding that names no new street thing ("about two feet wide", "full construction going on") joins that finding, including across chunk boundaries, where my unfinished-road finding had been split in two.
The same walk now gives 11 findings. Six are real: construction material, the barricaded road, cut trees, the unfinished road. The other five came from Whisper mishearing me: "volleyball court" became "vulnerable court", and a building I named became "a chair block", which Gemma reasonably reported as a chair blocking the footpath. In the review screen I deleted those, fixed the wording, and moved two findings to where I actually saw them. 47 → 11 → 6. That's the split I want: AI drafts, the person who walked there decides.
My GPS track taught me something too. The tracker app didn't have background location permission, so it stopped when my phone locked and logged only the first and last minute of the walk. That's why the report says 143 m walked and most points are marked "location approximate". Kerbside flagged every guess instead of quietly inventing positions, and the README now tells you to set location access to "Always" before you walk.
Measuring it
An evaluation script runs real Gemma on test sets with expected answers: my sample walk, a held-out set I never tuned against, split sentences, and my real walk. On the final version the three scripted sets get every category right, with nothing missed and nothing extra. Severity is sometimes one level off, and Gemma occasionally treats a detail like "pole number seventeen" as chatter, which is why there's a review step.
Speed on the i5: Gemma takes about 1–2 minutes per 12 sentences, so my 96-sentence walk took about 12 minutes. Kerbside shows a time estimate and unloads Gemma when it's done, to give the RAM back. (Lesson learned the hard way: ten forgotten background processes holding 15 GB of memory will make any local model crawl.)
Why Does Open Innovation Matter?
Because a voice recording plus a GPS track is a map of your life. It shows where you live, when you leave home and the route you walk, in your own voice. I would not ask a neighbour to upload that to a company's API to report a pothole. With open-weight models, Kerbside never has to: Whisper and Gemma run on the laptop, and only the findings you've reviewed get shared. (For this post, my walk's full transcript stays off the public repository; only the six reviewed findings are public.)
Open models also make it free to run, forever. A residents' association has no budget for API credits, and a tool that stops working when a free tier ends is worse than no tool.
And open means adaptable. The categories, severity rules and prompt live in one readable Python file, so a group in another city can add "stray cattle" or "no ramp for wheelchairs" without asking anyone's permission. The output is open data (GeoJSON and CSV), not locked in someone's dashboard.
What's Next
- Fully in the browser: small Gemma and Whisper models can already run inside a web page on the visitor's own device. Same privacy, zero install.
- Ward audits: several residents walk different streets, and one organiser combines their reviewed findings into one ward map and one joint report.
- Before and after: re-walk the same street a month later and get a report of what was fixed and what's still broken. Accountability is the missing half of civic complaints.
My Agent Session
I built Kerbside with the help of an AI coding assistant (Claude Code), which helped me spped up coding while I walked, recorded, tested on my own laptop and phone, and reviewed the results. The design decisions and this write-up's claims are mine to stand behind.
Prize Categories
Best Use of Gemma. Gemma 3 4B, running locally in Ollama, is the part of Kerbside that understands what the resident said. For every transcribed sentence it decides what kind of line it is, labels it with a category, writes the complaint summary and rates severity, all through Ollama's structured outputs so its answers always fit the schema. Per-line labelling, the "decide the kind first" step, the evaluation sets including a real walk, and the deterministic rules around the model all came from making a 4B model reliable enough for a civic report on a CPU-only laptop.
Top comments (0)