Ground Truth Walks: I Asked a Local AI What I'd See. Then I Went Outside

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

My routine is usually the gym, then hours at my desk. I can spend so much time working that even my own neighborhood becomes somewhere I pass through without really noticing.

For this Hacktoberfest challenge, I gave myself a reason to change that. Before each 15-minute walk, the plan was that I and a local AI model, Gemma 3 4B running offline on my laptop, would both predict what I'd encounter. (I didn't follow that order perfectly every time; more on that below.) Then I walked, phone in my pocket, and came back to see who was closer to reality.

I called it Ground Truth Walks. "Ground truth" is the machine-learning term for the real answer. I went and walked on the ground to get mine.

Each prediction covers six things: motorcycles that pass me, dogs I see, distinct bird sounds, people selling something, people who greet me, and whether I hear a generator. It's a small Rust command-line tool. It saves my prediction first, asks Gemma at three levels of context, doesn't show me the model's numbers until the walk is over, and then scores everybody.

I expected to learn how well a small model could imagine my street. What I didn't expect was how much of my own neighbourhood I hadn't explored. Every walk, including my trial run, took me through corners I'd never visited before. Some streets had so many turns that I realised a first-time visitor could easily lose their way without paying attention.

Demo

Code

Lymah123 / ground-truth-walks

Predict a walk with a local Gemma model, then check it against reality.

Ground Truth Walks

A small command-line experiment. Before a 15-minute walk, you and a local open-weight model (Gemma 3 4B via Ollama) both predict what you will encounter. You walk with your phone in your pocket. Afterwards you record what really happened, and the code scores everyone against reality.

The model is asked at three context levels (L1 generic, L2 city, L3 detailed). Your own prediction is saved to disk before the model is ever called.

Rules that protect the experiment

1. The six fields never change between walks. Surprises (goats, hens) go in a free-text note and are never scored.
2. Your prediction is saved before the model runs.
3. The CLI does not display model predictions untilthe walk is scored (they are stored in the walk's JSON file)
4. Reality cannot be recorded until all three model predictions exist.
5. Each model call is a separate stateless request, temperature 0, seed 42.
6. Walk…

View on GitHub

My own walk data stays on my machine. The repo ships a sample walk so you can try the scoring yourself.

How I Built It

• The model: Gemma 3 4B through Ollama, on my CPU. No GPU, no cloud.
• Three levels of context: L1 is a generic prompt, L2 adds the city, and L3 describes the place, the day and the time. The answer has the same JSON shape every time, and nothing tells the model it's part of an experiment.
• Fixed settings: every call is a separate request at temperature 0 with seed 42, so the model can't see its earlier answers. On my machine, the identical L1 and L2 prompts returned identical outputs on every recorded walk. It's slow on a CPU because the three predictions run one after another.
• Rules I set before the first walk: the six fields never change; my prediction is saved before the model runs; the CLI doesn't display the model's numbers until scoring; reality can't be recorded until all three predictions exist; surprises (goats, hens, a bridge) go in a note and are never scored.
• Rust does the validation and scoring. If the model returns something unusable, that's an error, not a quiet zero.

What Went Wrong

An experiment about leaving the screen turned out to be an experiment in how many ways I could break my own rules.

• My trial walk wasn't clean. I ran Gemma after I knew what I'd seen, and with default randomness. Earlier, when I repeated one prompt while testing the idea, it said 15, 20 and then 18 motorcycles. After that I fixed the settings, and the trial walk is left out of the results.
• I counted the wrong thing for birds. The field was distinct bird sounds, but on walk 001 I counted every bird I saw: 20. I could hear many distinct bird sounds, but I hadn't tallied the exact number. Typing 20 would have quietly changed the definition, so I left the field out of scoring for that walk. The tool now lets me skip anything I can't count properly. From then on, I scored the field only with a real tally: 10 on walk 002 and 14 on walk 003.
• I pasted a block of commands, and the tool called me out. The record command started straight after new, before I'd left the house. It noticed that no time had passed. Now it makes me type WALKED before it accepts results that early.
• I forgot to run the tool before walk 002. I'd written my prediction down at 4 am, before the walk, but Gemma's predictions were generated afterwards. The model can't see reality, and its L1 and L2 answers matched walk 001's exactly, so I counted the walk. But the point of the protocol was to prove the order of events, and this time the tool couldn't.
• The overall score misled me. Motorcycles are counted in the dozens and everything else in single digits, so the average was almost entirely a motorcycle score. I report each field separately now.

What the Walks Showed

It was a nice walk, I must say, from someone whose routine is the gym and her desk.

Three walks, all on weekdays around 5 pm. On walks 001 and 003 the full protocol held: my prediction and all three Gemma predictions were saved before I left. On walk 002 I had written my own prediction down beforehand, but Gemma ran after I got back. So I show the results both ways.

Strict comparison: walks 001 and 003 only (mean absolute error per field; lower is better)

Field	Me	L1	L2	L3	Walks scored
Motorcycles	5.00	44.00	37.00	37.00	2
Dogs	0.00	0.00	2.00	3.00	2
Bird sounds	1.00	12.00	9.00	11.00	1
People selling	9.50	12.50	6.50	8.50	2
People who greet me	1.50	1.50	1.50	1.50	2
Generator noise	0.00	0.00	1.00	1.00	2

Bird sounds were skipped on walk 001 (I had counted individual birds, not distinct sounds), so that row rests on one walk.

Exploratory sensitivity check: all three walks, including walk 002

Field	Me	L1	L2	L3	Walks scored
Motorcycles	4.67	44.33	37.33	37.33	3
Dogs	0.00	0.33	2.33	3.00	3
Bird sounds	3.50	10.00	7.00	8.50	2
People selling	6.67	11.33	5.33	8.00	3
People who greet me	2.00	2.00	1.00	1.33	3
Generator noise	0.00	0.00	1.00	1.00	3

Including walk 002 changes the apparent winner on two fields: my prediction pulls ahead of L1 on dogs, and Gemma's city-only prompt (L2) becomes closest on greetings. Because Gemma's walk 002 predictions were generated after I got back, read this table as a sensitivity check, not the main comparison. Two or three walks are too few to say that any predictor is generally better.

• It wasn't "human beats AI". In the strict comparison, I was closest on traffic and birds, and level with L1 on dogs. Gemma's city-only prompt (L2) was closest on people selling, and everyone was level on greetings.
• Gemma badly underestimated traffic. It predicted 1 to 8 motorcycles on streets that had 44 to 46.
• More context barely moved it. L3 predicted 8 motorcycles on all three walks, even though I described three different places, and said "generator: yes" every time, when there was no generator on any of them.
• I underestimated people selling every single time (errors of 11, 1 and 8).
• I could learn between walks, and Gemma couldn't. By the later walks I had seen earlier results, and each Gemma call starts from nothing. That gave me an informational advantage, so this was never a symmetrical contest.

What my six fields didn't capture: four hens, two goats and a lizard on one walk; an area transformer; a building store overflowing with water; carpenters working at the very top of a roof; a dam-bridge covered in bush; palm, coconut and banana trees full of ripe fruit; lots of butterflies; and a block-making industry, not advanced industry though, lol. On one walk, a car passed me too close, and the driver stopped to apologize because the road was too narrow. I told him it was fine.

After walk 003 I wrote down my reaction in one line: "The place looks too urban and yet kinda busy at the same time." On walk 002 I noticed that the area I'd walked the day before had more birds.

Limits. Three walks is a description, not proof. They were at similar times of day, on weekdays, with one model, and the places differed, so L3 mixes place with context. Counting while walking is error-prone. And the bird field is scored on two of the three walks, and on one in the strict comparison.

Why Does Open Innovation Matter?

• Repeatable settings. On my machine, identical L1 and L2 prompts returned identical predictions across the recorded runs with temperature 0 and seed 42. Those prompts don't contain street-specific details, so they can't distinguish between the places I visited. L3 adds a place description, but its motorcycle prediction remained eight across the three logged runs. That's an observation from this setup, not a guarantee about every machine or model version.
• Privacy. The experiment never sends my routes, times or routine to a third-party AI service. I tested it with my Wi-Fi switched off, and it still produced all three predictions. The only internet I needed was the one-time download of the model.
• A swappable part. The model is one component. I only tested Gemma 3 4B, and trying other open models on the same walks is the obvious next experiment.

What's Next

More walks in the mornings and evenings, other open models on the same walks, and a better way to count bird sounds.

Closing thought

I started this project wanting to find out whether an offline AI could predict my neighbourhood. I also ended up noticing parts of that neighbourhood I'd been missing. The experiment didn't need the model to understand my world perfectly. It just needed to give me a reason to go outside and pay attention to it.

Prize Categories

Best Use of Gemma