Footnotes: your walk, as a beat

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

Most of us walk the same streets every day and stop hearing them. Footsteps, traffic, a door, a dog two houses down: it all becomes background. I wanted a reason to listen again, and a way to bring a walk home as something you would actually play twice.

What I Built

Footnotes turns a walk into a beat.

Press Record a walk and go outside. The phone listens while you walk. When you come home, the real sounds you passed become a 30-second beat, on a rhythm measured from the walk itself. Then you hear that beat as three beatboxes side by side, while a pair of neon lips beatboxes along.

Other apps tell you what you heard outside. Footnotes makes music out of it. And it only works if you go out: the screen is just where the walk comes back to you. A trip to the shop is enough; you do not need birds or a view.

Demo

Live app (Android Chrome): https://gracelee087.github.io/footnotes/

No walk at hand? The home screen has two short samples. Their sounds are AI-generated with ElevenLabs (labelled in the app), and their Gemma 4 beat and beatboxes are saved, so trying them costs nothing.

Taking it outside

I tested Footnotes for two days, on walks and on my way to school, with the phone in my pocket.

Every step became fun. Knowing the walk would come back as a beat, I started listening on purpose, and every small sound felt worth keeping. My senses woke up on a route I thought I already knew.

Code

gracelee087 / footnotes

Your walk, as a beat. YAMNet listens on the phone, Gemma 4 makes the beat, ElevenLabs beatboxes it, DigitalOcean runs the server.

Footnotes

Your walk, as a beat. Record the street, and Footnotes turns the real sounds you brought home into a 30-second beat, then into three beatboxes, while neon lips beatbox along.

• Live app (Android Chrome): https://gracelee087.github.io/footnotes/
• Demo video: https://www.youtube.com/watch?v=rgViikcZVBI

Built for the Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass (repository started 6 October 2026).

How it works

1. Record. Tap Record a walk and go outside. The open model YAMNet listens on the phone and keeps one real snippet of each sound it hears (footsteps, cars, water, dogs…). The recording never leaves the phone.
2. Compress. The app measures the walk's own rhythm (tempo, loud and soft hits on a 16-step bar). Gemma 4 names the walk and makes the beat: which real snippets play it and the rhythm, with a one-line reason on the beat card. Recorded speech is never used as a beat sound.
3. Beatbox. The same beat as…

View on GitHub

How I Built It

Three steps, all automatic: Record → Compress → Beatbox.

1. Listen, on the phone. The browser records the microphone with voice processing turned off, so the street sounds like the street. At the same time the open model YAMNet, running in the browser through MediaPipe, recognises the sound every half second. I mapped its classes into 46 everyday sound types (footsteps, doors, dogs, water, bike bells…), and for each type the app keeps one real snippet. The recording never leaves the phone.

2. Compress, with Gemma 4. The app finds where the walk hit hard or soft and folds those hits onto one 16-step bar: the hardest become kicks, then snares, then hi-hats. Gemma 4 names the walk and makes the beat: it picks which real snippets play it, and the beat card shows why, for example "Gemma: Steady steps provide a pulse while splashing water offers a calm contrast." Speech is never used as a beat sound.

3. Beatbox, three ways.

• Human mouth: every hit replayed by a real recorded mouth sound. No generative AI.
• ElevenLabs Clean and ElevenLabs Deep: ElevenLabs Music, given only numbers: tempo, length and which steps carry the kick, snare and hi-hat.
• Surprise mix: the mouth takes cut and remixed on the beat grid.

Each card shows a Rhythm match and a Loud/soft match, so you can hear and see how closely each beatbox follows your walk.

The lips are real. I traced a human beatboxer frame by frame with MediaPipe Face Landmarker. Only the points ship, no video. On each hit the lips replay a real kick, snare or hi-hat.

The partners

• Gemma 4 (gemma-4-26b-a4b-it, via the Gemini API): names every walk (a short title and one line) and makes its beat: it picks 2 to 4 of the real snippets, a rhythm style for walks without a clear rhythm, and writes the one-line reason on the beat card. It sees only sound names and counts, never audio.
• ElevenLabs: three features. Music API makes the two AI beatboxes from the walk's numbers only: Clean with a composition plan, Deep with a prompt. Sound Effects made the sounds of the two sample walks. Text to Speech narrates the demo video.
• DigitalOcean App Platform: one small Node server, deployed from the GitHub repo. It holds the Gemini and ElevenLabs keys so the phone never does, accepts requests only from the app's page, runs every Gemma 4 and ElevenLabs call, and keeps a shared daily cap (the home screen shows how many walks are left today).

Open source used

Piece	Role	License
YAMNet	Recognises sounds on the phone	Apache-2.0
MediaPipe Tasks Audio	Runs YAMNet in the browser	Apache-2.0
MediaPipe Face Landmarker	Traced the lip motion (offline, once)	Apache-2.0
Gemma 4	Names the walk and makes the beat	Gemma license (open weights)
Mouth sounds by giddster and ipaghost (Freesound)	The human-mouth beatbox	CC0
Spencer X, One Beatboxer, 20 Beatbox Snares	Source of the lip motion	CC BY 3.0
Footnotes	Everything else	MIT

Why Does Open Innovation Matter?

Because the recording stays in your pocket. A walk recording is private: it catches voices, your street, your routine. An open model small enough to run in the browser means the listening happens on the phone, and only numbers leave it. A closed audio API would need the whole recording just to tell you a dog barked.

Because it is checkable. The rhythm, the snippets and the match scores are plain code anyone can read and change. The open core (YAMNet, the beat, the human-mouth beatbox, the lips) works without an account.

To be honest about the rest: ElevenLabs is a paid closed service, and Gemma runs through Google's API rather than on the phone.

My Agent Session

I built Footnotes in one week with Claude Code as my coding agent. The repo is the record: 42 commits between Oct 5 and Oct 11, 36 of them co-authored with Claude.

The commit history shows how the idea changed along the way:

• Day 1: a walk became a written score, with each kind of sound mapped to an instrument, then a short beat made from the real snippets.
• Day 2: beatboxes arrived to perform the beat, and the neon lips were traced from a real beatboxer.
• Day 3: Gemma 4 started naming each walk, and the server moved to DigitalOcean.
• Final days: Gemma 4 began picking the beat's sounds and rhythm, and I removed a remix feature ("Drop it, Gemma!") that the app no longer used.

I brought the idea and tested it on real walks; Claude Code and I wrote the code together.

Prize Categories

• Best Use of Gemma: Gemma 4 names every walk and makes its beat, with a one-line reason on the beat card.
• Best Use of DigitalOcean: the server on App Platform holds the keys and runs every Gemma 4 and ElevenLabs call.
• Best Use of ElevenLabs: two AI beatboxes, the sample sounds and the demo narration.

Thanks for reading. If you try it, take it somewhere ordinary: a bus stop, a market, your own street. Then see what beat comes back.