Japan turning footsteps into Electricity, I turned them into Music.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

I built Footwork, a music experience where going for a walk becomes part of the creative process. Instead of asking you to stare at a screen, it turns the way you move - your pace, pauses, turns, and loops into a short instrumental called a Soundprint.

Using it is intentionally simple:

1. Tap Start walking.
2. Put your phone away and explore.
3. End the walk and listen to what your movement created.

Every walk has its own rhythm. You might slow down to cross a road, make a sudden turn, speed up along an open stretch, or loop back toward somewhere you have already been. Footwork turns moments like these into musical decisions, such as changes in energy, transitions between sections, and brief pauses.

Before any AI processing happens, Footwork removes the location data. It sends only an anonymous timeline of movement events to Gemma, Google's open-weight model. Gemma interprets the shape of the journey and creates a structured musical plan. After that plan is checked and validated, a separate renderer turns it into the final piece of music.

In other words, the walk itself becomes the prompt. You do not have to sit down afterward and explain your experience to an AI. The way you moved already tells the story.

Footwork is for walkers, runners, and anyone curious about hearing an ordinary journey in a completely different way. It is not meant to score your fitness, recommend a route, or give you another reason to keep checking your phone. It is simply a creative reward for putting the screen away and spending some time outside.

Demo

Try the public app: footwork-m6g0.onrender.com

Fast judge path

1. Open the app and choose Hear an example.
2. Press play on the Soundprint.
3. Watch the traced path, movement ribbon, and active moment move together.
4. Jump to the turn near 0:18 and the pause near 0:36 to inspect two different movement-to-music ideas.

The example is prepared in the browser so anyone can evaluate the complete interaction without spending generation credits. It is clearly labeled Example walk and is not presented as a live Gemma or ElevenLabs Music call.

For the real experience, start a walk, keep the tab open, put the phone away, and move for at least 90 seconds of active time. Ending the walk starts the privacy-cleaning, arrangement, and rendering pipeline.

The source is public on GitHub: github.com/kernelKain/footwork

Architecture notes: Docs/architecture.md

flowchart LR
    A[Walk captured in browser] --> B[API cleans the trace]
    B --> C[Detect turn, pause, pace change, return]
    C --> D[Remove coordinates and original clock]
    D --> E[Gemma plans the arrangement]
    E --> F[Validate constrained JSON plan]
    F --> G[Eleven Music renders audio]
    G --> H[Soundprint follows one audio clock]
    C --> I[Route Sketch fallback]

How I Built It

Footwork is built as a series of small, clearly defined steps. I deliberately avoided sending the entire walk to one large, mysterious AI process. Each part of the pipeline has one job, and the next step only begins when the previous one has produced a valid result.

1. Record the walk without demanding attention

The browser records the journey while the user is outside. The recording screen has large controls, pause and resume options, a visible 90-second readiness target, and a hold-to-end button that helps prevent accidental taps.

Once the walk starts, the user should be able to put the phone away and focus on being outside.

When the walk ends, the backend cleans the recorded trace and looks for a few meaningful types of movement:

• sharp turns
• pauses
• changes in pace
• returns near an earlier part of the route

These events become the raw material for the music.

2. Keep the movement, remove the location

Privacy was an important design decision from the beginning. Gemma does not need to know the name of a street or even which country the walk happened in to understand its rhythm.

Before the data is sent to any model provider, the server removes the coordinates and original timestamps. What remains is an anonymous movement timeline containing event IDs, event types, normalized timing, and the approximate duration of the finished piece.

This allows the AI to understand how the walk unfolded without receiving the location where it happened.

3. Ask Gemma to arrange the journey

I use google/gemma-4-E2B-it, running inside a public Hugging Face ZeroGPU Space, as Footwork's musical director.

Gemma is not asked to describe the route or invent a story about where the user went. Its job is much more focused: it studies the anonymous movement events and creates a structured instrumental arrangement.

For example, it can decide that a sharp turn should introduce a new section, a pause should create a musical hold, or a faster stretch should increase the energy. This is where the shape of the walk begins to become the shape of the music.

4. Check Gemma's plan before generating audio

Gemma must return its arrangement as a small, well-defined JSON plan. Footwork validates that plan before making a music-generation request.

The validator checks that Gemma only refers to events that actually occurred during the walk. It also rejects unsupported moods, unknown values, and styles outside Footwork's intentionally limited instrumental vocabulary.

Only a valid arrangement is allowed to continue to Eleven Music. This makes the pipeline more predictable and avoids wasting a music-generation request on a malformed plan.

5. Keep the visuals and music on the same clock

Once the Soundprint is ready, the audio player becomes the single source of time for the whole experience.

The traced path, movement ribbon, selected event, playback progress, and current Soundprint moment all follow the position of the same audio element. There is no separate animation timer running beside it, so the explanation does not slowly drift away from the music.

The hardest parts

Some of the most difficult work happened around the details that users should never have to think about:

• Protecting privacy: I needed to preserve the shape and rhythm of a journey while removing the actual place where it happened.
• Handling failure honestly: a demo track, a Route Sketch, and a generated Soundprint must always be clearly distinguishable.
• Making retries safe: if a music request times out, Footwork cannot blindly send it again because the original request may already have consumed paid generation quota.
• Keeping everything synchronized: the path, movement events, and explanations must remain aligned when the user plays, pauses, seeks, or leaves the tab.
• Designing for outdoor use: important actions need large touch targets, strong contrast, minimal reading, and reduced-motion alternatives.

The result is a pipeline where each part can be inspected, tested, and replaced independently—while the experience itself remains simple: take a walk, put the phone away, and come back to a piece of music shaped by the journey.

Results

Footwork's result is more than an audio file. It is a synchronized explanation of how the walk became the piece:

• Soundprint — the finished playable artifact, pairing the instrumental with the walk's detected moments so movement and music remain readable as one result.
  

• Traced path — the cleaned route outline preserves the shape of the journey while model providers receive anonymized movement events instead of coordinates.
  

• Movement ribbon — a compact, time-ordered view of turns, pauses, pace changes, and returns. During playback, it shows which movement event is shaping the current moment.
  

• Journey pattern — the walk's larger structure—where motion changed, stopped, accelerated, or returned becomes the compositional plan rather than incidental activity data.
  

• Synchronized playback — the traced path, movement ribbon, and active Soundprint moment all follow one audio clock, keeping the explanation aligned with what the listener hears.

• Graceful fallback — if model planning or music generation fails, the same event timeline still produces a clearly labeled Route Sketch.

Gemma on Hugging Face

Gemma is the decision-making center of Footwork, not a decorative text generator.

• Model: google/gemma-4-E2B-it
• Space: kernelKain/footwork
• Runtime: public Gradio Space on free Hugging Face ZeroGPU (zero-a10g)

The Space exposes one narrow /arrange job. It receives turns, pauses, pace changes, returns, normalized timing, and duration but never coordinates or the original clock. Thinking output is disabled, and the model is asked for one bounded JSON arrangement.

A checked call returned a validated plan from the pinned model:

{
  "status": "valid",
  "model_id": "google/gemma-4-E2B-it",
  "arrangement": {
    "mood": "warm_cinematic",
    "style": ["instrumental", "warm", "cinematic", "rising"],
    "event_refs": ["turn-1"]
  }
}

The response is accepted only when its mood is warm_cinematic, its style stays within the closed instrumental vocabulary, and every referenced event exists in the submitted timeline. Invalid or unavailable responses remain blocked and fall back to Route Sketch. The Space never invents a successful plan, and only a validated arrangement can be sent to Eleven Music.

This separation is the architecture: Gemma interprets the journey, Footwork validates that interpretation, and Eleven Music renders the accepted plan.

Why Does Open Innovation Matter?

Open innovation matters here because the model controls the part of Footwork that makes the project more than a GPS visualizer: how a journey becomes musical structure.

The planner is inspectable

I can show the exact model, pin its revision, inspect the prompt and response contract, and prove which movement event IDs were accepted. With a closed route-to-music service, the most important creative decision would be hidden behind an endpoint.

The privacy boundary belongs to the project

Footwork decides what the model is allowed to know. Gemma receives turn-1 at normalized time 0.31; it does not receive a street, a coordinate, or a wall-clock timestamp. Because the planning layer is open and separately hosted, that boundary is visible in the code and testable in isolation.

The model can move or change without rewriting the product

The planner speaks a small JSON contract. I can tighten the vocabulary, compare a compatible checkpoint, or move inference closer to the device while keeping the capture, validation, renderer, and Soundprint interface intact. Footwork does not run fully offline today, so I am not claiming that benefit yet but the open-weight core keeps that route available.

Open inference keeps experimentation accessible

The checkpoint runs on a free Hugging Face ZeroGPU Space. That made it possible to test a real, pinned model and expose the planning service publicly without building the experience around paid planning calls. Specialized audio rendering is used only after the open planner has made and passed the decision.

A closed end-to-end service could produce audio, but it would weaken the two ideas I care about most: an enforceable location boundary and a replaceable, inspectable relationship between movement and music.

While shaping the submission, I compared the approach with two other outdoor-first community entries: WanderInk by @subhamduary and Paper Trail by @synthreaper. Both reinforced the value of making the device secondary to the outside experience, while Footwork takes a different path: movement is not context for generated content; it is the score itself.

My Agent Session

The agent-assisted development sessions for Footwork are tracked with Entire:

View the Footwork agent sessions on Entire

Together with the repository, these sessions provide a reviewable trail of the agent-assisted work behind the build.

Prize Categories

• Best Use of Gemma — Gemma is the musical director. It converts an anonymous movement timeline into the constrained arrangement that defines the generated piece; without that plan, the studio path does not proceed.

• Best Use of Render — Render hosts the public experience

• Best Use of ElevenLabs — Eleven Music renders the validated instrumental plan into the final audio

• Best Use of Entire — Provide a reviewable trail of the agent-assisted decisions, implementation steps, and verification work behind the project.

• Best Use of Copilot — For debugging purposes after code review by Code Rabbit.