The model heard the ocean. My demo was silent.

This is a submission for the MLH x DEV Writing Challenge. Built during MLH and DEV's Hacktoberfest Week 1: Touch Grass.

What I Built

YAMNet had an answer: Ocean, 0.721. The interface offered a water-themed listening exercise. The countdown worked. The shoreline itself was silent.

While recording SoundWalk's browser demo, native audio playback returned MEDIA_ELEMENT_ERROR: Format error. The same WAV had already decoded successfully for local inference. Two consumers of one recording had taken different paths through the browser.

That small failure became the most useful lesson from my build.

SoundWalk turns five seconds of audio into a sixty-second listening invitation. An open model suggests a direction; the listener chooses what deserves attention. For water, the prompt is: “Find one small sound inside the flow.” After a minute, you save a card with your own observation.

The microphone stops after capture. The model runs locally. Once the offline pack is ready, the app can travel outside with you.

I also built a shoreline rehearsal for people trying it at a desk. That route had a very concrete job: let someone hear what the invitation was talking about.

Demo

Try SoundWalk → · Source and setup →

The 77-second walkthrough shows the repaired rehearsal: local inference, a listening minute, a saved observation, and PNG export. The timer's middle section is edited out; the observation is scripted demonstration text.

The audio is HerbertBoland's “Branding_kort.wav”, via ESC-10, under CC BY 3.0. Its five-second segment loops through the rehearsal. The saved card keeps the sample label.

Partner Technologies

One recording, two browser paths

TensorFlow.js runs YAMNet in a Web Worker, using the CPU backend. The app prepares mono audio at 16 kHz and turns scores across 521 classes into a handful of listening directions.

Inference began with fetch() and decoded the returned bytes through Web Audio. Playback began with a URL assigned to an HTML media element. The cached WAV worked through the first path and failed through the second in the recording environment.

The repair was to give both paths the same complete local recording. This is the central sequence, shortened from analyzeSample():

const response = await fetch('./audio/sea-waves.wav');
if (!response.ok) throw new Error('The sample download failed.');
const bytes = await response.arrayBuffer();

const url = URL.createObjectURL(
  new Blob([bytes], { type: 'audio/wav' })
);
sampleAudio.src = url;
sampleAudio.load();

const context = new OfflineAudioContext(1, 1, 16_000);
const decoded = await context.decodeAudioData(bytes);

The Blob gives native playback a local source, independent of HTTP byte-range delivery. Creating it before decoding also preserves the recording for both consumers. The full implementation revokes the previous object URL when preparing another sample.

Playback then advanced through the full minute. After an offline reload, local inference and sample playback worked together again.

I also connected playback failure to the activity itself. A rejected play() pauses the timer and shows Resume. Injecting that failure, then restoring playback, exercised the recovery path. The listening clock now follows the availability of the sound.

Package the experience with the model

The first preparation downloads about 18 MB: application, TensorFlow.js, model graph, weight shards, labels, and sample. A service worker caches that pack. The footer announces “Ready for an offline soundwalk” after preparation completes.

GitHub Actions runs audio tests, checks required assets and licensing provenance, verifies offline dependencies, and deploys to GitHub Pages. This gave the open model a repeatable route from repository to browser.

The validation notes record the environment: Chromium 153 on Linux, TensorFlow.js 4.22.0, CPU inference. The first sample inference took 4.23 seconds. Browser automation checked the full minute, saved-card persistence, and image export. The offline microphone run used Chromium's file-backed audio simulation.

Hackathon Experience

I worked through the online Touch Grass challenge as a solo build, with Codex, OpenAI's coding agent, producing the implementation and this article. The event's prompt shaped a deliberate handoff: the model supplies a starting point, then the person takes over.

Recording the demo changed my definition of a finished rehearsal. A visible classification, a running timer, and an audible sample each needed their own observation. Following the whole sequence exposed the gap between them.

The repair was small enough to explain in a few lines. Its effect reached the first-time experience: a visitor could press the shoreline button, hear the waves, and understand the invitation.

That is the practice I am taking into the next build: keep a repeatable sample, and follow it all the way to the experience you promise.

Try the shoreline with sound on. Then take a minute in your own surroundings. What did you choose to listen to, and what did you notice?