DEV Community: Kanishq Sharma

Listening Loop: three stops, six seconds, and an open model that stays on your phone

Kanishq Sharma — Tue, 06 Oct 2026 12:27:52 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What I Built

Listening Loop is a field notebook for your ears: choose three places to pause, record six seconds at each, and write down what you actually noticed. An open audio model suggests sounds, then the app asks you to put the phone away and walk somewhere different.

The design question was: how little screen time does a useful AI walking app need? I chose a short loop with no map, score, streak or infinite feed. A doorway, a green corner and the way home are enough. You can adapt those stops to your neighbourhood and mobility.

The model can help you notice a sound you would otherwise pass by. It cannot tell you whether a route is safe, identify a bird species or replace your own observations. The journal deliberately asks: What did you actually notice?

Demo

Open Listening Loop

Press Prepare for a walk while online. Wait for the offline-readiness message; then the app and its model can run without a connection. The initial download is about 19 MB.

To try it at your desk, select Analyse example birdsong. This uses Oona Räisänen's public-domain chaffinch recording from Finland, not a recording I made outdoors. It never adds a stop to your journal.

For a walk, name your first stop and press Listen for 6 seconds. Microphone permission is requested at that point. Add a note, move somewhere different, and repeat twice. The finished journal downloads as JSON. Remembering it on the device is optional.

Imported audio is supported for testing and accessibility, but every imported stop is labelled as imported, not a verified outdoor observation.

Code

Public repository

Original application code and SVG artwork are MIT licensed. Google YAMNet and TensorFlow.js retain Apache 2.0 licences. Source links, notices and model-file hashes are included. This project and repository were created on October 6, during the Week 1 window.

How I Built It

The core is Google YAMNet, an open audio-event model with 521 labels, running through TensorFlow.js. The app averages stereo imports to mono, resamples to 16 kHz and averages the model's frame scores across a short clip. Broad Nature, People and City summaries use editable label groups.

A label is a suggestion, not a certainty. YAMNet scores are not calibrated probabilities. Quiet audio, clipping and ambiguous results receive different prompts instead of a confident-sounding claim.

The supplied birdsong gave a Bird score of approximately 0.556 and a Nature summary. That number is a model score, not a 55.6% confidence estimate. Silence produced a Silence label and a warning that the moment was too faint.

All model weights and the browser runtime are served as static files. A service worker caches a fixed list after successful preparation. There is no inference server, location request, analytics script or audio upload endpoint. Capture tracks are stopped after recording or cancellation; only notes, sources and model suggestions enter the journal.

Validation includes eight unit tests and twelve Chromium integration checks. The browser checks exercise the actual YAMNet model, permission denial, recording cancellation, a complete generated recording, invalid imports, silence, journal export, storage opt-in and forgetting, offline reload and a 390px layout. They also check for browser errors and network-upload requests.

Two details mattered during testing: a cancelled capture must release its track, and an example result must never quietly become a claimed walk observation. The integration tests check both. Repeated inference also returned to the same persistent tensor count.

Limits: the microphone test uses a generated MediaStream. I have not completed an outdoor, physical-phone or Safari trial, so I am not claiming one. Browser storage can be evicted, and overlapping outdoor sounds may confuse the model. Those are the next practical checks.

Why Does Open Innovation Matter?

A walk should keep working when the signal disappears. Having distributable model weights made offline inference possible without an API key or subscription. It also means a person's six seconds of sound do not need to leave their device.

The open implementation lets someone inspect the input processing, change the sound groups, adjust uncertainty thresholds or replace the model. They can run this static app on their own host. The tradeoff is a larger first download and work constrained by the device; that is visible in the preparation step rather than hidden behind a server call.

Build Process and Attribution

The implementation, documentation, tests and this write-up were generated by Codex under my instructions. DEV's AI disclosure is set to Fully Autonomous. I did not train YAMNet or collect the example recording. The test results are software evidence, not a claim of a human field trial.

Model: Google YAMNet. Runtime: TensorFlow.js. Example audio: Oona Räisänen, Fringilla coelebs, 2007. Full notices are in the repository.

For this entry I am entering the overall challenge; I am not claiming a partner-technology category.

Foley Field Notes: a tin box, an entire train

Kanishq Sharma — Mon, 28 Sep 2026 09:49:08 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange.

What I Built

Foley Field Notes is a tiny imaginary sound studio. A tin box becomes a train; a cup becomes a bell; tracing paper becomes rain.

Pick a sound recipe, arrange its entrance in a scene, adjust its level, and download the result as a real WAV file. The sounds are synthesized in the browser. They are interpretations of everyday materials, not recordings of those materials.

The interesting part is what lives in Sanity. A recipe contains the instructions that make its sound: noise or tone layers, frequencies, attack, decay, gain and pulse count. A scene references those recipes with start times and levels. Content is doing something audible here.

Demo

Open Foley Field Notes

No login or microphone permission is needed for the player. A quick route through it:

Listen to Paper rain and Pocket train.
Choose The last train is a tin box from the scene selector.
Move the cup bell's entrance, change a cue's level, and play the scene.
Download the WAV. The arrangement stays in your current tab.

The editor is separate and requires Sanity project membership. Judges can use the public player and inspect the project ID/schema below; no editor credentials are published.

Code

Source, tests and build notes

The application is MIT licensed. The original sound recipes use deterministic synthesis rather than stock audio, so the repository includes the entire sound-generation path.

My Build Process

AI disclosure: this was an autonomous AI-assisted build using Codex under my direction. Codex implemented and tested the application and helped write this account. I am not claiming I manually wrote every line or personally performed subjective listening tests.

The brief used in the build was to make a small, unusual app where structured content materially changes the experience, with a working Astro frontend and no paid runtime AI API. The resulting scope was six sound recipes, two starter scenes, an editable cue sheet and WAV export.

The schema came before the cards

soundRecipe describes a material, a gesture, the synthesis layers and a duration. soundScene contains cue objects with references to recipes. A reused recipe is one document, not a copied block in each scene.

This choice is easy to see in the app: change a recipe's synthesis parameters in the editor, approve and publish it, then refresh the public collection. Every scene referencing it uses the new sound.

Approval had to describe the current content

A status field alone would allow a recipe to be approved and then silently changed. The workflow therefore stores a signature of the approved content. An edit makes that signature stale.

The Studio actions are Request review, Approve for playback, and Return to draft. The publish action stays disabled until the current recipe matches its approval. Workflow patches use revision checks so a concurrent edit cannot be silently overwritten. The frontend checks approval too.

This is an editorial workflow for trusted project members, not an administrator-proof permission system. The starting collection explicitly names its reviewer as Demonstration seed (automated); it does not imply human approval.

A few things needed correction

The first test invocation failed before executing because the local environment could not spawn a child process. Running the same Node tests without process isolation fixed the runner issue without weakening the assertions.
The volume control initially affected the next playback only. Review caught that mismatch; it now changes the active gain node as well.
The initial Studio build command used an unsupported output flag. The current positional-output command built successfully.
The first upload draft flattened nested file paths. It was discarded before committing, and the saved repository was checked for the proper src, tests and docs directories.

What was verified

Eleven automated tests passed for deterministic sound generation, non-silence, finite/bounded samples, cue timing, overlapping-cue normalization, WAV headers, invalid input and approval invalidation.

Browser checks exercised the actual public Sanity data, recipe selection, scene presets, cue editing, playback start/stop, WAV download and the empty state. Layout checks found no horizontal overflow at 320, 393, 768 and 1440 pixels in Chromium.

Four additional live Studio checks used a temporary unpublished document: a draft could not publish, requesting review enabled approval, approval enabled publication, and a title edit disabled publication again. The temporary draft was then removed.

The fallback is intentionally visible. If the initial content request fails, the player says Offline demonstration and loads its bundled sample pack. That is not reported as a successful Sanity connection.

Where I stopped

There is no runtime LLM, App SDK integration or automatic realtime refresh. The frontend uses an explicit Refresh collection control. Public users arrange sounds locally; they cannot write into the shared dataset. The tests do not establish subjective sound quality or compatibility with every physical device or assistive technology.

That left room to finish a small app whose structured content, review state and generated output all have a clear purpose.

Sanity Project Details

Project ID: vkj2ioma
Dataset: production (public)
Document types: soundRecipe, soundScene
Nested types: soundLayer, sceneCue
Framework: Astro, with Sanity Studio for authoring

Inspect the schema or read the build notes.