DEV Community: BABBLED77

Building an Accessibility Agent with Hermes Agent: Sound to Music for People Who Couldn't Before

BABBLED77 — Sun, 24 May 2026 22:17:44 +0000

💎 One sound. Any sound. The agent hears it. Music appears.

𝕓𝕒𝕓𝕓𝕝𝕖𝕕 𝕟𝕠𝕥𝕖𝕤

✦ An Accessibility Agent Built on Hermes Agent Principles ✦

This post explores how Hermes Agent -- Nous Research's open-source autonomous agent platform -- maps perfectly to a real-world accessibility problem: giving people who cannot use traditional music tools a way to make music with any sound their body can produce.

The result is babbled notes v2: a Gemma 4-powered agent that turns a hum, a breath, a tap, or a tongue click into a real musical composition.

🔗 GitHub: https://github.com/brookehoward2008-droid/Babbled-notes-v2
📐 Agent docs: https://github.com/brookehoward2008-droid/Babbled-notes-v2/blob/main/HERMES.md

◈ What Is Hermes Agent

Hermes Agent is an autonomous system by Nous Research -- not a coding copilot tethered to an IDE, not a chatbot wrapper around a single API. It is a server-side agent with:

✦  Multi-platform reach    Telegram, Discord, Slack, WhatsApp, Signal, Email, CLI
✦  Persistent memory       Learns from past work, reapplies solutions automatically
✦  Scheduled automations   Natural language cron: "send me a briefing every morning"
✦  Subagent delegation     Parallel agents with isolated contexts, no context bleed
✦  Five sandbox backends   Local, Docker, SSH, Singularity, Modal
✦  Web capabilities        Search, browser automation, vision, image gen, TTS

Install in one command:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup

What makes Hermes Agent different from a chatbot: it operates independently on your server, can schedule work, delegate subtasks to subagents, and persist knowledge across sessions. That is a real agent architecture -- not a prompt-response loop.

💎 Why Hermes Agent and Accessibility Belong Together

Most music tools require two hands, ten fingers, perfect pitch, or years of training.

That shuts out a huge part of the world. People who are non-verbal. People with ALS, cerebral palsy, locked-in syndrome, quadriplegia, Parkinson's. People who have always heard music inside them and had no way to get it out.

Hermes Agent's architecture is exactly what an accessibility tool needs:

Hermes Agent capability	Accessibility use case
Multi-platform	User triggers music generation from Telegram with a voice message -- no keyboard needed
Persistent memory	Agent remembers "this user has Parkinson's -- treat tremor as vibrato, always use cello"
Scheduled automations	"Generate a new composition every morning at 7am" -- ambient music therapy, automated
Subagent delegation	One subagent handles DSP analysis; another handles Gemma 4 reasoning -- no context bleed
Web + browser	Agent could automatically post generated Lilt scores to a shared Notion page or email

A user with locked-in syndrome can send a single message to Hermes Agent via Telegram. Hermes Agent delegates to the babbled notes subagent. A composition comes back to their phone. No laptop required. No mouse. No keyboard.

◈ The Agent Loop: How babbled notes Maps to Hermes Agent

Hermes Agent's architecture -- perceive, reason, act, remember -- is exactly the loop babbled notes runs on every sound.

┌─────────────────────────────────────────────────────────────────┐
│                    HERMES AGENT ORCHESTRATION                   │
│                                                                 │
│  User message (Telegram / CLI)                                  │
│       "Turn my hum into music"                                  │
│              |                                                  │
│              v                                                  │
│  ┌───────────────────────────┐                                  │
│  │   babbled notes subagent  │                                  │
│  │                           │                                  │
│  │  PERCEIVE                 │                                  │
│  │  Web Audio API            │                                  │
│  │  FFT + onset detection    │                                  │
│  │  -> DspDigest             │                                  │
│  │          |                │                                  │
│  │  REASON                   │                                  │
│  │  Gemma 4 reads audio      │                                  │
│  │  + DspDigest              │                                  │
│  │  -> Lilt score            │                                  │
│  │          |                │                                  │
│  │  ACT                      │                                  │
│  │  Synthesizer plays music  │                                  │
│  └───────────────────────────┘                                  │
│              |                                                  │
│  Hermes Agent delivers result to user                           │
│  + stores preference in persistent memory                       │
└─────────────────────────────────────────────────────────────────┘

💎 Integrating babbled notes With Hermes Agent

Here is how to wire babbled notes into Hermes Agent as a callable subagent skill.

Step 1: Run babbled notes

git clone https://github.com/brookehoward2008-droid/Babbled-notes-v2.git
cd Babbled-notes-v2
npm install
# add GEMINI_API_KEY to .env.local
npm run dev
# server running at http://localhost:3000

Step 2: Install Hermes Agent

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup

Step 3: Create a babbled notes skill

Save this as ~/.hermes/skills/babbled_notes.py:

"""
Babbled Notes skill for Hermes Agent
Converts a DSP sound description into a musical Lilt score via Gemma 4.
"""
import requests, json

def generate_music(
    pitch_hz: float,
    duration_s: float,
    amplitude: float = 0.1,
    user_prompt: str = ""
) -> dict:
    """
    Ask babbled notes to compose music from a sound description.
    pitch_hz: dominant frequency in Hz (e.g. 220 for A3)
    duration_s: how long the sound lasted
    amplitude: loudness 0.0-1.0 (0.1 = soft breath, 0.9 = loud tap)
    user_prompt: optional intent hint ("make it a cello", "slow and gentle")
    """
    import math
    # build note name from Hz
    names = ["C","C#","D","D#","E","F","F#","G","G#","A","A#","B"]
    midi = round(12 * math.log2(max(pitch_hz, 20) / 440) + 69)
    midi = max(0, min(127, midi))
    pitch_name = f"{names[midi % 12]}{(midi // 12) - 1}"

    digest = {
        "duration": duration_s,
        "averageEnergy": amplitude,
        "peakOnsetCount": 1,
        "events": [
            {"time": 0.0, "frequency": pitch_hz,
             "pitchName": pitch_name, "amplitude": amplitude}
        ]
    }

    response = requests.post(
        "http://localhost:3000/api/interpret",
        json={"dspDigest": digest, "userPrompt": user_prompt},
        timeout=90
    )
    return response.json()


def generate_music_from_profile(profile: str) -> dict:
    """
    Generate music for a known disability profile.
    profile: one of 'breath', 'hum', 'tremor', 'tap', 'click', 'puff', 'whistle'
    """
    profiles = {
        "breath":  (180,  2.5, 0.03, "minimal breath, ambient drone"),
        "hum":     (220,  3.0, 0.11, "gentle sustained hum, cello"),
        "tremor":  (196,  2.0, 0.08, "tremor hum, treat as vibrato"),
        "tap":     (440,  0.2, 0.45, "single finger tap, percussive"),
        "click":   (800,  0.1, 0.60, "tongue click, sharp and short"),
        "puff":    (120,  1.5, 0.05, "breath puff, soft and round"),
        "whistle": (1047, 0.8, 0.30, "single whistle note, clear pitch"),
    }
    hz, dur, amp, prompt = profiles.get(profile, profiles["hum"])
    return generate_music(hz, dur, amp, prompt)

Step 4: Use it via Hermes Agent

you: Generate music from a hum at A3
hermes: [calls generate_music(220, 3.0, 0.11, "gentle hum")]

Gemma 4 returned:
  mood: pensive
  voice: cinematic cello
  notes: A3 soft @ 0.00s / C4 normal @ 1.20s / A2 soft @ 0.00s (drone)
  explanation: A sustained A natural, barely above a whisper.
               The cello holds it, lets it breathe.

Step 5: Add persistent memory

Tell Hermes Agent your preference once:

you: Remember that I have Parkinson's -- always treat tremor as vibrato,
     always use cinematic cello voice
hermes: Noted. I'll apply that to all future music generation for you.

Hermes Agent's persistent memory stores this. Every future generate_music call
is automatically informed by the user's disability profile -- no re-explaining needed.

◈ The Perception Layer in Detail

The babbled notes perception layer is what Hermes Agent's subagent would feed into.

Web Audio API runs in real time during recording:

Microphone  ->  AnalyserNode (FFT 256 bins)  ->  peak bin -> Hz -> note name
                ScriptProcessor              ->  RMS amplitude
                Onset detector               ->  timestamps when sound starts
                                             ->  DspDigest (JSON)

Onset detection threshold is set deliberately low (0.1 RMS) to catch breath inputs:

if (rms > 0.1 && elapsed - lastOnset > 0.1) {
  events.push({ time: elapsed, frequency: freq,
                pitchName: note, amplitude: rms });
}

A breath at 0.02 RMS in a quiet room barely registers. The threshold is at 0.1 because it needs to catch sounds that are 5x quieter than a normal speaking voice.

💎 The Reasoning Layer: Gemma 4

Gemma 4 (gemma-4-26b-a4b-it) is the reasoning engine. It receives both:

Raw audio (base64 WebM) -- texture, tremor quality, breath shape
DspDigest (JSON) -- precise onset timing, Hz, amplitude

And returns a complete Lilt score -- a musical composition in structured JSON:

{
  "mood": "pensive",
  "articulation": "legato",
  "voice": "cinematic cello",
  "notes": [
    { "note": "A3", "duration": 1.4, "velocity": "soft",   "time": 0.0 },
    { "note": "C4", "duration": 1.2, "velocity": "normal", "time": 1.2 },
    { "note": "A2", "duration": 4.0, "velocity": "soft",   "time": 0.0,
      "voice": "synthesizer ambient" }
  ],
  "explanation": "A breath, barely a sound. Steady. Like resolve."
}

The reasoning follows the Lilt Contract -- guidelines Gemma 4 interprets, not hardcoded rules:

◈  Slow, soft, hummed  ->  pensive/gentle + cello/piano + legato
◈  Sharp, rhythmic     ->  energetic/tight + marimba/drums + staccato
◈  Always harmonious pitches: C major, A minor, pentatonic
◈  Always include a synthesizer ambient drone layer

◈ 32 Profiles Tested Across 7 Disability Categories

The agent was validated against 32 live Gemma 4 responses -- no simulated data.

Result: 32 / 32 passed

Category	Tests	Notes range
Non-verbal autism (NV)	9	3-6 notes
Physical disabilities (PH)	13	3-7 notes
Mixed / cross-profile (MX)	10	3-7 notes

node test-runner.mjs   # run all 32 yourself

Full results in test-results.json -- 1,347 lines of live Gemma 4 output.

◈ The NeuralGem: Agent State Without Words

◇  IDLE        breathing silver ring. the agent is waiting.
◈  RECORDING   crystallizing polygon, purple to cyan.
               Hermes subagent is running perception layer.
⬡  PROCESSING  hexagon forming. Gemma 4 is reasoning.
⬡  LOCKED      hexagon, facets lit in the mood color.
               The agent has decided. Music is loading.

No text labels. Shape and color carry all the state. For users who cannot read, or who have cognitive differences: the gem is the interface.

💎 What Hermes Agent Makes Possible Next

With Hermes Agent as the orchestration layer:

◈  Telegram trigger       User sends voice note to Hermes Agent bot
                          Hermes transcribes audio -> babbled notes API -> Lilt score sent back

◈  Persistent memory      Agent knows: "this user uses breath puffs, always cello,
                          always soft dynamics" -- applied every session without re-explaining

◈  Scheduled music        "Every morning at 7am, generate a new ambient piece
                          from my baseline breath profile" -- Hermes cron triggers babbled notes

◈  Subagent pipeline      Agent 1: DSP analysis on uploaded audio file
                          Agent 2: Gemma 4 reasoning with profile context
                          Agent 3: Delivery to user's preferred channel

◈  Multi-platform         Same music generation accessible from phone, desktop,
                          Slack workspace, or Discord server -- wherever the user is

◈ Stack

┌──────────────────────────────────────────────────────┐
│  ORCHESTRATION  Hermes Agent (Nous Research)         │  ← autonomous agent platform
│  REASONING      Gemma 4  gemma-4-26b-a4b-it          │  ← the agent's brain
│  PERCEPTION     Web Audio API (FFT, RMS, onset)      │  ← the agent's ears
│  ACTION         Web Audio API (synthesis)            │  ← the agent's voice
│  FRONTEND       React + Vite + TypeScript            │
│  BACKEND        Express + @google/genai SDK          │  ← API key stays here
└──────────────────────────────────────────────────────┘

💎 The gem crystallizes.
Hermes delegates. Gemma 4 decides.
The music plays.
You made that. You made that with a breath.

GitHub: https://github.com/brookehoward2008-droid/Babbled-notes-v2
Agent architecture: https://github.com/brookehoward2008-droid/Babbled-notes-v2/blob/main/HERMES.md

by Brooke Chauntel

babbled notes: a sound-to-music agent for people who could not make music before

BABBLED77 — Sun, 24 May 2026 21:47:57 +0000

💎 You make a sound. Any sound. The agent hears it. Music comes back.

𝕓𝕒𝕓𝕓𝕝𝕖𝕕 𝕟𝕠𝕥𝕖𝕤

Hum into a microphone. Tap your desk. Exhale slowly. Click your tongue. Whistle once.

A Gemma 4 agent reads what you made, decides what music lives inside it, and plays it back as piano, cello, marimba, or drums.

You chose nothing. The agent chose everything.

Built for people who have never been able to make music before -- people who are non-verbal, people with ALS, cerebral palsy, locked-in syndrome, quadriplegia, Parkinson's. People who have always heard music inside them and had no way to get it out.

🔗 GitHub: https://github.com/brookehoward2008-droid/Babbled-notes-v2
🎵 Agent architecture: HERMES.md

◈ Why this is an agent, not a tool

A tool does what you tell it. You configure it. You choose the settings. You push the button.

An agent perceives its environment, reasons about what it observes, and takes action on its own judgment.

babbled notes runs a full agent loop on every sound:

Component	What it does
Perceive	Web Audio API reads the mic: FFT pitch analysis, RMS amplitude, onset detection. Outputs a structured DspDigest.
Reason	Gemma 4 (`gemma-4-26b-a4b-it`) receives the raw audio AND the DspDigest. Decides mood, instrument voice, articulation, and note timing.
Act	Web Audio API synthesizer plays the composition. Real instruments. Real time.
Reflect	User edits the Lilt score. Agent re-renders without re-recording.

The user never chooses a key, a tempo, a voice, or a mood. The agent reads the sound and decides all of it.

💎 The NeuralGem

The agent communicates its state through the NeuralGem -- a canvas visualizer with no text labels:

IDLE       ->  breathing silver ring. waiting for input.

RECORDING  ->  crystallizing polygon. sides grow as your audio level rises.
              color shifts purple to cyan as the sound builds.

PROCESSING ->  hexagon forming. the agent is reading your sound.

LOCKED     ->  hexagon. facets lit in the mood color the agent chose.
              the agent has heard you. music is loading.

For users who are non-verbal, have cognitive differences, or who cannot read: shape and color carry all the information. No labels to parse. No configuration panel to navigate. Tap once to start. Tap once to stop.

◈ How the agent reasons

The agent sends two things to Gemma 4 simultaneously:

1. Raw audio (base64 WebM)
The actual sound. Gemma 4 can hear the texture -- a tremor in a hum, the scrape of a breath, the sharp crack of a tongue click. These textures do not survive FFT analysis. They live in the audio.

2. DspDigest (structured JSON)
What the perception layer already calculated precisely:

{
  "duration": 3.2,
  "averageEnergy": 0.11,
  "peakOnsetCount": 2,
  "events": [
    { "time": 0.0,  "frequency": 220, "pitchName": "A3", "amplitude": 0.11 },
    { "time": 1.6,  "frequency": 261, "pitchName": "C4", "amplitude": 0.13 }
  ]
}

Two onsets. A3 moving to C4. 1.6 seconds apart. Average energy 0.11 -- a soft sound.

Gemma 4 reads both and decides: this is a sustained hum that rose in pitch. Mood: pensive. Voice: cinematic cello. Articulation: legato. Two melody notes, one drone pad underneath. Timestamps aligned to the 1.6-second interval in the digest.

The agent's output:

{
  "mood": "pensive",
  "articulation": "legato",
  "voice": "cinematic cello",
  "liltCode": "A3 ! soft @ 0.00s\nC4 ! normal @ 1.60s",
  "notes": [
    { "note": "A3", "duration": 1.4, "velocity": "soft",   "time": 0.0 },
    { "note": "C4", "duration": 1.2, "velocity": "normal", "time": 1.6 },
    { "note": "A2", "duration": 3.5, "velocity": "soft",   "time": 0.0, "voice": "synthesizer ambient" }
  ],
  "explanation": "A rising hum -- two tones, a minor third apart. The cello holds the first note soft, lifts into the second. The drone underneath gives it weight."
}

The agent turned a two-second hum into a composition with melody, countermelody, and an ambient drone. The user made one sound. The agent made the music.

◈ The Lilt Contract

The agent's reasoning follows a set of guidelines built into the system prompt. These are not hardcoded rules -- Gemma 4 interprets them against what it actually heard:

Slow, soft, or hummed sounds:
  mood = "gentle" or "pensive"
  voice = "cinematic cello" or "grand piano"
  articulation = "legato"

Sharp, rhythmic, or tapped sounds:
  mood = "energetic" or "tight"
  voice = "marimba" or "drum kit"
  articulation = "staccato"

Always keep pitches harmonious (C major, A minor, or pentatonic).
Timestamps must align with DSP onsets but feel musically polished.
Always include a drone layer using "synthesizer ambient" voice.

A tremor-affected tap does not fit cleanly into either category. The agent reads it as closer to a soft sound than a sharp one -- Parkinson's tremor in a hum becomes vibrato in the cello voice. A morse-style rhythm gets staccato articulation but the agent may still choose "grand piano" if the pattern feels musical rather than percussive.

The agent makes judgment calls. That is the point.

◈ The Lilt format

The agent outputs in Lilt -- a flat timestamp-based musical notation:

A3 ! soft   @ 0.00s
C4 ! normal @ 1.60s
E4 ! accent @ 2.80s
A2 ! soft   @ 0.00s   [synthesizer ambient]

Each line: pitch, velocity flag, timestamp, optional voice override.

The piano roll renders from this. The code is editable live. Change a velocity, shift a timestamp, swap a pitch, add a note. The synthesizer re-renders immediately. No new recording. No new API call.

This is the feedback loop. The agent interprets. The user adjusts. The agent re-renders.

💎 Who the agent serves

Profile	What they give	What the agent produces
💜 Non-verbal autism	Sustained hum, single tone	Cello or piano melody in that pitch
💙 Cerebral palsy	Tremor-affected taps	Percussive or piano rhythm
🤍 ALS	Minimal breath control	Ambient drone with gentle melody over it
💛 Locked-in syndrome	Single eye-blink switch click	One-trigger composition, loops
💚 Quadriplegia	Hard puff / soft puff contrast	Two-dynamic melody: accent and soft
🧡 Parkinson's	Tremor vocal hum	Cello composition that treats tremor as vibrato
🩷 Apraxia of speech	Broken phonation bursts	Legato phrase bridging the silence between bursts
💎 AAC / pre-verbal	Rising or falling hum	Interval-based melodic response
🔵 Spinal cord injury C4	Head tap on mic	Beat-based composition from impact events
⚪ Selective mutism	Barely audible breath	Gentle drone that validates the smallest input

The agent does not have a "minimum input" requirement. A breath at 0.02 RMS amplitude -- almost nothing -- produces a composition. This was a deliberate design decision. The quietest input a person can give must be enough.

◈ 32 profiles tested

The agent was validated against 32 real DSP profiles representing the disability communities it was built for.

Three difficulty levels:

Beginner     -- one event, one sound. proves the agent handles the minimum.
Intermediate -- 2-3 events, some rhythm or pitch shift.
Advanced     -- 4+ events, dynamics, intentional pattern.

Results across all 32 profiles: 32 passed. 0 failed.

Every result is a live Gemma 4 response -- no simulated data, no hardcoded fallback. The test suite fires real DSP payloads at the running Express server and logs every decision the agent made.

node test-runner.mjs   # run all 32 profiles yourself

Full results in test-results.json on GitHub.

◈ Technical stack

Gemma 4 (gemma-4-26b-a4b-it)   reasoning engine
Web Audio API                   perception layer + action layer (synthesis)
React + Vite + TypeScript       frontend / state machine
Express + @google/genai SDK     backend agent server

The API key stays server-side. The browser never sees it.

◈ How to run it

git clone https://github.com/brookehoward2008-droid/Babbled-notes-v2.git
cd Babbled-notes-v2
npm install

Add a free Gemini API key to .env.local:

GEMINI_API_KEY=your_key_here

npm run dev

Open http://localhost:3000. Allow microphone access. Tap the silver ring. Make any sound. Wait 30-60 seconds for Gemma 4 to reason. The music plays.

No API key? The app runs in simulation mode -- the full UI and audio play back immediately.

◈ Agent architecture (detailed)

Full technical breakdown in HERMES.md:

Perception layer: FFT signal chain, onset detector, DspDigest schema
Reasoning layer: dual-input Gemma 4 call, Lilt Contract, JSON extraction
Action layer: per-voice synthesis chains, scheduling via AudioContext
Feedback loop: live Lilt editor, re-render without re-recording
State machine: idle / recording / processing / playing

💎 The gem crystallizes. The music plays. You made that.
You made that with a breath.

GitHub: https://github.com/brookehoward2008-droid/Babbled-notes-v2
Agent docs: https://github.com/brookehoward2008-droid/Babbled-notes-v2/blob/main/HERMES.md

by Brooke Chauntel

babbled notes: any sound becomes music. built for people who couldn't before.

BABBLED77 — Sat, 23 May 2026 22:58:41 +0000

💎 One sound. Any sound. The gem listens. The music appears.

𝕓𝕒𝕓𝕓𝕝𝕖𝕕 𝕟𝕠𝕥𝕖𝕤

Make any sound. Hum. Tap. Breathe. Whistle.

Gemma 4 finds the music inside it and plays it back as piano, cello, marimba, or drums.

No keyboard. No music theory. No pitch-perfect voice.

Built for anyone who has ever felt shut out of making music.

🔗 GitHub: https://github.com/brookehoward2008-droid/Babbled-notes-v2
🎵 Live app: https://ai.studio/apps/4d235490-15ac-47a5-9599-f82aa85a2b57

◈ The problem

Most music tools require two hands, ten fingers, perfect pitch, or years of training.

babbled notes gives them a door.

A single breath. A tongue click. A finger tap. A hum with a tremor in it.

The app takes whatever you can give and turns it into a real musical composition, rendered in real time by a synthesized instrument of your choice.

💎 The NeuralGem

At the center of the app is the NeuralGem, a canvas visualizer with three states:

IDLE       →  a breathing silver ring. waiting.
RECORDING  →  a crystallizing polygon. sides grow with your audio level.
             color shifts purple → cyan as the sound builds.
LOCKED     →  a hexagon. facets lit in your mood color.
             the gem has heard you.

The gem is not decoration. It tells you what the app is doing without words.

◈ Who it is built for

Profile	What they give	What they get
💜 Non-verbal autism	Sustained hum, single tone	Cello or piano melody
💙 Cerebral palsy	Tremor-affected taps	Percussive rhythm, drum or marimba
🤍 ALS	Minimal breath	Ambient drone pad with gentle melody
💛 Locked-in syndrome	Single eye-blink switch click	One-trigger composition, looping
💚 Quadriplegia	Hard puff / soft puff	Two-dynamic melody: accent and soft
🧡 Parkinson's	Tremor vocal hum	Composition that treats tremor as vibrato
🩷 Apraxia of speech	Broken phonation bursts	Legato phrase bridging the gaps
💎 AAC / pre-verbal	Rising or falling hum	Interval-based melodic response

◈ How it works

1.  TAP THE ORB      →  microphone opens
2.  MAKE A SOUND     →  Web Audio API captures + analyzes in real time
                        (FFT pitch, RMS amplitude, onset detection)
3.  TAP AGAIN        →  recording stops
4.  GEMMA 4 READS    →  receives audio + DSP digest simultaneously
                        returns: mood, voice, articulation, Lilt score
5.  THE GEM LOCKS    →  mood-colored hexagon appears
6.  MUSIC PLAYS      →  synthesized instrument renders the Lilt score
7.  EDIT ANYTIME     →  piano roll + live Lilt code editor, re-render without re-recording

💎 Why Gemma 4

The app sends two things to the model at once:

Raw audio: the actual recorded sound
DSP digest: structured analysis of onset times, dominant frequency, pitch name, amplitude, tempo estimate

Gemma 4 (gemma-4-26b-a4b-it) reads both together and returns fast enough that a user with ALS or limited stamina hears their composition without waiting. That responsiveness matters. A slow model breaks the experience.

The system prompt enforces a strict JSON Lilt score every time. No freeform text. No guessing.

{
  "mood": "gentle",
  "articulation": "legato",
  "voice": "cinematic cello",
  "notes": [
    { "note": "A3", "duration": 1.2, "velocity": "soft", "time": 0.0 },
    { "note": "C4", "duration": 0.8, "velocity": "normal", "time": 1.2 }
  ],
  "explanation": "A slow exhale, barely a sound. But steady. Like resolve."
}

◈ Disability profiles tested

32 real DSP profiles. 7 disability categories. 3 difficulty levels.

Beginner: one event, one sound, one note
Intermediate: 2-3 events, some rhythm or pitch shift
Advanced: 4+ events, dynamics, intentional pattern

NV-01  Autism — slow exhale breath         (beginner)
NV-02  Autism — single sustained hum       (beginner)
NV-03  Autism — two-tone hum shift         (intermediate)
NV-04  Autism — melodic hum phrase         (advanced)
NV-05  Apraxia — disrupted single vowel    (beginner)
NV-06  Apraxia — broken phonation bursts   (intermediate)
NV-07  Apraxia — vowel glide attempt       (advanced)
NV-08  Selective mutism — barely audible   (beginner)
NV-09  Selective mutism — nose exhale      (intermediate)
PH-01  Cerebral palsy — single finger tap  (beginner)
PH-02  Cerebral palsy — tremor cluster     (intermediate)
PH-03  Cerebral palsy — intentional beat   (advanced)
PH-04  ALS — minimal breath control        (beginner)
PH-05  ALS — pulsed breath pattern         (intermediate)
PH-06  Locked-in — single switch click     (beginner)
PH-07  Locked-in — two-click phrase        (intermediate)
PH-08  Locked-in — morse-style rhythm      (advanced)
PH-09  Quadriplegia — single breath puff   (beginner)
PH-10  Quadriplegia — hard/soft contrast   (intermediate)
PH-11  Quadriplegia — rhythmic phrase      (advanced)
PH-12  Parkinson's — tremor hum            (beginner)
PH-13  Parkinson's — vocal tremor melody   (advanced)
MX-01  Whistle — single clear pitch        (beginner)
MX-02  Whistle — two-note call             (intermediate)
MX-03  Whistle — pentatonic phrase         (advanced)
MX-04  Tongue click — single event         (beginner)
MX-05  Tongue click — 4/4 rhythm           (intermediate)
MX-06  Tongue click — syncopated groove    (advanced)
MX-07  AAC — rising hum intention          (intermediate)
MX-08  AAC — call and response             (advanced)
MX-09  SCI C4 — head tap                   (beginner)
MX-10  SCI C4 — two-tap intentional gap    (intermediate)

Run them yourself: node test-runner.mjs

◈ Stack

Gemma 4 (gemma-4-26b-a4b-it)   multimodal audio + DSP digest to Lilt JSON
Web Audio API                   mic capture, FFT/RMS DSP, synthesized playback
React + Vite + TypeScript       frontend
Express + @google/genai SDK     backend (API key stays server-side)

💎 What the Lilt format looks like

A3 ! soft   @ 0.00s
C4 ! normal @ 1.20s
E4 ! accent @ 2.10s
G4 ! soft   @ 3.40s

Each line is a note trigger: pitch, velocity, timestamp. The piano roll renders from this. The code is editable live. Change a velocity, move a timestamp, swap a note, hit compile. The music changes without re-recording.

💎 The gem crystallizes. The music plays. You made that.
You made that with a breath.

GitHub: https://github.com/brookehoward2008-droid/Babbled-notes-v2
Live app: https://ai.studio/apps/4d235490-15ac-47a5-9599-f82aa85a2b57

by Brooke Chauntel