Reverb: Making Model Progress Audible - Two LLMs, One Prompt, Eight Seconds of Web Audio

How to build an app that runs AI-generated music code in sandboxed iframes, measures it with a real AnalyserNode, and exports the proof as WAV files — with zero audio libraries and zero mocking.

---

The premise

Ask a language model to describe music and you get words. Ask it to make music with a hard constraint — "Write JavaScript that plays an 8-second piece of music using ONLY the Web Audio API. No libraries, no external files, no samples. Output only code." — and you get something much more interesting: a tiny program that is either a composition or a crash, with no in-between.

Reverb takes that prompt and sends it to two model slots at once — an older model (A) and a newer one (B). Each reply is executed inside a sandboxed iframe with a genuine AudioContext. The app paints a live spectrogram of each performance on Canvas 2D, plays both pieces simultaneously as a duet, and exports WAVs rendered by a real OfflineAudioContext.

The purpose is demonstration, not benchmarking. "The new model is better" becomes audible and visible: two scrolling spectrograms, one cyan, one magenta, and a duet you can post as audio.

The rule that shaped every design decision: everything must be really synthesised and really analysed live. Never mocked. No canned animations, no fake meters, no invented token counts.

Architecture at a glance

┌─────────────────────────── Browser · localhost:5173 ───────────────────────────┐
│  React UI                                                                      │
│  ├── prompt + presets ───────────────► POST /api/compose (Vite /api proxy)     │
│  ├── iframe A (sandbox, srcdoc) ─┐  iframe B (same)                            │
│  │   ▲ {run|render|stop, code}   │   │                                         │
│  │   └ {frame|ended|rendered|error} ◄─ postMessage                            │
│  ├── Canvas 2D spectrograms (offscreen scroll @ 60fps)                         │
│  └── WAV encode · duet mix · FFT fingerprint                                   │
└────────────────────────────────────────────────────────────────────────────────┘
          │ /api/*                        ▲ OpenAI-compatible HTTP
          ▼                               │
┌─────────────────────────── Node backend · localhost:3001 ──────────────────────┐
│  POST /api/compose — both slots concurrently, fresh nonce per prompt,          │
│  reasoning_content stripped, code extracted + sha256'd, NO_AUDIO rejected      │
│  POST /api/models — live model-list proxy                                      │
└────────────────────────────────────────────────────────────────────────────────┘
     │                │                 │                  │
     ▼                ▼                 ▼                  ▼
 Particle.ai      Ollama            LM Studio        OpenRouter / Custom

Two processes, one contract. The browser never talks to a model provider directly — that is what makes Ollama and LM Studio work with zero CORS configuration, and it is where prompt hygiene lives (nonces, retries, stripping).

Trick #1 — Hearing and measuring the same graph

The central problem: model code connects its nodes to ctx.destination, and we need to both play it and read it. You cannot attach an analyser "after" the destination — the destination is the end of the graph.

The fix is to replace the AudioContext constructor inside the iframe with a wrapper that inserts a gain sink between the code and the speakers:

function Tapped() {
  var ctx = new Base();                     // the real AudioContext
  var an = ctx.createAnalyser();
  an.fftSize = 2048;                        // → 1024 frequency bins
  an.smoothingTimeConstant = 0;             // raw frames, no visual lying
  var sink = ctx.createGain();
  sink.connect(an);
  an.connect(ctx.destination);
  // the model code connects to ctx.destination; hand it the sink instead
  Object.defineProperty(ctx, 'destination', { get: function () { return sink; } });
  ctx.__an = an;
  return ctx;
}
window.AudioContext = Tapped;

The model's code cannot tell the difference — ctx.destination still behaves like a destination — but now every sample flows through our analyser on its way out. A requestAnimationFrame loop reads getByteFrequencyData (1024 bins) and getByteTimeDomainData, computes RMS, and posts a frame to the parent:

post({ type: 'frame', freq: freq, time: time, rms: rms, t: ctx.currentTime });

The parent paints each frame as a new column at the right edge of an offscreen canvas, shifts the image left with one drawImage, and blits. That is the entire spectrogram engine — one offscreen canvas, one blit, ~60 columns per second, with a log frequency axis so musical structure lands where the eye expects it.

Why an iframe and not eval?

<iframe sandbox="allow-scripts"> built from srcdoc gives the model code its own opaque origin: no parent DOM access, no cookies, no storage, no same-origin tricks — only postMessage in and out. The parent never evaluates model code in its own realm. If the code throws, the iframe catches it and posts back the real message and stack, and the UI shows the actual exception with the eval line number.

One subtlety: a hidden iframe stops firing requestAnimationFrame, which would freeze the spectrogram. Reverb keeps each iframe mounted at 2×2 pixels, technically visible, practically invisible.

Trick #2 — The OfflineAudioContext shim

Live playback proves the code runs. But for downloadable WAVs and the spectral fingerprint (duration, RMS, spectral centroid, peak frequency), you need deterministic PCM — which is exactly what OfflineAudioContext exists for.

You cannot force model code to use one. So Reverb re-evaluates the same code with the constructor replaced by a Proxy around a real OfflineAudioContext(2, 44100×8, 44100):

function Shim() {
  var ctx = new RealOAC(2, LEN, SR);
  return new Proxy(ctx, {
    get: function (target, prop) {
      if (prop === 'state') return 'running';
      if (prop === 'resume' || prop === 'suspend' || prop === 'close')
        return function () { return Promise.resolve(); };  // code may close() — ignore it
      var v = target[prop];
      return typeof v === 'function' ? v.bind(target) : v;
    },
    set: function (target, prop, value) { target[prop] = value; return true; }
  });
}

Three details make this work in practice:

1. currentTime stays 0 during evaluation. Code that schedules at ctx.currentTime + 0.1 schedules everything into the first beat — exactly what the offline render wants. (Code that schedules progressively via setTimeout chains is the known limitation; one-shot scheduling is the norm for "play 8 seconds" replies.)
2. close() is a no-op. Generated code loves to clean up after itself with ctx.close(); in offline mode that would kill the render.
3. startRendering() is passed through untouched — it is a real method on the real context, so when the shim's host calls it after evaluation settles, out comes a genuine 8-second PCM buffer.

The parent encodes 16-bit WAV itself (44-byte header + interleaved samples) and runs its own radix-2 FFT over Hann-windowed frames for the fingerprint. No dsp package, no wavefile package — the whole audio stack is ~200 lines of TypeScript.

The duet WAV is the honest kind: sample-wise average of the two renders, clamped — and the real duet is even more honest, both live AudioContexts playing through the speakers at the same moment.

Trick #3 — The nonce, or why cached replies lie

Provider-side response caches silently fake determinism. Ask the same question twice and you may get the same answer back instantly — which looks exactly like "the model is consistent" and is actually "the cache is warm."

Reverb appends a fresh random nonce to every prompt:

${userPrompt}\n\n[nonce:${randomBytes(8).toString('hex')}]

and records the SHA-256 of every full prompt it sends. A repeat is a hard 409. Two runs can never share a prompt, so every comparison you make on screen is between two genuinely fresh completions. The prompt SHA-256s are included in the exported JSON so anyone can audit the claim.

Capability detection, not assumptions

Local model servers (Ollama, LM Studio) are usually not reasoning models, and they do not return usage.completion_tokens_details.reasoning_tokens. Reverb's rules:

• Detect per slot whether usage carries reasoning tokens. Present → show the count. Absent → render n/a and hide the thinking toggle. Never 0, never invented.
• chat_template_kwargs {enable_thinking: false} is sent only when the slot's provider is Particle.ai and the model name starts with deepseek-. Other providers ignore or reject unknown fields, so they never see it.
• Reasoning models need a real budget: max_tokens has a floor of 900 and no artificial cap (it may go as high as 128000). An HTTP 200 with empty content means the hidden chain-of-thought ate the budget — that is not a refusal, so the backend escalates ×2 → 16k → 64k → 128k before giving up, and the error reports finish_reason, token counts, and the CoT's size (never its text) so you know exactly what happened.
• reasoning_content (the CoT text) is stripped in one place, stripMessage(), which copies only role and content. It is never logged, displayed, or persisted. Only its token count survives.

The same philosophy applies to the model picker: it is populated live from GET {base_url}/models, but you can always type a name by hand — deepseek-v4-flash-0731 responds even when it does not appear in Particle's model list, so a failing /models call must never gate a run. And a dead local server gets a specific message: Cannot reach http://127.0.0.1:11434 — is Ollama running? (ECONNREFUSED) — never "Something went wrong."

Proving it is real

The COPY RESULTS AS JSON button emits machine-checkable evidence:

{
  "verification": {
    "identicalPromptToBothSlots": true,
    "wavsDiffer": true,
    "wavsNotSilence": true,
    "spectrogramChangedA": true,
    "spectrogramChangedB": true,
    "framesA": 512,
    "framesB": 498
  }
}

• wavsDiffer — SHA-256 of both encoded WAVs, computed in the browser.
• wavsNotSilence — RMS > 0.0001 for both renders.
• spectrogramChanged* — a rolling window of frame checksums; a canned animation would collapse to one distinct value.

The backend has its own harness (npm run verify:api) that spins up a fake OpenAI-compatible provider and asserts the whole contract: fence extraction, reasoning-token detection and stripping, the empty-content retry, NO_AUDIO rejection, nonce uniqueness, the dead-Ollama message, and that the string reasoning_content never appears anywhere in a response.

What the spectrograms actually show

Run the "a hard techno loop" preset and both panels must show a periodic vertical pattern — a beat. That vertical stripe is the kick drum's broadband energy appearing once per interval; its spacing is the tempo; its height is the mix. An ambient drone instead shows slow horizontal bands that drift as the model detunes its oscillators. A lullaby in a minor key shows harmonic stacks — parallel horizontal lines whose spacing encodes the intervals.

When the duet plays, you are hearing two independent compositions collide. Sometimes it is a mess. Sometimes — this is the part that keeps you pressing PLAY DUET — the newer model's piece locks into a counterpoint with the older one's, and the two spectrograms braid.

What I would build next

• Tournament mode — bracket N models pairwise, advance by vote or by fingerprint score.
• Remix chain — feed Model B the code from A plus "improve this," and hear the delta.
• Spectrogram diff overlay — per-bin difference highlighting between A and B.
• Shareable runs — config + code in the URL, server-side render for OG audio embeds.

The deeper point: the browser's audio stack is a measurement instrument. AnalyserNode gives you ground truth about whether code did anything at all, and OfflineAudioContext gives you a reproducible artifact. Between those two, "the new model got better at code" stops being a vibe and becomes a waveform you can attach to a PR.

Code & more: https://www.dailybuild.xyz/project/265-reverb