DEV Community: Daniel García

0/55 three nights in a row: product bug or test bug?

Daniel García — Tue, 10 Nov 2026 08:49:01 +0000

Our nightly evaluation gate went 0/55 three nights in a row. The third time, the product was completely fine and the harness was the one lying. Telling those apart is the hardest testing problem we have hit this year, and the discipline that solved it is simple to state: no fix gets pushed until the red run has a named cause with evidence.

The setup. We sell conversational AI employees, so the thing that must work is a conversation. The nightly harness provisions a real employee through MCP for each of 55 cases across 6 suites (booking, student FAQs, email orders, ERP lookups, log review, YouTube backoffice), replays the conversation against production, and asserts two things per case: the right tools ran, and a judge model scores the reply against a rubric. The gate fails below 80% on product-backed cases. This is evaluation-driven development: the cases are written from the use case's intent, and the nightly is the reference curve, not the target.

Night one: 0/55, cause named. Every conversation died with a 401 on turn one. The client sent the standard X-Api-Key header; the auth code only matched a converted-underscores variant (x_api_key) because of how the framework lowercases parameter names. One customer-facing symptom, one line of auth code, every external client broken. Fixed, and a regression test now covers all three accepted spellings.

Night two: 0/55, a different cause. The policy engine's gate hook read arguments from a function signature that had been refactored away. Every reply became "name 'kwargs' is not defined". Two things made this one instructive. First, the error was invisible in the product: users just saw the fallback reply. Second, we had a day earlier merged a fix whose tests were green, because no test imported the module that broke. The suite now has a static check that fails when a handler references variables its own signature does not define.

Night three: 0/55, and the product was fine. Probing by hand, a booking conversation worked end to end: the agent checked the calendar, offered slots, booked, confirmed. But the gate still said 0.304. Reading the transcripts line by line found three harness defects stacked on top of each other. The fixtures (calendar events, client catalogs) were injected into the system prompt, so the agent never needed to call a tool, and tool-expecting cases failed with zero calls. A skill assignment wrote one table while activation read another, so assigned skills were silently inert. And the response contract changed shape, so the gate was reading evidence from an empty field. None of these were product bugs. All of them made the product look dead.

The discipline that paid. Each red run got a named cause with evidence before any fix was pushed: an API probe that reproduces the failure in isolation, or a transcript line that proves the claim. The alternative is what we did the first day: two theories, two pushed fixes, both wrong, four CI cycles wasted. Reading the actual log before theorizing sounds obvious. Under a red nightly at midnight it is not obvious at all; it is the only thing that works.

What the gate measures now. With the harness fixed, the number finally means what it says: the 0.304 pass rate measured a real gap, the gap between what the product serves today (booking, email handoff, lead capture) and what the case suites contract (ERP flows, invoicing, scheduled reports). That is not overfitting, that is a roadmap with a number on it. The gate only blocks on the product-backed suites, and the full 55 keep running as the curve.

If you are building anything conversational, the takeaway is one sentence: separate the judge from the judged, and never let either of them skip a named cause. The product behind this one is Optimly, and the nightly curve is how it earns the right to ship on a Tuesday afternoon.

One forward pass per level: replaying Next-Concept Modeling in the browser

Daniel García — Mon, 09 Nov 2026 16:07:46 +0000

Halfway through its 12 sweeps, a Next-Concept Model story is word salad with the characters already in place. I built a browser player that replays real generations level by level so you can watch that sharpening happen.

What NCM is. Next-Concept Modeling (NCM), from HazyResearch at Stanford, generates text coarse to fine (paper: arXiv 2610.14162). A multiscale residual VQ tokenizer represents a story as 12 nested sets of discrete codes, growing from 1 concept code to 128. A transformer predicts that whole hierarchy with one forward pass per level, not one token at a time.

What I built. A static page in vanilla JavaScript: no server, no GPU in the browser. The trajectories are real generations from the released TinyStories checkpoint (the generator is 356MB, the tokenizer 222MB), captured on CPU with a small script included in the repo. For each of the 12 sweeps the player shows the tokenizer's partial reconstruction of the codes generated so far. The same story, sharpening.

What the replays show. The split is sharp. By level 6 of 12 the partial decode is word salad, but the character names are already sitting in place. The text only becomes readable in the last three sweeps. Nine of twelve sweeps are not readable on their own: the model decides what the story is about early, and legibility arrives late.

Why a replay beats a demo video. A video plays once. A trajectory can be stopped at any level, scrubbed back, and compared sweep to sweep. You can see exactly when Lily and Ben appear, and what the model still gets wrong one level before the end.

Try the player here: NCM Trajectory. Code and the capture script are on GitHub. The paper is Next-Concept Modeling, arXiv 2610.14162 by HazyResearch; the trajectories come from the checkpoint they released, and the page explains the finding by itself.

The voice agent stack: what fits in a browser tab, and what you end up paying for

Daniel García — Mon, 09 Nov 2026 15:23:18 +0000

VAD, speech recognition and an 82M-parameter TTS now run free inside a browser tab. The interesting question is where that stops. Here is the line, and the cloud stack I would wire up on the other side.

Disclosure: three links below are affiliate links, marked with (af). If you sign up through them, I earn a commission at no extra cost to you. The criteria are the same ones I use when nothing is at stake.

What runs in the tab for free. Silero VAD as ONNX in the browser. Whisper-class ASR dueling live on your microphone. An 82M-parameter TTS (Kokoro) reading text in a tab, weights and all. All of it through transformers.js on WebGPU or WASM, no server, no API key, no bill per user. I keep these instruments up as static pages; open the latency waterfall and watch where a turn actually spends its milliseconds.

Where the tab ends. Three ceilings, in the order I hit them. First, the model: an 8B-class LLM fits in a tab, the model you want for real decisions does not. Second, the platform: a real deployment needs a domain, websocket endpoints, logs, a backend that survives a phone refresh; static hosting does not give you those. Third, the voice: an 82M TTS is impressive and still recognizably synthetic; brand-grade voices are server models, and the reference cascade in the literature measures them seriously.

The stack I would wire up on the other side.

Host the orchestrator on Railway: websockets, a domain and logs from a git push. The latency wins live in the event loop.
Serve open models through Novita AI: per-token API instead of GPUs and vLLM babysitting. OpenAI-shaped endpoint, so swapping providers is a config change.
Premium voice on ElevenLabs: the reference cascade measures 755ms to first audio with per-sentence streaming (arXiv 2603.05413). Premium voices are what you pay for.

Why this split is the honest default. In-browser parts have zero marginal cost per user, keep audio on-device, and ship as a static page that stays up. Cloud parts buy quality ceilings the tab cannot reach yet: cloud speech-to-speech sits around 702ms audio-to-audio in the current literature, and a well-tuned cascade of cheap parts can land close to that. Pay for the ceilings; run everything else in the tab.