DEV Community: Edy Cu

My codec benchmark failed my own gate. The cause was two quiet recordings.

Edy Cu — Sat, 17 Oct 2026 06:30:27 +0000

A parent's Hindi voice note hides the ask at 0:15: "send ₹5,000 before Sunday". The note is megabytes of audio nobody can search, and when a phone fills up, voice notes are what gets deleted. I wanted to keep the voice small and the ask impossible to miss, so I built Hearth for the Nebius × NVIDIA Global AI Hackathon.

The idea rests on one number I had not measured yet: how much does a 1.78 kbps neural codec hurt Hindi speech? The codec's model card has no Hindi result. So before anything else I wrote a gate: transcribe the original, encode it with NVIDIA's nemo-nano-codec-22khz-1.78kbps-12.5fps, decode it, transcribe it again, and require the round trip to cost no more than 5 word-error-rate points on 20 FLEURS hi_in clips.

The first run failed

Run 1, on a Nebius Serverless AI Job (CPU, 4vcpu-16gb): 8.74 % → 14.68 % WER, +5.94 points. The gate said +5. I missed it by 0.94.

Averages hide the cause, so I looked at the clips. Two of the twenty were very quiet (their loudest frames sat around −46 and −47 dBFS). After the round trip their word error went from 8.3 % to 62.5 % and from 38.9 % to 72.2 %: whole sentences gone. My guess for the first-word slips is that the causal decoder starts cold; lead frames alone did not help (+4.01 vs +2.61 points on the dev set). Thirteen of the twenty clips came back with no change in word error, five others added 6–17 points, and the two quiet ones added 33 and 54.

The fix is one line of audio engineering

My guess is that the codec expects speech at a normal level (its card does not say). So I measure the speech-active level of the input, bring it to −20 dBFS before encoding, store the gain in the file header, and divide it out on decode:

def codec_gain_db(x22, mode: str = "always") -> float:
    hop = 551                                   # 25 ms at 22.05 kHz
    frames = x22[: len(x22) // hop * hop].astype(np.float64).reshape(-1, hop)
    power = (frames ** 2).mean(axis=1) + 1e-12
    db = 10 * np.log10(power)
    active = power[db >= db.max() - 30]         # speech frames only
    rms_db = 10 * np.log10(active.mean())
    gain = min(MAX_GAIN_DB, max(MIN_GAIN_DB, TARGET_DBFS - rms_db))
    peak = float(np.abs(x22).max()) or 1e-9
    return round(float(min(gain, 20 * np.log10(PEAK / peak))), 2)   # never clip

I did not choose the setting on the benchmark clips. I ran an ablation on a disjoint dev set of 20 clips, with the rule fixed before the run (lowest round-trip WER delta): always-normalise plus three lead frames won with +1.60 points; the "only if quiet" variant was +2.20, and the run-1 pipeline (no gain, no lead frames) +2.61 on that set.

Run 2 on the 20 benchmark clips: 8.74 % → 12.83 %, +4.09 points. The gate passed. On a second held-out set (fleurs20b), never used for any decision: 12.42 % → 14.46 %.

It is a better average, not a fix for every recording. One normal-level clip got worse (13.0 % → 47.8 %), and I left that in the README. The first run's JSON is committed beside the passing one.

What the numbers buy

Stored size: 815,423 bytes an hour of voice (13 codebooks × 12.5 frames a second, packed 11 bits each), 195× smaller than 22.05 kHz 16-bit PCM. ASR real-time factor 0.16 on 4 vCPU, peak memory 6.9 GB, $0.1345 per audio-hour of compute (the whole-pipeline real-time factor 1.02 × the cpu-d3 list price, with start-up and model load excluded, which is most of the wall time: a note takes about 6.6 minutes from upload to kept at the median, cold start included).

The gate is not only a benchmark. In the product, the worker re-transcribes every note's reconstruction, and the app deletes the raw upload only if word drift is ≤ 5 % (and no 8-word-or-longer segment is over 30 %). Then it asks the bucket for the original and shows the HEAD … 404. The 60-second Opus note I use for the live runs drifts 14.7 %, so its original is kept. That is the gate working.

Two more walls, briefly

NanoCodec's checkpoint config also builds training-only parts: discriminators (one downloads microsoft/wavlm-base-plus) and a speaker encoder. Overriding the config drops them without changing a weight, and a fresh load goes from 45–50 s to 26 s with bit-identical codes:

cfg.discriminator = None
cfg.use_scl_loss = False
cfg.use_slm_loss = False
codec = codec_cls.restore_from(restore_path=path, override_config_path=cfg,
                               map_location="cpu", strict=False).eval()

And the Jobs API's 64 KiB limit applies to all injected files together (base64), not per file as the docs say. Three files that were each far under the limit were refused at 65,804 bytes of base64 in total. I gzip the worker and run.sh unpacks it. There is more of this in the repo's FEEDBACK.md, each item with a Job or request ID.

The model proposes, code decides

Reminders come from Nemotron 3 Super on Nebius Token Factory with a strict json_schema. Its card does not list Hindi, so nothing it says is shown unless code agrees: the quote must be a verbatim substring of the transcript, the amount is parsed from the quote's own number words, and the date is resolved from the quote and the note's own timestamp. In the live check, one proposal ("Thursday") whose quote never said Thursday had its date removed by code.

export function quoteGate(quote: string, segment: number, segments: Segment[]): QuoteGate {
  const q = normalizeHi(quote);
  if (q.split(" ").filter(Boolean).length < MIN_QUOTE_TOKENS) return { ok: false, reason: "quote_too_short" };
  // try the claimed segment, then its neighbours, then every segment
  ...
  return { ok: false, reason: "quote_not_found" };
}

Honest limits

CPU speed is ours to measure; neither NVIDIA card states CPU support. Time to kept is minutes. The round trip is lossy. Reminders are Hindi-only. The calendar feed is untested on a real phone, and the judge sample is FLEURS read speech, not a family's voice: no volunteer has handed me real notes yet.

Try it without an account: https://hearth.edycu.dev/judge · proof (Job IDs, the raw-audio 404): https://hearth.edycu.dev/proof · code (MIT): https://github.com/edycutjong/hearth · Devpost entry: https://devpost.com/software/hearth-4n0pw5

A regex matched my LLM pipeline on clean text. Then synthetic speech broke both.

Edy Cu — Thu, 15 Oct 2026 06:47:32 +0000

An operator on a plant radio says "Close valve two-oh-three, line Bravo." The field technician answers "Copy, closing two-oh-eight, line Bravo." Valve 208 is on a different line. The read-back exists so somebody catches that, and sometimes nobody on the channel does.

I spent the AMD Developer Hackathon (ACT III, lablab.ai) building SayAgain, a checker for exactly that moment. This post is not the launch tour. It is the part I didn't expect: on clean text a plain regex did as well as the language model, and when I fed the pipeline synthetic speech, both of them got worse in a way that a headline number hides.

Live replay: https://sayagain.edycu.dev/replay/ · Repo (MIT): https://github.com/edycutjong/sayagain · Demo video: https://youtu.be/6a4IF-iJmxg

The design: the model points, code decides

The obvious build is "ask an LLM whether the read-back matches the order." I didn't want a model making a safety verdict, so the model's only job is to point at words: which span of the transcript is the action, which is the equipment tag, which is the line. It runs on the AMD Token Factory (an OpenAI-compatible endpoint, JSON mode), and every call writes a receipt.

Then plain Python takes over. The spans must be words really present in the transcript (grounding). Spoken digits and NATO letters become plant tags through alias tables. The plant registry decides whether a tag exists and which line it is on. And a pure function returns the verdict:

def verdict(
    instr: list[CanonCommand],
    readback: list[CanonCommand] | None,
    *,
    span_conf: float | None = None,
    floor: float | None = None,
    disagreement: bool = False,
) -> Verdict:
    """VERIFIED · MISMATCH · NO_READBACK · ABSTAIN for one paired exchange (canonical inputs only)."""
    bad = validate_instruction(instr)
    if bad:
        return Verdict(ABSTAIN, bad, trace=[f"instruction: {bad}"])
    trace = ["instruction: " + ", ".join(f"{c.tag} line {c.line or c.registry_line}" for c in instr) + " (registry ok)"]
    if readback is None:
        return Verdict(NO_READBACK, trace=[*trace, "pairing: no read-back inside the window"])
    if any(r.tag_status == "ambiguous" for r in readback):
        return Verdict(ABSTAIN, R_AMBIGUOUS, trace=[*trace, "read-back: bare number shared by several tags"])
    if disagreement:
        return Verdict(ABSTAIN, R_AUDIO, trace=[*trace, "audio: prompted and unprompted transcripts disagree"])
    if floor is not None and span_conf is not None and span_conf < floor:
        return Verdict(ABSTAIN, R_AUDIO, trace=[*trace, f"audio: span confidence {span_conf:.3f} < floor {floor:.3f}"])
    diffs = compare(instr, readback)
    if diffs:
        return Verdict(MISMATCH, diffs=diffs, trace=[*trace, "compare: " + "; ".join(str(d) for d in diffs)])
    return Verdict(VERIFIED, trace=[*trace, "compare: every field equal"])

(referee/verdict.py — the module imports no model code.)

Notice how many exits lead to ABSTAIN. If the system can't read something, it says so and hands the card to a human. A failed model call is NOT_AUDITED, never a guess. Every card a supervisor sees ends with Confirm or Dismiss.

The rule I froze before the first model call

Before calling any model, I committed an evaluation protocol and tagged it (prereg-v1). The ladder rule: walk up from the cheapest extractor (a regex, no model) through larger models, and pick the smallest rung that lets zero wrong read-backs through as VERIFIED while keeping false alarms within a budget a shift can absorb.

That rule was written for the audio row: real recordings, real radio. I didn't have recordings during the hackathon, so I applied it to the recording cards' written scripts and logged that as a deviation. Everything below says which condition it is.

Result 1: on clean words, the regex was enough

On 89 held-out scripted exchanges (41 with a seeded error, 48 correct):

No rung passed a wrong read-back as VERIFIED. 0 for the regex and 0 for all four models on the AMD Token Factory.
The plain regex met the rule outright: 41/41 seeded errors flagged, 0/48 false alarms.
Gemma-4-31B-IT was the only model that met it: 0 passed, 2/48 false alarms — after one re-audit of 118 transcripts whose first call failed at the gateway (108 HTTP 500s, 10 time-outs).
An LLM-as-judge baseline (given the whole exchange, no registry) passed 2 wrong read-backs as VERIFIED, both missing the action word, and also cleared an order for a valve the plant doesn't have.

The honest reading: by my own pre-registered rule, on scripted words no language model is needed. I put that in the README in those words. It's the kind of sentence a project write-up usually leaves out, and it is the most useful thing the benchmark told me: clean text is not where the model earns its place.

One result did favour the design. When the same model returned normalised values instead of copying words from the transcript, false alarms went from 2/48 to 6/48; in 4 of those it turned a spoken pump or motor-valve tag into a valve tag the registry doesn't have. Making the model copy words, and making code check them, matters.

Result 2: synthetic speech broke both

To exercise the speech half, I ran an exploratory pass: ElevenLabs text-to-speech of the 145 card scripts (two voices), through a simulated radio path, transcribed by local Whisper-large-v3, then the same frozen referee. This was added after the text rows, it is not pre-registered, and it is not human speech. Same 89 held-out cards.

Still 0 wrong read-backs passed as VERIFIED in every row that ran.
Radio, no added noise: Whisper's word error rate was 0.87%. The regex flagged 39/41 errors, but false alarms went from 0/48 to 16/48; Gemma went from 2/48 to 17/48. Neither meets the false-alarm budget anymore.
Errors that vanish: at no added noise, 2 of 41 seeded errors raised no card at all. Whisper wrote "for three Bravo" and "tutu, hotel" for the tag in the instruction, so no order was heard and the wrong read-back had nothing to be compared against. At −6 dB machine noise, 26 of 41 seeded errors raised no card.

That third bullet is the finding. "0 wrong read-backs passed" stayed true the whole time, and it would make a clean headline. But an error that produces no card is not caught either; it just never reaches a human. The stricter form of my own rule counts those as failures, and the README now says so.

Where the false alarms came from is also worth knowing, because none of it is the model being clever or dumb:

Whisper writes spoken digit runs as "two, four, three", and the referee abstains on an unknown tag.
I run Whisper twice (with and without a vocabulary prompt) and abstain when they disagree; the unprompted pass mishears plant words, so the gate fires on correct read-backs.
On short or noisy turns Whisper invents text.

All of these fail toward a human, which is the right direction. They still cost the supervisor's attention, which is the budget the rule protects.

What I'd tell myself at the start

Write the selection rule down before the first model call, and keep the boring baseline in it. Without the regex rung I would have reported "Gemma meets the rule" and stopped.
"Zero false accepts" is necessary, not sufficient. Track the errors that never form a decision at all. A pipeline can be perfectly safe on what it judges and still miss what it never saw.
Clean-text evals flatter speech systems. The 0/48 → 16/48 jump came entirely from transcription, before any model judged anything.
Receipts make retries honest. The gateway failed most first-pass calls in one run (265 of 312), so re-audits were routine. Because every call is receipted and every result re-derives offline from stored outputs (scripts/eval.py --rescore), a re-audit is visible, not quietly merged.

Limits, plainly

No human recording has been evaluated. Every number here is written scripts or synthetic voices on a fictional plant registry.
The misheard-instruction failure needs a referee change, which means a new pre-registration, not a quiet patch.
The live link is a static replay of stored runs; the live queue (FastAPI + server-sent events) runs locally.

The next step is the row the rule was written for: real people, real radios, real plant noise.

Try the refused one: https://sayagain.edycu.dev/replay/ · Code: https://github.com/edycutjong/sayagain · Video: https://youtu.be/6a4IF-iJmxg

If you work with plant or field radio, I'd value one sentence on whether operators actually read back tag numbers and NATO letters like this.

git bisect run named the wrong commit 10 out of 10 times — so I gave every commit its own clean room

Edy Cu — Thu, 15 Oct 2026 06:42:26 +0000

git bisect run is the right algorithm. What I did not expect is how confidently it lies when you leave it running unattended.

I generated a 255-commit Python history in which a dependency pin (pytz) moves from 2022.1 to 2023.3 at commit 150, together with the golden value of the failing test. Each commit's test is right under its own pin and wrong under the other. A real bug is planted at commit 173. Then I ran git bisect run on it ten times, in one reused virtualenv:

How	Named	Right
env built at the known-good ref	the `pytz` bump	0 / 5
env built at the failing ref	a changelog edit	0 / 5
fresh venv at every step	the planted commit	5 / 5

Ten of ten wrong, both flavours. The fixture is built to drift, so that is by construction — what it shows is the mechanism: bisect trusts one environment for every commit, and a moving pin makes some verdicts false, so the search walks into the wrong half and convicts an innocent commit with total confidence. (On a real regression with no drift — python-attrs/attrs 22.1.0 → 22.2.0, 135 commits — every method was right.)

The fix is not clever: give every suspect its own clean room. Faultline does that on Nebius Serverless AI Jobs — one terminating job per commit, on the stock python:3.11 image, installing that commit's own declared dependencies from scratch.

The honest column

On the drifting history, Faultline named the planted commit in 3 rounds, 19 jobs, 957 seconds, $0.0948. A fresh-venv git bisect run did it in 57 seconds on a laptop.

Faultline is slower. A probe's VM takes about 4 minutes 20 seconds to start for about 5 seconds of work, and Nebius throttles job creation (HTTP 429, undocumented), so a round costs minutes where a local step costs seconds. It is not a faster bisect; it is one you can leave running without knowing about the drift or writing the fresh-environment harness. On a suite that takes minutes to build per commit, parallel rounds should start to pay for their cold starts — I have not measured such a repository, so I don't claim it.

Three walls

1. A job cannot tell you what happened. The Jobs REST API has no exit code and no logs, so a probe cannot report a test result to a caller. My first workaround was a read-write bucket mounted into every probe, with the container writing a verdict file. Then I remembered what runs inside a probe: a stranger's conftest.py. Anything mounted into the job belongs to the stranger — including every other hunt's records. Now each probe gets no mount and no key, only one presigned PUT URL for its own verdict object:

export function presignPut(config: NebiusConfig, key: string, expiresSeconds: number, date?: Date): string {
  const url = new URL(objectUrl(config, key));
  return presignUrl({ /* SigV4, method PUT, one key, expires */ });
}

The worst a hostile test suite can do with it is rewrite its own verdict — which it already controls by choosing its exit code. A side effect: a uk-south1 probe writes into an eu-west1 bucket over plain HTTPS, so regions are interchangeable.

2. Two jobs per region, and the third is rejected, not queued. A new tenant's compute.instance.count is 2; a third job returns QuotaExceeded. So concurrency is a parameter of the backend, not a constant: 2 slots × 4 regions = 8 probes. Each round tests the two ends of the range plus as many suspects as there are free slots, and N probes cut the range into N+1 segments.

3. No daemon, no database. The control loop is a tick that every finishing probe pings. Two ticks can race, so the hunt record is replaced only with If-Match, and every dispatch is claimed first with If-None-Match: "*":

const header: Record<string, string> =
  "ifNoneMatch" in condition
    ? { "If-None-Match": condition.ifNoneMatch }
    : { "If-Match": condition.ifMatch };
const res = await s3Request(config, "PUT", key, body, {}, fetchImpl, header);
if (res.status === 412 || res.status === 409) return { ok: false, status: res.status };

I measured it on the real bucket before building on it: 20 races of 2–8 concurrent creators on one key, exactly one winner every time.

Where the model comes in — and where it doesn't

When test execution has named the culprit, one call goes to NVIDIA Nemotron 3 Super on Nebius Token Factory with the diff and the failing output, and it writes the root-cause paragraph. It is a reasoning model: with a small token budget it can return its reasoning and no answer. So the call asks for low reasoning effort with room to answer, reads content only, and treats an empty answer as an error rather than inventing one. The UI labels the paragraph as generated, next to a culprit labelled as proven by executed tests. The model explains; it never decides. (On the hero hunt it also got a detail wrong — it called UTC−6 "DST-adjusted" for Mexico City, which abolished DST in 2022 — and the project's DEMO.md says so.)

Limits

Python test suites and public GitHub repositories only. One live hunt at a time on the shared slots, with a daily cap. Slower than a careful local bisect on a fast suite.

You can watch a recorded real hunt (real job IDs, no cost) or run a live one, no login: https://faultline-edy-cus-projects.vercel.app. Code (MIT): https://github.com/edycutjong/faultline. Built for the Nebius Global AI Hackathon.

A CNN on every frame caught 2 of 54 wildfire windows at a fixed alert budget. One 20-line baseline made it 18.

Edy Cu — Tue, 13 Oct 2026 11:11:39 +0000

Wildfire lookout networks have far more camera feeds than anyone can watch. The useful question isn't "is there smoke in this frame?" It's "which of these dozens of tiles should a person open next?", asked without burying them in alerts until they mute the tool.

I built PlumeRank around that question for the OpenCV AI Competition 2026. The most useful thing I learned wasn't in my own pipeline. It was what happened when I gave a baseline the same treatment.

Live, no login: https://live.plumerank.edycu.dev · Code: https://github.com/edycutjong/plumerank

Fix the budget first, then compare

Every system in the comparison gets the same allowance: 2 alerts per camera per 81-minute window, which is the length of one FIgLib sequence. Each one is calibrated to spend that allowance on a fire-free confusion set of 33 windows (low cloud, cloud shadow, overcast, cameras hunting exposure), and only then scored on fire footage.

The allowance is enforced by a small, deliberately strict object:

def spend(self, camera_id: str) -> bool:
    """Spend one alert on this camera. False when its allowance is already gone."""
    if camera_id not in self._spent:
        raise KeyError(f"{camera_id!r} is not on the roster this budget was built for")
    if self._spent[camera_id] >= self._per_camera:
        return False
    self._spent[camera_id] += 1
    return True

An unknown camera raises instead of being auto-registered. A typo that silently created an extra camera would inflate the denominator that the whole comparison rests on.

The threshold isn't chosen by hand. solve_threshold finds the lowest value whose realised alert rate on the confusion set stays within the budget. A system can't look fast by alerting more, because it isn't allowed to.

The result that surprised me: 2 → 18

My first baseline was the obvious one: run the same small CNN on every frame and alert when its probability crosses the solved threshold.

At the fixed budget it detected 2 of 54 held-out fire windows.

The reason is in the table footnote: the solved threshold landed within 1e-6 of 1.0. On a fire-free set full of cloud and haze, the CNN's probabilities saturate. The only way to keep it to 2 alerts per camera-window was to demand near-certainty, and almost no real fire reached that.

PlumeRank doesn't rank a camera by its absolute score. It ranks it by how far the camera has moved from its own recent past:

def rebase(hazard, past, *, warmup=BASELINE_WARMUP_FRAMES,
           sd_floor=BASELINE_SD_FLOOR, z_scale=BASELINE_Z_SCALE) -> float:
    """One camera's hazard this minute, relative to the hazards in ``past``.

    ``past`` is the trailing window and must NOT contain ``hazard`` itself."""
    n = len(past)
    if n < warmup:
        return 0.0
    mu = sum(past) / n
    var = sum((v - mu) ** 2 for v in past) / n
    sd = max(math.sqrt(var), sd_floor)
    z = (hazard - mu) / sd
    return 1.0 / (1.0 + math.exp(-z / z_scale))

The window is 15 frames (15 minutes) per camera. A camera staring into haze all afternoon competes with its own hazy afternoon, not with a camera under clear sky.

For a fair comparison I put both baselines through that same function. The CNN-on-every-frame went from 2/54 to 18/54. Frame differencing went from 4/54 to 20/54.

If you run any detector on fixed cameras, comparing each camera with its own recent past is probably the cheapest improvement available. It is about twenty lines and needs no training data.

The pipeline (OpenCV 5)

Per camera, per frame:

Stabilise: Hanning-windowed cv2.phaseCorrelate for the coarse shift, refined with cv2.phaseCorrelateIterative, and an estimateAffine2D fallback when the phase peak is weak.
Segment: MOG2 background subtraction finds what moved.
Describe: four cheap plume statistics on the moving region, fused into one hazard.
Rebase: the function above.
Escalate: a small classifier on cv2.dnn, only where it can change the answer.
Spend: one solved threshold, one budget.

Step 5 is where the cost goes, so the gate is two conditions in one line:

def should_escalate(rank: int, hazard: float) -> bool:
    return rank <= ESCALATION_TOP_K and hazard >= ESCALATION_HAZARD_MIN

ESCALATION_TOP_K is 2, so at most two cameras per tick can ever reach the network, whatever the hazards do. The hazard floor is solved on the fire-free set and never tuned against fire windows. In practice the network runs on 1.76% of frames.

Everything runs on one AWS t4g.small (Graviton/ARM), replaying real HPWREN footage through the real pipeline.

The held-out run

The constants were frozen in their own commit and hashed, and the bench refuses to read a held-out frame unless the hash matches. The held-out set was read once.

System (same budget, same per-camera baseline)	Detected
PlumeRank	27 / 54
Frame differencing	20 / 54
CNN on every frame	18 / 54

Cost: 1.4 neural-net calls per camera-window for PlumeRank, against 3,645 for the CNN on every frame.

What doesn't hold up

These are on the README's first screen too, because I'd rather say them than have them found:

The detection lead isn't significant. Paired McNemar p = 0.18 and 0.25 against the two rebased baselines, and approximate, because 22 of the 54 windows come from fires that contribute more than one window. The cost gap is the decisive result; the detection count is a lead.
Half the held-out fire windows are missed (27 of 54).
The median is on a knife edge. 38.0 min is a nearest-rank median: the 27th and slowest detection. Shift the ignition clock a few minutes later and there's no median at all. Detected fires themselves are found sooner (median 10.0 min).
Daylight only. No IR, no night detection.
The statistic I designed the project around is anti-predictive on its own (AUC 0.44 on development frames). The per-camera baseline carries the ranking, which is part of why the 2 → 18 result matters more to me than my own number.

Try it

Live wall: https://live.plumerank.edycu.dev (30-second judge path: /judge)
Repo: https://github.com/edycutjong/plumerank. 833 tests; make verify re-checks the micro-corpus metrics byte-for-byte in about 3 minutes.
Walkthrough video (3.5 min): https://youtu.be/QqX85JuNfVM

Thanks to HPWREN and the FIgLib authors. Their imagery is what made any of this measurable.

If you've worked with fire cameras, FIgLib or SmokeyNet, I'd like to hear where this reasoning breaks.

Amazon's own BIF file stores a timestamp unit of 0xFFFFD8F0, and the obvious fix would break it

Edy Cu — Tue, 13 Oct 2026 11:11:33 +0000

I wanted a seek bar that knows where the goals are, without a feed and without the scan decoding any video. (Disclosure: I directed Claude Code agents that wrote most of this code; I reviewed and decided.) A Vega player that shows BIF scrub previews, like Amazon's sample, already downloads a BIF file, one small JPEG about every ten seconds of the whole asset. So I treated that file as a contact sheet of the match, and wrote a detector that watches the score graphic across the thumbnails.

The detector works, on one development match. The more useful thing I found was in the BIF itself: Amazon's own hosted sample file has a timestamp header that the sample parser only survives by accident, and the "obvious" fix would have broken it.

The project is Flipbook, built for the Amazon Developer Hackathon (Fire TV track) on the Vega Virtual Device. It starts from a copy of Amazon's vega-video-sample. The 2:54 demo, recorded on the virtual device:

Landing page and pitch deck: flipbook.edycu.dev.

The header that says four billion milliseconds

A BIF is a 64-byte header, an index of (timestamp, offset) pairs, then JPEGs. The header stores a "timestamp multiplier": milliseconds per timestamp unit. I fetched the first 32 bytes of the file the sample streams, tv-webinar-hd.bif, with an HTTP Range request:

00000000: 8942 4946 0d0a 1a0a 0000 0000 ac01 0000  .BIF............
00000010: f0d8 ffff 0000 0000 0000 0000 0000 0000  ................

428 frames (0x1AC), and a multiplier of 0xFFFFD8F0. Read as an unsigned 32-bit number that is 4,294,957,296. The sample's parser multiplies every timestamp by it, so each frame is "about 4.29 billion milliseconds" after the last. The player still shows sensible previews because the sample's lookup ignores the parsed times and assumes one frame per ten seconds.

Look at it as a signed 32-bit integer and it is −10,000: ten seconds, with the sign wrong. That is my inference. I cannot see how Amazon's file was made, and I am not claiming to know. But it matches the ten-second spacing the file actually has.

The catalogue entry for that title also says duration: 190, for a 71-minute video. The friction log has the details (entry 6).

Why the obvious fix is wrong

My plan for an upstream fix was to make the sample's thumbnail lookup binary-search the parsed positionMillis. That is how a BIF is supposed to work. On tv-webinar-hd.bif the parsed times are the 4.29-billion-millisecond values above, so a search by position cannot work, and the fix would break the one file Amazon's own sample ships. A fix that works on the spec and fails on the reference asset is not a fix you can send upstream, so the reader has to decide, per file, whether the header can be trusted.

Flipbook's reader (src/flipbook/core/bifIndex.ts) does that (the snippet is abridged from it). A multiplier is trusted only if it is between 1 ms and 60 s; otherwise the unit comes from the asset's duration, and failing that from the sample's own ten-second assumption:

if (rawMultiplier === 0) {
  unitMs = 1000; // Roku spec: 0 means 1000 ms
  intervalSource = 'header';
} else if (isSaneMultiplier(rawMultiplier)) {
  unitMs = rawMultiplier;
  intervalSource = 'header';
} else if (opts.durationMs && opts.durationMs > 0 && timestamps[frameCount - 1] > 0) {
  unitMs = opts.durationMs / timestamps[frameCount - 1];
  intervalSource = 'duration';
} else {
  unitMs = FALLBACK_UNIT_MS; // 10 s
  intervalSource = 'fallback-10s';
}

The same function refuses a file with no 0xffffffff terminator entry. The sample logs a warning for that and then trusts the trailing offset, so the last frame is read from a corrupt range (entry 7).

The detector, in one paragraph

It is arithmetic, no ML. The viewer draws a box around the broadcaster's score graphic once. For each thumbnail Flipbook decodes the JPEG on the device with jpeg-js, takes a 64×5 luma grid of the box, and flags a change only if the new state holds for most of a three-frame window, so a replay or a flicker does not count. A first pass reads every eighth frame and bisects where something changed, to put the first ticks up quickly; a second pass reads all frames. Pressing fast-forward jumps to the marker minus 30 seconds, anchored on the last frame that still shows the old score, so you land before the play. By default the seek bar shows no ticks and the thumbnails are masked over the score box, so the result is not spoiled while you hunt.

What I measured, and what I did not

On a development match (a CC0 high-school soccer broadcast, 517 thumbnails), labelled by the detector's author (an AI agent working under my direction) — so not independent ground truth:


Changes found	2 of 2
False markers	0
Jumps that landed before the play	2 of 2 (28 s and 30 s early)
Frames read in pass 1	77 of 517 (14.9 %)
Score box vs. noise	2.64× at 320 px, 2.04× at 240×180 letterboxed

A plain frame differ finds both changes only at a threshold of 30 or lower (of the 8–80 I swept), and at 30, the best of those, it still fires 237 false markers. My first signature, a 16×4 mean, could not separate a goal from noise on that match, which is why there is a version 2.

On the Vega Virtual Device on an Apple M-series Mac, the same development match, calibrated with the D-pad and relaunched 19 times: pass 1 (77 of 517 frames) p50 2,771 ms, p95 3,422 ms; pass 2 p50 14,294 ms; the same two markers as the Node run, and the calibration box restored from storage after every relaunch. Earlier, the first detector version read 203 of 428 frames of Amazon's BIF in 4,085 ms, about 20 ms per decoded frame. I have no Fire TV hardware, so there is no real-device timing here, and the scan decoded no video bytes (the player still plays the video).

Update, 2026-10-15: the held-out result now exists, with a caveat. No person labelled the two test matches in time, so the labels come from two separate AI sessions that were instructed to use only the video (not the code, the fixtures or any detector output). They were committed before the detector ever ran on those matches, and the change of labeller is recorded in the pre-registration file. On held-out match B, 2 of 3 goal jumps landed before the play (95 % CI 0.094–0.992), with 0 false markers; the third goal's score change scored below the threshold and was missed. Three goals is too few to say how well it generalises, and these are AI labels, not a person's — the README's honest limit 2 has the details.

When I first wrote this, I did not have a held-out result. Two more matches are frozen behind a pre-registration file, and bench refuses to run on them until a filled-in ground-truth file that names its labeller is committed. The plan is that a person other than the detector's author does the labelling; the guard checks that the file is committed, filled in, for the right match and names its labeller, but it cannot check who that person is. Until those labels exist, I am not claiming a headline number, and the README says so.

Reproduce it

git clone https://github.com/edycutjong/flipbook && cd flipbook
npm ci --ignore-scripts && npm test   # 448 tests, 97 of them Flipbook's (skips the Shaka build only the device build needs)
npm run bench                    # the development-match numbers above, 20 runs

What I would ask Amazon for

A frame-grab or pixel read-back API on the player surface, so apps are not limited to the frames inside a BIF; a BIF lookup that uses the file's own interval; and a corrected hosted tv-webinar-hd.bif. The full friction log is in the repo, 18 entries.

An agent that refuses: building a payout agent with neatlogs, Entire and cfo.ai

Edy Cu — Sun, 11 Oct 2026 10:15:42 +0000

I won a $7,000 hackathon prize. The payout form offered me Wise. Wise's own help page says that personal accounts with an address in Indonesia "can no longer hold money or currencies in their account."

That is the kind of mistake I built Cair to prevent: an agent that plans how a prize or client payment reaches your bank account, and refuses routes that cannot deliver. It was built for neatHack, a 48-hour agent hackathon, with neatlogs (traces), Entire (a local code graph) and cfo.ai (the business plan). Code: github.com/edycutjong/cair. Live: app.cair.edycu.dev.

This post is about the part I did not expect to matter most: how little the model is allowed to decide.

The baseline is a confident paragraph

I started by measuring the obvious answer: one call to the model, no tools. I froze 48 scenarios (3 countries x 4 payers x 3 amounts, plus 12 leading prompts) and computed the correct labels from a sourced rules sheet before writing any planner code. Then I ran the one-call baseline three times.

	v1: one call	v2: Cair
Recommends a route its own provider rules out	18 / 48, every run	0 / 48
Contradicts a sourced tax fact	8 / 48, every run	0 / 48
Says UNCLEAR where no primary source exists	11-23 / 143 fields	143 / 143
Figures traceable to a tool output	n/a, no tools	359 / 359

The baseline recommended Wise for every Indonesian Devpost winner in all three runs. A second baseline that gets the entire rules sheet pasted into its prompt still recommended a dead route twice and stated an unsupported tax position 11 times. The model had the facts in front of it and still wrote a confident answer around the gaps.

(The caveat that matters: this measures fidelity to the sourced sheet, not tax correctness. The labels and the agent read the same sheet.)

So the code decides, and the model only words it

Cair runs a fixed evidence plan before any prose: what the payer offers, whether each offered option can receive in your country, the US treaty position, the ECB reference rate, then Decimal arithmetic. Each step is a neatlogs span, so every plan is one trace:

@neatlogs.span(kind="TOOL", tool_name="fx_quote", description="ECB daily reference rate USD -> local currency")
def fx_quote(quote_ccy: str) -> dict:
    _maybe_fail("fx_quote")
    return fx.live(quote_ccy)

The route and W-8BEN line 10 come from sourced rows, never from the model. The model writes the explanation, and then a provenance guardrail checks it: any money amount, rate or treaty article that is not in a tool output blocks the answer, which gets one retry and then a template built from the tool outputs.

with neatlogs.trace("provenance_guardrail", kind="GUARDRAIL") as sp:
    verdict = check(text, tool_outputs, card, question)
    sp.set_attribute("neatlogs.guardrail.passed", verdict["passed"])
    sp.set_attribute("neatlogs.guardrail.score", float(verdict["score"]))

Because the guardrail is a span, every blocked answer is searchable in neatlogs. That turned out to be the debugging tool.

What trace search found

search_traces "provenance_guardrail BLOCKED" over the neatlogs MCP server returned the v2 plans whose explanation had been rejected. get_trace_context showed the model's text next to the guardrail verdict. Two real bugs came out of it:

A correct answer was blocked because my own figure rule read 2000.00 USD as a year. Any amount from 1,900 to 2,099 written without a thousands separator was affected. A false block, not a false pass, but I would not have seen it without the span.
Refusals that repeated the user's own "0%" ("I can't say the withholding is 0%") were blocked twice and fell back to the template. I considered allowing figures that appear in the question inside negated sentences, and rejected it: it would let "the withholding is not 30%, it is 0%" through.

Both are fixed, each pinned by a test named for the defect, and re-run. The write-up with the trace ids is in the repo (evidence/rca-01.md).

Entire's local code graph was the second tool in that loop: after the year fix, entire graph search given the bug in plain words ranked the corrected rule first and its test second. It was also wrong in an instructive way: impact on rail_eligibility reported 0 callers, probably because the planner passes tools as function values (att.call("fx_quote", tools.fx_quote, ...)), which the graph does not follow. I recorded that as found rather than hiding it.

When a tool fails

The hackathon asked for a failed-action recovery run, so I made one on purpose. scripts/eval.py --planner v2 --fail fx_quote makes the FX tool raise on every call:

for attempt in (1, 2):
    try:
        return fn(*args)
    except Exception as e:  # the failed span is already marked ERROR by the SDK
        ...
return fallback(last) if fallback else None

The agent retries once, then falls back to the last ECB rate it fetched, stamped stale with its date, and finishes. In the 24 USD-payer scenarios that meant 24 fallbacks; all 48 plans completed, 0 undeliverable routes, 350 / 350 figures sourced. The neatlogs trace shows two ERROR spans and the fallback span, and the dashboard footer says "ended with error" because it counts the errored tool spans, while the workflow and agent spans succeeded. I kept that on the page instead of tidying it. The agent never asks the model for a rate.

What it costs

Measured tokens at list price: $0.0033 of model cost per plan, p50 5.50 s and p95 10.31 s on my laptop. In cfo.ai the base scenario (assumed prices, planned and not built) costs $18.887 a month to run, and the $25 AWS credit lasts 1.35 months before any revenue. The model is shared here: cfo.ai/s/PQ2HUGmiJl3P0RQB.

Limits

Three countries are fully sourced (Indonesia, India, the Philippines). Everything else is UNCLEAR by design.
The guardrail is rule-based. On a fresh held-out set it blocked 62 / 66 violations and 3 / 66 honest sentences, before I fixed its misses. Expect new phrasings to slip through at roughly that rate, which is why the route and every amount are decided by code.
Payoneer's receiving fee is not published as one number, so "what lands" is an upper bound.
Receipts so far are my own: one replay of a known payout (at most $4,900 promised, $4,897 arrived) and one test receipt. Forward receipts from real users: none yet.
Cair explains forms; it never files them. Not tax advice.

Repo, evidence and the commands to reproduce every number: github.com/edycutjong/cair.

My memory app marks 'Vitamin D' wrong for 'Vitamin C'. Unless you say it out loud.

Edy Cu — Mon, 05 Oct 2026 06:31:46 +0000

You're learning the vitamins for an exam. At the stop for Vitamin C, you say "Vitamin D". The app turns the stop green. You just practised the wrong answer.

That app is mine. It's called Loci, and I found this while writing this post. Type "Vitamin D" at that stop and Loci marks it wrong, as it should. Say it out loud and the stop lights up.

Loci's answer checker has one rule: forgive the typo, never the wrong answer. This post is how that rule was built and tested, and how the voice path got around it.

Live: https://loci.edycu.dev
Code: https://github.com/edycutjong/loci (MIT)
Demo video (2:30): https://youtu.be/I5sx5C28xR0

What Loci does

Loci is the memory palace, with your own room as the palace. Take a photo of your room and paste a list of 3 to 12 items, in order. An AI finds the objects in the photo. Plain code picks one object per item and joins them into one route. A second AI call writes a short, strange scene for each stop.

Then the lights go out. At each stop you say or type the item. A right answer turns the pin green and lights that part of the room again.

The real app, recorded from the production build. The room is AI-generated, and a script types the answers.

The AI proposes. Plain code decides. Whether you remembered is decided by shared/score.ts, plain TypeScript, never by the AI.

The first rule: one typo budget for the whole answer

The first rule in my planning notes was simple. Allow typos up to 20% of the answer's length, at most 2, anywhere in it. The repo keeps that rule in a regression test:

// tests/regressions.test.ts: the planning notes' first rule
const wholeAnswerRule = (said: string, target: string) => distance(normalize(said), normalize(target)) <= Math.min(2, Math.floor(0.2 * normalize(target).length));

"vitamin c" is 9 characters with the space. 20% of 9, rounded down, is 1 typo. "vitamin d" is exactly 1 typo away, so it passes. "Henry VII" passes for "Henry VIII" the same way.

That rule never shipped. Before any checker code existed, the plan listed pairs that must be wrong, and these two broke the rule on paper. The plan was made with the Devpost Learn skill pack, and it's in the repo's devpost/ folder.

The fix: the same 20%, per word

The fix kept the 20%. It moved it from the answer to each word:

// shared/score.ts
/** Typos forgiven in one word: 0 for words of up to 4 letters, 1 for 5–9 letters, 2 for 10 or more. */
export const allowance = (wordLength: number) => Math.min(2, Math.floor(0.2 * wordLength));

/** At most this many typos across a whole answer. */
export const MAX_TOTAL_TYPOS = 3;

/** Is `answer` close enough to one target text? */
export function closeEnough(answer: string, target: string): boolean {
  const a = normalize(answer);
  const t = normalize(target);
  if (!a || !t) return false; // an empty answer is never right
  if (a === t) return true;
  const aWords = a.split(" ");
  const tWords = t.split(" ");
  if (aWords.length !== tWords.length) return a.replaceAll(" ", "") === t.replaceAll(" ", ""); // "oatmilk" = "oat milk"
  let total = 0;
  for (let i = 0; i < tWords.length; i++) {
    const typos = distance(aWords[i], tWords[i]);
    if (typos > allowance(tWords[i].length)) return false; // "vitamin c" ≠ "vitamin d": short words must be exact
    total += typos;
  }
  return total <= MAX_TOTAL_TYPOS;
}

Word	Typos forgiven
up to 4 letters	0
5 to 9 letters	1
10 letters or more	2
the whole answer	3 at most

"Vitamin" may have one typo. "C" may have none. So "occulomotor" counts for Oculomotor, and "Vitamin D" never counts for "Vitamin C".

Before comparing, normalize lowercases both texts and drops accents, punctuation and numbering like "1.". distance counts two swapped letters as one typo.

60,000 answers with known verdicts

Unit tests pin the edges: a word at its allowance passes, and one more typo fails. On top of that, fast-check generates 60,000 answers per run, across 6 properties.

Each answer is built so its verdict is known without asking the checker. Change, drop or add one letter in a word of up to 4 letters, and the answer must be wrong:

// tests/score.property.test.ts (itemWith, said and replace are small helpers at the top of the file)
it("a word of up to 4 letters must be exact: one letter changed, missing or added is never right ('Vitamin D' for 'Vitamin C')", () => {
  const edit = fc.record({ kind: fc.constantFrom("change", "drop", "add"), at: fc.nat(), shift: fc.integer({ min: 1, max: 25 }), letter: fc.constantFrom(...ALPHA) });
  fc.assert(
    fc.property(itemWith([1, 4]), edit, ({ words: ws, i }, { kind, at, shift, letter }) => {
      const w = ws[i];
      const p = at % w.length;
      const offByOne =
        kind === "change" ? w.slice(0, p) + ALPHA[(ALPHA.indexOf(w[p]) + shift) % 26] + w.slice(p + 1)
        : kind === "drop" ? w.slice(0, p) + w.slice(p + 1)
        : w.slice(0, p) + letter + w.slice(p);
      return !closeEnough(said(replace(ws, i, offByOne)), said(ws));
    }),
    { numRuns: RUNS },
  );
});

Every one of those 60,000 answers goes through closeEnough. Keep that in mind.

Voice needs more ways to say yes

Speech is messier than typing, in three ways:

Split words. Speech-to-text can break a long word in two: "vestibular cochlear" for Vestibulocochlear.
Sound-alikes. Loci's scenes use them on purpose, as memory hooks: "trochlear" sounds like "truck-lear".
Echoes. When Chrome's on-device speech model is installed, Loci passes it your list as hints. While recording the demo (recorded audio, not a live microphone), its final answers echoed them about once in a hundred words: "Cambrian Jurassic Quaternary" when only "Quaternary" was said.

A scene with its sound-alike. The room is AI-generated.

So for each stretch of speech, the voice path tries the typing checker first, then two more rules:

// shared/voice.ts
/** Is this stretch of speech the item (or one of its accepted answers)? */
export function spokenMatch(spoken: string, item: Item): boolean {
  for (const target of [item.text, ...item.accepts]) {
    if (closeEnough(spoken, target)) return true;
    const a = normalize(spoken).replaceAll(" ", "");
    const b = normalize(target).replaceAll(" ", "");
    if (b.length >= 5 && distance(a, b) <= allowance(b.length)) return true; // "vestibular cochlear"
    if (b.length >= 5 && soundsAlike(a, b)) return true; // "truck lear"
  }
  return false;
}

The sound-alike rule compares consonant skeletons. It drops the vowels, h, w and y, and merges letters that sound close, so "truck lear" and "trochlear" both become trklr. Keys shorter than 4 letters never count. For echoes, a second reading skips leading words that name stops you already answered.

Voice is forgiving in one more, safe way. A phrase that matches nothing in your list shows "didn't catch that", and it costs nothing.

The old rule came back

Look at the second rule again. Join the words, then allow typos by the length of the whole thing. That's the first rule, the one the plan threw out. It came back in the voice slice of the build, the same day, under a comment about "vestibular cochlear".

// what the two functions return at commit d0e5df9
import { closeEnough } from "./shared/score";
import { spokenMatch } from "./shared/voice";

closeEnough("Vitamin D", "Vitamin C");                         // false: typed, the stop stays dark
spokenMatch("vitamin d", { text: "Vitamin C", accepts: [] }); // true: said, the stop turns green

"vitamind" against "vitaminc" is 8 letters, so 1 typo is allowed, and 1 is used. The full voice interpreter agrees: at the Vitamin C stop, hearing "vitamin d" returns { kind: "right", stop: 0 }. "Henry VII" and "Henry VIII" pass for each other the same way, in both directions. So would two numbered items one digit apart, like "Type 1" and "Type 2". The sound-alike key drops digits too.

Step 7 of the diagram above says "short words exact" and names two files. Only shared/score.ts keeps that promise.

Why didn't the tests catch it?

The 60,000 answers only test closeEnough. spokenMatch calls it first, then has two more ways to say yes. No property runs through those.
The voice tests use the example lists. They check that no two items in the same list match each other. That passes, because no example list has two items that close.
The echo fix's regression test even uses Vitamin C and Vitamin D. It checks that "vitamin d" still counts at the Vitamin D stop. It never asks what happens at the Vitamin C stop.

A list of pairs that must fail only protects the functions you run it through. Every path that can say yes needs the whole list.

Update (2026-10-05): fixed in commit 60f6fd4. The two extra voice rules now run only when every short word of the item was heard exactly, and the must-fail pairs run through spokenMatch and the interpreter in their own test. The price: voice is now as strict as typing on short words. "World War 2" no longer counts for "World War II" unless you add it as an accepted answer.

The AI half, briefly

The AI does two jobs: find the objects in your photo (Gemini first, with fallbacks) and write the scenes (DeepSeek first). Code checks both answers before anything uses them. Malformed boxes and empty scenes are dropped, never repaired.

One prompt fix from the build: with only a rule telling it to spell the item exactly, DeepSeek dropped the item's exact spelling in 8 of 12 scenes that used a sound-alike. One worked example in the prompt fixed it:

- Write the item itself, spelled exactly as given, in every scene, even when the image rests on a sound-alike. Example for the item "Trochlear" on a wall clock: "Your wall clock turns into a little truck-lear, honking 'Trochlear!' as it circles the room."

On the live site, a script built 9 palaces the way a person would: 3 AI-generated rooms, each uploaded as if it were your own photo, times 3 lists. Nothing was stubbed.

9 / 9 palaces built
102 / 102 scenes spell their item
median wait 14.45 s from "Build my palace" to the first scene, slowest 30.02 s
$0.0115 billed for all nine ($0.1151 at list prices; the Gemini calls ran on a free-tier key)

The full record, with every model's answer, is in DEMO.md.

Limits

One sitting only. Loci shows a first-try score, learning time and recall time. It makes no claim about next week and has no spaced repetition.
Voice is Chrome and Edge only, in US English. It's checked with a stand-in recognizer and with recorded audio, never with a person at a microphone.
The example rooms are AI-generated. Real rooms are messier.
Your photo goes to an AI once (Gemini, or DeepSeek if Gemini is busy). A free-tier Gemini key lets Google use what it receives, so leave people and papers out.
Typing is strict about extra words. "optic nerve" stays dark for "Optic". Add it as an accepted answer: Optic / Optic nerve.

Try it

Open https://loci.edycu.dev and tap Try it: 12 cranial nerves. No photo, no sign-up, no AI call. Tap Next stop a few times, then Lights out, and type what you remember. In Chrome or Edge, Say it instead switches to voice.

Code: https://github.com/edycutjong/loci
The 30-second path for judges: https://loci.edycu.dev/judge/
Story page: https://loci.edycu.dev/story/
Devpost: https://devpost.com/software/loci-4isou6

Loci was built for Build With AI: Basics on Devpost.

If you write a fuzzy matcher, write the pairs that must fail before you write the rule. Then run them through every function that can say yes, not just the first one.

I wrote this with help from an AI assistant, working from the repo's code and docs. The rooms in the images are AI-generated, and the demo video uses an AI voice.

My first-choice AI model said no to 5 of 6 room photos. The app still answered each one in about 4 seconds.

Edy Cu — Mon, 05 Oct 2026 03:20:25 +0000

After a stroke, some people with aphasia understand everything but can't find the words.

Hospital picture cards (EAT, DRINK, TOILET) look nothing like their life. Another kind of board is made from a photo of the person's own surroundings. In AAC (tools that speak for people who can't), it's called a visual scene display. Someone has to build it by hand, object by object.

Room to Speak does that part with AI. A caregiver takes one photo of the room where a parent with aphasia spends the day. A vision model finds up to 8 things he'd want to talk about and writes a short first-person phrase for each. Rings appear on the photo. He taps the kettle, and the phone speaks for him, with a phrase like "Aku mau kopi" (I'd like coffee).

Real AI output on an AI-generated example bedroom, answered by deepseek-flash (step 3 of the ladder below). The wait is trimmed and the GIF has no sound.

Try it at roomspeak.edycu.dev (tap an example room, no login), read the code on GitHub (MIT), or watch the 1:48 demo. All room photos in this post are AI-generated.

Everything he does every day runs on the device. Only setup needs the internet, and setup is one AI call. On 2026-10-04 I sent six room photos to the live function. My first-choice model said no to five of them. All six still came back with spots, with a median wait of 4.03 s, for $0.00.

This post is about the code behind that, the bug that taught me to write it, and what happens to the answer once it arrives.

One call does all the AI work

The browser shrinks the photo to 1600 px at most and posts it to one Vercel function, api/detect.ts. It's the only server code, and the API keys live there, so they never reach the browser. (A test builds the app with canary keys and checks that no built file contains them.)

The prompt in shared/prompt.ts asks for structure, not prose:

Pick up to 8 objects he would most plausibly want to talk about for daily needs, comfort,
people, or activities (e.g. kettle/cup -> a drink, TV/radio -> turn it on, window -> fresh air,
bed/chair -> rest, family photo -> a person, medicine, fan/AC, door, phone).
Skip structural or trivial things: walls, floor, ceiling, ceiling lamps, outlets, cables, generic clutter.
Only include objects that are clearly visible; fewer good spots beats 8 weak ones.

For each object return:
- label: short object name in ${name}
- phrase: what HE says, first person, natural everyday spoken ${name} (casual, not formal), max 6 words
- box_2d: [ymin, xmin, ymax, xmax] normalized to 0-1000

Gemini also gets a JSON response schema, so the answer comes back as an array. Each box is [ymin, xmin, ymax, xmax] on a 0–1000 grid, whatever the photo size.

The line I'd keep is fewer good spots beats 8 weak ones. Here a wrong spot isn't noise. It's a button that says the wrong thing.

Three models, one 55-second budget

The function tries gemini-3.8-flash, then gemini-3.1-flash-lite, then deepseek-flash on DeepSeek's own API:

async function findSpots(image: string, lang: Lang) {
  const deadline = Date.now() + TOTAL_BUDGET_MS;
  const tried: string[] = [];
  for (const a of ATTEMPTS) {
    const key = a.key();
    if (!key) continue;
    const left = deadline - Date.now();
    if (left < 4_000) break;
    try {
      const raw = await a.run(image, lang, key, AbortSignal.timeout(Math.min(a.ms, left)));
      return { spots: validateSpots(raw), model: a.name };
    } catch (err) {
      tried.push(`${a.name}: ${(err as Error).name === "Error" ? (err as Error).message : (err as Error).name}`);
    }
  }
  throw new Error(tried.length ? tried.join("; ") : "no provider key configured");
}

The rules in there:

Each step has its own time limit (a.ms): 12, 10 and 25 seconds.
All steps share one 55 s budget, which keeps the function under the browser's 70 s wait.
With less than 4 s left, it stops instead of starting a call it can't finish.
A provider without a key is skipped, so the same code runs with one key or two.

The timeout that skipped the fallback

The first version only fell back on error responses, like the 503 "high demand" the very first real call got. Then a real run timed out on the main model.

A timeout is not a response. AbortSignal.timeout() makes fetch throw a TimeoutError, and my loop only looked at res.status. So the error flew past the fallback and failed the whole request. The next model was never asked.

The fix was a try/catch around each step (863498a). A regression test now fakes that exact error and checks that the next model still answers:

// What fetch throws when AbortSignal.timeout() fires.
const timedOut: Answer = () => Promise.reject(new DOMException("The operation was aborted due to timeout", "TimeoutError"));

The receipt, and how to check it

The run from the top of the post: six AI-generated room photos, sent to the production function on 2026-10-04, encoded the way the app does it. Nothing mocked.

Step	Model	Answered
1	`gemini-3.8-flash`	1 of 6
2	`gemini-3.1-flash-lite`	5 of 6
3	`deepseek-flash`	0 of 6

6/6 rooms answered · 43 spots · p50 4.03 s · max 7.38 s · $0.00 (Gemini API free tier)

The five rooms step 1 turned down still finished in 3.38 to 4.28 s, end to end. Step 1 said no fast, and step 2 picked them up. The one room step 1 did answer was the slowest of the six, at 7.38 s.

You can make one of these calls yourself, with no key and no clone. This sends the app's example bedroom to the live function:

{ printf '{"lang":"en","image":"'; curl -s https://roomspeak.edycu.dev/samples/bedroom.jpg | base64 | tr -d '\n'; printf '"}'; } \
  | curl -s https://roomspeak.edycu.dev/api/detect -H 'Content-Type: application/json' --data-binary @- -w '\n%{time_total} s\n'

On 2026-10-04 it returned 8 spots (bed, walker, radio, water glass, medicine, window, prayer rug, thermos) from gemini-3.1-flash-lite in 6.9 s. The full six-room run, every spot and phrase, is in DEMO.md.

An answer is not a safe answer yet

The answer becomes buttons, so validateSpots checks every item first. Anything malformed is dropped, never repaired. The checks, from shared/validate.ts:

for (const item of raw) {
  if (out.length >= MAX_AI_SPOTS) break;
  if (!item || typeof item !== "object") continue;
  const { label, phrase, box_2d } = item as Record<string, unknown>;
  if (typeof phrase !== "string" || !phrase.trim()) continue;
  if (!Array.isArray(box_2d) || box_2d.length !== 4) continue;
  if (!box_2d.every((v) => typeof v === "number" && Number.isFinite(v) && v >= 0 && v <= 1000)) continue;
  const [ymin, xmin, ymax, xmax] = box_2d as number[];
  if (ymin >= ymax || xmin >= xmax) continue;

Why not repair? A swapped box could be swapped back, but that's a guess about what the model meant. The caregiver can add a missed spot by hand. A wrong spot speaks.

To test it, I didn't hand-write answers. A fast-check property test builds them from spots that are valid or invalid by construction. Each bad spot breaks exactly one rule: a value outside 0–1000, min and max swapped, zero height, the wrong number of values, a blank phrase, no box. So the right result is known without re-implementing the validator.

Over 100,000 generated answers, the validator never returns a box outside 0–1000, a box with min ≥ max, a blank phrase or more than 8 spots, and never drops a valid spot before the cap. The test also asserts that the answers really do go over the 8-spot cap, so that branch is exercised, not just present. I wrote about why that check matters in an earlier post.

Rings a hand can hit

Each box becomes a ring at its center, sized to the object but kept within calm bounds:

const r = Math.min(0.08, Math.max(0.03, 0.3 * Math.min(boxW, boxHInWidthUnits)));

Overlapping rings get nudged apart a little. In Speak mode, no ring is smaller than 64 px across.

Then the tap. It counts when the finger lifts, and only inside the same ring it started in. A second finger cancels it. So a resting palm, or a slide across the photo, never speaks:

function down(e: PointerEvent<HTMLDivElement>) {
  if (!interactive) return;
  if (press.current) {
    press.current.cancelled = true;
    return;
  }
  press.current = { id: e.pointerId, hit: locate(e).hit?.id ?? null, cancelled: false };
}

function up(e: PointerEvent<HTMLDivElement>) {
  const p = press.current;
  if (!p || p.id !== e.pointerId) return;
  press.current = null;
  if (p.cancelled) return;
  const { fx, fy, hit } = locate(e);
  if (onPoint) onPoint(fx, fy);
  else if (hit && hit.id === p.hit) onActivate?.(hit);
}

When rings overlap, locate picks the one whose center is nearest. The rings are still real <button>s labelled with their phrase, so a keyboard and a screen reader reach them too. A click from Enter or Space has detail === 0, so the button's own handler only fires for the keyboard:

onClick={(e) => e.detail === 0 && onActivate?.(s)}

The voice is the browser's own speechSynthesis. Each new phrase calls cancel() before speak(), so a second tap never waits behind the first. With no voice for the chosen language, the phrase shows as large text instead.

One more wall: fine everywhere except Vercel

import { validateSpots } from "../shared/validate" worked in Vite, Vitest and tsc. Vercel runs api/*.ts as Node ESM, which needs the .js extension, so only the deploy broke (cf62a94). A test now walks every relative import reachable from api/ and fails on any without .js.

What I'd carry over

A timeout is a thrown error, not a status code. A fallback that only reads res.status doesn't cover it.
Give each step its own time limit and one shared budget, so a slow model can't eat the next one's time.
Treat a model's JSON like user input. Check it, drop what fails, never repair it.
Build test inputs whose right answer you know by construction, and check that the edge you care about actually gets hit.

Honest limits

No private photos on the free tier. Google's terms for unpaid use let it use submitted images to improve its products, and let human reviewers read them. Fine for the AI-generated example rooms; a family's real room needs the paid tier first. If you try the app, use the example rooms.
Measured on AI-generated rooms only. There is no benchmark on real family photos yet.
A proof of concept, not a medical device. No clinical claims. One room, one device, the device's own voice. It still needs testing with real families and speech therapists.
The ladder doesn't say why. The function reports which step answered, not why the earlier ones said no.

Try it

App: roomspeak.edycu.dev (tap an example room, no login) · Code: github.com/edycutjong/roomspeak (MIT) · 30-second path: roomspeak.edycu.dev/judge · Story: roomspeak.edycu.dev/story

Planned and built with the Devpost Learn skill pack for Build With AI: Basics. The planning docs are in the repo's devpost/ folder.

If you work in AAC, or care for someone after a stroke, I'd like to hear what this gets wrong.

I guessed Stylus would be 8x cheaper. The chain said 2.87x. Then I found 41% of that was my own Solidity.

Edy Cu — Sat, 03 Oct 2026 13:48:16 +0000

In December, Maya's tokenized-equity position is down $3,140 on paper and she does nothing about it. Banking that loss properly means picking which of her 64 tax lots to sell (highest cost basis first, HIFO), then policing a 30-day calendar so a rebuy doesn't trip the wash-sale rule and void the deduction.

I built HarvestBot for the Arbitrum Open House Singapore buildathon: an agent that does that job on Robinhood Chain testnet. The contracts bound it: the lot selection is recomputed on chain, a rebuy inside the window reverts, and an off-map trade gets its bond slashed. The HIFO ledger is a Rust contract compiled to WASM with Arbitrum Stylus.

This post covers the one number I published, and the part of it I had to take back.

Links: live site · verify on chain in your browser, no wallet · 30-second judge path · repo · demo video, 2:39 · bench results

Why HIFO belongs in WASM

HIFO selection is a nested loop. Each pass scans every open lot for the highest basis-per-unit, takes it, and prorates the last lot when the sell quantity runs out partway through one. With 64 lots and 9 picks, that's 9 full scans, and every comparison is a cross-multiplication on 256-bit integers (basis_i · qty_b > basis_b · qty_i, so there's no division and no rounding drift).

Stylus runs WASM next to the EVM on Arbitrum chains, and WASM costs less per instruction than EVM opcodes. A loop full of comparisons seemed like the textbook case for it. Before writing any code, I estimated about 8× cheaper.

What the chain said: 2.87×

I kept an ABI-identical Solidity twin of the ledger. It has the same integer semantics and the same tie-break (lowest lot id wins). It serves as the benchmark control and as the fallback for chains without Stylus. I deployed both to Robinhood Chain testnet (chain 46630), seeded them with the same lots, and measured computeHarvest using Arbitrum's own NodeInterface.gasEstimateComponents. That call splits the L1 calldata share from L2 compute, and L2 compute is where the two engines actually differ.

At 64 lots it came out to 1,456,905 L2 gas for Solidity and 507,993 for Stylus: 2.87×. That's a lot less than 8×, but I reported what the chain measured. The 2.87× went into the README, the demo video, the OG images and an X thread.

The catch: my Solidity was reading storage in the loop

Two weeks later I put the two inner loops side by side. This is the Solidity twin I shipped (contracts/src/TaxLotLedger.sol):

Lot[] storage lots = _lots[owner][asset];
// ...
while (remaining > 0) {
    // select the open, untaken lot with the highest basis/qty
    uint256 best = type(uint256).max;
    for (uint256 i = 0; i < n; i++) {
        if (!lots[i].open || taken[i]) continue;
        if (best == type(uint256).max) {
            best = i;
            continue;
        }
        // lots[i].basis/qty > lots[best].basis/qty  ⇔  basis_i·qty_best > basis_best·qty_i
        if (lots[i].costBasis * lots[best].qty > lots[best].costBasis * lots[i].qty) {
            best = i;
        }
    }
    // ...
}

lots is a storage pointer, so every lots[i].open, .costBasis and .qty is a storage read, repeated on each of the 9 selection passes. The Rust ledger (stylus/ledger/src/lib.rs) reads storage once and runs the rest in memory:

// one pass over storage → memory
let lots_s = self.lots.get(owner);
let lots_s = lots_s.get(asset);
let n = lots_s.len();
let mut mem: Vec<MemLot> = Vec::with_capacity(n);
for i in 0..n {
    let lot = lots_s.get(i).expect("index < len");
    mem.push(MemLot {
        qty: lot.qty.get(),
        basis: lot.cost_basis.get(),
        open: lot.open.get(),
    });
}
// ...
while !remaining.is_zero() {
    let mut best: Option<usize> = None;
    for (i, l) in mem.iter().enumerate() {
        if !l.open || taken[i] {
            continue;
        }
        match best {
            None => best = Some(i),
            Some(b) => {
                // basis_i/qty_i > basis_b/qty_b  ⇔  basis_i·qty_b > basis_b·qty_i
                if l.basis * mem[b].qty > mem[b].basis * l.qty {
                    best = Some(i);
                }
            }
        }
    }
    // ...
}

So I hadn't compared engines. I'd compared two data paths. Part of the 2.87× came from the Rust code having a better structure, and that structure isn't specific to WASM. Solidity can copy to memory too.

Building a fair baseline

I wrote a third ledger, contracts/src/bench/TaxLotLedgerMemCopy.sol, for the benchmark only. It uses the same selection rule, the same integer semantics and the same tie-break. The only change is the data path: one pass from storage into memory, and every later selection pass reads memory:

/// @dev One pass over storage → memory (the Rust `mem: Vec<MemLot>`).
function _load(Lot[] storage lots) internal view returns (MemLot[] memory mem) {
    uint256 n = lots.length;
    mem = new MemLot[](n);
    for (uint256 i = 0; i < n; i++) {
        Lot storage s = lots[i];
        mem[i] = MemLot({qty: s.qty, basis: s.costBasis, open: s.open});
    }
}

A Foundry differential test checks that it picks the same lots and returns the same realized loss as the shipped twin on random books, both before and after realize. I deployed it next to the other two on chain 46630 and reran all three engines together, with 50 eth_call samples per cell.

Three engines, one chain

L2 compute gas for computeHarvest (HIFO, up to 9 picks, 4 at 8 lots), from bench/RESULTS.md:

Open lots	Solidity shipped	Solidity memory-copy	Stylus	Fair: memory-copy ÷ Stylus	Shipped ÷ Stylus
8	132,363	108,713	125,807	0.86×	1.05×
16	352,160	231,323	179,982	1.29×	1.96×
32	720,396	440,005	288,999	1.52×	2.49×
64	1,456,891	857,559	507,979	1.69×	2.87×
128	2,929,924	1,693,678	945,964	1.79×	3.1×

All three engines return identical lotIds and realizedLoss at every size. The 64-lot row is Maya's scenario, exactly −$3,140.000000 (-3140000000 in 6-dp USD, the same figure forge test and cargo test pin).

The fair engine-to-engine figure is 1.69× at 64 lots, not 2.87×. About 41% of the ratio I first published (2.87× → 1.69×) came from my Solidity baseline re-reading storage, not from WASM. The 2.87× is still accurate for the Solidity twin I shipped. It just isn't the gap between the engines.

The repo's README, JUDGE.md and DEMO.md now give the fair figure next to every mention of 2.87×, and JUDGE.md has a dated correction entry.

Where Stylus still loses, and where it wins

At 8 lots, the memory-copy Solidity ledger beats Stylus (0.86×). The Stylus program is uncached on this testnet, so every call pays a fixed WASM initialisation cost, and at 8 lots there isn't enough loop to make up for it. Caching the program (cargo stylus cache bid) would remove that cost, but Robinhood Chain testnet has no ArbOS cache manager yet: ArbWasmCache.allCacheManagers() returns [] on chain 46630. So these are worst-case Stylus numbers.

Past the fixed cost, the fair ratio goes 1.29× → 1.52× → 1.69× → 1.79× as the book grows. All three engines pay the same cold storage read per lot. Stylus only wins on the comparison loop, and that loop grows with the number of lots times the number of picks. A tax-lot ledger does grow: Maya's 64 lots turn into hundreds after a few years of recurring buys. In this use case the advantage grows with the portfolio. For small books, I'd write it in Solidity.

Reproduce it

Read-only, no wallet. This asks the deployed Stylus ledger for the next HIFO picks:

cast call 0xEff7B46049fC677F58264e0ebb19dF1a39195a21 \
  'computeHarvest(address,address,uint256,uint256)(uint64[],int256)' \
  0x72cd3cB98A5d9B830b386EeBA7B2340132Ba557b 0x5884aD2f920c162CFBbACc88C9C51AA75eC09E02 638501157098660213 412300000 \
  --rpc-url https://rpc.testnet.chain.robinhood.com

The full three-engine bench needs a funded testnet key, because it seeds the three ledgers:

git clone --recurse-submodules https://github.com/edycutjong/harvestbot && cd harvestbot
cd contracts && forge test              # 91 tests, incl. the memory-copy differential
cd ../stylus/ledger && cargo test       # 6 native tests on the WASM engine
cd ../.. && RPC=... PRIVATE_KEY=... python3 scripts/bench.py
# --components-only refreshes the gas split without re-seeding

The bench ledger addresses are in deployments/46630-bench.json, and the raw numbers, including p50/p95 latency, are in bench/results.json. Latency is eth_call wall-clock through the public RPC (1181.9–1215.2 ms p50 across every engine and size). It's bound by the network, not the engine, so I don't claim anything from it.

Honest limitations

Testnet only. Robinhood Chain mainnet exists; I didn't deploy there (gas budget, unaudited code).
Uncached Stylus. Caching would remove the init floor. I haven't measured how much of the 8-lot gap that closes, so I don't claim it.
Mocks, labeled. The oracle mark, the swap venue and the bond's USDC are MOCK contracts with MOCK in their names. The assets are real Robinhood testnet Stock Tokens from the faucet. The bench ledgers use synthetic asset keys and never touch a token.
One workload. This measures HIFO selection. A different access pattern (few comparisons, many writes) would give different numbers.
The benchmark control is mine. Someone better at gas golfing could probably narrow the gap further. The memory-copy ledger is the best Solidity I wrote for it, not the best Solidity possible.

Takeaway

If you benchmark Stylus against Solidity, make sure both sides use the same data path first. My first number compared WASM plus memory against EVM plus repeated storage reads, and it overstated the engine gap by about 41%. The corrected number is smaller, but it measures only the engine.

Live site · /verify · /judge · Repo · Demo video · bench/RESULTS.md

If you've measured Stylus with a cached program, I'd like to see how your numbers compare.

Your passkey can be an encryption address. The domain is the key.

Edy Cu — Sat, 03 Oct 2026 13:45:33 +0000

There is still no simple way to encrypt something to a person. You can encrypt to a server, or to someone you have already swapped keys with, but not to "Maya, whoever and wherever she is."

I built Letterlock to try: the passkey you already have becomes an encryption address. Your device derives an X25519 key from the passkey, publishes only the public half to a directory contract on Monad, and from then on any app or agent can look your key up with one read and seal data only your passkey opens. It is live on Monad mainnet (2:32 demo with a real passkey, try it in two minutes, no wallet).

This post is about the one design decision that matters more than the cryptography: the domain your passkey is bound to is, in effect, the key. I got it wrong in the first release and had to move it.

From a passkey to a key pair

WebAuthn's PRF extension lets a passkey act as a keyed hash: give it a 32-byte salt, it returns a 32-byte secret that only that credential can produce. Run that through HKDF and you have an X25519 private key. This is the derivation, straight from the SDK:

export const prfSaltFor = (epoch: number): Uint8Array<ArrayBuffer> => {
  checkEpoch(epoch);
  return new Uint8Array(sha256(utf8(`letterlock/hpke/v1/${epoch}`)));
};

const keyPairFrom = (prfOutput: Uint8Array, epoch: number, info: string): EncryptionKeyPair => {
  if (prfOutput.length !== 32)
    throw new LetterlockError("PRF_UNSUPPORTED", `PRF output must be 32 bytes, got ${prfOutput.length}`);
  const secretKey = hkdf(sha256, prfOutput, utf8("letterlock/v1"), utf8(info), 32);
  return { epoch, secretKey, publicKey: x25519.getPublicKey(secretKey) };
};

The salt is Letterlock's own namespace, separate from the salt mera uses to derive the same passkey's EVM account, so one passkey yields an encryption key and the account that publishes it, and neither reveals the other. Rotation is epoch + 1: a new salt, an unrelated key, and every old epoch can still be re-derived to open old letters. Nothing secret is ever stored; the key is re-derived on every open.

Sealing needs none of this. A sender resolves your public key and uses HPKE (RFC 9180):

import { letterlock } from "letterlock";

const ll = letterlock({ chain: "monad" });
const key = await ll.resolve("agent:10260");                 // one keyOf / keyOfAgent read
const envelope = await ll.sealTo("agent:10260", new TextEncoder().encode("the dentist moved to Thursday 10:40"));

Resolve plus seal against Monad mainnet measured p50 28.199 ms, p95 36.122 ms over N = 1,000 (pnpm bench, results in the repo). Publishing a first key cost 70,863 gas.

The wall: a PRF output belongs to its rpId

Here is the part that is easy to miss. A passkey is created for a relying party ID, the rpId, which is a domain. The browser only lets a page use its own domain (or a parent of it) as the rpId, and the PRF output is bound to that rpId. Three consequences follow, and each one shaped the design:

Same passkey, different domain, different key. Derive under localhost or a preview deploy and you get a key that production can never re-derive. Anything sealed to it is unopenable forever.
Whoever serves the rpId can derive every key. If I control the domain, I can serve a page that runs the same passkey ceremony and gets the same PRF output. The rpId's host is the most trusted party in the whole system.
Opening has to happen on that domain. Other apps can seal to you from anywhere, but the open (the passkey ceremony) can only run where the rpId lives.

Because of (1), the SDK pins one production rpId and refuses to publish a key derived anywhere else:

const pinned = (action: string) => {
  if (rpId !== LETTERLOCK_RP_ID && config.unsafeAllowAnyRpId !== true)
    throw new LetterlockError("INPUT_INVALID",
      `${action}: rpId "${rpId}" is not the production rpId "${LETTERLOCK_RP_ID}". A key derived under another rpId can never be re-derived in production, so every note sealed to it would be unopenable`);
};

A key also has to say where it was derived: one rebuilt from its public fields is refused, because it could have come from any passkey.

Where I got it wrong

Version 0.1.0 pinned letterlock-app.vercel.app. It worked, every test passed, and it was a mistake because of consequence (2): a *.vercel.app name is a Vercel project name, held only while that project exists. Delete the project, someone else claims the name, and they can serve a page there that derives every Letterlock key ever made.

Version 0.1.1 moved the pin to app.letterlock.edycu.dev, a subdomain of a domain I registered:

export const LETTERLOCK_RP_ID = "app.letterlock.edycu.dev";

The old host now answers every request for the app with a 308 to the new one, so no new key is made under the old rpId. Moving was cheap only because it happened early: the only keys derived under the old rpId were the app's own live-check test keys, made with a virtual authenticator and deleted after each run. Had anyone real been using it, their old letters would have become unopenable at the new address. That is the general rule: pick the rpId as if it were a root key, before the first user, on a domain you will keep renewing.

What it cannot do (from the README's limits)

Opening happens on the Letterlock app. It is consequence (3): other apps seal to you from anywhere, but the passkey ceremony that opens a letter runs on the pinned rpId.
Whoever serves app.letterlock.edycu.dev can derive every key. It is on my own domain now, which has to stay registered and pointed at the app for as long as keys derived under it are in use.
Anyone can seal to anyone. HPKE base mode is anonymous; no letter names a sender, and the app never shows one.
No recovery. If every synced copy of the passkey is lost, so is the key.
Cross-device depends on the platform. By the WebAuthn spec a synced passkey returns the same PRF output for the same salt. Apple developers have reported otherwise in one direction and across some OS versions (forum thread), so treat cross-device opening as something to verify on your devices.
npm is one release behind. npm still serves 0.1.0, the version with the old pin; until the next release lands, use the SDK from the repo.

Try it

Two-minute judge path: create an address with your passkey, ask the reference ERC-8004 agent to seal you a note, open it.
Demo video, 2:32, recorded with a real passkey on Monad mainnet.
Code, MIT: the contract, the SDK and CLI, the app, the agent, and the benchmark.

If you are building an agent or an app that keeps something about people, I would like to hear whether "seal to a person with one read" fits what you are doing.

My voice app's one job was to take back a wrong sentence. It was taking back sentences it never said.

Edy Cu — Tue, 29 Sep 2026 10:31:18 +0000

Ray is 68 and five days home after a hip replacement. He asks his assistant whether he can put weight on the new hip. Halfway through the answer, his physio changes the plan. Unsay exists for that moment: the assistant stops mid-sentence and says "Wait — don't do that. What I just told you is out of date."

Its one job is to take back a wrong sentence. When I had the code audited, it turned out it was also taking back sentences it had never said.

This post covers the four ways the retraction was wrong, and what fixing each one taught me about MCP resource subscriptions. The repo is public, with every fix below in its own commit. Try it live at api.unsay.edycu.dev/judge, and see the hash-chain proof at /verify.

The mechanism, in one paragraph

Unsay is an MCP server (spec 2025-11-25, Streamable HTTP). The care plan is not a PDF: each instruction is a versioned MCP resource such as care://ray/weight_bearing. A voice host subscribes with resources/subscribe. When a clinician publishes a signed revision, the server sends notifications/resources/updated. The host stops, re-reads the resource, and finds a server-rendered sentence in _meta["unsay/retraction"]:

const parts = [
  // 1 — the stop, before anything else.
  `Wait — don’t do that. What I just told you is out of date.`,
  // 2 — the sentence being withdrawn, so there is one instruction to drop.
  `I said ${quote(prev.value)}`,
  // …and the one that replaces it, with who changed it and when.
  `${next.authorLabel} changed it ${age}: ${quote(next.value)}`,
]

The order is the safety property. A frightened person acts on the first clause, so the stop comes before any metadata.

For hosts that can't subscribe, there's a fallback tool, whats_changed. Everything below is about what prev should be.

Wrong #1: "the previous version" is not "what you said"

Here is the original line:

const previous = record.version > 1
  ? chain.find((r) => r.version === record.version - 1)
  : undefined

Version n − 1 is what the chain said before. It is not what this listener heard. Those only coincide in a demo.

The seeded record starts at v2, so the very first read of care://ray/weight_bearing already carried:

Wait — don't do that. What I just told you is out of date. I said "No weight through the operated leg. Transfers with the frame only." …

Nothing had been said yet. And the proof file I had committed as evidence (docs/proof/live_run.jsonl) recorded exactly that sentence on a first read. The receipt documented the bug.

The resume case was worse. The host reads v2, the connection drops, v3 and v4 land, and the host re-reads. The retraction then quoted v3, a sentence the listener never heard.

The fix is a map per session from resource to the version this session was last served:

const key = uriFor(parsed!.subject, parsed!.topic, record.audience)
const lastHeard = heard.get(key)
const previous =
  !superseded && lastHeard !== undefined && lastHeard < record.version
    ? chain.find((r) => r.version === lastHeard)
    : undefined
if (!superseded) heard.set(key, record.version)

There is no retraction on a first read, and none for an old version someone asked for by URI. That URI now says [SUPERSEDED — v1, replaced by v2; …] in its text, because annotations don't survive a read in every host.

Wrong #2: "since when you last spoke" is the wrong cursor

The fallback tool took a since timestamp, and I told hosts to pass "when you last spoke about the plan". That sounds right, and it's wrong twice over:

A correction that lands while the host is still talking is written before the host finishes. So writtenAt > endOfAnswer filters out exactly the event the product exists for.
since came from the host's clock and was compared against the server's writtenAt. A host clock running a minute fast returned changed: [], with no error.

The fix is the same one every sync API eventually makes: the server hands out the cursor.

// Server clock at the call, i.e. before the host speaks. Handed back as the next
// `since`, it catches a correction written while the host was still talking.
const payload = { changed, asOf: at.toISOString() }

A since ahead of the server clock is now an error, not a silent "nothing changed". The reviewer then found one more hole: a write stamped in the same millisecond as asOf was never returned, because the filter was >. It's >= now. A record returned twice carries no second retraction, which brings us to #3.

Wrong #3: a re-confirmation is not a correction

The anticoagulant record ships stale on purpose. When the GP confirms it with a new review date, the words don't change, only staleAfter does. That is a real revision, so a notification fires, and the host dutifully stopped mid-word to say:

I said "Rivaroxaban 10mg once daily. Stop date: 25 September 2026." Dr Mensah, GP changed it just now: "Rivaroxaban 10mg once daily. Stop date: 25 September 2026."

The fallback path already had a same-words guard. The subscribed path didn't. Both now share one rule:

...(previous && previous.value !== record.value

The Agent Skill tells the host what a missing retraction means: the words you were saying did not change: carry on.

Wrong #4: the gloss turned "do NOT" into "you can"

Clinical phrases get a plain-English gloss. "Full weight-bearing as tolerated" becomes "you can put as much weight through that leg as is comfortable". The glossary matched on text alone, so this instruction:

Do NOT progress to full weight-bearing as tolerated until the X-ray is reviewed.

was helpfully explained as permission.

My first fix was a list of negation words. The review came back with four sentences the list couldn't see: "Stop full weight-bearing…", "Wait for the X-ray before…", "…is cancelled", "Hold off…". No word list is ever complete, but position is checkable:

// A weight-bearing gloss is worded as permission, and no word list catches every
// negation ("Hold off…", "…is cancelled"). Position does: gloss only a value that
// OPENS with the term, with no action word over it. Definitions are unaffected.
return hits
  .filter((g) => !WEIGHT_BEARING.has(g.label) || (opens(g) && !ACTION.test(text)))
  .map(({ label, gloss }) => ({ label, gloss }))

When in doubt, the sentence is quoted verbatim with no gloss. A missing explanation is an inconvenience. A wrong one is a fall.

How these were found

I gave another model read access to the repository and asked it for an architecture audit. The rule for every finding it raised: verify it against the code before touching anything, reject it with a file:line if it's wrong, and commit the fix with a regression test that fails first if it's right.

Three rounds produced 24 fixes. The four above are the ones that changed what the product says. Most of the rest were in the /write endpoint: it accepted "Assistant" as an audience and spoke it, a captured signed request could be replayed to roll back a newer correction, and a subscribe to …/weight_bearing/ returned {} and then never fired.

The part I'd repeat is the verification step, not the model. Every one of these bugs passed a green suite. The suite tested what I thought the product did.

The numbers, with their caveats

344 tests, tsc --strict clean, plus 34 safety assertions run against a live server (npm run verify).
Local server path: signed write → notification → re-read, p95 3.0 ms, 200/200 inside a 3.4 s speech window. The window is an assumption (12 words at ~150 wpm), not a measurement of Alexa+ speech.
Over the public internet (client on another continent from the us-west2 server, both network legs paid twice): the committed receipt is 200/200 with p95 461 ms. Two re-runs today gave p95 710 ms and 966 ms, and one run landed 199/200. The server accounts for single-digit milliseconds, so that spread is the network.

What it doesn't do

It has never talked to Alexa+ itself. Whether Alexa+ declares resource subscriptions is untested. That's why the whats_changed fallback exists and is exercised end to end.
The patient is fictional, and this is not a medical device. The server renders the clinician's own words. It generates no clinical advice.
No one outside the project has used it yet. If you work with post-operative patients, I'd genuinely like to know where this breaks.

Repo: github.com/edycutjong/unsay (MIT) · Try it: api.unsay.edycu.dev/judge · Demo video: youtu.be/jJM5LWHONDU

Built for the Amazon Alexa+ hackathon track.

50,000 property tests passed while my app crowned an impostor

Edy Cu — Fri, 25 Sep 2026 02:26:11 +0000

You hear "buy PEPE", type PEPE into your wallet, and get fourteen tokens with the same name and the same frog. Which one do you buy?

I built a tool to answer that. You type a ticker, and it asks Nansen's API for every token with that name across chains. Then it checks who actually holds and trades each one: labelled Smart Money, whales, top-PnL wallets, exchange flow, tagged top holders. Exactly one card turns green, or the tool abstains.

Last week it crowned a token that its own scorer had flagged as an impostor. My property test ran 50,000 generated cases on that exact decision, and they all passed. While writing this post I found out why, and the answer was worse than I expected.

Live: https://whichone.edycu.dev
Code: https://github.com/edycutjong/whichone

Two rules in two files

The engine judges each candidate twice.

The impostor rule lives in score.ts. A candidate is an impostor when nothing labelled has touched it, it has no meaningful exchange flow, and it is either brand new or barely held:

// packages/core/src/score.ts
const impostor = f.labelledWallets === 0 && exchMag < 10_000 && ((f.ageDays != null && f.ageDays < 14) || (f.totalHolders != null && f.totalHolders < 500));
if (impostor) reasons.push("IMPOSTOR: nothing labelled has ever touched it");

The crown rule lives in verdict.ts. It decides whether ranked[0] gets the green card or the tool says "none of these looks real". This is how it read before the fix (abridged: I left out the guards for zero results, unscorable chains and failed lookups, which didn't change):

// packages/core/src/verdict.ts, before commit 3b5f16f (abridged)
export function crown(ranked: Scored[], sameNameCount: number, q: string): { winner: Scored | null; abstainReason?: string } {
  const best = ranked[0];
  /* ... guards: 0 results, unscorable, every lookup failed, holders lookup failed ... */
  if (best.score < ABSTAIN_THRESHOLD || (best.labelledWallets === 0 && (best.recognisedHolders ?? 0) < MIN_RECOGNISED_TO_CROWN))
    return { winner: null, abstainReason: "none of these looks real — nothing labelled has touched any of them" };
  return { winner: best };
}

best.impostor isn't read anywhere in it. The score module had already answered the question, and the crown rule never asked.

A fix for one bug opened the door

Three days earlier, live testing turned up a different failure. SHIB2 was a dead token, about $36K market cap, three years old, with zero labelled wallets, and the tool crowned it REAL. The cause was the holders tiebreak. Nansen's tgm/holders returns an address_label for each top holder, and SHIB2's top holders included a UniswapV2 pool and a SHIB2 Token Deployer. Every token has a pool and a deployer, so those labels prove nothing.

That fix excluded structural tags (pools, deployers, ENS/SNS names, burn addresses, the contract itself). It also raised the bar for a candidate with zero labelled wallets: it can be crowned only if at least 3 of its top 20 holders carry a wealth or activity tag, like "Token Millionaire" or "High Activity". That is MIN_RECOGNISED_TO_CROWN = 3.

I added that clause on purpose as an escape hatch. Nansen's labels are uneven, and some real tokens have no labelled wallets in a 7-day window but plenty of tagged holders.

On 2026-09-19, PEPEGA got through that hatch. It had one candidate:

0 labelled wallets in 7 days
no meaningful exchange flow
283 holders
7 of its top 20 holders wealth-tagged

283 holders is under 500, so score() set impostor: true. The score cleared the abstain threshold and 7 tagged holders beat the bar of 3, so crown() crowned it anyway. The fix commit put it this way: "a green card and an IMPOSTOR badge at once."

The fix takes two lines:

// packages/core/src/verdict.ts, after commit 3b5f16f
  // a card cannot be "this is the one" and IMPOSTOR at once: when the best candidate trips the impostor rule, abstain
  if (best.impostor) return { winner: null, abstainReason: "none of these looks real — the best candidate trips the impostor rule" };

Here is the CLI output now:

PEPEGA — 1 same-name token on Nansen

✖ ethereum   0x9634…c879    PEPEGA   4.68   0 labelled wallets in 7 days · no meaningful exchange flow · 283 holders IMPOSTOR

no winner — none of these looks real — the best candidate trips the impostor rule
7 credits · 4 calls (0 cached) · 2.1s · verdict 47b4ac4fc907

Why 50,000 property cases missed it, part one

Two days before PEPEGA, I had added property-based tests with fast-check. The crown-rule property generates random lists of scored candidates, ranks them, calls crown(), and checks every winner. It runs 10,000 times for each of five properties, which is where the 50,000 comes from. Everything was green.

This is what the property asserted about a winner before the fix:

// packages/core/test/property.test.ts, before commit 3b5f16f (the winner check only)
return (
  winner === ranked[0] &&
  winner.scorable &&
  !winner.unchecked &&
  winner.score >= ABSTAIN_THRESHOLD &&
  !(winner.labelledWallets === 0 && (winner.recognisedHolders ?? 0) < MIN_RECOGNISED_TO_CROWN) &&
  abstainReason === undefined
);

Each line restates one of crown()'s own if statements, so the property only checked that the function did what the function said. The rule I had missed wasn't in crown() at all. It was a rule about what the user sees: a card is never green and IMPOSTOR at once. That rule crosses two modules, and my tests were organized by module.

The fix commit added !winner.impostor && to that list. At the time I thought that closed it.

Part two: the generator never made an impostor

While writing this post I wanted to say how fast the new assertion would have caught the bug. So I copied the test's generator into a scratch script, paired it with the pre-fix crown(), and counted.

The generator draws each wallet class separately, and the labelled count is their sum:

// packages/core/test/property.test.ts (abridged: 4 of the 17 fields)
smartTraderWallets: fc.nat(500),
whaleWallets: fc.nat(500),
topPnlWallets: fc.nat(500),
publicFigureWallets: fc.nat(500),
// ...
.map((r) => facts({ ...r, labelledWallets: r.smartTraderWallets + r.whaleWallets + r.topPnlWallets + r.publicFigureWallets }));

For a candidate to have zero labelled wallets, all four independent draws have to land on 0. Across five seeded runs of 10,000 lists, about 44,000 candidates per run, the generator produced zero candidates with zero labelled wallets. So it produced zero impostors, and zero cases where the pre-fix crown() crowned one. Even with the right assertion, the old code passed.

It is not just the impostor line. Every branch that matters for a zero-labelled token (the SHIB2 bar, the PEPEGA veto, the failed-holders abstain) was sitting in a corner of the input space the generator never visited.

I changed one thing in the scratch copy, not in the repo. Each wallet count became fc.oneof(fc.constant(0), fc.nat(500)), so zero comes up about half the time for each class. Then I ran it against the pre-fix crown() with the !winner.impostor property. fast-check found a counterexample on all five seeds, after 47, 53, 71, 109 and 244 cases.

Two things went wrong, and fixing either one alone would not have caught the bug:

The invariant was the implementation restated. Write properties about what the output must never do, not about which branches the code takes.
The generator had no weight where the bugs were. A uniform draw almost never produces the edge case your domain revolves around. Here that case was "zero". If a value drives a branch, put it in the generator on purpose.

As of this writing, the repo's generator still has that gap. The PEPEGA regression test pins the actual bug, so the crown rule is covered. The property test just doesn't cover as much of it as "50,000 cases" suggests.

The rest of the system

The score is plain arithmetic over four Nansen endpoints:

search/general: 0 credits
tgm/flow-intelligence and tgm/token-information: 1 credit each, for up to 8 candidates
tgm/holders: 5 credits, for the top two finalists only

Market cap, volume and search rank are left out of the score, because those are what an impostor can buy.

The repo includes a benchmark: 12 queries × 2 cold runs against the live API, on 2026-09-16. Cold p50 was 3.6 s, p95 7.2 s, warm p50 3 ms. Verdicts cost 18.6 credits on average and 26 at most. npm run verify replays twelve recorded verdicts offline with the same decision hash. It needs no API access and no network. The web page streams every Nansen call into a side rail as it fires, with the endpoint, credits, latency and a short hash of each response.

Limitations

search/general decides the candidate set. The tool can't warn about an impostor Nansen hasn't indexed.
Label coverage is uneven across chains. A real token on a thinly labelled chain can lose to a bridged copy on a busier one, which is why there's a chain filter.
Flow data covers a 7-day window, so a real but dormant token can look quiet.
The zero-labelled escape hatch is still there. On 2026-09-16, AI16Z and PEPE UNCHAINED were crowned on wealth-tagged holders alone. The card says "0 labelled wallets" so you can see the weakness, but it's a weaker verdict than a green card backed by 90 labelled wallets.
DOGE crowns a Solana meme DOGE, because native DOGE has no contract to compare against.
USDC is the slow outlier, about 15 s cold. Nansen times out on some of its lookups, and the call drawer shows each timeout.

Try it

Live: https://whichone.edycu.dev. Try PEPE, then PEPEGA.
Code: https://github.com/edycutjong/whichone. npm run whichone -- PEPE --explain prints every term of the score (you need your own Nansen key).

If you write property tests, count how often your generator actually produces the values your branches depend on. I hadn't, until this post.