Jev (by TypeSafe AI) launched last week and instantly became one of the most discussed releases on Hacker News. Every review we found tested the same thing: speed — latency benchmarks, structured output formats, injection resistance.
Nobody checked the other half: are the claimed results true?
So we did what we do. We took NanoJev — the open-source 0.6B replica that hit GitHub Trending on Sep 19 — and ran it through our three-gate verification protocol. Honest boundary up front: we could not verify TypeSafe's official Jev, because the official release ships no replayable evidence. NanoJev does — per-file SHA-256 manifests, frozen cohorts, the original controller rules, and frame-by-frame trajectory logs. That's exactly what makes this audit possible.
Gate 1: Evidence integrity
Every claimed data-file hash, recomputed and compared: builder/output/cohort manifests 3/3 PASS; random sample of source files 10/10 PASS.
One confession worth making: our first probe run failed every hash. Discipline says: when your probe reports red, suspect the probe first. It was our verifier's Windows CRLF conversion mutating bytes. Re-cloned with core.autocrlf=false — all green. Falsify yourself before you falsify others; that rule nearly saved us from slandering an honest repo.
Gate 2: Claim ↔ record reconciliation
All six README numbers (attempts / collisions / food / outcome across 3 systems × 2 games) checked against side_by_side_results.json. Zero discrepancy.
Gate 3: Rule-level replay
Numbers can be hand-written. So we replayed the recorded trajectories against the published controller rules, without model weights:
- Frame-level collision audit: claimed collisions exactly match in-frame flags — Jev 1044/1044, NanoJev 36/36, untuned Qwen 2044/2044. Zero mismatch across all three systems.
- Step accounting: frames = claimed steps + 1 in all six runs (initial-state frame convention, structurally consistent).
- Kinematic legality: every position change is a legal move or a stay — zero illegal jumps in NanoJev's 245-frame maze trajectory.
Verdict: agree — at integrity + consistency + rule-replay level.
Why this matters beyond one repo
Chat-model verification checks a response. Decision models are different: dozens of parallel decisions per second, where no single decision is "right" — the verifiable unit is the whole trajectory (initial state + frozen rules + claimed end state, replayed).
NanoJev also deserves credit: its evidence hygiene (per-file hashes, provenance categories, frozen cohorts, explicit controller rules) is the best we've audited in a public AI project to date. Verification of honest work is a gift, not a threat.
Reproduce it yourself
pip install assay-verify
The signed receipt (Genesis Receipt #2) is public on our receipt wall; anyone holding the public key can re-verify at zero cost. Challenge window: 90 days — and yes, that includes challenging our verdict.
We're a small independent org building an open verification protocol for AI claims (Assay Protocol v0). No funding from, or relationship with, TypeSafe or the NanoJev author. Disagree with the verdict? That's what the challenge window is for.
Top comments (0)