Jev is a brilliant decision model. Gemini Flash-Lite is the model nobody talks about, and it’s fast, accurate, and nearly free. Here's what we meas...
For further actions, you may consider blocking this person and/or reporting abuse
This landed at exactly the right time. I'm building with Jev this week for the AdaL hackathon, and your post changed how I'm writing my questions before I sent a single real one.
My job looks a lot like your router: I check whether a README's claims match the code. Jev labels each README line, deterministic verifiers decide what's true, and Claude Haiku explains the findings. Same split you landed on: Jev decides, a writer writes.
The line I'm stealing: "describe the checker's blind spot, not the check's shape." My worst failure is a false "Contradicted," which is your false green. When in doubt, my scanner says "can't tell."
I credited you in the README. Thanks for testing instead of hyping. 🚀
Good luck @earlgreyhot1701d , thanks for sharing!
it's interesting that examples made Jev perform worse. it shows that prompt examples aren't always helpful when a model starts matching patterns instead of checking what a tool can actually verify. i think this needs to be tested on a much larger and more diverse set of checks.
I thought this was one the most unintuitive parts TBH. It's a feature when something is generative (and therefore open ended), but a negative when it's constrained and precise.
The blind spot vs flip split is the part I'd lean on, and it changes how the numbers should be counted. Jev's 35 false-positive runs come from the same few checks failing every time, so the unit that carries information is the distinct check, not the run. "35 to 0" is then a count of checks (maybe 5 to 0), and zero out of N distinct unseen checks only bounds the true rate at about 3/N at 95%. Worth printing N next to the zeros, and for Flash-Lite a per-check flip rate (share of checks that are not unanimous across repeats) rather than pooled runs.
Two smaller things on the BFCL card. Both models scored the same items, so 1.1% vs 4.3% false positives is a paired comparison (count the items where only one of them called a function; McNemar on those), not two independent proportions. And if the 0.8 bar was picked while looking at that same slice, the 1.1% is optimistic; picking it on one half and reporting the other, as you did for the rewrite, fixes that.
The seven "none fits" items you found are a nice by-product: do the corrected labels move Flash-Lite's 96.9% too, or only the false-positive column?
Fair points on the counting. The unit is the check, not the run, so printing N next to the zeros and reporting a per-check flip rate for Flash-Lite would be more precise.
BFCL is in the post for one reason: our checks come from one project, so I wanted to see if the finding held on public data. It did. Jev's misses are a blind spot a prompt can close, Flash-Lite's are noise you measure for, and the pair gets Jev's false-positive rate with Flash-Lite's arguments for less than half the price. Paired test or not, that shape is the same on both datasets. Haven't seen McNemar before, it's a good suggestion!
On the seven items: we dropped them rather than relabel, so both columns exclude them. Crediting Flash-Lite's calls would move its accuracy up, not just the false-positive column.
@theycallmeswift , this is an absolute masterclass in constraint-driven ai architecture. the "jev + flash-lite" combo is exactly the kind of ruthless optimization i champion. when you're building on severe hardware constraints (like my $150 phone), every millisecond and token counts. using a cheap, precise router to decide if a heavier model is even needed is brilliant.
the insight that hit hardest: "describe the checker's blind spot, not the check's shape." that single sentence flips the entire mental model of prompt engineering from "what does the input look like" to "what can this tool actually fail to see?" it’s a profound shift that i will absolutely be applying to koda's own lightweight validation flows.
it’s also fascinating that examples made jev worse by triggering lazy pattern-matching, while a mechanical failure-mode test fixed it. it perfectly proves your point: smarter models don't fix bad prompts; they just mask them at a higher cost.
quick question on the production mix: when jev defers to flash-lite for argument generation, do you find that flash-lite occasionally hallucinates the arguments despite the strict schema, or does the shortened, highly-specific context (just the chosen checker) keep it perfectly locked in?
fantastic, deeply practical breakdown. this is going straight into my engineering playbook! 🐯⚡
Your blind-spot framing matches the jev-belay Stop-gate README, where Jev decides whether a coding agent's 'done' is honest: AUROC 0.976 versus 0.777 for judging the wording alone, 8 turns blocked in 100 and 7 of them rightly. Same caveat as the comment above on N: the 100 stops with 12 false dones were labeled by a Sonnet run, not by hand, so the interval is 0.915 to 1.000.
I think the paired setup makes sense: a cheap router, then a writer only when needed. One metric I'd add is cost per resolved assertion, including deferrals that need human review. It might change the economics compared with cost per 1,000 checks. Have you measured the rate of manual follow-up?
Routing each check to a deterministic checker (file_exists, glob_count, regex) before it hits a judge is the step most eval setups skip, and it is where the cost goes. Flash-Lite winning the router job lines up with what I saw when I looked at where a built-in judge quietly under-checks: dev.to/kartik-nvjk/aws-bedrocks-bu... . How do you score the router itself when it sends a regex-able check to the model?
the "brilliant but staying with what works" call is the most honest thing in applied AI right now. everyone feels pressure to adopt the hyped model but the ones who actually run the numbers usually end up exactly here: the contender wins the benchmark, the incumbent wins the real workload.
the calibrated probabilities framing is what makes Jev distinct though. for grading agent output you are turning a subjective rubric into a probability distribution over outcomes. that is where a decision model earns its place over a plain scorer. having benchspec as the eval layer makes this apples to apples.
what does the accuracy gap look like when agent output sits in truly ambiguous territory? is there a confidence threshold where Jev calibration starts mattering more than the cost delta?
Why is a "false positive" (sending a check to a deterministic code checker when it can't actually verify it) considered the most dangerous mistake for the router?
@rahul_r15 it costs user trust because you can end up with a passing assertion that is testing/validating the wrong thing. Imagine saying "No .DS_Store files exist" and the binder matches to the wrong function which passes even though your code generated a subfolder full of files named .DS_Store