Before any product description or post goes live on my shop, it gets checked against the thing it's describing. The point is simple. If the page says the guide covers X, the guide had better cover X, or someone's asking for a refund.
For a while that check was one LLM reading the source and saying PASS or FAIL. It worked, mostly. Then I added a second one, Jev, a small claim-check classifier on OpenRouter, and ran both against 62 test cases to see if it was worth it. It was, but not for the reason I expected.
How I tested it
62 cases, built from the real Gumroad product files and my own dev.to posts. 22 had a true claim about their source, 40 had a claim that went too far. The labels come from how each case was built, so there's no arguing about the answer key.
Each case went through both checkers on their own, then got scored.
- DeepSeek (deepseek-v4-flash), asked as a chat prompt: read the source, answer PASS or FAIL
- Jev (typesafe/jev-1.13), which just gives back a probability that the claim is supported
The numbers
| Checker | TP | FP | TN | FN | no answer | Accuracy (answered) | Mean time |
|---|---|---|---|---|---|---|---|
| DeepSeek (chat) | 22 | 2 | 35 | 0 | 3 | 96.6% | 21.5 s |
| Jev, pass at p ≥ 0.5 | 22 | 4 | 36 | 0 | 0 | 93.5% | 0.35 s |
| Jev, pass at p ≥ 0.9 | 21 | 0 | 40 | 1 | 0 | 98.4% | 0.35 s |
| Both combined | 22 | 1 | 39 | 0 | 0 | 98.4% |
FP here is the one that matters: a false claim that got waved through.
A few other things fell out of it:
- Jev is steady. I ran all 62 twice and the scores moved by 0.04 at most, 0.007 on average.
- Jev is cheap. All 62 checks cost $0.0018 in total. Not per check. In total.
- Jev is fast. Median 0.31 s against DeepSeek's 20.6 s.
- DeepSeek went quiet 3 times, no answer at all, and all 3 were false claims. Jev failed all 3 of them, with scores between 0.02 and 0.13.
Why two beats one
They get different things wrong. That's the whole story really.
DeepSeek passed 2 false claims. One was on my family holiday pricing guide. The page claimed it would help you "save at least £25" and that figure isn't a promise the guide makes anywhere. DeepSeek said PASS. Jev scored it 0.13, which is a clear fail.
Going the other way, Jev at 0.5 passed 4 false claims. Three were product summaries that oversold the thing a bit, like saying a guide covers "every password, token, and certificate" when the source never mentions tokens or certificates. DeepSeek failed all three of those, and even quoted the bit that was wrong.
So the rule that's live now is:
- if either checker says FAIL, it fails
- if DeepSeek goes quiet, Jev on its own has to be at 0.8 or above to pass
That combined rule got 61 of 62 right. The one that still gets through is a description of one of my own posts that claims more than the post actually says. DeepSeek passed it and Jev gave it 0.63, just over the line. So that's a known gap and I'm not pretending it isn't.
The number I didn't pick
If I moved Jev's fail line to 0.65, the score goes to 62 out of 62. Tempting. But 0.65 sits in a tiny gap between one false case at 0.63 and one true case at 0.69. Picking that number is just fitting it to these 62 cases. It proves nothing about case 63.
So the thresholds stay where they were: fail under 0.5, pass on Jev alone at 0.8. I'll look again once about 100 real checks from live publishing have built up, which is a fairer test than a set I built myself.
Speed and cost, for the record
- Jev: median 0.31 s, 90% under 0.39 s, slowest 1.58 s. $0.0018 for all 62.
- DeepSeek (answered checks): median 20.6 s, 90% under 26.9 s, slowest 31.4 s, fastest 11.0 s.
Bottom line
The slow one reads properly and catches the oversold summaries. The fast one catches the made-up numbers and doesn't go quiet on you. On their own each lets something through. Together they let through one case in 62, and I know which one it is. For well under a penny a run, I'll take that.
Top comments (3)
Yeah, it does this. Even its docs mentions occasional empty JSONs.
Although, I got at least 1 timeout response from Jev this week. It didn't reply in 1.5s. I'm a little worried that they might not be able to handle the demand. So yeah, I have an LLM backing Jev up in case of an issue. But it's not very convenient because they need different code to do the same job.
Yeah, the quiet DeepSeek thing caught me off guard first time too. Funny thing is, Jev's slowest here was 1.58s on those 62 runs, so I hadn't actually seen a timeout from it. Not something I wrote about, but if Jev went quiet I'd probably let DeepSeek's answer stand on its own since it's the better reader anyway, same fallback, no extra code path. Did you end up wrapping both in a unified interface or keeping two separate call paths?
Well... 1.58s is a lot, I had 1.5s as an upper limit. Set it to higher but I hope this won't become a regular thing. I have a common interface for all LLM providers, Jev sits aside because it's too different. So it's "if dev failed than use whatever model is backing it".