We gave Claude 70 claims, each with the evidence offered for it. Unguided, it almost never passed weak proof, yet it turned down a third of the sound ones.
Mostly by refusing evidence that was fine. We wrote 70 short cases, each a claim about a Cloudflare site, a payment on the Base chain or a web page, paired with the check someone ran to back it, then had Claude Sonnet and Claude Haiku decide whether the check settled the claim. With no guidance, Sonnet called a flawed check proven once in 45 tries, yet it turned down 9 of the 25 checks that did settle their claim. Haiku never passed a flawed check and turned down 12 of the 25 sound ones. The mistake we had built the test around, a model nodding at a status 200 that proves nothing, was the rare one.
Read the full report on AISkills402: https://aiskills402.com/blog/llm-doubts-sound-evidence
Top comments (0)