DEV Community

dodou
dodou

Posted on

Verify LLM Claims With a $3 Search Budget

An LLM confidently stating a wrong fact is worse than a blank answer — it sounds the same as a right one. The cheapest reliable fix is a search-backed verification pass: extract the claims, search each one, and mark unsupported claims.

At 1 credit per search, a verification pass is cheap. SerpBase's Starter Boost is $3 for 10,000 searches — enough for thousands of claim checks.

The verification loop 

import requests

API = "https://api.serpbase.dev"
KEY = "your_api_key"

def verify_claim(claim):
    """Search a claim, return whether the top results support it."""
    r = requests.post(
        f"{API}/google/search",
        headers={"X-API-Key": KEY},
        json={"q": claim, "hl": "en", "gl": "us"},
        timeout=10,
    )
    r.raise_for_status()
    organic = r.json().get("organic", [])
    if not organic:
        return {"claim": claim, "supported": False, "reason": "no results"}
    top = organic[0]
    return {
        "claim": claim,
        "supported": True,
        "top_title": top.get("title"),
        "top_link": top.get("link"),
    }
Enter fullscreen mode Exit fullscreen mode

How to extract claims

Ask the LLM to enumerate its own factual claims before verification:

List the factual claims in your answer, one per line, as standalone search queries.
Enter fullscreen mode Exit fullscreen mode

Each claim becomes a query. Run the loop, and any claim whose top result doesn't relate to it gets flagged.

What this catches

  • Fabricated statistics ("revenue grew 23% last year" with no source).
  • Confused entities ("the CEO of X is Y" when Y left in 2024).
  • Stale facts that changed since the training cut-off.

It won't catch subtle misreads, but it removes the loudest failures.

Cost

A 10-claim verification = 10 searches. At 1 credit each, the $3 Starter Boost covers ~10,000 claim checks. Even at full Starter rate, verification is a rounding error next to LLM inference tokens.

Honest limits

  • A top result is not proof; it's evidence. For high-stakes claims, pull a few results and read snippets, not just the top link.
  • Verification adds latency. Run it async for non-interactive pipelines.
  • The model can game the pass if you let it pick its own queries; a fixed extractor is more reliable.

Full parameter and response reference: serpbase.dev/docs.

Top comments (0)