Every AI agent that answers questions about the world needs web search, and choosing the API behind it is surprisingly hard. Vendor benchmarks are run by vendors, and most public evals use generic trivia that looks nothing like what an agent actually sends.
So I built a small, open web search API evaluation around agent-style calls, and tested You.com, Tavily, Exa and Parallel live.
TL;DR
| If your agent… | Pick | Why |
|---|---|---|
| Answers inside a live turn (voice, chat, alerts) | You.com | Fastest at p50 and p95, freshest results, never returned empty |
| Does deep research where seconds don't matter | Parallel | Highest accuracy overall and in finance |
| Needs the most on-topic top 5 for semantic or long-tail queries | Exa | Best precision@5, best insurance accuracy |
| Looks up docs or how-to answers | Tavily | Best GTM accuracy, right on the config-docs query |
There's no single winner. The right API depends on what your agent does.
How I tested
30 queries, written the way agents call search. Each one has an upstream task ("Any new mentions of Apache Iggy in the last day?"), the exact search string the agent sends ("Apache Iggy"), parameters (a 24-hour window), and what the agent needs back. They're split across three common agent use cases:
- GTM agent: fresh developer and company buying signals (new issues, releases, funding, incidents)
- Insurance agent: a voice and claims agent that needs live catastrophe status mid-call, with a 500 ms budget
- Finance agent: an analyst copilot that needs exact, current numbers with a primary-source citation
Every use case has at least four real-time queries, and some expect "none" as the right answer. That catches APIs that present stale pages as current.
Same request for every API: 10 results, full content where supported, the same freshness window and domain filters. Each query ran 3 times through promptfoo (90 calls per API). Latency is client wall-clock around the HTTP request after a discarded warm-up call.
Grading: answer keys were captured live and archived to the Wayback Machine. Answers were graded with the published SimpleQA rubric (correct = 2, partial list = 1, wrong = 0), with FreshQA strict rules for "none" answers and stale results.
Overall results
| Metric (90 calls per API) | You.com | Tavily | Exa | Parallel |
|---|---|---|---|---|
| Accuracy (% of max) | 63% | 51% | 53% | 67% |
| Queries fully correct in all 3 runs | 57% | 37% | 50% | 57% |
| Precision@5 | 37% | 26% | 40% | 26% |
| Results inside the query's window | 59% | 37% | 44% | 29% |
| Empty responses (of 90) | 0 | 27 | 12 | 3 |
| Latency p50 / p95, client (ms) | 593 / 804 | 1,864 / 3,305 | 1,106 / 3,997 | 2,038 / 4,117 |
| Search cost per correct answer | $0.026 | $0.016 | $0.012 | $0.008 |
What happened in each use case
GTM: "what's new?" favors freshness
You.com was the only API to surface the day's new Apache Iggy activity, and it was right on that week's funding rounds and agent-memory posts. Parallel won when the agent needed the official page, such as a project's release notes. Tavily won the documentation lookup (Kafka consumer settings). For an agent that watches for fresh buying signals, freshness matters most. For one that mostly looks up release notes or docs, Parallel or Tavily fit better.
Insurance: live status is hard for everyone
You.com returned the latest hurricane advisory and answered every call in under a second. For current wind speed, though, it returned a cached weather page instead of a live reading. Exa was the only API to get both the live wind speed and both county warnings, which is why it leads insurance accuracy. No API found the active binding suspension from the state insurer of last resort. An insurance agent still needs a direct check of carrier and regulator pages.
Finance: two reliable APIs, one tie-breaker
You.com and Parallel both reliably answered live market and filing questions: the NVDA price, the 10-year yield, Bitcoin, the Fed minutes, a fresh 8-K and an earnings date. Parallel edged ahead because it was the only API to find the right week's jobless-claims report. Tavily returned nothing on queries limited to government domains, and Exa mostly returned stale prices.
Developer setup is part of the product
I also timed signup to a working API key from screen recordings.
Tavily was fastest (25 seconds, 3 steps). Every provider's first API call worked on the first try.
Limits
- 10 queries per use case is a focused test, not a leaderboard. A difference of a query or two is noise.
- Results reflect one day (Oct 8, 2026). Real-time answer keys are only valid for that date.
- Costs are list prices and cover search only, not the answering model.
Use it, rerun it, extend it
Everything is open and MIT-licensed:
- Full results and charts: psinglap.github.io/websearch-api-eval
- Harness, answer keys and grades: github.com/psinglap/websearch-api-eval. Clone it, add your keys and run your own queries.
-
Ask about the results from Claude, ChatGPT or Cursor (MCP): add
https://websearch-api-eval.psberkeleymba.workers.dev/mcpas a connector, no sign-in. -
Run it locally, plus live tests with your own keys:
npx -y websearch-api-eval-mcp -
Claude Code plugin:
/plugin marketplace add psinglap/websearch-api-eval - Dataset: huggingface.co/datasets/psinglap/websearch-api-eval
Which queries or providers should I add next? I'd especially like ideas for making real-time answer keys reproducible.
Top comments (2)
Thirty queries tell you which API won on those thirty. To say what each one is good at, every query needs an answer you can check. Without that, the grading turns into reading, and reading rewards the answer that sounds most sure.
Useful breakdown — the per-use-case split is more actionable than the usual one-winner eval. The insurance result matches what I'd expect: none of these indices update fast enough for anything mid-call, so a fallback to the carrier/regulator page directly is unavoidable anyway.
Question on methodology: when you say same request for every API, did you normalize per-provider domain filtering/exclusions, or take each API's defaults? Some of these re-rank aggressively behind the scenes, so a query-by-query grid can partly reflect their threshold choices rather than raw retrieval quality. Also curious how you score the "none" expected answers — partial credit or binary? That's the part that best separates stale-from-current.