A 200 Response Does Not Prove Search Worked

TL;DR

• HTTP 200 proves that a server accepted a request and returned a response. It does not prove that the requested filter was applied, the evidence was current, the expected tool ran, the answer was supported, the job stopped, or the cost was zero.
• The evidence set contains 153 standardized or direct cases across AgentCore Web Search, Exa, Parallel, and Perplexity. Separate MCP and lifecycle work tests tool discovery, monitoring, webhooks, background runs, cancellation, and cleanup. This is an execution inventory, not a universal quality ranking.
• A successful page fetch contains stale PostgreSQL release data. A successful Perplexity Agent API call with People Search enabled records no invocation; an explicit people-search prompt records one. Successful generated answers introduce an unsupported 501 Not Implemented claim into the fact pattern being tested.
• One seeming AgentCore filter gap belongs to the local adapter and connector version, not the service. The failure sits in the evaluation path.
• Unknown cost is not free usage. Missing billing metadata stays unknown until a defensible rate or invoice establishes it.

---

HTTP success is transport evidence. Search quality requires evidence beyond transport.

That distinction sounds obvious until an evaluation counts every 2xx response as a pass. Agent-search products combine retrieval, page extraction, models, tools, background jobs, monitors, and remote MCP servers. Each layer can return a technically successful response while failing the reason it was called.

The evaluation contract therefore needs six separate proofs:

Layer	What success establishes	What it still does not establish
Transport	The endpoint responded and the client parsed it	The requested behavior occurred
Contract	Parameters, schema, and limits behave as specified	The returned evidence is correct
Evidence	Sources support the required facts and time window	The agent uses the intended tool
Tool invocation	The expected search, fetch, finance, or people tool runs	The final answer uses its output correctly
Lifecycle	Runs reach a terminal state and disposable state is cleaned up	The spend estimate matches the bill
Cost	Usage has a stated basis: provider-reported, list-price estimate, invoice, or unknown	The product is the best choice

A single green status cannot collapse these layers into one result.

153 cases, without a 153-cell leaderboard

The direct execution count across the provider-specific suites is 153:

Provider suite	Standardized or direct cases
Amazon Bedrock AgentCore Web Search	47
Exa Search, Contents, and Answer	41
Parallel Search, Extract, Responses, Chat, Task, and Entity Search	35
Perplexity Search, Agent API, Sonar, and Embeddings	30
Total	153

Those cases do not form one interchangeable benchmark. AgentCore contributes 47 retrieval cases. Exa contributes 41 cases across search, extraction, and generated answers. Parallel and Perplexity cover broader product surfaces. The suites establish which contracts and failure modes receive direct evidence.

MCP sessions, deep-research runs, task groups, monitors, webhook verification, background cancellation, and other lifecycle controls remain separate. Combining them into the 153 would inflate the number without making the comparison more useful.

The retrieval lane uses 30 fixed questions, four providers, and 1,200 results. It measures retrieved evidence under fixed conditions. It does not establish a universal answer-quality winner.

A successful fetch can return stale evidence

The page-fetch control sends the same four URLs through two extraction products. Every request succeeds. Every returned URL matches. Every body is nonempty and stays within the configured character cap.

One result demonstrates stale evidence.

For the PostgreSQL versioning page, one fetched copy reports PostgreSQL 18.4 and 17.10. The other reports 18.6 and 17.11, matching the official page in the frozen reference check. The stale response has valid text, a valid URL, and a successful status.

Transport: pass. Content: fail.

This changes the minimum test for page fetch. Nonempty text is only a structural assertion. Freshness-sensitive extraction also needs a dated reference fact, a live-source comparison, or a cache-control experiment. A fetch product can satisfy its API contract while returning evidence that no longer satisfies the user's question.

Enabling a tool does not prove the tool ran

Agent APIs add another gap between configuration and execution. Perplexity documents that tools must be configured in the request, and the model decides when to invoke them from the prompt and instructions.

That distinction appears in the People Search case. A request enables People Search and completes successfully, but its usage record shows no People Search invocation. The prompt is a plausible people-research request, yet the model answers without the specialized tool.

The explicit control names the people-search task. Its usage record contains one search_people invocation, matching the People Search usage field documented by Perplexity.

Both requests completed. Only one proved the intended capability.

Tool evaluations need three assertions:

1. The tool is present in the request or discovered through MCP.
2. The response records that the tool was invoked.
3. The output contains evidence produced by that invocation.

Tool availability is configuration evidence. Tool-call telemetry is execution evidence.

Generated answers can succeed and add unsupported facts

Generated-answer endpoints expose another kind of false positive. The response can be fluent, cited, and structurally valid while adding a claim that its cited source does not support for the tested operation.

The frozen question asks about Amazon S3 conditional writes using If-None-Match and If-Match. The official conditional-write reference documents 200 OK, 412 Precondition Failed, and concurrent-delete cases that produce 409 Conflict or 404 Not Found.

Across the saved Perplexity configurations, 13 comparable answers have frozen grades against that reference: eight complete and five incorrect. Four incorrect configurations add 501 Not Implemented to the tested write behavior. Parallel's saved Chat review contains three incorrect answers out of five, with unsupported 204 No Content or 501 Not Implemented claims added to the same fact pattern.

There is a nearby S3 policy page where 501 is valid for a different operation: CopyObject requests using those conditional headers. That is exactly why citation presence is insufficient. A true statement from a related page can become a false answer when moved to the wrong operation.

This is one frozen fact pattern. It does not rank either provider's general accuracy. It proves a narrower point: a successful generated answer needs claim-level checking against the operation, scope, and source text.

Capability exists at three layers

A vendor feature is usable only when three current layers agree: the vendor documents it, the active endpoint exposes it, and the adapter forwards it.

The AgentCore connector-version reference identifies request-level domain and published-date filters as a version 1.2.0 capability. The active target exposes that schema, the adapter forwards the filter object, and the fixed suite returns 40 of 40 compliant constrained results.

Checking only one layer can create a false product claim. Documentation alone does not prove account exposure. Endpoint discovery alone does not prove the client sends the field. A successful client request does not prove the provider honored the constraint.

A credible evaluation stores enough detail to locate the failing layer:

• endpoint and connector version;
• discovered tool schema;
• exact request shape emitted by the adapter;
• raw response shape;
• client-side validation result.

Without that trail, a local omission becomes a product claim.

Long-running search needs a cleanup proof

Background research and monitors create state that outlives the request. Their success criteria include termination.

Exa Monitors require a public HTTPS webhook at the final destination; HTTP, localhost, private addresses, and redirecting URLs do not satisfy the delivery contract. A useful monitor test therefore needs more than successful creation. It needs signature verification, a triggered or scheduled run, delivery evidence, lifecycle updates, and deletion or another documented terminal cleanup action.

The Exa Monitor row remains not run because the evaluation has no approved public webhook. A create response without delivery evidence would not satisfy the monitor contract.

Parallel's event-stream and snapshot monitor evidence covers create, retrieve, update, and cancel. The terminal read-back shows both as cancelled. The Parallel cancellation contract describes cancellation as irreversible and exposes active and cancelled states. Its webhook guide adds signature verification, retry behavior, and duplicate-event handling to the production contract.

Cancellation deserves the same discipline. One Exa Agent case contains an accepted graceful-stop response alongside an active run with increasing usage. Explicit cancellation plus a terminal read-back establishes cleanup. The accepted stop response alone does not.

For stateful search, cleanup evidence is part of correctness:

create -> retain identifier -> poll -> verify output -> stop or cancel
       -> read terminal state -> confirm no active test runs remain

Losing the identifier or skipping the final read converts a successful request into an unmanaged job.

Unknown cost is not zero

Search evaluations often make one last status-code mistake: treating absent cost metadata as $0.

The Perplexity pricing model separates model tokens from tool invocations and reports request cost when that field is available. The official MCP calls in this evaluation return no usage record, so their cost basis is a public list-price estimate. One Deep Research response reports zero cost alongside substantial citation tokens, reasoning tokens, and searches; the evidence therefore carries both the provider-reported value and a separate conservative list-price estimate.

Exa has three accepted or attempted paths whose billing cannot be established from the returned evidence. Their cost remains unknown. Parallel's accounting separates standardized-run totals from lifecycle additions and stops at public rates rather than claiming invoice proof.

Every cost number needs a basis:

• Provider-reported: returned by the API, not invoice-verified.
• List-price estimate: computed from a dated public rate card and observed usage.
• Invoice-verified: reconciled against billing data.
• Unknown: no defensible basis.
• Not incurred: a dry run that made no provider request.

Unknown and zero answer different questions.

The evaluation artifact is the product

The durable output of an agent-search test is not a pass count. It is a chain of evidence:

request manifest
  -> transport record
  -> contract checks
  -> source and claim review
  -> tool-call telemetry
  -> lifecycle ledger
  -> cost basis

That chain explains which layer passes, which layer fails, and whether the failure belongs to the vendor, the adapter, the model, the evidence, or the test itself.

The open thread is portability. Each provider exposes different usage fields, lifecycle states, citations, and tool-call traces. What is the smallest evidence contract an agent harness can require across all of them without discarding the details needed to explain a failure?

---

Part 4 of the Search for Agents series.
← Part 3: Search Is a Pipeline, Not a Tool · Part 2: What Actually Differs Across Search APIs Built for Agents · Part 1: Search for Agents Is Becoming Its Own Market