Most agent systems start with a simple integration decision: choose a provider for search, scraping, or browser automation, then call it whenever the agent needs the web.
That works in a prototype. It becomes fragile in production because “web access” is not one capability.
A search request needs relevant links. A scraping request needs clean content from a known URL. A sourced-answer request needs synthesis with citations. A browser-action request needs successful interaction with a live page. These tasks have different success conditions, latency profiles, costs, and failure modes.
The routing unit should therefore be the capability, not the vendor.
This article presents a practical way to build that routing layer. The goal is not to choose one universally best provider. It is to make each tool call measurable, replaceable, and aligned with the outcome the agent actually needs.
Start with the outcome, not the endpoint
Before selecting a provider, define what a successful result means for each capability.
A useful capability contract contains five parts:
- Input contract: query, URL, schema, document, or natural-language objective.
- Success predicate: the minimum conditions that make the response usable.
- Quality measure: relevance, extraction accuracy, coverage, content cleanliness, or task completion.
- Deadline: the latency budget available before the agent should fall back or stop.
- Failure policy: which errors are retryable, which require another provider, and which should be returned to the agent.
Consider web search. A 200 OK response is not enough. The result can still be useless if it contains irrelevant links, misses the requested time range, or returns snippets without enough context for the next reasoning step.
For document parsing, success might require readable text, a minimum page-coverage ratio, and OCR on scanned pages. For browser actions, success should be tied to the requested state change rather than the absence of an exception.
This distinction prevents a common observability mistake: counting transport success as task success.
Build one scorecard per capability
Once success is explicit, evaluate providers within each capability instead of combining unrelated tasks into one global ranking.
A practical scorecard needs four dimensions.
1. Quality
Quality must match the capability.
- Search: result recall or relevance at a chosen cutoff.
- Scraping: content completeness, markdown cleanliness, and success on difficult pages.
- Structured extraction: field-level precision and recall against a known schema.
- Crawling: discovered-page coverage and duplicate control.
- Document parsing: text accuracy, reading order, and OCR coverage.
- Browser actions: completion of the requested interaction and verification of the resulting state.
Avoid one generic “quality” test across all of them. The metric should describe what a downstream agent can safely use.
2. Latency
Measure wall-clock latency from the caller’s perspective. Median latency is useful for normal behavior, but it should not be the only number used for operational decisions.
A routing policy also needs a timeout based on the agent’s total deadline. If a workflow has ten seconds left, a provider with excellent quality and a twelve-second median is not a viable primary route for that call.
Tail latency matters when calls are chained. Several individually acceptable delays can consume the entire agent budget before the final synthesis step begins.
3. Cost per successful result
Price per request is easy to compare and often misleading.
The operational metric is:
Cost per successful result = total billed cost / number of usable outcomes
Assume one provider charges $0.001 per request and succeeds on half of the tasks. Another charges $0.0015 and succeeds on 90%. Ignoring retries and secondary processing, their effective costs are:
- Provider A: $0.002 per successful result.
- Provider B: about $0.00167 per successful result.
The nominally cheaper provider is more expensive when measured against the outcome.
Failures should remain in the denominator calculation even when they return a technically valid response. If the agent cannot use the data, the system still paid for an unsuccessful attempt.
4. Error rate
Track transport errors, timeouts, malformed responses, and capability-level failures separately.
This separation helps answer different questions:
- Is the provider unavailable?
- Is the integration parsing the response incorrectly?
- Is the provider returning valid but low-quality data?
- Is the target site blocking the request?
- Is the task outside the provider’s supported shape?
A single error-rate number hides the remediation path.
Keep the corpus fixed
Comparisons are only useful when providers receive the same work.
Use a versioned task corpus per capability. Every provider in a given evaluation should receive the same queries, URLs, target schemas, deadlines, and pass criteria. When the corpus changes, increment its version and record a new run date.
This controls a subtle source of bias: giving one provider easy pages and another provider pages protected by anti-bot systems, then comparing their success rates as if the workloads were equivalent.
The corpus should include difficult cases, not only demo-friendly inputs. Production failures tend to appear on scanned documents, dynamic pages, rate-limited domains, unusual schemas, and queries that require evidence from more than one source.
One public example of this design is the capability-based benchmark methodology operated by NativePort. It keeps a fixed corpus per capability and publishes quality, median latency, cost per successful call, and error rate with run dates. The important pattern is not its particular composite score; it is the separation of capabilities and the publication of the underlying measurements.
Separate selection from execution
The agent should ask for a capability. A routing layer should decide which provider executes it.
Conceptually, the request contains:
- required capability;
- input payload;
- deadline;
- minimum quality threshold;
- optional constraints such as geography or output format.
The router evaluates only providers that support that contract. It then selects a primary route using the latest eligible scorecard and current operational health.
Do not let provider-specific request fields leak into the agent’s planning interface unless they represent a real user requirement. Otherwise, every provider change becomes an agent-prompt change.
At the same time, avoid forcing every upstream response into one overly generic schema. Normalization is useful at the routing boundary, but capability-specific information should remain available when the agent needs it.
A good compromise is a stable envelope containing status, timing, cost, provenance, and an outcome classification, with the capability response preserved inside it.
Treat fallback as a policy, not a retry
Retries and provider fallbacks solve different problems.
A retry sends the same task to the same provider, usually after a transient failure. A fallback sends it to a different provider because the first route cannot satisfy the capability contract within the remaining budget.
The AWS Builders’ Library guidance on retries and backoff explains why retries need timeouts, limits, backoff, and jitter. Unbounded retries can amplify overload and turn a partial failure into a broader incident.
For agent tools, the fallback policy should consider:
- whether the failure is transient;
- whether the call is safe to repeat;
- remaining latency budget;
- cost already spent;
- whether another provider offers a genuinely independent path;
- whether duplicate execution could create an external side effect.
Search and read-only scraping are usually easier to retry than browser actions that submit forms or mutate state. For stateful actions, the system needs idempotency or post-action verification before it can safely try again.
A useful rule is to retry only when the same route is likely to produce a different outcome. Otherwise, switch routes or stop.
Instrument outcomes, not just requests
Routing improves only if production feedback reaches the scorecards.
Record at least:
- capability and corpus or task class;
- selected provider and fallback sequence;
- start time, deadline, and duration;
- billed cost when available;
- transport status;
- parse status;
- capability-level success;
- reason for fallback;
- final outcome used by the agent.
Use distributed traces to connect the agent decision, tool call, retry, fallback, and final result. The OpenTelemetry HTTP semantic conventions provide a standard basis for HTTP client spans, while capability and outcome fields can be added as application attributes.
Be careful with high-cardinality data. Raw queries, full URLs, and extracted content may contain sensitive information and can make telemetry expensive. Prefer bounded task classes, hashed identifiers where appropriate, and explicit retention rules.
Most importantly, preserve the difference between “request completed” and “agent received a usable result.” The second metric is the one the router is meant to improve.
Roll out without creating a routing black box
A capability router should be explainable to operators.
For every decision, retain enough information to answer:
- Which providers were eligible?
- Why was the primary route selected?
- Which threshold triggered the fallback?
- How old was the benchmark data?
- Did the final result meet the success predicate?
- What did the complete attempt cost?
Start in shadow mode. Let the existing integration continue serving traffic while the router computes the decision it would have made. Compare those decisions with real outcomes before enabling automatic switching.
Then introduce routing one capability at a time. Search is usually easier than stateful browser automation because the success predicate and retry behavior are simpler. Keep a manual override for incidents and a deterministic default for cases where score data is missing or stale.
The router should degrade predictably. Missing benchmark data must not silently become evidence that a provider is good.
A practical implementation checklist
Before enabling capability-first routing, confirm that:
- every capability has a written success predicate;
- providers are compared on identical, versioned tasks;
- failures are included in effective-cost calculations;
- transport errors and unusable responses are classified separately;
- retries have limits, timeouts, backoff, and jitter;
- fallbacks respect the remaining deadline and side-effect risk;
- traces connect provider attempts to the final agent outcome;
- benchmark dates are visible to operators;
- missing or stale data has an explicit default policy;
- routing decisions can be explained after the fact.
The central idea is simple: an agent does not need “a web provider.” It needs a successful search, extraction, crawl, parsed document, or completed browser action within a specific budget.
When routing follows those capability contracts, provider choice becomes an operational policy rather than a permanent architectural commitment. That makes the agent easier to measure, safer to fail over, and less expensive to evolve.
Disclosure: This article is published by NativePort, whose public benchmark methodology is cited as an implementation example. It does not recommend a product or provider. The draft was prepared with AI assistance and reviewed against the cited sources and public methodology before publication.
Top comments (0)