Cutting Agent Costs with Trace-Based Model Routing | Agent Lab Journal
AL
Agent Lab Journal
Guides
Glossary
PRACTICE · MODEL ECONOMICS AND EVALUATION
Cutting agent costs with trace-based model routing
Level: advanced
Reading time: 90 minutes
Outcome: a trained router and a verified operating report
Using the strongest model for every request is expensive, but replacing it with a cheaper model everywhere causes the difficult answers to fail. The useful middle ground is not a handwritten list of “easy” prompts. It is a router trained and tested on traces from the application it will actually serve. This guide builds that router, deploys it safely, and measures the complete outcome: cost, accepted quality, latency, wrong routes, escalations, automatic retries, and corrective user requests.
What you will build
You will build a two-route policy for an AI agent:
cheap sends an operation to the less expensive model;
strong sends it to the more capable model;
a failed cheap attempt may escalate to strong, but that escalation remains visible in the report.
The router does not claim to measure intelligence or universal prompt difficulty. It predicts a narrower event: whether the cheap route is likely to satisfy this application’s acceptance contract, with the current instructions, tools, validators, and input distribution.
The completed work should leave six reproducible artifacts:
a privacy-reviewed dataset derived from production-like traces;
paired cheap and strong outcomes for the same operations;
a versioned router with a calibrated decision threshold;
a shadow-mode comparison against fixed routing policies;
a canary rollout with an immediate strong-only fallback;
a report covering cost, quality, latency, routing errors, escalations, and retries.
No benchmark numbers are supplied in this article. Model prices, traffic mixes, quality requirements, and application behavior differ. Every numeric result must come from your own replay and live observations.
Concrete case: an internal support agent
Use an internal product-support agent as the working example. This is a reproducible design scenario, not a claim about a customer or a previously measured deployment.
The agent performs four operations:
answers questions from an approved knowledge base;
extracts required fields from a support request;
classifies the request and proposes the next action;
uses a read-only tool to retrieve ticket status.
The cheap model may be sufficient for direct retrieval, short classification, and constrained extraction. The strong model may be necessary when the context is contradictory, the policy is ambiguous, the conversation is long, or several dependent tool calls are required.
Both routes must receive equivalent inputs: the same normalized conversation, the same retrieved documents, the same prompt version, the same tool permissions, and the same output contract. If adapters differ because providers expose different interfaces, record the adapter version as part of the route.
The routing unit is one completed business operation, not one model call. An operation starts when the application accepts a user request and ends when it returns an accepted result, safely refuses, reaches a terminal error, or exhausts its retry budget.
1. Define acceptance before optimizing cost
Before collecting labels, write an operation contract. “The request returned HTTP 200” is not a quality criterion. The contract must describe an outcome the product can accept.
operation: support_answer:v4
starts_when:
- user_message_accepted
passes_when:
- output_schema_valid
- policy_checks_pass
- required_claims_supported_by_supplied_sources
- requested_read_action_completed_or_safely_declined
- no_critical_violation
- no_corrective_user_retry_inside_evaluation_window
fails_when:
- deterministic_validator_rejects
- expert_review_rejects
- unauthorized_action_attempted
- retry_budget_exhausted
- operation_deadline_exceeded
quality_floor:
critical_violation_rate: 0
accepted_operation_rate: PRODUCT_DEFINED_VALUE
latency_budget_ms: PRODUCT_DEFINED_VALUE
user_retry_window: PRODUCT_DEFINED_VALUE
Replace every PRODUCT_DEFINED_VALUE with an approved threshold. There is no universal acceptable error rate. A weak draft title and an incorrect account-status action do not carry the same risk.
Use three layers of evaluation:
Deterministic checks: schema validity, required fields, tool allowlists, argument constraints, supported citations, and prohibited values.
Expert checks: factual correctness, policy interpretation, sufficient explanation, and appropriate refusal.
Behavioral checks: a corrective user message, a human takeover, a repeated operation, or successful task completion.
Name each label according to what it proves. A JSON validator proves schema_valid, not answer_correct. Combining weak signals under an ambitious label is an easy way to train a confidently wrong router.
2. Establish policies worth comparing
Evaluate at least three policies on the same eligible operations:
Policy
Behavior
Purpose
strong_only
Every operation starts on the strong route
Reference quality, cost, and latency
cheap_only
Every operation starts on the cheap route
Maximum nominal savings and the resulting quality loss
rule_router
Predefined rules select the route
A simple baseline the learned router must beat
trace_router
A trained score plus safety rules selects the route
The policy being evaluated
A practical rule baseline can route operations with tools, large contexts, previous failures, or high-risk action types directly to strong. A learned router is justified only if it produces a better cost-risk trade-off than this understandable baseline.
3. Instrument the complete operation trace
A trace must connect the user request, routing decision, every attempt, every model call, validator results, tool activity, and the terminal outcome. This is the minimum observability needed to distinguish genuine savings from hidden retries.
{
"trace_id": "generated",
"operation_id": "generated",
"operation_type": "support_answer:v4",
"cohort": "eligible_support",
"policy_version": "strong_only:v1",
"router_version": null,
"feature_version": "features:v1",
"prompt_version": "support:v4",
"selected_route": "strong",
"router_score": null,
"decision_reason": "baseline",
"started_at": "ISO-8601",
"completed_at": "ISO-8601",
"terminal_status": "accepted",
"quality_pass": true,
"critical_violation": false,
"user_retry": false,
"retry_of_operation_id": null
}
Record each model call separately. At minimum, include:
call_id, operation_id, and attempt number;
provider, actual model identifier, route, and adapter version;
input, cached-input, reasoning, and output token counts when exposed;
confirmed charge, currency, and pricing-version identifier;
queue time, time to first token, and total call latency;
schema, policy, factual, and tool-result checks;
error class, automatic repair, retry, and escalation status.
CREATE TABLE router_operations (
operation_id TEXT PRIMARY KEY,
trace_id TEXT NOT NULL,
operation_type TEXT NOT NULL,
cohort TEXT NOT NULL,
policy_version TEXT NOT NULL,
router_version TEXT,
feature_version TEXT NOT NULL,
selected_route TEXT NOT NULL,
router_score DOUBLE PRECISION,
decision_reason TEXT NOT NULL,
started_at TIMESTAMP NOT NULL,
completed_at TIMESTAMP,
terminal_status TEXT,
quality_pass BOOLEAN,
critical_violation BOOLEAN,
user_retry BOOLEAN NOT NULL DEFAULT FALSE,
retry_of TEXT
);
CREATE TABLE router_calls (
call_id TEXT PRIMARY KEY,
operation_id TEXT NOT NULL,
attempt_no INTEGER NOT NULL,
route TEXT NOT NULL,
model_id TEXT NOT NULL,
prompt_version TEXT NOT NULL,
adapter_version TEXT NOT NULL,
input_tokens INTEGER,
cached_input_tokens INTEGER,
output_tokens INTEGER,
charged_cost NUMERIC,
currency TEXT,
price_version TEXT,
queue_ms INTEGER,
first_token_ms INTEGER,
latency_ms INTEGER,
schema_valid BOOLEAN,
validator_pass BOOLEAN,
error_class TEXT,
escalated BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMP NOT NULL,
FOREIGN KEY (operation_id) REFERENCES router_operations(operation_id)
);
CREATE INDEX router_calls_by_operation
ON router_calls(operation_id, attempt_no);
Store unknown charges as NULL, never zero. A zero means “confirmed free”; NULL means “not reconciled.” Keep a separate estimated-cost column if estimates are operationally useful.
4. Turn traces into a privacy-reviewed dataset
Production traces can contain personal data, secrets, internal URLs, document contents, and tool results. Do not export raw telemetry into a training directory by default.
Define the permitted operation types and time range.
Exclude opted-out, regulated, or legally restricted records.
Replace stable user and tenant identifiers with scoped pseudonyms.
Remove credentials, authorization headers, cookies, signed URLs, and tool secrets.
Preserve task structure: redaction must not erase the signals the router needs.
Record the redaction-policy version and dataset manifest.
Restrict retention and access independently from production logs.
{
"dataset_id": "router-replay-YYYYMMDD",
"source_window": {
"from": "APPROVED_START",
"to": "APPROVED_END"
},
"operation_types": ["support_answer:v4"],
"redaction_policy": "router-redaction:v2",
"prompt_version": "support:v4",
"tool_contract_version": "support-read-tools:v3",
"feature_version": "features:v1",
"record_count": "COMPUTED_VALUE",
"content_hash": "COMPUTED_VALUE"
}
Do not use raw text, email addresses, customer identifiers, or full URLs as metric labels. High-cardinality telemetry is expensive, hard to aggregate, and prone to leaking sensitive information.
5. Sample operations that represent the application
A random sample of successful calls is insufficient. It excludes the very cases that determine whether routing is safe.
Stratify the sample by variables known before routing:
operation type and product surface;
input-length and context-length buckets;
language, attachment presence, and retrieval use;
number and type of available tools;
new conversation versus follow-up;
previous attempt state;
historical terminal outcome;
time period, including recent traffic.
Deliberately include failures, timeouts, repairs, human escalations, and user retries. Weight the final aggregate back to the real traffic distribution; otherwise, oversampling rare failures will distort the cost projection.
Keep an untouched, recent holdout period. Model behavior, prompts, retrieval indexes, and traffic change over time, so a purely random split can make the test look easier than deployment.
6. Run paired counterfactual replay
In an isolated benchmark, execute each eligible saved input against both routes. Pairing matters: comparing cheap calls from Monday with strong calls from Friday confounds route quality with traffic differences.
for example in evaluation_set:
cheap = run_route(
route="cheap",
input=example.normalized_input,
prompt_version=example.prompt_version,
tools=read_only_replay_tools
)
strong = run_route(
route="strong",
input=example.normalized_input,
prompt_version=example.prompt_version,
tools=read_only_replay_tools
)
store_pair(
example_id=example.id,
cheap=cheap,
strong=strong,
evaluator_version="acceptance:v3"
)
Replay tools must not change live systems. Replace writes with recorded fixtures or a deterministic simulator. If a historical read cannot be reconstructed, mark the example unevaluable rather than silently giving one route different evidence.
Control nondeterminism:
pin model, prompt, adapter, tool contract, and evaluator versions;
use the same retrieved evidence for both routes;
record sampling parameters and provider request identifiers;
repeat a subset to measure outcome variance;
keep transient infrastructure errors separate from answer-quality failures.
A strong-model response is not automatically ground truth. Both routes can fail. Apply the same acceptance process to both outputs.
7. Build the routing label
The simplest target is whether the cheap route passes the acceptance contract:
cheap_success =
schema_valid
AND policy_pass
AND required_facts_supported
AND tool_outcome_valid
AND NOT critical_violation
For routing, paired results support four useful classes:
Cheap
Strong
Interpretation
Training treatment
Pass
Pass
Cheap route is sufficient
Positive cheap-route example
Fail
Pass
Routing to cheap would be harmful
High-cost false-cheap example
Pass
Fail
Strong is not a universal oracle
Inspect evaluator and model variance
Fail
Fail
Neither route solves the operation
Usually abstain, repair, or redesign
Keep infrastructure failures distinct from semantic failures. A provider timeout does not prove that the model was incapable of answering. Depending on the goal, either exclude these records from capability training or model availability as a separate routing signal.
Account for corrective user requests
A technically accepted response may still be wrong if the user immediately restates the request, corrects a field, or asks for the promised action again. Link likely retries using a conservative rule based on actor, operation type, time window, and semantic or business identifiers.
user_retry =
same_scoped_actor
AND same_operation_type
AND within_configured_window
AND (
explicit_correction
OR same_business_object
OR reviewed_semantic_match
)
Do not declare every follow-up a failure. A user may ask a legitimate next question. Report confirmed retries separately from inferred retries and manually review a sample of both.
8. Split data without leakage
Near-duplicate conversations commonly appear in support workloads. If one copy enters training and another enters testing, the reported performance will be optimistic.
Group exact duplicates by normalized-input hash.
Group related turns by conversation or case identifier.
Where appropriate, group repeated templates or the same business object.
Assign complete groups to train, validation, or test.
Reserve the newest time window as an additional temporal holdout.
train:
time: earlier_window
groups: assigned_train_groups
validation:
time: earlier_window
groups: assigned_validation_groups
test:
time: later_untouched_window
groups: assigned_test_groups
Fit text vocabularies, encoders, scalers, and missing-value rules on training data only. Do not inspect test errors repeatedly while adjusting features; that turns the test set into another validation set.
9. Use only features available before the decision
The router may use properties known before the first model call:
operation type;
input character or token-length bucket;
conversation-turn count;
retrieved-document count and aggregate length;
attachment types;
language category;
tool availability and required action risk;
whether structured output is required;
previous attempt count;
coarse policy or product area;
privacy-safe text representation, if approved.
Do not use fields produced after routing, including cheap-model confidence, cheap output length, validator result, final latency, escalation status, or the user’s later correction. Those are future information and will not exist when the live router makes its choice.
{
"operation_type": "support_answer",
"input_token_bucket": "513-1024",
"conversation_turn_bucket": "4-7",
"retrieval_document_bucket": "1-3",
"has_attachment": false,
"requires_tool": true,
"tool_risk": "read_only",
"requires_structured_output": true,
"attempt_bucket": "first",
"language": "en"
}
Prefer stable, interpretable features first. A small router is easier to calibrate, debug, and run within a tight latency budget. Add embeddings or a compact language model only if simpler baselines leave a material, verified gap.
10. Train a calibrated probability model
A regularized logistic model or gradient-boosted tree is often an adequate first router. The desired output is an estimate of:
p_cheap_success = P(cheap route passes | pre-route features)
One possible configuration shape is:
router:
objective: predict_cheap_success
model_family: regularized_logistic_regression
class_weight:
cheap_success: 1
cheap_failure_strong_success: PRODUCT_RISK_WEIGHT
calibration:
method: isotonic_or_sigmoid
fit_on: validation_only
versions:
dataset: router-replay-YYYYMMDD
features: features:v1
evaluator: acceptance:v3
The risk weight is a product decision, not a value to copy from this article. Route errors that trigger an incorrect high-impact action should carry more weight than errors that merely require reformatting.
Measure discrimination and calibration separately. Ranking metrics show whether safer examples tend to receive higher scores. Calibration shows whether a score such as 0.90 corresponds to roughly that success frequency in the evaluated cohort. Threshold decisions depend on the latter.
Why accuracy is inadequate
If most operations are easy, a router that always chooses cheap can have high routing accuracy while still sending the small, costly failure class to the wrong model. Report at least:
false-cheap rate: cheap selected where cheap fails and strong passes;
false-strong rate: strong selected where cheap would pass;
critical false-cheap count and rate;
accepted-operation rate by route;
calibration by score bucket;
metrics by operation type, language, tool use, and length bucket.
11. Select the threshold using cost and risk
Do not select 0.5 merely because the router is a classifier. Evaluate candidate thresholds on validation data using complete operation economics.
if p_cheap_success >= threshold:
route = "cheap"
else:
route = "strong"
For each threshold, simulate:
expected_operation_cost =
router_inference_cost
+ initial_model_cost
+ automatic_retry_cost
+ repair_cost
+ escalation_cost
+ evaluation_cost
+ configured_failure_penalty
subject to:
accepted_operation_rate >= APPROVED_FLOOR
critical_violation_rate <= APPROVED_LIMIT
p95_operation_latency_ms <= APPROVED_BUDGET
Keep monetary spend and business-loss assumptions in separate columns. Provider charges can often be reconciled. Failure penalties are modeled assumptions and must be labeled as such.
Add an uncertainty zone
A practical policy can use two thresholds:
if safety_rule_requires_strong(features):
route = "strong"
elif score >= cheap_threshold:
route = "cheap"
elif score <= strong_threshold:
route = "strong"
else:
route = uncertainty_policy
The uncertainty policy may choose strong, apply a deterministic rule, request clarification, or use a separately tested intermediate route. Do not hide uncertainty by forcing every input onto cheap.
Apply hard safety rules first
Keep known high-risk categories out of the learned cheap route until they have dedicated evidence. Examples include write-capable tools, irreversible actions, authentication changes, money movement, legal commitments, and inputs outside the training distribution.
force_strong_when:
- action_risk in ["write", "irreversible"]
- operation_type not in trained_operation_types
- required_feature_missing
- router_unavailable
- prompt_version not in compatible_prompt_versions
- context_size outside_trained_range
12. Implement the runtime policy
def execute_operation(operation):
features = build_features(operation)
safety = evaluate_safety_rules(features)
if safety.force_strong:
decision = Decision(
route="strong",
score=None,
reason=safety.reason
)
else:
score = router.predict_proba(features)
route = "cheap" if score >= CHEAP_THRESHOLD else "strong"
decision = Decision(
route=route,
score=score,
reason="trained_policy"
)
persist_decision_before_call(
operation_id=operation.id,
router_version=ROUTER_VERSION,
feature_version=FEATURE_VERSION,
decision=decision
)
first = call_route(
operation=operation,
route=decision.route,
idempotency_key=operation.id + ":attempt:1"
)
checked = validate(first)
if checked.pass_all:
return finalize(operation, first, status="accepted")
if decision.route == "cheap" and checked.escalation_allowed:
second = call_route(
operation=operation,
route="strong",
idempotency_key=operation.id + ":attempt:2"
)
return finalize_escalation(operation, first, second)
return finalize(operation, first, status="rejected")
Persist the routing decision before the billable call. Otherwise, a worker crash can leave a charge with no recorded policy or score.
Use an idempotency key when the provider or your application boundary supports it. A retry caused by an uncertain network result must not accidentally create another tool action.
Separate these concepts:
call retry: the same route is called again after a transient error;
repair: another call corrects schema or formatting;
escalation: a cheap result fails validation and the strong route is invoked;
user retry: the user repeats or corrects the logical request.
All four consume resources, but they indicate different failures.
13. Start in shadow mode
In shadow mode, production continues using strong_only. The router computes a hypothetical decision but does not control the response.
{
"effective_route": "strong",
"effective_policy": "strong_only:v1",
"shadow_route": "cheap",
"shadow_router_version": "trace-router:v1",
"shadow_score": 0.93,
"shadow_reason": "trained_policy"
}
Shadow mode verifies feature availability, schema compatibility, score distributions, decision latency, version logging, and cohort assignment without exposing users to route errors.
It does not prove cheap-route quality unless the cheap route is also evaluated. You can obtain that evidence through offline paired replay or a separately approved shadow call. Extra shadow calls cost money and may process sensitive data, so include them in the experiment budget and privacy review.
Exit criteria
no missing required features in the eligible cohort;
router and feature versions present on every decision;
decision latency within its allocated budget;
score distribution consistent with offline observations or explained;
no critical false-cheap cases in the reviewed release sample;
cost projections include retries and escalations;
strong-only fallback tested successfully.
14. Run a controlled canary
Enable the router only for an eligible low-risk cohort. Assignment must be stable at the appropriate unit—usually actor, conversation, or operation—so a user is not moved unpredictably between policies.
routing_experiment:
eligibility:
operation_types: ["support_answer:v4"]
action_risk: ["read_only", "no_tool"]
supported_languages: ["en"]
assignment_unit: conversation_id
policies:
control: strong_only:v1
treatment: trace_router:v1
exclusions:
- write_capable_tool
- unresolved_previous_failure
- unsupported_prompt_version
- missing_required_feature
emergency_fallback: strong_only:v1
Choose traffic exposure and duration from your power analysis, operational risk, and rollback capability. A percentage copied from another system has no evidential value.
Freeze the router, thresholds, prompt, evaluator, and cohort definition during the measurement window unless safety requires a change. If anything changes, start a new report segment rather than merging incompatible versions.
15. Compute operation-level metrics
Total cost
confirmed_model_cost =
SUM(charged_cost WHERE charge_status = "confirmed")
complete_operation_cost =
confirmed_model_cost
+ confirmed_router_cost
+ confirmed_evaluator_cost
+ confirmed_tool_cost
cost_per_started_operation =
SUM(complete_operation_cost) / COUNT(started_operations)
cost_per_accepted_operation =
SUM(complete_operation_cost) / COUNT(accepted_operations)
Report total spend and cost per accepted operation. A cheap policy can lower call cost while increasing cost per accepted result through repairs, escalations, and failures.
Show the share of costs still estimated or unknown. Do not merge currencies without a dated, documented conversion rule.
Quality
accepted_operation_rate =
accepted_operations / eligible_started_operations
critical_violation_rate =
critical_violations / evaluated_operations
review_acceptance_rate =
reviewer_accepted / reviewer_evaluated
Include evaluator coverage. A high pass rate based on a small or selectively reviewed subset is not equivalent to broad evidence.
Latency
Measure end-to-end operation latency, not just model latency:
operation_latency_ms =
terminal_timestamp - accepted_request_timestamp
Report median, p90, p95, and p99 where sample size permits. Separate queue time, router time, first call, validation, retry, and escalation so the tail has an explanation.
Wrong routes
false_cheap =
selected_route = "cheap"
AND cheap_acceptance = false
AND strong_counterfactual_acceptance = true
false_strong =
selected_route = "strong"
AND cheap_counterfactual_acceptance = true
false_cheap_rate =
false_cheap / evaluated_cheap_decisions
false_strong_rate =
false_strong / evaluated_strong_decisions
Live traffic usually lacks both counterfactual outcomes. Estimate wrong-route rates from paired replay, approved exploration, or a reviewed sample. Label the source and coverage instead of presenting an incomplete denominator as universal truth.
Retries, repairs, and escalations
automatic_retry_rate =
operations_with_call_retry / started_operations
repair_rate =
operations_with_repair / started_operations
cheap_to_strong_escalation_rate =
cheap_started_then_strong / cheap_started_operations
user_retry_rate =
confirmed_or_inferred_user_retries / eligible_completed_operations
Also report average calls per operation and the cost of escalated operations. A high escalation rate can make routing look safe because the strong model repairs the output, while eliminating the expected savings.
16. Produce a report without invented results
Generate the table below from your warehouse or evaluation files. Leave missing values explicitly unavailable rather than filling them with assumptions.
Metric
Strong only
Cheap only
Rule router
Trace router
Eligible started operations
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Accepted-operation rate
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Critical violations
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Confirmed total cost
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Cost per accepted operation
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Unknown-cost share
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Median operation latency
COMPUTE
COMPUTE
COMPUTE
COMPUTE
p95 operation latency
COMPUTE
COMPUTE
COMPUTE
COMPUTE
False-cheap rate
N/A
COMPUTE
COMPUTE
COMPUTE
False-strong rate
COMPUTE
N/A
COMPUTE
COMPUTE
Automatic retry rate
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Cheap-to-strong escalation rate
N/A
COMPUTE
COMPUTE
COMPUTE
User retry rate
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Accompany the summary with these slices:
operation type;
input and context-length bucket;
language;
tool requirement and action risk;
new conversation versus follow-up;
router-score bucket;
prompt, router, feature, adapter, and evaluator version;
failure class.
Every report should state the evaluation window, cohort definition, exclusion count, sample size, evaluator coverage, price version, currency, unknown-charge count, and confidence interval or uncertainty method where appropriate.
Routing-error register
Field
Purpose
operation_id
Links the error to the full trace
router_score
Shows whether the decision was confident or near threshold
selected_route
Records the effective route
counterfactual_outcome
Supports false-cheap or false-strong classification
failure_class
Schema, policy, factual, tool, timeout, or infrastructure
feature_snapshot
Supports reproducible diagnosis
review_status
Separates detected from confirmed errors
corrective_action
Tracks rule, feature, threshold, prompt, or workflow changes
17. Verify the report with independent queries
Aggregate at operation level before comparing policies. Joining calls directly to evaluations can multiply rows and inflate cost.
WITH call_totals AS (
SELECT
operation_id,
COUNT(*) AS call_count,
SUM(CASE WHEN charged_cost IS NOT NULL
THEN charged_cost ELSE 0 END) AS confirmed_model_cost,
SUM(CASE WHEN charged_cost IS NULL
THEN 1 ELSE 0 END) AS unknown_charge_calls,
MAX(CASE WHEN escalated THEN 1 ELSE 0 END) AS had_escalation
FROM router_calls
GROUP BY operation_id
),
operation_facts AS (
SELECT
o.operation_id,
o.policy_version,
o.selected_route,
o.quality_pass,
o.critical_violation,
o.user_retry,
EXTRACT(EPOCH FROM (o.completed_at - o.started_at)) * 1000
AS operation_latency_ms,
COALESCE(c.call_count, 0) AS call_count,
COALESCE(c.confirmed_model_cost, 0) AS confirmed_model_cost,
COALESCE(c.unknown_charge_calls, 0) AS unknown_charge_calls,
COALESCE(c.had_escalation, 0) AS had_escalation
FROM router_operations o
LEFT JOIN call_totals c USING (operation_id)
WHERE o.cohort = 'eligible_support'
)
SELECT
policy_version,
COUNT(*) AS started_operations,
SUM(CASE WHEN quality_pass THEN 1 ELSE 0 END) AS accepted_operations,
SUM(CASE WHEN critical_violation THEN 1 ELSE 0 END)
AS critical_violations,
SUM(confirmed_model_cost) AS confirmed_model_cost,
SUM(confirmed_model_cost)
/ NULLIF(SUM(CASE WHEN quality_pass THEN 1 ELSE 0 END), 0)
AS model_cost_per_accepted_operation,
AVG(call_count) AS calls_per_operation,
AVG(had_escalation) AS escalation_rate,
AVG(CASE WHEN user_retry THEN 1.0 ELSE 0.0 END) AS user_retry_rate,
SUM(unknown_charge_calls) AS unknown_charge_calls
FROM operation_facts
GROUP BY policy_version
ORDER BY policy_version;
Check repeated calls independently:
SELECT
operation_id,
route,
COUNT(*) AS calls,
MIN(created_at) AS first_call,
MAX(created_at) AS last_call,
SUM(COALESCE(charged_cost, 0)) AS confirmed_cost
FROM router_calls
GROUP BY operation_id, route
HAVING COUNT(*) > 1
ORDER BY calls DESC;
Reconcile these aggregates with provider billing exports for the same model identifiers and time window. Explain differences caused by delayed charges, credits, cached-token accounting, failed calls, or timezone boundaries.
18. Verification before increasing traffic
Trace completeness: every billable call maps to one operation and one recorded decision.
Counterfactual comparability: paired routes used equivalent inputs, evidence, prompts, and tools.
Leakage prevention: no feature depends on the selected route’s output or later user behavior.
Split integrity: duplicate and related conversations do not cross dataset partitions.
Evaluator validity: deterministic and expert labels have documented coverage and versions.
Calibration: score buckets on the holdout set match observed cheap-route success closely enough for the decision policy.
Safety rules: unknown and high-risk operations default to strong.
Retry accounting: call retries, repairs, escalations, and user retries are counted separately.
Financial reconciliation: confirmed, estimated, and unknown costs are not mixed.
Latency accounting: the report uses end-to-end operation latency.
Version isolation: incompatible router, prompt, adapter, feature, or price versions are not aggregated.
Fallback: an immediate rollback to strong_only has been exercised.
Minimum automated tests
test_unknown_operation_routes_to_strong
test_missing_feature_routes_to_strong
test_router_timeout_routes_to_strong
test_write_tool_routes_to_strong
test_decision_is_persisted_before_model_call
test_retry_reuses_logical_operation_id
test_idempotency_key_changes_only_by_attempt
test_escalation_cost_is_included
test_unknown_charge_is_not_zero
test_future_features_are_rejected
test_duplicate_groups_do_not_cross_splits
test_report_aggregates_calls_before_joining
test_policy_versions_are_not_merged
test_strong_only_fallback_bypasses_router
Common failure cases
The strong model is treated as ground truth
The strong route can also hallucinate, misuse a tool, or violate policy. Evaluate both routes against the same contract and review disagreements.
Training uses future information
A feature such as validator outcome makes offline performance excellent and live routing impossible. Enforce a schema of fields available at decision time.
Training and test contain duplicate conversations
The router memorizes templates or cases. Group before splitting and maintain a recent temporal holdout.
The report counts calls instead of operations
A cheap attempt followed by a strong escalation appears as two successes or two observations. Aggregate the entire trace into one terminal operation first.
Escalation hides bad routing
Final quality remains acceptable because strong repairs cheap failures, but cost and tail latency increase. Report initial-route quality and escalation rate separately.
User follow-ups are all labeled as retries
Legitimate next questions become false failures. Use explicit correction signals, business identifiers, conservative matching, and reviewed samples.
Only averages are monitored
Average latency and quality can conceal a failing language, tool, or long-context segment. Inspect tails and cohort slices.
Current prices are applied to historical traffic
The apparent savings change when the price table changes. Store confirmed charges or a versioned price rule effective at call time.
The prompt changes without retraining or reevaluation
A prompt update can alter which operations the cheap model handles. Treat prompt compatibility as part of the router release.
Router failure defaults to cheap
This converts an infrastructure incident into a quality incident. For risk-sensitive operations, fail closed to the strong route.
Retries have no budget
A route that looks cheap per call can become expensive through repeated repairs. Cap attempts per operation and stop unchanged retries on permanent errors.
Limitations
A trace-trained router learns the distribution represented in its dataset. It may not generalize to a new product surface, language, prompt, retrieval index, tool, or model revision.
Counterfactual replay is imperfect when external state cannot be reconstructed. Live user reactions are also unavailable for hypothetical responses, so behavioral quality requires careful sampling or controlled deployment.
Evaluator errors become label errors. Deterministic checks are narrow, expert review is costly, and model-based judging requires its own validation. No single evaluator should silently define overall quality.
Provider behavior and prices can change. Recalculate economics with versioned charges and reevaluate route capability after material model changes.
A two-route policy does not solve every workload. Some operations need retrieval repair, clarification, a deterministic program, a human reviewer, or refusal rather than a stronger model.
Finally, routing itself adds operational complexity: another model or ruleset, feature computation, monitoring, version compatibility, and failure handling. If traffic is small or the models have similar economics, that complexity may cost more than it saves.
Operating the router after launch
The router becomes part of the production workflow. Monitor it as a versioned decision service, not as a static experiment.
track input-feature and score-distribution drift;
compare accepted-operation, escalation, and retry rates by route;
review false-cheap cases and critical failures immediately;
replay a stable regression set after model, prompt, tool, or evaluator changes;
sample live operations for expert review;
reconcile charges on a regular schedule;
expire router versions that are incompatible with current prompts or features;
retain a tested strong-only fallback.
Retrain on a declared trigger—material drift, a changed model, a changed prompt, enough newly reviewed errors, or an agreed schedule. Automatic retraining without gated evaluation can turn temporary anomalies into production policy.
Final checklist
The routing unit is a completed business operation.
Acceptance criteria exist before model comparison.
Cheap and strong routes are evaluated on paired inputs.
Both routes use equivalent evidence and permissions.
Training data is privacy-reviewed and versioned.
Related traces remain in one dataset partition.
Features exist before the route decision.
The router outputs a calibrated score.
The threshold reflects cost and asymmetric risk.
Hard safety rules override learned routing.
Unknown inputs and router failures default safely.
Shadow mode validates integration before control.
The canary has stable assignment and explicit eligibility.
Cost includes retries, repairs, evaluations, and escalations.
Quality includes deterministic, expert, and behavioral evidence.
Latency is measured end to end and includes tail percentiles.
False-cheap and false-strong routes are reported separately.
Automatic and user retries have distinct definitions.
Unknown financial data remains unknown.
The report records every relevant version and denominator.
Rollback to strong_only is tested.
The router is ready only when it reduces verified cost per accepted operation without crossing the approved quality and latency limits. A lower model bill alone is not success if difficult answers fail, users repeat themselves, or cheap attempts routinely escalate to the strong route.
Continue the work
Find more reproducible implementation material in the Agent Lab Journal guides, and use the glossary for definitions of agents, traces, prompts, context, tokens, tool calling, benchmarks, observability, idempotency, rollback, and workflows.
Agent Lab Journal
Privacy
Top comments (0)