Production agents exhibit wildly different performance across domains. A customer support agent might ace billing questions but fail on technical troubleshooting. A trading agent might handle equity orders cleanly but struggle with options spreads. You need disaggregated evaluation, but labeling every interaction is expensive.
The standard approach is to sample and extrapolate. The problem is that direct estimators (including prediction-powered inference) use only a domain's own labels. When you have few labeled examples in a domain, your confidence intervals blow out. You ship blind or you pay for more labels.
A new ArXiv paper (2609.20758v1) proposes prediction-powered smoothing (PP-S), a Bayesian model that borrows strength across domains to tighten estimates where labels are sparse. The key insight: treat your evaluation set as a finite population and use small-area estimation techniques to share information across related domains.
The Evaluation Sampling Problem
When you deploy an agent, you face a labeling budget constraint:
- Exhaustive testing means human grading every interaction (conversation, trade, query). Infeasible at scale.
- Stratified sampling means labeling a subset per domain. Works if domains are balanced and you have enough samples everywhere.
- Reality means some domains are rare (edge cases, niche asset classes, low-traffic conversation types) but still matter for compliance, risk, or user trust.
Direct estimators fail in sparse domains. If you label 10 out of 1,000 interactions in a rare domain, your point estimate might be decent but your interval estimate is useless. You cannot confidently say whether the agent is safe to deploy for that domain.
How Prediction-Powered Smoothing Works
PP-S combines three components:
Prediction-powered inference (PPI) as the base layer. You run a cheap predictor (a smaller model, a heuristic, a rule-based grader) on all interactions, then label a sample. PPI uses the predictor's output to reduce variance in your estimate.
Bayesian smoothing across domains. Instead of treating each domain's estimate independently, you fit a hierarchical model that shares information. Domains with few labels borrow strength from domains with many labels.
Taxonomic extension (PP-TS) for nested domains. If your domains form a hierarchy (e.g., "customer support" contains "billing" and "technical"), you can borrow strength along the taxonomy tree.
The workflow:
# Pseudocode for PP-S estimation
def prediction_powered_smoothing(domains, predictor, labels, budget):
# Step 1: Run cheap predictor on all interactions
predictions = {d: predictor(d.interactions) for d in domains}
# Step 2: Allocate labeling budget across domains
sample_allocation = allocate_budget(domains, budget)
# Step 3: Collect labels for sampled interactions
labels = {d: label_sample(d, sample_allocation[d]) for d in domains}
# Step 4: Compute PPI estimates per domain
ppi_estimates = {d: ppi_estimate(predictions[d], labels[d]) for d in domains}
# Step 5: Fit Bayesian hierarchical model
smoothed_estimates = bayesian_smooth(ppi_estimates, domain_structure)
return smoothed_estimates
The Bayesian model treats each domain's PPI estimate as a noisy observation of the true domain mean. The prior encodes the assumption that related domains have similar performance. The posterior tightens intervals in sparse domains by pulling them toward the global or parent-domain mean.
Cross-Validation for Estimator Selection
You now have multiple estimators: direct PPI per domain, PP-S, PP-TS. Which one should you trust?
The paper derives a design-based cross-validation score that is approximately unbiased. The key: you split your labeled sample into training and validation folds, fit each estimator on the training fold, and score it on the validation fold. The score estimates the mean squared error of the estimator on the full population.
This matters because:
- You do not need a separate validation sample (which would consume part of your labeling budget).
- The score is calibrated to the finite population you care about (your evaluation set), not an infinite superpopulation.
- You can compare direct and smoothed estimators on equal footing.
The workflow becomes:
- Allocate labeling budget.
- Collect labels.
- Fit multiple estimators (direct PPI, PP-S, PP-TS).
- Run cross-validation to select the best estimator per domain.
- Report the selected estimator's point and interval estimates.
Architecture for Production Evaluation
Here is how this fits into a production agent evaluation pipeline:
| Component | Role | Implementation |
|---|---|---|
| Predictor | Cheap grader for all interactions | Smaller LLM, rule-based heuristic, or fine-tuned classifier |
| Sampler | Selects interactions for human labeling | Stratified sampling with domain-aware allocation |
| Labeling queue | Routes sampled interactions to human graders | Task queue (Celery, SQS) with domain metadata |
| Estimator | Computes point and interval estimates per domain | PPI, PP-S, or PP-TS depending on CV score |
| Validator | Runs cross-validation to select estimator | Offline batch job, rerun when new labels arrive |
| Dashboard | Displays disaggregated performance metrics | Confidence intervals, coverage diagnostics, domain hierarchy |
The predictor runs in real time or near-real time. The sampler decides which interactions to label based on domain rarity and current label counts. The labeling queue routes to human graders. The estimator and validator run offline, typically daily or weekly, to update performance metrics.
Failure Modes and Boundaries
Domain shift. If the predictor is biased differently across domains, PPI's variance reduction breaks down. You need to validate that the predictor's errors are uncorrelated with the true labels within each domain.
Sparse taxonomy. If your domain hierarchy is shallow or poorly structured, PP-TS offers little benefit over PP-S. You need at least two levels of nesting and enough domains per level to share information.
Budget allocation. If you allocate too few labels to rare domains, even smoothing cannot save you. The paper does not prescribe an allocation strategy. You need to balance exploration (labeling rare domains to detect failures) and exploitation (labeling common domains to tighten global estimates).
Coverage diagnostics. The paper reports near-nominal coverage (e.g., 95% intervals contain the true mean 94-96% of the time) in experiments. In production, you should track empirical coverage by holding out a validation set or using sequential testing. If coverage degrades, your model assumptions (domain similarity, predictor quality) may be violated.
Labeling latency. Human grading introduces delay. If your agent's behavior changes faster than you can collect labels, your estimates lag reality. You need to decide whether to use stale estimates or to increase labeling throughput.
When to Use This
Use prediction-powered smoothing when:
- You have multiple domains with unbalanced interaction counts.
- Labeling is expensive (human grading, expert review, compliance checks).
- You need confidence intervals, not just point estimates, to make ship/no-ship decisions.
- Your domains are related (similar tasks, shared failure modes, hierarchical structure).
Avoid it when:
- You can afford exhaustive testing (small evaluation sets, cheap labeling).
- Domains are independent or adversarial (no shared structure to exploit).
- Your predictor is unreliable or biased in ways you cannot diagnose.
- You need real-time evaluation (the Bayesian model fitting and cross-validation add latency).
Comparison to Alternatives
| Approach | Label efficiency | Interval quality | Complexity | Best for |
|---|---|---|---|---|
| Exhaustive testing | Low (label everything) | Perfect | Low | Small eval sets |
| Stratified sampling | Medium | Poor in sparse domains | Low | Balanced domains |
| Direct PPI | High | Poor in sparse domains | Medium | Single-domain eval |
| PP-S | High | Good across domains | High | Multi-domain with shared structure |
| PP-TS | High | Best for hierarchical domains | Highest | Nested taxonomies |
Implementation Notes
The paper uses Stan for Bayesian inference. You could also use PyMC, NumPyro, or TensorFlow Probability. The key is to specify a hierarchical prior that encodes domain similarity.
For the predictor, start simple. A fine-tuned BERT classifier or a GPT-3.5 grader often suffices. The predictor does not need to be perfect; it just needs to be correlated with the true labels and cheap to run.
For cross-validation, use k-fold with k=5 or k=10. The paper's design-based score is more efficient than traditional CV because it accounts for the finite population structure.
For labeling allocation, consider Neyman allocation (allocate proportional to domain variance) or power allocation (allocate to maximize the probability of detecting failures in rare domains).
Technical Verdict
Use prediction-powered smoothing when you need disaggregated agent evaluation with tight confidence intervals and cannot afford to label everything. The technique is most valuable when domains are sparse, related, and hierarchical. It requires a reliable predictor, a structured domain taxonomy, and the infrastructure to run Bayesian inference offline.
Avoid it if your domains are independent, your predictor is unreliable, or you need real-time evaluation. In those cases, stick with direct PPI or increase your labeling budget.
The paper fills a gap in production agent evaluation: how to decide what to test when you cannot test everything. The math is solid, the experiments are convincing, and the workflow is practical. If you are shipping agents with domain-specific performance requirements, this is the eval plumbing you need.
Top comments (0)