KillianBerg5391

Posted on Sep 19

Marketplace Agent Budgets Need Pre-Call Cost Estimates and Post-Hoc Reports

#python #ai #architecture

Short answer: put a conservative cost estimate in the admission path, then reconcile it against reported usage after every model call. The estimate protects a marketplace spend ceiling while events are arriving. The report corrects the ledger and improves the next estimate. Neither can replace the other. If the estimate is unavailable, stale, or over budget, route the event to a bounded fallback or defer it instead of pretending a later report can undo spend.

This matters most after an outage. A recovered marketplace may release a backlog of listing updates, buyer messages, and fraud-review events all at once. Letting every event start an agent can consume the remaining budget before a usage dashboard catches up; rejecting everything preserves money but refuses useful traffic. The practical decision is an admission policy: reserve estimated spend, preserve headroom for high-value work, and settle the reservation from actual usage.

Should agent budgets use a pre-call cost estimate or post-hoc usage report?

Treat the pre-call number as a reservation, not an invoice. An event enters the backend with a stable event ID, workload class, prompt revision, model class, and bounded input. The gate calculates an upper estimate from locally countable input plus configured output and tool-call limits. It atomically reserves that amount against the relevant budget window. Only then does the worker call the model.

The estimate will be imperfect because the response does not exist yet. Agent loops make the uncertainty larger: a tool result can add context, and another turn can follow. The useful question is not "Can this be exact?" It is "Is this conservative enough to prevent overspend without refusing too much traffic?" That is an eval problem. Replay representative marketplace events, compare reserved and settled amounts, and track false refusals alongside budget overruns.

Exact is impossible.

Post-call usage has a different job. When the provider response includes input and output usage, the worker records it with the event ID and prompt revision, converts it under the same versioned rate card used by the estimator, and settles the reservation. A missing report must not become zero cost. Keep the reservation open or settle to a documented conservative amount, then reconcile asynchronously when authoritative data arrives.

Reports arrive too late.

The data flow is small enough to reason about: queue to admission ledger, admission ledger to agent worker, worker to model and tools, then usage back to the ledger. Metrics and traces observe the flow, but they do not own the balance. That separation is important during recovery because telemetry may be delayed while admission still has to be correct.

A runnable reservation and settlement core

The following example uses integer micro-units for money, a lock for atomic reservations, and no network dependency. It models the core rule that production storage must enforce with a transaction or compare-and-swap operation. The numbers are synthetic policy inputs, not market prices.

from __future__ import annotations

from dataclasses import dataclass
from threading import Lock
from typing import Literal

MICRO_UNITS = 1_000_000


@dataclass(frozen=True)
class RateCard:
    input_per_million: int
    output_per_million: int
    version: str


@dataclass(frozen=True)
class RequestPlan:
    event_id: str
    workload: Literal["buyer_message", "listing_enrichment", "fraud_review"]
    estimated_input_tokens: int
    max_output_tokens: int
    max_agent_turns: int


@dataclass(frozen=True)
class Usage:
    input_tokens: int
    output_tokens: int


def token_cost(tokens: int, rate_per_million: int) -> int:
    if tokens < 0 or rate_per_million < 0:
        raise ValueError("tokens and rates must be non-negative")
    return (tokens * rate_per_million + 999_999) // 1_000_000


def estimate(plan: RequestPlan, rates: RateCard) -> int:
    # Reserve every allowed turn; evals should tune these bounds by workload.
    per_turn = token_cost(plan.estimated_input_tokens, rates.input_per_million)
    per_turn += token_cost(plan.max_output_tokens, rates.output_per_million)
    return per_turn * plan.max_agent_turns


def actual_cost(usage: Usage, rates: RateCard) -> int:
    return token_cost(usage.input_tokens, rates.input_per_million) + token_cost(
        usage.output_tokens, rates.output_per_million
    )


class BudgetLedger:
    def __init__(self, ceiling: int) -> None:
        self.ceiling = ceiling
        self.settled = 0
        self.reservations: dict[str, int] = {}
        self._lock = Lock()

    def try_reserve(self, event_id: str, amount: int) -> bool:
        with self._lock:
            if event_id in self.reservations:
                return True  # Idempotent redelivery of an admitted event.
            committed = self.settled + sum(self.reservations.values())
            if committed + amount > self.ceiling:
                return False
            self.reservations[event_id] = amount
            return True

    def settle(self, event_id: str, amount: int) -> None:
        with self._lock:
            if event_id not in self.reservations:
                raise KeyError("event has no reservation")
            del self.reservations[event_id]
            self.settled += amount


rates = RateCard(input_per_million=200_000, output_per_million=800_000, version="eval-3")
plan = RequestPlan(
    event_id="evt_01",
    workload="buyer_message",
    estimated_input_tokens=1_200,
    max_output_tokens=300,
    max_agent_turns=2,
)
ledger = BudgetLedger(ceiling=50_000)
reserved = estimate(plan, rates)

if ledger.try_reserve(plan.event_id, reserved):
    # Replace this fixture with the model response's reported usage.
    reported = Usage(input_tokens=1_080, output_tokens=214)
    ledger.settle(plan.event_id, actual_cost(reported, rates))
else:
    # Defer, use a non-model path, or reject according to workload policy.
    pass

There is a deliberate rough edge in this tiny example: settlement can take the ledger above its ceiling if actual cost exceeds the reservation. Production policy must decide what happens next. A sensible response is to stop new low-priority admissions and raise an alert, not to rewrite the actual amount. Hiding the variance destroys the signal needed to fix the estimator.

The in-memory lock only protects one process. A deployed gate needs durable uniqueness on the event ID, an atomic conditional reservation, and explicit states such as reserved, settled, expired, and disputed. Queue delivery can repeat. Ten workers can read the same remaining balance before any one of them writes its reservation. Each local decision looks valid, yet their combined commitment crosses the ceiling. A retry can then reserve again if event identity is lost between the queue and ledger. The estimator's arithmetic is irrelevant in both failures because transaction boundaries, not prediction error, caused the overspend. Test the storage primitive under contention and force a worker to stop after reservation, after the external call, and before settlement. Idempotency is mandatory.

Choosing refused traffic instead of accidental spend

A single first-in, first-out budget treats a fraud review and a cosmetic listing rewrite as equal. They are not equal to the marketplace. Partition the ceiling into policy buckets, or reserve headroom for the workload whose refusal has the highest business cost. Keep the policy understandable enough to test.

Some traffic should wait.

During normal traffic, a listing enrichment might allow two turns and a short output. During backlog recovery, the same class can move to one turn, a smaller context, or a deterministic template. Buyer messages might be deferred in arrival order. Fraud review may retain its allocation. These are product decisions expressed as code, rather than a provider SDK deciding implicitly through retries.

The spend ceiling and refusal rate should appear on the same dashboard. Measure admitted events, deferred events, hard refusals, open reservations, settlement lag, estimate-to-actual ratio, and cost by prompt revision. A low spend graph can mean the gate worked, or it can mean customers were silently refused. You need both sides to tell which.

Protect the control plane too. Rate cards and credentials belong in a secrets-management process with least-privilege access, rotation, auditing, and a defined lifecycle. Do not put credentials in event payloads, notebook cells, trace attributes, or prompt logs. The OWASP guidance in Further reading gives a broader treatment of those controls.

Where estimates and reports fail differently

Control	Available	Best use	Main failure mode	Safe response
Pre-call estimate	Before admission	Reserve budget and choose a degraded path	Underestimates loops, tool context, or output	Apply hard bounds and tune from evals
Post-call usage report	After a response	Settle actual usage and calibrate forecasts	Arrives late, is absent, or cannot prevent the completed spend	Retain a conservative reservation and reconcile
Provider invoice or aggregate export	Later	Financial reconciliation	Too delayed and coarse for per-event admission	Compare aggregates; investigate drift

Token counts alone are not a universal cost interface. Some workloads include tools, cached input, media, or other billable dimensions. Preserve the raw reported fields and the rate-card version instead of flattening everything into one unexplained number. Then a rate change or mapping correction can be audited without inventing history.

This design has a real limitation: conservative reservations can refuse useful events even when their eventual responses would have been cheap. It is not a good fit for workloads where requests cannot be classified, output cannot be bounded, and refusal is more damaging than exceeding a soft internal target. In that case, use post-hoc reporting for visibility and enforce a coarser upstream quota; do not present the result as a hard per-event spend ceiling. The trade-off is explicit: stronger protection requires more refused or degraded traffic when uncertainty grows.

Observability requires similar discipline. Propagate a trace identifier across queue consumption, admission, model calls, and tools using a standard context format, but keep budget correctness in the transactional ledger. OpenTelemetry metrics define counters and histograms that fit admitted-event counts, settlement latency, and estimate error distributions. Avoid event IDs, user IDs, and prompt text as metric labels; unbounded label sets raise storage cost and make dashboards hard to operate.

There is also a notebook-to-production trap. A notebook usually runs one event at a time and can calculate spend after each cell. Production workers race. Ten workers that each read "40 units remaining" can all admit a 10-unit request unless reservation is atomic. The estimator may be excellent while the budget control is still wrong. Test concurrency separately from prediction accuracy.

Operational acceptance before backlog release

Before deployment, replay a fixed evaluation set by workload and prompt revision. Record estimate error, refusal decisions, and the actual path selected; fail the release if a prompt change pushes a critical class outside its agreed reservation envelope. Include long tool outputs, empty usage fields, timeouts, duplicated queue deliveries, and a settlement that exceeds its reservation. Synthetic happy paths will miss the expensive branches.

Then load-test the ledger with concurrent reservations at the exact ceiling. Verify that committed spend never exceeds the policy limit, that a repeated event ID cannot reserve twice, and that settlement can be retried without double charging. Inject delayed usage and worker termination between the model response and settlement. An expiry job should flag abandoned reservations for investigation rather than silently freeing them while a call may still be running.

Roll out with a small traffic slice and compare shadow decisions before enabling refusal. Watch estimate-to-actual distributions by workload, not only the average; a good mean can hide a dangerous tail. Keep the rate card versioned, review access to its update path, and alert on settlement lag or unexplained aggregate drift.

The final runbook should be short enough to use during an outage: confirm the active ceiling and rate-card version, inspect open reservations and settlement lag, select the approved degradation level for each workload, release the backlog gradually, and stop the release when refusal or variance crosses its reviewed threshold. This closes the loop. The pre-call estimate makes the immediate admission decision; the post-call report makes that decision better next time and keeps the ledger honest.

DEV Community