Short answer: use a pre-call estimate to decide whether an agent may start, then use a post-hoc usage report as the billable record. Neither one is the budget. The budget is the reconciliation between them, keyed to an event that survives retries and an outage.
I run a small developer-tools SaaS, so every infrastructure choice competes with a feature I could ship this week. The question is not which number looks nicer in a dashboard. It is whether I can explain one customer charge when the model provider reports usage late, a webhook is retried, or my ingestion service is down.
Should Python AI agent budgets use pre-call estimates or post-hoc usage reports?
Treat the two measurements as different jobs. A pre-call estimate is a guardrail: it reserves a worst-case amount before work begins. A post-hoc report is evidence: it records what the provider says happened.
The failure mode is using either value as if it were both. A prompt can be larger after tool output is added. A provider can round tokens differently from your tokenizer. A report can arrive after the request that triggered it. If I block on the estimate forever, useful work gets rejected. If I trust the report alone, a runaway loop can spend money before accounting catches up.
For an agent that processes platform events, I attach an immutable run_id and event_id at ingress. The estimate reserves against run_id; every provider response and later usage record points back to the same run. That makes a retry an accounting event instead of a second customer charge.
Here is the small state model I use. It is deliberately boring.
| Record | Created | Purpose | Can change? |
|---|---|---|---|
| Reservation | Before the model call | Decide if the run may proceed | Released or adjusted |
| Usage fact | After provider data arrives | Calculate billable consumption | Append corrections |
| Reconciliation | After both exist | Explain the difference | Re-run safely |
The catch is timing. A reservation is not a forecast of the invoice, and a usage fact is not permission to spend. Keeping those meanings separate is more valuable than squeezing another decimal place out of a token estimate.
It is a ledger problem.
The build log: an outage-safe attribution path
My first sketch put the budget check inside the webhook handler. That looked tidy until the handler retried an event while the model call was still running. Two reservations appeared for one customer action. The totals were technically consistent, but attribution was wrong. That is the kind of bug that costs a Friday afternoon and a trust conversation.
I moved the decision to an idempotent command path. The event is accepted, assigned an ID, and placed in a durable queue. A worker creates one reservation with a deterministic key. The worker can then call the model, or stop when the reservation would cross the tenant limit.
type Budget = {
limitCents: number;
reservedCents: number;
usedCents: number;
};
type Estimate = {
inputTokens: number;
outputTokens: number;
unitInputMicros: number;
unitOutputMicros: number;
};
function estimateCents(e: Estimate): number {
const micros =
e.inputTokens * e.unitInputMicros +
e.outputTokens * e.unitOutputMicros;
return Math.ceil(micros / 10_000);
}
function canStart(budget: Budget, estimateCentsValue: number): boolean {
return budget.usedCents + budget.reservedCents + estimateCentsValue <= budget.limitCents;
}
The estimate intentionally rounds up. It includes a small policy buffer for tool output and provider rounding, but it does not pretend to know the final price. I store the input assumptions with the reservation: tokenizer version, rate-card revision, and the maximum output I allowed. Without those fields, a later reconciliation cannot answer why two estimates differed.
After the call, I append a usage fact rather than overwriting the reservation. The fact includes provider request ID, actual input and output units, and the same run_id. A unique constraint on (tenant_id, run_id, provider_request_id) protects the ledger when a report is delivered twice.
type UsageFact = {
tenantId: string;
runId: string;
providerRequestId: string;
inputTokens: number;
outputTokens: number;
reportedAt: string;
};
function reconcile(estimate: number, actual: number) {
return {
releaseCents: Math.max(estimate - actual, 0),
extraCents: Math.max(actual - estimate, 0),
};
}
If the usage report is late, the reservation remains visible and the tenant's available balance stays conservative. If the report never arrives, an operator can identify an aging reservation without inventing a usage number. Your mileage may vary on the buffer size; measure the gap by model and workload before setting a global percentage.
What breaks when the backend is offline?
An outage changes delivery, not ownership. The platform event still belongs to a tenant, and its event_id must remain the same when it is replayed. I persist the envelope before attempting any external call, then acknowledge only after the envelope is durable. A replay sees the same idempotency key and does not create a second reservation.
The queue carries a small state machine: accepted, reserved, sent, reported, and reconciled. Transitions are append-only facts. A timeout does not mean reported; it means the run needs a poll, provider export, or manual review. That distinction prevents an outage from becoming free usage or a double charge.
The ugly case is a worker crash in the narrow gap after a provider accepts a request and before the worker records sent. On restart, the queue sees an accepted event with no terminal state. The worker retries using the same provider idempotency key, then records whichever response the provider returns. If the provider does not offer that key, I mark the run as ambiguous and hold its reservation for review rather than silently issuing a second call. During the outage, the customer sees a pending run and a conservative balance; after recovery, the usage fact or an explicit cancellation closes it. This is slower than optimistic accounting, but the audit trail explains every cent and the retry behavior is testable. A nightly job can flag reservations older than the published correction window without making up usage data.
I also keep customer-facing balance and internal provider cost separate. The first answers “may this tenant start another run?” The second answers “what did this run cost us?” They converge during reconciliation, but they should not share a mutable counter.
There is a practical limitation here: a pre-call gate cannot guarantee a hard ceiling when a provider bills asynchronous tools or charges outside the request response. For those workloads, cap concurrency and tool steps as well, and make the billing policy explicit. Stick with a simple reservation-only flow when the provider emits complete usage synchronously and your retry window is short.
The trade-offs I would make at scale
| Choice | Helps with | Cost or limitation |
|---|---|---|
| Reserve the maximum output | Stops most runaway calls | Rejects legitimate long answers |
| Reserve a percentile estimate | Improves admission rate | Tail usage can exceed the reserve |
| Wait for post-hoc data | Accurate provider accounting | No protection during a delayed report |
| Append corrections | Auditable history | Queries need a current-balance projection |
| One ledger per tenant | Clear customer attribution | Cross-tenant campaigns need a second aggregation |
At higher volume, I would separate the write path from projections. The ledger stays append-only in durable storage; a stream builds tenant balances, aging-reservation alerts, and daily cost reports. Rebuilding a projection from facts is slower than updating a counter, but it gives me a way to repair a bad deploy without guessing which events were lost.
I would also test the accounting path with generated sequences: duplicate delivery, out-of-order usage, a worker crash after the provider accepts a request, and a report that arrives 48 hours late. The assertions are about invariants, not a happy-path total: one event maps to one billable run, reservations never make available balance positive by accident, and every correction names its predecessor.
Short version: spend a little engineering time on keys and state transitions. It buys back hours that would otherwise go into reconciling spreadsheets.
A weekly operating checklist
Every week I sample runs across tenants and compare three numbers: reserved cents, reported provider cost, and charged customer units. I look for drift by model, not just a single aggregate. A stable total can hide one tenant being overcharged while another is undercharged.
Secrets deserve their own boundary. Store provider credentials in a managed secret system, restrict access by workload, rotate them, and keep them out of event payloads and logs. The OWASP Secrets Management Cheat Sheet is a useful baseline for ownership, rotation, and audit controls.
Do not promise a perfect estimate. Publish the rounding rule, the correction window, and what happens when a report is missing. That policy is part of the product, especially when customers use your platform to set their own AI agent budgets.
References
- https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
- https://www.rfc-editor.org/rfc/rfc9110
- https://opentelemetry.io/docs/specs/otel/logs/
Top comments (0)