Fintech code review systems should allocate inference spend per tenant before they try to reduce it. Token totals alone cannot explain a bill: retries, model routing, and optional operations can all multiply work. The practical control is an append-only usage ledger that records the decision, attempt, and reason for each charge.
The factors are easy to miss. An LLM API call can increase spend through repeated delivery, a more capable model than the review needs, or an unnecessary feature enabled by default.
Start with accounting.
That is an accounting problem before it is a prompt-tuning problem. I use the exactly-once mindset of payment and ledger systems: a request may be delivered twice, but it must be posted once; a failed attempt remains visible; and a tenant total is reconstructable from immutable records.
Which factors increase LLM API spend beyond token cost?
Start with a cost event, not a dashboard total. For every review job, record tenant, job identifier, model class, token counts when available, attempt number, feature flags, outcome, and timestamp. Add a provider-neutral request key. This answers the hard question later: did the tenant pay for one useful review, or for three attempts and an unused explanation pass?
A timeout may trigger a retry even though the upstream service completed. A broad model may be selected for every file although only a fraction needs its extra capacity. A workflow may request summaries, remediation suggestions, or embeddings by default, turning one required finding into several operations. Unchanged files, generated artifacts, and duplicate snippets can expand context too. The trade-off is real: aggressive retries may improve completion rate, while conservative retries preserve a tenant budget but leave more jobs awaiting reconciliation; a smaller model may lower spend, while a larger one may reduce manual review for security-sensitive changes. The policy should expose that choice instead of pretending there is one universally correct threshold.
A retry record is an audit trail, not noise. Retain the minimum metadata needed to reconcile usage while excluding source code and secrets from the ledger.
How should a tenant ledger decide what to charge?
Use a deterministic charge key: tenant ID, review job ID, attempt number, operation, and provider request ID when one exists. The attempt number is intentional. Deduplicating every event by job ID hides repeated upstream work; deduplicating by request ID alone fails when a timeout leaves the client without a response. Mark an attempt submitted, completed, or unknown, then reconcile unknowns through usage records or replay. Never silently convert uncertainty to zero.
Here is a compact Go model for the accounting boundary. It does not calculate a provider price; that policy belongs behind an explicit rate table with an effective timestamp because rates and model catalogs change.
package spend
import (
"crypto/sha256"
"encoding/hex"
"fmt"
)
type UsageEvent struct {
TenantID string
JobID string
Attempt int
Operation string
ModelClass string
InputTokens int
OutputTokens int
Status string
}
func (e UsageEvent) ChargeKey() string {
raw := fmt.Sprintf("%s|%s|%d|%s", e.TenantID, e.JobID, e.Attempt, e.Operation)
sum := sha256.Sum256([]byte(raw))
return hex.EncodeToString(sum[:])
}
ChargeKey makes replay idempotent for the same attempt, while Attempt preserves a genuinely repeated call. The sink needs a unique constraint on that key and a durable handoff so a process crash cannot leave the review result committed while its usage event disappears. Exactly-once delivery is rarely a network property; exactly-once posting is a ledger property.
Which control belongs at each failure boundary?
Put a retry budget beside the request, not in a generic HTTP client. Permit at most two retries for a transient failure, with a new attempt record for each retry and a stable job ID for the review. A timeout is not proof that no work occurred. If the response is ambiguous, reconcile it before an automatic repeat, or label the repeat as a new billable attempt.
Model selection is another budget boundary. Choose a model class from review evidence such as changed-file count, language coverage, security-sensitive paths, and the tenant quality tier. Log the policy result before dispatch. The useful metric is the distribution of attempts and model classes by tenant, job type, and outcome, not average tokens.
Make summary generation, remediation text, and secondary analysis explicit operations with independent event types. A reviewer can disable one operation for a tenant without changing the accounting semantics of required findings. Stop scheduling the optional operation, leave existing events intact, and compare review quality with the prior window.
A useful review cadence is a 30-day reconciliation window with a daily anomaly check. Alert when a tenant's retry ratio, optional-operation ratio, or unknown count moves above its own baseline. A new tenant has no meaningful baseline and should be marked accordingly.
What should the implementation prove before rollout?
Test invariants with failure injection. Send one completion event twice and require one ledger row. Send a timeout followed by a successful retry and require two attempt rows under one job. Replay an event after deployment and require the same charge key. Compare the ledger aggregate with a raw usage export for a fixed window, including unknown records. These tests catch the expensive bug class: the review looks correct while the financial record is incomplete.
Roll out in three stages: observe and write events without changing routing; enforce tenant budgets and expose optional operations; then require an explanation for every model override and retry beyond policy. Keep raw event, derived charge, and reconciliation result separate so one can be corrected without rewriting history.
If a control cannot identify tenant, operation, attempt, and outcome, it cannot explain spend. If it cannot be replayed idempotently, it cannot support an audit. If it hides optional work inside the required review request, it cannot identify which feature to remove.
Top comments (0)