Instrument the bill, not just the model

Most AI observability stacks answer "is it up." Very few answer "what did it do, and can you prove it."

That distinction stopped being academic for me when an agent I was responsible for silently approved $2.4M in non-covered procedures over six weeks. Uptime was green throughout. A CFO found it in a month-end review. The audit pipeline never did.

This post is about the minimum record you need so that never happens, and why the same record is what lets you bill on outcomes.

What is actually missing?

Standard telemetry captures request, latency, status, token count. That tells you the system ran. It does not tell you what decision was made, on what evidence, under which version of anything.

The gap looks like this in practice. A customer disputes an output. You need to answer four questions, and you have ninety minutes before someone senior asks for an update.

1. Which action produced this?
2. Which model and which prompt or policy version was live?
3. What inputs and retrieved context were used?
4. Can you replay it and get the same result?

If any of those requires archaeology, you do not have an evidence layer. You have logs.

The minimum viable decision record

Not a framework. A shape. Anything that captures these fields survives most disputes.

from dataclasses import dataclass, asdict, field
from typing import Any
import hashlib, json

@dataclass(frozen=True)
class DecisionRecord:
    decision_id: str          # stable, referenced on the invoice line
    action: str               # "approve_claim", "resolve_ticket"
    outcome: str              # "approved" | "escalated" | "refused"
    billable: bool            # did this become a charge?
    model_id: str             # provider + exact version, never "gpt-latest"
    policy_version: str       # your prompt/rules version, semver
    input_digest: str         # hash, not the payload
    context_refs: list        # document ids + revision, not the text
    confidence: float | None
    ts: str                   # RFC3339, UTC

def digest(payload: Any) -> str:
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

Three choices in there are load-bearing.

Hash the input, do not store it. You get tamper-evidence and replay verification without inheriting a retention and privacy problem. If the hash matches on replay, the inputs matched.

billable is a first-class field, not derived later. The moment billing logic lives somewhere other than the decision record, the two drift, and reconciling them at renewal is worse than the original problem.

Pin the exact model version. "gpt-latest" is not a version. When a provider silently updates a model behind an alias, your replay is no longer a replay and you cannot say why behaviour changed in March.

Canonical JSON, or the hash is useless

If you hash a dict without a canonical encoding, key order changes break equality and your tamper-evidence becomes noise.

def record_digest(rec: DecisionRecord) -> str:
    d = asdict(rec)
    d.pop("decision_id", None)     # id is assigned after hashing
    return digest(d)

sort_keys=True plus tight separators is enough for most cases. If you later sign these records, use a real envelope format rather than inventing one. DSSE with PAE encoding is well specified and the implementations are small.

Wiring it to the invoice

This is the part that turns a governance artifact into a revenue one.

def billable_units(records, period_start, period_end):
    """One charge, one decision_id. No aggregate-only billing."""
    return [
        {"decision_id": r.decision_id,
         "action": r.action,
         "outcome": r.outcome,
         "ts": r.ts}
        for r in records
        if r.billable and period_start <= r.ts < period_end
    ]

The rule that matters: every charge on an invoice resolves to exactly one decision_id. Not a count. Not a rollup. If a customer questions line 4,127, you return that record.

Vendors who cannot do this end up defending totals instead of transactions, which is an argument you lose slowly.

Why this is a pricing decision, not a compliance one

Look at what the AI customer service market charges right now.

Vendor	Unit	Price
Intercom Fin	Resolution, free if escalated	$0.99
HubSpot	Resolved conversation	$0.50
Zendesk	Verified resolution, LLM-confirmed within 72h	~$1.20 to $2.00
Salesforce Agentforce	Conversation, resolved or not	$2.00

Same category. Different units. The vendors billing on outcome are carrying the failure rate themselves, and they can only do that if they can evidence the outcome.

One leading vendor reports 76% average resolution across 8,000+ customers. Independent reports put the same metric at 42 to 50%. That spread is not fraud. It is what happens when the billable unit has no verifiable definition.

Zendesk's answer was not a discount. They restructured in May 2026 to bill only on a resolution confirmed by a separate evaluation model within 72 hours. They added an auditor and removed the argument.

Four checks before you price anything

1. Can the customer see the unit? Tokens are your problem. A resolved ticket is theirs.
2. Can you measure it without argument? Write the definition as if procurement will read it, because eventually they will.
3. Does it scale with their value, not your cost? A unit indexed to compute makes you a reseller of inference.
4. Can you defend the bill nine months later? Which action, which version, what evidence, on demand.

Most teams pass 1 and 3, and fail 2 and 4.

The metric nobody tracks

Add time to reconstruct to your dashboard. If a customer disputes an output today, how long until you can show exactly how it was produced?

It is measurable, it is cheap to instrument, and it predicts your next bad quarter better than accuracy does. In my experience teams discover the honest answer is measured in days, and they discover it during the dispute rather than before it.

Caveats

This is a shape, not a library, and I have deliberately kept it small. Signing, retention policy, PII handling in context_refs, and replay determinism under a non-deterministic runtime are all real problems this sketch does not solve. Determinism for RL rollouts shipped in vLLM this year as a beta flag at roughly double the latency, and almost nobody turned it on, which tells you something about how much appetite there is for the strict version.

If you have built this properly in production, I would like to know where the record schema broke first. Mine broke on context references, because document revisions were not stable identifiers and I had assumed they were.