The Answer Looked Fine. The AI Bill Didn’t.

A Friday release adds a friendly touch to a support chatbot. Greet customers by loyalty tier. Give a Gold member a more personal welcome before answering their question.

The answer still looks reasonable. The order status is correct. Nobody changed the model.

But the request costs more.

That was the fictional case at the centre of Beyond the Prompt: The Engineering of AI Cost, the session I presented with Balaji Saravanan at Presidio India Exchange 2026 on 4 November. We used an application called AI Cost Engineering Lab to follow a support request, inspect its receipt, investigate a regression and evaluate a release gate.

I wanted the engineering work to be visible. A good answer on a chat screen hides a lot of decisions about context, retrieval, tools, retries and pricing. So we gave the audience a chatbot to start with and a detective game to investigate what happened behind it.

Presidio’s published schedule listed our 50-minute talk in the Day 1 Tech Talk programme. Event announcement.

This article follows that investigation, then goes deeper into the part I would build around a production application: a CI/CD gate that can refuse a cost regression without rewarding a worse answer.

The game uses synthetic execution and teaching rates. The chatbot has a separate live-provider path. The production pipeline described later is a reference design that extends the demo’s ideas. The screenshots and numbers below show the teaching application, with their evidence source identified.

Start with the request the customer actually made

Our fictional customer, Asha, asks about an order. ShopBot has customer-scoped, read-only tools for orders, shipments, profiles, policies and support cases. The visible experience is a support conversation. Underneath it, the application records the model invocations, tool work and evidence needed to explain the answer.

The final deployed ShopBot interface, shown with viewer access. The customer and records are fictional. Chat supports a separately selected live or synthetic execution source.

A single customer message can lead to several model calls. A tool result can become input to the next call. The next call can carry previous messages, tool definitions and policy passages again. A failed attempt may have already consumed billable usage before the application retries.

That is why the first useful question is more specific than “which model did we use?”

What did the complete request do, and what did that work cost?

Read the receipt before choosing an optimization

For a provider that bills these categories separately, a normalized model charge can be written as:

model charge =
    fresh input tokens × fresh input rate
  + cache-write tokens × cache-write rate
  + cache-read tokens × cache-read rate
  + output tokens × output rate

Apply the rate’s unit conversion, such as dividing by one million for a per-million-token price. Then sum across every charged invocation and attempt in the request.

The normalization matters. Providers can expose different usage fields and token inclusion rules. An adapter must map those fields into mutually exclusive billing categories before applying a dated price table. Adding cached tokens to an input total that already includes them would charge for the same work twice.

Amazon Bedrock’s Converse API exposes usage, stop reasons and latency metrics. Its cache-read and cache-write fields give an application a way to retain the provider’s observations alongside its own trace. Bedrock Converse response documentation

I keep four distinctions in the receipt:

• Provider usage records the counters returned by the inference API.
• Estimated composition explains how much context appears to come from instructions, retrieval or tools. A character-based token estimate belongs here.
• Calculated charge applies a versioned price table to usage.
• Invoice reconciliation accounts for what the provider actually billed, including applicable commercial terms.

Our lab calculates execution charges. A production cost model also needs to define how it treats embeddings, retrieval infrastructure, tool APIs, orchestration and human escalation. Keep that boundary attached to every number.

If a required usage counter or price is unavailable, record an unknown value. Replacing it with zero makes the next comparison look cheaper without establishing any saving.

Cost per resolved conversation changes the question

Suppose two versions each process 100 conversations. This is an illustrative calculation:

	Version A	Version B
Total measured execution charge	$1.00	$0.80
Conversations resolved under the rubric	90	60
Charge per attempted conversation	$0.0100	$0.0080
Charge per resolved conversation	$0.0111	$0.0133

Version B spends less in total and more per successful outcome.

The calculation is:

cost per resolved conversation =
    total in-scope cost of all attempted conversations
    / number of conversations successfully resolved

The numerator includes work spent on unsuccessful attempts. The resolution definition needs a versioned rubric. If nothing was resolved, the ratio is undefined and the release decision must handle that explicitly.

For monthly planning, multiply expected conversation volume by expected cost per conversation. For a resolution target, account for the resolution rate and follow-up work. Report traffic volume separately so a larger workload does not masquerade as a per-request regression.

Five places to investigate

We organised the investigation around five layers. They are useful places to look for a cause. Their effects can overlap, so adding five independent “savings” estimates is unreliable.

Layer	What I would inspect	A possible experiment
Model	Exact model identity, routing, input/output rates, answer quality	Route a bounded task to a smaller model and score the same cases
Context	Stable instructions, message history, dynamic prefixes, cache observations	Move changing data after the cache boundary
Retrieval / RAG	Retrieved passages, relevance, duplication, grounding	Reduce redundant passages while checking answer coverage
Tools	Fields returned, payload size, repeated reads, failure behaviour	Return the fields needed for this task
Agent behaviour	Calls, attempts, retries, termination and escalation	Bound the loop and inspect failed cases

The Journey view animates recorded events from a completed synthetic run. Playing the trace does not make another model call.

The journey connects those layers. A broad tool response can increase context on several subsequent calls. A retrieval change can reduce input tokens and also remove the evidence needed to answer correctly. A smaller model can require more attempts. The full workflow decides whether the change helped.

Case #1 — The Expensive Greeting

The game takes the investigator through Case file → Crime scene → Line-up → Lab → Verdict. Each location has a job in the investigation.

The map makes the investigation sequence visible. The clues come from stored run evidence.

The crime scene establishes what changed

CR-2041 is the fictional festival-weekend release. It introduces loyalty-tier greetings. We reproduce the healthy and changed versions on the same questions, then compare cost, quality and test volume.

This two-question reproduction is a diagnostic sample. The displayed charges use synthetic usage and teaching rates. It cannot establish a production regression rate.

The investigator opens evidence records, reads the before-and-after summaries and labels each contact. Those labels are notes. An accusation in the Line-up still has to survive a one-change lab test.

The final game gives the player three Lives for wrong accusations, followed by unlimited attempts in Free Play. Those mechanics keep the investigation moving. They do not alter the evidence required to close the case.

The first clue is where the changing text appears

Prompt caching depends on the content before the cache boundary remaining eligible for reuse. In this case, customer-specific material changes that prefix.

The pattern looks like this:

Before CR-2041
  Stable instructions and tool definitions
  Cache boundary
  Customer-specific context and current question

After CR-2041
  Changing personalization
  Stable instructions and tool definitions
  Cache boundary
  Current question

The teaching trace changes from a cache write followed by reads to repeated writes. Moving the varying material after the stable prefix is the Context experiment. Claude’s caching documentation describes exact prefix matching and explains why placing a breakpoint on changing content can keep producing fresh writes. Claude prompt caching documentation

A production test also needs the provider’s minimum cacheable length, TTL, supported model and request format. A stable hash is evidence about prompt construction. Cache counters establish what the provider observed. An unknown cache observation stays unknown.

The second clue is what the profile tool returns

The greeting needs a small projection of customer information. Returning the full profile can send unrelated fields into the prompt and carry them through later calls.

A minimal response might look like:

{
  "display_name": "Asha",
  "loyalty_tier": "Gold"
}

The actual allowlist follows the task and policy. If answering the question requires an order record, fetch that scoped record through the appropriate tool.

Our Tools experiment changes the profile projection and measures the result. The lab also keeps a Tools-only variant. That lets the investigator compare the effect of fixing the payload while the context problem is still present.

The lesson is to measure the combined change. Fixing one contributor does not prove the remaining request is within budget.

An alibi needs a boundary

The same model identity before and after rules out a model switch in that comparison. It does not establish that every generation will behave identically.

Similarly, a familiar successful-call count does not explain rejected attempts or retries. The game can leave Agent as Alibi unclear when the evidence does not justify clearing it. A production investigation needs that same ability to withhold a conclusion.

What the gate actually demonstrated

The game’s larger gate trial uses 60 matched synthetic questions per comparison. This is separate from the two-question crime-scene reproduction.

Teaching verdicts from the final app. The values are calculated using synthetic execution and invented fixture rates.

Teaching gate metric	Healthy reference	CR-2041	Both fixes
Calculated charge per resolved chat	$0.007263	$0.011300	$0.007504
Eligible-call cache hit rate	66.7%	0.0%	66.7%
Profile output, p95 estimated tokens	20	1,616	20
Agent steps, p95 successful model calls	3	3	3
Resolution under the fixture rubric	100%	100%	100%

The as-shipped version is BLOCK. The version with both fixes is PASS under the teaching policy. Its charge remains slightly above the reference but within the allowed increase.

The implementation also carries an evidence-admissibility flag. A passing synthetic teaching verdict does not authorize a production release. The repository’s hosted cost-quality workflow exercises synthetic evaluations and gate plumbing. Production model replay, deployment approval and canary automation need their own implementation.

That separation is where I would begin the production work.

Building a production cost-and-quality gate

A gate needs an evidence-producing pipeline, a policy evaluator and an enforcement point that prevents release. Printing a red or green report only implements the middle part.

The delivery path I would build is:

PR and candidate artifact
       ↓
Gate 0 — static contracts and budget checks
       ↓
Gate 1 — bounded, matched real-model evaluation
       ↓
Gate 2 — evidence validation and policy decision
       ↓
Required check for the evaluated artifact
       ↓
Canary → observe → promote or roll back

1. Decide what success means before running the comparison

Use a versioned evaluation set with known expectations. Include common support paths, ambiguous requests, tool failures, policy boundaries, long histories and cases that must escalate. Add slices for material production segments such as language or request category.

Define resolution in observable terms. For an order-status question, that might require the correct order, the recorded shipment state and an estimate presented with the proper qualification. For a refund question, check eligibility, policy grounding and the permitted next action. A fluent answer alone does not satisfy those checks.

Use deterministic checks for facts that the application can verify. If a model judge is useful for a broader rubric, version its model and prompt, calibrate it against human labels, and track disagreement. The judge’s output is another measurement with an error rate.

Keep evaluation cases held out from prompt tuning. When a production incident becomes a regression case, version the new dataset and retain the previous one for comparison history.

2. Freeze the inputs and identify both artifacts

A meaningful comparison records more than a branch name:

{
  "candidate_digest": "sha256:<candidate artifact digest>",
  "baseline_digest": "sha256:<approved baseline digest>",
  "dataset_sha256": "<versioned evaluation set hash>",
  "policy_sha256": "<approved policy hash>",
  "rubric_version": "support-resolution-v3",
  "price_table_version": "provider-region-date-version",
  "evidence_source": "live_provider_evaluation",
  "cache_protocol": "cold-and-warm-measured-separately"
}

These are example field names, not a record of a production deployment.

Freeze the customer question, conversation history, tool fixtures and policy corpus for both sides. Pin model identifiers and inference parameters where the provider supports it. If model routing is the variable being tested, record that difference explicitly.

A baseline must come from an approved registry of release artifacts. Allowing a candidate to select an unusually expensive reference or edit the acceptance policy would undermine the test.

A content hash detects changed bytes. Establishing who produced the evidence also requires a trusted evaluator and authenticated artifact storage or attestations.

3. Run cheap checks first

Gate 0 can reject obvious contract problems without buying inference:

• Render prompts for different customers and clocks, then compare the designated stable prefix.
• Check model-visible tool payloads against a schema, field allowlist and size budget.
• Validate retrieval limits, output caps, retry limits and termination conditions.
• Check that a proposed model or route is allowed for the data classification.

These checks would detect the two specific construction problems built into our case. They cannot predict all runtime cost or quality behaviour.

For payload sizing, record the tokenizer or estimator version. A character estimate can be a useful local budget check, while provider usage remains the source for billed model-token accounting.

4. Bound the real-model replay

Gate 1 runs the candidate and baseline on matched cases. It should have a maximum number of calls, a reservation for worst-case spend, output-token caps, a deadline and a cancellation path.

Account for retries and evaluator calls in the evaluation budget. Reserve spend before dispatching concurrent work so several workers cannot each approve against the same remaining balance. Keep final usage and outstanding reservations consistent when a worker times out.

Use recorded tool responses when the goal is isolating prompt or agent behaviour. Use controlled integration tests when a tool contract itself changes. A replay of a historical model answer does not measure a new prompt’s effect on the model. Record what was replayed and what actually executed.

Test cold and warm cache conditions separately. Interleave or randomize baseline/candidate order, control cross-run cache reuse, and document warm-up charges. Otherwise the version that ran second may inherit an advantage unrelated to its change.

5. Preserve every scheduled case and attempt

The evidence contract should account for the complete scheduled set. A useful per-attempt record includes:

case ID and repetition ID
artifact and configuration identities
attempt number and status
provider request reference
usage categories and pricing version
calculated charge or missing-cost reason
latency, tool payload size and model-call count
rubric results and critical-failure flags

Retain failures, timeouts and cancellations. Define whether a retry belongs to the same customer attempt and make that aggregation rule consistent. Reject duplicate identifiers and mismatched cohorts. Never compare only the intersection of successful cases and silently lose the difficult cases.

Store raw traces behind access controls. CI reports can expose redacted summaries, case IDs and evidence links. Prompts and tool responses can contain customer data, so collecting full content needs an explicit data-handling policy. OpenTelemetry’s GenAI guidance provides a useful tracing vocabulary and treats content capture as an opt-in concern. OpenTelemetry GenAI observability

6. Choose thresholds with their meaning attached

The lab policy provides a concrete example:

# Teaching values from the case, requiring calibration for production.
minimum_matched_cases: 60
cost_per_resolved_max_increase: 0.10
cache_hit_rate_max_drop_percentage_points: 15
profile_payload_p95_max_increase: 0.25
profile_payload_p95_budget_tokens: 1000
agent_steps_p95_max: 5
resolution_rate_floor: 0.90
resolution_rate_max_drop_percentage_points: 2
critical_failures_max: 0

An increase from 100% to 98% is a two-percentage-point drop. A 10% relative cost increase uses a different calculation. Encode units in the schema and the report.

A production policy should also define absolute spend, latency, coverage by important segment, permitted evidence sources and what happens when a metric is inapplicable. Treat cache hit rate as a diagnostic guardrail with a defined denominator. A deliberate cache trade-off can require a reviewed policy change while still being evaluated on cost and quality.

Sixty is the demo’s minimum, not a universal sample-size recommendation. Choose production sample size around tolerated regression, variance, business risk and the cases you need to cover. Zero observed safety failures in a small sample does not establish a zero failure rate.

7. Handle model variability before deciding

Repeated runs help distinguish a change from ordinary variation. Pair baseline and candidate by case. For repeated observations on the same conversation, resample or analyse at the conversation level so correlated attempts do not become fictitiously independent evidence.

For cost per resolved conversation, one practical analysis is a paired bootstrap over cases. In each resample, calculate each side’s total cost divided by its resolved count, then calculate the ratio. A resample with no resolutions needs an explicit undefined-result policy.

Choose the statistical rule in advance. For example, require the upper confidence bound of the cost ratio to remain below the allowed increase, alongside the quality constraints. If the interval crosses the boundary, return INCONCLUSIVE and use the predetermined additional-evidence procedure. Re-running until one batch turns green invalidates the rule.

Use an equivalent uncertainty treatment for resolution and other noisy metrics. Review important slices and tail behaviour as well as the overall mean. A global saving can hide a serious regression in one support path.

8. Make the decision fail closed

The decision model needs at least these outcomes:

Outcome	Meaning	Release action
INVALID	Schema, identity or evidence-integrity checks failed	Refuse and repair the evidence
BLOCK	A known policy or critical-safety constraint failed	Refuse and fix the change
INCONCLUSIVE	Required coverage, metrics or statistical confidence are missing	Refuse automatic promotion
WARN	An explicitly defined review band was reached	Follow the documented review policy
PASS	Every required condition is satisfied	Permit the next release stage

This is the shape of the decision core after evidence validation and metric computation:

def decide(validation, metrics, policy):
    if not validation.valid:
        return "INVALID"

    failures = evaluate_known_hard_limits(metrics, policy)
    if failures:
        return "BLOCK"

    if not metrics.complete or not metrics.coverage_sufficient:
        return "INCONCLUSIVE"

    if not metrics.uncertainty_within_policy:
        return "INCONCLUSIVE"

    if metrics.in_review_band:
        return "WARN"

    return "PASS"

This is decision pseudocode. The validator, metric calculations and uncertainty checks are explicit responsibilities of the evaluator. They must also reject an evidence source that the release policy does not admit.

Only an explicitly admissible PASS should produce a successful release check. A missing report, parser error, cancelled evaluation or controller crash must leave promotion blocked. Upload a sanitized report even when evaluation fails, while preserving the failing exit status.

9. Put credentials and authority in a trusted control plane

I would separate the production gate into three roles:

1. Candidate execution runs the application artifact in an isolated sandbox, with fixture data and tightly bounded tool access.
2. Evaluation control selects approved datasets and policy, meters inference through a constrained service, and writes authenticated results.
3. Release enforcement validates the results and publishes the required check for the exact candidate identity.

The candidate must not receive a credential that can change policy, overwrite the baseline, forge usage, or mark itself approved. Keep provider credentials in the metered inference service and reconcile its request records with the run evidence.

Untrusted PR code also needs isolation from CI secrets and write tokens. GitHub documents how checking out and executing PR code in a privileged pull_request_target workflow can expose the base repository’s credentials. A label or review on an earlier revision does not make later pushed code safe. GitHub Actions security guidance

For an initial implementation, a trusted controller can accept a reviewed candidate digest, start the bounded evaluation and await its attested result. The submission contract can be small:

{
  "repository": "approved-repository",
  "candidate_sha": "<full commit SHA>",
  "candidate_digest": "sha256:<reviewed image digest>",
  "baseline_release": "approved-baseline-v12",
  "evaluation_suite": "support-heldout-v3",
  "request_id": "<unique idempotency key>"
}

The server resolves the dataset, price table and policy from trusted versions. It validates artifact provenance, authorizes the requester, reserves budget and records the job. Retries with the same idempotency key return the same evaluation identity. The candidate cannot supply a larger budget in this request.

10. Connect the verdict to the merge and deploy controls

Create a required check such as ai-cost-quality/release through the trusted controller’s GitHub App. Attach it to the evaluated commit and retain the artifact digest in the report. Configure the expected App as the source of that required check.

A new commit needs fresh evidence. If the merge result differs from the evaluated artifact, evaluate that result or rebuild and verify its identity before deployment. Merge queues need checks for their merge-group commits. GitHub’s documentation covers latest-commit requirements, expected check sources and the merge_group trigger. Required status checks

Protect changes to the evaluator, policy and workflow with designated reviewers. Make any bypass explicit, time-limited and attributable. Apply the repository’s restrictions to the relevant privileged roles as well. GitHub provides branch protection and ruleset controls for this enforcement. Protected branches

The deployment job checks that the approved digest, policy version and evidence validity period still match. It promotes that exact image. It must not rebuild an unreviewed artifact after the gate has passed.

11. Keep the production observation loop

An evaluation set cannot represent every future conversation. Roll out to a bounded canary cohort with a defined observation window, spend limit and rollback procedure.

Track cost per attempted and resolved conversation, resolution and escalation, critical failures, latency, calls, retries, tool payloads and cache observations. Compare like traffic segments and cache conditions. Quality labels can arrive later than usage, so define how long promotion waits for them and how late failures trigger rollback.

Tie the rollback to a known-good artifact and configuration. Test the mechanism before relying on it. Also define failure behaviour when telemetry disappears or the rollout controller stops responding.

An exception should identify the failed criterion, the accountable approver, the permitted scope and the expiry. That keeps an accepted risk distinguishable from a silent green check.

What I took away from building the lab

The detective game made it possible to open a clue, make an accusation and test an explanation. The underlying method applies without the game UI:

Measure → Diagnose → Optimize → Validate → Monitor.

Measure the whole attempted workflow. Diagnose through the request trace. Change a bounded control. Validate outcome quality beside cost. Keep monitoring after release.

The production gate is the delivery version of that method. It needs enough evidence to justify a decision and enough authority to enforce it.

Thank you to Balaji Saravanan, the Presidio India Exchange organisers and everyone who joined the session.

The sandbox used for the talk has since been torn down. These screenshots preserve the application and its teaching examples. The next step for this work is applying the evaluation and enforcement design to a workload with an approved production dataset, cost boundary and release policy.

Written with AI assistance. The application behaviour and teaching figures were checked against the project’s implementation and final release evidence.