Spot-priced LLM inference changes how runtime budget guards should work

Most AI cost guards assume one thing:

You know the price before you make the request.

That's been true for most LLM APIs.

It's becoming less true as inference marketplaces introduce dynamic pricing, where providers compete for requests and the final price is determined at runtime.

That creates an interesting engineering problem.

Traditional budget guards

A typical pre-call budget check is simple:

1. Look up the model price.
2. Estimate token usage.
3. Estimate request cost.
4. Compare against the remaining session budget.
5. Allow or block the request.

That works well when prices are fixed.

It doesn't work as well when the final price isn't known until the request completes.

The wrong solutions

There are two obvious approaches.

Ignore pricing uncertainty

This removes one of the main reasons the guard exists.

Always assume list price

Safe, but often overly conservative. You end up rejecting requests that would have cleared well within budget.

Neither is ideal.

Reserve a maximum, settle the actual

A pattern borrowed from cloud infrastructure works much better.

Instead of estimating the exact cost, reserve the maximum amount you're willing to spend.

interface SpotRequest {
  maxBidPerMillion: number;
  estimatedInputTokens: number;
  estimatedOutputTokens: number;
}

Before the request:

• Reserve the worst-case spend.
• Reject the request if that reservation exceeds the session budget.

After the request:

• Replace the reservation with the actual amount charged.
• Release any unused budget back to the session.

The important property is that your budget is never temporarily overstated.

Budget enforcement becomes deterministic

With this approach, the guard doesn't ask:

How much will this request cost?

Instead, it asks:

Can this session afford the maximum amount we're willing to pay?

That's a question you can answer before the request leaves your process.

Bid ceilings become another optimization tool

Once you introduce a maximum acceptable price, that ceiling can become part of your routing strategy.

For example:

• low-value classification → low bid ceiling
• code review → medium ceiling
• architecture tasks → higher ceiling

As the session budget decreases, those ceilings can become more conservative.

Instead of simply rejecting requests when money runs low, your system can automatically become more selective about what it's willing to pay for.

Why this matters

Today it's spot-priced inference.

Tomorrow it may be:

• dynamic provider marketplaces,
• region-specific pricing,
• time-based pricing,
• SLA-based pricing,
• or real-time routing across multiple providers.

The common pattern is the same:

The exact price is no longer guaranteed before the request is sent.

That means runtime cost controls need to evolve as well.

Rather than assuming every request has a fixed price, they should be designed to operate under uncertainty—enforcing budgets using maximum acceptable cost, then reconciling against the actual price afterward.

That's a much more flexible architecture for the direction AI infrastructure appears to be heading.
https://github.com/salimassili62-afk/ai-costguard