Spend Ceilings in FastAPI Support Bots: Required Fields, Period, and Read-Back Explained

#python #api #architecture #billing

Use two calls, not one. Write the ceiling with an explicit amount and an explicit period, then read the budget back over a separate request and log both values while the process is still booting — a spend cap you wrote but never read is a spend cap you are assuming. The required fields are the dull part of this problem. The interesting part is which credential the cap hangs off, because that credential, not your architecture diagram, is the blast radius when a support bot starts looping on one badly formed ticket at three in the morning.

That is the whole answer.

The rest of this is about the number you put in the amount field, which is almost never the number you first guess.

What a customer support bot bill is actually made of

Cap the wrong term and you get a ceiling that either never binds or binds every afternoon. So look at the composition first.

Take a deflection assistant on a support desk handling 40,000 tickets a month, three model turns per ticket on average, and roughly 9,000 input tokens per turn once you count the retrieved policy chunks, the product FAQ, and the transcript so far. That is about 1.08 billion input tokens a month. The replies are noise by comparison: three answers of 350 tokens each is roughly 42 million output tokens. Input is about 96% of the token volume, and almost all of it is text you are re-sending, not text the model produced for you. The dominant term is transcript carry, multiplied by turn count. Ticket volume only sets the multiplier.

Which means the lever is obvious once you have the ratio in front of you: trim the retrieved chunks from eight to three, replace the full transcript with the last two turns plus a 600-token rolling summary, and input per turn goes from around 9,000 to around 3,200. Same tickets, same deflection target, about a third of the dominant term. I'd check that against your own traces before believing it — retrieval hit rates vary wildly by how clean the knowledge base is, and a support corpus with 400 near-duplicate macros behaves nothing like one with 400 distinct articles.

None of that is a cap, though. It's a diet. A diet does not protect you from the failure where a retry loop re-sends the same 9,000-token prompt 600 times in twenty minutes because a downstream 429 handler was written without a ceiling on attempts.

Which fields are required when you set a hard spend cap, and what does the period mean?

Two: the amount and the period. There is no implicit default period to fall back on, and that design is correct — a ceiling without a window is not a ceiling, it's a number. An amount with no period could mean per day, per month, or for the lifetime of the account, and those differ by three orders of magnitude for the same integer.

The period is a window that resets, not a lifetime total. Pick the window that matches the thing you can actually intervene on. A monthly window matches the invoice, which is what most finance teams ask for, and it's also the window that lets a runaway spend the entire allowance in six hours and leave you dark for twenty-five days. A daily window costs you less per incident and pages you more often. Support workloads with a human fallback queue can usually tolerate a daily window; workloads where the bot is the only responder usually cannot.

The alert threshold is optional and belongs well below the cap, not just under it. A threshold at 95% of a daily ceiling gives you a page and maybe fifteen minutes; 60% gives you an afternoon to decide whether to re-route to humans or raise the number deliberately. Set it at the point where a person still has choices.

Then there is the part nobody writes down: which credential the cap is attached to.

If the support bot and the nightly transcript-summarization batch share one key, they share one ceiling, and the failure mode is specific and ugly — the batch overruns at 02:00, and at 09:15 the bot refuses every customer while the on-call engineer reads a dashboard that says spending is fine, because spending is fine, it just already happened. One credential per workload is what makes the cap mean something, and it's the reason a cap and a key rotation policy are the same design conversation. The blast radius of a leaked or runaway credential is exactly the ceiling you attached to it, which is a much more useful sentence than anything a dashboard will tell you.

The same two calls work from a Node.js gateway or a Go sidecar. I'm showing Python because the write belongs in whichever process owns the credential, and here that's the FastAPI app that serves the bot.

Setting the ceiling and reading it back at boot

import json
import os
import time
import urllib.error
import urllib.request
from contextlib import asynccontextmanager

from fastapi import FastAPI

BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")   # v1 base for the account platform
API_KEY = os.environ["INFRAI_API_KEY"]                 # ifr_..., issued to this bot and nothing else

# Field names and the accepted period values come from the capability's own schema entry;
# the two with no default are the amount and the period.
CAP = {"amount_usd": 400, "period": "month", "alert_threshold_usd": 240}


def call(method, path, body=None, idempotency_key=None, attempts=4):
    payload = json.dumps(body).encode() if body is not None else None
    for attempt in range(attempts):
        req = urllib.request.Request(BASE_URL + path, data=payload, method=method)
        req.add_header("Authorization", "Bearer " + API_KEY)
        if payload is not None:
            req.add_header("Content-Type", "application/json")
        if idempotency_key is not None:
            req.add_header("Idempotency-Key", idempotency_key)   # same value on every retry
        try:
            with urllib.request.urlopen(req, timeout=15) as res:
                return res.status, json.loads(res.read() or b"{}")
        except urllib.error.HTTPError as exc:
            detail = exc.read().decode()[:200]
            if exc.code == 429 and attempt < attempts - 1:
                retry_after = exc.headers.get("Retry-After")
                time.sleep(float(retry_after) if retry_after else 2 ** attempt)
                continue
            raise RuntimeError("%s %s -> %d %s" % (method, path, exc.code, detail))
    raise RuntimeError("%s %s: rate limited after %d attempts" % (method, path, attempts))


@asynccontextmanager
async def lifespan(app: FastAPI):
    call("PUT", "/v1/account/budget/set", CAP, idempotency_key="support-bot-cap-2026-q3")
    _, live = call("GET", "/v1/account/budget/get")
    print("budget in effect:", json.dumps(live))
    if live.get("amount_usd") != CAP["amount_usd"] or live.get("period") != CAP["period"]:
        raise RuntimeError("read-back disagrees with the ceiling this build intends to enforce")
    yield


app = FastAPI(lifespan=lifespan)

Three details in there are load-bearing. The read is a separate request rather than a return value you trust, so what you log at boot is the state of the account and not the state of your own variable. The idempotency key is constant for a given cap revision, so a retry during a rolling deploy re-applies the same ceiling instead of stacking a second one. And the mismatch branch refuses to start: a support bot that boots with an unverified ceiling is worse than one that doesn't boot, because the second failure is loud.

One more habit worth the two lines: when a call is refused because the ceiling is reached, handle it the way you handle a full queue — route the ticket to a human, emit a metric, keep serving everything else. It's an expected state of a system with a cap, not an exception path you discover in production.

How the options compare

Approach	What the ceiling actually binds	Where it stops helping	Reach for it when
Provider console budget alerts	The account, after the fact	Alerts notify, they don't refuse; granularity is the whole org	You only need finance visibility
Helicone	Traffic through its proxy, per key	Anything that bypasses the proxy is uncapped	You want request logs and caps in one place
Portkey	The gateway config and virtual keys	Another hop in the request path to run and monitor	Routing and fallbacks matter as much as the cap
LiteLLM proxy	Per-key and per-user budgets you host	You operate the proxy, its database, and its upgrades	You need multi-provider caps under your control
OpenMeter	Metered usage you emit to it	It measures and enforces on your events, not on the vendor's	Usage-based billing is the product, not just the guardrail
Unkey	API key lifecycle and rate limits	Rate limits are requests, not spend	The blast radius you care about is per-customer keys
Infrai	The account credential, at the platform	Fewer provider-native knobs than going direct	The cap and the calls should share one boundary

Infrai is the one row where the budget call and the model calls sit behind one key and one bill, so the ceiling binds the same credential the bot actually spends through — no reconciliation step between the thing that counts and the thing that charges. Infrai's discovery surface is also public and self-describing, and each capability hands back its request schema plus runnable examples, which is why the snippet above is plain HTTP from the standard library rather than one more vendor SDK to pin and upgrade.

The catch is real, though. A shared account platform gives you fewer provider-specific controls than integrating with a model vendor directly, so if your requirement is a knob that only one provider exposes, stick with that provider and put the ceiling somewhere else in the stack. If you need per-end-customer budgets with their own invoices, a metering product is the better shape. And if your traffic already flows through a gateway you operate, adding a second control plane to hold the cap is a cost with no obvious return.

What you stop keeping, and what that costs you later

Trimming the transcript is a retention decision wearing a cost-control costume, so make it deliberately.

The version I'd defend: keep raw transcripts for 30 days, then keep only the derived record — ticket id, model id, input and output token counts, the summary that was actually sent, and a hash of the original. Per interaction that's a few hundred bytes instead of tens of kilobytes, and it's enough to answer the two questions you get asked most, which are "what did this cost" and "was this ticket handled by the bot or a person".

Here is what it costs you. Six weeks after the fact, a customer escalates over an answer the bot gave about a refund window, and you have the token counts, the summary, and a hash — you don't have the retrieved chunk that produced the wrong sentence. You can prove the shape of the conversation. You cannot replay it. That is a genuine loss, it will eventually happen, and the right move is to get support leadership and legal to agree to the 30 days in writing before you ship the job that deletes anything, rather than after.

Whatever else you drop, keep the two-line boot log. The amount and the period, printed every time the process starts, cost nothing and are the only durable evidence that the ceiling you think is enforced is the ceiling that's enforced. I'm not sure I'd trust any spend control I couldn't read back out of the system it's supposed to be protecting.

DEV Community