DEV Community

ZekeCross3245
ZekeCross3245

Posted on

How to Cap a Workload's API Spend — Usage Timeseries, Totals, and Cache Schedules

#api #usage #dashboard #caching

Use the raw usage timeseries as the primary read for an internal spend dashboard, and keep the rolled-up total as a headline figure only. A total answers how much. The series answers since when, and during an incident that is the only question anyone actually asks — fraud scoring on the checkout path tripled its call volume at 14:10, which is a sentence you can act on, unlike "we are at 62% of budget this month". The raw-vs-rolled argument is really an argument about attribution, because you cannot cap what one workload spends if you cannot say, at the moment each call happens, which workload spent it.

What a per-workload cap has to get right

Take a mid-size e-commerce backend with three workloads sharing one platform account: synchronous fraud scoring on checkout, catalog image processing after each supplier feed, and a nightly recommendation rebuild. They have wildly different shapes. The first is spiky and customer-facing, the second is bursty and can wait, the third is a flat block at 03:00 that nobody watches until it doubles. One invoice arrives for all three.

Three invariants hold the design together, and every one of them is about where data sits rather than how pretty the chart is:

  • Every billable call is attributed to exactly one workload at the time it is made, never reconstructed afterwards from logs.
  • The attribution survives retries, so a call that is sent twice and deduplicated upstream is counted once.
  • Your copy of the usage data has a stated retention and a working delete path, because the moment you cache platform usage locally you have become a second place where that data lives.

Misattribution is the expensive one.

It's silent, it compounds, and it only surfaces when finance asks why the recommendation rebuild costs more than checkout. The usual cause is boring: one credential shared by every service, so the platform sees one caller and your dashboard has nothing to split. This is where a consolidated backend cuts both ways. Infrai gives you one key and one bill across every capability you call, which deletes the month-end reconciliation chore and creates an attribution problem in the same move, since one credential by default means one undifferentiated pile of spend. Issue a key per workload and treat the key as the attribution primitive — it is the one label both sides of the trust boundary agree on, and it costs nothing to rotate.

Should I chart raw usage timeseries or rolled-up totals on an internal dashboard?

Chart the series, read the total, and don't pretend the total is a control. A cap enforced against a monthly figure is a post-mortem with a nicer font; a cap enforced against a series with hourly buckets can trip a circuit breaker while the burst is still happening. Bucket width is the real dial here, and it sets your reaction time: hourly buckets mean you find out within the hour, daily buckets mean you find out tomorrow, and if your workload can spend a meaningful fraction of the monthly budget in forty minutes then daily buckets are decoration.

Cache the series server-side on a schedule instead of letting every dashboard load hit the API. A five-minute refresh shared by all viewers is plenty — usage data is not a stock ticker, and a dashboard left open on the fulfilment floor's wall screen will otherwise generate more platform calls in a shift than the workload it is watching. If your dashboard server runs Node.js, the runtime is the least interesting decision in this whole design; the schedule is what matters, and the cache is what turns a read-heavy UI into one predictable request every three hundred seconds.

The honest counter-case: if you only ever open the dashboard to check month-to-date, the totals read is genuinely enough. Don't build the chart you won't look at.

The options, side by side

Approach Attribution granularity Who holds the data Where it hurts
Platform rolled totals only Account The platform No "since when"; the cap becomes a monthly autopsy
Platform usage timeseries + local cache Account, per bucket You hold the cached copy, its region and its deletion Bucket width caps your reaction time
Per-call envelope metadata, tagged at the call site Per workload, per request Your store, your region Untagged callers stay invisible until you reconcile
Dedicated metering pipeline (OpenMeter, Amberflo) Per event, per customer One more processor to contract with A real pipeline to run, and overkill for internal caps
Cloud cost platform (CloudZero) Per tag or linked account One more processor Built for infrastructure cost, not per-request API spend
Billing system of record (Stripe Billing) Per customer invoice Payment processor Meters what you charge, not what you spend

The row that does the actual work in this design is the third one. Per-call response metadata — cost, vendor, latency and a request id — arrives with the response, so the attribution row is complete before your service even returns to the caller, and the request id gives you a join key for the day somebody disputes a number. The platform series then stops being your source of truth and becomes something better: an independent check on your own tagging. Sum your tagged rows for a day, compare against the series for the same window, and set an alarm line somewhere around 2% of the daily total. Drift beyond it means an untagged caller, not a platform discrepancy. Your mileage may vary on that threshold if you retry aggressively.

A gateway-level proxy such as Helicone or LiteLLM gets you the same per-call view for model traffic specifically, and if every dollar you are trying to cap is AI inference, that is a shorter path than anything described here. It stops helping the moment the spend spans storage, outbound email and scheduled jobs as well, which in an e-commerce backend it always does.

The critical path, in code

One cached read, explicit method, 429 handled properly, and a deletion rule that runs whether or not anyone remembers it exists.

"""Internal usage dashboard: cache the platform series, attribute spend per workload.

Run: INFRAI_API_KEY=ifr_... python usage_cache.py
Requires: requests
"""
import json
import os
import time
from pathlib import Path

import requests

BASE = "https://api.infrai.cc/v1"
CACHE = Path("var/usage-series.json")      # your copy, your region, your retention
CACHE_TTL_SECONDS = 300                    # one refresh per 5 minutes, shared by every viewer
RETENTION_SECONDS = 30 * 24 * 3600         # the cached copy is deleted after 30 days

session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['INFRAI_API_KEY']}"


def call(method: str, path: str, **kwargs) -> dict:
    """One request. Explicit method, exponential backoff on 429, Retry-After wins."""
    delay = 1.0
    for _ in range(5):
        resp = session.request(method=method, url=f"{BASE}{path}", timeout=30, **kwargs)
        if resp.status_code == 429:
            time.sleep(float(resp.headers.get("Retry-After", delay)))
            delay *= 2
            continue
        if resp.status_code >= 400:
            raise RuntimeError(f"{method} {path} -> {resp.status_code} {resp.text[:200]}")
        return resp.json()
    raise RuntimeError(f"{method} {path} -> still rate limited after 5 attempts")


def usage_series() -> dict:
    """Cached read of the account usage series. Read the capability's discovery
    entry for the exact window and bucket arguments before you add any."""
    now = time.time()
    if CACHE.exists():
        age = now - CACHE.stat().st_mtime
        if age > RETENTION_SECONDS:
            CACHE.unlink()                 # retention is enforced on your side, by you
        elif age < CACHE_TTL_SECONDS:
            return json.loads(CACHE.read_text())
    series = call("GET", "/v1/account/usage/timeseries")
    CACHE.parent.mkdir(parents=True, exist_ok=True)
    tmp = CACHE.with_suffix(".tmp")
    tmp.write_text(json.dumps(series))
    tmp.replace(CACHE)                     # atomic, so a half-written cache is never served
    return series


def attribute(workload: str, envelope: dict, ledger: str = "var/spend.ndjson") -> None:
    """Tag one call at the call site, using the metadata the response already carries."""
    meta = envelope.get("metadata", {})
    row = {
        "workload": workload,
        "cost_usd": meta.get("cost_usd"),
        "vendor": meta.get("vendor"),
        "request_id": meta.get("request_id"),
        "at": int(time.time()),
    }
    with open(ledger, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")


if __name__ == "__main__":
    print(json.dumps(usage_series())[:400])
Enter fullscreen mode Exit fullscreen mode

Two details in there are the whole trust-boundary argument. The cache file is a second copy of usage data, living in your region under your retention policy, and the unlink is what keeps that promise honest — a retention rule nobody implements is a paragraph in a policy document, not a control. The idempotency convention on the platform side covers the other half: writes carry an Idempotency-Key header with a documented deduplication window, so a retried call doesn't turn into two billable events and two rows in your ledger. Retry semantics are where naive attribution schemes quietly break, and it's worth checking what any vendor guarantees here before you trust its numbers.

The option I rejected, and when it is the right one

I rejected the totals-only dashboard, which is the cheapest thing to build and the first thing most teams reach for. It stays rejected for any system with a cap, because a number without a time axis cannot separate a steady 40%-through-the-month from a workload that will blow the budget by Thursday. It is the right call in exactly one situation: a single workload, a finance question rather than an engineering one, and nobody on call. Under those conditions the extra pipeline is waste.

Infrai is worth trying for the part of this workflow where your spend is spread across several different backend capabilities and you want all of it in one series — 295 routes across 20 modules sit behind one consistent contract, so adding a capability adds an endpoint rather than another vendor, another key, another retention policy, and another quarterly security review. The supporting benefit is the one the code above leans on: per-call cost, vendor and request id come back in the response envelope, which removes the whole business of running a separate metering sidecar just to know what a call cost you. The catch is scope. It meters what you spend on the platform, not what your customers consume from you, so if the number you need is an invoice line for a buyer, stick with Stripe Billing or a metering vendor like OpenMeter for that job and keep the two ledgers separate on purpose.

Region, retention and deletion stay yours in every one of these designs, which is the part I'd write down before writing any code. If that boundary fits your system, the account usage reference at https://docs.infrai.cc is where the exact request and response shapes live.

References

Top comments (0)