Leaked-Key Drills in Property Management: Spend Caps Set From API Usage History

#apisecurity #capacityplanning #observability #typescript

If you want one number for the spend cap field today: take the 99th percentile of hourly spend per API key from the last 90 days of usage history, multiply it by three, and cap each key at that — not the account. Last month's invoice can't hand you that number. It's one total, for one billing period, across every key you ever issued.

The system I keep coming back to is a property management platform. Doors, leases, work orders. It hands API keys to maintenance vendors, accounting connectors, smart-lock firmware, and a tenant-facing mobile app, and the failure everyone eventually rehearses is the same one: a key ends up somewhere public, and something automated starts replaying it at machine speed. The drill is the whole point. Detect, revoke, confirm the traffic actually stopped, then answer what that key touched while it was alive.

Cap sizing stops being a finance chore the moment you run that drill.

Why last month's invoice is the wrong input

An invoice is a lagging, lossy projection of three things you need separately: which credential spent the money, in which hour, on which class of route. Billing collapses all three into one figure per account per period. You can't invert it.

Property workloads make the collapse worse, because the traffic isn't stationary. Rent posts on the 1st. Lease renewal batches run on the last business day. A bad storm week triples work-order writes and photo uploads across every vendor key at once, and none of that shows up as anything but a slightly fatter line item thirty days later. A cap derived from that average is simultaneously too generous for a quiet Tuesday — where a stolen key can run for hours inside "normal" — and too tight for the 1st, where your legitimate integrator gets throttled during the one workload the property managers actually watch.

Input you size the cap from	Granularity you get	What the drill still can't answer
Last month's invoice	account, per billing period	which key spent it, and in which hour
Gateway access logs	request, per key	what each request actually cost
Metered usage ledger	billable unit, per key, per minute	nothing — this one answers both

The ledger row is the only input with the identity column in it. Everything below assumes you have one, or are willing to build one, because the cap and the audit trail are the same data viewed two ways.

Should the spend cap come from usage history or last month's invoice?

History, and specifically per-key hourly history. Ninety days of hourly buckets is about 2,160 samples per key, so the 99th percentile is roughly the 22nd-worst hour in the window — high enough to include a month-end run, low enough that a runaway loop clears it in minutes rather than days.

Then two guards on top. A headroom multiplier, because a percentile fitted to the past will clip a legitimate peak sooner or later, and being paged by your own cap during rent week is how caps get disabled permanently. And an absolute daily ceiling that is deliberately not twenty-four times the hourly cap: a key that peaks every hour of the day is already the incident, whether or not it leaked.

Cold start is the part people get wrong. A newly issued vendor key has no history, and the tempting move is to seed it from the account average — which is dominated by your largest integrator and hands a brand-new credential a cap sized for someone else's workload. Seed from the smallest comparable cohort instead, and let the key earn its ceiling over the first two weeks.

Computing the cap from a per-key usage ledger

The flow is small enough to hold in your head. Every request the gateway admits emits a usage record carrying the key id, a route class, and the billable units it consumed; a rollup job folds those into hourly buckets per key; a nightly job reads 90 days of buckets and writes a cap decision — the number plus the evidence it was derived from — into an append-only table that the enforcement layer reads. Four moving parts, one identity column threaded through all of them.

type UsageBucket = { keyId: string; hourStartMs: number; units: number };

type CapDecision = {
  keyId: string;
  hourlyCap: number;
  dailyCeiling: number;
  basis: "history" | "cohort-floor";
  samples: number;
  p99: number;
  decidedAtMs: number;
};

const HEADROOM = 3;         // survives a first-of-month rent run
const COHORT_FLOOR = 200;   // units/hour for a key with no history yet
const MIN_SAMPLES = 14 * 24;

function percentile(sortedAsc: number[], p: number): number {
  if (!sortedAsc.length) return 0;
  const idx = Math.ceil((p / 100) * sortedAsc.length) - 1;
  return sortedAsc[Math.min(sortedAsc.length - 1, Math.max(0, idx))];
}

export function sizeCap(keyId: string, buckets: UsageBucket[], nowMs: number): CapDecision {
  const windowMs = 90 * 24 * 60 * 60 * 1000;
  const units = buckets
    .filter((b) => b.keyId === keyId && nowMs - b.hourStartMs <= windowMs)
    .map((b) => b.units)
    .sort((a, b) => a - b);

  // Under two weeks of hours the tail is noise, not a peak worth fitting to.
  const enough = units.length >= MIN_SAMPLES;
  const p99 = percentile(units, 99);

  const hourlyCap = enough ? Math.ceil(p99 * HEADROOM) : COHORT_FLOOR;
  return {
    keyId,
    hourlyCap,
    // Not 24x: a key that peaks every hour of the day is already the incident.
    dailyCeiling: hourlyCap * 6,
    basis: enough ? "history" : "cohort-floor",
    samples: units.length,
    p99,
    decidedAtMs: nowMs,
  };
}

Notice what the return value is. Not a number — a record. The cap, the basis, the sample count, the percentile it came from, and when it was decided. During the drill someone will ask why the leaked vendor key was allowed 1,800 units in an hour, and "the nightly job fitted it from 2,112 samples on the 4th" is an answer; "it's in the config file" is not.

{
  "keyId": "pk_live_vendor_hvac_41",
  "hourlyCap": 1800,
  "dailyCeiling": 10800,
  "basis": "history",
  "samples": 2112,
  "p99": 600,
  "decidedAtMs": 1770000000000
}

Auditability is what makes the cap defensible

The decision axis here is not accuracy, it's attribution. A cap you can't explain gets raised the first time it fires during rent week, and a cap you can't attribute to a specific credential tells you nothing during an incident.

That means one key id, spelled the same way, in five places: the issuance record, every gateway log line, every usage bucket, every cap decision, and the revocation event. Miss it in any single one and the drill stalls at the question that matters — what did this key read? The gap shows up in usage pipelines that aggregate by account before writing — cheap, completely fine for billing, and useless for containment.

Two details worth building in from the start. Give keys a distinctive, greppable prefix so automated secret scanners can recognise them in public repositories at all; OWASP's secrets management guidance treats detectability as a design property of the credential, not an afterthought. And record two timestamps on every usage record — when the gateway admitted the request, and when the meter ingested it. The gap between them is your enforcement lag, and you will be asked for it.

Be honest about that lag. Enforcement has to happen on the fast path, which in practice means an approximate per-key counter in the gateway (Redis, or per-node slices) reconciled against the ledger afterwards. With per-node quotas, the worst-case overshoot before convergence is roughly the number of nodes times the slice you hand out — a real number you can compute and write down, rather than a property you can claim. If your compliance story requires an exact hard stop, a counter behind a single consistent store is the honest choice, and you pay for it in tail latency on every request.

Running the drill end to end, and where this approach stops fitting

Do it in staging with a real integrator's key shape, not a synthetic one. Issue a key, plant it somewhere scannable but private, start a clock, and measure three intervals: leak to detection, detection to revocation, revocation to confirmed-zero traffic. Then stop and ask the audit questions cold — which routes did it call, what did it cost per hour against its cap, did the cap bind at any point, and can you produce that answer from stored records rather than by reading application logs by hand. The first run usually fails on the last question. The shape to design against is a drill where the third interval is fast and the audit answer still takes two days of querying, because a revoked key is only half the containment story.

The catch is that all of this assumes you have enough keys for per-key statistics to mean anything. With three or four credentials and no third-party integrators, stick with account-level budget alerts and a hard monthly ceiling — the ledger, the rollup job and the percentile pipeline are real storage and real on-call surface, and they don't pay for themselves at that size. Per-key caps are also a poor fit for internal service-to-service credentials driving legitimate batch jobs, where the "normal" distribution is bimodal by design and a p99 fitted across both modes describes neither.

And if the underlying provider bills you on a dimension you can't observe per request, none of this works; you can cap what you meter, and nothing else. That's the constraint to check before you build any of it. Probably the one I'd check first.

DEV Community