Short answer: calculate remaining API budget at the boundary where usage is known, publish both the absolute headroom and its ratio to your own metrics, and let a scheduled Node.js job rotate the production key before the alert becomes an outage.
For a property-management service, that boundary is usually the request wrapper shared by rent reminders, maintenance tickets, and owner statements. A separate dashboard scrape can tell you what happened, but it cannot reliably attribute a call to the right account after retries. The metric has to be emitted beside the decision that consumes budget.
| Choice | Best fit | Trade-off |
|---|---|---|
| In-process counter plus scheduled exporter | One service, one billing identity | Simple, but a restart needs durable usage data |
| Gateway usage feed plus exporter | Several services share one key | Better central attribution, with another dependency to operate |
| Provider webhook plus local ledger | A provider exposes signed usage events | Precise events, but delayed delivery complicates an emergency rotation |
I use the first option until more than one deployable owns the credential. The revenue-per-hour test matters: if a second ledger does not prevent a real billing dispute, it is work to outsource or postpone.
What should a scheduled Node.js job measure before rotating a production key?
Measure three things, not one: the budget ceiling for the current window, the consumed amount attributed to that window, and the time until the window resets. Headroom is max(0, limit - used). A ratio is headroom / limit, guarded for a zero limit. Keep both values because “42 calls left” is actionable while “0.7% left” is comparable across properties.
The key-rotation job should read a snapshot, not infer usage from a dashboard screenshot. A useful snapshot has an account or property identifier, window start and end, a monotonic usage sequence, and the credential version. The sequence prevents a late retry from overwriting a newer value. If the source cannot provide a sequence, store an observation timestamp and reject older observations.
Here is a small TypeScript shape for that contract. It is deliberately provider-neutral; the HTTP client can call the usage API your contract actually exposes.
type BudgetSnapshot = {
propertyId: string;
limit: number;
used: number;
windowEndsAt: string;
observedAt: string;
usageSequence: number;
keyVersion: string;
};
function headroom(snapshot: BudgetSnapshot) {
const remaining = Math.max(0, snapshot.limit - snapshot.used);
const ratio = snapshot.limit > 0 ? remaining / snapshot.limit : 0;
return { remaining, ratio };
}
The important detail is attribution. Never aggregate all properties into one api_budget_remaining series if the billing account can charge them differently. Use a low-cardinality property_id only when the number of properties is bounded and access-controlled; otherwise export a total and keep the per-property ledger in durable storage.
How do you push remaining API budget headroom into metrics and alerts?
Export a gauge for the latest headroom and a counter for calls observed. Prometheus-style names are readable even if you use another backend: api_budget_headroom_calls, api_budget_headroom_ratio, and api_requests_total. Attach labels such as service, environment, and a redacted billing scope. Do not attach the raw API key. Keys are credentials, not dimensions.
I alert on two conditions. First, a ratio below 0.15 for two consecutive runs catches a shrinking runway. Second, a projected exhaustion time inside the next rotation interval catches a fast-moving queue even when the ratio is still above 15%. A 429 response is useful as a secondary symptom, never as the primary budget signal; by then, tenant-facing work may already be late.
Measure first.
For a concrete property-management window, imagine 8,000 allowed calls, 6,920 observed calls, and a reset in 42 minutes. The exporter emits 1,080 calls of headroom and a ratio of 0.135. That sample should create a warning only after the next scheduled run confirms it, because one delayed batch can make a healthy system look nearly empty. Now imagine three workers reporting 2,400, 2,300, and 2,220 calls from the same window. Summing those values is valid only if the workers own disjoint request streams; if they can retry the same request, the sum double-counts usage. I keep a request id or provider sequence in the ledger, record the key version used for the attempt, and reconcile the daily total against the upstream statement. When the numbers disagree, the alert should say “attribution uncertain” instead of pretending the remaining budget is precise. That wording changes the operator's next action: inspect the ledger and retry path, rather than rotate a key blindly. It also protects the billing conversation with a property owner, because the team can show which observation, credential version, and window produced the number.
No guesswork.
type Metrics = {
setGauge(name: string, value: number, labels: Record<string, string>): void;
increment(name: string, value: number, labels: Record<string, string>): void;
};
function recordBudget(metrics: Metrics, snapshot: BudgetSnapshot) {
const { remaining, ratio } = headroom(snapshot);
const labels = {
service: "property-api",
environment: "production",
scope: snapshot.propertyId,
};
metrics.setGauge("api_budget_headroom_calls", remaining, labels);
metrics.setGauge("api_budget_headroom_ratio", ratio, labels);
return { remaining, ratio };
}
The alert expression should include a freshness test. An old value can look healthy after the exporter stopped. For example, page only when the ratio is under 0.15 and the sample age is under ten minutes; send a separate stale-metric alert when the age exceeds ten minutes. Your mileage may vary because a batch-heavy property portfolio may need a five-minute window, while a low-volume portfolio may tolerate thirty.
A scheduled key rotation that does not race billing attribution
Rotation is a two-phase operation: create the new credential, then switch consumers, then revoke the old credential after a grace period. The budget job should not rotate merely because a single request failed. It should require a fresh snapshot, a policy threshold, and an idempotency record.
type CredentialStore = {
readActive(propertyId: string): Promise<{ version: string; secret: string }>;
writePending(propertyId: string, version: string, secret: string): Promise<void>;
promote(propertyId: string, version: string): Promise<void>;
revoke(propertyId: string, version: string): Promise<void>;
};
type UsageClient = {
readBudget(propertyId: string): Promise<BudgetSnapshot>;
createKey(propertyId: string): Promise<{ version: string; secret: string }>;
};
export async function rotateIfNeeded(
propertyId: string,
usage: UsageClient,
credentials: CredentialStore,
now = Date.now(),
) {
const snapshot = await usage.readBudget(propertyId);
const { ratio } = headroom(snapshot);
const windowEnd = Date.parse(snapshot.windowEndsAt);
const fresh = now - Date.parse(snapshot.observedAt) < 10 * 60 * 1000;
if (!fresh || ratio >= 0.15 || windowEnd <= now) {
return { rotated: false, reason: "policy_not_met" };
}
const next = await usage.createKey(propertyId);
await credentials.writePending(propertyId, next.version, next.secret);
await credentials.promote(propertyId, next.version);
// Revoke only after consumers have loaded the promoted version.
const previous = await credentials.readActive(propertyId);
if (previous.version !== next.version) {
await credentials.revoke(propertyId, previous.version);
}
return { rotated: true, version: next.version };
}
Run this function from a scheduler with a distributed lock keyed by propertyId. The lock duration must exceed the normal API latency but be shorter than the job interval. Store the last successful usage sequence and rotation decision, so a retried invocation cannot create a chain of keys. Secrets belong in a managed secret store with access logging and short-lived access where possible; OWASP's guidance covers the operational controls better than an ad hoc .env file.
I once treated a key version as a deployment detail and lost the link between a retry and its billing scope. The fix was boring: include keyVersion in the ledger row and reject an update whose usageSequence is lower than the stored value. Boring is good here.
When is this budget-headroom pattern the wrong choice?
It is not suitable when the upstream provider reports only account-wide usage but your invoices depend on per-property attribution. In that case, keep a gateway or an event ledger that sees every request before selecting a key. It is also a poor fit for a high-volume system where a ten-minute exporter lag can exhaust the budget; use streaming usage events and make the scheduler a backstop.
Stick with a provider-managed rotation workflow when compliance requires its audit trail or when your service cannot safely hold a credential-generation permission. The local metric still has value, but it should observe that workflow rather than replace it. I'm not sure any single threshold fits every portfolio, so I start with 15%, replay the last two billing windows, and tune from observed false pages.
The decision rule is simple: choose the smallest architecture that preserves attribution through retries, exposes fresh headroom, and can prove which credential version paid for each call. Ship that slice weekly. Add a gateway only when the ledger shows why it earns its operational cost.
Top comments (0)