"The 45-Hour Silent 403: How I Learned to Actually Monitor LLM API Keys"

On September 18, 2026, 19 of my customer API keys silently returned 403 for 45 hours.

No alert fired. No email went out. The gateway admin dashboard looked perfectly healthy — balance was fine. I only found out because a customer opened a support ticket asking why their integration was dead.

That's the moment I realized my "monitoring" was a lie.

This post is the incident write-up and the five small scripts I built afterward so it never happens again. They're gateway-agnostic (One API / New API / LiteLLM) and plain Python + SQL — no SaaS, no lock-in.

---

The lie I was telling myself

I run a self-hosted LLM gateway (One API) serving ~20 paying customers. My "health check" was a cronjob that queried the admin account balance once an hour.

Balance-checking catches one failure mode: "I'm about to run out of credits." It completely misses another, far more common one: a key that's been silently revoked, expired, or rate-limited and now returns 403 on every single call.

The balance was fine. The keys were corpses. The gateway happily kept routing traffic to dead keys, customers got 403s, and the only signal was human complaints.

Root cause, in one sentence

I was monitoring the account, not the keys.

A pool can look alive while every key in it is silently failing. You have to actually probe each key.

The fix: 5 scripts

1. oneapi_pool_monitor.py — actually ping every key

This is the one that would have caught the incident. It loops over every key (not just the admin balance) and fires a tiny completion request. Any non-200 (especially a silent 403) triggers an alert.

# pseudocode of the core loop
for key in list_customer_keys():
    try:
        r = probe(key, model="gpt-4o-mini", prompt="ping")
        if r.status != 200:
            alert(f"KEY {key} unhealthy: HTTP {r.status}")
    except Exception as e:
        alert(f"KEY {key} threw: {e}")

Key design point: it pings with the cheapest possible model and a 1-token prompt so the health check itself costs fractions of a cent. And — important — it has a n_cust == 0 guard so an empty result doesn't silently "pass".

2. customer_quota_alert.py — tell the customer before they're blocked

When a customer's balance drops to <=10%, email them (not just you) with a top-up link. Surprise bill-shock is how you lose customers.

if remain_pct <= 0.10:
    send_email(customer.email,
               "Your LLM quota is low",
               f"You have {remain_pct:.0%} left. Top up: {BUY_LINK}")

3. llm_billing_calculator.py — ratio → cost → price

If you resell LLM access, you set a gateway ratio (e.g. 1.3× upstream). This turns that ratio into actual cost, margin, and a suggested retail price at a target margin:

# suggested retail at a target gross margin
def suggested_price(upstream_cost, margin=0.7):
    if upstream_cost <= 0:
        raise ValueError("fill in real upstream cost first")
    return upstream_cost / (1 - margin)

4. usage_sql_examples.sql — per-customer reports

Copy-paste queries for per-customer token usage, remaining quota %, and low-quota customers — the kind of report you need when a customer asks "why am I throttled?"

SELECT t.name,
       t.used_quota,
       t.remain_quota,
       CASE WHEN (t.used_quota + t.remain_quota) > 0
            THEN t.remain_quota * 1.0 / (t.used_quota + t.remain_quota)
            ELSE 1 END AS remain_pct
FROM tokens t
WHERE t.status = 1
ORDER BY remain_pct ASC;

5. grafana_cost_dashboard.json — drop-in spend panel

A Grafana dashboard (import-ready) showing daily spend and quota burn, with a ${pool_user_id} template variable so it maps to your setup, not mine.

What I'd tell past-me

1. Probe keys, not balances. A single tiny completion call per key per cycle is cheap and catches the silent failures.
2. Alert on empty results too. n_cust == 0 should be its own alarm, not a green checkmark.
3. Tell the customer, not just yourself. Low-quota emails to the customer prevent the "why did I get blocked" ticket.
4. Make health-check cost near-zero. Cheapest model + 1 token. Otherwise you'll disable it to save money and be back to square one.

The full templates

I cleaned these five scripts up into open, documented templates — with a README, the SQL, and the Grafana dashboard — as a one-time pack:

👉 LLM Ops Pack — self-hosted LLM cost & key monitoring templates

Plain Python + SQL, works with One API / New API / LiteLLM. Buy once, keep forever, free updates.

If you've lived through a similar silent outage, I'd genuinely like to hear how you caught it — drop a comment.