DEV Community: Sanghyeok Yoon

LLM gateway on-call: the 4 incidents that actually happen

Sanghyeok Yoon — Mon, 05 Oct 2026 22:15:00 +0000

An LLM gateway is a small system with a big trust load. It's small enough for one on-call to own, and it holds your org's model credentials, its spend attribution and its audit trail. That combination shapes how you run it: conservative where the trust is, pragmatic everywhere else.

The runbook set I keep for one has four incidents, not forty. These are the ones that actually happen, and every runbook has the same four steps:

Detect: the metric or the report that tells you.
Assess: the one query that sizes it.
Act: the sequence, in order.
Close: the artifact you leave behind.

The last step is the one teams skip. An incident without its artifact is a memory, and the memory is the next incident.

First, decide what is allowed to page

Before any runbook, the pager needs rules, or every incident review starts with "we ignored the alert". Three severities with fixed meanings:

info: a forecast. No action needed; the owner glances at it. Goes to the team channel.
warn: act this week. Team channel plus the team lead.
crit: spend or integrity is at risk now. The platform on-call channel (page-worthy) plus a webhook into the incident system.

Two rules keep the pager trusted:

Fire on state entry, not state persistence. Crossing 95% of a budget fires once. Sitting at 96% doesn't fire again. A re-fire is allowed after a 24-hour cooldown, and only if the state got worse by a level.
One escalation, then stop. An unacknowledged crit escalates once, after 4 hours, then goes quiet. An alert that pages forever gets muted, and a muted crit is the incident.

Here's how my default thresholds split. Treat them as starting points to tune, not laws:

Signal	warn (this week)	crit (now)
Budget, per team or feature	95% of the hard limit	100% (hard) and 110% (overage)
Cost anomaly	a feature's 7-day cost ≥ 3× its 30-day average	a team's daily cost ≥ 5× its 30-day daily average, or one request over your per-request cap
Metering completeness	below the 99.9% SLO	below 99.5% for an hour
Provider breaker	open, and fail-over worked	open, and the alias has no secondary (users get 503s)
Metering bus lag (p95)	over 5 minutes	no page: dashboards must say "stale" instead
Credential age	past its rotation schedule	suspected leak: revoke first, investigate second

Now the four incidents.

IR-1: a provider degrades (the most common one)

Detect. The provider's circuit breaker opens, or your services start reporting 503 model_unavailable on an alias. The provider's status page is corroboration, not detection. Your breaker sees it first.

Assess. One query: request outcomes by provider over the last 15 minutes. One provider failing is a provider incident. Every provider failing at once is your gateway, and that's a different runbook.

Act. If the alias has a secondary binding, fail-over already happened. Your job is to confirm it: watch the share of responses marked X-LLM-Router: failed-over=1 (or metering events where routed_to != requested_binding), and check the cost delta of the secondary model. If the alias has no secondary, bind one in the model registry. That's a registry change, not a deploy, which is the whole point of routing through aliases.

Close. The record names the window, the fail-over share, the cost delta, and the provider's acknowledgment if one came. Add a note to that provider's availability history. Your provider mix is a portfolio, and you rebalance it on that evidence, not on loyalty.

IR-2: a cost anomaly (the one finance calls about)

Detect. The crit cost alert, or finance asking. Finance's question is the same alert with a slower detector.

Assess. Look at the per-feature, per-user and per-model views, in that order:

Feature first. A feature's spike is a workload change or a loop.
User second. A principal's spike is a human, or a script holding someone's grant.
Model third. A model's spike is a price change, or traffic that failed over onto an expensive secondary.

Act. It depends on what you found:

A loop. The feature's hard budget should already be denying. Confirm the denials counter is climbing for that feature, then tell the team.
A price change. Force a price refresh in the registry, re-forecast the budget at the new price, and warn whoever reconciles the invoice to expect the variance.
A fail-over cost. Review the binding. The secondary was the design; its price is the data.

Close. Name the cause class (loop, price, fail-over, one-off), the cost, and the budget state afterwards. If a budget should change, that goes through the normal budget workflow, not the incident. The incident doesn't grant budgets; it finds them.

IR-3: an auth failure (the one that looks like everything)

Detect. A wall of 401/403, a degraded identity platform, or "my token works in staging but not in prod."

Assess. This is why denial reasons deserve their own label instead of one unauthorized bucket. Split the denials by reason and the shape names the cause:

token_invalid spikes. A JWKS problem: the identity platform rotated keys out from under the gateway, or the JWKS cache is stuck.
token_expired across the whole org. A token-TTL misconfiguration, or a clock problem in the identity platform.
scope_missing or role_denied in one team. A grant change: someone's role changed, or a migration forgot a scope.

Act. For the JWKS shape, force a JWKS refresh through the gateway's admin path and check the identity platform's key state. For the TTL shape, the fix is upstream in the token mint; your job meanwhile is checking whether the affected services can degrade safely. For the grant shape, restore the grant through the normal access workflow, so the fix carries its own audit trail.

Close. Record the shape, the blast radius (which teams, which services), the recovery action, and whether anyone used break-glass. If they did, the break-glass audit line goes in the record. This is the incident that touches the trust load, so the record matters most here.

IR-4: metering integrity (the one users never see)

Detect. Metering completeness drops below the 99.9% SLO, the monthly reconciliation finds a gap, or bus lag alerts. Below 99.5% for an hour is an incident, because your books are now wrong.

Assess. Find where the events went missing. Missing from the bus is a transport loss. On the bus but missing from the rollups is an aggregation loss. Bus lag catches the first; completeness catches the sum. If lag is normal while completeness drops, look at the rollups.

Act. For a transport loss, follow the bus's own runbook while the gateway's bounded queue holds events, and watch completeness recover. For a rollup loss, recompute the rollups from the bus. The recompute is a job, and its run is itself an audit event.

One rule I don't bend: never back-fill metering from the provider's invoice. The invoice is the reconciliation's check, not metering's source. Back-fill from it and your books become the provider's books, with no team, feature or user attribution, and next month's reconciliation has two gaps instead of one.

Close. Record the window, the lowest completeness during it, the recompute if one ran, and the expected variance for month-end, so the finance close isn't a surprise.

Put the first query in the alert

The cheapest on-call improvement I know: write each runbook's assess query into the alert itself, so whoever gets paged at 3 a.m. doesn't have to remember it. A Prometheus sketch with the metric names used above. Thresholds come from the table; the for: durations are illustrative.

groups:
  - name: llm-gateway
    rules:
      # IR-1: breaker gauge, 1 = open in this sketch
      - alert: LLMProviderBreakerOpen
        expr: max by (provider) (llm_provider_breaker_state) == 1
        for: 2m
        labels: { severity: warn }   # raise to crit if the alias has no secondary
        annotations:
          first_query: 'sum by (provider, outcome) (increase(llm_requests_total[15m]))'

      # IR-4: the books are wrong
      - alert: LLMMeteringBooksWrong
        expr: llm_metering_completeness < 0.995
        for: 1h
        labels: { severity: crit }
        annotations:
          first_query: 'max_over_time(llm_metering_bus_lag_seconds[1h])'   # high = transport, normal = rollups

      - alert: LLMMeteringBelowSLO
        expr: llm_metering_completeness < 0.999
        for: 1h
        labels: { severity: warn }

      # IR-2 (cost) fires from the budget service on state entry, not from here.
      # IR-3 first query: sum by (reason) (increase(llm_requests_total{outcome="unauthorized"}[1h]))

The cost alerts deliberately live in the budget service, not in Prometheus. They come from rollups, they need the state-entry dedup, and they should carry the budget percent, the forecast and the top features in the webhook payload, so the person who gets it can act without opening five dashboards.

Two things that never page but will bite

Credential drift. Every credential the gateway holds rotates on a schedule (90 days is my default for provider credentials) through a two-secret swap: mint the new version, verify it with a tiny tagged probe call, flip the binding, and only then retire the old one. On a suspected leak, the order flips: revoke first, investigate second. The schedule is also what bounds a leak's exposure window: a 90-day rotation means a 90-day exposure, not a forever one.

The degrade-mode direct keys. If the gateway is down, some services that can't afford the gap may keep a scoped, audited direct-provider key. Fine, but every such key needs a named owner and a deletion review date. A direct key without an owner is a static key with a runbook around it.

The cheat sheet

IR-1, provider: detect with the breaker; first query is outcomes by provider over 15 minutes; close with the window, fail-over share and cost delta.
IR-2, cost: detect with the crit cost alert; assess feature, then user, then model; close with the cause class, cost and budget state.
IR-3, auth: detect the 401/403 wall; first query is denials by reason; close with the shape, blast radius and any break-glass use.
IR-4, metering: page below 99.5% completeness for an hour; first query is bus lag; close with the window, floor, recompute and month-end variance.

This is a condensed version of Chapter 7 of my book The AI Gateway Playbook, which also covers the audit-log retention contract, keyless credential rotation, the metrics catalog behind these alerts, and the deployment and migration checklists. Related reading: MCP at 90+ tools: the catalog breaks before the server does.

Disclosure: this article and the book were drafted with AI assistance from my own experience designing and running this kind of platform. All examples are generic.

MCP at 90+ tools: the catalog breaks before the server does

Sanghyeok Yoon — Sun, 04 Oct 2026 20:08:21 +0000

Exposing one internal tool over the Model Context Protocol is a weekend project. Exposing ninety of them to humans and agents, without turning the MCP endpoint into a credential dump and the audit log into a fire hazard, is a platform problem.

This post is the short version of how I'd design that platform. The one-line answer: treat the MCP server as a tool broker in the same family as your LLM gateway. Same identity rules (no static keys), same metering shape (every invocation is an event), same audit discipline. Plus two things the LLM path doesn't need: an envelope for stored third-party credentials, and a PII masking step for what flows back.

1. Organize the catalog by category, not by tool

At 90+ tools, the first failures are organizational, not performance. A few rules keep the catalog sane:

Naming is a contract: {category}.{verb}_{object}, e.g. issues.create_ticket, mail.send_draft, ci.get_pipeline_logs. Renaming a tool is a breaking change with a deprecation window, not a silent edit.
One tool = one unit of permission, audit, and test. A "do everything in the issue tracker" tool is a permission you can't grant safely, an audit row you can't read, and a test you can't write. Split until each tool is all three.
Version in the schema, not the name (issues.create_ticket@v2), and serve all live versions during the deprecation window.
Grant tool groups, not the universe. The server barely notices 94 schemas. The client does: an agent that loads every schema can burn 30k+ tokens of context on tool definitions alone. Clients request a group (ci, mail), and the group is the grant unit.

2. Tool design patterns that keep agents (and auditors) sane

A tool definition worth reviewing looks roughly like this:

id: issues.create_ticket@v2
input_schema:
  type: object
  required: [project, summary]
  properties:
    project: { type: string }
    summary: { type: string, maxLength: 256 }
errors: [permission_denied, validation_failed, upstream_error, rate_limited]
pii: { inputs: [description], policy: mask-on-store }
permissions: { min_role: contributor, write: true, idempotency: client_key }

The patterns behind it:

Idempotency keys on retryable writes. A retried create with the same key returns the same ticket, not a duplicate. Make it a registration-time lint for write tools.
Errors are part of the contract. Normalize upstream failures into a small taxonomy with a retryable flag (and Retry-After when you have it). Agents retry on the flag; humans read the code.
Reads are cheap, writes are loud. Writes need a minimum role and a scope (project, space, folder). The audit event for a write carries the full (masked) input; for a read, just the shape.
Bound every list. Default limit 50, server cap 200, cursor pagination. An unbounded list tool is a context-window bomb.
Return what was asked, not what you know. Enrichment ("related tickets") is a second tool the agent can choose to call. Tools that over-return are tools agents can't plan around.

3. Auth: OAuth 2.1 + PKCE, with a few non-negotiables

The happy path is simple: the client runs the authorization-code flow with PKCE against your identity platform, gets a short-lived access token, and presents it on the MCP endpoint. The server verifies it, resolves the principal, and checks tool grants. What makes it actually safe:

PKCE for every client type, including IDEs and agent runtimes. MCP clients usually can't hold a secret.
Resource indicators. The token's audience is the MCP server. A valid token minted for some other service gets rejected.
Dynamic client registration is not a grant. DCR keeps "bring your own agent runtime" possible (redirect URI allowlist, auth method checks), but tool access still flows through an explicit grant.
One token-time posture for the org. For example: 15-minute access tokens for interactive clients, 1 hour for workloads, no refresh tokens for agents (they re-authenticate through their workload identity).
Scopes per tool group (mcp:ci:read, mcp:issues:write), so the consent screen is readable instead of a wall of 94 strings.
The client's token never travels to the SaaS. The broker calls downstream systems with its own credential and carries the principal in-band for audit and metering. Where a downstream system has a clean per-user delegation model, use that instead; it's the stronger design.

4. Stored secrets: envelope encryption, and why it isn't the control

The broker has to store credentials for the systems behind it. Do it with a standard envelope:

KMS key-encryption key (never leaves KMS, auto-rotated)
  └── data-encryption key per category (AES-256, wrapped by the KEK)
        └── secret value: AES-256-GCM(DEK, value), fresh 96-bit nonce per write

Per-category DEKs keep the blast radius at one category. Per-user tokens get a per-user DEK under the category DEK.
Rotation is cheap by design: the KMS rotates the KEK; a scheduled job re-wraps DEKs. Neither touches the secret ciphertexts.
But encryption at rest only answers "a DB leak is not a credential leak." The real control is that no role (except break-glass) can read a secret. The broker uses it in-process for one call, zeroes the buffer, and the audit event records the call and the principal, never the credential. A portal should show "credential present, last rotated N days ago", nothing more.

5. Mask on the way back

The LLM gateway can stay payload-blind. The MCP path can't: the tool result is the product. So results pass through a PII masking step before they reach the client and before anything is stored, and the per-tool pii field in the catalog says which inputs and outputs need it. This is also why write audits store masked inputs, not raw ones.

The checklist version

Broker, not proxy: no static keys, every invocation metered and audited
Category naming, one-tool-one-permission, schema versioning, group grants
Idempotency, error taxonomy, bounded lists, no over-returning
OAuth 2.1 + PKCE, audience-bound tokens, DCR ≠ grant, group scopes
Envelope encryption with per-category DEKs, secrets used and never read
PII masking on results and on stored audit inputs

This is a condensed version of Chapter 5 of my book The AI Gateway Playbook, which also covers the LLM gateway itself: model registry, keyless auth and RBAC, quotas and budgets, a RAG team assistant, and operations runbooks. Free sample on the Leanpub page.

Disclosure: this article and the book were drafted with AI assistance from my own experience designing and running this kind of platform. All examples are generic.

The LLM Gateway I Put in Production: 4 Decisions That Actually Mattered

Sanghyeok Yoon — Sun, 04 Oct 2026 11:43:18 +0000

Your team is wiring services directly to provider APIs. Every app has its own API key, its own retry logic, its own hardcoded model name. Nobody can answer "how much are we spending on LLM this month," and when a model gets deprecated, you find out through a 500 error in production.

I've been working on LLM infrastructure for a few years now, and the pattern I keep coming back to is an in-house LLM gateway: a stateless proxy and a small control plane. Not an "AI platform" - just a boring, dependable layer. Here are the four decisions that actually mattered when putting one in production.

The shape

flowchart LR
    subgraph Clients
        A[Service A<br/>team: payments]
        B[Service B<br/>team: growth]
        H[Developer<br/>IDE / CLI]
    end
    subgraph GW["LLM Gateway (stateless)"]
        AUTH[Auth: JWT, RBAC]
        ROUTE[Routing: alias, quotas, failover]
        METER[Metering: usage, cost]
    end
    subgraph CP["Control plane (async)"]
        REG[Model registry: aliases, prices]
        Q[Quotas / budgets]
    end
    subgraph Providers
        PA[Provider A]
        PB[Provider B]
    end
    A -->|short-lived JWT| AUTH
    B -->|short-lived JWT| AUTH
    H -->|user token, OBO| AUTH
    AUTH --> ROUTE --> PA
    ROUTE --> PB
    ROUTE -.-> REG
    PA -->|usage| METER
    PB -->|usage| METER
    METER --> BUS[(message bus)] --> ROLL[(rollups)] --> DASH[Dashboards + alerts]

Two planes, four rules:

One control plane, one data plane. Everything a human changes (registry, roles, budgets, prices) lives in the control plane. The proxy is stateless and only reads control-plane state.
Everything is metered. If a request does not produce a metering event, it did not happen. The metering event is as real as the HTTP response.
No secrets in requests. The only credentials on the wire are short-lived identity tokens. Provider credentials never leave the gateway's secret view.
Boring technology. One language, one message format, one stream. The gateway's job is to be the least interesting component in your architecture.

The load path: clients → auth → routing → provider → stream back → metering → message bus → rollups → dashboards → alerts. The control plane is never in the hot path; it writes state, and the gateway picks up changes on a short TTL (1-5 seconds is fine).

Decision 1: A model registry that syncs itself

Lock-in happens quietly. One day a provider deprecates a model name and your services die. The fix is a model registry with aliases:

Apps talk to aliases (chat-fast, chat-reasoning, embeddings-1k), not provider model IDs.
A sync job polls each provider's /models endpoint and updates availability and pricing.
When a provider model disappears, the alias routes to a fallback model instead of failing.
Price updates flow through the same sync, so your cost dashboard is never stale.

Two tables, one scheduled job. No microservice forest.

Decision 2: Auth that works for services, humans, and agents

Services get short-lived JWTs from your identity platform, verified against a JWKS cache.
Humans (developers in IDEs or CLIs) get on-behalf-of tokens, so you can attribute usage to people, not shared service accounts.
Agents get service tokens with narrower scope than the services they call.
Provider credentials are resolved inside the gateway via managed identities - no API keys in your config files.

Fail closed on unknown alias, fail closed on invalid token. Quota counters are best-effort under high load (over-admit and alert); policies fail closed.

Decision 3: Cost tracking where the metering event is the source of truth

Every request produces a usage object: tokens in/out, model, alias, team, user.
Usage events go to a durable message bus; rollup jobs aggregate them into per-model / per-tenant / per-user tables.
Budgets are counters (Redis) plus policies (SQL), with alerts at 80% and 100% of monthly budget per team and per model.
The rollups DB can lag. It cannot lose. Recompute from the bus.

Once this is in place, "how much does this feature cost?" stops being a product meeting and becomes a dashboard question.

Decision 4: MCP as the interface for your AI tools

If agents or tools will talk to models, they should talk through the same gateway, with the Model Context Protocol (MCP) as the stable interface. One endpoint, any provider behind it. The gateway becomes the only place in your org that knows which provider is actually serving your traffic today.

What v1 looks like

One stateless service plus scheduled sync jobs. No orchestration framework, no second database. Everything else (portal, per-user budgets, fine-grained failover) can be added later without breaking the seams.

I wrote everything I learned into a book - reference architecture, configuration walkthroughs, an operations runbook, and a production launch checklist: The AI Gateway Playbook (32,000 words, $19, PDF + diagrams). Proceeds keep this kind of writing going.

If you're building an LLM gateway (or you built the one that's a mess of API keys and hardcoded model names), what's the hardest production problem you hit? I'd love to hear about it.