LiraelVex6403

Posted on Sep 18

Per-Tenant API Keys — Verify Actual Capability at Startup Before Traffic Arrives

#security #apikeys #go #sre

Use a boot-time capability probe. Before the process accepts its first request, ask the issuer what the API key in front of it is actually permitted to do, compare that answer against a declared manifest of the scopes the service needs, and refuse to report ready when the two disagree. That is the least complex control that stops a mis-scoped tenant credential from being discovered by real traffic, and it is the one I'd put in first — everything below is about where the self-check earns its keep, and where it will page you for nothing.

The page that fires is the one you cannot act on

Picture the alert an on-call engineer gets on a multi-tenant grading platform at the end of a school district's marking window: tenant_sync_error_ratio > 0.05 for 10m, one district, no other symptoms. The dashboard shows a wall of 403s against a downstream roster provider, all of them for the same tenant, all of them starting eleven minutes after a routine deploy.

Nothing is down. The service is healthy, the pods are green, the queue is draining. The credential issued for that district simply doesn't carry grades:write, because whoever provisioned the tenant copied a template that only had read scopes, and no code path in the deploy ever asked the question.

That is the shape of the problem. The page fires after the blast, measured in failed student-grade writes, and by then the on-call engineer's only lever is to roll back a deploy that wasn't the cause. The interesting design question isn't how to alert on 403 ratios faster. It's why a process was allowed to become ready while holding a credential it had never verified.

What can this API key actually do before real traffic arrives?

There are three honest answers, and they differ mainly in what the issuer is willing to tell you.

The cleanest is introspection. RFC 7662 defines a token introspection endpoint that returns active, the granted scope as a space-delimited string, and whatever extra claims the issuer chooses to publish — for a per-tenant key that usually includes the tenant binding itself. One round trip, no side effects, and the answer is authoritative because it comes from the thing that minted the credential rather than from a guess encoded in config.

The second is a self-review endpoint: the caller asks "what am I allowed to do", and the authorization layer enumerates it. Kubernetes has shipped this pattern for years as SelfSubjectRulesReview, which is what kubectl auth can-i --list is really calling. It is the same idea as introspection with the polarity reversed — instead of describing a token, the server describes the subject's effective permissions.

The third is synthetic probing: make one deliberately harmless call per capability and read the status code. It works everywhere, which is its only real virtue. It also costs one request per scope per tenant per boot, and on any surface where "harmless" is a judgement call rather than a documented guarantee, it's the wrong instrument.

Whichever answer the issuer supports, the shape of the check is the same. Declare what you need, ask what you have, and treat the difference as a startup failure rather than a runtime surprise.

// Scopes this grader service refuses to serve without, per tenant.
// true = hard requirement; false = optional, degrade the feature instead.
var manifest = map[string]bool{
    "roster:read":    true,
    "grades:write":   true,
    "reports:export": false,
}

// RFC 7662 introspection response, trimmed to the fields the gate reads.
type grant struct {
    Active   bool   `json:"active"`
    Scope    string `json:"scope"` // space-delimited
    TenantID string `json:"tenant_id"`
    Exp      int64  `json:"exp"`
}

func introspect(ctx context.Context, tenantKey string) (grant, error) {
    form := url.Values{"token": {tenantKey}, "token_type_hint": {"access_token"}}
    req, err := http.NewRequestWithContext(ctx, http.MethodPost,
        os.Getenv("INTROSPECTION_URL"), strings.NewReader(form.Encode()))
    if err != nil {
        return grant{}, err
    }
    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
    req.Header.Set("Authorization", "Bearer "+os.Getenv("PROBE_CLIENT_TOKEN"))

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return grant{}, err
    }
    defer resp.Body.Close()

    var g grant
    if err := json.NewDecoder(resp.Body).Decode(&g); err != nil {
        return grant{}, err
    }
    return g, nil
}

Working backwards to the signal that should have fired earlier

A boot probe is still late. The credential was mis-scoped at issue time, which may have been three weeks before the deploy that exposed it, so the earliest honest signal lives in the issue-and-revoke path, not in the application.

The instrumentation change is small and it is mostly about what you record rather than what you measure. At issue time, write the requested scope set and the granted scope set as separate fields on the same structured event. At boot, emit the granted set again, keyed by tenant. At revoke, emit the set that was withdrawn. Three events with the same schema turn an unanswerable question — "which tenants hold more than they should" — into a join you can run without touching production.

Then the gate itself becomes a diff, and the diff becomes a gauge:

// Called once per tenant credential during startup, before the listener binds.
func gate(ctx context.Context, tenantID, tenantKey string) error {
    g, err := introspect(ctx, tenantKey)
    if err != nil {
        probeErrors.WithLabelValues(tenantID).Inc()
        return err // issuer unreachable is a different decision; see below
    }
    if !g.Active || g.TenantID != tenantID {
        return fmt.Errorf("credential is not bound to tenant %s", tenantID)
    }

    granted := map[string]bool{}
    for _, s := range strings.Fields(g.Scope) {
        granted[s] = true
    }

    var missing, undeclared []string
    for scope, hard := range manifest {
        if hard && !granted[scope] {
            missing = append(missing, scope)
        }
    }
    for scope := range granted {
        if _, declared := manifest[scope]; !declared {
            undeclared = append(undeclared, scope) // blast radius wider than the manifest
        }
    }
    scopeDrift.WithLabelValues(tenantID).Set(float64(len(missing) + len(undeclared)))

    if len(missing) > 0 {
        return fmt.Errorf("tenant %s is missing hard scopes %v", tenantID, missing)
    }
    if len(undeclared) > 0 {
        log.Printf("tenant %s holds %d undeclared scopes: %v", tenantID, len(undeclared), undeclared)
    }
    return nil
}

The SLI worth carrying on a dashboard is not the 403 ratio. It's the fraction of live tenant credentials whose granted scope set equals the declared manifest, and unlike an error ratio it can sit at 100% for months and still be meaningful, because every drop has exactly one cause. Set the objective wherever your provisioning discipline actually lives; a platform issuing keys through a reviewed pipeline can hold 100% and alert on any single deviation, while one that still allows manual issuance should pick a number it can defend and then work the number down.

Capacity is the part teams skip. A district-scale edtech tenant list of 1,200 credentials, times forty replicas, is 48,000 introspection calls in the sixty seconds after a rolling deploy, which is a thundering herd aimed at the one service that must never be the reason a deploy stalls. Two mitigations are boring and sufficient: cache the grant per credential with a short TTL shared across replicas, and probe only the credentials the replica will actually use rather than the whole tenant list.

Where the check belongs is a buy-versus-build question, and the honest table has four rows rather than two:

Control point	Catches	Ongoing cost	Wrong choice when
Policy test at key issuance	Over-broad grants, template drift	One test suite in the provisioning pipeline	Keys can be minted outside the pipeline
Gateway-level scope enforcement	Requests that exceed the grant	A gateway on the critical path, plus its own SLO	You don't terminate the traffic yourself
In-process startup probe	Mis-scoped credentials, wrong tenant binding	One round trip per credential per boot	Processes run for weeks between restarts
Per-request authorization	Everything, at the moment it matters	Latency and a hard dependency on the issuer	Your budget for added p99 is already spent

Nobody picks one. Issuance tests plus a startup probe covers the failure that produced the 3 a.m. page, and per-request authorization stays where it always was, doing the job the probe was never meant to do.

Where a startup self-check is the wrong control

The catch is time-of-check versus time-of-use. A probe proves what the credential could do at boot; it says nothing about what happens when an administrator narrows that scope forty minutes later, and a service that only checks at startup will keep assuming a capability it no longer holds until something restarts it. If your credentials rotate on a schedule measured in hours, a startup gate is not suitable on its own and you should stick with authorization decisions made per request, with the probe demoted to a deploy-time smoke test.

Some issuers don't offer introspection or a self-review endpoint at all, and publish only the scope names in documentation. That's a design boundary rather than a defect, and it pushes you back to synthetic probing or to reading a scope header off a cheap authenticated call — the way a X-OAuth-Scopes style response header is used to discover what a token carries after the fact.

There's also the recursion nobody enjoys: the probe needs its own credential to call introspection, and that credential can see every tenant. I'm not convinced there's a clean answer. Scoping the probe client to introspection-only and keeping it out of the workload's own secret material is the version I'd defend, but it does concentrate a capability in one place, which is exactly the property the whole exercise was meant to avoid.

One more, specific to how this gets wired. In Node.js the temptation is to run the probe inside the same handler that answers the orchestrator's readiness endpoint, which is convenient and wrong, because a slow issuer then looks identical to a slow application; run it once before the listener binds, cache the verdict, and let the readiness endpoint serve a boolean.

The false-positive cost of getting the threshold wrong

Fail-fast has a bill, and it's paid by the deploy. If the gate treats every scope as hard, one unreachable issuer during a rolling restart converts a partial degradation — a district that can't export reports — into forty pods in CrashLoopBackOff and a platform-wide incident that your own safety control caused.

So the threshold is not a number, it's a classification. Hard scopes are the ones where serving traffic would produce wrong data or a silent write failure; those fail closed. Soft scopes disable a feature, emit an event, and let the process start. And when the probe itself errors rather than returning a verdict, the safe default is to start with the last known-good grant if it's within TTL, and to fail closed only when there's nothing cached — because an issuer having a bad minute is a much more common event than a scope changing without anyone noticing.

The rule I'd write into the runbook: a readiness gate is only worth having when the incident it prevents is more expensive than the deploys it will block. For per-tenant keys in an edtech platform, where the failure mode is writing grades to the wrong place or not writing them at all, that math is easy. For a reporting sidecar it probably isn't, and your mileage may vary with how much of your tenant list is provisioned by hand.

Verify the scope, publish the drift, and keep the fail-fast surface as small as the blast radius you're actually defending.

DEV Community