DEV Community

RhettFletcher9678
RhettFletcher9678

Posted on

Implementing Containment for One Scoped API Permission Error in Node.js

#node #sre #security

A permission failure isolated to one Node.js path usually means the credential reached a capability boundary, not that authentication is broken. During a leaked-key drill, treat that failure as useful evidence: identify the exact operation, compare its required capability with the key's effective grant, and replace only the affected credential. Do not widen the key until the mismatch is proved. That preserves the main safety property of scoped credentials: one leak should have a bounded blast radius.

TL;DR: reproduce the failing operation with the same tenant and credential class, record a sanitized authorization decision, diff required versus effective capabilities, then rotate the exposed key through an idempotent runbook. A successful neighboring path proves very little beyond basic key validity.

Why does a scoped API permission error hit one code path?

Authentication answers who presented the credential. Authorization answers whether that identity may perform this operation on this resource. A key can pass the first check and still fail the second because the route needs a capability absent from its grant, the resource belongs to a different tenant, or a policy condition narrows the grant. The HTTP status alone cannot distinguish those cases.

The misleading signal is the healthy path beside it. Suppose a developer-tools account service accepts a scoped automation key for reading projects, while the leaked-key drill also invokes an endpoint that revokes active sessions. Reads continue to work; revocation fails. That pattern is consistent with least privilege. It is not evidence that the client library, network, or key parser is intermittently broken.

My operational rule is blunt: a neighboring 2xx response never authorizes the failing operation. I start at the authorization decision and work outward. This ordering matters during a drill because speculative scope expansion turns a contained credential into a broader incident.

Healthy is not authorized.

Build a capability diff before changing policy

Model the evidence as sets. The code path declares the capabilities it needs, while the authorization layer reports the effective capabilities for the presented principal and resource. The missing set is required - effective. Keep resource and tenant constraints alongside the set; a matching capability name does not override a tenant boundary.

Use a small offline probe to make that comparison repeatable. This Go program consumes a sanitized decision record, prints missing capabilities, and exits nonzero when the path cannot proceed. It never prints a token.

package main

import (
    "encoding/json"
    "fmt"
    "os"
    "sort"
)

type Decision struct {
    RequestID string   `json:"request_id"`
    Tenant    string   `json:"tenant"`
    Resource  string   `json:"resource"`
    Required  []string `json:"required"`
    Effective []string `json:"effective"`
}

func main() {
    var d Decision
    if err := json.NewDecoder(os.Stdin).Decode(&d); err != nil {
        fmt.Fprintln(os.Stderr, "invalid decision record:", err)
        os.Exit(2)
    }

    granted := make(map[string]struct{}, len(d.Effective))
    for _, capability := range d.Effective {
        granted[capability] = struct{}{}
    }

    missing := make([]string, 0)
    for _, capability := range d.Required {
        if _, ok := granted[capability]; !ok {
            missing = append(missing, capability)
        }
    }
    sort.Strings(missing)

    result := struct {
        RequestID string   `json:"request_id"`
        Tenant    string   `json:"tenant"`
        Resource  string   `json:"resource"`
        Missing   []string `json:"missing"`
    }{d.RequestID, d.Tenant, d.Resource, missing}

    if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
        fmt.Fprintln(os.Stderr, "encode result:", err)
        os.Exit(2)
    }
    if len(missing) != 0 {
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

For the drill, feed it a redacted fixture derived from the authorization decision rather than a production key. This test locks down the important boundary: reading projects does not imply permission to revoke sessions.

package main

import (
    "reflect"
    "testing"
)

func TestMissingCapabilities(t *testing.T) {
    required := []string{"projects:read", "sessions:revoke"}
    effective := map[string]struct{}{"projects:read": {}}

    var got []string
    for _, capability := range required {
        if _, ok := effective[capability]; !ok {
            got = append(got, capability)
        }
    }

    want := []string{"sessions:revoke"}
    if !reflect.DeepEqual(got, want) {
        t.Fatalf("missing = %v, want %v", got, want)
    }
}
Enter fullscreen mode Exit fullscreen mode

Two details prevent false fixes. First, capability names must come from the service's policy contract, not from guessed route names. Second, capture the effective decision at evaluation time. A dashboard showing the key's intended grant may lag deployment, omit conditions, or describe a different tenant binding.

The trade-off is extra policy metadata in the diagnostic path. That cost is justified for privileged operations and cross-tenant systems, where a vague 403 can otherwise send an operator toward a dangerously broad grant. It is less suitable for a tiny, single-purpose service with one fixed role and no resource conditions; there, a reviewed static capability manifest and a focused integration test may provide enough evidence with less machinery. The diff also has a hard limitation: it explains only the decision record it receives. It cannot prove that the record came from the same policy revision, tenant, and resource as the failed request, so those fields must be correlated before anyone edits a grant.

Run the drill as a bounded incident

Start with containment. Mark the leaked credential identifier as compromised, stop new work that uses it, and preserve request identifiers needed to find its use. Never put the secret value in a ticket, trace attribute, command history, or test fixture. OWASP recommends centralizing secrets, applying least privilege, automating rotation, and logging secret-management events without logging the secrets themselves.

Then reproduce exactly one failing operation in a controlled environment. Hold four dimensions constant: credential class, tenant, resource type, and operation. Compare it with one known-good operation using the same dimensions. If more than one variable changes, the comparison cannot isolate the policy boundary.

The useful evidence is compact:

  1. A correlation identifier and timestamp.
  2. The principal or key identifier, hashed or otherwise non-secret.
  3. The tenant and resource identifiers in their approved redacted form.
  4. Required capabilities, effective capabilities, and the policy revision.
  5. The final allow or deny result and a stable reason code.

Do not log the bearer credential. RFC 6750 describes bearer tokens as usable by any party in possession of them, so disclosure in observability data creates another exposure surface. Redaction should occur before serialization, not downstream in a log shipper.

Containment comes first.

Next, revoke and replace the exposed credential according to the runbook. Make the operation idempotent: retrying a revocation must leave the credential revoked, and retrying deployment of the replacement must converge on one active configuration. Record states such as identified, revoked, replacement_issued, deployed, and verified. A job may be delivered twice. The second delivery must not mint a second replacement or roll the service back to an older secret.

Only after containment should you change policy. If the path legitimately requires sessions:revoke, issue a separate narrowly scoped credential or adjust the workload role through review. If the path should never revoke sessions, the deny is correct; fix the caller or split the workflow. Passing the drill does not mean making every call succeed.

Make capability discovery observable without leaking credentials

A production authorization record should support diagnosis while remaining useless as a credential. Prefer structured fields and stable reason codes over prose. For example, missing_capability, tenant_mismatch, resource_condition_failed, and credential_revoked let an operator choose the next check without exposing policy internals to an untrusted client.

Keep client responses deliberately less detailed than internal decision logs. RFC 6750 permits an insufficient-scope response to indicate the scope required for the requested resource, but revealing resource existence or policy shape may be inappropriate in some threat models. Decide that boundary explicitly. Internal operators can correlate a public error with a protected decision record through a request ID.

Metrics need similar restraint. Count denies by operation, credential class, reason code, and policy revision. Avoid labels containing token values, unbounded resource IDs, or full URLs; they leak data and create high-cardinality telemetry. Alert on a change in the deny ratio for a stable operation, not on every expected least-privilege denial. During the drill, verify both sides: the intended forbidden action is denied, and the replacement credential performs only its documented job.

This is also where scheduling discipline matters. Rotation jobs need a deduplication key tied to the compromised credential identifier and drill run, durable state transitions, bounded retries, and an owner-visible terminal failure. A cron expression is not evidence that rotation happened. The verification event is.

Retries will happen.

Know when this method does not apply

A capability diff is the wrong first tool when all paths fail, the credential cannot be parsed, TLS or DNS fails before authorization, or the identity provider is unavailable. Those symptoms point earlier in the request chain. Likewise, an intermittent deny across identical inputs may require checking policy propagation, clock-dependent conditions, cache keys, or replica consistency rather than adding a grant.

Stop the drill if evidence shows the credential was used outside the expected tenant or time window. That is no longer a narrow capability exercise; it is an incident requiring the organization's response process, broader log preservation, and an expanded impact assessment.

The decision rule remains small: widen nothing until the required operation, effective grant, resource boundary, and policy revision explain the deny. Then replace the exposed credential, verify the narrow workflow, and prove that prohibited operations still fail. A clean leaked-key drill measures containment, not convenience.

References

Sources

Top comments (0)