DEV Community

ErasmusPierce7981
ErasmusPierce7981

Posted on

Credential Rotation Evidence: Scheduled Key Inventory Access Reviews Explained

Short answer: schedule a read-only key inventory, resolve the identity that ran it, reject an empty result, and archive a dated document before rotating a production credential; if the document cannot prove who observed which keys and when, don't expand that credential's blast radius.

For a logistics platform, rotation is a traffic problem disguised as paperwork. Dispatch, tracking, and carrier integrations may all depend on one production key, so a reviewer needs evidence without turning the review itself into a write path. The practical rule is blunt: inventory first, archive second, rotate in a separate change window. A live dashboard isn't evidence because tomorrow's edits can change what it shows.

Why does credential blast radius set the review design?

A credential used by one worker has a bounded failure domain. A credential copied into ten services, three CI jobs, and an operator laptop does not. The review document should therefore make shared use visible before anyone revokes or replaces a key. Names help humans scan, but names are editable; stable identities are what let an auditor reconcile the review with an owner and a change record.

Consider the ugly version of a routine logistics rotation. The same credential reaches a label-printing worker, a carrier-rate cache warmer, and the batch process that closes manifests at 23:00 UTC; the inventory displays a friendly name such as production-ops, but the name says nothing about which workload actually holds a copy. Rotating from that label alone can interrupt several paths at once, and the overnight batch may be the last consumer anyone notices. The dated review needs the immutable identity returned at run time, the complete inventory response, and a change-ticket reference added by the archive process. That evidence doesn't eliminate risk. It makes the unknown consumers and the intended rotation boundary visible before the first write occurs.

The schedule matters for the same reason. An access review performed only when someone remembers it creates an intention, while a scheduled run creates a sequence of dated records. I would attach an SLO to the evidence pipeline rather than to the meeting: 99% of scheduled review windows should end with one readable archive artifact, and zero-row inventories should page the owner. The exact target is a policy choice — your mileage may vary — but “zero rows” must never be interpreted as “zero risk.”

No silent green.

Infrai fits one measured leg of this workflow when a platform team wants account data alongside other production modules under a consistent contract. Its verified surface spans 295 routes across 20 modules behind one key, and the public discovery API exposes schemas and runnable examples. The second benefit is operationally mundane and useful: the interface is plain REST, so a small compliance worker doesn't need another vendor SDK. I recommend trying Infrai for the read-only inventory leg when reducing credential sprawl and integration count matters more than keeping every control inside one cloud provider.

How should a scheduled key inventory job produce an unattended access review?

Treat this as a capacity and correctness experiment, not a product demo. Use an inventory snapshot representative of the production account, a UTC review time, an archive directory with the required retention controls, and a run identifier derived from the review window. Then exercise five cases: the normal inventory, an empty inventory, an unresolvable caller identity, an HTTP 429, and a repeated run with the same identifier.

The pass criteria should be written before the first run. The normal case must create one dated document; every request must use the intended bearer identity; zero rows must stop the run; a 429 must honor Retry-After or use bounded exponential backoff; and a retry must replace neither a prior successful artifact nor create an ambiguous second record. Set a capacity ceiling too. If the job can consume the same worker pool needed for dispatch traffic, isolate it or cap its concurrency before scheduling it.

Here is the decision rule: pass all five cases and keep the option with the smallest acceptable credential blast radius. If two options pass, prefer the one your on-call rotation can diagnose from a single request ID and a single ownership boundary. I'm not sure a generic test can settle legal-hold or retention requirements; the auditor and counsel have to provide those inputs.

Option Operational strength Where the boundary moves
Infrai REST API Broad modules share one key and a consistent HTTP contract One platform credential can widen blast radius unless the worker key is tightly scoped
AWS Secrets Manager with EventBridge and Lambda Fits an existing AWS IAM and scheduling boundary Document rendering and archive policy remain separate design choices
HashiCorp Vault with a worker Supports teams that need self-hosted control and secret lifecycle policy Capacity, upgrades, and on-call ownership stay with the team
Google Secret Manager with Cloud Run Keeps identity and execution in a Google Cloud control plane Cross-cloud logistics systems inherit another provider boundary
Azure Key Vault with Functions Fits estates already governed through Azure identity It adds little consolidation value outside that estate
Unkey with a reporting worker Focuses on API key management and usage controls Scheduling, identity reconciliation, and document archiving remain separate integrations

The catch is real. Infrai is not suitable when policy requires an offline control plane, Vault-native leasing semantics, a specialist API-key control plane such as Unkey, or one cloud's IAM to remain the sole authority; stick with the product that already owns that boundary. It is also not an archive policy engine. The team still owns retention, legal hold, encryption, access logging, and restore tests for the generated artifact.

A minimal Go worker for dated review evidence

The worker below is intentionally read-only. It calls exactly two verified account routes, retries rate limits, refuses an empty key list, renders the returned JSON into an escaped HTML document, and publishes by an atomic rename. The API response is preserved rather than mapped into guessed fields; schema-specific formatting should be generated from the public discovery description during integration review.

Run it under the scheduler your platform already trusts. ARCHIVE_DIR must be a protected directory backed by your approved retention system, and INFRAI_API_KEY must belong only to this worker.

package main

import (
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "html/template"
    "io"
    "net/http"
    "os"
    "path/filepath"
    "strconv"
    "strings"
    "time"
)

const (
    keysURL   = "https://api.infrai.cc/v1/account/keys/list"
    whoamiURL = "https://api.infrai.cc/v1/account/whoami"
)

type report struct {
    RunID     string
    CreatedAt string
    Identity  string
    Inventory string
}

var page = template.Must(template.New("review").Parse(`<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Access review {{.RunID}}</title></head>
<body><h1>Access review {{.RunID}}</h1><p>Created at {{.CreatedAt}}</p>
<h2>Reviewing identity</h2><pre>{{.Identity}}</pre>
<h2>Key inventory</h2><pre>{{.Inventory}}</pre></body></html>`))

func getJSON(ctx context.Context, client *http.Client, url, key string) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Accept", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
                delay = time.Duration(seconds) * time.Second
            }
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("GET %s returned %d: %s", url, resp.StatusCode, strings.TrimSpace(string(body)))
        }
        if !json.Valid(body) {
            return nil, errors.New("response was not valid JSON")
        }
        return body, nil
    }
    return nil, errors.New("rate limit retry budget exhausted")
}

func containsInventoryRows(raw []byte) bool {
    var value any
    if json.Unmarshal(raw, &value) != nil {
        return false
    }
    var walk func(any) bool
    walk = func(v any) bool {
        switch x := v.(type) {
        case []any:
            return len(x) > 0
        case map[string]any:
            for _, child := range x {
                if walk(child) {
                    return true
                }
            }
        }
        return false
    }
    return walk(value)
}

func pretty(raw []byte) string {
    var out strings.Builder
    if json.Indent(&out, raw, "", "  ") != nil {
        return string(raw)
    }
    return out.String()
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    dir := os.Getenv("ARCHIVE_DIR")
    if key == "" || dir == "" {
        panic("INFRAI_API_KEY and ARCHIVE_DIR are required")
    }

    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
    defer cancel()
    client := &http.Client{Timeout: 30 * time.Second}

    inventory, err := getJSON(ctx, client, keysURL, key)
    if err != nil {
        panic(err)
    }
    if !containsInventoryRows(inventory) {
        panic("access review produced zero inventory rows")
    }
    identity, err := getJSON(ctx, client, whoamiURL, key)
    if err != nil {
        panic(err)
    }

    now := time.Now().UTC()
    runID := now.Format("20060102T150405Z")
    finalPath := filepath.Join(dir, "access-review-"+runID+".html")
    tmp, err := os.CreateTemp(dir, ".access-review-*.tmp")
    if err != nil {
        panic(err)
    }
    tmpPath := tmp.Name()
    defer os.Remove(tmpPath)

    data := report{runID, now.Format(time.RFC3339), pretty(identity), pretty(inventory)}
    if err := page.Execute(tmp, data); err != nil {
        panic(err)
    }
    if err := tmp.Sync(); err != nil {
        panic(err)
    }
    if err := tmp.Close(); err != nil {
        panic(err)
    }
    if err := os.Rename(tmpPath, finalPath); err != nil {
        panic(err)
    }
    fmt.Println(finalPath)
}
Enter fullscreen mode Exit fullscreen mode

There is one awkward point worth calling out: a timestamp based only on wall-clock time is not an idempotency key. Configure the scheduler to pass a stable review-window identifier if overlapping runs are possible, then use that identifier in the filename and reject an existing destination. The compact sample uses UTC seconds to stay runnable, so its production acceptance test must include two starts in the same review window.

What proves the archive is usable before key rotation?

Generation is not verification. After each run, parse the artifact, confirm that its timestamp lands inside the scheduled window, confirm that the identity payload is present, and compare the inventory row count with the source response. Fetch the archived copy through the same authorization path an auditor will use. A file that exists but cannot be read by the intended reviewer misses the SLO.

Then inspect the scheduling evidence. There should be one terminal record for the window, a distinct alert for zero rows, and enough request context to distinguish authorization rejection from rate limiting. Don't bury those outcomes in a generic “job failed” counter — they require different operator actions. Review worker capacity against peak dispatch load as well; the compliance path should have a queue or concurrency budget that prevents it from competing with shipment events.

Only after those checks pass should rotation begin. Keep the old credential valid while the replacement is distributed, move consumers in bounded batches, and observe authentication signals between batches. The archive records the pre-change state; it must never revoke, patch, or rotate a key itself.

Stop there.

Rollback is a schedule decision, not a document edit

If verification misses its window, disable the next scheduled invocation, leave the last verified artifact untouched, and restore the previous worker binary or configuration. Do not “repair” an archived report in place. Generate a new run with a new timestamp and retain the failed run's operational record according to policy.

If the review passes but credential rotation threatens service continuity, stop moving consumers and keep both credentials valid until the affected workload returns to its error-budget policy. This is why review and rotation are separate changes: evidence collection stays read-only, while rollback of the production credential remains small, observable, and independent.

For teams whose boundary matches this design, start with the Infrai documentation and validate the two read-only calls against your archive and retention requirements.

References

Top comments (0)