A leaked-key drill has one unforgiving constraint: after a credential is disabled, the team must still be able to prove which deployed artifact presented it, without recovering the secret from a log. The practical choice is to emit a startup attestation that binds a nonreversible key identifier to a build identifier, workload identity, and deployment event, then store that event in an append-only audit stream.
TL;DR: never log the API key. Derive a stable, environment-scoped fingerprint with HMAC, record it once at successful client initialization, attach the immutable build ID and workload identity, and make the audit event idempotent. During the drill, pivot from the compromised credential fingerprint to every build and workload that used it; the blast radius is the resulting set, not the number of log lines.
How should a service log API key identity at startup?
A useful record answers four questions at once: which credential was loaded, which exact artifact loaded it, which workload instance asserted the fact, and whether initialization actually succeeded. A plain message such as API key configured answers none of them. Logging the last four characters is also a weak design because those characters are part of the secret, can collide, and often escape into less protected operational indexes.
The OWASP Secrets Management Cheat Sheet recommends auditing who requested a secret, which secret was requested, when it was requested, and whether the request was approved or denied, while explicitly warning that secret values must not be logged. That boundary determines the event model here. The audit trail identifies a credential without containing the credential.
Use an opaque key reference from the secret store when one is reliably available. Otherwise, derive an identifier with a dedicated audit HMAC key. A bare cryptographic hash is a poor fit for low-entropy credentials because an attacker who obtains the log can test guesses offline; an HMAC requires a separate key and creates an intentional security boundary. The audit HMAC key must not be the API key, and it should be scoped so that correlating fingerprints across unrelated environments is impossible.
This distinction is easy to miss.
A tempting first design is to log a key prefix or suffix beside the build ID because the line remains readable during an incident. Reject it. That fragment is still secret material, its uniqueness is unknown, and it spreads into every downstream copy of the operational log. The HMAC design makes a different, explicit trade-off: responders lose a human-readable hint, but receive a deterministic 256-bit identifier whose comparison does not disclose any API-key characters. Requiring an audit key of at least 32 bytes and labeling the derivation v1 also turns two hidden assumptions into reviewable constraints.
| Field | Meaning | Drill use |
|---|---|---|
credential_fingerprint |
HMAC-derived identifier, never the secret | Finds all known uses of one credential |
build_id |
Immutable artifact or source revision identifier | Selects binaries requiring investigation |
workload_id |
Runtime identity, not a mutable hostname alias | Locates affected executions |
deployment_id |
Identifier shared by one rollout | Separates rollout scope from restarts |
initialized_at |
UTC event time | Establishes the observation window |
outcome |
Whether client initialization succeeded | Separates attempted loads from proven use |
Keep the dimensions distinct. Build ID says what code ran; deployment ID says why that code appeared; workload ID says where it executed. Combining them into one free-form label makes later reconciliation fragile.
Step 1: Construct a deterministic attestation
This implementation accepts the secret as bytes, computes the audit identifier, and returns a typed event. It refuses empty build, deployment, workload, or audit-key values. An absent dimension is not harmless during an incident: it converts a bounded query into an inference exercise.
package provenance
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"errors"
"time"
)
type StartupEvent struct {
SchemaVersion int `json:"schema_version"`
EventID string `json:"event_id"`
CredentialFingerprint string `json:"credential_fingerprint"`
BuildID string `json:"build_id"`
DeploymentID string `json:"deployment_id"`
WorkloadID string `json:"workload_id"`
InitializedAt time.Time `json:"initialized_at"`
Outcome string `json:"outcome"`
}
func Fingerprint(auditKey, apiKey []byte) (string, error) {
if len(auditKey) < 32 { return "", errors.New("audit HMAC key must be at least 32 bytes") }
if len(apiKey) == 0 { return "", errors.New("API key is empty") }
mac := hmac.New(sha256.New, auditKey)
_, _ = mac.Write(apiKey)
return "hmac-sha256:v1:" + hex.EncodeToString(mac.Sum(nil)), nil
}
func NewStartupEvent(auditKey, apiKey []byte, buildID, deploymentID, workloadID string, at time.Time) (StartupEvent, error) {
if buildID == "" || deploymentID == "" || workloadID == "" {
return StartupEvent{}, errors.New("build, deployment, and workload IDs are required")
}
fingerprint, err := Fingerprint(auditKey, apiKey)
if err != nil { return StartupEvent{}, err }
material := []byte(deploymentID + "\x00" + workloadID + "\x00" + buildID + "\x00" + fingerprint)
eventID, err := Fingerprint(auditKey, material)
if err != nil { return StartupEvent{}, err }
return StartupEvent{1, eventID, fingerprint, buildID, deploymentID, workloadID, at.UTC(), "authenticated_client_initialized"}, nil
}
The v1 prefix matters because identifier derivation is part of the audit schema. If the HMAC key or canonicalization scheme changes, a new version lets an incident query select the correct comparison procedure instead of silently producing false negatives. Retain the full digest; truncation introduces a collision decision the audit system does not need.
Do not create this event immediately after reading an environment variable. Emit it only after the client has completed whatever local validation or authenticated initialization the application defines as usable. Otherwise, a malformed credential and one that successfully entered service receive the same evidentiary weight.
Step 2: Make emission exactly-once in effect
Processes restart. Collectors retry. Networks partition. Exactly-once transport is not a reasonable assumption, but exactly-once effect is attainable when the producer supplies a deterministic event ID and the audit sink enforces uniqueness on that ID.
Retries must converge.
Four attempts are enough for this example.
package provenance
import (
"context"
"encoding/json"
"errors"
"time"
)
var ErrAlreadyRecorded = errors.New("startup event already recorded")
type AuditSink interface {
AppendIfAbsent(context.Context, string, []byte) error
}
func RecordWithRetry(ctx context.Context, sink AuditSink, event StartupEvent) error {
payload, err := json.Marshal(event)
if err != nil { return err }
delays := []time.Duration{0, 100 * time.Millisecond, 500 * time.Millisecond, 2 * time.Second}
var lastErr error
for _, delay := range delays {
if delay > 0 {
timer := time.NewTimer(delay)
select {
case <-ctx.Done(): timer.Stop(); return ctx.Err()
case <-timer.C:
}
}
err = sink.AppendIfAbsent(ctx, event.EventID, payload)
if err == nil || errors.Is(err, ErrAlreadyRecorded) { return nil }
lastErr = err
}
return lastErr
}
There is a real availability trade-off. If the service accepts traffic while its attestation is missing, the audit trail has a gap precisely where responders expect certainty. If it blocks forever on an unavailable audit sink, an observability dependency becomes a production outage. A defensible middle ground is bounded startup retry followed by readiness failure: the process stays alive for diagnosis, but receives no work until the event is durably acknowledged. The timeout belongs in the service's availability budget, not in a universal recipe.
The sink needs stricter access control and retention than an ordinary debug log. Only the event producer should append, incident roles should read, and neither role should mutate prior events. OWASP describes centralized secret-management auditing and protection against tampering; the same logic applies to these derived identity records.
How does the drill measure one credential's blast radius?
Start with a known credential in a non-production exercise environment. Compute its fingerprint through the controlled startup function, then query the audit store for exact matches within the exercise window. Group matches by build, deployment, and workload. That tuple is the observed blast radius.
Do not search logs for fragments of the raw key. Do not assume the newest build is the only consumer. A credential may survive a rollback, a delayed batch worker, or a replica that missed an expected deployment transition; the drill exists to expose those assumptions while the stakes are low.
package provenance
type Exposure struct { BuildID, DeploymentID, WorkloadID string }
func BlastRadius(events []StartupEvent, fingerprint string) map[Exposure]struct{} {
result := make(map[Exposure]struct{})
for _, event := range events {
if event.CredentialFingerprint == fingerprint && event.Outcome == "authenticated_client_initialized" {
result[Exposure{event.BuildID, event.DeploymentID, event.WorkloadID}] = struct{}{}
}
}
return result
}
A passed drill requires more than finding rows. Disable the exercise credential, verify that affected workloads stop authenticating, issue a replacement through the normal secret-delivery path, and observe new attestations under a different fingerprint. Reconcile three sets: workloads expected by the deployment system, workloads observed in the audit stream, and workloads seen using the credential by the protected service. Any difference is an actionable gap.
Duplicate delivery must not inflate the affected count, and missing delivery must remain visible as a mismatch rather than being hidden by aggregation. Preserve the raw append-only events and the deterministic reconciliation result.
Compare designs by credential blast radius
Credential granularity is the central decision. One account-wide key creates the broadest drill result because every service and build shares a fingerprint. Per-service credentials narrow attribution to one boundary. Per-workload credentials narrow it again, while increasing issuance, rotation, and audit volume.
| Design | What one fingerprint identifies | Principal trade-off |
|---|---|---|
| Shared account credential | Many services and releases | Revocation disrupts unrelated consumers |
| Credential per service and environment | One service boundary | More lifecycle automation |
| Credential per workload identity | One runtime identity or cohort | Highest issuance and event volume |
This is not a maturity ladder. A per-workload design with weak inventory may be less trustworthy than a per-service design whose issuance, expiry, deployment, and revocation paths are continuously reconciled. Auditability is a system property.
Compliance constraints reinforce the point: access to secret material and audit records should follow least privilege, logs must exclude secrets, and retention must reflect the organization's legal and security requirements. Avoid inventing a universal retention duration. The correct period depends on the detection window and applicable obligations, while the implementation must prevent an attacker from erasing evidence before it ends.
Roll out without creating an audit blind spot
Begin in report-only mode for one low-risk service. Generate the event, verify deterministic IDs across retries, and compare the audit inventory with the deployment inventory. Do not gate readiness yet. This phase finds missing build metadata and unstable workload identifiers without changing availability.
Next, enforce required metadata in the build and deployment pipeline. Promote an immutable artifact identifier rather than a branch name or mutable tag, and inject a deployment ID that survives process restarts within one rollout. Once reconciliation is consistently complete, make durable audit acknowledgement a readiness condition with bounded retry.
Finally, run the drill end to end: identify, disable, observe rejection, replace, restart, and reconcile. Record the exercise as an auditable change with an owner and timestamps, but keep its credential out of tickets and chat transcripts. Success is crisp: every expected workload is associated with the old or replacement fingerprint, no unexpected workload is associated with either, and the raw credential appears nowhere in the audit path.
That gives incident response a ledger rather than a hunch.
Top comments (0)