API Key Inventory and Application Logs During Checkout Rotation (Audit Evidence Limits)

An e-commerce access review needs two evidence sets because it asks two different questions. A credential inventory answers what can authenticate now: owner, scope, environment, creation time, rotation state, and expiry. Application audit logs answer what happened: which stable principal attempted which operation, against which tenant or resource, with what outcome. During production API-key rotation, use the inventory to control the overlap window and the logs to prove traffic has left the retiring key. Never treat one as a substitute for the other.

The dominant bill is usually retained event volume, not the inventory row for each key. Model it before choosing a retention period: events per day x average stored bytes x retained days x replication and index factor. If checkout emits 18 million auditable authorization events per day and the encoded record averages 700 bytes, the raw stream is 12.6 GB per day before indexes, replicas, or backups. Those figures are an illustrative capacity model, not a benchmark. An inventory of even 100,000 compact credential records is small beside that stream.

The practical decision is to retain detailed events long enough to cover the review and investigation window, while retaining compact lifecycle evidence much longer. Aggregate old operational events only after preserving the fields and integrity evidence required by policy. What I would deliberately stop keeping is an indefinitely searchable copy of every successful request payload. The cost is real: after the detailed window closes, an investigation can establish credential lifecycle and summarized use, but may no longer reconstruct every individual success.

Why can't API key inventory answer application audit log questions?

Inventory is state-oriented. It should tell a reviewer that credential cred_checkout_02 is active, belongs to the checkout service, is restricted to the production merchant tenant, was introduced for rotation, and must be retired at a defined deadline. A useful inventory also records the credential's public identifier or fingerprint, never the secret value itself. OWASP's Secrets Management Cheat Sheet recommends metadata about who created a secret, when it was created, rotation, expiry, and access, while warning that secrets must not be logged.

Audit evidence is event-oriented. It should tell a reviewer that a stable service principal used cred_checkout_02 to request an order authorization, that policy allowed or denied the action, and that the event received a correlation identifier. The principal matters more than a mutable display name. The credential identifier matters because a single principal may legitimately have old and new keys active during rotation.

The distinction exposes two common false assurances. An unused-looking inventory entry does not prove non-use if the application never records the credential identifier. A busy audit trail does not prove that the observed key is the only valid key; an undiscovered or orphaned credential may remain capable of authenticating without appearing in recent events. Absence of an event is not evidence of absence unless collection coverage, clock handling, delivery, and retention are known.

The join is mandatory.

Review question	Authoritative evidence	Typical failure if used alone
What credentials can authenticate?	Current inventory plus lifecycle history	Misses actual use and denied attempts
What did a principal attempt?	Application audit events	Misses dormant but still-valid credentials
Is the old key safe to revoke?	Inventory state joined to recent use by credential ID	Revokes live traffic or leaves needless overlap
Who approved the change?	Immutable lifecycle event with actor and reason	Current state erases the decision path

Keep authentication logs separate from business audit semantics, even if they share transport. A gateway can report that a key authenticated; only the application can reliably state that an order refund was requested for merchant A and denied because the principal lacked the required scope.

The rotation protocol is a state machine

Zero-downtime rotation requires bounded dual validity, not an instantaneous replacement. Create the successor, distribute it through the approved secret-delivery path, deploy consumers, observe adoption, revoke the predecessor, and verify that attempts using it are refused. Each transition needs an idempotency key and an append-only audit event. Retries are normal; duplicate credential creation is not.

I use states such as pending, active, retiring, and revoked because a Boolean enabled field cannot represent an overlap deadline or distinguish an intentional retirement from an emergency revocation. The state machine also makes reconciliation mechanical: inventory state can be compared with the sequence of accepted lifecycle events, and any impossible transition becomes reviewable evidence rather than silent drift.

package rotation

import (
    "context"
    "errors"
    "time"
)

type State string

const (
    Active   State = "active"
    Retiring State = "retiring"
    Revoked  State = "revoked"
)

type Transition struct {
    CredentialID  string
    From, To      State
    EffectiveAt   time.Time
    IdempotencyID string
    ActorID       string
    Reason        string
}

type Store interface {
    ApplyOnce(context.Context, Transition) (alreadyApplied bool, err error)
}

func Revoke(ctx context.Context, store Store, t Transition) error {
    if t.From != Retiring || t.To != Revoked {
        return errors.New("invalid credential transition")
    }
    _, err := store.ApplyOnce(ctx, t)
    return err
}

ApplyOnce must commit the state change and its audit record atomically, or through a transactional outbox that is reconciled. Otherwise a crash can produce a revoked key without evidence, or evidence claiming revocation while the key remains valid. The exactly-once claim should be narrow: the transition has one durable effect for one idempotency identifier. Message transport may still deliver more than once, so consumers must deduplicate.

Do not log the API key, authorization header, secret-manager response, or request body. Record a non-secret credential ID assigned at creation, a stable principal ID, tenant ID, operation class, result, reason code, event time, ingestion time, correlation ID, and policy version. Redaction belongs at the producer as well as the collector because downstream filters are not a reliable containment boundary.

Spend ceiling versus refused orders

The overlap window is an operational risk budget. A short window limits how long two credentials are valid but raises the chance that a lagging checkout instance will present the revoked predecessor. A long window reduces refused traffic during deployment but enlarges the period in which either credential can be abused. There is no universal duration. Set it from measured deployment propagation, rollback time, event-delivery lag, and the business tolerance for refused checkout requests, then add an explicit margin approved by the control owner.

Use high-cardinality detail where it changes a decision. During rotation, retain per-attempt events keyed by the non-secret credential ID so operators can see the last accepted use of the predecessor and the first sustained use of the successor. Outside that window, success events can often move to a cheaper tier or become policy-approved aggregates, while denials, privilege changes, credential lifecycle transitions, and integrity checkpoints remain individually searchable for the required period. The exact periods belong in a retention schedule derived from legal, contractual, incident-response, and payment-control obligations; they should not be invented by an engineer or copied from a generic article.

PCI DSS 4.0.1 Requirement 10 applies specific audit-log controls and retention expectations to entities in scope. NIST SP 800-92 provides broader guidance for log-management infrastructure, including retention and analysis planning. Neither source makes every application event equally valuable. Scope must be determined by the organization's qualified compliance and legal owners.

A useful spend ceiling is expressed as a daily ingest limit and a retained searchable-byte limit, with reserved capacity for denials and lifecycle events. Sampling successful reads may be acceptable under an approved policy; sampling credential creation, revocation, authorization denial, or privilege escalation destroys the evidence an access review needs. Keep those complete.

This method has limitations. It is not suitable when the application cannot attach a stable, non-secret credential identifier to each decision, and aggregates are the wrong alternative when an investigator must reconstruct every in-scope transaction. In that case, choose complete event retention in access-controlled archival storage and accept slower queries, rather than claiming that a sampled operational index is an audit trail. The opposite boundary matters too: retaining complete request bodies merely to prove key adoption increases exposure and storage load without improving the inventory-to-event join, so a narrow event schema is the better choice there.

That trade-off has a hard edge.

Revoke the predecessor only after all expected deployment units report the successor version, the old credential has no accepted events for a window longer than observed delivery lag plus rollback time, and a synthetic request proves that the successor works. If the evidence budget cannot support that window, shorten searchable retention elsewhere before weakening the rotation signal. Refused traffic is visible immediately; a dormant valid key can remain invisible for months.

Testing the evidence, not just the happy path

A staging rotation proves little unless it exercises retries and partial progress. Test duplicate create and revoke commands with the same idempotency identifier. Delay audit delivery. Restart the worker after the inventory commit but before publication, then confirm that reconciliation emits the missing event exactly once at the logical level. Skew producer clocks and verify that ingestion time remains available for ordering analysis.

The production drill should include a canary consumer, a rollback path that does not reactivate a revoked key, and an alert on any accepted use of a retiring credential after the deployment deadline. After revocation, an attempt with the predecessor must be denied and audited without echoing the secret. A request with the successor must preserve the same stable service principal and tenant restrictions. This is a subtle failure: the new key authenticates, but it is attached to broader scope than the old one.

Three reconciliations catch most control gaps:

1. Compare active and retiring inventory records with the deployment registry; every production credential needs an accountable owner and an expected consumer.
2. Compare lifecycle transitions with current state; every state must be derivable from an accepted, ordered transition history.
3. Compare authentication decisions with application events by correlation ID; unexplained gaps become incidents or documented collection loss, never assumed success.

Keep the alert payload sparse. Operators need credential ID, principal, environment, result, reason, timestamps, and correlation ID. They do not need secret material or full order details to decide whether to halt a rotation.

A review packet with explicit limits

The final access-review packet should contain an inventory snapshot at a declared time, lifecycle events for the review period, evidence of the join to application activity, exceptions, reconciliation results, and the retention policy that explains unavailable detail. Sign or otherwise integrity-protect exported evidence, restrict access to it, and record who generated and reviewed it. NIST's log-management guidance treats confidentiality, integrity, and availability of logs as operational concerns, not optional archival polish.

Inventory proves capability; audit logs prove observed behavior. Together they support a defensible decision to revoke an old production key without guessing which checkout instances still use it. They still cannot prove that an unobserved request never occurred, that every collector delivered every event, or that a compromised principal was operated by its legitimate owner. State those limits in the packet.

For this e-commerce rotation, the defensible outcome is modest: the successor is active with equivalent scope, the predecessor is revoked after a measured quiet period, post-revocation attempts are denied, and every control transition is attributable and idempotent. The system preserves compact lifecycle evidence longer than bulky success traffic. That boundary respects a fixed evidence budget without purchasing lower cost through unexplained refused orders.

Further reading

• OWASP Secrets Management Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
• NIST SP 800-92, Guide to Computer Security Log Management: https://csrc.nist.gov/pubs/sp/800/92/final
• PCI DSS v4.0.1 document library: https://www.pcisecuritystandards.org/document_library/
• OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html