A developer finds an API key in a public repository, generates a replacement, and pushes the fix commit. The incident appears closed. The key persists in 3 places the rotation never reached.
Git history on every existing clone retains the original blob. Nginx access logs record every query parameter at request time. An attacker who watched the fix commit knew the old key was still valid and had hours to act.
Rotation acts on the active credential. It does not act on version control history, log archives, or telemetry data. GitHub's 2024 secret scanning report documented 39 million secrets exposed in public repositories. Push protection caught 4.4 million before they reached repositories. The remaining 34.6 million are now in git histories, log files, and monitoring tools.
Three Surfaces Where Rotated Keys Survive
Three records remain unchanged. These are 3 surfaces that independently preserve the exposed key after rotation completes.
Git stores each file version as a content-addressed blob. Removing a file from HEAD creates a new commit object. The original blob remains in .git/objects. Anyone with read access to the repository has access to the full history.
API keys passed as query parameters appear in nginx's $request log field, which records the full URL with query string. CloudFront's cs-uri-query field and Datadog APM traces also capture the key. The log archive receives the key at request time. Rotation does not update the archive.
CVE-2024-28778 documents an Artifactory API key exposed in IBM Cognos Controller versions 11.0.0 through 11.0.1. The key affected downstream CI/CD pipelines and allowed unauthorized package publishing under the organization's namespace. An infrastructure credential with broad blast radius that no rotation of the application credential could address.
Git History Preserves Secrets After HEAD Deletion
Removing a secret from the latest commit does not remove it from the repository. Every clone, fork, and cached snapshot retains the blob. Rewriting tools reach the remote origin; they cannot reach distributed copies already in existence.
Git retains all reachable objects in packfiles. A secret that lived in any commit remains in the local object store of every clone made before the history rewrite. BFG Repo-Cleaner rewrites history but requires force-pushing and notifying all collaborators. In an organization with 50 or more engineers, a fork may preserve the original history.
CVE-2021-41077 documents the Travis CI case from September 2021. Over 8 days (September 3-10), environment variables were injected into fork PR builds across 900,000+ open-source projects. Affected variables included API keys, signing keys, and access tokens. Travis CI patched silently with no post-mortem and no user notification.
Organizations had no mechanism to know which secrets were captured during the window. Ethereum developer Felix Lange identified the issue on September 7. Travis CI fixed it on September 10. No affected project was notified.
GitGuardian's State of Secrets Sprawl 2026 report recorded 28.65 million secrets, a 34% year-over-year increase, with 70% of secrets committed in 2022 still active in 2026.
Query Parameters in Access Logs: The Copy Rotation Does Not Reach
API keys embedded in URL query parameters appear verbatim in every log layer. Teams rotate the application credential. Log archives receive no update.
Nginx's default log format (combined) includes $request, which records the full URL with query string. A caller using ?api_key=xxx generates a permanent record per request in access_log. CloudFront logs include the cs-uri-stem and cs-uri-query fields. Query strings are logged unredacted by default and shipped to S3 with the configured retention policy.
OWASP API8:2023 (Security Misconfiguration) classifies API keys in query parameters as a misconfiguration class. Logging utilities with insecure defaults write request data, including authentication parameters, to access logs. This is classified as misconfiguration, not an application bug.
Datadog APM captures full request URLs in traces. A ?api_key= parameter is indexed, searchable, and visible to any team member with APM read access. Log access controls are typically weaker than the original API's access controls. An S3 bucket with CloudFront logs is often readable by all engineers while the rotated key may have had restricted distribution.
The Rotation Event Is an Attacker Signal
A passive observer monitoring a repository detects the exact moment a key is being rotated. The fix commit starts a countdown. The old key remains valid until deployment propagates and revocation completes.
A git commit that removes a hardcoded credential, updates a .env.example, or bumps a key version variable is a public event. On public repositories, GitHub Actions pipeline runs triggered by the fix PR are observable. The time between the fix commit and deployment is measurable from outside the organization.
AWS Secrets Manager uses 3 staging labels to document the grace period. AWSPENDING: new credential generated. AWSCURRENT: new credential active. AWSPREVIOUS: old credential, still valid during grace period. The grace period is configurable and defaults to non-zero.
The CircleCI January 2023 incident documents the revocation window at real scale. Exfiltration occurred December 22, 2022. All API tokens revoked January 5, 2023; CircleCI announced the revocation publicly on January 6. AWS tokens January 7-12. The window between exfiltration and complete revocation reached 21 days.
The Incident Record: CVE-2021-41077 and CircleCI 2023
Travis CI and CircleCI demonstrate the same structural failure. Secrets remained accessible far longer than the initial exposure window because revocation lagged the compromise by days. Detection existed in both cases. The delay was operational.
Travis CI (CVE-2021-41077, September 2021): 8-day exposure window, 900,000+ projects affected, silent patch, zero notification. Organizations had no mechanism to know which secrets were captured during the exposure window.
CircleCI (January 2023): laptop compromised December 16 via 2FA-backed SSO session cookie. Unauthorized reconnaissance December 19. Exfiltration December 22. All API tokens revoked January 5, 2023; CircleCI announced the revocation publicly on January 6. AWS tokens January 7-12. Gap from exfiltration to complete revocation: up to 21 days.
GitGuardian's State of Secrets Sprawl 2026 report recorded that 70% of secrets leaked in 2022 were still active in 2026. The industry-wide revocation rate is failing to outpace the exposure rate across all organizations, not just these two incidents.
Rotation achieves its goal when the credential has never appeared in version control, has never passed as a query parameter, and the attacker had no prior read access — these are the prerequisites, not the common case.
Fix: Atomic Rotation, Log Scrubbing, and Eliminating Stored Credentials
The correct response to API key exposure combines 3 controls. Simultaneous generation and revocation, with no overlap window. Query parameter masking at the log level. Elimination of stored API keys in favor of short-lived credentials, where the provider supports it.
Atomic rotation. AWS Secrets Manager does not provide atomic rotation by default. When a new key becomes AWSCURRENT, the previous key moves to AWSPREVIOUS and remains valid at the target service until the next rotation cycle completes. True atomic rotation requires the rotation Lambda's finishSecret step to explicitly call the target service's API to invalidate the previous key before completion. AWS does not handle this automatically.
Log scrubbing. Change nginx log_format to use $uri instead of $request to exclude query strings from access_log. For CloudFront: configure the cache policy to exclude sensitive query parameters from logging. Audit Datadog APM trace parameters for credential patterns.
Credential elimination. Replace API keys with OIDC federation where the provider supports it (AWS, GitHub Actions, GCP). Short-lived tokens from OIDC have no rotation gap because they expire by design. Rotation stops being a problem when the persistent credential stops existing.
The MAGO Intel tool (intel.mago.team) scans API gateway access logs and CI/CD pipeline configurations during security assessments. The scan covers API keys in query parameters and rotation-gap indicators.
GitHub push protection blocked 4.4 million secrets in 2024 out of 39 million leaked. The 34.6 million that passed through are now in log archives, git histories, and monitoring tools. Rotation acts on the credential. The records are still there.
Top comments (0)