A developer finds a hardcoded API key in a production JavaScript file. He deletes the file, rotates the key, closes the incident. What went unchecked: whether web.archive.org captured a snapshot before deletion. In 2025, TruffleSecurity scanned the December 2024 Common Crawl archive and found 11,908 active, verified credentials across 2.76 million public pages.
Deleting a page or rotating a key after exposure is not remediation. If the Wayback Machine or Common Crawl captured the content first, the secret remains publicly accessible indefinitely. Practitioners who skip archive reconnaissance are ignoring a permanent layer of the attack surface.
The Archive Captures in Hours. Deleting Afterward Does Not Undo the Damage.
The Wayback Machine stores over 1 trillion web page captures, 99 petabytes of data, and ingests roughly 498 million pages per day. The December 2024 Common Crawl snapshot contains 400TB of compressed data, 2.67 billion pages, and 38.3 million registered domains. The two archives operate in parallel and independently: overlapping coverage is not redundancy, it is surface area.
Crawlers are not triggered by deletion events. A page exposed at 2pm can be archived by 4pm, well before any incident response process detects the problem. JavaScript files from popular domains, exactly the ones that concentrate the most hardcoded credentials, are crawled more frequently by both archives.
The irony of robots.txt reveals the permanent nature of the problem. Adding Disallow: / instructs future crawlers not to index, but does not remove captures that already exist. A file exposed on Monday and covered by robots.txt on Tuesday still appears in every CDX query with the original timestamp. Deletion on the origin server is invisible to the archive.
The Internet Archive itself demonstrated the cost of this dynamic in October 2024. Attackers found GitLab authentication tokens exposed in the organization's repository for roughly 2 years. With them, they breached the platform and exfiltrated 31M user records, including emails, usernames, and bcrypt password hashes. The platform that preserves third-party exposed credentials was compromised by the same category of exposure.
Four Documented Cases: What Survived Deletion in Public Archives
TruffleSecurity's research on the December 2024 Common Crawl establishes the scale of the problem with a precision rare in security research. The team scanned 2.76 million public pages and found 11,908 active, verified credentials. Mailchimp had roughly 1,500 unique API keys hardcoded in front-end JavaScript, spread across clients who integrated the SDK directly into public code. A single WalkScore API key appeared 57,029 times across 1,871 distinct subdomains, showing how a single leak propagates through an entire infrastructure. An AWS root key was found in front-end HTML being used directly for S3 authentication.
watchTowr Labs demonstrated a complementary approach using Amazon Athena to query Common Crawl at petabyte scale. The result: over 1,500 exposed SQL backup files and hundreds of config.php files serving database credentials as plain text. The exposure vector was a server misconfiguration that delivered PHP code as source instead of executing it, a mistake that misconfigured web servers have been making for decades and that Common Crawl faithfully recorded in every captured instance.
OAuth tokens in URL parameters form a third category. Authentication flows that pass tokens via GET, such as OAuth callbacks, session tokens, and password reset links, are captured verbatim in CDX records. The CDX API returns these URLs permanently, with the token visible in the original field, even if the session expired and the page was deleted minutes after the crawler's first access.
In April 2024, HackerOne established leaked credentials as an official vulnerability category. Roughly 18% of valid bug bounty reports involved endpoints discovered through passive reconnaissance, with a large portion sourced from web archives. Archive reconnaissance moved from niche technique to formal component of the triage pipeline on bug bounty platforms.
The CDX API: Programmatic Archive Queries at Reconnaissance Scale
The CDX endpoint turns archive queries into an automated pipeline. The base URL https://web.archive.org/cdx/search/cdx accepts unauthenticated queries with subdomain wildcards, MIME type filters, HTTP status filters, date ranges, and field collapsing. The default response is JSON, directly processable by jq or any scripting language with no authentication layer.
For discovering JavaScript files on a target:
https://web.archive.org/cdx/search/cdx?url=target.com/*&output=json&fl=original,timestamp,statuscode,mimetype&filter=mimetype:text/javascript&filter=statuscode:200
Each capture returns a digest field with a SHA1 hash of the content. Comparing digests across snapshots from different dates identifies when a file changed, pinpointing the exact moment credentials were introduced or removed. This technique eliminates the need to download every snapshot: a digest change signals which specific snapshot contains the relevant version.
For subdomain enumeration via archive:
https://web.archive.org/cdx/search/cdx?url=*.target.com&collapse=urlkey&output=json
The collapse=urlkey parameter returns one result per normalized URL, surfacing subdomains that no longer resolve in DNS but whose history remains intact. Staging subdomains, discontinued development environments, and internal tools appear in this query with a frequency that rarely matches what the security team knows exists.
gau and waybackurls: Aggregating Multiple Sources Against a Single Target
No single archive captures everything. gau aggregates four sources simultaneously: Wayback Machine, Common Crawl, AlienVault OTX, and URLScan.io. A single command produces a URL list that no isolated source can replicate, with overlapping snapshots from distinct periods and crawlers covering different exposure windows.
gau --subs target.com | tee urls.txt
grep -iE '\.js$|\.json$|\.env$|backup|config' urls.txt | sort -u > high_value.txt
cat high_value.txt | trufflehog filesystem --only-verified
For queries restricted to the Wayback Machine, waybackurls is the direct option:
echo target.com | waybackurls | grep -iE '\.js$|\.env$|config'
The waymore R mode downloads full HTTP responses offline, enabling analysis with grep, TruffleHog, and LinkFinder without generating logs on the target server. Patterns like apiKey=, Authorization:, password=, and SECRET appear frequently in historical JavaScript files removed from the origin server but preserved in the archives. Offline analysis eliminates WAF or IDS detection: no request reaches the live target.
LinkFinder analyzes archived JavaScript offline to extract undocumented API endpoints. Combined with TruffleHog for entropy-based secret detection, the pipeline goes from URL list to verified credentials with no interaction with the live server.
Your Archive History Is an Asset: Query It Before Attackers Do
Offensive archive reconnaissance applies directly to defensive monitoring. A CDX wildcard query against your own domains reveals every path ever captured, including forgotten staging subdomains, old API versions, and internal tools never intended to be crawled. What the attacker sees in the query, the defender can see first with the same query.
Historical robots.txt files are particularly revealing. One documented bug bounty program had nearly every internal endpoint listed in its robots.txt, expecting the file to instruct crawlers not to index those paths. All of them were preserved in Wayback Machine snapshots and are accessible via CDX. The robots.txt meant to hide endpoints created a persistent map of sensitive surface area in the archives.
The <meta name="robots" content="noarchive"> tag prevents future captures, but does not remove snapshots that already exist. Only the Internet Archive exclusion process can purge historical captures, and that process takes months. Treating any exposed credential as permanently compromised, regardless of when it was rotated, is the only posture consistent with what public archives preserve permanently.
The concrete defensive cadence: run a CDX wildcard query weekly per domain, compare JavaScript file digests against live versions, incorporate archive history into the attack surface management program. The 11,908 secrets TruffleSecurity found in Common Crawl were already available to any attacker who ran the same scan. The difference between detection and compromise is who ran the query first.
Top comments (0)