Before opening a headless browser, the right question is: has the target already published this data in JSON? Certificate Transparency logs, the Wayback Machine CDX API, and the GitHub REST API answer that question for free, with no authentication, without touching a single HTML page. Most OSINT practitioners never ask it.
Treating HTML scraping as the starting point is an operational mistake. The most productive OSINT targets already expose structured data at public endpoints with open documentation. Every unnecessary request to a target's HTML burns scraping budget, increases exposure, and triggers defenses before any relevant data is collected.
HTML scraping is the right starting point when a target has no structured API, rate-limits its API more aggressively than its web interface, or when the required data exists only in rendered DOM elements. For those cases, the techniques in Section 2 apply directly. The issue is making HTML the default — most targets publish structured data, and practitioners who skip the API check generate avoidable noise.
Start With the API, Not the HTML
The crt.sh JSON endpoint returns all SSL certificates issued for a domain since Certificate Transparency logs began. The query ?q=<domain>&output=json returns issuer, CN, SANs, notBefore/notAfter timestamps, and log entry date, with no authentication and no documented rate limit for normal use. A single request covers the complete subdomain history of any target with a web presence, including wildcard certificates and staging environment certificates that rarely appear in active scanners.
The Wayback CDX API works on the same model: http://web.archive.org/cdx/search/cdx?url=<target>&output=json returns all archived snapshots with timestamps, MIME types, and HTTP status codes. The endpoint supports status filters, URL collapse, and resume keys for domains with massive history. Zero cost, zero authentication, zero contact with the target's infrastructure.
The GitHub REST API with a personal token allows 5,000 requests per hour across the /search/repositories, /search/users, and /search/code endpoints. Results come paginated and filterable by language, organization, and file content. Even a handful of authenticated API requests return clean, structured JSON; a thousand HTML requests per hour against an anti-bot target return blocks after the third page. Shodan and Censys cover the entire IPv4 space at banner and certificate level. All these services deliver JSON by default. Treating HTML as the first option is a decision that increases cost and exposure without increasing data coverage.
When HTML Is Unavoidable: Anti-Detection in Three Layers
Modern anti-bot systems operate across three simultaneous layers: TLS fingerprint, JavaScript environment, and behavioral pattern. Bypassing only one layer does not survive Cloudflare-level defenses, which weigh all three signals together before deciding on blocking or CAPTCHA.
The TLS layer is the first barrier: the JA3 hash generated by Playwright's Chromium differs from real Chrome and exposes the scraper at the handshake, before the first HTTP request. Stealth plugins patch navigator.webdriver, chrome.runtime, plugin enumeration, language settings, and WebGL fingerprint. These patches are necessary, but Cloudflare adds cryptographic proof-of-work and timing analysis that operates independently of the TLS fingerprint. The behavioral layer is the hardest to fake: fixed-interval loading followed by immediate extraction is statistically distinguishable from human navigation even with a perfect fingerprint. Since 2023, Cloudflare Turnstile and similar ML-based behavioral classifiers have added a fourth detection layer that goes beyond JA3 and JS environment checks — they model the timing and pattern of user interactions, making static patches insufficient against modern enterprise anti-bot deployments.
Residential proxies eliminate the IP reputation layer. Datacenter ranges are blocklisted across all major anti-bot providers. Residential proxies (USD 10-15/GB) rotate through ISP-assigned IPs, indistinguishable from legitimate traffic by ASN. CAPTCHA correlates with the absence of prior session cookies, JavaScript challenge failure, and IPs with no history, not with raw request volume. Sessions with prior-page context and 5-15 second jitter between requests reduce CAPTCHA rate significantly more than user-agent rotation alone.
The LinkedIn Problem: No API, Civil Risk
LinkedIn shut down its public developer API in 2015. Current APIs are restricted to LinkedIn Marketing Solutions contracts and do not expose individual profile data. No legitimate structured path exists for profile enumeration at scale, making LinkedIn the canonical hard case in OSINT.
LinkedIn's automation detection operates by session fingerprinting before any IP-based block. Session token patterns, request timing uniformity, and the absence of interaction events trigger account restrictions before IP limits activate. IP blocking is the most rudimentary defense mechanism in LinkedIn's security stack.
The legal outcome of the hiQ case and what it means for OSINT practitioners is covered in the Legal Boundary section below.
Common Crawl: The Internet Already Collected for You
Common Crawl makes monthly web snapshots available since 2008 on Amazon S3 via AWS Open Data. Each crawl covers approximately 3 billion pages in around 100 TB compressed. The columnar parquet index is queryable via AWS Athena without downloading the full dataset.
The query for subdomain enumeration is straightforward:
SELECT DISTINCT(url_host_name)
FROM ccindex.ccindex
WHERE crawl = 'CC-MAIN-2024-10'
AND subset = 'warc'
AND url_host_name LIKE '%target.com%'
The cost per single-domain query on the optimized parquet runs under USD 5 per TB scanned. The result covers all subdomains captured that month without a single request reaching the target's infrastructure: zero IDS entries, zero firewall logs, zero IP burn. Historical crawls reveal subdomains and pages that were deleted, rotated, or decommissioned after incidents.
Research published by watchTowr Labs in 2024 ('All Around The World: The Common Crawl Dataset') demonstrated that Common Crawl can map expired government domain reuse across multiple countries, identifying live infrastructure risk from domains no longer under government control. Sixteen years of crawls represent historical attack surface that active scanners cannot reconstruct.
Legal Boundary: ToS vs. CFAA vs. Authorized Scope
The CFAA is the wrong framework for evaluating scraping risk. The 9th Circuit ruling in hiQ (April 2022) established that publicly accessible sites cannot impose unauthorized access liability through terms of service: publishing the data constitutes access authorization for purposes of the federal statute. Analyzing scraping risk through the CFAA leads to incorrect conclusions about where real exposure lies.
The real risk lies in state civil law. California trespass to chattels (server resource interference at scale), misappropriation (commercial exploitation of collected data), and breach of contract (ToS as an enforceable civil agreement) are the three vectors the hiQ settlement activated simultaneously, all without CFAA. The CFAA carries criminal penalties and federal jurisdiction; California torts carry civil exposure and state jurisdiction. The distinction matters for scoping decisions, contracting, and insurance.
robots.txt has no legal force in US law. The ToS is enforceable as a civil contract, not as criminal law. The CFAA is a criminal statute inapplicable to public data per the 9th Circuit. All three are distinct, and all three are routinely conflated in offensive security discussions. Explicit written authorization from the asset owner, with signed Rules of Engagement covering the target scope, defines the boundary between authorized reconnaissance and unauthorized access in any automated collection engagement.
Automating OSINT Collection Pipelines
A production OSINT pipeline treats source type as a routing decision: structured APIs first at maximum speed, semi-structured feeds at moderate rate, HTML scraping last with full browser cost and session management. The correct ratio by request volume is approximately 80/15/5.
The pattern works in three layers: parallel APIs (crt.sh, Wayback CDX, GitHub API, Shodan) consume zero scraping budget and deliver maximum data density. RSS and JSON-LD for news, forums, and job listings operate at 1 request per 30 seconds. The headless browser pool is restricted to targets without a machine-readable endpoint, with a maximum of 3 concurrent sessions and 5-15 second jitter between requests. Deduplication before fetch (URL hash against the existing corpus, cosine similarity above 0.85) eliminates re-scraping of already-indexed content.
The cost of an HTML scraping pool with residential proxies on a single VPS runs between USD 20-50/month including proxy bandwidth. Structured APIs cost essentially zero by comparison. intel.mago.team (MAGO team tool) applies this pipeline: target profiles trigger parallel queries across CT logs, GitHub, Wayback, and Shodan; gaps route to an HTML scraper with automatic session rotation; normalized results aggregate to the same schema regardless of source type.
The question before any OSINT collection task: does a structured endpoint already answer this? CT logs, the Wayback CDX API, and Common Crawl collectively cover subdomain history, archive history, and web enumeration without a single request to the target's infrastructure. Build the HTML scraper last, size it small, and route to it only when the structured layer has nothing left to offer.
Top comments (0)