DEV Community

artmcclellan2521995-sys
artmcclellan2521995-sys

Posted on

How to Scrape Behind Cloudflare with 0 API Keys: Pay-Per-Query on Base L2 (x402)

Every developer building an AI agent or LLM pipeline eventually hits the same roadblock:

  • You need clean Markdown from a Cloudflare-protected page.
  • Traditional scrapers get blocked immediately.
  • Commercial scraping APIs force a $50/mo minimum subscription and credit card KYC just to test 10 pages.

Here is how to extract protected pages using HTTP 402 (Payment Required) and instant Base L2 micropayments.


Specifications

  • Protocol: HTTP 402 with on-chain verification
  • Network: Base Mainnet (Chain ID 8453)
  • Cost: 0.05 USDC per extraction
  • Settlement Token: USDC (0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913)
  • Gateway Wallet: 0x379B6643675421867f8436513826781748Ec8329
  • Gateway Endpoint: https://agentfi-gatewaye.onrender.com/v1/extract

How to Call (cURL)

Once you send 0.05 USDC on Base to the gateway wallet, pass the transaction hash in x-402-signature:


bash
curl -X POST [https://agentfi-gatewaye.onrender.com/v1/extract](https://agentfi-gatewaye.onrender.com/v1/extract) \
  -H "Content-Type: application/json" \
  -H "x-402-signature: <YOUR_BASE_TX_HASH>" \
  -d '{"url": "[https://example.com](https://example.com)"}'
Enter fullscreen mode Exit fullscreen mode

Top comments (0)