How to Scrape Webpages for AI Agents in 3 Lines of Code (No API Keys, 0.05 USDC on Base)

The Problem

Building an autonomous AI Agent often requires scraping web pages and converting them to clean Markdown.
However, traditional scraping services impose heavy friction:

1. Mandating credit card registrations.
2. Forcing recurring monthly subscriptions ($20-$50/month).
3. Requiring complex API key management and quota monitoring.

For autonomous AI Agents, this model is fundamentally broken. AI Agents should pay per query autonomously using on-chain micropayments.

The Solution: AgentFi (x402 Micropayment Protocol)

AgentFi Web Extraction Gateway allows AI Agents and developers to scrape and convert any target page to Markdown by paying 0.05 USDC per query on Base Layer-2.
No accounts, no credit cards, zero API keys.

1. Quick cURL Test

You can trigger the gateway directly from your terminal:
curl -X POST "https://agentfi-gatewaye.onrender.com/v1/extract" \
-H "Content-Type: application/json" \
-H "x-402-signature: " \
-d '{"url": "https://news.ycombinator.com"}'

(If called without payment, the gateway returns HTTP 402 with billing metadata and recipient wallet parameters).

2. Python / LangChain Integration

Integrate this into your Agent workflows:

import requests

GATEWAY_URL = "https://agentfi-gatewaye.onrender.com/v1/extract"
tx_hash = "YOUR_BASE_USDC_TX_HASH"

response = requests.post(
GATEWAY_URL,
json={"url": "https://news.ycombinator.com"},
headers={"x-402-signature": tx_hash, "Content-Type": "application/json"}
)

print(response.json())

Gateway & Settlement Specifications

• Live Endpoint: https://agentfi-gatewaye.onrender.com/v1/extract
• Cost: 0.05 USDC per extraction call
• Settlement Network: Base Mainnet (Chain ID: 8453)
• Token Contract (Base USDC): 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913
• Recipient Address: 0x379B6643675421867f8436513826781748Ec8329