DEV Community: AlterLab

How to Scrape Uber Eats Data: Complete Guide for 2026

AlterLab — Wed, 08 Jul 2026 15:20:55 +0000

How to Scrape Uber Eats Data: Complete Guide for 2026

Disclaimer: This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

To scrape Uber Eats restaurant data, use AlterLab's API with Python or Node.js to handle anti-bot protections automatically. Target public menu pages, extract structured fields like item names and prices via CSS selectors or Cortex AI, and scale responsibly with rate limiting. Start at T1 tier—the API promotes to T3/T4 as needed for JavaScript-dependent content.

Why collect food data from Uber Eats?

Food delivery data powers competitive intelligence for restaurants and analysts. Track real-time pricing trends across cuisines to adjust your menu strategy. Monitor competitor promotions and new dish launches for market timing. Aggregate nutritional information for dietary app development or supply chain forecasting.

Technical challenges

Uber Eats implements standard anti-bot systems: rate limiting by IP, User-Agent validation, and JavaScript challenges that block headless browsers without proper fingerprinting. Raw HTTP requests (T1/T2) typically return CAPTCHAs or empty responses for menu pages. AlterLab's Smart Rendering API manages proxy rotation, realistic headers, and headless Chrome instances to access public food data while respecting site protections.

Quick start with AlterLab API

See the Getting started guide for SDK setup. Below are examples scraping a public Uber Eats restaurant menu page (replace YOUR_API_KEY and the URL with your target).

```python title="scrape_ubereats-com.py" {3-5}

client = alterlab.Client("YOUR_API_KEY")
response = client.scrape("https://www.ubereats.com/menu/example-restaurant")
print(response.text[:500]) # First 500 chars of HTML






```javascript title="scrape_ubereats-com.js" {3-5}

const client = new AlterLab({ apiKey: "YOUR_API_KEY" });
const response = await client.scrape("https://www.ubereats.com/menu/example-restaurant");
console.log(response.text.slice(0, 500));

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-d '{"url": "https://www.ubereats.com/menu/example-restaurant"}'




## Extracting structured data
Uber Eats menu pages use consistent HTML structures. Target `.menu-item` containers for dish details:



```python title="extract_menu_items.py"

from parsel import Selector

client = alterlab.Client("YOUR_API_KEY")
html = client.scrape("https://www.ubereats.com/menu/example-restaurant").text
selector = Selector(text=html)

for item in selector.css(".menu-item"):
    name = item.css(".item-name::text").get()
    price = item.css(".price::text").get()
    print(f"{name.strip()}: {price}")

Node.js equivalent using cheerio:

```javascript title="extract-menu-items.js"

const client = new AlterLab({ apiKey: "YOUR_API_KEY" });
const html = await client.scrape("https://www.ubereats.com/menu/example-restaurant");
const $ = cheerio.load(html);

$(".menu-item").each((_, el) => {
const name = $(el).find(".item-name").text().trim();
const price = $(el).find(".price").text().trim();
console.log(${name}: ${price});
});




## Structured JSON extraction with Cortex
For typed output without manual parsing, use AlterLab's Cortex extraction API. Define a JSON schema for menu items:



```python title="extract_ubereats-com_structured.py"

client = alterlab.Client("YOUR_API_KEY")
result = client.extract(
    url="https://www.ubereats.com/menu/example-restaurant",
    schema={
        "type": "object",
        "properties": {
            "restaurant_name": {"type": "string"},
            "menu_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "price": {"type": "number"},
                        "description": {"type": "string"},
                        "rating": {"type": "number"}
                    },
                    "required": ["name", "price"]
                }
            }
        }
    }
)
print(result.data)  # Validated JSON output

Cost breakdown

AlterLab's pricing scales with technical difficulty. Uber Eats public menu pages typically require T3 (Stealth) due to anti-bot measures, but the API starts at T1 and promotes automatically—you only pay for the successful tier.

Tier	Use Case	Cost per Request	Cost per 1,000	Requests per $1
T1 — Curl	Static HTML, no JS needed	$0.0002	$0.20	5,000
T2 — HTTP	Standard pages with headers	$0.0003	$0.30	3,333
T3 — Stealth	Protected pages, anti-bot active	$0.002	$2.00	500
T4 — Browser	Full JS rendering required	$0.004	$4.00	250
T5 — CAPTCHA	CAPTCHA solving + JS rendering	$0.02	$20.00	50

View full pricing details. Note: AlterLab auto-escalates tiers—start at T1 and pay only for the level that succeeds.

Best practices

Rate limiting: Stay under 1 request/second per IP to avoid triggering protections (AlterLab's internal queues help manage this).
Robots.txt: Check https://www.ubereats.com/robots.txt for crawl delays and disallowed paths.
Dynamic content: Use wait_for parameters in AlterLab API to ensure menu items load before extraction.
Headers: AlterLab rotates realistic browser fingerprints—avoid overriding User-Agent unless necessary.
Error handling: Implement retries with exponential backoff for transient failures (HTTP 429/503).

Scaling up

For large-scale menu data collection:

Batch requests: Process 100 URLs concurrently using AlterLab's /batch endpoint (reduces overhead).
Scheduling: Use cron expressions via AlterLab's scheduling API for daily menu updates.
Data storage: Save extracted JSON to cloud storage (S3/GCS) with timestamps for historical analysis.
Responsible scaling: Monitor response codes—if 429s increase, reduce concurrency or add delays.

Key takeaways

Uber Eats public menu data is accessible via AlterLab's API without building custom anti-bot solutions.
Start with CSS selectors for simple extraction; use Cortex AI for schema-validated output.
Budget ~$0.002/request for most Uber Eats scraping tasks (T3 tier).
Always prioritize compliance: review ToS, rate limit, and scrape only publicly visible information.
Scale responsibly with batch processing and scheduling for ongoing market intelligence.

Explore our dedicated Uber Eats scraping guide for advanced patterns and maintenance tips.

How to Scrape Redfin Data: Complete Guide for 2026

AlterLab — Wed, 08 Jul 2026 15:20:54 +0000

TL;DR

Scrape Redfin with AlterLab API. Send a request with the target URL. Receive JSON response.

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

Why collect real-estate data from Redfin

Market research teams monitor price trends. Investors track inventory changes. Analysts build property databases. Each use case needs fresh public listings.

Technical challenges

Real‑estate sites like Redfin enforce anti-bot rules. They check headers, rate limits, and JavaScript execution. Simple curl calls often fail. Use AlterLab smart rendering API to bypass these hurdles.

Quick start with AlterLab API

Follow the Getting started guide for setup. Then run a Python script or a Node.js snippet.

Python example

```python title="scrape_redfin-com.py" {3-5}

client = alterlab.Client("YOUR_API_KEY")
response = client.scrape("https://www.redfin.com/city/32/DCA/virginia")
print(response.text)




Node.js example


```javascript title="scrape_redfin-com.js" {3-5}

const client = new AlterLab({ apiKey: "YOUR_API_KEY" });
const response = await client.scrape("https://www.redfin.com/city/32/DCA/virginia");
console.log(response.text);

cURL example

```bash title="Terminal" {3-5}
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-d '{"url": "https://www.redfin.com/city/32/DCA/virginia"}'




## Extracting structured data
Identify key elements on a Redfin listing page. Common public fields include title, price, rating, and description. Use CSS selectors that match these elements. Example selectors: .heading-location, .price, .rating, .description.

## Structured JSON extraction with Cortex
Cortex extracts typed JSON without manual parsing. Define a schema that matches the fields you need. The API returns clean data ready for analysis.



```python title="extract_redfin-com_structured.py"

client = alterlab.Client("YOUR_API_KEY")
result = client.extract(
    url="https://www.redfin.com/city/32/DCA/virginia",
    schema={
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "price": {"type": "number"},
            "rating": {"type": "number"},
            "description": {"type": "string"}
        }
    }
)
print(result.data)

Cost breakdown

Tier	Use Case	Cost per Request	Cost per 1,000	Requests per $1
T1 - Curl	Static HTML, no JS needed	$0.0002	$0.20	5,000
T2 - HTTP	Standard pages with headers	$0.0003	$0.30	3,333
T3 - Stealth	Protected pages, anti-bot active	$0.002	$2.00	500
T4 - Browser	Full JS rendering required	$0.004	$4.00	250
T5 - CAPTCHA	CAPTCHA solving + JS rendering	$0.02	$20.00	50

For Redfin start at T1. If the call fails AlterLab upgrades the tier automatically. You only pay for the tier that succeeds. See pricing details at AlterLab pricing.

Best practices

Throttle your calls. Respect rate limits set by the target site. Use the AlterLab monitoring endpoint to track scrape health. Store results in a durable location. Review the site’s Terms of Service before large scale collection.

Scaling up

Create schedules with cron expressions. Use webhook destinations to receive results in real time. Process large datasets in chunks to avoid memory spikes. Monitor API usage to stay within budget.

Key takeaways

Scraping Redfin works best with AlterLab. Choose the right tier for the page complexity. Follow legal and technical guidelines. Your pipeline will stay stable and affordable.

Related resources

Explore the full Redfin scraping guide at Redfin scraping guide.

Quick start with AlterLab API

Getting started guide walks you through API key creation. Use the Python or Node.js snippets above to test a single URL. For batch jobs integrate the Smart Rendering API to handle dynamic content.

How to Give Your AI Agent Access to AngelList Data

AlterLab — Tue, 07 Jul 2026 11:35:47 +0000

TL;DR

Extract AngelList job listings with AlterLab’s Extract API or Search API to get clean JSON for your agent, handle anti bot automatically, and feed results directly into your LLM context window.

Why AI agents need AngelList data

AI agents that monitor startup ecosystems can use AngelList data for several practical tasks. Job postings reveal hiring trends and emerging technologies. Founder activity signals market interest and potential investment targets. Investor watchlists can be built from publicly listed fund activity. These use cases feed directly into RAG pipelines and knowledge base updates.

Why raw HTTP requests fail for agents

Direct HTTP calls to AngelList often trigger rate limiting and bot detection. The site uses JavaScript rendering for many job listings, meaning a simple GET request returns an empty page until the client executes scripts. Agents that attempt to scrape without a headless browser see many empty responses and must retry, consuming extra tokens and time. CAPTCHAs appear when request patterns look automated, forcing manual intervention or additional workarounds. Without a dedicated proxy pool, the source IP can be blocked after a few hundred requests, halting the pipeline entirely. All of this creates unpredictable latency and unnecessary cost.

Connecting your agent to AngelList via AlterLab

AlterLab provides two paths for agentic access. Use the Extract API for structured output or the Scrape API for raw HTML when you need full control. Both endpoints automatically rotate proxies across residential IP pools, handle CAPTCHA solving, and retry failed requests internally. The Extract API returns JSON that matches a schema you define, eliminating the need for downstream parsing. The Scrape API returns the full HTML payload, which you can process further if you need custom extraction logic. Because the service manages anti bot protection, agents can focus on data consumption rather than bot evasion.

Structured extraction example

```python title="agent_angellist-com.py" {3-7}

client = alterlab.Client("YOUR_API_KEY")

Structured extraction — get clean data without parsing HTML

result = client.extract(
url="https://angellist.com/example-page",
schema={"title": "string", "price": "string", "description": "string"}
)
print(result.data) # Clean structured dict, ready for your LLM




### Raw HTML fetch example


```bash title="Terminal" {2-5}
curl -X POST https://api.alterlab.io/api/v1/extract/templates/{template_id} \
  -H "X-API-Key: YOUR_KEY" \
  -d '{"url": "https://angellist.com/example-page", "schema": {"title": "string", "price": "string"}}'

See the quickstart guide for installation instructions. Refer to the Extract API docs for schema options. Check the pricing page for cost details.

Using the Search API for AngelList queries

Search API lets agents query AngelList with natural language filters. You can specify filters such as location, funding stage, or industry to narrow results. An example query for “seed stage SaaS founders in New York” returns a list of matching profiles with titles, URLs, and basic metadata. The response includes a stable schema that lists each result’s ID, name, and tags. Pagination is supported via a cursor parameter, allowing you to retrieve large result sets in batches. Rate limits are enforced per API key, so you should implement back‑off logic when approaching the quota.

MCP integration

AlterLab offers an MCP server that integrates with Claude Cursor and GPT agents. Add the server URL to your agent configuration to call AlterLab tools natively. Documentation and quickstart guides are available at the AlterLab for AI Agents page.

Building a startup job market monitoring pipeline

An end‑to‑end pipeline might look like this. The agent requests data from AngelList through the service. The service returns structured JSON that matches a schema you define, such as job title, location, and tags. The agent passes this JSON to an LLM for summarization, sentiment analysis, or extraction of key signals. The resulting insight can be stored in a vector database for similarity search later. Schedule the function with a cron expression to run daily, and use monitoring to detect changes in job count or new postings. When a change is detected, trigger a notification or update the knowledge base automatically.

```python title="pipeline_example.py" {4-9}

client = alterlab.Client("YOUR_API_KEY")

def get_jobs():
resp = client.extract(
url="https://angellist.com/startups",
schema={"title": "string", "location": "string", "tags": "string"}
)
return resp.data

jobs = get_jobs()

Send jobs to an LLM for summarization

Store embeddings in a vector database

Update knowledge base on schedule




Schedule the function with a cron expression to run daily, and use monitoring to detect changes in job count or new postings. When a change is detected, trigger a notification or update the knowledge base automatically.

## Key takeaways
Agents can access AngelList data reliably through the Extract and Search APIs. Structured output reduces parsing overhead and token waste, keeping cost predictable. MCP integration simplifies tool calling for major LLM platforms such as Claude, GPT, and Gemini. Always respect robots.txt and rate limits when scaling scrapes, and monitor response health to avoid pipeline breaks.

<div data-infographic="stats">
  <div data-stat data-value="99.2%" data-label="Request Success Rate"></div>
  <div data-stat data-value="<1s" data-label="Avg Structured Response"></div>
  <div data-stat data-value="0" data-label="HTML Parsing Required"></div>
</div>

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Agent requests data" data-description="LLM agent calls AlterLab tool with target URL"></div>
  <div data-step data-number="2" data-title="AlterLab fetches plus extracts" data-description="Handles anti bot and returns structured JSON"></div>
  <div data-step data-number="3" data-title="Agent uses clean data" data-description="No parsing, no retries, data goes straight to LLM context"></div>
</div>

<div data-infographic="try-it" data-url="https://angellist.com" data-description="Extract structured AngelList data for your AI agent"></div>

Building a Scalable Proxy Rotation System for AI Agents

AlterLab — Tue, 07 Jul 2026 11:05:47 +0000

TL;DR

A scalable proxy rotation system for AI agents continuously monitors tunnel health, automatically removes unhealthy endpoints, and switches to fresh proxies without interrupting the agent’s workflow. This approach keeps success rates high and reduces manual intervention.

Why Proxy Rotation Matters for AI Agents

AI agents often perform repetitive HTTP(S) requests to gather data, interact with APIs, or drive headless browsers. Target websites employ rate limiting, IP‑based blocking, or bot detection that can halt an agent after a handful of requests. By rotating proxies, each request appears to come from a different IP address, spreading the load and lowering the probability of triggering defenses.

A robust rotation system does more than randomly pick an IP. It must:

Verify that a proxy is actually reachable and returns expected content.
Remove failing proxies quickly to avoid wasted attempts.
Re‑introduce proxies after a cool‑down period, assuming they may recover.
Provide low‑latency failover so the agent experiences minimal delay.

Core Components

Proxy Pool – a mutable list of candidate endpoints (HTTP/HTTPS or SOCKS5).
Health Checker – a background process that periodically tests each proxy.
Selector – chooses the next proxy based on health scores and recent usage.
Failover Handler – switches to an alternative proxy when a request fails or times out.
Metrics Store – records latency, success/failure counts, and timestamps for each proxy.

Health Verification Techniques

A proxy is considered healthy if it meets three criteria:

Connectivity: TCP handshake succeeds within a timeout (e.g., 2 s).
Response Validity: A simple GET request to a known endpoint (like https://httpbin.org/ip) returns the expected IP address.
Performance: Average latency over the last N checks stays below a threshold (e.g., 500 ms).

If any check fails, the proxy’s health score is decremented. After M consecutive failures, it is marked unhealthy and removed from the active pool. Successful checks increment the score, allowing recovery.

Example Health Checker (Python)

```python title="health_checker.py" {2-8}

from typing import List, Dict

class ProxyHealthChecker:
def init(self, proxies: List[str], test_url: str = "https://httpbin.org/ip"):
self.proxies = proxies
self.test_url = test_url
self.scores: Dict[str, int] = {p: 10 for p in proxies}
self._session: aiohttp.ClientSession | None = None

async def start(self):
    self._session = aiohttp.ClientSession()
    asyncio.create_task(self._run())

async def _run(self):
    while True:
        await asyncio.gather(*[self._check(p) for p in self.proxies])
        await asyncio.sleep(30)  # check every 30 s

async def _check(self, proxy: str):
    try:
        timeout = aiohttp.ClientTimeout(total=5)
        async with self._session.get(self.test_url, proxy=proxy, timeout=timeout) as resp:
            if resp.status == 200:
                data = await resp.json()
                if data.get("origin"):  # basic validity
                    self.scores[proxy] = min(self.scores[proxy] + 1, 20)
                else:
                    self.scores[proxy] = max(self.scores[proxy] - 2, 0)
            else:
                self.scores[proxy] = max(self.scores[proxy] - 2, 0)
    except Exception:
        self.scores[proxy] = max(self.scores[proxy] - 2, 0)

def get_healthy(self, threshold: int = 10) -> List[str]:
    return [p for p, s in self.scores.items() if s >= threshold]



The checker maintains a score per proxy; the selector uses `get_healthy()` to fetch the current viable list.

## Selector and Failover Logic
The selector picks the proxy with the highest health score that hasn’t been used recently (to balance load). When a request raises an exception or returns a non‑2xx status, the failover handler instantly retries with the next candidate.

### Example Selector
python
title="selector.py" {2-6}
def select_proxy(healthy: List[str], used_recently: set[str]) -> str | None:
    candidates = [p for p in healthy if p not in used_recently]
    if not candidates:
        return healthy[0] if healthy else None
    # simple round‑robin among candidates; replace with weighted choice if desired
    return candidates[0]


```python
Used proxies are tracked in a short‑lived set (e.g., last 5 requests) to avoid hammering the same endpoint.

## Putting It All Together – Request Wrapper
The wrapper combines the health checker, selector, and failover logic into a single `fetch` function that AI agents can call.

python
title="agent_fetch.py" {3-12}

from health_checker import ProxyHealthChecker
from selector import select_proxy

class ResilientFetcher:
    def __init__(self, proxy_list: List[str]):
        self.checker = ProxyHealthChecker(proxy_list)
        self.used_recently: set[str] = set()
        self.max_retry = 3

    async def start(self):
        await self.checker.start()

    async def fetch(self, url: str) -> str:
        attempt = 0
        while attempt < self.max_retry:
            healthy = self.checker.get_healthy()
            proxy = select_proxy(healthy, self.used_recently)
            if not proxy:
                raise RuntimeError("No healthy proxies available")

            try:
                timeout = aiohttp.ClientTimeout(total=10)
                async with aiohttp.ClientSession() as session:
                    async with session.get(url, proxy=proxy, timeout=timeout) as resp:
                        if resp.status == 200:
                            self.used_recently.add(proxy)
                            if len(self.used_recently) > 5:
                                self.used_recently.pop()
                            return await resp.text()
                        # treat non‑2xx as failure for this proxy
                        raise aiohttp.ClientError(f"Bad status {resp.status}")
            except Exception as exc:
                # penalize the proxy and try next
                self.checker.scores[proxy] = max(self.checker.scores[proxy] - 3, 0)
                attempt += 1
                await asyncio.sleep(0.5 * attempt)  # brief backoff
        raise RuntimeError(f"Failed to fetch {url} after {self.max_retry} attempts")

The fetcher updates the used‑recently set to spread load and relies on the health checker’s scores to avoid bad proxies.

Equivalent cURL Workflow

For environments where a Python SDK isn’t available, the same logic can be scripted with shell commands and a small state file.

bash
title="fetch_with_proxy.sh" {2-9}

!/usr/bin/env scroll="nowrap"

PROXY_FILE="/tmp/proxies.list"
SCORE_FILE="/tmp/proxy_scores.txt"
TEST_URL="https://httpbin.org/ip"
TARGET_URL="$1"

Initialize scores if missing

if [[ ! -f $SCORE_FILE ]]; then
while IFS= read -r line; do
echo "$line 10" >> $SCORE_FILE
done < $PROXY_FILE
fi

get_healthy() {
awk '$2 >= 10 {print $1}' $SCORE_FILE
}

update_score() {
local proxy=$1 delta=$2
awk -v p="$proxy" -v d="$delta" '
$1==p { $2+=d; if($2<0) $2=0; if($2>20) $2=20 }
{ print }
' $SCORE_FILE > "${SCORE_FILE}.tmp" && mv "${SCORE_FILE}.tmp" $SCORE_FILE
}

attempt=0
MAX_ATTEMPTS=3
while (( attempt < MAX_ATTEMPTS )); do
healthy=$(get_healthy)
if [[ -z $healthy ]]; then
echo "No healthy proxies" >&2; exit 1
fi
# pick first healthy line (round‑robin)
proxy=$(echo "$healthy" | head -n1)

resp=$(curl -s -x "$proxy" --max-time 10 -w "%{http_code}" "$TARGET_URL" -o /dev/null || echo "000")
if [[ $resp -eq 200 ]]; then
    # success: boost score
    update_score "$proxy" 2
    echo "Success via $proxy"
    exit 0
else
    # failure: penalize
    update_score "$proxy" -3
    ((attempt++))
    sleep $((attempt * 1))
fi

done
echo "All attempts failed" >&2
exit 1



This script mirrors the Python version: it reads a proxy list, maintains scores, picks a healthy endpoint, and retries on failure.

## Infographic: Proxy Rotation Flow
<div data-infographic="steps">
  <div data-step data-number="1" data-title="Initialize Proxy List" data-description="Load HTTP/SOCKS5 endpoints from config or service discovery."></div>
  <div data-step data-number="2" data-title="Run Health Checks" data-description="Background pings test connectivity, latency, and IP echo validity."></div>
  <div data-step data-number="3" data-title="Score Update" data-description="Successful checks increase score; failures decrease it."></div>
  <div data-step data-number="4" data-title="Select Proxy" data-description="Choose highest‑scoring unused proxy for the next request."></div>
  <div data-step data-number="5" data-title="Execute Request" data-description="Send the agent’s HTTP request via the selected proxy."></div>
  <div data-step data-number="6" data-title="Evaluate Outcome" data-description="On success, mildly boost score; on failure, penalize and retry with alternate proxy."></div>
</div>

## TryIt: Test Proxy Health with AlterLab
To see how a managed scraping API handles proxy rotation and health verification, you can run a quick test against a sample endpoint.
<div data-infographic="try-it" data-url="https://example.com" data-description="Try scraping this page with AlterLab"></div>

## Integration Tips
- **Pool Size**: Maintain at least 3× the expected concurrent request count to ensure healthy fallbacks.
- **Test URL Choice**: Use a lightweight, reliable endpoint that returns the caller’s IP (e.g., `https://httpbin.org/ip`). Avoid heavy pages that add noise to health checks.
- **Metrics Export**: Expose scores and latency via Prometheus or similar to observe trends and tune thresholds.
- **Legal Compliance**: Only rotate proxies for accessing publicly available data; respect each site’s terms of service and rate‑limit policies.

## Why a Managed Service Helps
Building and operating a health‑checking layer at scale demands continuous monitoring, fast failure detection, and intelligent rerouting. Services like AlterLab’s [web scraping API](https://alterlab.io/web-scraping-api-python) already embed automatic proxy rotation, tunnel health verification, and smart retry logic, letting engineers focus on the data extraction logic rather than networking plumbing. For quick experimentation, the [Python SDK](https://alterlab.io/web-scraping-api-python) provides a ready‑to‑use client, while the [API reference](https://alterlab.io/docs) details all available parameters.

## Takeaway
A scalable proxy rotation system combines continuous health verification, intelligent scoring, and fast failover to keep AI agents productive despite anti‑bot measures. By separating concerns—pool management, health checking, selection, and request handling failures—you create a resilient layer that can be tuned, observed, and replaced with a managed offering as your needs evolve. Start small, measure success rates, and iterate on thresholds to achieve the reliability your pipelines demand.

How to Scrape Best Buy Data: Complete Guide for 2026

AlterLab — Mon, 06 Jul 2026 15:05:47 +0000

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

Scrape Best Buy product pages using AlterLab's API with Python or Node.js. Start at T1 tier, let auto-escalation handle anti-bot, and extract structured data via CSS selectors or Cortex AI. For typical product pages, expect T2-T3 tiers ($0.0003-$0.002/request).

Why collect e-commerce data from Best Buy?

Best Buy's public product pages offer valuable signals for:

Price monitoring: Track competitor pricing strategies across electronics categories
Inventory analysis: Gauge stock levels and product availability trends
Review aggregation: Collect customer sentiment for market research

Technical challenges

Best Buy implements standard e-commerce anti-bot measures including rate limiting, IP reputation checks, and JavaScript rendering requirements. Raw HTTP requests often fail with 403/429 responses or incomplete HTML. AlterLab's Smart Rendering API automatically handles proxy rotation, header management, and headless browser rendering to access public product data reliably.

Quick start with AlterLab API

See the Getting started guide for setup. Below are examples scraping a Best Buy product page:

```python title="scrape_bestbuy-com.py" {3-5}

client = alterlab.Client("YOUR_API_KEY")
response = client.scrape("https://www.bestbuy.com/site/apple-iphone-15-pro-128gb-black-titanium/6501342.p")
print(response.text)






```javascript title="scrape_bestbuy-com.js" {3-5}

const client = new AlterLab({ apiKey: "YOUR_API_KEY" });
const response = await client.scrape("https://www.bestbuy.com/site/apple-iphone-15-pro-128gb-black-titanium/6501342.p");
console.log(response.text);

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-d '{"url": "https://www.bestbuy.com/site/apple-iphone-15-pro-128gb-black-titanium/6501342.p"}'




## Extracting structured data
AlterLab returns raw HTML by default. Use CSS selectors to extract specific public data points:



```python title="parse_bestbuy-com.py"

from parsel import Selector

client = alterlab.Client("YOUR_API_KEY")
html = client.scrape("https://www.bestbuy.com/site/apple-iphone-15-pro-128gb-black-titanium/6501342.p").text
selector = Selector(text=html)

title = selector.css(".sku-header h1::text").get()
price = selector.css(".price-current .sr-only::text").get()
rating = selector.css(".rating-stars::attr('aria-label')").get()

print({"title": title.strip() if title else None, 
       "price": price, 
       "rating": rating})

Structured JSON extraction with Cortex

For typed data without parsing HTML, use AlterLab's Cortex AI extraction:

```python title="extract_bestbuy-com_structured.py"

client = alterlab.Client("YOUR_API_KEY")
result = client.extract(
url="https://www.bestbuy.com/site/apple-iphone-15-pro-128gb-black-titanium/6501342.p",
schema={
"type": "object",
"properties": {
"title": {"type": "string"},
"price": {"type": "number"},
"rating": {"type": "number"},
"availability": {"type": "string"},
"sku": {"type": "string"}
}
}
)
print(result.data) # Typed JSON output: {"title": "...", "price": 999.99, ...}




## Cost breakdown
AlterLab auto-escalates tiers — start at T1 and pay only for the tier that succeeds. For Best Buy product pages (standard anti-bot protections), expect T2 or T3 tiers. See [AlterLab pricing](/pricing) for full details.

| Tier | Use Case | Cost per Request | Cost per 1,000 | Requests per $1 |
|------|----------|-----------------|----------------|------------------|
| T1 — Curl | Static HTML, no JS needed | $0.0002 | $0.20 | 5,000 |
| T2 — HTTP | Standard pages with headers | $0.0003 | $0.30 | 3,333 |
| T3 — Stealth | Protected pages, anti-bot active | $0.002 | $2.00 | 500 |
| T4 — Browser | Full JS rendering required | $0.004 | $4.00 | 250 |
| T5 — CAPTCHA | CAPTCHA solving + JS rendering | $0.02 | $20.00 | 50 */

<div data-infographic="stats">
  <div data-stat data-value="99999.2%" data-label="Success Rate"></div>
  <div data-stat data-value="1.2s" data-label="Avg Response"></div>
  <div data-stat data-value="$0.002" data-label="Per Request (T3)"></div>
</div>

## Best practices
- **Rate limiting**: Start with 1 request/second, adjust based on response headers
- **Robots.txt**: Check `https://www.bestbuy.com/robots.txt` for crawl delays and disallowed paths
- **Dynamic content**: Use Cortex AI or wait for network idle state instead of fixed timeouts
- **Error handling**: Implement retry logic with exponential backoff for 429/5xx responses
- **Data freshness**: For price monitoring, schedule requests during off-peak hours (2-5 AM local store time)

## Scaling up
For large-scale data collection:
- **Batch requests**: Use AlterLab's batch endpoint (up to 100 URLs/request)
- **Scheduling**: Implement cron-based scrapes via AlterLab's scheduling feature
- **Storage**: Stream results directly to data warehouses or cloud storage
- **Monitoring**: Set up alerts for failed requests or data anomalies

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Configure scraper" data-title="Configure scraper" data-description="Set URL, parameters, and extraction schema"></div>
  <div data-step data-number="2" data-title="Execute request" data-description="AlterLab handles tier selection and anti-bot"></div>
  <div data-step data-number="3" data-title="Process results" data-description="Store structured JSON or trigger webhooks"></div>
</div>

## Key takeaways
- AlterLab simplifies Best Buy scraping by managing anti-bot, proxies, and rendering
- Extract public product data via CSS selectors or Cortex AI for type-safe JSON
- Costs scale with complexity: $0.0002-$0.004/request depending on required tier
- Always comply with robots.txt, rate limits, and Terms of Service
- See the [Best Buy scraping guide](/scrape/best-buy) for advanced patterns

<div data-infographic="try-it" data-url="https://bestbuy.com" data-description="Try scraping Best Buy with AlterLab"></div>
Hit reply if you have questions.

Designing a Fault-Tolerant Proxy Rotation Wrapper

AlterLab — Mon, 06 Jul 2026 10:55:39 +0000

TL;DR

A fault-tolerant proxy rotation wrapper ensures high success rates by validating proxy tunnels via a lightweight health check before passing them to a headless browser. This architecture prevents browser timeouts, reduces resource waste, and ensures that only active, high-quality IPs are used for data extraction.

The Problem with Naive Proxy Rotation

Most basic proxy implementations follow a "try-and-fail" pattern. The application picks a random proxy from a list, attempts to load a page, and if the request fails, it catches the exception and tries again.

For headless browsers (Playwright, Puppeteer, Selenium), this is inefficient. Loading a full browser instance and navigating to a URL is resource-heavy. When a proxy is dead, the browser often hangs for 30 to 60 seconds before hitting a timeout. In a high-concurrency environment, this leads to "zombie" browser processes and massive latency spikes.

A professional architecture separates tunnel verification from page execution.

Designing the Health-Check Wrapper

The goal is to create a layer that acts as a gatekeeper. Instead of the browser requesting a page directly, it requests a "healthy" proxy from the wrapper. The wrapper verifies the proxy's connectivity and latency before returning it.

The Verification Logic

A healthy proxy must meet three criteria before it is deemed viable:

TCP Connectivity: The tunnel must be open.
HTTP Response: The proxy must return a valid status code from a neutral endpoint.
Latency Threshold: The response time must be under a specific limit (e.g., < 2000ms) to prevent slow-loading pages.

Implementation in Python

The following implementation uses a ProxyManager class to maintain a pool of proxies and a verify_proxy method to ensure health.

```python title="proxy_manager.py" {15-24}

from collections import deque

class ProxyManager:
def init(self, proxy_list, timeout=2):
self.pool = deque(proxy_list)
self.timeout = timeout
self.unhealthy_proxies = {}

def verify_proxy(self, proxy):
    """Check if a proxy is healthy using a lightweight request."""
    try:
        # Use a neutral, fast endpoint for verification
        response = requests.get(
            "https://httpbin.org/ip", 
            proxies={"http": proxy, "https": proxy}, 
            timeout=self.timeout
        )
        return response.status_code == 200
    except requests.RequestException:
        return False

def get_healthy_proxy(self):
    """Rotates and verifies proxies until a healthy one is found."""
    attempts = 0
    max_attempts = len(self.pool)

    while attempts < max_attempts:
        proxy = self.pool[0]
        self.pool.rotate(-1)  # Move to end of list

        # Check if proxy is currently in cooldown
        if proxy in self.unhealthy_proxies:
            if time.time() < self.unhealthy_proxies[proxy]:
                attempts += 1
                continue

        if self.verify_proxy(proxy):
            return proxy

        # Mark as unhealthy for 5 minutes
        self.unhealthy_proxies[proxy] = time.time() + 300
        attempts += 1

    return None




## Integrating with Headless Browsers

Once you have a verified proxy, you can inject it into your browser launch configuration. This ensures that your browser session starts with a known-good connection, reducing the chance of an immediate failure.



```python title="scraper.py" {8-12}
from playwright.sync_api import sync_playwright
from proxy_manager import ProxyManager

proxies = ["http://user:pass@ip1:port", "http://user:pass@ip2:port"]
manager = ProxyManager(proxies)

def run_scrape(url):
    proxy_server = manager.get_healthy_proxy()
    if not proxy_server:
        print("No healthy proxies available")
        return

    with sync_playwright() as p:
        browser = p.chromium.launch(
            proxy={"server": proxy_server}, 
            headless=True
        ) # Start browser with verified proxy
        page = browser.new_page()
        page.goto(url)
        print(page.title())
        browser.close()

run_scrape("https://example.com")

Handling Advanced Anti-Bot Systems

While proxy rotation handles connectivity, it does not solve fingerprinting or sophisticated behavioral analysis. High-value targets often employ anti-bot handling that detects headless browsers regardless of the IP address.

To scale this, you need to combine proxy rotation with:

Header Randomization: Rotating User-Agents and Accept-Language headers.
TLS Fingerprinting: Ensuring the TLS handshake matches a real browser.
JS Rendering: Using a browser that can execute JavaScript to prove it is not a simple script.

For those who prefer not to manage the infrastructure of health-checks and rotations, utilizing a Python scraping API abstracts the entire proxy management layer, providing a single endpoint that handles rotation and verification internally.

Performance Comparison

Using a pre-verification wrapper significantly changes the failure profile of a scraping pipeline.

Metric	Naive Rotation	Health-Check Wrapper
Avg. Request Latency	High (due to timeouts)	Low (pre-verified)
Browser Resource Use	Wasted on dead IPs	Optimized
Success Rate	Variable	Consistent

Managing State and Cooldowns

In a production system, you should not just rotate proxies but also track their "health score." If a proxy fails three times in a row on the target site (even if it passes the httpbin health check), it should be flagged as "site-blocked" and put into a longer cooldown period.

The Circuit Breaker Pattern

Implementing a circuit breaker prevents your system from hammering a target site with IPs that are already flagged. If your failure rate exceeds a certain threshold (e.g., 20% over 100 requests), the circuit "opens," and the system pauses all requests for a set duration to avoid a total IP range ban.

Key Takeaways

Decouple Verification: Never let the headless browser be the primary tool for testing proxy health.
Use Lightweight Pings: Use fast, neutral endpoints to verify tunnels before assigning them to heavy browser instances.
Implement Cooldowns: Track unhealthy proxies in a dictionary with timestamps to avoid repeated attempts on dead tunnels.
Combine Layers: Pair your rotation wrapper with TLS fingerprinting and JS rendering for the highest success rates.

How to Scrape Stack Overflow Data in 2026

AlterLab — Sun, 05 Jul 2026 14:53:45 +0000

TL;DR

Scrape stack overflow with Python, Node.js, or cURL via the AlterLab API. Use T1 for static pages, T3 for protected content, and Cortex for structured JSON extraction.

Why collect developer data from Stack Overflow?

Market research, price monitoring, and analytical dashboards often rely on publicly listed questions, answers, and tags. The data is openly available and can inform product decisions without violating access rules.

Technical challenges

Stack Overflow enforces rate limits and delivers much of its content through JavaScript. Simple HTTP requests fail on heavy query patterns or on pages that load content dynamically. To handle these realities, use the Smart Rendering API for full page rendering and automatic bot detection mitigation.

Quick start with AlterLab API

Create an account and obtain an API key. Then follow the Getting started guide at /docs/quickstart/installation to install the SDK. Below are minimal examples in Python, Node.js, and cURL that target a public question page.

```python title="scrape_stackoverflow-com.py" {3-5}

client = alterlab.Client("YOUR_API_KEY")
response = client.scrape("https://stackoverflow.com/questions")
print(response.text)






```javascript title="scrape_stackoverflow-com.js" {2-4}

const client = new AlterLab({ apiKey: "YOUR_API_KEY" });
const response = await client.scrape("https://stackoverflow.com/questions");
console.log(response.text);

```bash title="Terminal" {3}
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-d '{"url": "https://stackoverflow.com/questions"}'




## Extracting structured data
Public pages expose predictable HTML structures. For example, question titles use `<h1 class="question-title">`, while answer counts appear in `<div class="answer-count">`. Use CSS selectors that match these classes to pull the exact fragments you need.

## Structured JSON extraction with Cortex
Cortex simplifies schema‑driven extraction. The following Python sample pulls a question’s title, score, and answer count into a typed JSON object.



```python title="extract_stackoverflow-com_structured.py" {3-6}

client = alterlab.Client("YOUR_API_KEY")
result = client.extract(
    url="https://stackoverflow.com/questions",
    schema={
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "score": {"type": "number"},
            "answer_count": {"type": "number"}
        }
    }
)
print(result.data)

Cost breakdown

Pricing depends on the tier you select. The table below shows cost per request and per 1,000 requests.

Tier	Use Case	Cost per Request	Cost per 1,000	Requests per $1
T1 — Curl	Static HTML, no JS needed	$0.0002	$0.20	5,000
T2 — HTTP	Standard pages with headers	$0.0003	$0.30	3,333
T3 — Stealth	Protected pages, anti‑bot active	$0.002	$2.00	500
T4 — Browser	Full JS rendering required	$0.004	$4.00	250
T5 — CAPTCHA	CAPTCHA solving + JS rendering	$0.02	$20.00	50

Stack Overflow’s dynamic nature typically requires T3 or higher. AlterLab auto‑escalates tiers automatically; you only pay for the tier that succeeds. See the full AlterLab pricing details at /pricing.

Best practices

Respect robots.txt and any posted usage limits. Limit request frequency to avoid triggering rate‑limit defenses. When targeting pages with heavy query load, start at T1 and let the system upgrade as needed. Always handle failures gracefully and log response codes for debugging.

Scaling up

For large projects, batch requests using cron schedules or the Scheduler feature. Store results in a durable bucket and process them in parallel workers. Monitor success rates and adjust min_tier settings to control costs while maintaining reliability.

Key takeaways

Use the AlterLab API for reliable access to public Stack Overflow data.
Choose a tier that matches the page’s rendering needs; the system upgrades automatically.
Extract structured JSON with Cortex to avoid manual parsing.
Keep requests polite, stay within rate limits, and review the site’s Terms of Service.
Consult the related guide at /scrape/stack-overflow for deeper examples and patterns.

How to Give Your AI Agent Access to PubMed Data

AlterLab — Sun, 05 Jul 2026 13:23:44 +0000

How to Give Your AI Agent Access to PubMed Data

TL;DR: Equip your AI agent with structured PubMed data by using AlterLab's Extract API to bypass anti-bot measures and return clean JSON. This enables reliable medical research monitoring, clinical trial tracking, and biotech intelligence without parsing HTML or managing proxies.

Disclaimer: This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access.

Why AI agents need PubMed data

AI agents in healthcare and life sciences require current PubMed data for:

Medical research monitoring: Tracking new publications on specific diseases or treatments to update knowledge bases.
Clinical trial tracking: Identifying emerging trial results or protocol changes for real-time intelligence.
Biotech intelligence: Monitoring competitor research, grant publications, and emerging science for strategic decisions.

Why raw HTTP requests fail for agents

Direct requests to PubMed often fail for agents due to:

Rate limiting: PubMed blocks IPs exceeding request thresholds, causing failed tool calls.
JavaScript rendering: Dynamic content (like abstracts loaded via JS) returns incomplete HTML to naive scrapers.
Bot detection: Advanced anti-bot systems challenge requests with CAPTCHAs, wasting agent context windows on retries.
Token budget waste: Failed requests consume LLM tokens without yielding usable data, increasing costs and reducing pipeline reliability.

Connecting your agent to PubMed via AlterLab

Use AlterLab's Extract API (Extract API docs) to get structured data from PubMed pages. This handles anti-bot bypass, JavaScript rendering, and returns clean JSON ready for your LLM.

Getting started guide shows how to install the AlterLab SDK. Here’s a Python example extracting structured data from a PubMed article:

```python title="pubmed_extract_agent.py" {3-8}

from alterlab import Client

client = Client("YOUR_API_KEY")

Define schema for PubMed article structure

schema = {
"title": "string",
"authors": "string",
"journal": "string",
"pub_date": "string",
"abstract": "string",
"doi": "string"
}

Extract structured data from a PubMed article URL

result = client.extract(
url="https://pubmed.ncbi.nlm.nih.gov/34567890/",
schema=schema
)

Result.data is a dict, ready for LLM context or RAG pipeline

print(result.data)




Equivalent cURL command:



```bash title="Terminal" {2-6}
curl -X POST https://api.alterlab.io/api/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://pubmed.ncbi.nlm.nih.gov/34567890/",
    "schema": {
      "title": "string",
      "authors": "string",
      "journal": "string",
      "pub_date": "string",
      "abstract": "string",
      "doi": "string"
    }
  }'

For raw HTML (e.g., if you need full page content), use the Scrape API (/api/v1/scrape). However, structured extraction via Extract API is recommended for agents to minimize post-processing.

Using the Search API for PubMed queries

To search PubMed for articles matching a query, use AlterLab's Search API (/api/v1/search). This returns structured search results without needing to parse PubMed's search page.

```python title="pubmed_search_agent.py" {3-8}

from alterlab import Client

client = Client("YOUR_API_KEY")

Search PubMed for recent articles on cancer immunotherapy

search_params = {
"query": "cancer immunotherapy 2024",
"site": "pubmed.ncbi.nlm.nih.gov",
"num_results": 10
}

response = client.search(**search_params)

Response contains structured list of articles

for article in response.data:
print(f"{article['title']} - {article['journal']}")






```bash title="Terminal" {2-7}
curl -X POST https://api.alterlab.io/api/v1/search \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "cancer immunotherapy 2024",
    "site": "pubmed.ncbi.nlm.nih.gov",
    "num_results": 10
  }'

MCP integration

AlterLab provides an MCP server that lets Claude, GPT, or Cursor agents call web data extraction as a native tool. This simplifies agent configuration by abstracting API keys and request handling.

See the AI agent tutorial to set up the MCP server and integrate it with your agent framework.

Building a medical research monitoring pipeline

Here’s an end-to-end example of an agent monitoring PubMed for new diabetes research:

Agent triggers a daily search for "type 2 diabetes treatment 2024" via AlterLab's Search API.
For each new article (comparing against a known ID set), the agent extracts structured data (title, abstract, DOI) using the Extract API.
The agent summarizes key findings and updates a medical knowledge base in vector store for RAG.
If high-impact findings are detected (e.g., new mechanism), the agent alerts researchers via Slack.

```python title="diabetes_monitoring_pipeline.py" {5-15}

from alterlab import Client

from datetime import datetime, timedelta

Initialize client (in production, load API key from secure vault)

client = Client("YOUR_API_KEY")

Track seen articles to avoid duplicates

SEEN_ARTICLES_FILE = "seen_articles.json"

def load_seen_articles():
try:
with open(SEEN_ARTICLES_FILE) as f:
return set(json.load(f))
except FileNotFoundError:
return set()

def save_seen_articles(seen_set):
with open(SEEN_ARTICLES_FILE, "w") as f:
json.dump(list(seen_set), f)

def monitor_diabetes_research():
seen = load_seen_articles()

# Search for new diabetes articles from last 7 days
seven_days_ago = (datetime.now() - timedelta(days=7)).strftime("%Y/%m/%d")
search_query = f"type 2 diabetes treatment {seven_days_ago}[Date - Publication] : 3000[Date - Publication]"

search_response = client.search(
    query=search_query,
    site="pubmed.ncbi.nlm.nih.gov",
    num_results=20
)

new_articles = []
for article in search_response.data:
    # Create unique ID from PMID or DOI
    article_id = article.get("pmid") or article.get("doi") or hashlib.md5(article["title"].encode()).hexdigest()

    if article_id not in seen:
        seen.add(article_id)

        # Extract full structured data for new article
        extract_result = client.extract(
            url=article["url"],
            schema={
                "title": "string",
                "authors": "string",
                "journal": "string",
                "pub_date": "string",
                "abstract": "string",
                "doi": "string"
            }
        )

        new_articles.append(extract_result.data)

# Update knowledge base with new articles (pseudo-code)
if new_articles:
    update_knowledge_base(new_articles)
    save_seen_articles(seen)
    print(f"Added {len(new_articles)} new diabetes research articles to knowledge base")
else:
    print("No new articles found")

def update_knowledge_base(articles):
# In practice: embed abstracts and store in vector DB (e.g., Pinecone, Weaviate)
pass

if name == "main":
monitor_diabetes_research()




## Key takeaways

- AI agents need reliable, structured web data to function effectively in knowledge-intensive domains like healthcare.
- AlterLab eliminates anti-bot, rendering, and parsing complexity, letting agents focus on data utilization rather than data acquisition.
- Structured extraction via Extract API delivers PubMed data in LLM-ready JSON, preserving token budgets for reasoning.
- Always comply with robots.txt and rate limits; users bear responsibility for reviewing PubMed's Terms of Service.
- Scale agentic workloads efficiently with usage-based pricing—see [pricing](/pricing) for details.

How to Give Your AI Agent Access to arXiv Data

AlterLab — Sat, 04 Jul 2026 12:36:09 +0000

How to Give Your AI Agent Access to arXiv Data

This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access.

TL;DR

Give your AI agent reliable access to arXiv data by using AlterLab's Extract API for structured paper metadata or Search API for query-based retrieval. This avoids rate limits, CAPTCHAs, and HTML parsing overhead while delivering clean JSON directly to your LLM context.

Why AI agents need arXiv data

AI agents require arXiv data for three core agentic workflows: monitoring new publications in specific ML domains for RAG knowledge base updates, tracking citation networks to assess paper impact automatically, and building ML paper pipelines that trigger retraining when novel architectures appear. These use cases demand timely, structured access without manual intervention.

Why raw HTTP requests fail for agents

Direct requests to arxiv.org fail agent pipelines due to rate limiting (60 seconds/minute per IP), JavaScript-dependent content rendering that breaks simple parsers, and bot detection mechanisms triggering CAPTCHAs. Failed requests waste LLM token budgets on retries and error handling, increasing costs by 3-5x while reducing pipeline reliability below 70% success rates.

Connecting your agent to arXiv via AlterLab

AlterLab's Extract API (/api/v1/extract) returns structured arXiv data ready for LLM consumption. For raw HTML needs, use the Scrape API (/api/v1/scrape). Both handle anti-bot challenges automatically.

Structured extraction example

Extract paper metadata without parsing HTML:

```python title="agent_arxiv-org.py" {3-8}

client = alterlab.Client("YOUR_API_KEY")

Get structured data for a specific arXiv page

result = client.extract(
url="https://arxiv.org/abs/2301.00001",
schema={
"title": "string",
"authors": "array",
"abstract": "string",
"categories": "array",
"submitted_date": "string"
}
)

Feed clean data directly to your LLM

print(result.data)

Output: {"title": "Attention Is All You Need", ...}





```bash title="Terminal"
curl -X POST https://api.alterlab.io/api/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -d '{
    "url": "https://arxiv.org/abs/2301.00001",
    "schema": {
      "title": "string",
      "authors": "array",
      "abstract": "string",
      "categories": "array",
      "submitted_date": "string"
    }
  }'

Raw HTML example (when needed)

```python title="scrape_arxiv-org.py" {3-6}
result = client.scrape(
url="https://arxiv.org/list/cs.LV/recent",
formats=["html"] # Get clean HTML without JS challenges
)





```bash title="Terminal"
curl -X POST https://api.alterlab.io/api/v1/scrape \
  -H "X-API-Key: YOUR_KEY" \
  -d '{"url": "https://arxiv.org/list/cs.LV/recent", "formats": ["html"]}'

See Extract API docs for full schema options.

Using the Search API for arXiv queries

For dynamic paper discovery, AlterLab's Search API (/api/v1/search) queries arXiv through AlterLab's infrastructure:

```python title="search_arxiv-org.py" {3-7}
results = client.search(
query="large language model transformer",
site="arxiv.org",
num_results=10
)

for paper in results.data:
# Process structured search results
print(f"{paper['title']} by {paper['authors'][0]}")





```bash title="Terminal"
curl -X POST https://api.alterlab.io/api/v1/search \
  -H "X-API-Key: YOUR_KEY" \
  -d '{
    "query": "large language model transformer",
    "site": "arxiv.org",
    "num_results": 10
  }'

This bypasses arXiv's native search limitations while respecting their usage policies.

MCP integration

AlterLab provides an MCP server that exposes web data capabilities as tools for Claude, GPT, and Cursor agents. Install it to let your agent call alterlab_extract or alterlab_search as native functions. See the AlterLab for AI Agents tutorial for setup.

Building a research paper monitoring pipeline

Here's a complete agentic pipeline for tracking new diffusion model papers:

Agent triggers search: LLM agent calls AlterLab Search API for query="diffusion model" AND date:[now-7d TO now]
AlterLab returns structured data: Clean JSON with paper metadata, no HTML parsing needed
Agent evaluates relevance: LLM checks abstracts against research goals
Agent extracts full papers: For relevant papers, calls Extract API to get structured metadata
Agent updates knowledge base: Stores embeddings in vector DB for RAG
Agent schedules next run: Uses cron expression via AlterLab's scheduling feature (set min_tier=3 for JS-heavy pages)

```python title="research_pipeline.py" {5-12,18-25}

from datetime import datetime, timedelta

client = alterlab.Client("YOUR_API_KEY")

def monitor_arxiv():
# Step 1: Search for recent papers
search_result = client.search(
query="diffusion model",
site="arxiv.org",
num_results=20,
date_range=f"[(datetime.now() - timedelta(days=7)).isoformat() TO {datetime.now().isoformat()}]"
)

# Step 2: Process results
relevant_papers = []
for paper in search_result.data:
    # Step 3: LLM relevance check (simplified)
    if "transformer" in paper["abstract"].lower():
        # Step 4: Get full structured data
        full_data = client.extract(
            url=paper["link"],
            schema={"title": "string", "authors": "array", "categories": "array"}
        )
        relevant_papers.append(full_data.data)

# Step 5: Update knowledge base (pseudo-code)
if relevant_papers:
    update_vector_db(relevant_papers)

return len(relevant_papers)

Step 6: Schedule via AlterLab (would be configured in dashboard)

cron: "0 9 * * *" # Daily at 9 AM




## Key takeaways
- AI agents need reliable, structured arXiv data for research pipelines and RAG
- Direct HTTP requests fail due to anti-bot measures, wasting agent resources
- AlterLab's APIs handle extraction, search, and anti-bot challenges automatically
- Structured output eliminates HTML parsing, saving LLM tokens and reducing latency
- MCP integration lets agents call web data as native tools in Claude/GPT/Cursor
- Always comply with robots.txt and ToS when building agentic data pipelines

<div data-infographic="stats">
  <div data-stat data-value="99.2%" data-label="Request Success Rate"></div>
  <div data-stat data-value="<1s" data-label="Avg Structured Response"></div>
  <div data-stat data-value="0" data-label="HTML Parsing Required"></div>
</div>

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Agent requests data" data-description="LLM agent calls AlterLab tool with target URL"></div>
  <div data-step data-number="2" data-title="AlterLab fetches + extracts" data-description="Handles anti-bot, returns structured JSON"></div>
  <div data-step data-number="3" data-title="Agent uses clean data" data-description="No parsing, no retries — data goes straight to LLM context"></div>
</div>

<div data-infographic="try-it" data-url="https://arxiv.org/list/cs.LV/recent" data-description="Extract structured arXiv data for your AI agent"></div>

How to Give Your AI Agent Access to CNBC Data

AlterLab — Sat, 04 Jul 2026 12:21:10 +0000

TL;DR

To give an AI agent access to CNBC data, call AlterLab’s Extract API with a target URL and a schema. You receive clean JSON ready for your LLM’s context window.

Why AI agents need CNBC data

Financial news drives market signals.

Agents use it for earnings alerts.

Pipelines ingest headlines for RAG retrieval.

A single structured article can power multiple downstream decisions.

Why raw HTTP requests fail for agents

Direct GET requests hit Cloudflare challenges.

JavaScript rendering wastes token budget.

Agents see repeated 429 responses.

Each retry consumes compute and delays the pipeline.

Connecting your agent to CNBC via AlterLab

Use the Extract API to get structured output.

No HTML parsing required.

Set a schema that matches the fields you need.

```python title="agent_cnbc-com.py" {3-5}

client = alterlab.Client("YOUR_API_KEY")

Structured extraction — get clean data without parsing HTML

result = client.extract(
url="https://cnbc.com/example-page",
schema={"title": "string", "price": "string", "description": "string"}
)
print(result.data) # Clean structured dict, ready for your LLM






```bash title="Terminal" {2-4}
curl -X POST https://api.alterlab.io/api/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -d '{"url": "https://cnbc.com/example-page", "schema": {"title": "string", "price": "string"}}'

The platform handles anti‑bot protection behind the scenes.

You only pay for successful extractions.

No need to manage CAPTCHAs or rotating proxies yourself.

Using the Search API for CNBC queries

Search lets you discover articles by keyword.

It returns a list of URLs with metadata.

Pick the most relevant link and feed it to Extract.

```bash title="Terminal" {2-5}
curl -X GET https://api.alterlab.io/api/v1/search \
-H "X-API-Key: YOUR_KEY" \
-d '{"query": "quarterly earnings cnbc"}'




The response includes a `results` array.  
Each entry has a `url` field you can pass to `/extract`.

## MCP integration
Add AlterLab as a tool in your agent’s MCP server.  
Your LLM can call `alterlab_extract` without writing HTTP code.  
See the full guide at [AlterLab for AI Agents](https://alterlab.io/docs/tutorials/ai-agent).

## Building a financial news pipelines pipeline
An end‑to‑end flow looks like this:

1. Agent requests CNBC data via MCP.  
2. AlterLab fetches the page and returns structured JSON.  
3. The LLM consumes the JSON in its context window.  
4. The LLM generates a market summary.

<div data-infographic="stats">
  <div data-stat data-value="99.2%" data-label="Request Success Rate"></div>
  <div data-stat data-value="<1s" data-label="Avg Structured Response"></div>
  <div data-stat data-value="0" data-label="HTML Parsing Required"></div>
</div>

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Agent requests data" data-description="LLM agent calls AlterLab tool with target URL"></div>
  <div data-step data-number="2" data-title="AlterLab fetches + extracts" data-description="Handles anti‑bot, returns structured JSON"></div>
  <div data-step data-number="3" data-title="Agent uses clean data" data-description="No parsing, no retries — data goes straight to LLM context"></div>
</div>

<div data-infographic="try-it" data-url="https://cnbc.com" data-description="Extract structured CNBC data for your AI agent"></div>

A typical code snippet for the pipeline:



```python title="pipeline_cnbc.py" {4-8}
from alterlab import Client

client = Client("YOUR_API_KEY")

# Step 1: Search for relevant article
search_res = client.search(query="cnbc market recap")
url = search_res.results[0].url

# Step 2: Extract structured data
data = client.extract(url=url, schema={"title":"string", "summary":"string", "sentiment":"string"})
summary = data.data["summary"]

# Step 3: Feed to LLM
# llm.generate(prompt=f"Write a 2‑sentence market outlook using: {summary}")

This pattern scales to dozens of articles per hour.

Your agents stay fast and reliable.

Key takeaways

Use Extract API for clean, structured CNBC output.
Avoid raw HTML parsing; let AlterLab handle anti‑bot.
Combine Search and Extract for targeted data retrieval.
Integrate via MCP for zero‑code tool calls.
Monitor cost with AlterLab pricing as your pipeline grows.

Always review a site's robots.txt and Terms of Service before automated access. This guide covers accessing publicly available data.

How to Give Your AI Agent Access to Reuters Data

AlterLab — Fri, 03 Jul 2026 12:11:08 +0000

How to Give Your AI Agent Access to Reuters Data

TL;DR: To give an AI agent access to Reuters data, use AlterLab's Extract API to transform raw news pages into structured JSON. This bypasses JavaScript rendering and anti-bot protections, providing your LLM with clean data that fits directly into its context window.

Disclaimer: This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access.

Why AI agents need Reuters data

For an AI agent to be effective in financial or geopolitical intelligence, it cannot rely solely on its training data. Training data is static; real-world markets and political landscapes move in real-time. To build high-utility agentic workflows, you must connect them to live news sources like Reuters.

Common agentic use cases include:

News Monitoring Pipelines: Agents that monitor specific keywords (e.1., "Federal Reserve" or "semiconductor supply chain") and trigger workflows when significant news breaks.
RAG-enhanced Intelligence: Providing an LLM with the most recent news as context to prevent hallucinations and ensure responses are grounded in current events.
Event Detection & Signal Tracking: Using agents to parse news sentiment or supply chain disruptions to trigger automated actions in trading or logistics systems.

Why raw HTTP requests fail for agents

If you attempt to build a tool-calling loop where an agent uses a standard requests or fetch call to reach Reuters, your pipeline will fail almost immediately. Modern news sites employ sophisticated edge protections to prevent scraping.

Common failure points include:

JavaScript Rendering: Much of the content on Reuters is hydrated via client-side JavaScript. A basic HTTP GET request returns a nearly empty HTML shell.
Bot Detection: Servers identify the lack of browser fingerprints, leading to 403 Forbidden errors or endless CAPTCHAs.
Rate Limiting: Without rotating residential proxies, your agent's IP will be flagged after a few requests.
Token Budget Waste: Even if you successfully fetch a page, sending raw, uncleaned HTML to an LLM is expensive and fills the context window with noise (scripts, nav bars, ads) instead of signal.

Connecting your agent to Reuters via AlterLab

Instead of building a browser-based scraping-engine, you should treat data acquisition as a structured tool call. AlterLab provides two primary methods for this: the Scrape API for raw data and the Extract API for structured intelligence.

Method 1: Extracting structured news via Extract API

For most agentic workflows, you don't want HTML. You want a JSON object containing the headline, the body text, and the publication timestamp. This minimizes token usage and maximizes reasoning accuracy.

```python title="reuters_extractor.py" {2-8}

client = alterlab.Client("YOUR_API_KEY")

Extract clean news data without writing a single CSS selector

result = client.extract(
url="https://www.reuters.com/business/finance-industry/example-news-article/",
schema={
"headline": "string",
"body": "string",
"timestamp": "string",
"author": "string"
}
)

print(result.data) # Returns a clean dictionary for your LLM




Using the cURL equivalent for testing your tool definitions:



```bash title="Terminal"
curl -X POST https://api.alterlab.io/api/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://reuters.com/...",
    "schema": {
      "headline": "string",
      "body": "string"
    }
  }'

For more advanced schema definitions, refer to our Extract API docs.

Method 2: Broad search via the Search API

If your agent needs to find news rather than process a known URL, use the Search API. This allows the agent to perform a query and receive a list of relevant URLs or snippets.

```python title="agent_search.py" {4-7}

client = alterlab.Client("YOUR_API_KEY")

The agent performs a search to find recent context

search_results = client.search(
query="impact of interest rates on tech stocks",
site_limit_only="reuters.com"
)

for article in search_results.items:
print(f"Found: {article.title} at {article.url}")




## Using MCP for seamless integration

If you are building custom agents using Model Context Protocol (MCP), you can integrate AlterLab as a dedicated tool. This allows Claude or other LLM-based agents to fetch Reuters data directly within their reasoning loop without extra boilerplate code. By exposing AlterLab as an MCP server, your agent gains a "web-search" capability that returns structured,-ready data instead of messy HTML.

<div data/instruction="link_to_tutorial">
Learn how to implement this in our <a href="https://alterlab.io/docs/tutorials/ai-agent">AI Agent Guide</a>.
</div>

## Building a news monitoring pipeline

A production-grade agentic pipeline follows a specific flow: the agent identifies a need for data, triggers a tool call, receives structured JSON, and then performs reasoning.

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Agent requests data" data-description="LLM agent calls AlterLab tool with target URL or search term"></div>
  <div data-step data-number="2" data-title="AlterLab fetches + extracts" data-description="Handles TLS fingerprints, JS rendering, and returns JSON"></div>
  <div data-step data-number="3" data_description="Agent uses clean data" data-description="No parsing, no retries — data goes straight to LLM context"></div>
</div>

### Full Pipeline Implementation

Here is how a production pipeline looks when an agent is tasked with monitoring a topic:



```python title="news_monitoring_pipeline.py"

from openai import OpenAI # Or any LLM provider

# Initialize clients
llm = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
data_client = alterlab.Client(api_key=os.environ["ALTERLAB_API_KEY"])

def news_monitoring_agent(topic: str):
    # Step 1: Search for news via AlterLab
    print(f"Searching for: {topic}")
    search_results = data_client.search(query=f"latest news about {topic}", site_limit_only="reuters.com")

    if not search_results.items:
        return "No recent news found."

    # Step 2: Deep dive into the top result
    top_url = search_results.items[0].url
    print(f"Extracting content from: {top_url}")

    content = data_client.extract(
        url=top_url,
        schema={"summary": "string", "sentiment": "string", "key_entities": "list[string]"}
    )

    # Step 3: LLM Reasoning
    prompt = f"Based on this news: {content.data['summary']}, what is the sentiment toward {topic}? Entities: {content.data['key_entities']}"

    response = llm.chat.complet_messages(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )

    return response.choices[0].message.content

# Execute the agentic loop
print(news_monitoring_agent("NVIDIA earnings"))

Key takeaways

Don's scrape, extract: Don't try to parse HTML with regex or BeautifulSoup. Use the Extract API to get clean JSON that fits your agent's schema.
Handle the heavy lifting: Let the API manage JavaScript rendering,-proxy rotation, and anti-bot measures so your agent can focus on reasoning.
Optimize for context: Delivering raw HTML to an LLM is a waste of money. Always transform web data into minimal, high-signal structured formats.

Hit reply if you have questions.

AlterLab // Web Data, Simplified.

How to Give Your AI Agent Access to TechCrunch Data

AlterLab — Fri, 03 Jul 2026 12:11:07 +0000

How to Give Your AI Agent Access to TechCrunch Data

Disclaimer: This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access.

TL;DR

To give an AI agent access to TechCrunch data, connect your agent's tool-calling interface to a structured data API. By using the AlterLab Extract API, agents can request a specific URL and receive a JSON object matching a predefined schema, removing the need for the LLM to parse raw HTML or handle bot detection.

Why AI agents need TechCrunch data

For AI engineers building agentic systems, live web data is the difference between a static chatbot and a functional autonomous agent. TechCrunch serves as a primary source of truth for the technology sector, making it essential for several agentic workflows:

1. Startup News Monitoring
Agents can be programmed to monitor specific categories (e.g., "AI" or "Fintech") to identify emerging players. Instead of a human reading a feed, an agent can filter for specific keywords and summarize the impact of a new product launch in real-time.

2. Funding Round Detection
By monitoring the "Startups" section, agents can trigger workflows the moment a funding announcement is published. This allows a pipeline to automatically update a CRM, notify a venture capital team, or trigger a competitive analysis report.

3. Tech Trend Pipelines
RAG (Retrieval-Augmented Generation) pipelines often suffer from "knowledge cutoff." Giving an agent access to TechCrunch allows the LLM to ground its responses in today's news, ensuring that answers about the latest LLM releases or hardware breakthroughs are accurate and current.

Why raw HTTP requests fail for agents

Most developers attempt to give their agents web access by providing a simple requests.get() or axios.get() tool. In a production agentic pipeline, this approach fails for four specific reasons:

Rate Limiting and IP Blocking
TechCrunch employs sophisticated bot detection. When an agent makes multiple requests in rapid succession to track a trend, the server identifies the non-browser behavior and returns a 403 Forbidden or 429 Too Many Requests error.

JavaScript Rendering
Modern news sites often load content dynamically. A raw HTTP request retrieves the initial HTML shell, but the actual article content or the latest headlines may be injected via JavaScript. Without a headless browser, your agent sees an empty page.

Token Budget Waste
Feeding raw HTML into an LLM's context window is inefficient. A single TechCrunch page can contain thousands of lines of boilerplate HTML, navigation menus, and tracking scripts. This consumes thousands of tokens, increasing costs and introducing noise that leads to hallucinations.

The Retry Loop
When an agent hits a CAPTCHA or a block, the LLM often attempts to "fix" the problem by retrying the request or changing the URL. This creates an infinite loop that drains your API budget without ever retrieving the data.

Connecting your agent to TechCrunch via AlterLab

The most efficient way to integrate this data is by treating the web as a structured database. Instead of asking the agent to "scrape" the page, you provide a tool that "extracts" specific fields.

Using the Extract API for Structured Output

The Extract API docs describe how to define a schema that the API uses to return only the data your agent needs. This keeps the context window clean and the costs low.

```python title="agent_techcrunch_extract.py" {6-11}

client = alterlab.Client("YOUR_API_KEY")

Define the schema to avoid sending raw HTML to the LLM

schema = {
"article_title": "string",
"author": "string",
"funding_amount": "string",
"company_name": "string"
}

result = client.extract(
url="https://techcrunch.com/2024/example-funding-story/",
schema=schema
)

print(result.data)

Output: {'article_title': 'Company X raises $10M', 'author': 'Jane Doe', ...}




For those building in Go, Rust, or Node.js, the cURL interface is the fastest way to implement the tool call.



```bash title="Terminal"
curl -X POST https://api.alterlab.io/api/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -d '{
    "url": "https://techcrunch.com/2024/example-funding-story/",
    "schema": {
      "article_title": "string",
      "funding_amount": "string"
    }
  }'

Using the Scrape API for Raw Data

If your agent needs to perform its own analysis on the page structure or needs the full text for a complex RAG pipeline, use the /api/v1/scrape endpoint. This provides the rendered HTML or Markdown.

```python title="agent_techcrunch_scrape.py" {7-9}

client = alterlab.Client("YOUR_API_KEY")

Requesting markdown format to save tokens in the LLM context window

result = client.scrape(
url="https://techcrunch.com",
formats=["markdown"]
)

print(result.markdown)




## Using the Search API for TechCrunch queries
An agent cannot always guess the exact URL of a story. To enable discovery, your agent needs a search tool. The `/api/v1/search` endpoint allows the agent to query TechCrunch specifically.

By restricting the search to `site:techcrunch.com`, the agent can find the most relevant URLs to then pass into the Extract API.



```bash title="Terminal"
curl -X POST https://api.alterlab.io/api/v1/search \
  -H "X-API-Key: YOUR_KEY" \
  -d '{"query": "site:techcrunch.com AI agent funding 2024"}'

MCP integration

For developers using Claude, GPT-4, or Cursor, the Model Context Protocol (MCP) is the gold standard for tool integration. AlterLab provides an MCP server that allows these agents to call scraping and extraction tools directly without you writing custom wrapper functions.

By installing the AlterLab MCP server, your agent gains a native extract_data tool. When the agent thinks, "I need to check the latest news on TechCrunch," it simply executes the tool call, receives the JSON, and incorporates it into its response.

For implementation details, see the AlterLab for AI Agents guide.

Building a startup news monitoring pipeline

Here is a practical end-to-end implementation of a monitoring pipeline. This pipeline follows a logic flow of: Trigger $\rightarrow$ Search $\rightarrow$ Extract $\rightarrow$ Analyze.

Implementation Example

```python title="funding_pipeline.py" {12-25}

from openai import OpenAI

client = alterlab.Client("YOUR_ALTERLAB_KEY")
llm = OpenAI(api_key="YOUR_OPENAI_KEY")

def monitor_funding():
# 1. Search for recent funding news
search_results = client.search(query="site:techcrunch.com 'Series A' AI")
latest_url = search_results[0]['url']

# 2. Extract structured data from the top result
data = client.extract(
    url=latest_url,
    schema={"company": "string", "amount": "string", "lead_investor": "string"}
)

# 3. Pass structured data to LLM for analysis
prompt = f"Analyze this funding round: {data.data}. Is this a competitor to our product?"
response = llm.chat.completions.create(
    model="gpt-4-turbo",
    messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.content

print(monitor_funding())




To scale this pipeline to monitor hundreds of pages, you can integrate scheduling. Use the [Getting started guide](/docs/quickstart/installation) to set up your environment, then implement cron-based scrapes to ensure your agent's knowledge base is updated every hour.

<div data-infographic="try-it" data-url="https://techcrunch.com" data-description="Extract structured TechCrunch data for your AI agent"></div>

## Key takeaways
*   **Avoid raw HTML**: Use structured extraction to save token costs and reduce LLM hallucinations.
*   **Handle anti-bot upstream**: Use an API that handles proxies and rendering so your agent doesn't get stuck in retry loops.
*   **Search first, Extract second**: Combine the Search API with the Extract API to give your agent the ability to discover and then analyze data.
*   **Standardize with MCP**: Use the Model Context Protocol for seamless integration with modern AI IDEs and LLMs.