DEV Community: AlterLab

Extract Structured Data from Websites Using AI Instead of CSS Selectors

AlterLab — Sun, 12 Apr 2026 10:35:25 +0000

The Problem with CSS Selectors

You write a scraper targeting .product-price .amount. It works. Two weeks later, the site ships a redesign and your selector returns null. You inspect the DOM, find the new class, patch your code, and move on. This repeats every few months for every site you scrape.

CSS selectors couple your extraction logic to implementation details you do not control. Class names change. DOM structures shift. A/B tests swap element order. Each change breaks your pipeline silently until you notice missing data downstream.

AI extraction removes this coupling. You describe the data you want in plain text. The model reads the page, understands the semantic structure, and returns clean JSON. No selectors to maintain. No DOM inspection when layouts change.

How AI Extraction Works

The process has three steps:

Fetch the page content (rendered, with JavaScript executed)
Pass the content and your extraction schema to a language model
Return structured JSON matching your schema

The model does not guess. It reads the actual rendered DOM, identifies elements matching your description, and extracts their values. If a product page has a price, name, and rating, you describe those fields and get them back as typed JSON.

Setting Up

Install the Python SDK:

```bash title="Terminal"
pip install alterlab




Or use the REST API directly with curl. Both approaches are covered below. You will need an API key from your [dashboard](https://alterlab.io/signup).

## Example: Extracting Product Data

Here is a product page on an e-commerce site. You need the product name, price, rating, and number of reviews. With CSS selectors, you would inspect the DOM, write four selectors, and hope they survive the next deploy.

With AI extraction, you describe the fields:



```python title="extract_product.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

response = client.scrape(
    url="https://example-store.com/products/wireless-headphones",
    formats=["json"],
    cortex={
        "prompt": "Extract: product_name (string), price (float), rating (float out of 5), review_count (integer)"
    }
)

data = response.json["cortex"]
print(data)

Output:

```json title="response.json"
{
"product_name": "Sony WH-1000XM5 Wireless Headphones",
"price": 348.00,
"rating": 4.7,
"review_count": 2841
}




The same request via curl:



```bash title="Terminal" {4-7}
curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-store.com/products/wireless-headphones",
    "formats": ["json"],
    "cortex": {
      "prompt": "Extract: product_name (string), price (float), rating (float out of 5), review_count (integer)"
    }
  }'

Structured Schemas with JSON Schema

For production pipelines, you want type guarantees. Pass a JSON Schema instead of a plain text prompt. The model validates its output against your schema before returning it.

```python title="extract_with_schema.py" {8-25}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"price": {"type": "number"},
"in_stock": {"type": "boolean"},
"sku": {"type": "string"}
},
"required": ["name", "price", "in_stock"]
}
}
}
}

response = client.scrape(
url="https://example-store.com/category/electronics",
formats=["json"],
cortex={"prompt": "Extract all products from this category page", "schema": schema}
)

for product in response.json["cortex"]["products"]:
print(f"{product['name']}: ${product['price']}")




This returns an array of products with typed fields. Missing optional fields are omitted. Required fields are always present. If the model cannot confidently extract a required field, it returns an error you can handle in your pipeline.

## Handling Dynamic Content

Many sites load data client-side. A product listing might render empty HTML, then populate via JavaScript fetches. Traditional scrapers that only fetch raw HTML get nothing back.

AI extraction requires the rendered DOM. The platform handles this automatically: it launches a headless browser, waits for the page to stabilize, then passes the rendered content to the model. You do not need to configure wait times or detect network idle.

For sites with aggressive bot detection, the [anti-bot bypass](https://alterlab.io/anti-bot-bypass-api) layer handles fingerprint rotation, TLS fingerprint matching, and challenge solving before the page ever reaches the extraction step.

## When to Use AI Extraction vs CSS Selectors

AI extraction is not a replacement for every scraping pattern. It is a tool for specific scenarios.

<div data-infographic="comparison">
  <table>
    <thead><tr><th>Criteria</th><th>AI Extraction</th><th>CSS Selectors</th></tr></thead>
    <tbody>
      <tr><td>Setup time</td><td>Seconds &mdash; describe fields in text</td><td>Minutes &mdash; inspect DOM, write selectors</td></tr>
      <tr><td>Maintenance</td><td>None &mdash; model adapts to layout changes</td><td>Ongoing &mdash; selectors break on redesign</td></tr>
      <tr><td>Cost per request</td><td>Higher &mdash; includes model inference</td><td>Lower &mdash; raw extraction only</td></tr>
      <tr><td>Type safety</td><td>Strong &mdash; JSON Schema validation</td><td>Manual &mdash; you parse and validate</td></tr>
      <tr><td>Best for</td><td>Dynamic pages, complex layouts, prototyping</td><td>Stable pages, high volume, simple structures</td></tr>
    </tbody>
  </table>
</div>

Use AI extraction when:
- The site changes its layout frequently
- You are prototyping and need data fast
- The page structure is complex or inconsistent
- You need to extract from many different sites with one pipeline

Use CSS selectors when:
- The page structure is stable and predictable
- You are scraping at very high volume and cost matters
- You need sub-second response times
- The data is in simple, consistent locations

You can mix both approaches in the same pipeline. Use AI extraction for complex pages and selectors for stable ones. The [Python SDK](https://alterlab.io/web-scraping-api-python) supports both patterns with the same client interface.

## Real-World Pattern: Monitoring Competitor Prices

Here is a practical pipeline that combines scheduling with AI extraction. You want to track prices for a list of competitor products daily.



```python title="price_monitor.py" {10-18}

client = alterlab.Client("YOUR_API_KEY")

competitors = [
    {"url": "https://competitor-a.com/product/123", "name": "Competitor A"},
    {"url": "https://competitor-b.com/p/abc", "name": "Competitor B"},
]

for competitor in competitors:
    response = client.scrape(
        url=competitor["url"],
        formats=["json"],
        cortex={
            "prompt": "Extract: product_name (string), price (float), availability (string)"
        }
    )

    data = response.json["cortex"]
    print(f"{competitor['name']}: {data['product_name']} @ ${data['price']} - {data['availability']}")

Wrap this in a scheduled job and store results in your database. When prices change, your pipeline detects the delta automatically. The monitoring feature can also handle this natively by watching pages for content changes and pushing diffs to your webhook endpoint.

Error Handling

AI extraction can fail when the page does not contain the requested data, the model cannot parse the structure, or the schema validation fails. Handle these cases explicitly:

```python title="error_handling.py" {12-18}

client = alterlab.Client("YOUR_API_KEY")

try:
response = client.scrape(
url="https://example.com/page",
formats=["json"],
cortex={"prompt": "Extract: email (string), phone (string)"}
)

if "error" in response.json.get("cortex", {}):
    print(f"Extraction failed: {response.json['cortex']['error']}")
else:
    print(response.json["cortex"])

except alterlab.APIError as e:
print(f"API error: {e.status_code} - {e.message}")




Common errors include pages that require authentication, content behind CAPTCHAs that exceed your tier, and schemas with impossible constraints. The API returns structured error messages so you can retry, adjust your prompt, or skip the page.

## Performance Considerations

AI extraction adds latency compared to raw HTML fetching. A typical request takes 3-8 seconds depending on page complexity and model load. For most pipelines, this is acceptable. Price monitoring, lead generation, and market research do not require sub-second responses.

If you need speed, use a two-tier approach:
1. Fetch raw HTML with a basic tier (fast, cheap)
2. Only escalate to AI extraction when the raw response is insufficient

Set `min_tier` in your request to skip lower tiers for known-difficult sites. This avoids the retry loop and gets you to the rendering tier on the first attempt.

Check the [pricing page](https://alterlab.io/pricing) for current tier costs and rate limits.

## Takeaway

CSS selectors tie your scraping logic to markup you do not control. AI extraction breaks that dependency. Describe the data you need, get back typed JSON, and stop maintaining selectors every time a site redesigns.

Use AI extraction for dynamic pages, prototyping, and multi-site pipelines. Use selectors for stable, high-volume targets. Mix both in the same pipeline based on each site's characteristics.

The [quickstart guide](https://alterlab.io/docs/quickstart/installation) covers installation and your first request in under five minutes.

Automate Web Scraping in n8n with AlterLab API

AlterLab — Sat, 11 Apr 2026 10:35:25 +0000

Automate Web Scraping in n8n with AlterLab's API

n8n is a workflow automation tool that connects APIs, databases, and services. Pair it with a scraping API that handles anti-bot bypass, proxy rotation, and headless rendering, and you get a pipeline that pulls structured data from any website on a schedule.

This tutorial shows how to build that pipeline. You will configure an n8n workflow that sends scrape requests, receives clean JSON, and routes the data to a database, spreadsheet, or webhook.

Prerequisites

An n8n instance (self-hosted or cloud)
An API key from alterlab.io/signup
Basic familiarity with n8n's node-based workflow editor

Step 1: Configure the HTTP Request Node

Create a new workflow in n8n. Add an HTTP Request node and configure it as follows:

Method: POST
URL: https://api.alterlab.io/v1/scrape
Authentication: Header Auth
Header Name: X-API-Key
Header Value: Your API key
Send Body: JSON

Set the JSON body to:

```json title="HTTP Request Body"
{
"url": "https://example.com/products",
"formats": ["json"],
"min_tier": 3
}




The `min_tier` parameter controls the scraping tier. Tier 3 enables JavaScript rendering. Set it higher for sites with aggressive bot detection. The [anti-bot bypass](https://alterlab.io/anti-bot-bypass-api) system auto-escalates if the initial tier fails.

## Step 2: Test with cURL First

Before building the full workflow, verify the endpoint works from your terminal. This isolates API issues from n8n configuration problems.



```bash title="Terminal" {1-4}
curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products", "formats": ["json"]}'

A successful response returns structured data:

```json title="Response" {3-8}
{
"status": "success",
"data": {
"products": [
{"name": "Widget A", "price": 29.99},
{"name": "Widget B", "price": 49.99}
]
},
"metadata": {
"url": "https://example.com/products",
"timestamp": "2026-04-11T10:30:00Z"
}
}




<div data-infographic="try-it" data-url="https://example.com" data-description="Try scraping this page with AlterLab"></div>

## Step 3: Build the Full n8n Workflow

A production workflow needs more than a single HTTP request. You need error handling, data transformation, and a destination for the scraped data.

### Workflow Structure



```plaintext
[Schedule Trigger] -> [HTTP Request (Scrape)] -> [Code (Parse)] -> [Database/Sheet/Webhook]

Add these nodes in order:

1. Schedule Trigger

Set a cron expression for your scrape frequency. Daily at 6 AM UTC:

0 6 * * *

2. HTTP Request Node

Use the configuration from Step 1. Enable "Continue On Fail" so one failed scrape does not block the entire workflow.

3. Code Node (Data Transformation)

Parse the JSON response and extract the fields you need:

```python title="n8n Code Node" {5-12}

Access the HTTP Request output

response = json.parse($input.first().json.body)

Extract product data

products = response.get("data", {}).get("products", [])

Transform to your schema

items = []
for product in products:
items.append({
"json": {
"name": product["name"],
"price": product["price"],
"scraped_at": response["metadata"]["timestamp"],
"source": response["metadata"]["url"]
}
})

return items




**4. Destination Node**

Connect your output node. Common choices:

- **Postgres/MySQL**: Use the database node to upsert records
- **Google Sheets**: Append rows for lightweight tracking
- **Webhook**: Push to your own API or a Slack channel

## Step 4: Handle Multiple URLs

Scraping a single page is straightforward. Real pipelines scrape dozens or hundreds of URLs. Use n8n's Split Out node to fan out requests.



```python title="URL List Generator" {3-7}
# Code node that outputs multiple URLs
urls = [
    "https://example.com/products/page/1",
    "https://example.com/products/page/2",
    "https://example.com/products/page/3"
]

return [{"json": {"url": u}} for u in urls]

Connect this to a Split Out node, then to your HTTP Request node. Each URL becomes a separate execution branch. n8n processes them in parallel up to your concurrency limit.

Add rate limiting between requests if the target site requires it. Use the Wait node between the Split Out and HTTP Request nodes:

Wait: 2 seconds

Step 5: Add Error Handling and Retries

Scraping fails. Pages change structure, sites go down, anti-bot systems update. Your workflow should handle failures gracefully.

Retry Configuration

In the HTTP Request node settings:

Retry On Fail: Enable
Max Retries: 3
Retry Backoff: Exponential

Error Routing

Add an error output branch from the HTTP Request node:

[HTTP Request] --(success)--> [Parse] --> [Database]
       |
       --(error)--> [Error Handler] --> [Alert/Log]

The error handler can log failures to a separate sheet, send a Slack notification, or queue the URL for a retry with a higher tier.

```python title="Error Handler Code Node" {4-9}

Capture failed URLs for retry

error_data = $input.first().json

failed_urls.append({
"url": error_data.get("url"),
"error": error_data.get("error"),
"timestamp": datetime.utcnow().isoformat(),
"retry_tier": 4 # escalate tier on retry
})

return [{"json": {"failed": failed_urls}}]




## Step 6: Use Cortex AI for Structured Extraction

Some pages do not have clean HTML structures. Product listings buried in JavaScript, unstructured text, or dynamic content require a different approach. Cortex AI extracts structured data using natural language instructions.



```bash title="Terminal" {5-9}
curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/reviews",
    "formats": ["json"],
    "cortex": {
      "prompt": "Extract reviewer name, rating (1-5), and review text from each review block"
    }
  }'

The response returns data matching your schema:

```json title="Cortex AI Response" {4-12}
{
"status": "success",
"data": {
"reviews": [
{
"reviewer_name": "Jane D.",
"rating": 5,
"review_text": "Excellent product, fast shipping."
},
{
"reviewer_name": "Mark S.",
"rating": 4,
"review_text": "Good quality, slightly overpriced."
}
]
}
}




In n8n, the Cortex output works identically to standard JSON output. Route it through the same Code and Database nodes.

## Step 7: Monitor and Alert on Changes

Scraping is not always about collecting new data. Sometimes you need to detect changes on existing pages. Price drops, stock availability, competitor updates, regulatory filings.

Configure monitoring by storing previous scrape results and comparing them on each run:



```python title="Change Detection Code Node" {6-15}
# Compare current scrape with previous state
current = $input.first().json
previous = get_previous_state(current["url"])  # from database

changes = []
for key in current["data"]:
    if key not in previous:
        changes.append({"field": key, "action": "added", "value": current["data"][key]})
    elif current["data"][key] != previous[key]:
        changes.append({
            "field": key,
            "action": "changed",
            "old": previous[key],
            "new": current["data"][key]
        })

# Only pass through if changes detected
if changes:
    return [{"json": {"url": current["url"], "changes": changes}}]
return []

When changes exist, route to an alert node. When nothing changed, the workflow exits silently.

Cost Considerations

Scraping pipelines can run expensive if you are not careful. A few practices:

Cache aggressively: Do not re-scrape pages that have not changed. Store hashes of previous responses and skip identical results.
Use the lowest tier that works: Start with min_tier: 1 for static pages. Only escalate to tier 3+ for JavaScript-heavy sites.
Batch URLs: Group related URLs into single workflow runs rather than triggering separate workflows per URL.
Set spend limits: API keys support spend caps. Set them per workflow to prevent runaway costs.

Check pricing for current rates. You pay for what you use with no monthly minimums.

Complete Workflow Example

Here is the full n8n workflow JSON for a daily product price scrape:

```json title="n8n Workflow Export" {10-20}
{
"name": "Daily Price Scraper",
"nodes": [
{
"name": "Schedule",
"type": "n8n-nodes-base.scheduleTrigger",
"parameters": {
"rule": { "interval": ["days"], "triggerAtHour": 6 }
}
},
{
"name": "Scrape Products",
"type": "n8n-nodes-base.httpRequest",
"parameters": {
"method": "POST",
"url": "https://api.alterlab.io/v1/scrape",
"authentication": "headerAuth",
"body": {
"url": "={{ $json.url }}",
"formats": ["json"],
"min_tier": 3
},
"options": {
"retryOnFail": true,
"maxTries": 3
}
}
},
{
"name": "Parse Response",
"type": "n8n-nodes-base.code",
"parameters": {
"jsCode": "const data = $input.first().json.body;\nreturn data.data.products.map(p => ({ json: p }));"
}
},
{
"name": "Save to Database",
"type": "n8n-nodes-base.postgres",
"parameters": {
"operation": "upsert",
"table": "product_prices",
"columns": "name,price,scraped_at"
}
}
],
"connections": {
"Schedule": { "main": [[{ "node": "Scrape Products", "type": "main" }]] },
"Scrape Products": { "main": [[{ "node": "Parse Response", "type": "main" }]] },
"Parse Response": { "main": [[{ "node": "Save to Database", "type": "main" }]] }
}
}




Import this into n8n via the workflow editor, replace the authentication credentials with your API key, and adjust the URL and database schema to match your use case.

## Troubleshooting

**Empty responses**: The page may require a higher tier. Increase `min_tier` to 4 or 5. Check the [API docs](https://alterlab.io/docs) for tier descriptions.

**Rate limit errors**: Add a Wait node between requests. Start with 1-2 seconds and increase if needed.

**CAPTCHA blocks**: Set `min_tier: 5` to enable CAPTCHA solving. This costs more per request but eliminates manual intervention.

**Schema drift**: Websites change their HTML structure. Cortex AI handles this better than CSS selectors since it uses semantic understanding. Switch to Cortex if your selectors break frequently.

**n8n timeout**: Long-running scrapes can exceed n8n's execution timeout. For large batches, use the webhook pattern. Configure AlterLab to push results to an n8n webhook URL instead of polling.

## Takeaway

n8n handles orchestration. AlterLab handles extraction. Together they give you a scraping pipeline that runs on a schedule, handles failures, and delivers clean data to your systems.

Start with a single URL and a basic HTTP Request node. Add error handling, multi-URL support, and change detection as your needs grow. The [quickstart guide](https://alterlab.io/docs/quickstart/installation) covers API setup in under five minutes.

Feed Clean Web Data to RAG Pipelines Without Wasting LLM Tokens

AlterLab — Sat, 04 Apr 2026 10:35:11 +0000

How to Feed Clean Web Data to RAG Pipelines Without Wasting 90% of Your LLM Tokens

Raw HTML is the worst possible input for a RAG pipeline. A single product page carries 15,000 to 25,000 tokens of navigation chrome, analytics scripts, CSS classes, and ad placeholders. Your embedding model processes all of it. Your vector store stores all of it. Your retrieval step searches through all of it.

You pay for every token.

The fix is straightforward: extract only the content that matters before it reaches your embedding model. Strip the noise. Keep the signal. Structure it so retrieval actually works.

Here is how to build that pipeline.

The Token Math Behind Dirty Web Data

A typical e-commerce product page breaks down like this:

Product title, description, specs: ~800 tokens
Navigation menus, footer, sidebar: ~3,000 tokens
JavaScript bundles, tracking pixels, ad scripts: ~8,000 tokens
CSS class names, inline styles, layout divs: ~4,000 tokens
Schema markup, meta tags, Open Graph: ~1,200 tokens

Your RAG pipeline cares about the first line. The rest is infrastructure for a browser, not context for a language model.

When you embed raw HTML, the noise drowns out the signal. Two product pages with identical descriptions but different ad networks produce wildly different embeddings. Retrieval quality drops. You compensate by increasing chunk overlap and top-k results, which drives costs higher.

Extract clean content first. Embed only what matters.

Step 1: Get Clean Content at the Source

The most efficient place to strip noise is during extraction, not after. Fetching raw HTML and cleaning it locally means you still transfer the full page, parse the full DOM, and run your own selector logic. Doing it server-side through a scraping API cuts the work in half.

Here is the same operation using the Python SDK and a direct cURL call. Both request Markdown output instead of raw HTML.

```python title="scraper.py" {1,4-6}
from alterlab import AlterLab

client = AlterLab(api_key="YOUR_API_KEY")
response = client.scrape(
url="https://example.com/product/12345",
formats=["markdown"]
)
print(response.markdown)






```bash title="Terminal" {3-5}
curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -d '{
    "url": "https://example.com/product/12345",
    "formats": ["markdown"]
  }'

The response arrives as clean Markdown. No HTML tags. No script blocks. Just headings, paragraphs, lists, and code blocks in a format embedding models already understand.

For sites that require JavaScript rendering, set min_tier=3 to skip the basic HTTP fetcher and go straight to a headless browser. The API handles Cloudflare challenges, CAPTCHAs, and rotating proxies automatically. You get the rendered content without managing browser instances.

Step 2: Structure Data for Retrieval, Not Display

Markdown output works well for articles, documentation, and blog posts. But product pages, job listings, and pricing tables need structure. A flat text blob loses the relationships between fields.

Use Cortex AI extraction to pull structured data directly from the page. You describe what you want in plain English. The API returns JSON.

```python title="structured_extraction.py" {5-12}
from alterlab import AlterLab

client = AlterLab(api_key="YOUR_API_KEY")

response = client.scrape(
url="https://example.com/jobs",
cortex={
"prompt": "Extract all job listings. For each listing, return: title, department, location, salary_range, posting_date, and apply_url."
},
formats=["json"]
)

for job in response.json["listings"]:
print(f"{job['title']} - {job['location']} ({job['salary_range']})")




The JSON output maps directly to your embedding pipeline. Each job listing becomes a single document with typed fields. You can embed the full record, or embed specific fields separately for hybrid search.

Compare this to the alternative: scraping raw HTML, writing CSS selectors for each site, parsing dates from inconsistent formats, and handling layout changes that break your selectors every few weeks. Cortex handles the variation. You get consistent JSON regardless of how the page renders.

## Step 3: Chunk Strategically

Clean content solves the noise problem. Chunking strategy solves the retrieval problem.

Bad chunking cuts sentences in half. It splits tables across chunks. It separates a heading from the paragraphs it governs. Your embedding model sees fragments without context, and retrieval returns partial matches.

Good chunking respects document structure. Markdown makes this straightforward.



```python title="chunker.py" {6-15}

from typing import List

def chunk_markdown(text: str, max_tokens: int = 500) -> List[str]:
    chunks = []
    sections = re.split(r'\n## ', text)

    for section in sections:
        if not section.strip():
            continue

        heading = ""
        if "\n" in section:
            heading, body = section.split("\n", 1)
        else:
            heading, body = section, ""

        current_chunk = f"## {heading}\n" if heading else ""

        paragraphs = body.split("\n\n")
        for para in paragraphs:
            if len(current_chunk) + len(para) > max_tokens * 4:
                chunks.append(current_chunk.strip())
                current_chunk = f"## {heading}\n" if heading else ""
            current_chunk += para + "\n\n"

        if current_chunk.strip():
            chunks.append(current_chunk.strip())

    return chunks

This approach keeps headings attached to their content. It respects paragraph boundaries. It produces chunks that embedding models can reason about as complete units.

The token estimate uses a 4:1 character-to-token ratio for planning. Your embedding provider's tokenizer gives exact counts. Use that for production.

Step 4: Build the Ingestion Pipeline

Tie extraction, cleaning, chunking, and embedding together. The pipeline should handle three scenarios:

Initial index: Scrape a list of URLs, extract clean content, chunk, embed, store.
Incremental update: Monitor pages for changes. Re-extract and re-embed only what changed.
Scheduled refresh: Run on a cron to catch pages that changed without triggering monitoring alerts.

```python title="pipeline.py" {8-10,18-22}
from alterlab import AlterLab
from datetime import datetime

client = AlterLab(api_key="YOUR_API_KEY")

def ingest_page(url: str, embedding_fn):
response = client.scrape(
url=url,
formats=["markdown"],
min_tier=3
)

if not response.markdown:
    return

chunks = chunk_markdown(response.markdown)

for i, chunk in enumerate(chunks):
    vector = embedding_fn(chunk)
    store_vector(url, i, chunk, vector, datetime.utcnow())

def ingest_batch(urls: list, embedding_fn):
for url in urls:
try:
ingest_page(url, embedding_fn)
except Exception as e:
print(f"Failed {url}: {e}")




For incremental updates, use the monitoring feature. Set up watchers on your indexed URLs. When content changes, the API notifies you via webhook. You re-run `ingest_page` for that URL only. No full re-index required.



```python title="monitoring_setup.py" {4-9}
client.monitor(
    url="https://example.com/pricing",
    schedule="0 9 * * 1",
    webhook="https://your-server.com/webhooks/alterlab",
    diff=True
)

The webhook payload includes a diff showing what changed. You can decide whether the change warrants a re-embedding. A price update does. A typo fix in the footer does not.

Approach	Tokens per Page	Retrieval Quality	Maintenance
Raw HTML	15,000-25,000	Low	High (selector breaks)
Local HTML cleaning	5,000-8,000	Medium	High (DOM changes)
Server-side Markdown	1,500-3,000	High	Low (handled by API)
Cortex JSON extraction	200-800	Highest	Lowest (AI adapts)

Step 5: Handle Anti-Bot Pages Without Infrastructure

Many sites you want to index block automated requests. Cloudflare challenges, CAPTCHAs, rate limits. Managing bypass logic yourself means running browser instances, solving CAPTCHAs through third-party services, rotating proxies, and handling fingerprinting.

That infrastructure costs more than the scraping itself.

Use tiered scraping to handle this automatically. Start with a lightweight HTTP request. If the site blocks it, the API escalates to a headless browser with anti-bot bypass. You set the floor with min_tier to skip the试探 phase for sites you know are protected.

```python title="tiered_scraping.py" {4-7}
response = client.scrape(
url="https://protected-site.com/data",
min_tier=3,
formats=["markdown"]
)
print(response.status)
print(response.markdown[:500])




Tier 1 handles simple static pages. Tier 3 adds JavaScript rendering and anti-bot bypass. Tier 5 includes CAPTCHA solving. The API picks the right tier for each URL. You get clean content regardless of what stands between you and the data.

<div data-infographic="try-it" data-url="https://alterlab.io/docs" data-description="Try extracting clean Markdown from this documentation page"></div>

## Cost Breakdown

Token waste compounds across three stages of a RAG pipeline:

**Embedding**: You pay per token sent to the embedding model. Feeding 20,000 tokens of raw HTML instead of 2,000 tokens of clean Markdown costs 10x more per page. Index 10,000 pages and the difference is measurable.

**Storage**: Vector databases charge by dimension count and record volume. Storing embeddings for noise chunks wastes space. It also degrades query performance as the index grows with low-signal vectors.

**Retrieval**: Each query searches the entire index. A bloated index with noisy chunks returns worse results. You compensate by fetching more candidates (higher top-k), which increases the context window for your generation model. That costs more per query.

Clean extraction at the source addresses all three. Smaller chunks. Better embeddings. Faster retrieval. Lower generation costs because the context window contains relevant content, not navigation footers.

## When to Use Each Output Format

**Markdown**: Articles, documentation, blog posts, help centers. Any page where the content flows as prose with headings and lists. This is your default for knowledge base ingestion.

**JSON with Cortex**: Product catalogs, job boards, pricing tables, real estate listings. Any page with repeating structured elements. The AI extraction handles layout variation across sites without custom selectors.

**Plain text**: Simple pages with minimal formatting. API response pages. Status pages. Use it when you want the smallest possible output and document structure does not matter for retrieval.

**HTML**: Rarely. Only when you need to preserve specific formatting that Markdown cannot represent, like complex tables with merged cells or embedded SVG diagrams. Most RAG pipelines do not need this.

## Putting It Together

A production RAG ingestion pipeline looks like this:

1. Maintain a URL registry with metadata (category, last indexed, change hash).
2. On schedule or webhook trigger, scrape each URL with `formats=["markdown"]` or Cortex extraction.
3. Chunk the output using structure-aware splitting.
4. Embed chunks and upsert into your vector store with URL and timestamp metadata.
5. Monitor URLs for changes. Re-index only what changed.

The scraping layer handles rendering, anti-bot bypass, and format conversion. Your pipeline handles chunking, embedding, and storage. Clean separation. Each layer does one job well.

Check the [Python SDK documentation](https://alterlab.io/web-scraping-api-python) for the full API reference, including webhook configuration and scheduling options. The [quickstart guide](https://alterlab.io/docs/quickstart/installation) covers account setup and your first API call.

## Takeaway

Raw HTML wastes tokens on infrastructure code that embedding models cannot use. Extract clean Markdown or structured JSON before the content reaches your pipeline. Chunk with respect to document boundaries. Monitor for changes and re-index incrementally.

The result: 85 to 90 percent fewer tokens per page, better retrieval accuracy, and lower costs at every stage of the RAG pipeline. The scraping API handles rendering and anti-bot bypass. You handle the data.

Build a Web Scraping Pipeline with n8n and AlterLab

AlterLab — Tue, 31 Mar 2026 10:36:52 +0000

n8n is a workflow automation platform built around HTTP nodes, visual routing, and an in-process JavaScript runtime. When you pair it with AlterLab — a scraping API that handles anti-bot detection, headless rendering, and proxy rotation — you get a complete data extraction pipeline without managing browser pools, proxy credentials, or retry logic from scratch.

This tutorial builds a production-ready pipeline: URL inputs → scraping API → HTML parsing → structured storage, driven by a cron schedule with proper error handling.

Prerequisites

n8n instance (self-hosted via Docker or n8n Cloud)
API key — follow the quickstart guide to get one in under two minutes
Familiarity with n8n's workflow editor and basic JavaScript

Step 1: Store the API Key in n8n Credentials

Never hardcode secrets into HTTP Request nodes. Go to Settings → Credentials → Add Credential → Header Auth and fill in:

Field	Value
Name	`Scraping API Key`
Header Name	`X-API-Key`
Header Value	`YOUR_API_KEY`

Reference this credential in every HTTP Request node in the workflow. Rotating the key means updating one credential, not hunting through nodes.

Step 2: Configure the HTTP Request Node

Drop an HTTP Request node into the canvas. Set Method to POST, URL to https://api.alterlab.io/v1/scrape, authenticate with the credential created above, and set Body Content Type to JSON.

```json title="HTTP Request — Payload"
{
"url": "https://books.toscrape.com/catalogue/page-1.html",
"render_js": false,
"premium_proxy": false,
"country": "us",
"timeout": 30000
}




For targets protected by Cloudflare, Akamai, or PerimeterX, set `render_js: true` and `premium_proxy: true`. The [anti-bot bypass](https://alterlab.io/anti-bot-bypass-api) layer handles TLS fingerprinting, browser emulation, and CAPTCHA solving transparently — no extra configuration on your end.

The same request in cURL for testing before wiring into n8n:



```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/catalogue/page-1.html",
    "render_js": false,
    "premium_proxy": false
  }'

The equivalent single-URL call in Python:

```python title="single_scrape.py" {7-12}

API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.alterlab.io/v1/scrape"

def scrape(url: str, render_js: bool = False) -> dict:
with httpx.Client() as client: # synchronous single fetch
r = client.post(
BASE_URL,
headers={"X-API-Key": API_KEY},
json={"url": url, "render_js": render_js},
timeout=30.0,
)
r.raise_for_status()
return r.json()

result = scrape("https://books.toscrape.com/catalogue/page-1.html")
print(result["status_code"], result["elapsed_ms"], "ms")




The API response shape:



```json title="API Response"
{
  "success": true,
  "status_code": 200,
  "url": "https://books.toscrape.com/catalogue/page-1.html",
  "html": "<!DOCTYPE html>...",
  "elapsed_ms": 712
}

Try it against a live target to see the response before building the rest of the pipeline:

Step 3: Parse HTML in the Code Node

Add a Code node immediately after the HTTP Request. n8n bundles Cheerio in its runtime — use it to walk the DOM and emit structured records.

```javascript title="n8n Code Node — Extract Book Listings" {7-18}
const { load } = require('cheerio');

const results = [];

for (const item of $input.all()) {
const $ = load(item.json.html);

$('article.product_pod').each((_, el) => { // iterate product cards
const title = $(el).find('h3 a').attr('title');
const price = $(el).find('.price_color').text().trim();
const rating = $(el).find('p.star-rating').attr('class')?.split(' ')[1];
const relHref = $(el).find('h3 a').attr('href');

results.push({                                    // emit flat record
  title,
  price,
  rating,
  url: `https://books.toscrape.com/catalogue/${relHref}`,
  scraped_at: new Date().toISOString(),
});

});
}

return results.map(r => ({ json: r }));




For targets that return JSON from an XHR endpoint (scraped through the proxy), skip Cheerio and parse directly:



```javascript title="n8n Code Node — Parse JSON from html Field" {2-3}
const raw = $input.first().json.html;
const data = JSON.parse(raw);            // html field contains the raw JSON string
return data.products.map(p => ({ json: p }));

If Cheerio is missing in a self-hosted setup, run npm install cheerio in the n8n working directory and restart the service.

Step 4: Scrape Multiple Pages

Use a Code node to generate a URL list, then feed it through Split In Batches → HTTP Request:

```javascript title="n8n Code Node — Generate Paginated URL List" {3-6}
const BASE = 'https://books.toscrape.com/catalogue/page-';
const PAGES = 50;

const urls = Array.from( // generate range of page URLs
{ length: PAGES },
(_, i) => ({ json: { url: ${BASE}${i + 1}.html } })
);

return urls;




Set **Split In Batches** to a batch size of 5 to avoid hammering the target. The HTTP Request node processes each batch item as a separate request automatically.

For high-volume pipelines where n8n acts as the orchestrator and Python handles the heavy lifting, use async fan-out:



```python title="batch_scrape.py" {15-21}

API_KEY  = "YOUR_API_KEY"
ENDPOINT = "https://api.alterlab.io/v1/scrape"

async def fetch(client: httpx.AsyncClient, url: str) -> dict:
    r = await client.post(
        ENDPOINT,
        headers={"X-API-Key": API_KEY},
        json={"url": url, "render_js": False},
        timeout=30.0,
    )
    r.raise_for_status()
    return r.json()

async def scrape_batch(urls: list[str]) -> list[dict]:  # fan-out entry point
    async with httpx.AsyncClient() as client:           # single connection pool
        tasks   = [fetch(client, u) for u in urls]      # build coroutine list
        results = await asyncio.gather(*tasks, return_exceptions=True)
        return results

if __name__ == "__main__":
    pages = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 11)]
    data  = asyncio.run(scrape_batch(pages))

    for i, result in enumerate(data):
        if isinstance(result, Exception):
            print(f"Page {i+1} failed: {result}")
        else:
            print(f"Page {i+1}: {len(result['html']):,} bytes — {result['elapsed_ms']}ms")

The Python scraping API client wraps this pattern with built-in retry logic, concurrency throttling, and typed responses — worth switching to once you move beyond prototyping.

Step 5: Route Data to Storage

Wire the Code node output to whichever storage node fits your stack.

Postgres — recommended for structured pipelines:

Node: Postgres, Operation: Insert, Table: scraped_books
Map title, price, rating, url, scraped_at directly from Code node output fields

Google Sheets — minimal setup for low-volume runs:

Node: Google Sheets, Operation: Append or Update
Same column mapping

Webhook forward — for downstream microservices or event buses:

```json title="Webhook Payload"
{
"source": "n8n-book-scraper",
"run_id": "{{ $execution.id }}",
"count": 20,
"records": [
{ "title": "A Light in the Attic", "price": "£51.77", "rating": "Three", "url": "..." }
]
}




---

## Step 6: Schedule and Add Error Handling

Swap the manual trigger for a **Schedule Trigger** node before going to production.

| Cadence | Cron Expression | Typical Use Case |
|---------|-----------------|------------------|
| Hourly | `0 * * * *` | Price monitoring |
| Daily 06:00 UTC | `0 6 * * *` | News/content aggregation |
| Every 15 minutes | `*/15 * * * *` | Inventory feeds |
| Weekdays 09:00 UTC | `0 9 * * 1-5` | B2B lead enrichment |

For event-driven scraping — e.g., new URLs inserted into a database — replace the Schedule Trigger with a **Postgres Trigger** node watching for new rows.

**Error handling — configure before going live:**

1. HTTP Request node → enable **Retry On Fail**: 3 retries, 2000ms backoff
2. Code node → enable **Continue On Fail** if partial runs are acceptable
3. In **Settings → Error Workflow**, assign a dedicated workflow that captures and routes failures:



```javascript title="Error Workflow — Log Failures to Dead-Letter Table" {5-11}
// Runs inside the error workflow's Code node
const err = $input.first().json;

return [{
  json: {
    workflow:     err.workflow?.name,
    node:         err.execution?.lastNodeExecuted,   // which node threw
    message:      err.execution?.error?.message,
    failed_at:    new Date().toISOString(),
    execution_id: err.execution?.id,
  }
}];

Route the output to a Postgres scrape_errors table or a Slack node. Silent failures are harder to diagnose than loud ones.

Approach	Anti-Bot Handling	Setup Time	Maintenance	Scaling	Cost Model
DIY Playwright + Proxies	Manual (fingerprinting, stealth)	Days–weeks	High (browser updates, proxy churn)	Complex (concurrency, queueing)	Infrastructure + proxy fees
n8n + Scraping API	Automatic (TLS, CAPTCHA, headers)	<1 hour	Low (API versioned separately)	Batch nodes + API concurrency	Per successful request
Commercial ETL (Apify, etc.)	Varies by actor	Minutes (pre-built actors)	Low but opaque	Platform-managed	Platform subscription + compute

Monitoring Pipeline Health

Don't rely solely on n8n's execution log. Instrument your pipeline explicitly:

Log success: false responses from the scraping API to a monitoring table — the API returns this field even on 200 responses if the target blocked the request
Store elapsed_ms per run in a scrape_metrics table; trend upward means proxy pool degradation
Row count guard — after the storage node, add a Code node that alerts if results.length < EXPECTED_MINIMUM:

```javascript title="n8n Code Node — Row Count Guard" {5-9}
const MINIMUM = 15; // expect at least 15 records per page

const count = $input.all().length;

if (count < MINIMUM) { // trigger alert path
throw new Error(Low yield: got ${count}, expected >= ${MINIMUM});
}

return $input.all(); // pass through if OK




Place this node between the Code parser and the storage node. When it throws, n8n's error workflow catches it.

---

## Takeaways

- n8n's HTTP Request node integrates with any REST scraping API in minutes — no custom nodes required
- Use `render_js: true` selectively; static fetches are faster and cheaper than headless browser requests
- Keep parsing logic inside the Code node to maintain self-contained, debuggable workflows
- Cheerio handles the majority of HTML extraction cases; fall back to a dedicated parser service only for complex XPath requirements
- Configure retries on the HTTP node and a global error workflow before scheduling — silent data loss compounds across runs
- For event-driven ingestion triggered by new URLs in a queue or database, swap the Schedule Trigger for a Postgres Trigger or AMQP node without changing the rest of the workflow

How to Scrape Glassdoor: Complete Guide for 2026

AlterLab — Sat, 28 Mar 2026 14:25:57 +0000

How to Scrape Glassdoor: Complete Guide for 2026

Glassdoor exposes salary data, company reviews, and job listings that are genuinely useful for compensation benchmarking, recruiting analysis, and labor market research. The catch: Glassdoor runs Cloudflare, gates salary data behind authentication, and renders all meaningful content client-side with React. This guide cuts through those obstacles with working Python code.

Why Scrape Glassdoor?

Three use cases that justify the engineering effort:

Compensation benchmarking — HR teams and SaaS products aggregate salary ranges by role, level, location, and company size. Glassdoor's crowdsourced compensation data is one of the richest publicly accessible sources for this kind of analysis. Refreshing it weekly catches market shifts before they show up in annual survey reports.

Competitive talent intelligence — Track hiring velocity at competitors. Which roles are they posting? How quickly are positions closing? Job listing volume is a reliable leading indicator of engineering and product priorities six to nine months out.

Employer brand monitoring — Tracking review sentiment over time — overall ratings, CEO approval, interview difficulty scores — gives recruiting teams early warning of culture problems before they surface as churn events. Companies also benchmark their own standing against direct competitors.

Anti-Bot Challenges on glassdoor.com

Glassdoor deploys several overlapping protections that make DIY scraping expensive to maintain:

Cloudflare WAF and Bot Management — Glassdoor sits behind Cloudflare's bot management layer. A standard Python requests call receives a JS challenge page requiring a valid cf_clearance cookie before any real HTML is served. This blocks virtually every naive scraper immediately.

Login wall for salary data — Salary ranges and detailed compensation breakdowns are gated behind authentication. Unauthenticated sessions see truncated results or get redirected to a signup modal. Full data access requires managing authenticated sessions with valid cookies.

Client-side rendering — Job listings, reviews, and salary cards are all React components. The initial HTML response from Glassdoor's server is a near-empty shell. You need a JavaScript runtime to execute the page and produce actual content.

Browser fingerprinting and behavioral detection — Glassdoor combines static browser fingerprinting with behavioral signals (scroll cadence, mouse movement, click timing) to identify headless browsers. Playwright and Puppeteer with default configurations are reliably flagged within a few page loads.

Maintaining your own bypass stack — refreshing cf_clearance cookies, managing residential proxy pools, spoofing browser fingerprints — is a real ongoing engineering commitment. AlterLab's Anti-bot bypass API handles all of this at the infrastructure level, so your scraping code stays focused on data extraction.

Quick Start with AlterLab API

Install the SDK and you can make your first Glassdoor request in under a minute. See the getting started guide for full environment setup, including API key management and optional async configuration.

```bash title="Terminal"
pip install alterlab beautifulsoup4 lxml






```python title="scrape_glassdoor.py" {4-11}

from bs4 import BeautifulSoup

client = alterlab.Client("YOUR_API_KEY")

# Scrape a Glassdoor job search results page
response = client.scrape(
    "https://www.glassdoor.com/Job/python-developer-jobs-SRCH_KO0,16.htm",
    render_js=True,                          # Required: Glassdoor is a React SPA
    wait_for="[data-test='jobListing']",     # Wait for job cards before returning
)

soup = BeautifulSoup(response.html, "html.parser")
job_cards = soup.select("[data-test='jobListing']")
print(f"Found {len(job_cards)} job listings")

The equivalent cURL call for testing from a shell or integrating with non-Python pipelines:

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.glassdoor.com/Job/python-developer-jobs-SRCH_KO0,16.htm",
"render_js": true,
"wait_for": "[data-test=\"jobListing\"]"
}'




<div data-infographic="stats">
  <div data-stat data-value="99.1%" data-label="Glassdoor Success Rate"></div>
  <div data-stat data-value="2.4s" data-label="Avg JS Render Time"></div>
  <div data-stat data-value="100%" data-label="Cloudflare Bypass Rate"></div>
  <div data-stat data-value="0ms" data-label="Proxy Setup Time"></div>
</div>

## Extracting Structured Data

With fully rendered HTML in hand, here is how to pull the most useful data points from Glassdoor's DOM.

### Job Listings

Glassdoor uses `data-test` attributes on stable semantic elements — always prefer these over generated class names, which change with every React build deployment.



```python title="parse_jobs.py" {9-22}
from bs4 import BeautifulSoup

def parse_job_listings(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    jobs = []

    for card in soup.select("[data-test='jobListing']"):
        def text(selector):
            el = card.select_one(selector)
            return el.get_text(strip=True) if el else None

        jobs.append({
            "title":    text("[data-test='job-title']"),
            "company":  text("[data-test='employer-name']"),
            "location": text("[data-test='emp-location']"),
            "salary":   text("[data-test='detailSalary']"),
            "rating":   text("[data-test='rating']"),
            "age":      text("[data-test='job-age']"),
        })

    return jobs

Company Reviews

Review pages are paginated at 10 entries per page. The _IP{n} path segment in the URL controls the page number.

```python title="parse_reviews.py" {6-22}

from bs4 import BeautifulSoup

client = alterlab.Client("YOUR_API_KEY")

def scrape_company_reviews(company_slug: str, pages: int = 5) -> list[dict]:
"""
company_slug: e.g. 'Google' (as it appears in the Glassdoor URL)
"""
reviews = []
slug_len = len(company_slug)

for page in range(1, pages + 1):
    url = (
        f"https://www.glassdoor.com/Reviews/{company_slug}-reviews"
        f"-SRCH_KE0,{slug_len}_IP{page}.htm"
    )
    response = client.scrape(url, render_js=True, wait_for="[data-test='review']")
    soup = BeautifulSoup(response.html, "html.parser")

    for review in soup.select("[data-test='review']"):
        def text(selector):
            el = review.select_one(selector)
            return el.get_text(strip=True) if el else None

        reviews.append({
            "headline": text("[data-test='review-title']"),
            "rating":   text("[data-test='overall-rating']"),
            "pros":     text("[data-test='pros']"),
            "cons":     text("[data-test='cons']"),
            "date":     text("[data-test='review-date']"),
            "role":     text("[data-test='author-jobTitle']"),
        })

return reviews




### Salary Data

Salary pages require an authenticated session. Pass `glassdoor_session` and `tguid` cookies obtained from a logged-in browser profile. The API accepts a `headers` dict for this purpose:



```python title="parse_salaries.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

response = client.scrape(
    "https://www.glassdoor.com/Salaries/software-engineer-salary-SRCH_KO0,17.htm",
    render_js=True,
    headers={
        "Cookie": "JSESSIONID=YOUR_SESSION_ID; tguid=YOUR_TGUID"
    },
    wait_for="[data-test='salaryRow']",
)

Key selectors once authenticated: [data-test='salaryRow'] for each salary entry, [data-test='salary-estimate'] for the reported range, and [data-test='total-compensation'] for the total comp breakdown.

Common Pitfalls

Per-IP rate limiting — Glassdoor throttles at the IP level, not just by User-Agent. Exceeding roughly 25–30 requests per minute from a single IP triggers 429 responses or silent result degradation, where fewer listings are returned without any error signal. Distributed requests across rotating proxies are required for sustained collection.

Session expiry on gated content — Glassdoor sessions expire within a few hours. For pipelines that scrape salary or authenticated review data, implement cookie refresh logic. Detect redirects to /profile/login as the signal that your session has expired and re-authenticate before continuing.

Hard pagination cap — Glassdoor limits job search results to 30 pages (300 results) per query regardless of how many matching listings exist. Paginating past page 30 returns the first page again. The correct approach is to narrow queries by location, fromAge (days posted), or jobType parameter rather than paginating deeper on a broad query.

Selector drift — Glassdoor ships frontend updates frequently. Class names change with every React build. The data-test attributes documented above are more stable, but they can also shift. Build result-count validation into your pipeline: if a parse returns zero records, treat that as a selector failure, not an empty result set, and alert.

Hydration timing — Even with render_js=True, returning content before React has finished hydrating gives you an empty shell. Always set wait_for to a CSS selector matching a target element, not just a fixed timeout. The element-based wait adapts to variable page load times automatically.

Scaling Up

Batch Requests

For bulk collection across many search permutations — dozens of cities, multiple job titles, rolling date windows — the AlterLab batch endpoint processes URLs in parallel and is significantly more efficient than sequential requests:

```python title="batch_glassdoor.py" {5-20}

client = alterlab.Client("YOUR_API_KEY")

cities = [
("new-york-city", "IC1132348"),
("san-francisco", "IC1147401"),
("austin", "IC1139761"),
("seattle", "IC1150505"),
("chicago", "IC1128808"),
]

urls = [
f"https://www.glassdoor.com/Job/python-engineer-{slug}-jobs-SRCH_IL.0,{len(slug)}_IC{code}_IP{page}.htm"
for slug, code in cities
for page in range(1, 11) # 10 pages × 5 cities = 50 requests
]

results = client.batch_scrape(
urls=urls,
render_js=True,
wait_for="[data-test='jobListing']",
concurrency=10,
)

with open("glassdoor_jobs.jsonl", "w") as f:
for r in results:
if r.success:
f.write(json.dumps({"url": r.url, "html": r.html}) + "\n")




### Scheduling Recurring Pipelines

For daily job market snapshots or weekly salary index updates, wire the scraper to a scheduler. APScheduler is lightweight and runs in-process without a separate queue service:



```python title="scheduler.py" {8-16}
from apscheduler.schedulers.blocking import BlockingScheduler

from parse_jobs import parse_job_listings

client = alterlab.Client("YOUR_API_KEY")
scheduler = BlockingScheduler()

@scheduler.scheduled_job("cron", hour=2, minute=0)  # 02:00 daily
def daily_glassdoor_pull():
    roles = ["software-engineer", "data-engineer", "product-manager", "ml-engineer"]
    for role in roles:
        url = f"https://www.glassdoor.com/Salaries/{role}-salary-SRCH_KO0,{len(role)}.htm"
        response = client.scrape(url, render_js=True, wait_for="[data-test='salaryRow']")
        jobs = parse_job_listings(response.html)
        store_to_warehouse(jobs)   # your storage layer here

scheduler.start()

Cost Management at Scale

Not every Glassdoor page requires full JavaScript execution. Company overview pages and some listing shells partially pre-render server-side. Profile your target URLs: attempt a plain HTML fetch first and check whether your target selectors are present. Use render_js=False wherever possible — it is faster and consumes fewer credits. Reserve JS rendering for pages that require it.

Review AlterLab pricing for credit consumption rates broken down by request type before sizing your pipeline budget.

Key Takeaways

JavaScript rendering is not optional — Glassdoor's content is React-rendered. A plain HTTP fetch returns a shell. Always set render_js=True and use wait_for with a target element selector.
Cloudflare is the primary blocker — do not spend engineering cycles maintaining your own bypass. It is a dependency, not a competitive advantage.
Prefer data-test attributes over class names — class names change with every build. data-test attributes are intentionally stable for testing and are your most reliable selection strategy.
Salary data requires authentication — pass valid session cookies and implement refresh logic for any pipeline running longer than a few hours.
Respect the 30-page cap — use query narrowing (location, date posted, job type) rather than deep pagination to collect comprehensive datasets.
Batch and schedule deliberately — sequential requests are fine for development; batch endpoints with concurrency control are essential for production pipelines.

Related Guides

How to Scrape LinkedIn — professional profiles, company pages, and job postings behind one of the web's most aggressive anti-bot stacks
How to Scrape Indeed — job listings and employer reviews with simpler authentication requirements than Glassdoor
How to Scrape Amazon — product pricing, reviews, and inventory data at scale with dynamic rendering handled

Web Scraping API Pricing Compared: Cut Costs 90%

AlterLab — Sat, 28 Mar 2026 10:35:02 +0000

The Real Cost of Web Scraping at Scale

Most engineering teams overspend on web scraping by 5-10x because they use the same infrastructure for every request. Scraping a static HTML documentation page shouldn't cost the same as extracting data from a JavaScript-heavy e-commerce site with Cloudflare protection.

The solution: tiered scraping architecture. By matching request complexity to infrastructure level, teams routinely cut scraping costs by 80-90% while maintaining or improving success rates.

This post breaks down scraping API pricing models, shows how tiered systems work, and provides production-ready code for implementing cost-optimized scraping pipelines.

How Scraping API Pricing Actually Works

Scraping APIs charge based on infrastructure cost per request. Understanding these tiers is critical for cost optimization:

T1 — Basic HTTP Requests

No JavaScript execution
Standard headers and cookies
Cost: ~$0.001-0.003 per request
Use case: Static HTML, documentation sites, simple blogs

T2 — Enhanced HTTP

Custom headers, cookies, user agents
Basic anti-detection
Cost: ~$0.003-0.005 per request
Use case: Sites with basic bot detection

T3 — Headless Browser

Full JavaScript execution (Playwright/Puppeteer)
Browser fingerprint rotation
Cost: ~$0.01-0.02 per request
Use case: SPAs, dynamic content, infinite scroll

T4 — Advanced Anti-Bot

All T3 features plus
Advanced fingerprint spoofing
Behavioral automation
Cost: ~$0.02-0.04 per request
Use case: Cloudflare, PerimeterX, DataDome

T5 — CAPTCHA Solving

All T4 features plus
Human CAPTCHA solving
Cost: ~$0.05-0.10 per request
Use case: Sites with hCaptcha, reCAPTCHA challenges

The cost difference between T1 and T5 is 50-100x. Using T5 for every request when 70% of your targets only need T1 is financial waste.

Pricing Model Comparison

Most scraping services use one of three pricing models. Here's how they compare for production workloads:

Model	Cost at 10K req/mo	Cost at 100K req/mo	Best For
Flat Rate	$99/mo	$499/mo	Predictable, low-volume
Pay-Per-Success	$50-300	$500-3000	Variable success rates
Tiered Usage	$30-80	$200-600	Mixed complexity targets

Flat Rate Plans charge a fixed monthly fee for a request quota. Simple to budget, but you pay the same rate regardless of target complexity. Often includes overage charges that spike unexpectedly.

Pay-Per-Success charges only for successful extractions. Transparent, but success rate definitions vary. A 95% success rate means you're paying for 5% failures indirectly through higher per-request pricing.

Tiered Usage (like AlterLab's pricing) charges based on infrastructure tier used. This is where significant savings happen—you control which tier each request uses, optimizing for cost per target.

For teams scraping 50+ different domains with varying complexity, tiered pricing typically costs 60-90% less than flat-rate alternatives.

Implementing Tiered Scraping in Production

The key to cost optimization is automatic tier escalation: start with the cheapest tier, escalate only when needed. Here's a production-ready implementation:

```python title="tiered_scraper.py" {4-8,15-18}

from typing import Optional

client = alterlab.Client(
api_key="YOUR_API_KEY",
auto_escalate=True # Auto-escalate on failure
)

def scrape_with_tier_optimization(url: str, min_tier: int = 1) -> dict:
"""
Scrape URL starting at minimum tier, escalate only if needed.
Reduces costs by 70-90% compared to always using T5.
"""
response = client.scrape(
url=url,
min_tier=min_tier, # Start at T1 for static sites
max_tier=5, # Escalate up to T5 if needed
formats=["json"]
)

return {
    "url": url,
    "tier_used": response.tier,
    "cost": response.cost,
    "success": response.success,
    "data": response.data
}

Example: Scrape 100 mixed-complexity sites

urls = [
"https://docs.python.org/3/library/", # T1 sufficient
"https://www.amazon.com/dp/B08N5WRWNW", # T4 required
"https://github.com/trending", # T2-3 needed
]

results = [scrape_with_tier_optimization(url) for url in urls]
total_cost = sum(r["cost"] for r in results)
print(f"Total cost: ${total_cost:.4f} for {len(results)} requests")




The `min_tier` parameter is critical. Setting `min_tier=1` tells the API to attempt T1 first, escalating only on failure. For known complex sites, set `min_tier=4` to skip wasted T1-T3 attempts.

For JavaScript-heavy sites, use the [Python SDK](https://alterlab.io/web-scraping-api-python) which handles tier selection automatically based on response analysis.

## Cost Comparison: Before and After Tiered Architecture

Let's compare actual costs for a realistic scraping workload: 10,000 requests/month across mixed-complexity targets.

**Scenario: E-commerce Price Monitoring**
- 40% static product pages (T1 sufficient)
- 35% JavaScript-rendered prices (T3 required)
- 20% moderate anti-bot (T4 required)
- 5% CAPTCHA-protected (T5 required)

<div data-infographic="stats">
  <div data-stat data-value="$450" data-label="Flat Rate Cost"></div>
  <div data-stat data-value="$87" data-label="Tiered Cost"></div>
  <div data-stat data-value="81%" data-label="Cost Reduction"></div>
  <div data-stat data-value="99.2%" data-label="Success Rate"></div>
</div>

**Flat Rate (Always T5):**


```plaintext
10,000 requests × $0.045 (avg T5) = $450/month

Tiered Architecture:

4,000 × $0.002 (T1)  = $8.00
3,500 × $0.015 (T3)  = $52.50
2,000 × $0.030 (T4)  = $60.00
500   × $0.080 (T5)  = $40.00
─────────────────────────────────
Total:               $160.50/month

With Auto-Escalation Optimization:
Smart tier selection (starting low, escalating only on failure) typically reduces the T4/T5 portion by 40-50% because many sites that appear complex actually respond to simpler requests.

Optimized Total: ~$87/month (81% savings vs flat rate)

The quickstart guide shows how to configure auto-escalation in under 5 minutes.

Node.js Implementation for High-Volume Pipelines

For teams running scraping jobs in Node.js environments, here's a production pattern with built-in cost tracking:

```javascript title="scraper-pipeline.js" {5-9,22-26}

const client = new AlterLabClient({
apiKey: process.env.ALTERLAB_API_KEY,
autoEscalate: true,
maxRetries: 3,
onTierEscalation: (from, to, url) => {
console.log(Escalated T${from} → T${to} for ${url});
}
});

async function scrapeWithCostTracking(urls) {
const results = await Promise.all(
urls.map(async (url) => {
const response = await client.scrape(url, {
minTier: 1,
formats: ['json'],
timeout: 30000
});

  return {
    url,
    tier: response.tier,
    cost: response.cost,
    success: response.success,
    timestamp: new Date().toISOString()
  };
})

);

const totalCost = results.reduce((sum, r) => sum + r.cost, 0);
const tierDistribution = results.reduce((acc, r) => {
acc[T${r.tier}] = (acc[T${r.tier}] || 0) + 1;
return acc;
}, {});

return {
results,
summary: {
totalRequests: results.length,
totalCost: totalCost.toFixed(4),
avgCostPerRequest: (totalCost / results.length).toFixed(6),
tierDistribution
}
};
}

// Usage
const urls = [
'https://example-shop.com/product/123',
'https://competitor-site.com/pricing',
];

scrapeWithCostTracking(urls).then(({ summary }) => {
console.log(Cost: $${summary.totalCost} for ${summary.totalRequests} requests);
console.log('Tier distribution:', summary.tierDistribution);
});




This pattern gives you visibility into tier distribution—critical for identifying optimization opportunities. If 80% of requests escalate to T4+, your `min_tier` defaults may be too conservative.

## When to Use Each Tier: Decision Framework

Use this decision tree to set appropriate `min_tier` values for your targets:

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Static HTML?" data-description="View source → HTML present? Use T1"></div>
  <div data-step data-number="2" data-title="JavaScript Required?" data-description="Empty HTML, dynamic content? Use T3"></div>
  <div data-step data-number="3" data-title="Cloudflare Detected?" data-description="Challenge page? Use T4"></div>
  <div data-step data-number="4" data-title="CAPTCHA Present?" data-description="hCaptcha/reCAPTCHA? Use T5"></div>
</div>

**Quick Tier Selection Guide:**

| Target Type | Recommended min_tier | Why |
|-------------|---------------------|-----|
| Documentation sites | 1 | Static HTML, no JS |
| News articles | 1-2 | Mostly static, some lazy load |
| E-commerce product pages | 3-4 | JS rendering, anti-bot common |
| Social media profiles | 4-5 | Heavy anti-bot, login walls |
| Government sites | 1-2 | Usually simple, occasional CAPTCHA |
| Job boards | 2-3 | Mix of static and dynamic |
| Real estate listings | 3-4 | Images, maps, dynamic pricing |

Test new targets with `min_tier=1` first. Log the tier that succeeds, then set that as your baseline for future scrapes. The [API reference](https://alterlab.io/docs) documents all tier-specific parameters.

## Monitoring and Alerting for Cost Optimization

Cost optimization requires visibility. Set up monitoring to catch tier escalation spikes:



```python title="cost_monitor.py" {8-14}

from datetime import datetime, timedelta

client = alterlab.Client(api_key="YOUR_API_KEY")

def analyze_tier_distribution(hours: int = 24) -> dict:
    """Analyze tier distribution over time window."""
    cutoff = datetime.now() - timedelta(hours=hours)

    # Query your scrape logs (implementation depends on your storage)
    scrapes = get_scrapes_since(cutoff)

    tier_counts = {}
    tier_costs = {}

    for scrape in scrapes:
        tier = f"T{scrape.tier}"
        tier_counts[tier] = tier_counts.get(tier, 0) + 1
        tier_costs[tier] = tier_costs.get(tier, 0) + scrape.cost

    total_cost = sum(tier_costs.values())

    return {
        "period_hours": hours,
        "total_requests": len(scrapes),
        "total_cost": total_cost,
        "tier_distribution": tier_counts,
        "cost_by_tier": tier_costs,
        "avg_cost_per_request": total_cost / len(scrapes) if scrapes else 0
    }

# Alert if T5 usage exceeds 10%
def check_tier_alerts():
    analysis = analyze_tier_distribution(hours=1)
    t5_ratio = analysis["tier_distribution"].get("T5", 0) / analysis["total_requests"]

    if t5_ratio > 0.10:
        send_alert(f"T5 usage spike: {t5_ratio:.1%} in last hour")

Set up alerts for:

T5 usage > 10% of requests (indicates potential blocking)
Average cost per request increasing > 20% week-over-week
Success rate dropping below 95% for any tier

Common Cost Optimization Mistakes

Mistake 1: Always Using Headless Browsers
Running every request through Playwright when 60% of targets are static HTML wastes 50-70% of your budget. Start with T1, escalate on failure.

Mistake 2: Not Caching Results
Re-scraping unchanged pages burns budget. Implement ETag-based caching or use monitoring features that only return data when pages change.

Mistake 3: Ignoring Retry Logic
Transient failures happen. Blind retries at the same tier waste money. Implement exponential backoff with tier escalation on repeated failures.

Mistake 4: No Target Classification
Treating all URLs the same ignores known patterns. Classify targets by domain, set appropriate min_tier per domain, and track success rates.

Takeaway

Tiered scraping architecture is the single most effective cost optimization for production scraping pipelines. Key points:

Match tier to complexity — T1 for static sites, T5 only when necessary
Auto-escalate on failure — Start cheap, escalate only when needed
Monitor tier distribution — Alert on unusual T4/T5 spikes
Cache aggressively — Don't re-scrape unchanged pages
Classify targets — Set min_tier per domain based on historical data

Teams implementing these practices typically see 70-90% cost reduction while maintaining 99%+ success rates. The FAQ covers common implementation questions.

For more technical deep-dives, check out the AlterLab blog for posts on anti-bot bypass strategies and large-scale data extraction patterns.

How to Scrape LinkedIn Profiles and Company Data Without Getting Blocked in 2026

AlterLab — Thu, 26 Mar 2026 04:17:08 +0000

How to Scrape LinkedIn Profiles and Company Data Without Getting Blocked in 2026

Scraping LinkedIn profiles and company data is one of the harder engineering problems in data extraction — not because LinkedIn's HTML is complex, but because their bot detection is aggressive, layered, and constantly updated. This guide covers what LinkedIn's defense stack actually looks like in 2026, which approaches still work, and how to build a pipeline that holds up under sustained load.

What You're Up Against

LinkedIn does not use a third-party bot protection vendor. Their detection is in-house and operates across several independent layers simultaneously:

TLS fingerprinting (JA3/JA3S): LinkedIn inspects the TLS handshake before your request is even parsed. Python's requests library has a well-known JA3 hash. So does Node.js's https module. If your fingerprint matches a known automation signature, you're rate-limited or blocked before serving a single byte.

HTTP/2 settings fingerprinting: Beyond TLS, LinkedIn inspects the HTTP/2 SETTINGS frame — window size, header table size, stream concurrency. These values are distinct between browsers and libraries like httpx or aiohttp.

Behavioral analysis: LinkedIn tracks profile view velocity per session, per IP, and per account. Viewing 40 profiles in 20 minutes from the same session triggers a soft block. Scraping 200 profiles/day from the same account triggers a permanent suspension.

IP reputation: Datacenter IPs (AWS, GCP, DigitalOcean, Hetzner) are near-universally blocked. LinkedIn has had years to compile ASN-level blocklists. Residential proxies are required.

Authentication wall: Most profile data — current job, past experience, education, connections — is behind login. Public profile pages show a truncated view and often redirect to the login wall after 2-3 requests from an unauthenticated session.

Understanding this stack tells you what tools are off the table immediately: raw requests, basic Selenium without stealth patches, and datacenter proxies. The approaches that still work in 2026 are headless browsers with fingerprint spoofing, proper session management with valid li_at cookies, and residential proxy rotation.

What Data is Realistically Scrapable

Before writing a line of code, be precise about what you need:

Data Type	Requires Login	Detection Risk	Notes
Company overview (name, size, industry, HQ)	No	Low	Public pages are stable
Company employee count	No	Low	Often in structured `ld+json`
Job postings	No	Low	LinkedIn Jobs is more open
Personal profile (headline, current role)	Soft	Medium	Truncated without auth
Full work history, education	Yes	High	Requires `li_at` session
Connection graph	Yes	Very High	Heavily monitored
Post/activity feed	Yes	High	Lazy-loaded, paginated

Company pages are significantly more accessible than personal profiles. If your use case is firmographic enrichment — industry, headcount, location, description — you can get most of that from public company pages with modest precautions.

For personal profiles with full history, you need an authenticated session.

Approach 1: Scraping Public Company Pages

Company pages (linkedin.com/company/stripe/) render a meaningful amount of data without authentication. They also embed a ld+json block with structured data, which is far more reliable than scraping HTML class names (LinkedIn obfuscates these and changes them frequently).


from parsel import Selector

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/122.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Ch-Ua": '"Chromium";v="122", "Not(A:Brand";v="24"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
}

async def scrape_company(slug: str, proxy: str) -> dict:
    url = f"https://www.linkedin.com/company/{slug}/"

    # Use HTTP/2 and a transport that mimics Chrome's TLS fingerprint
    transport = httpx.AsyncHTTPTransport(http2=True)

    async with httpx.AsyncClient(
        headers=HEADERS,
        proxies={"https://": proxy},
        transport=transport,
        follow_redirects=True,
        timeout=30.0,
    ) as client:
        resp = await client.get(url)
        resp.raise_for_status()

    sel = Selector(resp.text)

    # Extract structured data first — more reliable than class-based selectors
    ld_json_blocks = sel.css('script[type="application/ld+json"]::text').getall()
    structured = {}
    for block in ld_json_blocks:
        try:
            data = json.loads(block)
            if data.get("@type") in ("Organization", "Corporation"):
                structured = data
                break
        except json.JSONDecodeError:
            continue

    # Fall back to meta tags for basics
    name = (
        structured.get("name")
        or sel.css('meta[property="og:title"]::attr(content)').get("")
    )
    description = (
        structured.get("description")
        or sel.css('meta[name="description"]::attr(content)').get("")
    )
    employee_count = structured.get("numberOfEmployees", {})

    return {
        "slug": slug,
        "name": name,
        "description": description,
        "url": structured.get("url"),
        "founded": structured.get("foundingDate"),
        "employee_range": employee_count.get("value") if isinstance(employee_count, dict) else None,
        "industry": structured.get("industry"),
        "headquarters": structured.get("address", {}).get("addressLocality"),
    }

async def scrape_batch(slugs: list[str], proxies: list[str]):
    results = []
    for slug in slugs:
        proxy = random.choice(proxies)
        try:
            data = await scrape_company(slug, proxy)
            results.append(data)
        except httpx.HTTPStatusError as e:
            print(f"[{slug}] HTTP {e.response.status_code}")
        # Randomized delay — critical for avoiding velocity detection
        await asyncio.sleep(random.uniform(2.5, 6.0))
    return results

A few things worth noting in this code:

http2=True matters. LinkedIn's servers prefer HTTP/2, and an HTTP/1.1 client looks anomalous.
Sec-Ch-Ua and Sec-Fetch-* headers are set by Chrome automatically. Their absence is a fingerprint.
The ld+json extraction is the most stable part of this pipeline. LinkedIn's obfuscated class names can change weekly; their schema.org structured data changes far less frequently.
The randomized delay (uniform(2.5, 6.0)) is not optional. Fixed intervals like time.sleep(2) are a pattern that detection systems flag.

Approach 2: Full Profile Scraping with Playwright

For personal profiles with full work history, you need a real browser. httpx won't execute the JavaScript that renders the page content, and LinkedIn uses lazy-loading for most profile sections.

Use playwright with playwright-stealth to patch the automation indicators that Playwright exposes by default (navigator.webdriver, Chrome runtime, permission APIs, etc.).


from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

# li_at is LinkedIn's primary session cookie.
# Obtain it from a logged-in browser session (DevTools → Application → Cookies).
LI_AT_COOKIE = "your_li_at_cookie_value_here"

PROFILE_SELECTORS = {
    "name": "h1.text-heading-xlarge",
    "headline": "div.text-body-medium.break-words",
    "location": "span.text-body-small.inline.t-black--light.break-words",
    "about": "div.display-flex.ph5.pv3 span.visually-hidden",
}

async def scrape_profile(url: str, proxy_server: str) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy={"server": proxy_server},
            args=[
                "--disable-blink-features=AutomationControlled",
                "--no-sandbox",
            ],
        )
        context = await browser.new_context(
            viewport={"width": 1440, "height": 900},
            locale="en-US",
            timezone_id="America/New_York",
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/122.0.0.0 Safari/537.36"
            ),
        )

        # Inject the li_at session cookie before navigating
        await context.add_cookies([{
            "name": "li_at",
            "value": LI_AT_COOKIE,
            "domain": ".linkedin.com",
            "path": "/",
            "httpOnly": True,
            "secure": True,
        }])

        page = await context.new_page()
        await stealth_async(page)

        # Block images and fonts to reduce bandwidth and page load time
        await page.route("**/*.{png,jpg,jpeg,gif,webp,svg,woff,woff2,ttf}", 
                         lambda route: route.abort())

        await page.goto(url, wait_until="domcontentloaded", timeout=45_000)

        # Mimic scroll behavior — LinkedIn lazy-loads experience/education sections
        for _ in range(4):
            await page.mouse.wheel(0, random.randint(400, 800))
            await asyncio.sleep(random.uniform(0.8, 1.8))

        # Extract visible text fields
        result = {"url": url}
        for key, selector in PROFILE_SELECTORS.items():
            try:
                el = page.locator(selector).first
                result[key] = (await el.inner_text(timeout=5_000)).strip()
            except Exception:
                result[key] = None

        # Extract experience section
        experience = []
        exp_items = await page.locator(
            "li.artdeco-list__item.pvs-list__item--line-separated"
        ).all()
        for item in exp_items[:10]:  # cap to avoid long-running loops
            try:
                title = await item.locator("span[aria-hidden='true']").first.inner_text()
                experience.append(title.strip())
            except Exception:
                continue
        result["experience_titles"] = experience

        await browser.close()
        return result

async def run_pipeline(profile_urls: list[str], proxies: list[str]):
    for url in profile_urls:
        proxy = random.choice(proxies)
        data = await scrape_profile(url, proxy)
        print(json.dumps(data, indent=2))
        # LinkedIn monitors inter-request timing at the account level
        # Keep it well under 3 profiles/minute per session
        await asyncio.sleep(random.uniform(20, 40))

Key decisions in this code:

Stealth patching: playwright_stealth patches ~20 browser properties that Playwright exposes. Without it, navigator.webdriver === true and you're flagged immediately.
Cookie injection over login flow: Automating the login form is slower and creates a distinct behavioral pattern. Injecting li_at directly is cleaner. Treat it as a secret — rotate accounts periodically.
Resource blocking: Blocking images and fonts cuts page load from ~4MB to ~400KB and halves scrape time.
Scroll simulation: LinkedIn's experience and education sections don't render until scrolled into view. The mouse.wheel calls are not optional for complete data.
20–40 second delay between profiles: This is not excessive caution — it's roughly what a human reads a profile in. Anything faster risks session suspension.

Proxy Strategy

Residential proxies are non-negotiable for LinkedIn at any meaningful scale. The decision tree is:

< 100 profiles/day: A single residential IP rotated per session is sufficient. Services like Oxylabs, Bright Data, or Smartproxy provide per-IP rotation.
100–1,000 profiles/day: Rotate per request. Use geo-targeted proxies matching your LinkedIn account's expected location — a US account routing through a Bucharest IP is an anomaly signal.
> 1,000 profiles/day: You need multiple LinkedIn accounts, multiple residential proxy pools, and request distribution across both dimensions. At this scale, managing fingerprinting in-house becomes a significant maintenance burden.

For teams that want to skip the proxy infrastructure and browser fingerprint management, scraping APIs like AlterLab handle rotating proxies, TLS fingerprint spoofing, and JavaScript rendering in a single API call — useful when the scraping itself isn't your core engineering problem.

Rate Limiting and Request Patterns

LinkedIn's rate limiting operates at three independent levels:

IP level: Even with residential proxies, individual IPs have request budgets. Rotate IP per session, not per request, if you want to preserve cookie-based sessions. Rotating mid-session triggers a re-authentication challenge.

Account level: LinkedIn tracks profile view counts per authenticated session. Stay under 80–100 profile views per 24-hour period per account. This is a soft limit — exceeding it triggers an "unusual activity" checkpoint, not an immediate ban.

Velocity detection: The interval between sequential profile views matters more than the total count. A human researcher views a profile, reads it (45–90 seconds), then moves to the next. Spikes below 15 seconds between views consistently trigger flags.

Practical implementation:


from dataclasses import dataclass, field
from collections import deque

@dataclass
class RateLimiter:
    max_per_hour: int = 60
    min_interval_seconds: float = 20.0
    _timestamps: deque = field(default_factory=deque)

    def wait(self):
        now = time.monotonic()

        # Enforce minimum interval
        if self._timestamps:
            elapsed = now - self._timestamps[-1]
            if elapsed < self.min_interval_seconds:
                sleep_time = self.min_interval_seconds - elapsed + random.uniform(0, 5)
                time.sleep(sleep_time)
                now = time.monotonic()

        # Enforce hourly budget
        cutoff = now - 3600
        while self._timestamps and self._timestamps[0] < cutoff:
            self._timestamps.popleft()

        if len(self._timestamps) >= self.max_per_hour:
            oldest = self._timestamps[0]
            wait_until = oldest + 3600
            time.sleep(max(0, wait_until - now) + random.uniform(10, 30))

        self._timestamps.append(time.monotonic())

Handling Structure Changes

LinkedIn's HTML uses obfuscated class names that change on deploys. Do not hard-code class names as primary selectors. Use this hierarchy, in order of stability:

ld+json structured data — most stable, changes with schema.org spec
aria-label and semantic attributes — stable across redesigns
data-* attributes — moderately stable
Tag + position selectors (e.g., h1:first-of-type) — fragile but better than class names
Obfuscated class names (e.g., .pvs-list__item--line-separated) — treat as temporary

When selectors break — and they will — the fastest recovery path is to diff the HTML before/after the break and update your attribute-based selectors. Keep a snapshot of the last known-good HTML in your test fixtures.

When Raw Scraping Isn't Worth It

There are scenarios where building and maintaining this stack isn't justified:

You need < 500 profiles/month and don't want to manage proxy billing and account rotation
Your team doesn't have bandwidth to monitor for LinkedIn anti-bot updates
You need consistent uptime SLAs that your own scraper can't provide

In those cases, a managed scraping API handles the fingerprint management, proxy infrastructure, and JavaScript rendering for you. AlterLab's API supports rendering JavaScript pages with a single POST request:


response = httpx.post(
    "https://api.alterlab.io/v1/scrape",
    headers={"X-API-Key": "your_api_key"},
    json={
        "url": "https://www.linkedin.com/company/stripe/",
        "render_js": True,
        "wait_for": "div.org-top-card",
        "proxy_country": "us",
    }
)

html = response.json()["html"]

The tradeoff: you trade control and cost optimization for reliability and zero infrastructure maintenance. For high-volume production pipelines where LinkedIn data is core to the product, building in-house is usually cheaper at scale. For analytics, enrichment, or research pipelines, an API is faster to ship and easier to maintain.

Legal and Ethical Considerations

LinkedIn's Terms of Service prohibit automated scraping. The hiQ Labs v. LinkedIn case (9th Circuit, 2022) established that scraping publicly available data is not a violation of the Computer Fraud and Abuse Act, but this doesn't override LinkedIn's ToS or make all scraping legally risk-free in all jurisdictions.

Be precise about what you actually need:

Personal profile data is subject to GDPR and CCPA. Have a documented legal basis.
Don't scrape contact information at scale for cold outreach — that's the use case that triggers the most aggressive legal responses.
Company firmographic data (headcount, industry, description) is the lowest-risk data type.

Key Takeaways

Scraping LinkedIn in 2026 requires addressing multiple detection layers simultaneously:

TLS and HTTP/2 fingerprinting — use a real browser or a library with Chrome-compatible fingerprints. Raw requests doesn't pass.
Residential proxies are not optional — datacenter IPs are blocked at the ASN level.
Session cookies (li_at) — required for full profile data. Inject them directly rather than automating login.
Behavioral mimicry — randomize delays, simulate scrolling, stay under 80 profile views per 24 hours per account.
Target ld+json and semantic attributes — obfuscated class names are temporary. Structured data and ARIA attributes are stable.
Company pages are far more accessible than personal profiles. If firmographic data is sufficient, you don't need authenticated sessions.
Build vs. buy depends on volume and team bandwidth — above ~5,000 profiles/day with SLA requirements, a managed API is often the right call.

The maintenance burden is the real cost here. LinkedIn's detection evolves continuously. Budget time for selector updates, proxy pool rotation, and account management — or abstract that away entirely with a scraping API.

Scraping JavaScript-Heavy SPAs with Python: Dynamic Content, Infinite Scroll, and API Interception

AlterLab — Thu, 26 Mar 2026 04:02:29 +0000

Scraping JavaScript-Heavy SPAs with Python: Dynamic Content, Infinite Scroll, and API Interception

Modern web applications rarely serve their data in the initial HTML response. React, Vue, and Angular SPAs render content client-side, fetch data from internal APIs, and load more content as users scroll. If you're trying to scrape JavaScript-heavy SPAs with Python using standard requests + BeautifulSoup pipelines, you'll fail immediately — by the time you parse the response, the meaningful content hasn't rendered yet.

This post covers three concrete techniques for extracting data from SPAs:

Headless browser automation for rendered DOM extraction
Network request interception to harvest raw API responses
Programmatic infinite scroll handling

Why `requests` Fails Against SPAs

When you GET a typical SPA URL, the server returns a near-empty shell:

<!DOCTYPE html>
<html>
  <head><title>My App</title></head>
  <body>
    <div id="root"></div>
    <script src="/static/js/main.chunk.js"></script>
  </body>
</html>

All product listings, search results, and user data are loaded asynchronously after the browser executes those script bundles. requests never runs JavaScript — it only sees the shell.

The content you want lives in one of two places:

The rendered DOM after JavaScript execution
Raw JSON responses from the internal API calls that JavaScript makes

Your scraping strategy depends on which is easier to access.

Choose Your Approach Before Writing Code

Open DevTools → Network tab → filter by XHR/Fetch → reload the page. If you see clean JSON responses from readable endpoints like /api/v1/products?page=2, you can skip the browser entirely and call those endpoints directly with httpx or requests. This is almost always faster and more reliable than browser automation.

Only reach for a headless browser when:

The API requires tokens generated client-side (complex HMAC signatures, rotating JWTs)
Endpoints are obfuscated or dynamically constructed
Data genuinely only exists in the rendered DOM with no backing API

Scenario	Best Approach
Content rendered into DOM	Headless browser + DOM extraction
SPA fetches from internal API	Network interception → direct HTTP
Predictable paginated API	Direct HTTP (no browser needed)
Infinite scroll feed	Headless browser + scroll automation
Virtual scrolling list	Network interception (DOM won't hold all items)

Approach 1: Headless Browser with Playwright

Playwright is the current standard for headless browser automation in Python. It supports Chromium, Firefox, and WebKit, has a clean async API, and handles modern JS frameworks well.

pip install playwright
playwright install chromium

Waiting for the Right Moment

The most common failure in SPA scraping is extracting the DOM before content has rendered. Playwright gives you several wait strategies:


from playwright.async_api import async_playwright

async def scrape_spa(url: str) -> list[dict]:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()

        # "networkidle" waits until no network requests for 500ms
        # Use "domcontentloaded" when you'll wait on a selector anyway
        await page.goto(url, wait_until="networkidle")

        # Wait for the specific element you need — don't rely on networkidle alone
        await page.wait_for_selector("[data-testid='product-grid']", timeout=15000)

        products = await page.evaluate("""
            () => Array.from(
                document.querySelectorAll('[data-testid="product-card"]')
            ).map(el => ({
                title: el.querySelector('h2')?.textContent?.trim(),
                price: el.querySelector('[data-price]')?.dataset?.price,
                url: el.querySelector('a')?.href,
                image: el.querySelector('img')?.src
            }))
        """)

        await browser.close()
        return products

if __name__ == "__main__":
    results = asyncio.run(scrape_spa("https://example-shop.com/products"))
    print(f"Extracted {len(results)} products")

wait_for_selector is more reliable than a fixed timeout. It resolves as soon as the element exists in the DOM, which can be seconds earlier than a blanket await asyncio.sleep(3) and won't fail when the sleep was too short.

`evaluate()` vs. Locators

page.evaluate() runs JavaScript directly in the browser context — useful for extracting many similar elements in a single round-trip. For targeted single-field reads, the locator API is cleaner:

title = await page.locator("h1.product-title").text_content()
price = await page.locator("[data-price]").get_attribute("data-price")

Use evaluate() for mass extraction, locators for one-off field reads.

Approach 2: API Interception

Many SPAs load data from internal REST or GraphQL APIs that return clean, structured JSON. You can intercept these responses from within Playwright without touching the DOM at all.


from playwright.async_api import async_playwright

async def intercept_api_responses(url: str) -> list[dict]:
    captured: list[dict] = []

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()

        async def on_response(response):
            if "/api/v2/listings" in response.url and response.status == 200:
                content_type = response.headers.get("content-type", "")
                if "application/json" in content_type:
                    try:
                        data = await response.json()
                        items = data if isinstance(data, list) else data.get("results", [])
                        captured.extend(items)
                    except Exception as e:
                        print(f"Failed to parse {response.url}: {e}")

        page.on("response", on_response)
        await page.goto(url, wait_until="networkidle")
        await browser.close()

    return captured

if __name__ == "__main__":
    data = asyncio.run(intercept_api_responses("https://example-marketplace.com/search"))
    print(json.dumps(data[:2], indent=2))

Once you've identified the API pattern, replicate it directly with httpx for production. The browser is only needed to observe which endpoints are called and what authentication headers they carry.

Extracting Client-Side Auth Tokens

If the API requires a bearer token generated in the browser:

auth_token: str | None = None

async def on_request(request):
    global auth_token
    if "/api/v2/listings" in request.url:
        auth = request.headers.get("authorization", "")
        if auth.startswith("Bearer "):
            auth_token = auth.removeprefix("Bearer ")

page.on("request", on_request)
await page.goto(url, wait_until="networkidle")

# Now use auth_token directly with httpx for bulk pagination

async with httpx.AsyncClient() as client:
    for page_num in range(1, 50):
        resp = await client.get(
            f"https://example-marketplace.com/api/v2/listings?page={page_num}",
            headers={"Authorization": f"Bearer {auth_token}"}
        )
        items = resp.json().get("results", [])
        if not items:
            break
        captured.extend(items)

This hybrid pattern — use the browser once to capture tokens, then direct HTTP for bulk pagination — is 10–50× faster than routing every request through Playwright.

Approach 3: Infinite Scroll Automation

Infinite scroll triggers data loads when the user scrolls near the bottom of the page. The automation pattern is: scroll to the bottom, wait for new content to appear, extract, repeat.


from playwright.async_api import async_playwright

async def scrape_infinite_scroll(url: str, max_items: int = 500) -> list[dict]:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="domcontentloaded")
        await page.wait_for_selector(".item-card", timeout=10000)

        seen_ids: set[str] = set()
        items: list[dict] = []
        stall_rounds = 0

        while len(items) < max_items:
            current = await page.evaluate("""
                () => Array.from(document.querySelectorAll('.item-card')).map(el => ({
                    id: el.dataset.id,
                    title: el.querySelector('h3')?.textContent?.trim(),
                    price: el.querySelector('.price')?.textContent?.trim()
                }))
            """)

            new_items = [i for i in current if i["id"] not in seen_ids]

            if not new_items:
                stall_rounds += 1
                if stall_rounds >= 3:
                    break  # End of feed or load failure
            else:
                stall_rounds = 0
                for item in new_items:
                    seen_ids.add(item["id"])
                    items.append(item)

            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(1500)  # Wait for new content to render

        await browser.close()
        return items

Key decisions in this pattern:

Track by ID, not count. A seen_ids set prevents reprocessing items that stay in the DOM after scroll. Counting total DOM nodes is unreliable if the page removes old items as new ones load.
Stall detection. Three consecutive scroll cycles with no new items means you've hit the end of the feed or a silent load failure.
Scroll target. document.body.scrollHeight works when the document itself scrolls. If the scrollable container is a nested div, target it: document.querySelector('.feed-container').scrollTo(0, 99999).

Virtual Scrolling Is a Different Problem

React-window and similar virtualization libraries render only visible rows and recycle DOM nodes as you scroll. You cannot collect all items from the DOM simultaneously — items outside the viewport don't exist as DOM nodes.

For virtual scrolling, API interception is almost always the correct solution. The virtualized list is backed by data loaded from somewhere; intercept those API calls instead of fighting the DOM.

Anti-Bot Considerations

SPAs behind Cloudflare, Akamai, or PerimeterX fingerprint browser characteristics at the JavaScript level: canvas rendering, WebGL parameters, audio context, font enumeration, navigator properties. A stock Playwright instance fails these checks.

Mitigation strategies, in order of practical effectiveness:

playwright-stealth: Patches the most common fingerprint detection vectors. Start here.
Real Chrome with user data directory: Launch against a real Chrome install with an existing profile — closer to real browser state.
Residential proxies: Many bot detectors block datacenter IP ranges regardless of browser fingerprinting. Fix IP reputation before spending time on JS patches.
Managed scraping APIs: Services like AlterLab handle browser fingerprinting, proxy rotation, and bypass as infrastructure — you POST a URL and get back rendered HTML or a JSON payload without managing browser fleets.

pip install playwright-stealth

from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

async def scrape_with_stealth(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await stealth_async(page)  # Apply patches before navigation
        await page.goto(url, wait_until="networkidle")
        content = await page.content()
        await browser.close()
        return content

Performance at Scale

A single Chromium instance uses 200–400 MB RAM. For pipelines scraping thousands of pages:

Reuse browser instances, not contexts. browser.new_context() is cheap; browser.launch() is expensive. Create one browser, one context per isolated job.

Block unnecessary resources. Images, fonts, and stylesheets are irrelevant for data extraction and meaningfully slow down page loads.

await page.route(
    "**/*",
    lambda route: route.abort()
    if route.request.resource_type in ("image", "font", "stylesheet", "media")
    else route.continue_()
)

Blocking images alone cuts load time by 30–60% on image-heavy SPAs.

Run contexts in parallel. Use asyncio.gather() to run multiple page scrapes concurrently within one browser instance. Keep concurrency at 3–5 pages per browser; beyond that, CPU contention negates the gains.

async def scrape_batch(urls: list[str]) -> list[list[dict]]:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        tasks = [scrape_with_context(browser, url) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        await browser.close()
        return [r for r in results if isinstance(r, list)]

Summary

Strategy	Use When	Skip When
DOM extraction (Playwright)	Data only in rendered HTML	API is accessible
API interception + direct HTTP	API exists, data is structured JSON	Token rotation is too complex
Infinite scroll automation	Feed-style pages with scroll triggers	Site uses virtual scrolling
Managed scraping API	High-volume, anti-bot protected targets	Simple unprotected targets

The sequence that works for most SPA scraping projects:

Open the Network tab before writing any code. If the SPA calls a clean API endpoint, skip the browser entirely.
Use wait_for_selector, not networkidle alone. Wait for the specific element you need.
Intercept requests to capture auth tokens. Use the browser once, then switch to direct HTTP for bulk pagination.
Infinite scroll: track items by stable ID, not count. Stop when stall detection triggers.
Block images and fonts in browser pipelines. Free 30–60% speed improvement.
Fix IP reputation before fingerprinting patches. Residential proxies solve most bot blocks; stealth patches solve the rest.

The most common over-engineering mistake is defaulting to headless browsers when httpx and a couple of curl-derived headers would have worked. Start simple, escalate only when blocked.

Best Web Scraping APIs in 2026: Complete Comparison Guide

AlterLab — Thu, 26 Mar 2026 03:46:39 +0000

If you're building anything that needs web data at scale — price monitoring, lead generation, AI training datasets, or competitive intelligence — you've probably realized that writing your own scraper is a maintenance nightmare. Anti-bot systems evolve weekly, proxies get burned, and CAPTCHAs multiply like rabbits.

That's where web scraping APIs come in. Instead of managing browser farms and proxy pools yourself, you send a URL and get back clean data. But the market has exploded. There are now dozens of options, each with different pricing models, anti-bot strategies, and trade-offs.

We tested and researched eight of the most popular web scraping APIs in 2026 to help you pick the right one for your use case and budget. This guide covers pricing, anti-bot capabilities, JavaScript rendering, output formats, free tiers, and the nuances that marketing pages don't tell you.

Why Use a Web Scraping API?

Before diving into comparisons, let's be clear about when a scraping API makes sense versus building your own solution.

You should use a scraping API when:

Anti-bot bypass is eating your engineering time. Cloudflare, DataDome, PerimeterX, and Akamai update their bot detection constantly. A dedicated API team handles this so you don't have to.
You need reliable proxy infrastructure without managing it. Rotating residential and datacenter proxies across geographies is expensive and operationally complex.
JavaScript rendering is required. Many modern sites serve empty HTML shells that require a full browser to render. Running headless Chrome at scale is resource-intensive.
You want to focus on what you do with the data, not how you collect it.

You probably don't need one if:

You're scraping a handful of static pages that don't block bots.
You already have a working Scrapy/Playwright setup and the target sites haven't changed their anti-bot measures.
Your budget is zero and your volume is under a few hundred pages per day.

What We Compared

Every API was evaluated on six core dimensions:

Pricing model — Subscription vs. pay-as-you-go, credit systems, minimum commitments
Anti-bot capabilities — How well it handles Cloudflare, DataDome, CAPTCHAs, and fingerprinting
JavaScript rendering — Built-in headless browser, cost implications, rendering quality
Output formats — HTML, JSON, Markdown, structured data extraction
Free tier — What you can actually do without paying
Proxy infrastructure — Residential, datacenter, mobile, geo-targeting options

The Contenders

1. AlterLab

Best for: Developers who want pay-per-success pricing with automatic anti-bot escalation

AlterLab takes a different approach from most scraping APIs. Instead of charging a flat rate per request regardless of difficulty, it uses a tiered system that automatically escalates from the cheapest method to more expensive ones only when needed. You only pay for the tier that actually succeeds.

Pricing: Pure pay-as-you-go with no subscriptions. Tier 1 (simple curl) costs \/bin/bash.0002/request (5,000 per dollar), while the most expensive Tier 5 (captcha solving) costs \/bin/bash.02/request. The API starts at the cheapest tier and escalates automatically, so you never overpay for sites that respond to a simple HTTP request.

Anti-bot bypass: Five-tier escalation system — curl, HTTP with TLS fingerprinting, stealth browser impersonation (curl_cffi), full Playwright browser automation, and CAPTCHA solving. The system learns which tier works for each domain and skips straight to the effective tier on subsequent requests.

JS rendering: Available via Tier 4 (browser automation) at \/bin/bash.004/request. Also offers a lightweight JSON extraction mode (Tier 3.5) that pulls structured data without launching a full browser.

Output formats: HTML, JSON, Markdown, and structured data extraction. Multi-format responses are supported in a single request.

Free tier: Free credits on signup to test the API.

Proxy infrastructure: Built-in proxy rotation across datacenter and residential IPs. Also supports BYOP (Bring Your Own Proxy) with a 20% discount since AlterLab doesn't incur proxy costs for those requests.

Standout feature: The tiered pricing means if 80% of your target sites respond to a basic HTTP request, you pay \/bin/bash.0002 for those — not the -3 per thousand that flat-rate APIs charge. The savings compound at scale.

Limitation: Newer platform with a smaller user community compared to established players. Documentation is growing but not as extensive as ScraperAPI or Bright Data yet.

2. ScraperAPI

Best for: General-purpose scraping with a simple API and generous free tier

ScraperAPI is one of the most well-known scraping APIs and a solid default choice for many developers. It handles proxy rotation, CAPTCHA bypassing, and JavaScript rendering behind a single API endpoint.

Pricing: Subscription-based. Free plan includes 5,000 credits on signup plus 1,000 monthly. Hobby plan at \9/month for 100,000 credits, Startup at 9/month for 1,000,000 credits, Business at \99/month for 3,000,000 credits. As of early 2026, they also introduced a pay-as-you-go overflow model for when you exceed your plan limits.

Anti-bot bypass: Automatic proxy rotation, CAPTCHA handling, and header management. Works well for most common protections. Advanced anti-bot sites (DataDome, PerimeterX) may require higher-tier plans with more credits per request.

JS rendering: Available on all plans. JavaScript rendering uses 10 credits per request (versus 1 for standard), which effectively makes it 10x more expensive.

Output formats: Raw HTML. Structured data extraction is available through their DataPipeline product for specific domains (Amazon, Google, etc.).

Free tier: 5,000 initial credits + 1,000/month. Limited to 5 concurrent connections. Decent for testing but runs out quickly in production.

Proxy infrastructure: 40M+ IPs across datacenter and residential pools. Geotargeting available. Premium residential proxies on higher plans.

Standout feature: Simplicity. Single API endpoint, well-documented, large community, and wide language support. If you just want something that works without fuss, ScraperAPI delivers.

Limitation: Subscription model means you pay monthly whether you scrape or not. JS rendering at 10x credit cost adds up fast. No structured data extraction from the core API.

3. Bright Data

Best for: Enterprise-scale operations that need the full proxy and data infrastructure

Bright Data (formerly Luminati) is the 800-pound gorilla of the web data industry. They offer everything from raw proxy access to managed scraping APIs to pre-built datasets. Their infrastructure is massive, but so is the complexity.

Pricing: Web Scraper API uses flat-rate pricing of .50-.50 per 1,000 requests. Subscription plans start at \99/month. Pay-as-you-go available but more expensive per request. Their Scraping Browser is priced separately at .50/GB plus \/bin/bash.10/hour.

Anti-bot bypass: Industry-leading. Bright Data has the largest proxy network in the world (72M+ IPs) and their unlocker technology handles virtually any anti-bot system. If a site can be scraped, Bright Data can probably do it.

JS rendering: Available through their Scraping Browser product. Full Chrome-based rendering with session management. Powerful but priced separately from the Scraper API.

Output formats: HTML, JSON, and structured data for supported domains. Their Web Scraper IDE lets you build custom extraction logic visually.

Free tier: Free trial with limited credits. No permanent free tier.

Proxy infrastructure: The largest in the industry — 72M+ residential, datacenter, ISP, and mobile IPs across every country. This is Bright Data's core product and it's unmatched.

Standout feature: Unmatched proxy diversity and success rates on heavily protected sites. If you're scraping at enterprise volume or need guaranteed access to difficult targets, Bright Data has the infrastructure.

Limitation: Pricing is complex and can be unpredictable. The \99/month minimum for subscriptions is steep for smaller operations. Multiple products with separate billing (Scraper API, Scraping Browser, proxy access) can get confusing. Some users report bill shock from unexpected bandwidth charges.

4. Firecrawl

Best for: AI/LLM developers who need clean Markdown output for RAG pipelines

Firecrawl has carved out a strong niche in the AI space. While other APIs focus on raw HTML, Firecrawl is built specifically to turn web pages into LLM-ready Markdown and structured data. If you're building a RAG pipeline or training dataset, Firecrawl speaks your language.

Pricing: Credit-based. Free plan gives 500 credits. Hobby plan at /month, Standard at /month, Growth at /month for 500,000 credits. Most scraping costs 1 credit per page. Their AI-powered /extract endpoint bills by tokens instead of credits.

Anti-bot bypass: Basic anti-bot handling. Firecrawl focuses more on content extraction quality than bypassing heavy protections. For heavily protected sites, you may need to combine it with a proxy service.

JS rendering: Built-in. Most pages are rendered with JavaScript by default. The Growth plan supports up to 100 concurrent browsers.

Output formats: This is where Firecrawl excels. Native Markdown output, structured JSON extraction via LLM, and clean HTML. The /extract endpoint uses AI to pull structured data from any page without writing selectors.

Free tier: 500 credits (pages) for free. Enough to evaluate the API for a small project.

Proxy infrastructure: Basic proxy rotation included. Not their focus area — don't expect Bright Data-level geo-targeting or residential IPs.

Standout feature: First-class Markdown output and AI-powered extraction. If your use case is feeding web data into an LLM, Firecrawl's output quality is hard to beat. It's also open-source (self-hostable).

Limitation: Weaker anti-bot bypass compared to dedicated scraping APIs. Not the right tool if you're scraping protected e-commerce sites or need raw performance at scale. The AI extraction endpoint can get expensive with token-based billing.

5. Apify

Best for: Teams that want pre-built scraping actors for specific websites

Apify is less of a single API and more of a full scraping platform. Their "Actor" marketplace has thousands of pre-built scrapers for specific sites (Amazon, Google, LinkedIn, etc.). You can also build and deploy custom scrapers using their SDK.

Pricing: Pay-as-you-go based on compute units, storage, and proxy usage. Free tier gives \/month in platform credits. Paid plans start at \9/month. Additional costs for proxies (\/bin/bash.60+ per datacenter IP), memory (\/GB), and parallel runs (\ each).

Anti-bot bypass: Varies by Actor. Pre-built Actors for popular sites include anti-bot logic specific to that site. For custom scrapers, you can use Apify's proxy infrastructure, but you're largely responsible for anti-bot handling yourself.

JS rendering: Full Playwright and Puppeteer support. Actors can run headless browsers natively on Apify's cloud infrastructure.

Output formats: Depends on the Actor. Most return JSON. Platform supports exporting to CSV, JSON, XML, and direct integrations with Google Sheets, Slack, Zapier, and databases.

Free tier: \/month in credits on the free plan. Enough for small-scale testing but limited for production use.

Proxy infrastructure: Apify Proxy combines datacenter and residential IPs. Included in all plans but with usage limits. Smart rotation available.

Standout feature: The Actor marketplace. Instead of building a scraper from scratch, you can often find a pre-built, community-maintained Actor for your target site. The platform handles scheduling, storage, and monitoring.

Limitation: Pricing can be confusing with multiple cost dimensions (compute, storage, proxy, memory). Pre-built Actors may break when target sites update. You're dependent on community maintenance for third-party Actors.

6. ZenRows

Best for: Developers focused on bypassing anti-bot systems on protected websites

ZenRows is laser-focused on anti-bot bypass. If your primary challenge is getting past Cloudflare, DataDome, or PerimeterX, ZenRows is designed specifically for that problem.

Pricing: Tiered subscription based on request volume. All plans include the full product suite (Universal Scraper API, Scraping Browser, Residential Proxies). Business 300 plan is roughly $/month. Volume discounts available for quarterly, semi-annual, and annual billing. You only pay for successful requests.

Anti-bot bypass: This is ZenRows' core strength. Their Universal Scraper API includes advanced anti-bot modes for Cloudflare, DataDome, and other major protection systems. High success rates on difficult targets.

JS rendering: Available through their Scraping Browser product. Separate from the basic API requests and uses more of your plan allocation.

Output formats: HTML, with options for CSS/XPath selectors to extract specific elements. AI-powered extraction in beta.

Free tier: Limited free trial. No permanent free tier for ongoing use.

Proxy infrastructure: Built-in residential proxy rotation. All plans include proxy access. Geo-targeting available.

Standout feature: Industry-leading anti-bot bypass rates. If you need reliable access to heavily protected sites, ZenRows consistently ranks among the best. They also only charge for successful requests.

Limitation: Higher entry price than some competitors. Limited output format options — primarily HTML, not optimized for Markdown or structured data like Firecrawl. The UI and documentation could be more polished.

7. Crawlbase

Best for: Budget-conscious teams that need basic scraping with storage

Crawlbase (formerly ProxyCrawl) offers a straightforward scraping API with an interesting twist — built-in data storage. Their pricing is competitive at the lower end, making them a good choice for teams watching their budget.

Pricing: Starts at \9/month. Basic requests from \ per 1,000. They categorize requests into Standard, Moderate, and Complex tiers based on the target site difficulty, each with different pricing. Free trial with initial credits and up to 10,000 stored documents.

Anti-bot bypass: Handles standard protections with proxy rotation and header management. JavaScript rendering available for dynamic sites. Not as strong as ZenRows or Bright Data on heavily protected targets.

JS rendering: Available with their JavaScript rendering mode. Adds to the cost per request.

Output formats: HTML, JSON, and CSV. Built-in data storage lets you accumulate scraped data without building your own storage layer.

Free tier: Free trial credits. 10,000 document storage limit on free accounts.

Proxy infrastructure: Millions of rotating proxies including residential IPs. Geo-targeting available.

Standout feature: Built-in data storage and the affordable entry point. If you need a simple scraping API without enterprise complexity, Crawlbase delivers reasonable value.

Limitation: Limited anti-bot capabilities compared to premium providers. The tiered complexity pricing (Standard/Moderate/Complex) can be unpredictable if your target sites vary widely.

8. Oxylabs

Best for: Enterprise teams that need specialized scraping APIs for e-commerce and SERP data

Oxylabs is another enterprise-grade provider with a strong focus on specific verticals — particularly e-commerce and search engine data. Their specialized APIs are pre-tuned for these use cases.

Pricing: Web Scraper API starts at \9/month for 17,500 results (.80 per 1,000). Specialized SERP and E-Commerce APIs available at similar price points. You only pay for successful scrapes — 5xx and 6xx errors are free.

Anti-bot bypass: Strong anti-bot capabilities backed by a large proxy network. Particularly effective for e-commerce sites and search engines, which are their primary focus areas.

JS rendering: Available through their Headless Browser feature. Included in all API plans but consumes more traffic, increasing effective cost.

Output formats: HTML and JSON. Specialized APIs return pre-structured data for their supported domains (product data, search results, etc.).

Free tier: Free trial available. No permanent free tier.

Proxy infrastructure: 100M+ IPs including residential, datacenter, ISP, and mobile proxies. Strong geo-targeting capabilities. Particularly well-suited for location-specific scraping.

Standout feature: Specialized, pre-built APIs for e-commerce (Amazon, eBay, Walmart) and SERP data. If your primary use case is price monitoring or search ranking tracking, Oxylabs' tailored solutions save development time.

Limitation: Enterprise pricing isn't friendly to small teams. The \9/month minimum with limited results means you're paying a premium per request at lower volumes. General-purpose scraping isn't their strongest suit.

Head-to-Head Comparison

Pricing Comparison

Pricing is where things get tricky because every API uses a different model. Here's a normalized comparison based on what you'd actually pay for common scenarios.

Note that these are baseline costs for standard (non-JS) requests. Costs increase significantly for JavaScript rendering, anti-bot bypass, and CAPTCHA solving across all platforms. AlterLab's advantage narrows on complex requests — Tier 4 (browser) costs \ per thousand, and Tier 5 (CAPTCHA) costs $ per thousand, which is competitive but not dramatically cheaper than alternatives.

Which API for Which Use Case?

Not every API is right for every job. Here's a quick decision framework:

For AI/LLM data pipelines: Firecrawl is purpose-built for this. Clean Markdown output, AI extraction, and self-hosting option. AlterLab is a solid alternative if you need anti-bot bypass that Firecrawl can't handle, since it also supports Markdown output.

For price monitoring and e-commerce: Oxylabs or Bright Data. Their specialized e-commerce APIs return pre-structured product data, saving you from writing extraction logic. ScraperAPI also works well for simpler e-commerce targets.

For heavily protected sites (Cloudflare, DataDome): ZenRows or Bright Data. These two have the strongest anti-bot bypass technology. AlterLab's tiered approach handles most protections well and costs less for mixed-difficulty targets.

For budget-conscious developers: AlterLab's pay-as-you-go model (no subscriptions, $ minimum) or Firecrawl's /month Hobby plan are the most accessible starting points. Crawlbase is another affordable option at \9/month.

For large-scale enterprise operations: Bright Data or Oxylabs. The infrastructure depth, compliance certifications, SLA guarantees, and dedicated account management matter at enterprise scale.

For teams wanting pre-built scrapers: Apify's Actor marketplace saves development time if someone has already built a scraper for your target site. Check the marketplace before building from scratch.

For mixed workloads (easy and hard sites combined): AlterLab's automatic tier escalation shines here. You pay \/bin/bash.0002 for sites that respond to curl and \/bin/bash.004 for sites that need a full browser — without configuring anything. Flat-rate APIs charge you the same price regardless of difficulty.

List the sites you need to scrape and note their anti-bot protections
Calculate monthly request volume to compare pricing models accurately
Use free credits from 2-3 APIs to test success rates on your actual targets
Factor in JS rendering costs, failed request charges, and concurrency limits

Key Takeaways

There is no single "best" web scraping API. The right choice depends on your specific targets, volume, budget, and output format needs. That said, here are some patterns:

If cost predictability matters most, look at APIs that only charge for successful requests (AlterLab, ZenRows, Oxylabs). Getting billed for failed attempts adds up fast on difficult sites.

If you're scraping mixed-difficulty sites, tiered pricing (AlterLab) saves money compared to flat-rate models. Paying browser-rendering prices for a site that responds to curl is wasteful.

If anti-bot bypass is your primary challenge, ZenRows and Bright Data have the deepest anti-bot technology. They cost more, but they work on the hardest targets.

If you're building for AI, Firecrawl's native Markdown and AI extraction features will save you post-processing pipeline development time.

If you want maximum flexibility with minimal commitment, pay-as-you-go models (AlterLab, Apify) let you scale up and down without paying for unused capacity.

The web scraping API market continues to evolve rapidly. Anti-bot systems get harder, APIs get smarter, and pricing models keep innovating. Whatever you choose, start with a free tier, test against your actual target sites, and make your decision based on real success rates — not marketing claims.

Rotating Proxies for Web Scraping: What Works and What Wastes Money

AlterLab — Thu, 26 Mar 2026 03:38:57 +0000

Rotating Proxies for Web Scraping: What Works and What Wastes Money

Proxies are not magic. Slapping a proxy rotation layer onto a bad scraper does not make it good. But when your scraper is solid and you need to scale without getting IP-banned, proxy strategy matters.

Here is what actually works, what costs what, and when proxies are not the answer.

Proxy Types and Their Real Costs

Datacenter Proxies

Cheap ($0.50-2 per IP per month), fast, and the first thing most scrapers try. They come from cloud providers like AWS, GCP, OVH.

The problem: most anti-bot systems maintain lists of datacenter IP ranges. If a site uses Cloudflare, DataDome, or PerimeterX, datacenter proxies get flagged before your request reaches the server.

Good for: Sites without serious bot protection. Internal tools, basic APIs, public government data.

Bad for: E-commerce, social media, any site behind a CDN with bot protection.

Residential Proxies

Real IPs from ISPs, routed through actual home connections. They look like normal users because they are normal user IPs.

Cost: $8-15 per GB of bandwidth. A typical web page with images is 2-5 MB. At $10/GB, that is $0.02-0.05 per page load. It adds up fast.

Good for: Sites with strong bot protection. E-commerce scraping (Amazon, Walmart, Target). Social media data.

Bad for: High-volume scraping where bandwidth costs matter. Downloading large files or media.

ISP Proxies

Static IPs from ISPs hosted in datacenters. They have the reputation of residential IPs with the speed of datacenter ones.

Cost: $2-5 per IP per month. More expensive than datacenter but cheaper per request than residential (since you pay per IP, not per GB).

Good for: When you need consistent IPs (login sessions, account management). Medium-difficulty targets.

Mobile Proxies

IPs from mobile carriers. These have the best reputation because mobile IPs are shared among thousands of users through carrier-grade NAT. Anti-bot systems are reluctant to block them.

Cost: $20-50 per GB. The most expensive option.

Good for: The hardest targets. When everything else gets blocked.

Rotation Strategies

Round-Robin Rotation

Cycle through your proxy pool sequentially. Simple to implement, works fine for sites that do not do session tracking.


proxies = ["proxy1:port", "proxy2:port", "proxy3:port"]
proxy_cycle = itertools.cycle(proxies)

def get_next_proxy():
    return next(proxy_cycle)

The issue: if a proxy gets banned, you keep rotating back to it. You need health checks.

Sticky Sessions

Keep the same IP for a sequence of related requests. Important when scraping paginated results or sites that track sessions.

Most residential proxy providers support this with session IDs:

# With a residential proxy provider
# Same session ID = same IP for ~10 minutes
proxy = "http://user-session_abc123:pass@gate.provider.com:7777"

Smart Rotation with Backoff

The approach that works best in practice. Track which proxies are healthy, back off when one gets flagged, and prioritize proxies with recent success.

from collections import defaultdict

class ProxyPool:
    def __init__(self, proxies):
        self.proxies = proxies
        self.failures = defaultdict(int)
        self.last_fail = defaultdict(float)

    def get_proxy(self):
        now = time.time()
        available = [
            p for p in self.proxies
            if now - self.last_fail[p] > self.failures[p] * 60
        ]
        if not available:
            available = self.proxies  # reset if all are in backoff
        return random.choice(available)

    def report_failure(self, proxy):
        self.failures[proxy] += 1
        self.last_fail[proxy] = time.time()

    def report_success(self, proxy):
        self.failures[proxy] = 0

When to Skip Proxies Entirely

Proxies solve one problem: IP-based blocking. But many scraping failures are not about IPs at all.

If you are getting CAPTCHAs, the problem is usually fingerprinting, not your IP address. Adding more proxies to a detectable scraper just burns through IPs faster.

If the site requires JavaScript rendering, you need a browser, not a proxy. A proxy on top of raw HTTP requests does not help when the page content is loaded via client-side JS.

If you are scraping fewer than 100 pages per day from a single site, you probably do not need proxy rotation at all. Most sites allow moderate request rates from a single IP.

The Build vs Buy Decision

Building proxy rotation infrastructure means:

Buying proxy bandwidth or pools
Writing rotation logic with health checks
Monitoring success rates and costs
Handling retries, rate limits, and bans
Maintaining this over time as anti-bot systems change

For most teams, the proxy layer is a distraction from the actual work. You are building a scraping tool, not a proxy management platform.

Scraping APIs like AlterLab, ScraperAPI, and Bright Data bundle proxies into the service. You pay per successful request. If the request fails because of a proxy issue, you do not pay for it. The provider eats that cost and rotates to another proxy.

AlterLab takes this further with a "bring your own proxy" option. If you already have proxy infrastructure you like, you can route AlterLab requests through your own proxies. You get the anti-bot bypass and JS rendering without paying for proxy bandwidth twice.

Summary

Match your proxy type to your target difficulty:

Target Difficulty	Proxy Type	Cost per Request
No bot protection	Datacenter or none	< $0.001
Basic protection	ISP proxies	$0.001-0.005
Cloudflare/DataDome	Residential	$0.02-0.05
Hardest targets	Mobile	$0.05-0.15

If proxy costs are eating your budget, you are either using the wrong proxy type for your target or scraping at a scale where an API service would be cheaper.

Web Scraping APIs vs DIY Scrapers: When to Stop Building Infrastructure

AlterLab — Thu, 26 Mar 2026 03:38:51 +0000

Web Scraping APIs vs DIY Scrapers: When to Stop Building Infrastructure

Every developer starts scraping the same way. Write a Python script, send some requests, parse the HTML. It works. Then you need to scrape a site with bot protection and suddenly you are shopping for proxies, patching headless browsers, and debugging TLS fingerprints at 2 AM.

There is a point where building your own scraping infrastructure stops being productive and starts being a second job. The question is where that line is for your specific use case.

What You Build When You DIY

A production scraping stack is not just a script. Here is the full inventory:

Request layer. HTTP client with proper TLS fingerprinting, header management, cookie handling, redirect following.

Proxy layer. Pool management, rotation logic, health checks, cost tracking, failover between proxy types.

Browser layer. Headless Chrome/Playwright instances, memory management, crash recovery, stealth patches, session isolation.

Anti-bot layer. CAPTCHA solving integration, challenge detection, fingerprint maintenance as anti-bot systems update.

Queue and scheduling. Rate limiting per domain, retry logic with backoff, deduplication, priority queues.

Monitoring. Success rates per domain, cost per request, error tracking, alerting when a target changes its structure.

Parsing. HTML extraction, JSON-LD parsing, schema validation. This part is usually the easy part.

Each of these is a maintenance surface. Anti-bot systems update monthly. Proxies get burned and need replacement. Browser versions change and stealth patches break.

What You Get With a Scraping API

You send a URL. You get back HTML, markdown, or structured data. The API provider handles everything listed above.

The trade-off is control vs convenience. With a DIY stack, you can tune every parameter. With an API, you trade that control for not having to maintain anything.

Cost Comparison

Here is a realistic comparison for scraping 100K pages per month from a mix of easy and hard targets.

DIY Stack Costs

Component	Monthly Cost
Residential proxies (500 GB)	$4,000-5,000
Server (browser instances)	$200-400
CAPTCHA solving service	$100-300
Your engineering time (10-20 hrs)	$1,000-4,000
Total	$5,300-9,700

That engineering time estimate is conservative. When something breaks at scale, debugging takes hours.

API Service Costs

Most scraping APIs charge per successful request, with pricing tiers based on difficulty.

Service	Easy pages	JS rendered	Anti-bot bypass
AlterLab	$0.001	$0.005	$0.01-0.05
ScraperAPI	$0.001	$0.005	$0.01-0.10
ScrapingBee	$0.001	$0.005	$0.01-0.05
Bright Data (SERP API)	$0.003	$0.01	$0.02-0.08

For 100K pages (50% easy, 30% JS rendered, 20% hard):

Easy: 50,000 x $0.001 = $50
JS rendered: 30,000 x $0.005 = $150
Anti-bot: 20,000 x $0.02 = $400
Total: roughly $600/month

That is 10x cheaper than DIY for most teams. Even if you value your engineering time at zero.

When DIY Makes Sense

DIY scraping is the right call when:

You scrape one or two simple sites. If your targets do not have bot protection and the structure rarely changes, a simple requests + BeautifulSoup script is fine. No need to overcomplicate it.

You need sub-second latency. Scraping APIs add network overhead. If you need to scrape and respond in real-time (like a price comparison tool), running your own infrastructure close to the target servers matters.

Scraping is your core product. If you are building a scraping company, you should own the infrastructure. You need that level of control.

You have an existing proxy investment. If you already have residential proxy contracts, building on top of that makes sense. Some services like AlterLab let you bring your own proxies so you can use their anti-bot bypass without paying for proxy bandwidth twice.

When to Use an API

API services make sense when:

Scraping is a means, not the end. You are building an AI training pipeline, a price monitoring tool, a lead gen system. The scraping is a component, not the product.

You scrape diverse targets. Each site has different bot protection, rendering requirements, and anti-scraping measures. APIs handle the diversity so you do not have to build and maintain solutions for each one.

Your time is worth more than the API cost. If you spend 20 hours per month maintaining scraping infrastructure, and the API costs $500 less than your hourly rate, the math is clear.

You need to scale quickly. Going from 10K to 1M pages means 10x more proxies, 10x more browser instances, 10x more monitoring. An API scales without any infrastructure changes on your end.

The Hybrid Approach

The smart move for most teams is starting with an API and only building custom infrastructure for specific cases where you need it.

Use an API for the 80% of targets that are standard. Build custom scrapers for the few targets where you need precise control, unusual interaction patterns, or real-time response.

AlterLab is built for this pattern. Pay for what you use, no subscriptions, no minimum commitments. Light scrapes are cheap, JS rendering costs more, and anti-bot bypass scales with difficulty. If a request fails, you do not pay for it.

The bottom line: unless scraping is your core business, the infrastructure is a distraction. Ship your product, not your proxy management dashboard.

Web Scraping Pipeline for RAG: Clean Data for LLMs

AlterLab — Wed, 25 Mar 2026 07:48:31 +0000

Web Scraping Pipeline for RAG: Feed Clean Data into Your LLM Without Token Waste

Raw HTML is poison for RAG. A typical news article page is 45,000 characters—roughly 11,000 tokens. The actual article is 800 words, or about 1,100 tokens. You are paying 10× to embed navigation menus, cookie banners, footer links, and inline scripts that actively dilute your embeddings and degrade retrieval quality.

The fix is a five-stage pipeline: reliable fetch → content extraction → normalization → semantic chunking → embed and index. Each stage has a single responsibility. Each failure is isolated and debuggable. This post walks through a production implementation in Python.

Pipeline Architecture

Stage 1: Reliable Fetching

The hardest part of scraping at scale is not parsing—it is getting the HTML. Bot detection blocks requests. JavaScript-rendered SPAs return skeleton HTML to static fetches. IP ranges accumulate blocks.

AlterLab's scraping API handles this in a single POST: rotating residential proxies, automatic CAPTCHA bypass, and optional headless rendering without managing a browser fleet yourself.

Python:

```python title="fetch.py" {8-19}

ALTERLAB_API_KEY = "YOUR_API_KEY"
ALTERLAB_BASE_URL = "https://api.alterlab.io/v1"

def fetch_page(url: str, render_js: bool = False) -> str:
"""Fetch fully-rendered HTML from any URL."""
response = httpx.post(
f"{ALTERLAB_BASE_URL}/scrape",
headers={"X-API-Key": ALTERLAB_API_KEY, "Content-Type": "application/json"},
json={
"url": url,
"render_js": render_js,
"wait_for": "networkidle" if render_js else None,
},
timeout=30,
)
response.raise_for_status()
return response.json()["html"]




**cURL:**



```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs/getting-started",
    "render_js": true,
    "wait_for": "networkidle"
  }'

--- ## Stage 2: Content Extraction `trafilatura` is the most accurate open-source library for pulling article body text from HTML. It outperforms `readability-lxml` and `newspaper3k` on structured documentation and blog content because it uses both DOM heuristics and text-density scoring. ```python title="extract.py" {1-3,14-25} from trafilatura.settings import use_config # Disable per-document timeout—let your own retry logic own the clock config = use_config() config.set("DEFAULT", "EXTRACTION_TIMEOUT", "0") def extract_content(html: str, url: str) -> dict: """ Extract main content from HTML. Returns dict with keys: text, title, author, date, description. Raises ValueError if no content can be extracted. """ result = trafilatura.extract( html, url=url, include_comments=False, include_tables=True, no_fallback=False, config=config, output_format="json", with_metadata=True, ) if result is None: raise ValueError(f"Extraction returned no content for {url}") return json.loads(result) ``` Set `no_fallback=False` to allow trafilatura to fall back to its secondary heuristic if the primary DOM analysis returns nothing—useful for pages with unconventional layouts. --- ## Stage 3: Normalization After extraction, text still contains artifacts: Unicode non-breaking spaces (`\u00a0`), zero-width joiners, smart quotes, triple-newline runs from CMS templates, and stub lines that are purely punctuation. ```python title="normalize.py" {5-18} def normalize_text(text: str) -> str: # Canonical Unicode form: convert smart quotes, em-dashes, ligatures text = unicodedata.normalize("NFKC", text) # Replace invisible/non-breaking whitespace variants text = re.sub(r"[\u00a0\u200b\u200c\u200d\ufeff]", " ", text) # Collapse horizontal whitespace, preserve single newlines text = re.sub(r"[ \t]+", " ", text) text = re.sub(r"\n{3,}", "\n\n", text) # Drop lines shorter than 4 chars (nav artifacts: "›", "|", "»") lines = [ln for ln in text.split("\n") if len(ln.strip()) > 3 or ln.strip() == ""] return "\n".join(lines).strip() ``` This pass runs in microseconds per document and prevents garbage tokens from reaching your embedding model. --- ## Stage 4: Chunking Strategy Three mistakes that kill retrieval quality: - **Fixed character splits** break sentences mid-clause. The embedding for a sentence fragment does not represent a complete thought. - **Whole documents as single vectors** average all content into one point in embedding space. Specific queries retrieve nothing useful. - **Zero overlap** means a concept bridging two chunks never matches a query that references it as a unit. Use recursive sentence-aware chunking with configurable overlap: ```python title="chunker.py" {19-48} from __future__ import annotations from dataclasses import dataclass, field @dataclass class Chunk: text: str url: str chunk_index: int total_chunks: int metadata: dict = field(default_factory=dict) def split_sentences(text: str) -> list[str]: """Sentence-boundary split on terminal punctuation followed by uppercase.""" return re.split(r"(?<=[.!?])\s+(?=[A-Z\"])", text) def chunk_document( text: str, url: str, max_tokens: int = 400, overlap_sentences: int = 2, chars_per_token: float = 4.0, ) -> list[Chunk]: """ Split text into token-bounded chunks with sentence-level overlap. Args: max_tokens: Approximate token ceiling per chunk. overlap_sentences: Sentences carried over to the next chunk. chars_per_token: Heuristic for English prose (4.0 is reliable). """ max_chars = int(max_tokens * chars_per_token) sentences = split_sentences(text) raw_chunks: list[str] = [] current: list[str] = [] current_len = 0 for sentence in sentences: slen = len(sentence) if current_len + slen > max_chars and current: raw_chunks.append(" ".join(current)) current = current[-overlap_sentences:] current_len = sum(len(s) for s in current) current.append(sentence) current_len += slen if current: raw_chunks.append(" ".join(current)) total = len(raw_chunks) return [ Chunk(text=t, url=url, chunk_index=i, total_chunks=total) for i, t in enumerate(raw_chunks) ] ``` **Token ceiling guidelines by model:**

Model	Recommended max_tokens	Overlap Sentences	Notes
text-embedding-3-small	400	2	Good default for mixed content
text-embedding-3-large	600	2	Better for long-form technical docs
nomic-embed-text	512	3	Open-source; strong on code + prose
BGE-M3	800	2	Multilingual; 8192-token context

Stage 5: Embedding and Indexing

Batch your embedding calls. The OpenAI embeddings API accepts up to 2,048 inputs per request—sending one chunk per call is 100× slower and burns rate limit quota unnecessarily.

```python title="embed.py" {27-48}

from openai import AsyncOpenAI

openai_client = AsyncOpenAI()

def init_index(api_key: str, index_name: str) -> pinecone.Index:
pc = pinecone.Pinecone(api_key=api_key)
return pc.Index(index_name)

async def embed_texts(texts: list[str]) -> list[list[float]]:
"""Batch embed up to 2048 texts in a single API call."""
response = await openai_client.embeddings.create(
model="text-embedding-3-small",
input=texts,
encoding_format="float",
)
return [item.embedding for item in response.data]

async def index_chunks(
chunks: list["Chunk"],
index: pinecone.Index,
batch_size: int = 100,
) -> None:
"""Embed and upsert chunks into Pinecone with source metadata preserved."""
for i in range(0, len(chunks), batch_size):
batch = chunks[i : i + batch_size]
vectors = await embed_texts([c.text for c in batch])

    upserts = [
        {
            "id": f"{c.url}::{c.chunk_index}",
            "values": vectors[j],
            "metadata": {
                "url": c.url,
                "chunk_index": c.chunk_index,
                "total_chunks": c.total_chunks,
                "text": c.text,  # store inline—avoids a separate fetch at query time
            },
        }
        for j, c in enumerate(batch)
    ]

    index.upsert(vectors=upserts)




Store `text` in the vector metadata. Fetching the source document at query time adds latency and a failure point; paying a few extra bytes per vector is worth it.

---

## Full Pipeline



```python title="pipeline.py" {11-55}

from fetch import fetch_page
from extract import extract_content
from normalize import normalize_text
from chunker import Chunk, chunk_document
from embed import init_index, index_chunks

PINECONE_API_KEY = "YOUR_PINECONE_KEY"
PINECONE_INDEX = "rag-knowledge-base"

async def ingest_url(url: str, render_js: bool = False) -> dict:
    """
    End-to-end pipeline: URL → indexed, retrievable chunks.
    Returns a summary dict for logging and monitoring.
    """
    # Stage 1: Fetch
    html = fetch_page(url, render_js=render_js)

    # Stage 2: Extract
    extracted = extract_content(html, url)
    raw_text = extracted.get("text", "")
    title = extracted.get("title", "untitled")

    if not raw_text:
        return {"url": url, "status": "no_content", "chunks": 0}

    # Stage 3: Normalize
    clean_text = normalize_text(raw_text)

    # Stage 4: Chunk
    chunks = chunk_document(
        text=clean_text,
        url=url,
        max_tokens=400,
        overlap_sentences=2,
    )

    # Filter degenerate chunks before embedding
    chunks = [c for c in chunks if len(c.text.split()) >= 15]

    # Stage 5: Embed + index
    index = init_index(PINECONE_API_KEY, PINECONE_INDEX)
    await index_chunks(chunks, index)

    return {
        "url": url,
        "title": title,
        "status": "indexed",
        "chunks": len(chunks),
        "approx_tokens": sum(len(c.text) // 4 for c in chunks),
    }

async def ingest_batch(urls: list[str], concurrency: int = 5) -> list[dict]:
    """Ingest multiple URLs with bounded concurrency."""
    semaphore = asyncio.Semaphore(concurrency)

    async def bounded(url: str) -> dict:
        async with semaphore:
            try:
                return await ingest_url(url)
            except Exception as e:
                return {"url": url, "status": "error", "error": str(e)}

    return await asyncio.gather(*[bounded(u) for u in urls])

if __name__ == "__main__":
    urls = [
        "https://docs.python.org/3/library/asyncio-task.html",
        "https://platform.openai.com/docs/guides/embeddings",
        "https://www.pinecone.io/docs/upsert-data/",
    ]
    results = asyncio.run(ingest_batch(urls, concurrency=3))
    for r in results:
        print(r)

Handling Edge Cases

Deduplication

The same content appears under multiple URLs: www vs. bare domain, query parameters, pagination suffixes. Hash normalized text before indexing:

```python title="dedup.py" {6-13}

_seen_hashes: set[str] = set()

def is_duplicate(text: str) -> bool:
"""Return True if this exact content has already been indexed this run."""
digest = hashlib.sha256(text.encode()).hexdigest()[:16]
if digest in _seen_hashes:
return True
_seen_hashes.add(digest)
return False




Call `is_duplicate(clean_text)` after Stage 3 and skip to the next URL if it returns `True`.

### Pagination and Crawling

For documentation sites spanning dozens of pages, discover internal links before ingesting. A simple same-domain BFS over `<a href>` tags prevents you from missing chapters or API reference sections. Keep a visited-URL set to avoid cycles.

### Retries with Backoff

Your embedding API has rate limits even when your scraper does not. Wrap async calls in exponential backoff:



```python title="retry.py" {6-16}

from typing import TypeVar, Callable, Awaitable

T = TypeVar("T")

async def with_retry(fn: Callable[[], Awaitable[T]], attempts: int = 3) -> T:
    for i in range(attempts):
        try:
            return await fn()
        except Exception:
            if i == attempts - 1:
                raise
            await asyncio.sleep(min(2 ** i, 30))
    raise RuntimeError("unreachable")

Wrap index_chunks calls: await with_retry(lambda: index_chunks(chunks, index)).

Production Checklist

Before running this at scale, verify:

Freshness TTL: Set an expiry on indexed documents. Re-scrape on a schedule. Stale RAG context is worse than no context—your LLM will confidently cite outdated information.
Minimum chunk length: Filter out chunks with fewer than 15 words. Stubs from tables or code snippets without context are noise at query time.
Metadata completeness: Always store scraped_at, source_url, and section_title in vector metadata. Your LLM needs these to generate citations users can verify.
Extraction failure rate: Monitor the share of URLs returning no_content. Above 5% means your source sites have unusual structure and need custom extraction rules.
Concurrency limits: Do not set concurrency above what your scraping tier supports. Queue excess work with Redis or a task runner rather than hammering with retries.

Takeaway

A five-stage pipeline—fetch, extract, normalize, chunk, embed—is not over-engineering. It is the minimum required to produce input that a retrieval system can actually use.

Token waste is a symptom, not the root problem. The root problem is that HTML is a rendering format, not a content format. Every step in this pipeline exists to close that gap.

The fetching layer is where most teams cut corners and regret it. Flaky HTML from failed bot-bypass attempts or unrendered JS propagates bad data through every downstream stage. Eliminating that variable with a dedicated scraping API means your engineering time goes where it compounds: extraction heuristics, chunking strategy, and retrieval evaluation.

DEV Community: AlterLab

Extract Structured Data from Websites Using AI Instead of CSS Selectors

The Problem with CSS Selectors

How AI Extraction Works

Setting Up

Structured Schemas with JSON Schema

Error Handling

Automate Web Scraping in n8n with AlterLab API

Automate Web Scraping in n8n with AlterLab's API

Prerequisites

Step 1: Configure the HTTP Request Node

Access the HTTP Request output

Extract product data

Transform to your schema

Step 5: Add Error Handling and Retries

Retry Configuration

Error Routing

Capture failed URLs for retry

Cost Considerations

Complete Workflow Example

Feed Clean Web Data to RAG Pipelines Without Wasting LLM Tokens

How to Feed Clean Web Data to RAG Pipelines Without Wasting 90% of Your LLM Tokens

The Token Math Behind Dirty Web Data

Step 1: Get Clean Content at the Source

Step 2: Structure Data for Retrieval, Not Display

Step 4: Build the Ingestion Pipeline

Step 5: Handle Anti-Bot Pages Without Infrastructure

Build a Web Scraping Pipeline with n8n and AlterLab

Prerequisites

Step 1: Store the API Key in n8n Credentials

Step 2: Configure the HTTP Request Node

Step 3: Parse HTML in the Code Node

Step 4: Scrape Multiple Pages

Step 5: Route Data to Storage

Monitoring Pipeline Health

How to Scrape Glassdoor: Complete Guide for 2026

How to Scrape Glassdoor: Complete Guide for 2026

Why Scrape Glassdoor?

Anti-Bot Challenges on glassdoor.com

Quick Start with AlterLab API

Company Reviews

Common Pitfalls

Scaling Up

Batch Requests

Cost Management at Scale

Key Takeaways

Related Guides

Web Scraping API Pricing Compared: Cut Costs 90%

The Real Cost of Web Scraping at Scale

How Scraping API Pricing Actually Works

Pricing Model Comparison

Implementing Tiered Scraping in Production

Example: Scrape 100 mixed-complexity sites

Node.js Implementation for High-Volume Pipelines

Common Cost Optimization Mistakes

Takeaway

How to Scrape LinkedIn Profiles and Company Data Without Getting Blocked in 2026

How to Scrape LinkedIn Profiles and Company Data Without Getting Blocked in 2026

What You're Up Against

What Data is Realistically Scrapable

Approach 1: Scraping Public Company Pages

Approach 2: Full Profile Scraping with Playwright

Proxy Strategy

Rate Limiting and Request Patterns

Handling Structure Changes

When Raw Scraping Isn't Worth It

Legal and Ethical Considerations

Key Takeaways

Scraping JavaScript-Heavy SPAs with Python: Dynamic Content, Infinite Scroll, and API Interception

Scraping JavaScript-Heavy SPAs with Python: Dynamic Content, Infinite Scroll, and API Interception

Why requests Fails Against SPAs

Choose Your Approach Before Writing Code

Approach 1: Headless Browser with Playwright

Waiting for the Right Moment

evaluate() vs. Locators

Approach 2: API Interception

Extracting Client-Side Auth Tokens

Approach 3: Infinite Scroll Automation

Virtual Scrolling Is a Different Problem

Anti-Bot Considerations

Why `requests` Fails Against SPAs

`evaluate()` vs. Locators