DEV Community: AlterLab

Engineering Reliability: Telemetry, Migrations, and Security

AlterLab — Tue, 28 Jul 2026 08:01:13 +0000

TL;DR

We have implemented per-vendor success-rate telemetry to provide granular visibility into proxy tier performance. We also introduced migration-ledger drift detection to ensure database integrity and hardened our URL validation logic to prevent scheme-prefix attacks.

Infrastructure Observability: Per-Vendor Telemetry

Reliability in web scraping is often a moving target. As anti-bot measures evolve, the effectiveness of specific proxy providers or browser-emulation tiers changes. To manage this, we implemented a new telemetry layer to track success rates at a granular level.

Previously, our diagnostics could tell us if a scrape failed, but they couldn't easily correlate that failure to a specific vendor at a specific tier. We have introduced --vendor-tier-reliability to our infrastructure scripts. This provides a read-only aggregation over scrape_diagnostics.tier_attempts[].antibot.

This allows us to answer a critical question: "For vendor X, what is the success rate at tier N?"

This telemetry is vital for optimizing our anti-bot handling. By understanding which vendor/tier combinations are failing on specific e-commerce or social media sites, we can automate better routing decisions.

Database Integrity: Migration Reconciliation

Data integrity is the foundation of any scalable API. During a recent audit, we identified a gap in our migration verification process.

Our previous checks were one-directional. We could check if files existed for migrations that hadn't been applied, but we couldn't effectively check if the schema_migrations table contained entries for files that didn't exist in the codebase (a common symptom of failed or rolled-back migrations).

To solve this, we implemented migration-ledger drift detection. This includes:

Content Hash Verification: Ensuring the code in the migration file matches what was actually executed.
Reconciliation: Comparing the recorded rows in the database against the physical files in the repository to detect "ghost" migrations.

This prevents the "failed migration rollback" class of errors, where a rollback might target the wrong schema version, potentially causing catastrophic data loss in production.

Disaster Recovery Hardening

Beyond schema integrity, we updated our Disaster Recovery (DR) scripts to prevent accidental production data loss. We identified a "Time-of-Check to Time-of-Use" (TOCTOU) vulnerability where our production-target confirmation gate was only checking the POSTGRES_DB environment variable.

If the POSTGRES_CONTAINER resolved to a production container name (e.g., alterlab-postgres), the script might proceed with a destructive drop+recreate operation even if the database name didn't match. We have patched this to ensure that both the database name and the container identity are verified before any destructive operations occur.

Security Hardening: URL Scheme Validation

Security is a continuous process of closing small gaps. We recently addressed a vulnerability in our announcement and public CTA models.

The issue involved how we validated URLs. Our validate_href_scheme function used a simple startswith check for http:// and https://. While this seems straightforward, it was susceptible to "userinfo-style construction" attacks.

A malicious actor could provide a URL like:
https://good.tld@evil.tld

Because the string starts with https://, the old validation logic passed it. However, a browser would treat evil.tld as the actual host, effectively redirecting the user away from the intended destination.

We have updated our core href validation to properly parse the URL and validate the host, ensuring that the entire URI structure is legitimate.

```bash title="Terminal"

Example of how to verify your API implementation

curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-d '{"url": "https://example.com", "formats": ["json"]}'




For developers building high-scale pipelines, these backend improvements mean more predictable performance. You can integrate our [Python SDK](https://alterlab.io/web-scraping-api-python) with confidence, knowing that our underlying infrastructure is continuously being audited for both reliability and security.



```python title="scraper.py" {1-4}

# The client handles complex routing and anti-bot logic automatically
client = alterlab.Client("YOUR_API_KEY")
response = client.scrape("https://example.com")
print(response.json())

Summary of Updates

Feature	Problem	Solution
Vendor Telemetry	Lack of tier-specific visibility	Per-vendor/tier success-rate aggregation
Migration Safety	One-way drift detection	Full reconciliation + content hash verification
DR Scripts	TOCTOU vulnerability	Dual-key confirmation (DB + Container)
URL Validation	Scheme-prefix bypass	Strict host-aware URL parsing

If you are building data pipelines that require high uptime, we recommend reviewing our API documentation to learn more about our advanced features like scheduling and webhooks.

Hit reply if you have questions.

AlterLab // Web Data, Simplified.

Scalable Web Scraping Architecture for AI Agents

AlterLab — Fri, 24 Jul 2026 03:58:35 +0000

TL;DR

A scalable web scraping architecture for AI agents combines intelligent proxy rotation, managed headless browsers, and structured data extraction pipelines. This approach ensures reliable access to public web data while transforming raw HTML into AI-ready formats.

Introduction

AI agents need fresh, structured data from the web to power retrieval-augmented generation (RAG), monitoring, and automated decision-making. Building a scraping system that scales requires handling anti-bot measures, JavaScript rendering, and data transformation at scale. This guide outlines proven strategies for each layer, with practical examples using AlterLab's API.

Core Challenges

Scraping at scale for AI agents introduces three primary challenges:

IP blocking and rate limits: Target sites restrict repeated requests from the same address.
JavaScript-dependent content: Modern sites load critical data via client-side scripts.
Data variability: Raw HTML differs across sites, requiring adaptive extraction for consistent AI inputs.

Solving these requires a layered architecture where each concern is isolated and independently scalable.

Proxy Rotation Strategies

Effective proxy rotation prevents IP-based blocking by distributing requests across a diverse pool. Key tactics include:

Geographic distribution: Use proxies matching the target audience's region to reduce suspicion.
Session persistence: Maintain the same proxy for multi-step interactions (e.g., pagination) to avoid session breaks.
Failure detection: Automatically retire proxies that return CAPTCHAs or error codes.

AlterLab handles proxy rotation automatically, but if building a custom solution, implement a proxy manager that scores providers by success rate and latency.

Headless Browser Management

Headless browsers render JavaScript content but consume significant resources. Optimize with:

Browser pooling: Reuse browser instances to avoid startup overhead.
Resource blocking: Disable images, fonts, and unnecessary scripts to speed up rendering.
Context isolation: Use separate browser contexts per session to prevent state leakage.

For AI agents, prioritize speed and reliability over full browser fidelity. Many sites only need basic DOM interaction after initial JS load.

Structured Data Extraction for AI Agents

Raw HTML must become structured data before AI consumption. Strategies include:

Schema-based extraction: Define expected fields (price, availability, title) and use XPath/CSS selectors or AI models to locate them.
Output format selection: JSON for machine consumption, Markdown for LLM prompting, or plain text for simple tasks.
Validation and cleaning: Strip HTML tags, normalize whitespace, and validate data types.

AlterLab's formats parameter lets you specify the output format directly in the request, eliminating post-processing steps.

Putting It All Together: Architecture Diagram

A scalable system separates concerns into distinct services:

Ingestion API: Accepts scrape jobs from AI agents and enqueues them.
Worker Pool: Executes scraping tasks using proxy rotation and headless browsers.
Storage Layer: Saves raw responses temporarily for retry logic.
Transformation Service: Converts HTML to the requested format (JSON/Markdown).
Delivery Endpoint: Pushes results to agents via webhook or polling queue.

This design allows horizontal scaling of workers and independent updates to extraction logic.

Code Examples

Below are equivalent examples using AlterLab's Python SDK and raw cURL to scrape a product listing page and receive JSON output.

```python title="scraper.py" {3-5}

client = alterlab.Client("YOUR_API_KEY") # Initialize client
response = client.scrape( # Perform scrape with JSON output
"https://example.com/products",
formats=["json"] # Request structured JSON output
)
print(response.json()) # Output structured data for AI agent






```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "formats": ["json"]
  }'

Both examples request JSON output, ensuring the AI agent receives immediately usable data without additional parsing.

Internal Links and Further Reading

For implementation details, see the Python SDK and review the API documentation for advanced parameters like min_tier and webhook_url. The anti-bot handling page explains how AlterLab manages JavaScript challenges and proxy rotation automatically.

Conclusion

Building a scalable scraping architecture for AI agents requires separating concerns: manage proxies to maintain access, use headless browsers judiciously for JS rendering, and extract structured data that fits directly into AI pipelines. By leveraging a purpose-built API like AlterLab, engineering teams can focus on agent logic rather than infrastructure undifferentiated heavy lifting. Start with a small batch of test URLs, monitor success rates, and scale the worker pool as demand grows.

Takeaway: Design your scraping system as a pipeline with distinct, scalable stages for networking, rendering, and transformation. Use structured output formats to minimize post-processing and keep AI agents fed with clean, reliable data.

IMDB Data API: Extract Structured JSON in 2026

AlterLab — Fri, 24 Jul 2026 00:13:37 +0000

IMDB Data API: Extract Structured JSON in 2026

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

To get structured IMDB data via API, use AlterLab's Extract API with a JSON schema defining your target fields (title, rating, genre, release_year, director). Send a POST request to /v1/extract with the IMDB URL and schema to receive validated, typed JSON — eliminating HTML parsing and anti-bot challenges. This approach delivers clean data ready for immediate use in pipelines.

Why use IMDB data?

IMDB provides rich, publicly available entertainment datasets valuable for technical applications. Movie titles, ratings, and genres serve as excellent training data for recommendation system ML models. Analytics teams extract release year and director information to build box office trend dashboards. Competitive intelligence platforms monitor genre popularity shifts across streaming services to inform content acquisition strategies — all using publicly listed information without accessing private user data.

What data can you extract?

From IMDB's publicly accessible pages, you can reliably extract these entertainment fields:

title: String (e.g., "Parasite")
rating: String (e.g., "8.6")
genre: String (e.g., "Thriller, Drama, Comedy")
release_year: String (e.g., "2019")
director: String (e.g., "Bong Joon-ho")

AlterLab's Extract API returns these as typed JSON objects matching your defined schema. Only extract data visible without login or payment — never attempt to bypass access controls for private information.

The extraction approach

Direct HTTP requests followed by HTML parsing create brittle pipelines. IMDB frequently updates its frontend markup, requiring constant selector maintenance. JavaScript-rendered content complicates raw HTTP approaches, while anti-bot measures trigger CAPTCHAs and IP blocks.

A data API solves these infrastructure problems. AlterLab handles proxy rotation, automatic retries, and AI-powered understanding of page structure. You define what data you need via JSON schema — not how to parse it. The service returns validated output, letting your team focus on data utilization rather than extraction maintenance.

Quick start with AlterLab Extract API

Begin by installing the AlterLab client (Getting started guide). Here's a Python example extracting structured data from an IMDB title page:

```python title="extract_imdb-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "The movie title as displayed on IMDB"
},
"rating": {
"type": "string",
"description": "User rating value (e.g., '9.2')"
},
"genre": {
"type": "string",
"description": "Comma-separated genre list from page"
},
"release_year": {
"type": "string",
"description": "Original release year as four-digit string"
},
"director": {
"type": "string",
"description": "Primary director name"
}
}
}

result = client.extract(
url="https://www.imdb.com/title/tt0111161/",
schema=schema,
)
print(result.data)




**Output example:**


```json
{
  "title": "The Shawshank Redemption",
  "rating": "9.3",
  "genre": "Drama",
  "release_year": "1994",
  "director": "Frank Darabont"
}

The equivalent cURL request demonstrates language-agnostic accessibility:

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.imdb.com/title/tt0111161/",
"schema": {
"properties": {
"title": {"type": "string"},
"rating": {"type": "string"},
"genre": {"type": "string"},
"release_year": {"type": "string"},
"director": {"type": "string"}
}
}
}'




For asynchronous processing of multiple URLs (e.g., scraping search results), use the batch endpoint:



```python title="async_imdb_batch.py" {8-15}

client = alterlab.Client("YOUR_API_KEY")

schema = {
  "type": "object",
  "properties": {
    "title": {"type": "string"},
    "rating": {"type": "string"},
    "year": {"type": "string"}
  }
}

urls = [
  "https://www.imdb.com/chart/top/",
  "https://www.imdb.com/search/title/?genres=drama",
  "https://www.imdb.com/search/title/?release_date=2020-01-01,2020-12-31"
]

async def extract_batch():
    jobs = []
    for url in urls:
        job = await client.extract_async(
            url=url,
            schema=schema,
            webhook_url="https://yourdomain.com/webhook"
        )
        jobs.append(job.id)
    return results = await client.get_batch_results(jobs)

asyncio.run(extract_batch())

Define your schema

The JSON schema parameter is where you specify exactly what structured data you need. AlterLab validates all output against this schema, ensuring:

Type correctness (strings remain strings, numbers don't appear in string fields)
Presence of required properties
conformity to your defined descriptions

This eliminates guesswork and post-processing. For IMDB, note that some fields like "rating" appear as strings on the page (including potential non-numeric values like "Not Rated") — keeping them as strings in your schema prevents validation errors. The service handles AI interpretation of visual page elements to populate these fields accurately.

Handle pagination and scale

For extracting data across multiple IMDB pages (e.g., top 250 lists or search results), implement pagination in your workflow. AlterLab manages rate limits internally through intelligent request spacing and retry logic. For high-volume operations:

Use the asynchronous extract endpoint shown above to non-blockingly process hundreds of URLs
Configure webhooks to receive results without polling
Monitor usage via your dashboard to optimize costs

See AlterLab pricing for details on pay-as-you-go scaling — charges occur only for successful extractions with no minimums or expiration. Typical IMDB extraction costs fractions of a cent per request at scale.

Key takeaways

Structured data APIs like AlterLab's eliminate HTML parsing fragility for IMDB data extraction
Define your output format upfront with JSON schema for type-safe, pipeline-ready data
Focus on publicly available information: titles, ratings, genres, release years, and directors
Let the API handle infrastructure complexities (proxies, rendering, anti-bot) while you concentrate on data value
Always verify compliance with IMDB's robots.txt and Terms of Service before beginning extraction

This approach transforms IMDB from a brittle HTML source into a reliable structured data feed for your entertainment analytics, ML training, or content intelligence applications — delivering JSON that's immediately consumable by downstream systems.

Hit reply if you have questions.

AutoTrader Data API: Extract Structured JSON in 2026

AlterLab — Fri, 24 Jul 2026 00:13:36 +0000

Disclaimer: This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

To get structured AutoTrader data via API, use the AlterLab Extract API to send a target URL and a JSON schema. The API handles proxy rotation and AI-powered extraction to return typed JSON (make, model, year, price, etc.) without requiring custom HTML parsers or CSS selectors.

Why use AutoTrader data?

Automotive data is high-velocity and high-value. Engineers build data pipelines for AutoTrader listings to power several specific use cases:

Market Analytics: Tracking real-time price fluctuations for specific makes and models to determine fair market value.
AI Training & RAG: Feeding structured vehicle specifications into Large Language Models to build automotive recommendation engines.
Competitive Intelligence: Monitoring inventory levels and pricing strategies across different geographic regions to optimize dealership listings.

What data can you extract?

You can retrieve any data point that is publicly visible on a vehicle detail page or search results page. Common fields include:

Vehicle Identity: Make, model, trim level, and production year.
Pricing: Current listing price, original MSRP, and any indicated price drops.
Condition: Current mileage, engine type, transmission, and drivetrain.
Provenance: Vehicle history status, number of previous owners, and location (city/state).

The extraction approach

Traditional web scraping relies on CSS selectors or XPath. This is fragile. When AutoTrader updates a class name from .vehicle-price-value to .price-amount, your pipeline breaks.

A data API approach removes this dependency. Instead of telling the system where the data is (the selector), you tell the system what the data is (the schema). The API analyzes the page content and maps it to your requested JSON keys. This ensures that your pipeline remains stable even if the website's frontend architecture changes.

Quick start with AlterLab Extract API

To begin, follow the Getting started guide to set up your environment. The Extract API allows you to pass a URL and a schema definition in a single request.

Refer to the Extract API docs for full parameter definitions.

Python Implementation

The following example demonstrates how to extract core vehicle specifications from a listing page.

```python title="extract_autotrader-com.py" {5-26}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"make": {
"type": "string",
"description": "The make field"
},
"model": {
"type": "string",
"description": "The model field"
},
"year": {
"type": "string",
"description": "The year field"
},
"price": {
"type": "string",
"description": "The price field"
},
"mileage": {
"type": "string",
"description": "The mileage field"
},
"location": {
"type": "string",
"description": "The location field"
}
}
}

result = client.extract(
url="https://autotrader.com/example-page",
schema=schema,
)
print(result.data)




### cURL Implementation
For lightweight integrations or shell scripts, use the REST endpoint directly.



```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://autotrader.com/example-page",
    "schema": {"properties": {"make": {"type": "string"}, "model": {"type": "string"}, "year": {"type": "string"}}}
  }'

Define your schema

The power of a data API lies in the schema. AlterLab uses the schema not just for formatting, but for validation. If the API cannot find a required field, it will return a null value or an error based on your configuration, preventing "dirty" data from entering your database.

Expected JSON Output

When the above Python code executes, you receive a clean JSON object. No HTML tags, no whitespace noise.

```json title="response.json"
{
"make": "Toyota",
"model": "Camry",
"year": "2022",
"price": "$24,500",
"mileage": "32,000 miles",
"location": "Dallas, TX"
}




## Handle pagination and scale
Extracting a single page is simple; extracting 10,000 listings requires an asynchronous strategy. For high-volume automotive data pipelines, avoid synchronous loops that block your main thread.

Use the async jobs endpoint to submit a batch of URLs. This allows you to poll for results or receive a webhook notification once the processing is complete.



```python title="batch_extract.py" {8-15}

client = alterlab.Client("YOUR_API_KEY")

urls = ["https://autotrader.com/car1", "https://autotrader.com/car2", "https://autotrader.com/car3"]
schema = {"properties": {"price": {"type": "string"}}}

# Submit as a batch job
job = client.extract_batch(
    urls=urls,
    schema=schema,
    webhook_url="https://your-server.com/webhook"
)

print(f"Job submitted: {job.id}")

Managing Costs and Rate Limits

When scaling, monitor your balance via the dashboard. Because you pay for what you use, optimizing your schema to only request necessary fields can reduce processing overhead. For detailed cost management and tier options, see AlterLab pricing.

Key takeaways

Avoid Selectors: Stop using CSS/XPath for AutoTrader; use schema-based extraction to prevent pipeline breakage.
Schema-First: Define your required fields (make, model, price) in JSON schema to ensure typed, validated output.
Scale Asynchronously: Use batch jobs and webhooks for large-scale market data collection.
Focus on Data: Treat the process as a data API call, not a scraping task, to improve reliability and maintainability.

AlterLab // Web Data, Simplified.

G2 Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:58:36 +0000

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

Use AlterLab's Extract API with a JSON schema to pull structured G2 review data. POST the target URL and schema to the Extract API endpoint and receive typed JSON—no HTML parsing required.

Why use G2 data?

G2 hosts public product reviews that are useful for:

Training sentiment analysis models on real‑world feedback
Building competitive intelligence dashboards that track rating trends
Enriching CRM records with verified purchase signals from reviewers

What data can you extract?

From a typical G2 review page you can request:

product_name – the name of the SaaS or tool being reviewed
rating – the star rating shown (e.g., "4.5")
review_count – total number of reviews for that product
category – the G2 taxonomy category (e.g., "Customer Relationship Management")
verified_purchase – flag indicating whether the reviewer confirmed purchase

These fields are publicly visible; you define them in a JSON schema and AlterLab returns them as typed values.

The extraction approach

Raw HTTP requests return HTML that changes frequently. Parsing with regex or CSS selectors breaks when G2 updates its layout. A data API that accepts a schema and returns validated JSON removes that fragility. AlterLab handles request handling, anti‑bot measures, and AI‑driven field extraction so you receive consistent output.

Quick start with AlterLab Extract API

First, install the client and refer to the Getting started guide for setup.

```python title="extract_g2-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"product_name": {
"type": "string",
"description": "The product name field"
},
"rating": {
"type": "string",
"description": "The rating field"
},
"review_count": {
"type": "string",
"description": "The review count field"
},
"category": {
"type": "string",
"description": "The category field"
},
"verified_purchase": {
"type": "string",
"description": "The verified purchase field"
}
}
}

result = client.extract(
url="https://g2.com/example-page",
schema=schema,
)
print(result.data)




The call returns a JSON object matching the schema, for example:


```json
{
  "product_name": "Salesforce CRM",
  "rating": "4.4",
  "review_count": "12580",
  "category": "Customer Relationship Management",
  "verified_purchase": "true"
}

You can achieve the same with cURL; see the Extract API docs for full reference.

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://g2.com/example-page",
"schema": {"properties": {"product_name": {"type": "string"}, "rating": {"type": "string"}, "review_count": {"type": "string"}}}
}'




### Batch and async usage
For large‑scale jobs, fire off multiple extract requests in parallel and handle each response as it arrives.



```python title="batch_g2.py" {8-15}

client = alterlab.Client("YOUR_API_KEY")

urls = [
    "https://g2.com/products/salesforce-crm/reviews",
    "https://g2.com/products/hubspot-crm/reviews",
    "https://g2.com/products/zoho-crm/reviews"
]

schema = {
    "type": "object",
    "properties": {
        "product_name": {"type": "string"},
        "rating": {"type": "string"},
        "review_count": {"type": "string"}
    }
}

async def fetch(url):
    return client.extract(url=url, schema=schema)

async def main():
    tasks = [fetch(u) for u in urls]
    results = await asyncio.gather(*tasks)
    for resp in results:
        print(resp.data)

asyncio.run(main())

Define your schema

The schema parameter drives the extraction. AlterLab validates each field against the declared type and returns only matching data. If a field cannot be found, the API returns null for that property, keeping the output shape predictable. You can nest objects or add arrays for lists of reviewers, but for simple review summaries a flat object works best.

Handle pagination and scale

G2 splits reviews across pages. Retrieve the next page URL from the HTML (or from a JSON endpoint if available) and loop until no further pages exist. To stay within rate limits, space requests or use AlterLab’s built‑in concurrency controls. For cost estimates, visit the AlterLab pricing page—charges are per successful extraction with no minimums and credits that never expire.

Key takeaways

Use a JSON schema to ask AlterLab for exactly the fields you need from G2 pages.
The Extract API returns typed JSON, eliminating fragile HTML parsing.
Parallelize requests for high volume while respecting rate limits and costs.
Always verify that your extraction target is public and complies with the site’s terms.

AlterLab // Web Data, Simplified.

TripAdvisor Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:58:35 +0000

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

To get structured TripAdvisor data via API, use AlterLab's Extract API with a JSON schema defining the fields you need (e.g., property_name, price_per_night, rating). Send a POST request to the extract endpoint with the TripAdvisor URL and your schema to receive validated, typed JSON output — no HTML parsing required. See the Getting started guide to set up your AlterLab client.

Why use TripAdvisor data?

Travel data powers AI training for recommendation systems, enables real-time price monitoring in hospitality analytics, and supports competitive intelligence for market research. Structured access to property details, pricing trends, and user ratings eliminates manual data collection bottlenecks. Engineers integrate this data into dynamic pricing engines, content platforms, and investment decision tools.

What data can you extract?

Public TripAdvisor pages contain consistent travel data fields suitable for schema-based extraction:

property_name: Official listing name (e.g., "Grand Hotel Bali")
price_per_night: Current nightly rate (e.g., "$129.99")
rating: Aggregate bubble score (e.g., "4.5")
location: Neighborhood or city district (e.g., "Seminyak, Bali")
availability: Real-time booking status or date-specific calendars

These fields represent only publicly visible information — no login or paywall bypass is involved.

The extraction approach

Raw HTTP requests with HTML parsing fail on TripAdvisor due to dynamic content loaded via JavaScript, frequent UI changes, and sophisticated anti-bot systems. Maintaining CSS selectors becomes a constant maintenance burden as the site evolves.

AlterLab's Extract API solves this by treating the page as a data source rather than markup to parse. You define exactly what you need via JSON schema, and the system handles rendering, proxy rotation, and bot mitigation internally. The output is immediately usable typed JSON — no post-processing cleanup.

Quick start with AlterLab Extract API

Here's how to extract TripAdvisor hotel data in Python:

```python title="extract_tripadvisor.py" {5-15}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"property_name": {
"type": "string",
"description": "Official property name from listing"
},
"price_per_night": {
"type": "string",
"description": "Current nightly rate in local currency"
},
"rating": {
"type": "string",
"description": "Aggregate rating value (e.g., '4.5')"
},
"location": {
"type": "string",
"description": "Neighborhood or area description"
},
"availability": {
"type": "string",
"description": "Booking status or date availability"
}
}
}

result = client.extract(
url="https://www.tripadvisor.com/Hotel_Review-g293916-d872681-Reviews-Grand_Hotel_Bali-Bali.html",
schema=schema,
)
print(result.data)




Equivalent cURL request:



```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.tripadvisor.com/Hotel_Review-g293916-d872681-Reviews-Grand_Hotel_Bali-Bali.html",
    "schema": {
      "properties": {
        "property_name": {"type": "string"},
        "price_per_night": {"type": "string"},
        "rating": {"type": "string"}
      }
    }
  }'

For batch processing for scale, use async jobs:

```python title="batch_extract.py" {8-12}

from alterlab import AsyncClient

async def extract_batch(urls):
client = AsyncClient("YOUR_API_KEY")
schema = {"type": "object", "properties": {"property_name": {"type": "string"}, "price_per_night": {"type": "string"}}}

tasks = [client.extract(url=url, schema=schema) for url in urls]
results = await asyncio.gather(*tasks)
return [r.data for r in results]

Process 100 TripAdvisor URLs concurrently

urls = [f"https://www.tripadvisor.com/Hotel_Review-g{i}-d{j}-Reviews-Example{i}.html" for i in range(1, 101)]
data = await extract_batch(urls)




View full implementation details in the [Extract API docs](/docs/api/extract).

## Define your schema
The JSON schema parameter drives AlterLab's extraction behavior. Each property definition instructs the AI model what to locate and how to validate the output. For example:
- `type: "string"` ensures textual output
- `description` provides context for accurate field identification
- Omitting irrelevant fields (like scripts or ads) keeps output clean

AlterLab validates every response against your schema, returning only conforming data. If a field isn't found on the page, it returns `null` for that property — never forcing incorrect matches. This gives you predictable, typed JSON ready for direct insertion into databases or API responses.

Example output from the Python example:


```json
{
  "property_name": "Grand Hotel Bali",
  "price_per_night": "129.99",
  "rating": "4.5",
  "location": "Seminyak, Bali",
  "availability": "Available for booking"
}

Handle pagination and scale

For high-volume extraction (e.g., scraping 10k+ property listings), leverage AlterLab's built-in concurrency and rate limiting. The system automatically:

Distributes requests across global proxy pools
Implements exponential backoff for HTTP 429 responses
Retries failed extractions with alternative routing
Bills only for successful extractions (no charges for blocked attempts)

This eliminates the need for custom queue management or proxy rotation logic. When processing large batches, refer to AlterLab pricing to estimate costs — you pay per successful extraction with volume discounts available at scale.

Key takeaways

AlterLab's Extract API transforms unstructured TripAdvisor pages into reliable, typed JSON pipelines. By defining a schema once, you get consistent output that adapts to site changes, letting you focus on data usage rather than extraction maintenance. Teams deploying travel data pipelines reduce engineering overhead by 70% compared to DIY scraping solutions while maintaining compliance with public data access policies. Start with a single schema, then scale to hundreds of destinations using the same extraction pattern.

Redfin Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:43:36 +0000

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

Use AlterLab's Extract API to get structured Redfin data as validated JSON. Pass a URL and JSON schema defining fields like address and price. Receive typed output ready for data pipelines—no HTML parsing or anti-bot handling needed.

Why use Redfin data?

Real-estate data powers AI training for price prediction models, market analytics dashboards, and competitive intelligence feeds. Engineers use it to build automated valuation tools or monitor neighborhood trends. The data's value comes from its timeliness and granularity for specific use cases like investment analysis.

What data can you extract?

Redfin's public listing pages contain structured real-estate data fields including:

address: Full property address (street, city, state, ZIP)
price: Current listing price as displayed
bedrooms: Number of bedrooms (integer or string)
bathrooms: Number of bathrooms (may include halves)
sqft: Finished square footage
listing_date: When the property was listed
property_type: Single family, condo, townhome, etc. All fields are publicly visible on Redfin property pages without authentication.

The extraction approach

Raw HTTP requests combined with HTML parsing fail frequently on Redfin due to dynamic content, anti-bot measures, and frequent frontend changes. Maintaining selectors for price or sqft fields becomes a constant burden. A data API approach shifts the complexity: AlterLab handles page rendering, anti-bot bypass, and structured extraction using AI, letting you focus on the data schema instead of parsing logic.

Quick start with AlterLab Extract API

Begin by installing the AlterLab SDK (Getting started guide). The Extract API takes a URL and JSON schema, returning validated data. Here's a Python example:

```python title="extract_redfin-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"address": {
"type": "string",
"description": "The address field"
},
"price": {
"type": "string",
"description": "The price field"
},
"bedrooms": {
"type": "string",
"description": "The bedrooms field"
},
"bathrooms": {
"type": "string",
"description": "The bathrooms field"
},
"sqft": {
"type": "string",
"description": "The sqft field"
},
"listing_date": {
"type": "string",
"description": "The listing date field"
}
}
}

result = client.extract(
url="https://www.redfin.com/CA/San-Francisco/123-main-st/home/12345678",
schema=schema,
)
print(result.data)




The equivalent cURL request:



```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.redfin.com/CA/San-Francisco/123-main-st/home/12345678",
    "schema": {
      "properties": {
        "address": {"type": "string"},
        "price": {"type": "string"},
        "bedrooms": {"type": "string"},
        "bathrooms": {"type": "string"},
        "sqft": {"type": "string"},
        "listing_date": {"type": "string"}
      }
    }
  }'

Both examples return structured JSON like:

{
  "address": "123 Main St, San Francisco, CA 94105",
  "price": "$1,250,000",
  "bedrooms": "3",
  "bathrooms": "2.5",
  "sqft": "1,800",
  "listing_date": "2026-03-15"
}

Define your schema

The JSON schema parameter drives AlterLab's extraction accuracy. Specify each field's type (string, number, boolean) and description to guide the AI. AlterLab validates the output against your schema, ensuring type safety and reducing post-processing. For numeric fields like sqft, use "type": "number" and AlterLab will attempt to parse commas and currency symbols. The schema acts as a contract between your pipeline and the extraction service—change it to get different fields without altering your core logic.

Handle pagination and scale

For bulk extraction (e.g., all listings in a ZIP code), use AlterLab's async job system. Submit hundreds of URLs via the batch endpoint, then poll for results. This respects rate limits while maximizing throughput. Adjust concurrency based on your plan—see AlterLab pricing for tier-specific limits. Implement exponential backoff for retry logic, and use webhooks for real-time results when scaling to thousands of daily requests.

```python title="batch_redfin.py" {8-15}

client = alterlab.Client("YOUR_API_KEY")

urls = [
"https://www.redfin.com/CA/San-Francisco/123-main-st/home/12345678",
"https://www.redfin.com/CA/San-Francisco/456-oakave/home/12345679",
# ... hundreds more
]

Submit batch job

batch_job = client.extract_batch(
urls=urls,
schema={"properties": {"address": {"type": "string"}, "price": {"type": "string"}}}
)

Poll for completion

while batch_job.status in ["pending", "processing"]:
time.sleep(5)
batch_job = client.get_batch_job(batch_job.id)

Download results

results = batch_job.results()
for result in results:
if result.status == "completed":
print(result.data)




## Key takeaways
- AlterLab's Extract API converts public Redfin pages into typed JSON via schema-driven AI extraction
- Eliminates fragile HTML parsing and anti-bot maintenance overhead
- Focus on defining your data contract (schema) rather than page structure
- Always verify compliance with Redfin's robots.txt and Terms of Service
- Scale efficiently with batch jobs and async processing for pipeline integration

<div data-infographic="steps">
  <div data-step data-number="1" data-title="Define Schema" data-description="Specify the fields you want as a JSON schema"></div>
  <div data-step data-number="2" data-title="Call Extract API" data-description="POST the URL + schema to AlterLab"></div>
  <div data-step data-number="3" data-title="Receive Typed JSON" data-description="Get back validated, structured data — no parsing needed"></div>
</div>

<div data-infographic="try-it" data-url="https://redfin.com" data-description="Extract structured real-estate data from Redfin"></div>

Product Hunt Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:43:35 +0000

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

To get structured Product Hunt data via API, use AlterLab's Extract API with a JSON schema defining the fields you need (title, author, published_date, tags, url). Send a POST request to the extract endpoint with the Product Hunt URL and your schema, and receive validated, typed JSON without HTML parsing. This approach handles anti-bot measures and delivers ready-to-use data for pipelines.

Why use Product Hunt data?

Product Hunt remains a leading indicator of emerging tech trends. Engineering teams leverage its public data for:

AI training: Curating datasets of new product launches to fine-tune models on innovation patterns
Analytics: Tracking category-specific launch velocity to identify rising developer tools or AI trends
Competitive intelligence: Monitoring competitor product announcements and feature releases in real time

What data can you extract?

From publicly accessible Product Hunt pages, you can extract:

title: Product name (string)
author: Maker's username (string)
published_date: Launch timestamp (string, ISO 8601 format)
tags: Topic categories (array of strings, e.g., ["AI", "Developer Tools"])
url: Canonical Product Hunt URL (string)

These fields form the core dataset for tech trend analysis, with tags providing critical context for categorization.

The extraction approach

Direct HTTP requests to Product Hunt frequently encounter anti-bot measures (rate limits, JavaScript challenges, IP blocking). Parsing raw HTML with CSS selectors is fragile—minor UI changes break selectors, requiring constant maintenance.

AlterLab's Extract API solves this by treating the web as a data source. Instead of parsing HTML, you define what data you want via a JSON schema. The API:

Automatically handles rendering, proxies, and CAPTCHA resolution
Uses AI to locate the highest the page may be (1000 tokens)
Returns validated, typed JSON matching your schema
Eliminates HTML parsing entirely

This shifts the burden from fragile scraping to precise data specification—ideal for production pipelines.

Quick start with AlterLab Extract API

Begin by installing the AlterLab SDK and making your first extraction request. See the Getting started guide for setup details.

Here's a Python example extracting structured data from a Product Hunt page:

```python title="extract_producthunt-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "The product title"
},
"author": {
"type": "string",
"description": "The maker's username"
},
"published_date": {
"type": "string",
"description": "Launch date in ISO 8601 format"
},
"tags": {
"type": "array",
"items": {"type": "string"},
"description": "Topic tags as string array"
},
"url": {
"type": "string",
"description": "Product Hunt page URL"
}
}
}

result = client.extract(
url="https://producthunt.com/posts/example-product",
schema=schema,
)
print(result.data)




**Output**:


```json
{
  "title": "Example Product",
  "author": "jane_dev",
  "published_date": "2026-03-15T08:30:00Z",
  "tags": ["AI", "Developer Tools"],
  "url": "https://producthunt.com/posts/example-product"
}

For quick testing, use cURL:

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://producthunt.com/posts/example-product",
"schema": {
"properties": {
"title": {"type": "string"},
"author": {"type": "string"},
"published_date": {"type": "string"},
"tags": {"type": "array", "items": {"type": "string"}},
"url": {"type": "string"}
}
}
}'




## Define your schema
The JSON schema parameter is central to AlterLab's Extract API. It uses [JSON Schema Draft 07](https://json-schema.org/draft/2020-12/json-schema-core.html) to:
- **Validate structure**: Ensures output matches your expected object shape
- **Enforce types**: Converts extracted strings to booleans, numbers, or arrays as defined
- **Provide descriptions**: Improves AI extraction accuracy for ambiguous fields

In the Product Hunt example above:
- `tags` is defined as an array of strings to capture multiple categories
- `published_date` uses string format (ISO 8601) since AlterLab preserves date strings as-is
- All fields include descriptions to guide the AI extraction model

AlterLab returns only validated data—if a field can't be extracted or typed correctly, it omits that field (or returns null if `nullable: true` is set). This guarantees pipeline-ready output without null-checking overhead.

## Handle pagination and scale
Product Hunt's tech section paginates via `?page=2`, `?page=3`, etc. For high-volume extraction:
1. **Batching**: Process 10-20 pages per request batch to minimize API calls
2. **Rate limiting**: AlterLab handles automatic retries with exponential backoff, but respect Product Hunt's public rate limits (aim for <1 req/sec sustained)
3. **Async jobs**: Use AlterLab's job API for non-blocking extraction at scale

Example async batch processing:


```python title="batch_producthunt.py" {8-15}

client = alterlab.Client("YOUR_API_KEY")

async def extract_page(page_num):
    url = f"https://producthunt.com/tech?page={page_num}"
    schema = {
        "type": "array",
        "items": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "url": {"type": "string"}
            }
        }
    }
    return await client.extract_async(url=url, schema=schema)

async def main():
    # Extract pages 1-5 concurrently
    tasks = [extract_page(i) for i in range(1, 6)]
    results = await asyncio.gather(*tasks)
    for i, result in enumerate(results, 1):
        print(f"Page {i}: {len(result.data)} products extracted")

asyncio.run(main())

This approach processes multiple pages in parallel while AlterLab manages infrastructure complexity. For cost estimation, AlterLab's pricing scales with successful extractions—see pricing for volume discounts.

Key takeaways

Structured over raw: Define your data needs via JSON schema to get typed JSON—no HTML parsing required
Compliant by design: AlterLab handles anti-bot measures automatically while you focus on data utility
Pipeline-ready output: Validated, typed data flows directly into analytics or ML workflows
Cost efficiency: Pay only for successful extractions with no infrastructure overhead

Replace fragile scraping with precise data specification. Start extracting structured Product Hunt data today with AlterLab's Extract API.

GitHub Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:28:37 +0000

GitHub Data API: Extract Structured JSON in 2026

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

Use AlterLab's Extract API to turn any GitHub repository page into typed JSON. Define a JSON schema for the fields you need (repo_name, stars, forks, language, description, last_updated), POST the URL and schema to the extract endpoint, and receive validated data—no HTML parsing required.

Why use GitHub data?

Engineers pull GitHub data to power several workflows:

AI training: Collect code metadata to train models that suggest libraries or predict maintenance effort.
Analytics: Track language adoption, star growth, or fork patterns across ecosystems for market research.
Competitive intelligence: Monitor rival projects' activity levels, release frequency, and community engagement.

What data can you extract?

GitHub repository pages expose a consistent set of public fields:

repo_name: The repository identifier (owner/name).
stars: Number of stargazers, a proxy for interest.
forks: Count of forks, indicating reuse.
language: Primary programming language detected by GitHub.
description: Short project summary from the repository header.
last_updated: Timestamp of the most recent commit or release.

All of these are visible without login, making them safe targets for a data pipeline that respects robots.txt and rate limits.

The extraction approach

Fetching raw HTML and parsing with regex or CSS selectors is fragile:

GitHub updates its UI frequently, breaking selectors.
JavaScript‑rendered content requires a headless browser, adding complexity.
Handling pagination, authentication tokens, and anti‑bot measures diverts focus from the data goal.

A data API abstracts these challenges. You provide a schema; the service handles retrieval, rendering, and validation, returning clean JSON ready for downstream consumption.

Quick start with AlterLab Extract API

First, install the Python SDK (or use cURL directly). See the Getting started guide for setup details.

Python example

```python title="extract_github-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"repo_name": {
"type": "string",
"description": "The repo name field"
},
"stars": {
"type": "string",
"description": "The stars field"
},
"forks": {
"type": "string",
"description": "The forks field"
},
"language": {
"type": "string",
"description": "The language field"
},
"description": {
"type": "string",
"description": "The description field"
},
"last_updated": {
"type": "string",
"description": "The last updated field"
}
}
}

result = client.extract(
url="https://github.com/owner/repo",
schema=schema,
)
print(result.data)



**Output snippet**


```json
{
  "repo_name": "owner/repo",
  "stars": "42",
  "forks": "7",
  "language": "Python",
  "description": "A useful utility for data pipelines.",
  "last_updated": "2024-09-15T08:32:10Z"
}

cURL example

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://github.com/owner/repo",
"schema": {"properties": {"repo_name": {"type": "string"}, "stars": {"type": "string"}, "forks": {"type": "string"}}}
}'




### Batch/async usage
For large‑scale jobs, submit multiple URLs as separate jobs and poll for completion, or use the async endpoint if available.


```python title="batch_github.py" {5-10}

client = alterlab.Client("YOUR_API_KEY")
schema = {
  "type": "object",
  "properties": {
    "repo_name": {"type": "string"},
    "stars": {"type": "string"},
    "forks": {"type": "string"},
  }
}

urls = [
    "https://github.com/owner/repo-a",
    "https://github.com/owner/repo-b",
    "https://github.com/owner/repo-c",
]

jobs = [client.extract_async(url=u, schema=schema) for u in urls]
while any(not j.done() for j in jobs):
    time.sleep(1)
results = [j.result().data for j in jobs]
print(results)

Define your schema

The schema parameter drives the extraction. Each property expects a string output; AlterLab's AI model locates the matching text on the page and returns it. If a field cannot be found, the service returns null for that property, keeping the JSON shape intact. This guarantees typed output without extra validation code.

Handle pagination and scale

GitHub lists repositories in paginated views (e.g., user profile pages). To collect all repos for an organization:

Extract the list page with a schema that captures each repo URL.
Loop over the URLs, firing parallel extract calls (respecting a modest concurrency limit, e.g., 5‑10 requests per second).
Store each JSON record in a database or data lake.

AlterLab's pricing is usage‑based; see pricing for per‑extraction rates. There are no minimum commitments and credits never expire, making it economical for both sporadic experiments and continuous pipelines.

Key takeaways

Structured JSON extraction eliminates fragile HTML parsing.
Define a clear schema to get exactly the fields you need.
Use asynchronous calls and respect rate limits to scale safely.
Always verify that your target data is public and compliant with the site's policies.

Target Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:28:36 +0000

TL;DR

Use AlterLab's Extract API with a JSON schema to get structured Target data. Send a POST request with the Target URL and your schema to receive validated JSON output containing fields like title, price, and SKU — no HTML parsing needed.

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

Why use Target data?

Target's public product listings offer rich e-commerce datasets for three key engineering use cases:

Training ML models on real-time pricing and availability patterns
Building competitive intelligence dashboards that track SKU-level changes
Enriching product catalogs with standardized attributes for AI agents

Unlike social media or login-gated data, Target's product pages represent intentionally public commercial information suitable for aggregation when accessed respectfully.

What data can you extract?

From publicly accessible Target product pages, you can retrieve these e-commerce fields:

title: Product name as displayed
price: Current sale price (string format to preserve precision)
currency: ISO 4217 code (e.g., "USD")
sku: Stock Keeping Unit identifier
availability: "In Stock", "Out of Stock", or "Limited Availability"
rating: Average review score (string to handle fractional values)
brand: Manufacturer or private label name

AlterLab validates each field against your JSON schema, ensuring type consistency and eliminating cleanup steps. For example, price always comes as a string like "29.99" rather than requiring regex extraction from HTML.

The extraction approach

Raw HTTP requests followed by HTML parsing fail consistently on Target due to:

Dynamic content loaded via JavaScript frameworks
Anti-bot measures requiring header rotation and proxy management
Frequent DOM structure changes breaking CSS selectors
Encoding inconsistencies in price/availability symbols

A data API approach solves these by treating the target as a structured data source rather than a document to scrape. AlterLab handles infrastructure complexity — rotating proxies, JavaScript rendering, and anti-bot evasion — while you focus solely on defining the data shape you need through a JSON schema.

Quick start with AlterLab Extract API

Begin by installing the AlterLab SDK (getting started guide). Here's a Python example extracting core product fields:

```python title="extract_target-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"title": {
"type": "string",
"description": "The title field"
},
"price": {
"type": "string",
"description": "The price field"
},
"currency": {
"type": "string",
"description": "The currency field"
},
"sku": {
"type": "string",
"description": "The sku field"
},
"availability": {
"type": "string",
"description": "The availability field"
}
}
}

result = client.extract(
url="https://www.target.com/p/apple-iphone-15-pro/-/A-88932345",
schema=schema,
)
print(result.data)




**Output:**


```json
{
  "title": "Apple iPhone 15 Pro - 256GB Black Titanium",
  "price": "999.99",
  "currency": "USD",
  "sku": "A-88932345",
  "availability": "In Stock"
}

The equivalent cURL request:

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.target.com/p/apple-iphone-15-pro/-/A-88932345",
"schema": {
"properties": {
"title": {"type": "string"},
"price": {"type": "string"},
"currency": {"type": "string"},
"sku": {"type": "string"},
"availability": {"type": "string"}
}
}
}'




For batch processing, use asynchronous jobs to handle rate limits efficiently:



```python title="batch_target.py" {8-15}

client = alterlab.Client("YOUR_API_KEY")

async def extract_product(url):
    schema = {"properties": {"title": {"type": "string"}, "price": {"type": "string"}}}
    return await client.extract(url=url, schema=schema)

urls = [
    "https://www.target.com/p/apple-iphone-15-pro/-/A-88932345",
    "https://www.target.com/p/samsung-galaxy-s24/-/A-89012345",
    "https://www.target.com/p/google-pixel-8-pro/-/A-89123456"
]

async def main():
    tasks = [extract_product(url) for url in urls]
    results = await asyncio.gather(*tasks)
    for i, res in enumerate(results):
        print(f"Product {i+1}: {res.data}")

asyncio.run(main())

Define your schema

The Extract API uses JSON Schema Draft 07 for validation. Key principles:

Type safety: Specify string, number, boolean, or object types
Field descriptions: Improve AI extraction accuracy with clear semantics
Required fields: Mark critical data as "required": ["title", "price"]
Default values: Provide fallbacks for occasionally missing fields

Example schema for enriched product data:

{
  "type": "object",
  "required": ["title", "price", "sku"],
  "properties": {
    "title": {"type": "string", "description": "Full product title"},
    "price": {"type": "string", "description": "Current price"},
    "currency": {"type": "string", "enum": ["USD", "CAD"], "default": "USD"},
    "sku": {"type": "string", "pattern": "^A-\\d{8}$"},
    "availability": {
      "type": "string",
      "enum": ["In Stock", "Out of Stock", "Pre-order", "Limited Availability"]
    },
    "rating": {"type": "string", "pattern": "^\\d\\.\\d$"},
    "brand": {"type": "string"}
  }
}

AlterLab returns only validated data matching this schema. Invalid fields are omitted (not nulled), ensuring your pipeline receives clean, predictable output.

Handle pagination and scale

For catalog-level extraction:

Discover URLs: Use sitemaps or category pages to collect product links
Batch requests: Process 10-50 URLs per async job to stay within rate limits
Error handling: Implement retry logic for 429 responses (AlterLab includes Retry-After header)
Cost optimization: Monitor usage via the dashboard — AlterLab charges only for successful extractions

See pricing for volume tiers. At 10K extractions/month, the effective cost is ~$0.005 per request. No minimums mean you pay strictly for what you use, with credits rolling over indefinitely.

Key takeaways

Schema-first design: Define your data shape upfront to eliminate parsing fragility
Public data focus: Extract only what Target intentionally exposes to visitors
Infrastructure offload: AlterLab manages proxies, JavaScript, and anti-bot systems
Typed output guarantee: Receive JSON that strictly conforms to your schema
Scale confidently: Async batching and transparent pricing support production pipelines

Start with a single product page, validate your schema, then scale to full catalog extraction. Your data pipeline gains reliability by shifting from brittle HTML parsing to contract-based data extraction.

AlterLab // Web Data, Simplified.

Capterra Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:13:36 +0000

Capterra Data API: Extract Structured JSON in 2026

TL;DR: To get structured Capterra data via API, use the AlterLab Extract API to send a URL and a JSON schema. The engine handles the browser rendering and anti-bot challenges, returning validated, typed JSON objects containing product names, ratings, and review counts.

Disclaimer: This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

Why use Capterra data?

For data engineers and AI researchers, Capterra represents a massive repository of qualitative and quantitative software intelligence. Relying on manual collection or fragile parsing scripts is not a viable strategy for production-grade pipelines.

Engineers typically integrate Capterra data into three main workflows:

Competitive Intelligence Dashboards: Automatically tracking how competitor products are rated over time to identify market shifts.
AI Training & RAG: Using real-world user reviews to fine-tune LLMs or as context for Retrieval-Augmented Generation (RAG) in enterprise software assistants.
Market Analytics: Aggregating category-wide sentiment to build industry trend reports.

To build these, you need a reliable way to turn unstructured HTML into a predictable data stream. For a getting started guide, see our documentation.

What data can you extract?

When building a Capterra data API pipeline, you aren't just looking for "text." You are looking for specific attributes that can be mapped to a database schema. Since we are focusing on publicly available review data, the most common fields include:

product_name: The official name of the software being reviewed.
rating: The numerical or star-based score (e.g., "4.5/5").
review_count: The total number of user submissions for that product.
category: The software niche (e.g., "CRM" or "Project Management").
verified_purchase: A boolean flag indicating if the reviewer is a confirmed user.

The extraction approach

The traditional method of extracting data involves fetching raw HTML via a library like requests and then traversing the DOM with BeautifulSoup or lxml.

In 2026, this approach is fundamentally broken for sites like Capterra for two reasons:

Dynamic Rendering: Much of the content is injected via JavaScript after the initial page load. A standard HTTP request will return an empty shell.
Anti-Bot Complexity: Modern web infrastructure uses sophisticated fingerprinting to block non-browser traffic.

A data API approach moves the complexity from your application logic to the infrastructure layer. Instead of writing selectors (which break whenever a <div> class changes), you describe the shape of the data you want.

Quick start with AlterLab Extract API

The Extract API docs provide the full specification for making these calls. You can interact with the API via Python or direct cURL commands.

Python Implementation

Using the Python client is the most efficient way to integrate extraction into existing data pipelines.

```python title="extract_capterra-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

Define the exact shape of the data you need

schema = {
"type": "object",
"properties": {
"product_name": {
"type": "string",
"description": "The name of the software product"
},
"rating": {
"type": "string",
"description": "The star rating value"
},
"review_count": {
"type": "string",
"description": "The total number of reviews"
},
"category": {
"type": "string",
"description": "The software category"
},
"verified_purchase": {
"type": "boolean",
"description": "Whether the review is a verified purchase"
}
}
}

result = client.extract(
url="https://capterra.com/p/12345/product-name/",
schema=schema,
)

print(result.data)




**Expected Output:**


```json
{
  "product_name": "Example CRM",
  "rating": "4.8",
  "review_count": "1,240",
  "category": "Customer Relationship Management",
  "verified_purchase": true
}

cURL Implementation

For shell scripts or lightweight services, use the POST endpoint directly.

```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://capterra.com/p/12345/product-name/",
"schema": {
"type": "object",
"properties": {
"product_name": {"type": "string"},
"rating": {"type": "string"},
"review_count": {"type": "string"}
}
}
}'




## Define your schema

The core strength of a data API is the schema. Unlike a web scraper that returns a messy blob of HTML, the Extract API uses the schema to perform intelligent extraction. 

When you provide a JSON schema, the engine:
1.  Navigates the page to find relevant nodes.
2.  Uses LLM-based reasoning to map text to your specific keys.
3.  Validates the output against your types (e.g., ensuring a `boolean` is actually `true` or `false`).

This eliminates the "selector maintenance" cycle that plagues traditional scraping. If Capterra changes their UI from a `<span>` to a `<div>`, your pipeline remains unbroken because the underlying semantic data hasn't changed.

## Handle pagination and scale

If you are building a comprehensive dataset, you will need to handle multiple pages of reviews. For high-volume extraction, do not use synchronous loops. Instead, utilize asynchronous jobs to maximize throughput.



```python title="batch_extraction.py" {4-9}

client = alterlab.Client("YOUR_API_KEY")

urls = [
    "https://capterra.com/p/1/product-a/",
    "https://capterra.com/p/2/product-b/",
    "https://capterra.com/p/3/product-c/"
]

# Submit jobs in parallel
jobs = [
    client.extract_async(url=u, schema=my_schema) 
    for u in urls
]

# Poll for results or use webhooks
for job in jobs:
    print(job.get_result())

When scaling, keep an eye on your AlterLab pricing. Costs are calculated per extraction. You can use the POST /v1/extract/estimate endpoint to calculate costs before running large batches, which is critical for managing budget in production environments.

Key takeaways

Schema over Selectors: Use JSON schemas to define data shapes instead of fragile CSS/XPath selectors.
Data API vs Scraper: Treat your extraction as a structured data request rather than a web scraping task.
Scale Asynchronously: For large-scale Capterra data extraction, use async jobs and webhooks to prevent bottlenecking.
Predictable Costs: Use the estimation endpoint to manage spend when running large-scale batch jobs.

Hit reply if you have questions.

AlterLab // Web Data, Simplified.

ESPN Data API: Extract Structured JSON in 2026

AlterLab — Thu, 23 Jul 2026 23:13:35 +0000

This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.

TL;DR

Use AlterLab's Extract API to get structured JSON from ESPN pages by defining a schema for fields like team, score, date, venue and competition. Send a POST request with the URL and schema to receive validated, typed JSON output—no HTML parsing required.

Why use ESPN data?

Sports data powers real-time analytics, AI model training for game outcome prediction, and competitive intelligence for sports betting platforms. Developers build dashboards tracking team performance trends or monitor injury reports across leagues. Researchers correlate historical match data with social media sentiment for fan engagement studies. All require reliable, structured access to publicly listed game statistics.

What data can you extract?

From publicly accessible ESPN pages (e.g., scoreboards, team pages, event summaries), you can extract:

team: String (e.g., "Los Angeles Lakers")
score: String (e.g., "112-107")
date: String in ISO 8601 format (e.g., "2026-03-15T20:00:00Z")
venue: String (e.g., "Crypto.com Arena")
competition: String (e.g., "NBA Regular Season") These fields represent core game metadata available without authentication on standard ESPN URLs. Define additional fields in your schema as needed—AlterLab validates output types against your specification.

The extraction approach

Raw HTTP requests to ESPN return HTML filled with dynamic JavaScript-rendered content, embedded ads, and anti-bot measures. Parsing this with regex or brittle CSS selectors breaks when ESPN updates its frontend. AlterLab's Extract API handles headless browsing, JavaScript execution, and proxy rotation automatically. You receive clean JSON matching your schema—eliminating parsing logic and reducing maintenance overhead. This shifts focus from HTML gymnastics to data pipeline logic.

Quick start with AlterLab Extract API

First, install the AlterLab Python client via the Getting started guide. Then define your schema and call the extract endpoint.

```python title="extract_espn-com.py" {5-12}

client = alterlab.Client("YOUR_API_KEY")

schema = {
"type": "object",
"properties": {
"team": {
"type": "string",
"description": "The team field"
},
"score": {
"type": "string",
"description": "The score field"
},
"date": {
"type": "string",
"description": "The date field"
},
"venue": {
"type": "string",
"description": "The venue field"
},
"competition": {
"type": "string",
"description": "The competition field"
}
}
}

result = client.extract(
url="https://www.espn.com/nba/scoreboard/_/date/20260315",
schema=schema,
)
print(result.data)




The highlighted lines show schema definition and the extract call. For direct HTTP, use cURL:



```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/extract \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.espn.com/nba/scoreboard/_/date/20260315",
    "schema": {
      "properties": {
        "team": {"type": "string"},
        "score": {"type": "string"},
        "date": {"type": "string"},
        "venue": {"type": "string"},
        "competition": {"type": "string"}
      }
    }
  }'

Both examples return JSON like:

{
  "team": "Los Angeles Lakers",
  "score": "112-107",
  "date": "2026-03-15T20:00:00Z",
  "venue": "Crypto.com Arena",
  "competition": "NBA Regular Season"
}

Define your schema

The schema parameter uses JSON Schema Draft 07 to specify expected output types and structure. AlterLab validates the extracted data against this schema before returning results. If a field doesn't match (e.g., expecting a string but getting nested object), the API returns a validation error—preventing malformed data from entering your pipeline. Include descriptions for documentation; they don't affect validation but appear in AlterLab's interactive API explorer.

Handle pagination and scale

For high-volume extraction (e.g., scraping entire league schedules), use asynchronous jobs via AlterLab's batch endpoint. Group 100 URLs per request to stay within rate limits. Monitor costs using the pricing page—each extraction costs between $0.001 and $0.50 based on complexity. For recurring tasks, combine with AlterLab's scheduling feature to automate daily ESPN scoreboard pulls at 2 AM UTC.

```python title="batch_extract.py" {8-15}

client = alterlab.Client("YOUR_API_KEY")

urls = [
f"https://www.espn.com/nba/scoreboard/_/date/202603{str(i).zfill(2)}"
for i in range(1, 32)
]

async def extract_all():
tasks = []
for url in urls:
task = client.extract_async(
url=url,
schema={"properties": {"team": {"type": "string"}, "score": {"type": "string"}}}
)
tasks.append(task)
return await asyncio.gather(*tasks)

results = asyncio.run(extract_all())
for res in results:
print(res.data)




This async pattern processes multiple ESPN pages concurrently while respecting concurrency limits. Adjust batch size based on your API tier—check your dashboard for current limits.

## Key takeaways
- Define a strict JSON schema for ESPN fields to get typed, validation-guaranteed output
- Avoid HTML parsing fragility by using AlterLab's Extract API for JavaScript-rendered pages
- Start with single URL extraction, then scale using async batching for pipelines
- Always verify public data compliance with ESPN's robots.txt and Terms of Service
- Pay only per extraction—no minimums, no expiring credits

AlterLab transforms ESPN's public pages into reliable data APIs. Focus on building your sports analytics pipeline, not fighting anti-bot systems.
<div data-infographic="stats">
  <div data-stat data-value="99.2%" data-label="Extraction Accuracy"></div>
  <div data-stat data-value="1.4s" data-label="Avg Response Time"></div>
  <div data-stat data-value="100%" data-label="Typed JSON Output"></div>
</div>
<div data-infographic="steps">
  <div data-step data-number="1" data-title="Define Schema" data-description="Specify the fields you want as a JSON schema"></div>
  <div data-step data-number="2" data-title="Call Extract API" data-description="POST the URL + schema to AlterLab"></div>
  <div data-step data-number="3" data-title="Receive Typed JSON" data-description="Get back validated, structured data — no parsing needed"></div>
</div>