DEV Community: Minexa.ai

The complete web scraping process: what each stage actually involves

Minexa.ai — Wed, 01 Jul 2026 12:06:43 +0000

Web scraping is not one task. It is a sequence of distinct stages, each with its own failure modes. Understanding what each stage does makes it easier to decide where to invest time and where to offload work.

Stage 1: Define what you actually need

Before touching any code or tool:

What data do you need? Be specific. Product prices, job titles, property addresses, and review scores all live in different parts of a page.
Where is it? Identify the exact pages. Is it a list page, a detail page, or both?
How often do you need it? A one-off export is a different problem from a weekly recurring dataset.

Vague goals produce broken scrapers. Specificity at this stage saves hours later.

Stage 2: Inspect the site structure

Open your browser's developer tools and look at the HTML before writing anything.

Is the content in the initial HTML response, or does it load after the page via JavaScript?
Are the data points inside consistent, repeating containers?
How does pagination work? Next page button, infinite scroll, or a load more trigger?

Static content is straightforward to parse. Dynamic content requires JavaScript rendering, which adds complexity to any custom build.

Stage 3: Check the ethical and legal boundaries

Rate limiting: Do not hammer a server. Introduce delays between requests. One request per second is a common starting point.
Personal data: Avoid collecting personally identifiable information without a clear legal basis.

Stage 4: Choose your approach

Three broad paths exist:

1. Write it yourself
Python with requests and BeautifulSoup handles static pages well. For JavaScript-heavy sites, you need a headless browser like Playwright.

import requests
from bs4 import BeautifulSoup

response = requests.get('https://example.com/listings')
soup = BeautifulSoup(response.content, 'html.parser')
items = soup.find_all('div', class_='listing-card')
for item in items:
    print(item.find('span', class_='price').text)

This works. But you are also writing pagination logic, error handling, retry logic, and output validation yourself.

2. Use a dedicated extraction tool
Tools like Minexa.ai handle detection, pagination, JavaScript rendering, and output formatting automatically. You confirm what it found rather than specifying it manually.

3. Pass pages to an AI model
Works for one-off tasks on small volumes. Becomes unreliable and expensive at scale, particularly when pages contain multiple similar values that the model has to disambiguate.

Stage 5: Extract the data

Whether you write selectors manually or use a tool, extraction has the same sub-steps:

Fetch the HTML
Parse it into a navigable structure
Locate the elements containing your target data
Pull the values out
Clean them (strip whitespace, normalize formats)

One thing worth knowing: many pages have two layers of data. The list page shows summary information. Each result links to a detail page with fuller content. If you need both, your scraper has to follow those links and repeat the extraction on each detail page.

Minexa handles this natively. After detecting the list, you can instruct it to follow each result's link and extract the detail page content in the same run, no extra configuration needed.

Stage 6: Handle pagination

Most datasets span multiple pages. Your options:

Find the next page URL and loop
Simulate scroll events for infinite scroll
Click a load more button programmatically

Each requires different logic. Minexa detects the pagination type automatically and follows it across all pages without any setup.

Stage 7: Store and validate the output

Storage options by scale:

Scale	Format
Small	CSV, Excel, JSON
Medium	PostgreSQL, MySQL
Large	NoSQL, data warehouse

Validation checks to run:

Are any expected fields missing?
Are numeric fields stored as numbers, not strings?
Are there duplicate rows from overlapping pagination?

This step is often skipped and causes problems downstream. Minexa returns null for missing values rather than fabricating a substitute, which makes validation simpler because you are checking for nulls rather than hunting for plausible-looking wrong values.

Stage 8: Monitor and maintain

Websites change. A class name update, a layout redesign, or a new anti-bot layer can break a scraper silently or noisily.

Monitor output quality on each run
Set up alerts for empty results or format changes
Have a retraining or rewrite process ready

With Minexa, retraining after a site redesign takes the same few minutes as the original setup. The scraper ID stays stable, so downstream integrations do not break.

For recurring data needs, Minexa supports scheduled runs so the job executes automatically without manual triggering each time.

Where Minexa fits in this workflow

Minexa does not replace understanding the process. It replaces the implementation of the hardest parts:

No selector writing
No pagination logic
No JavaScript rendering setup
No output schema definition
Automatic field discovery across any page structure

The extension trains on a page once, then reuses that structure indefinitely. The same scraper that took a few minutes to set up can run against thousands of structurally similar pages without repeating setup.

Install the Minexa.ai extension and run your first extraction in under ten minutes.

For more on how extraction actually works under the hood, read: What actually happens when Minexa extracts data from a page

10 things every developer learns the hard way about web scraping

Minexa.ai — Fri, 26 Jun 2026 17:38:16 +0000

Starting with a simple GET request feels natural. Then the site returns a 403. Then you add headers, and it still fails. Then you discover the page needs JavaScript to render. Sound familiar? Here are the lessons most developers pick up the hard way.

1. A plain HTTP request is rarely enough

Sending a basic request works on static pages, but most modern sites require JavaScript execution before any useful content appears. The HTML you receive without a browser runtime is often just a loading shell.

2. Headers matter more than you think

Browsers send a detailed fingerprint with every request: browser type, version, OS, accepted encodings, and more. A minimal HTTP client sends almost none of that. Sites use this difference to flag automated requests almost immediately.

3. Cookies are not optional

Many sites tie session state to cookies set on the first visit. If your scraper does not carry those cookies forward, subsequent requests either fail or return incomplete data. A proper cookie jar is not a nice-to-have.

4. Headless browsers are powerful but expensive to scale

Puppeteer and Playwright solve the JavaScript rendering problem well. But running a headless browser per page at scale requires real infrastructure: memory, concurrency management, and a solid retry strategy. It is not a lightweight solution.

5. Anti-bot systems track more than your IP

IP blocking is just the start. Modern protection layers also analyze mouse movement patterns, timing between requests, TLS fingerprints, and behavioral signals across sessions. Rotating IPs alone does not get you far on well-protected sites.

6. Raw HTML is not your output format

Puppeteer gives you a DOM. BeautifulSoup gives you a parse tree. Neither gives you structured data. Turning raw HTML into clean, typed, consistently named fields is a separate problem that takes real effort to get right at scale.

7. Selectors break when sites update

CSS selectors and XPath expressions are tied to the current structure of a page. When a site updates its layout, those selectors silently stop working or start returning wrong values. Maintenance overhead grows with every scraper you own.

8. Scaling changes everything

A scraper that works for 50 pages often breaks at 5,000. Concurrency limits, rate limiting, memory leaks in long-running browser instances, and downstream parsing failures all become real problems only at volume.

9. Silent failures are the worst kind

If a field is missing and your scraper does not surface that explicitly, you end up with gaps in your dataset that are hard to detect later. A scraper that returns null loudly is far easier to operate than one that silently returns the wrong value.

10. You can skip most of this with the right extraction layer

Tools like the Minexa.ai API handle JavaScript rendering, anti-bot evasion, and structured field extraction without requiring you to write or maintain selectors. You train a scraper once using the browser extension, get a stable scraper ID, and call the API with that ID against any list of URLs.

import requests

response = requests.post(
    'https://api.minexa.ai/data',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    json={
        'scraper_id': 4821,
        'columns': 'top_25',
        'urls': ['https://example.com/listings']
    }
)
print(response.json())

Extraction is DOM-based and deterministic: every field maps to a fixed position in the page structure. If a value is not found, the output returns null rather than a fabricated substitute. No guessing, no hallucination risk.

For developers managing many different URLs across recurring jobs, the practical approach is to set up your own cron jobs and pass URL batches to the API on each run. The extraction side stays stable while your scheduling logic stays in your own infrastructure.

The Minexa.ai API docs cover the full request structure, credit consumption by page type, and how to handle paginated API responses in Python.

If you want more context on what breaks in production scraping pipelines and why, this is worth reading: Why your scraping setup works in testing but breaks in production.

AI agents for web scraping: what actually works vs. what gets oversold

Minexa.ai — Fri, 26 Jun 2026 17:32:04 +0000

The AI agent hype cycle has reached web scraping. Teams are shipping agentic workflows that use LLMs to browse pages, extract fields, and pipe structured data into downstream systems. Some of it works well. A lot of it is solving the wrong problem in the most expensive way possible.

Here is a more grounded look at what is actually happening.

'Agents are better than traditional scrapers because they self-heal'

Partially true. The tradeoff is often ignored.

The self-healing argument is real. When a site changes its layout, an LLM-based agent can sometimes reason its way to the right field without requiring a rewrite. That is genuinely useful, especially for teams that do not have dedicated scraping engineers.

But self-healing is not free. It means the agent is making judgment calls about which value on the page maps to which field in your schema. For pages with a single unambiguous value per field, this is fine. For pages with multiple similar values, like two prices, two dates, or two address lines, the model picks one. It does not always pick the right one. And it rarely signals that it was uncertain.

At small scale, this is manageable. Across tens of thousands of pages running on a schedule, it becomes a data quality problem that is hard to detect because the output still looks valid.

Structural extraction tools avoid this entirely. They bind each output field to a specific position in the page's DOM. If the value is not there, the field returns empty. No guessing, no silent substitution.

'Non-technical people can now build scrapers using prompts'

This is the most underrated real win.

Several engineering teams have noted that the biggest productivity gain from agentic scraping is not raw performance. It is that non-technical staff can now describe what they want in plain language instead of filing a ticket and waiting for an engineer to write Playwright selectors.

This is a legitimate shift. Freeing engineers from writing and maintaining scraping scripts is valuable, and it is happening in practice across teams in compliance, operations, and research.

The same shift is available without agents. Tools like Minexa.ai let non-technical users browse to a page, confirm what was detected automatically, and export structured data, without writing a single line of code or understanding how the page is built. The scraper trains itself based on the page structure, and the same configuration runs again on any structurally similar page.

'Token costs are manageable'

At low volume, yes. At scale, this is where budgets break.

A full HTML page can contain a very large amount of text, far more than the visible content. When you pass that to an LLM for extraction, you are paying for all of it, including navigation menus, scripts, footers, and boilerplate. Even with stripped HTML, the token count per page is significant.

For a one-off extraction of a handful of pages, this is not a concern. For a recurring job that processes thousands of pages per day, the cost scales directly with volume and page size. There is no efficiency gain at scale.

Structural extraction does not work this way. The cost per page stays flat regardless of how much HTML the page contains, because the tool is not reading the content, it is reading the structure. Whether you extract ten rows or ten thousand from the same page type, the setup cost is the same.

Try Minexa.ai: Install the Chrome extension and run your first extraction in a few minutes.

'LLMs are good enough for production data pipelines'

For specific use cases, yes. For general structured extraction at scale, the failure modes matter.

LLMs are genuinely good at tasks that require reading and synthesis, summarizing documents, classifying content, extracting information from unstructured text where the schema is loose. These are areas where the model's ability to reason adds real value.

For structured web extraction, the requirements are different. You want the same field to return the same value every time, across every page, with no variance. You want empty results when data is missing, not fabricated placeholders. You want costs that are predictable at volume.

LLM-based extraction can meet these requirements for simple pages. For pages with dense or ambiguous content, the accuracy and cost profile becomes harder to control.

What actually works well together

The most practical pattern is not 'agents vs. traditional scrapers' but knowing which tool fits which job.

One-off extraction from a handful of pages: An LLM or a quick manual copy-paste is often faster than setting up any tool.
Recurring structured extraction from many pages: DOM-based tools are more consistent and cheaper at volume.
Unstructured content that needs interpretation: LLMs add real value here because the task requires reasoning, not just reading.
Non-technical users who need recurring datasets: A no-code tool with automatic field detection and scheduling handles this without engineering involvement.

Minexa.ai covers the second and fourth cases. It detects page structure automatically, handles all common pagination types without configuration, supports scheduled runs, and exports to Excel, Google Sheets, or JSON. For teams that need data from many pages on a regular basis, that is a more reliable foundation than an agent making field-level judgment calls on every run.

The honest summary

AI agents have made scraping more accessible and reduced the engineering overhead for certain workflows. That is real progress. But the framing that agents simply replace structured extraction tools is not accurate, and teams that have shipped both in production know the difference.

Accuracy at scale, predictable costs, and consistent output are not marketing claims. They are the actual requirements of a production data pipeline. Choose your tooling based on those requirements, not on which approach has more momentum in the current hype cycle.

See how Minexa.ai handles structured extraction: minexa.ai

For more on what LLM-based extraction actually costs at volume, this breakdown is worth reading: Why using AI to collect web data at scale costs more than you think

Scraping real estate data: what actually works and where most pipelines break

Minexa.ai — Fri, 26 Jun 2026 17:31:00 +0000

Real estate data is publicly visible on dozens of platforms. Prices, listing dates, square footage, agent details, location components — all sitting right there on the page. Yet collecting it reliably at any meaningful scale is genuinely difficult, and most pipelines that start clean eventually break.

Here is what actually goes wrong, and what a working setup looks like.

The real problems with real estate scraping

Real estate sites are among the most scraper-hostile categories on the web. Heavy JavaScript rendering, aggressive anti-bot layers, geo-targeted content, and frequent layout changes all compound into maintenance work that never really stops.

Beyond the infrastructure problems, there is a subtler issue: field consistency. A property listing page often contains multiple address components, multiple price figures (asking price, last sale price, estimated value), and multiple date fields. When you build a selector-based scraper, you map each field manually. When a site updates its layout, those mappings silently break and return wrong values or nothing at all.

LLM-based extraction sounds like a fix but introduces a different problem. A language model parsing a listing page may swap asking price and last sale price because both look like prices and context is ambiguous. It may merge street address, city, and postcode into a single string instead of preserving them as separate fields. These errors do not always produce an exception. They produce plausible-looking data that requires validation downstream.

At a few hundred pages that validation is manageable. At tens of thousands of pages per month, it becomes a significant and ongoing cost.

What deterministic extraction changes

Minexa.ai is a web extraction platform that takes a different approach. Instead of writing selectors or passing HTML to a language model, you train a scraper once using a browser extension, and Minexa binds each data field to its exact DOM position. That binding is stable across all structurally similar pages.

For real estate specifically, this matters because:

Asking price and last sale price always come from their respective DOM elements, not inferred from context
Address components are extracted as separate columns, not merged
Missing values return null, never a fabricated fallback
The same scraper runs identically on page 1 and page 50,000

Install the Minexa Chrome extension and train your first real estate scraper in under five minutes.

How the workflow looks in practice

You open a listing page in Chrome, select the HTML container holding the data block you want, and click 'Create Scraper'. Minexa analyzes the structure and automatically discovers all data points within that container. No schema definition required. The whole process takes two to five minutes.

Once the scraper exists, you get a scraper_id. From that point, extraction runs through the API:

import requests

url = "https://api.minexa.ai/data/"
headers = {"Content-Type": "application/json", "api-key": "YOUR_API_KEY"}

data = {
  "batches": [{
    "scraper_id": 6241,
    "columns": ["top_30"],
    "urls": [
      "https://example-realestate.com/listing/101",
      "https://example-realestate.com/listing/102"
    ],
    "scraping": {
      "js_render": True,
      "proxy": "verified",
      "retry": 3
    }
  }],
  "threads": 5
}

response = requests.post(url, json=data, headers=headers)
print(response.json())

The columns: ["top_30"] parameter tells Minexa to return the thirty highest-ranked data points from the scraper. You can also pass explicit column names once you know which fields you need. Both approaches cost the same and return the same underlying data.

When the site changes layout

Minexa fails loudly, not silently. If a listing site updates its page structure and the trained scraper no longer matches, affected fields return null or an explicit error rather than a wrong value. You open the updated page in the extension, select the new container, and create a new scraper. Same two-to-five minute process. The only required code change is updating scraper_id in your request body.

This is a meaningful operational difference from selector-based scrapers, which can quietly match the wrong element for days before anyone notices the data is wrong.

A note on scale and cost

For real estate pipelines running tens of thousands of pages per month, the cost structure of LLM-based extraction becomes a real constraint. A full HTML property page can run to hundreds of thousands of tokens. At that size, even the cheapest available models cost significantly more per page than a flat-rate extraction platform. Minexa's pricing is per page, not per token, so page size does not affect cost.

For teams already collecting real estate data and hitting reliability or cost walls, the Minexa API docs cover the full request structure, scraping configuration options, and how to handle dynamic content.

If you are building or maintaining a real estate data pipeline, the extraction layer is worth getting right once rather than patching repeatedly.

Minexa cost breakdown: when LLM-based extraction stops making financial sense

Minexa.ai — Fri, 19 Jun 2026 09:25:58 +0000

Most developers building data pipelines start with an LLM. It feels like the obvious move: pass the HTML, describe what you want, get structured JSON back. It works fine at low volume. Then the bills arrive.

This article breaks down exactly where the math stops working, using real token counts from real pages across six content categories.

The token problem nobody talks about upfront

When you pass a page to an LLM for extraction, you are paying for every token in that HTML. The average full HTML page, with whitespace cleaned but otherwise intact, runs around 572,000 tokens. That is not a worst-case number. That is the average across job listings, ecommerce pages, property results, review pages, search results, and hotel booking pages.

You have two options:

Strip the HTML down to DOM tags and text only, which brings the average to roughly 39,000 tokens. Cheaper, but you risk removing markup that contains the data you need, and it requires preprocessing logic you now have to maintain.
Pass full HTML and pay for every token. Safe and low-maintenance, but the cost per page becomes significant fast.

There is no clean middle ground. A context cap saves money but silently truncates pages mid-content with no error signal.

What the numbers actually look like

Here is the cost to process 120,000 pages per month using stripped HTML (39k tokens/page):

Model	Cost
GPT-5 nano	$285
GPT-4o-mini	$773
GPT-5 mini	$1,410
Claude Haiku 4.5	$5,280
Claude Sonnet 4.6	$15,820
Minexa Startup	$60

Switch to full HTML and every figure above scales by roughly 15x. GPT-5 nano goes from $285 to $3,480. Claude Sonnet 4.6 goes from $15,820 to $207,980. Minexa stays at $60 because its pricing is per page, not per token.

The only range where LLMs are competitive

Below roughly 10,000 pages per month on stripped HTML, the cheapest nano-class models (GPT-5 nano, GPT-4.1 nano, Gemini Flash Lite) land between $24 and $43. Minexa Personal is $15 as a flat monthly floor, so even here Minexa is cheaper. But the gap is small enough that tooling familiarity might reasonably win the decision.

Beyond that threshold, the gap widens fast and does not close.

Try Minexa free and train your first scraper in under 10 minutes

The costs that do not show up in token pricing

Token cost is only part of the picture. LLM extraction pipelines carry indirect costs that compound at volume:

Validation overhead. LLMs can return plausible but wrong values without any error signal. A job listing with salary, equity, and bonus in similar formats might come back with equity assigned to the salary field. At 100,000 pages, that translates to thousands of rows requiring downstream validation logic or manual review.

Retry logic. Inconsistent JSON field naming across responses, occasional fabricated values, and schema drift all require retry and normalization code. That is engineering time spent on infrastructure rather than the actual product.

Prompt maintenance. When a target site updates its layout, LLM prompts may need adjustment to keep extraction accurate. This is not always obvious until data quality degrades silently.

Minexa's DOM-based extraction returns null when a field is absent and raises an explicit error when a page does not match the trained scraper. There is no silent failure mode.

Three real scenarios

Ecommerce price monitoring (~80,000 pages/month). Using GPT-5 nano on stripped HTML with a 20% retry overhead: approximately $230/month. After training a single Minexa scraper on the product page structure: $60/month on the Startup plan. Price fields always pulled from the correct DOM element regardless of page layout updates.

Real estate listings (~200,000 pages/month, full HTML). GPT-5 mini on full HTML: over $29,000/month. Minexa Business plan: $500/month. The LLM had been occasionally swapping asking price and last sale price; Minexa eliminated the issue by binding each column to its specific DOM element.

Lead generation from directories (~50,000 pages/month). Mistral Small 2 on stripped HTML: approximately $485/month. Minexa Startup: $60/month. Inconsistent JSON field naming across LLM responses required a normalization step that was eliminated entirely after switching.

How Minexa's pricing works

Minexa charges per page extracted, not per token. Page size does not affect credit consumption. A 600,000-token page costs the same credit as a 20,000-token page.

The three plans:

Personal ($15/month): 10,000 credits, 3 threads
Startup ($60/month): 120,000 credits, 10 threads
Business ($500/month): 2,000,000 credits, 100 threads

Note: credit consumption can be higher for pages requiring JavaScript rendering or aggressive anti-bot handling. The baseline figures above apply to standard pages.

Getting started

Training a scraper takes 2 to 5 minutes via the Minexa Chrome extension. You hover over the HTML container holding your target data, confirm the selection, and Minexa generates a reusable scraper with a stable scraper_id. From there, extraction runs through the API with a straightforward POST request:

{
  "batches": [{
    "scraper_id": 6241,
    "columns": ["top_30"],
    "urls": ["https://example.com/listing/1"],
    "scraping": {"js_render": true, "proxy": "verified"}
  }],
  "threads": 5
}

The extension generates this code for you after training. You update the URLs and run it.

If you are currently running an LLM extraction pipeline above 10,000 pages per month, the full API documentation has everything needed to evaluate a migration.

Scraping app store listings at scale with the Minexa API

Minexa.ai — Fri, 19 Jun 2026 09:17:44 +0000

App store pages are structured consistently. Every listing has a title, a rating, a review count, a description, a category, a developer name, and often a price or in-app purchase flag. For app analytics platforms, that consistency is exactly what makes them worth scraping at scale.

The challenge is volume. Tracking thousands of apps across multiple store pages, refreshing data regularly, and keeping fields aligned across runs is not something you want to rebuild every time a page shifts. This is where Minexa API fits in.

How the workflow starts: train once in the extension

Before you write a single API call, you train a scraper using the Minexa Chrome extension. Browse to an app listing detail page, let Minexa detect the structure automatically, confirm the fields, and save the scraper. That scraper gets a stable scraper_id you will reuse in every API call going forward.

This is the core developer pattern with Minexa: visual setup in the browser, programmatic execution via API.

Once trained, the scraper works on any app listing page that shares the same structure. You do not retrain for every app. One scraper, thousands of pages.

Making the API call

The endpoint for extracting data is https://api.minexa.ai/data. You send a POST request with your scraper_id, the list of URLs you want to process, and the columns you want back.

Here is a Python example for detail-mode extraction across a batch of app listing pages:

import requests

API_KEY = "your_api_key_here"

app_urls = [
    "https://appstorehub.com/app/4821",
    "https://appstorehub.com/app/4822",
    "https://appstorehub.com/app/4823"
]

payload = {
    "scraper_id": 6374,
    "urls": app_urls,
    "columns": [
        "app_name",
        "developer",
        "rating",
        "review_count",
        "category",
        "description",
        "price"
    ]
}

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

response = requests.post(
    "https://api.minexa.ai/data",
    json=payload,
    headers=headers
)

print(response.json())

If you are not sure which column names are available, you can use "columns": "top_40" instead of listing them manually. Minexa will return the top 40 ranked data points it found on the page.

Handling paginated responses

When you submit a large batch of URLs, the API returns results in pages. Each response includes a next_token field. If it is present, there are more results to fetch. Here is a checkpoint-based loop to collect everything:

import json

all_results = []
next_token = None

while True:
    if next_token:
        payload["next_token"] = next_token

    response = requests.post(
        "https://api.minexa.ai/data",
        json=payload,
        headers=headers
    ).json()

    all_results.extend(response.get("data", []))
    next_token = response.get("next_token")

    with open("checkpoint.json", "w") as f:
        json.dump(all_results, f)

    if not next_token:
        break

print(f"Total records collected: {len(all_results)}")

Saving a checkpoint after each iteration means a network interruption does not cost you the work already done.

Credit consumption on app store pages

App store listing pages often load ratings, reviews, and media assets dynamically via JavaScript. Pages with heavy dynamic content or anti-bot protection may consume more than one credit per page. Plan your batch sizes accordingly.

Why not just use an LLM for this?

App listing pages contain multiple similar numeric fields: aggregate rating, rating count, number of reviews per version, and sometimes a separate score for the current version. An LLM reading the raw HTML has to decide which number maps to which field. It does not always get this right, and it does not always signal when it is uncertain.

Minexa binds each column to a specific position in the DOM. The same field returns the same value every run, regardless of what else is on the page. If a value is missing, the output is null, not a guess.

For an analytics platform ingesting data from thousands of apps on a recurring basis, that determinism matters more than flexibility.

Get started: Read the full API docs and train your first app listing scraper in the extension before writing any code.

Scheduling your runs

The API itself does not manage scheduling. If you need to refresh app data daily or weekly, set up a cron job on your end and pass the updated URL list to the API on each run. This gives you full control over timing, batching, and retry logic within your own infrastructure.

For smaller, fixed lists of app URLs, the Chrome extension's built-in scheduling is simpler to configure. For larger or dynamic URL sets, the cron-plus-API approach scales better.

What app analytics platforms actually get out of this

With a trained scraper and a few dozen lines of Python, an analytics platform can maintain a structured, refreshable dataset of app listings covering name, developer, category, rating, review volume, pricing model, and description text. That data feeds directly into trend analysis, competitive benchmarking, category ranking models, and review sentiment pipelines.

The extraction setup is done once. After that, running it again on a new batch of URLs takes the same amount of engineering effort as the first run.

Explore Minexa API docs to see the full request schema and response format before you build.

Scraping environmental data from OpenEI with Minexa.ai

Minexa.ai — Fri, 19 Jun 2026 09:09:56 +0000

OpenEI (Open Energy Information) is a platform maintained by the U.S. Department of Energy that hosts a large catalog of energy-related datasets. The search page at data.openei.org/search lists datasets across topics like solar resources, utility rates, building energy use, and grid data. Each listing includes a title, organization, tags, license type, and a link to the full dataset record.

If you need to collect this data at scale — to build a dataset index, track what gets published over time, or feed an internal research tool — copying it manually is not realistic. This is where the Minexa.ai Chrome extension comes in.

Watch the full walkthrough first

Before going through the screenshots below, the video tutorial covers the entire process end to end. It is the fastest way to understand how the extraction works on OpenEI.

Watch full video demo

How the extraction works, stage by stage

Rather than listing steps in isolation, here is what each stage of the process actually does and what you see on screen.

Starting point: the Minexa extension

Once the extension is installed, opening it from any page brings up the Minexa home screen. This is where all your scrapers and jobs are managed.

The extension works directly in your browser — no separate app, no dashboard to log into from another tab.

Navigating to the target page

Browse to data.openei.org/search. This is the dataset search listing page. You can see all the dataset cards that Minexa will detect and extract from.

Minexa works on the page currently open in your browser, so there is no URL to paste into a separate interface. You are already on the right page.

Confirming the page

After opening the extension popup, you click 'I'm on the right page'. This tells Minexa to begin analyzing the current page structure.

From this point, Minexa takes over the detection process automatically.

Pagination detection

Minexa scans the page and identifies how it paginates. For OpenEI, it detects the next page mechanism and shows you a list of the pagination it found. You review it and click Continue.

You do not configure this manually. Minexa reads the page structure and figures out the pagination pattern on its own.

Choosing your scraping depth

After pagination is confirmed, Minexa asks whether you want to scrape just the list page or also follow each dataset link and extract detail page data. For most research use cases, list-only is sufficient. For deeper extraction, the detail mode pulls additional fields from each individual dataset record page.

This two-layer extraction capability means a single job can produce both the summary data from the list and the full metadata from each dataset page.

Simple or advanced mode

Before the job starts, you choose between simple mode (Minexa picks the most relevant fields automatically) and advanced mode (you can review and adjust the field selection). For most users, simple mode produces a clean, complete output without any additional configuration.

Container detection

Minexa automatically highlights the repeating container on the page — the element that wraps each dataset listing. This is the structural anchor it uses to identify where one result ends and the next begins.

Field discovery

After detecting the container, Minexa surfaces all the data points it found within each listing. These appear as labeled columns — title, organization, tags, license, and more. You do not need to specify these upfront.

This is one of the more useful aspects of the tool: if you are not sure what fields are available on a page, Minexa shows you rather than asking you to define them first.

API and code samples

At the configuration stage, Minexa also surfaces ready-to-use code samples in JSON and Python, along with an API request view. This is useful if you want to integrate the scraper into an existing pipeline.

Job summary with scheduling and Google Sheets options

Before running, you see a summary screen. From here you can connect a Google Sheet for live output or set up a recurring schedule so the job runs automatically without manual triggering.

Scheduling is particularly relevant for OpenEI since new datasets are added regularly. A weekly scheduled run keeps your local dataset index current without any manual work.

Running the job

The scraper appears in your jobs list with a Run button. Once triggered, extraction begins across all detected pages.

Results during and after the run

As the job runs, data populates in a table view in real time. Once complete, you can export to Excel or JSON.

What the extracted data looks like

Here is a sample of what the JSON output contains after a completed run on the OpenEI search page:

[
  {
    "title": "U.S. Solar Resource Data",
    "organization": "National Renewable Energy Laboratory",
    "tags": "solar, irradiance, GHI, DNI",
    "license": "Public Domain"
  },
  {
    "title": "Utility Rate Database",
    "organization": "NREL",
    "tags": "electricity rates, tariffs, utilities",
    "license": "Creative Commons"
  }
]

Each row corresponds to one dataset listing. Fields are clean and consistently named across all pages.

Working with the exported data in Python

import json

with open('openei_datasets.json', 'r') as f:
    datasets = json.load(f)

for dataset in datasets:
    print(dataset.get('title'), '|', dataset.get('organization'))

This gives you a quick way to scan titles and organizations, or pipe the data into a pandas DataFrame for further filtering and analysis.

The scraper configuration is saved after the first run. The next time you trigger it, Minexa skips the detection phase entirely and goes straight to extraction. If you want to get started, the extension is available at minexa.ai.

What actually happens when Minexa extracts data from a page

Minexa.ai — Tue, 16 Jun 2026 10:21:37 +0000

If you have looked at Minexa before and wondered what is actually going on under the hood when it extracts data, this article walks through the internal mechanics in plain terms. Not the marketing pitch. The actual behavior.

What does 'container locking' mean?

Before collecting any values, Minexa identifies and locks onto the exact section of a page that holds the target data. This prevents it from accidentally pulling values from visually similar but unrelated sections, like a sidebar showing related products with their own prices, or a footer with duplicate navigation text.

The user selects a parent HTML element in the browser extension, not individual fields. Minexa works from that container outward, discovering columns within it rather than scanning the whole page.

How does it pick which selector to use for each field?

For each discovered data point, Minexa evaluates multiple candidate selectors and ranks them by structural stability and content regularity across pages. The final selector is the one that consistently targets the correct element even when minor layout differences exist between pages on the same site.

This is why a scraper trained on one product page works correctly across thousands of structurally similar product pages without modification.

What is the top_n columns parameter?

When you make an API request, you do not have to know the column names upfront. You can use "top_30" or "top_10" or any value, and Minexa returns that many columns ranked by its relevance algorithm.

{
  "batches": [{
    "scraper_id": 4712,
    "columns": ["top_30"],
    "urls": ["https://example.com/product/99"],
    "scraping": {"js_render": true, "proxy": "verified"}
  }],
  "threads": 5
}

The ranking is deterministic, so "top_30" always maps to the same set of columns in the same order. Once you know which columns matter, you can switch to named fields like ["price", "availability", "brand"] with no other changes needed.

What does nested data look like in the output?

When extracted content maps to multiple elements, Minexa returns a list of objects instead of a flat string. Each object includes a tag, type, attribute, and value field.

{"study_locations": [
  {"tag": "span", "type": "text", "value": "Austin", "attribute": null},
  {"tag": "span", "type": "text", "value": "Denver", "attribute": null}
]}

In most cases you only need the value. In Python:

locations = [item["value"] for item in row["study_locations"]]

For deeply structured content like article body text, the tag and attribute metadata let you filter and reconstruct the original content precisely. This does require some extra handling compared to flat columns, which is a real tradeoff worth knowing about before you start.

Is extraction deterministic?

Yes. Running the same scraper on the same page always produces identical JSON output as long as the underlying HTML has not changed. This differs from LLM-based extraction where outputs can vary between runs due to temperature settings, prompt drift, or model updates.

For testing and validation pipelines, this matters a lot. You can run the same page twice and diff the outputs with confidence.

What happens when something goes wrong?

Minexa is designed to fail loudly. If a page structure changes and the scraper no longer matches the HTML, affected fields return null or the scraper raises an explicit error. It never silently returns a wrong value.

If you accidentally submit a URL with the wrong scraper_id (for example, a category page when the scraper was trained on a detail page), Minexa returns an error flagging the mismatch rather than attempting extraction on mismatched content.

This is meaningfully different from selector-based scrapers that can quietly match the wrong element, and from LLM pipelines that may return a plausible-looking but fabricated value with no error signal at all.

What if the site redesigns?

When a site changes its layout substantially, the scraper will start returning errors or null values. That is the signal to retrain. You open an affected page in the extension, select the updated container, and create a new scraper. It takes the same 2 to 5 minutes as the original setup.

Retraining creates a new scraper with a new scraper_id. Column names may also change since the scraper is generated fresh. The only required code update is swapping the scraper_id in your request body and verifying the column names you rely on.

Can you skip live crawling if you already have the HTML?

Yes. If you have pages stored as HTML files on something like AWS CloudFront or a public URL, you can pass them via file_urls and set js_render to false. This is the cheapest scraping configuration since no live fetching or rendering is needed.

{
  "scraping": {"js_render": false, "proxy": "verified"},
  "file_urls": [
    "https://9343.cloudfront.net/html-page-1.html"
  ],
  "urls": [
    "https://original-site.com/page-1"
  ]
}

The urls field here holds the original source URLs so extracted data can be mapped back to the real page it came from. The two arrays are 1-to-1 by index.

Where to go from here

The Minexa Chrome extension is the fastest way to train your first scraper and get the pre-generated Python code. The extension also has a drop-down of ready-made scraping scenarios you can copy directly, which saves time compared to reading through the full API docs.

Full API reference is at minexa.stoplight.io/docs/minexa if you want to go deeper on request parameters and scraping configurations.

Nested data, null fields, and the quiet failures nobody talks about in web scraping

Minexa.ai — Sun, 14 Jun 2026 16:01:25 +0000

Most scraping bugs are not crashes. They are wrong values that look right.

A price field returns a number. The number is plausible. It passes your type check. It lands in your database. Three weeks later someone notices the sale price and the original price have been swapped on roughly 8% of records. No error was raised. No log entry flagged it. The pipeline just quietly extracted the wrong element.

This is the failure mode that actually costs time, and it shows up in three distinct places.

The selector drift problem

CSS selectors and XPath expressions are written against a snapshot of a page. When a site updates its layout, the selector either breaks visibly (returns nothing) or drifts silently (matches a different element that happens to exist at the same path). The second case is worse. A selector targeting .price-now that starts matching .price-was after a redesign will not throw an exception. It will just return the wrong number, consistently, at scale.

Traditional scraping gives you no structural guarantee. You write the selector, you hope the site does not change, and you build monitoring on top to catch drift after the fact.

The LLM ambiguity problem

LLM-based extraction has a different failure signature. On pages with multiple visually similar fields, like a job listing with a salary range, an equity range, and a bonus figure, the model picks based on proximity and pattern rather than structural position. It is usually right. At 100,000 pages, 'usually right' means thousands of incorrectly attributed rows, with no error signal attached.

The hallucination that is hardest to catch is not fabrication. It is field swapping: the correct value extracted into the wrong column. Schema conformance failures are also common. If a price is unavailable, some models return 0 or a nearby value rather than null. Both pass downstream validation.

The nested data problem

This one is underappreciated. Many pages contain fields that are not flat strings. A clinical trials page might have four separate date fields rendered as sibling span elements. A property listing might have address components spread across multiple tags.

When Minexa extracts nested content, it returns a list of objects rather than a collapsed string:

{
  "study_locations": [
    {"tag": "span", "type": "text", "value": "Akishima"},
    {"tag": "span", "type": "text", "value": "Atsugi"}
  ]
}

To get the values in Python:

values = [item["value"] for item in row["study_locations"]]

The tag and attribute metadata lets you filter when multiple object types are present. This is a real tradeoff: nested fields require more handling than flat columns, but the structure is explicit and accurate rather than collapsed and potentially wrong.

Check the Minexa API docs for the full parameter reference

How Minexa handles this structurally

Minexa is a deterministic, DOM-based extraction platform. You train a scraper once using the Chrome extension by selecting the HTML container that holds your target data block. Minexa locks onto that container and discovers all data points within it automatically.

Each column is bound to a specific DOM element via a consolidated selector chosen for structural stability. Running the same scraper on the same page always returns identical JSON. No temperature variance, no prompt sensitivity.

When a field is absent from the HTML, the output is null. Never a fabricated default.

When a URL is submitted with the wrong scraper_id, the API returns an explicit mismatch error rather than attempting extraction on the wrong structure.

A minimal API call looks like this:

{
  "batches": [{
    "scraper_id": 6241,
    "columns": ["top_30"],
    "urls": ["https://example.com/listing/99"],
    "scraping": {"js_render": true, "proxy": "verified"}
  }],
  "threads": 5
}

The columns parameter accepts either top_N for automatic ranked field selection or explicit column names generated during training. Both cost the same.

When a site redesigns and the scraper starts returning nulls or explicit errors, you retrain in 2 to 5 minutes via the extension. The only required code change is updating the scraper_id.

The fail-loudly design matters more than it sounds

Selector-based scrapers fail silently. LLMs fail silently. Both produce outputs that look valid and are not. Minexa is designed to surface structural problems as explicit errors rather than letting wrong data propagate.

This changes what your validation layer needs to do. Instead of checking whether extracted values are plausible, you can trust that a non-null value came from the correct DOM position. Your checks shift from 'does this look right' to 'is this field present'.

At scale, that is a meaningful reduction in downstream cleanup work.

Start with the extension, get your first dataset in under 10 minutes

Scraping scholarship data from Scholarship America with Minexa.ai

Minexa.ai — Sun, 14 Jun 2026 15:59:28 +0000

Scholarship America maintains one of the largest publicly browsable scholarship databases in the US. The browse page at scholarshipamerica.org lists hundreds of programs with award amounts, deadlines, eligible institutions, and geographic scope. If you are building a scholarship aggregator, doing financial aid research, or just trying to track what programs are open and when, that data is useful but locked inside paginated HTML.

This article walks through how to pull it out using the Minexa.ai Chrome extension without writing any scraping code.

What the page looks like

The browse page at scholarshipamerica.org/students/browse-scholarships/ renders a paginated list. Each entry includes a program name, award amount, deadline, eligible institution types, and a location tag. The pagination uses a URL pattern (?_paged=2, ?_paged=3, etc.), which Minexa detects automatically.

Opening Minexa on the page

Once the extension is installed and you are on the browse page, click the Minexa icon. The popup asks you to confirm you are on the right page.

After confirming, Minexa scans the page and surfaces the pagination it found.

No configuration needed here. Hit Continue.

Choosing scraping depth

This is a decision point most scraping tools skip entirely. Minexa asks whether you want to scrape the list only, or the list plus the detail page behind each link.

For most research use cases, the list data is sufficient. If you need full program descriptions or eligibility criteria from each individual scholarship page, the detail option handles that in the same run.

Automatic container and field detection

Minexa highlights the repeating container it identified. You do not click on individual fields. It finds all data points inside the container on its own and presents them for review.

This is useful when you are not sure what fields are available. You see everything Minexa found before committing to a run.

Video walkthrough

Watch full video walkthrough

Sample output

Here is a cleaned sample from the extracted JSON (meta fields and field prefixes removed):

[
  {
    "scholarship_program_name": "#RAREis Scholarship Fund",
    "award_amount": "$5,000",
    "event_date_time": "April 28, 2026 3:00 PM CT",
    "eligible_institutions": "Community or Technical College, Four-Year University, Graduate School",
    "location": "National",
    "status": "Closed",
    "program_link": "https://scholarshipamerica.org/scholarship/rareis/"
  },
  {
    "scholarship_program_name": "Amazon Future Engineer Scholarship",
    "award_amount": "Up to $40,000",
    "event_date_time": "January 26, 2026 3:00 PM CT",
    "eligible_institutions": "Community or Technical College, Four-Year University",
    "location": "National",
    "status": "Closed",
    "program_link": "https://scholarshipamerica.org/scholarship/amazonfutureengineer/"
  }
]

Each row is one scholarship. Fields map directly to what is on the page.

Working with the data in Python

import json

with open('scholarships.json', 'r') as f:
    data = json.load(f)

for item in data:
    name = item.get('scholarship_program_name', '')
    amount = item.get('award_amount', 'TBD')
    deadline = item.get('event_date_time', '')
    location = item.get('location', '')
    print(f"{name} | {amount} | {deadline} | {location}")

The export from Minexa is already structured, so there is no parsing or cleanup step before this runs.

Scheduling and export options

Once the scraper is configured, you can schedule it to run on a recurring basis and push results directly to Google Sheets. For a scholarship database that updates seasonally, this means your dataset stays current without manual re-runs.

Export to Excel, JSON, or Google Sheets directly from the results view.

If you want to try this on Scholarship America or any other paginated listing site, the Minexa.ai extension is available at the Chrome Web Store.

Stop reaching for an LLM every time you need to extract web data

Minexa.ai — Sun, 14 Jun 2026 15:54:43 +0000

There is a pattern that has become surprisingly common in backend and data engineering work: someone needs structured data from a website, reaches for an LLM API, feeds it raw HTML, and calls it done. It works well enough at small scale. Then the bill arrives, or the extracted fields start drifting, and the whole thing needs rethinking.

Using an LLM to parse HTML is not wrong by default. But it is often the wrong tool chosen for the wrong reason — convenience at prototype stage, not fitness for production.

What actually goes wrong with LLM extraction at scale

The issues are not dramatic. They are quiet and cumulative.

A product page shows a sale price and a crossed-out original price. Both look like prices to a language model. Depending on surrounding text, token context, and model temperature, the model may return either one under sale_price. It will not tell you it is uncertain. You find out during a data audit three weeks later.

A clinical trials page has four date fields. An LLM assigns the wrong date to a label roughly once per hundred rows or more, because the values are structurally similar and the model picks based on proximity rather than DOM position. At 50,000 pages, that is hundreds of silently wrong rows.

This is not a criticism of LLMs as a category. It is a description of what probabilistic text generation does when applied to a task that requires structural precision.

The alternative: train once, extract deterministically

The Minexa.ai API takes a different approach. You train a scraper once using the browser extension — point at the HTML container holding your data, confirm the selection, and Minexa generates a reusable scraper with a stable scraper_id. That scraper is backed by consolidated DOM selectors, not prompts. Every field maps to a specific element. Running the same scraper on the same page always produces identical JSON.

Once trained, you call the API with your URLs:

data = {
  "batches": [{
    "scraper_id": 4821,
    "columns": ["top_30"],
    "urls": ["https://example.com/listing/9981"],
    "scraping": {
      "js_render": True,
      "proxy": "verified",
      "timeout": 30,
      "js_code": [
        {"wait_time": 2},
        {"page_init": True},
        {"wait_time": 4}
      ],
      "retry": 3
    }
  }],
  "threads": 5
}

The columns parameter accepts either a named list like ["price", "availability", "brand"] or a top_n shorthand. Using "top_30" returns the 30 highest-ranked columns by Minexa's relevance algorithm. The ranking is deterministic, so top_30 always maps to the same 30 fields — safe for production without locking in a schema upfront.

Up to 50,000 URLs can go into a single batch request. The threads value controls parallel processing up to your plan's limit.

When a field is missing, you get null — not a guess

Minexa is designed to fail loudly. If a page structure changes and a trained selector no longer finds its target, the affected field returns null or an explicit error. It never borrows a value from a nearby element or invents a plausible substitute.

This contrasts directly with LLM behavior, where a missing price might come back as $0.00 or a missing date might be filled from another date field on the page — both plausible, both wrong, neither flagged.

If you submit a URL with the wrong scraper_id (a page type the scraper was not trained on), Minexa returns an error indicating the mismatch rather than attempting extraction on mismatched structure.

Handling pre-scraped HTML

If you already have HTML stored (on CloudFront, S3, or anywhere publicly accessible), you can skip live crawling entirely using file_urls:

{
  "scraping": {"js_render": false, "proxy": "verified"},
  "file_urls": [
    "https://your-cdn.cloudfront.net/page-1.html"
  ],
  "urls": [
    "https://original-site.com/listing/1"
  ]
}

file_urls and urls are 1-to-1. Minexa reads from the stored HTML and maps output back to the original URL. This is the lowest-credit configuration since no rendering is needed.

The cost reality at scale

LLMs price by token. A full DOM-rendered HTML page averages around 572,000 tokens. At that size, GPT-4o-mini costs roughly $0.086 per page. At 120,000 pages per month, that is $10,320. Minexa's Startup plan handles the same volume for $60. Even with stripped HTML at 38,965 tokens, GPT-4o-mini costs $773 for 120,000 pages versus $60 on Minexa.

Minexa's cost does not change based on page size. A 600K-token page costs the same credit as a 10K-token page.

Start extracting structured data from any site in under 10 minutes: minexa.ai

Nested data and what to do with it

When extracted content is structurally nested, Minexa returns a list of objects with metadata:

{"locations": [
  {"tag": "span", "type": "text", "value": "Berlin"},
  {"tag": "span", "type": "text", "value": "Vienna"}
]}

In most cases you only need the values:

values = [item["value"] for item in row["locations"]]

The tag and attribute metadata are available when you need to filter by element type, which is useful for pages with long mixed content like article bodies.

What retraining looks like

When a site redesigns and the existing scraper starts returning nulls or errors, you open the updated page in the extension, select the new container, and create a new scraper. This takes the same 2-5 minutes as the original training. The only required code change is updating scraper_id in your request body and checking whether any column names you depend on have changed.

Retraining creates a new scraper from scratch. Column names may differ because they are generated fresh. Minexa does not attempt to preserve labels from the previous scraper.

Read the full API docs and explore scraping scenarios: minexa.stoplight.io/docs/minexa

The actual takeaway

LLM extraction is a reasonable starting point for low-volume, exploratory work where output variance is acceptable. Once you are running tens of thousands of pages per month, or once field accuracy matters for downstream use, the tradeoffs shift significantly: token costs compound, silent errors accumulate, and validation logic adds engineering overhead that was never in the original estimate.

Deterministic DOM-based extraction does not solve every problem, but for structured data at scale from consistent page types, it is the more predictable and cost-stable path.

Install the Minexa Chrome extension, train a scraper on your target page, and pull the auto-generated Python code directly from the extension: chromewebstore.google.com/detail/minexa-ai-scraper/ddljgbflolmninnkfcbdikabbjeapdnh

What actually happens when your scraping pipeline hits 100,000 pages

Minexa.ai — Sun, 14 Jun 2026 15:45:41 +0000

Most scraping projects start small. A few hundred pages, a quick script, maybe an LLM call to parse the HTML. It works fine. Then the scope grows.

At 100,000 pages per month, the decisions you made at 1,000 pages start costing real money and real engineering time. This post walks through what that scaling curve actually looks like, and where the hidden costs appear.

The token problem nobody budgets for

When you pass a full HTML page to an LLM for extraction, you are not passing a few paragraphs. Real pages across job listings, ecommerce, real estate, and review sites average around 572,739 tokens per page after DOM rendering with whitespace cleaned.

At that size, here is what a single page costs across common models:

Model	Cost per page (full HTML)
GPT-5 nano	$0.029
GPT-4o-mini	$0.086
GPT-5 mini	$0.145
Claude Haiku 4.5	$0.584
GPT-4o	$1.330
Claude Sonnet 4.6	$1.733

At 100,000 pages per month, GPT-5 nano costs roughly $2,900. GPT-4o-mini costs $8,600. Claude Sonnet 4.6 costs $173,300.

You can strip the HTML first to reduce tokens. Stripped pages average around 38,965 tokens. That brings GPT-5 nano down to $0.0024 per page, or $240 for 100,000 pages. But stripping HTML requires preprocessing work, and you risk removing markup that contains the data you actually need. There is no clean middle ground.

Flat cost regardless of page size

Minexa.ai is a Chrome extension-based scraper training tool that connects to an API for batch extraction. You train a scraper once by selecting an HTML container on a page in the extension. Minexa identifies all data points inside that container automatically. The result is a scraper_id you reference in every subsequent API call.

The extraction cost is credit-based, not token-based. A page costs the same whether it is 10,000 tokens or 600,000 tokens. At 100,000 pages per month, the Startup plan at $60 covers up to 120,000 pages. The Business plan at $500 covers up to 2,000,000.

What the API request looks like

After training a scraper in the extension, you click 'API Request' to get pre-generated Python code. The core request structure looks like this:

import requests

url = 'https://api.minexa.ai/data/'
api_key = 'YOUR_API_KEY'

data = {
  'batches': [{
    'scraper_id': 6241,
    'columns': ['top_30'],
    'urls': [
      'https://example.com/listing/1',
      'https://example.com/listing/2'
    ],
    'scraping': {
      'js_render': True,
      'timeout': 30,
      'js_code': [
        {'wait_time': 2},
        {'page_init': True},
        {'wait_time': 4}
      ],
      'proxy': 'verified',
      'retry': 3
    }
  }],
  'threads': 8
}

headers = {'Content-Type': 'application/json', 'api-key': api_key}
response = requests.post(url, json=data, headers=headers)
print(response.json())

The columns parameter accepts top_N notation to return the top N ranked fields automatically, or explicit column names if you want specific fields only. The threads parameter controls how many URLs are processed in parallel, up to your plan limit. Up to 50,000 URLs can be submitted in a single batch request.

If you already have HTML stored on S3 or CloudFront, you can pass file_urls pointing to those files and set js_render: false. This is the cheapest scraping configuration since no live crawling is needed.

What happens when a site redesigns

This is where most scraping pipelines require the most maintenance. With selector-based scrapers, a layout change can silently return wrong values for weeks before anyone notices.

With Minexa, a structural mismatch produces an explicit error or null values rather than a plausible-looking wrong value. When that happens, you open the updated page in the extension, select the new container, and create a new scraper. This takes the same 2 to 5 minutes as the original setup.

The only required code change is updating the scraper_id in your request body and checking whether any column names you rely on have shifted.

Try it yourself: Install the Minexa Chrome extension and train your first scraper in one session.

The reliability cost that does not show up in token pricing

LLM extraction pipelines at scale require validation logic. Field mapping errors, swapped values between visually similar fields, and fabricated defaults when a value is missing all produce rows that look correct but are not.

Minexa binds each column to a specific DOM element identified during training. The same field always maps to the same element. If the element is absent, the output is null. No value is invented.

At 100,000 pages, even a 1% error rate means 1,000 rows requiring manual review or retry logic. That indirect cost does not appear in any token pricing table.

Practical takeaway

If your extraction volume is below roughly 10,000 pages per month and you are already using stripped HTML, the cheapest LLM models are competitive on price. Above that threshold, or any time you are working with full HTML, the cost gap widens sharply and does not close.

The engineering overhead also compounds at scale. Prompt maintenance, schema drift, validation pipelines, and retry logic all grow with volume. A scraper trained once in the Minexa extension and called via API does not require any of that.

Full API documentation is at minexa.stoplight.io/docs/minexa if you want to explore the complete parameter reference before starting.