A scraping API should do more than start a scraper you built somewhere else

A common trap with scraping tools is assuming that “has an API” means “can be driven entirely from code.” Sometimes it only means the API can start a project you already built in a GUI. That difference matters once scraping becomes part of a backend job, data pipeline, agent workflow, or monitoring system.

The difference between a trigger API and a scraping API

A trigger API wraps an existing scraper. You build the scraper somewhere else, usually by clicking through pages in a desktop or web UI, then your code starts that saved project.

The API flow usually looks like this:

# Start a saved scraping project
curl -X POST "https://api.example.com/v2/projects/PROJECT_TOKEN/run" \
  -d "api_key=$API_KEY"

# Poll for results
curl "https://api.example.com/v2/runs/RUN_TOKEN/data?api_key=$API_KEY"

That can be useful. If a non-developer built a scraper once and wants it scheduled or triggered from another system, this model works fine.

But it has a hard boundary: your code does not define the extraction. It only presses run.

A scraping API should let your code describe the job at runtime:

curl -X POST "https://api.example.com/v1/scrape" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/widget-123",
    "renderJavascript": true,
    "extract": {
      "title": "product name",
      "price": "current price",
      "availability": "stock status"
    }
  }'

That distinction changes how you maintain the system. With the first model, a new URL pattern may require someone to reopen the visual builder and edit selectors. With the second, the extraction contract lives next to the code that consumes the result.

Where the trigger model breaks down

The trigger model usually fails in boring production ways.

A selector changes and the job still “succeeds,” but returns an empty CSV. A site adds a login step and the run starts timing out. A product page template changes and half the rows get shifted into the wrong columns. Your pipeline sees HTTP 200, then discovers that price is missing downstream.

The failure mode is not always an exception. Often it is bad data.

That makes observability harder. If your code only knows “run project 123,” it cannot easily explain which selector failed, which extraction field was missing, or whether the target page changed. You end up debugging in the visual tool instead of in logs, tests, and version control.

For authenticated or repeated site-specific workflows, Wire treats those actions as API-callable units rather than desktop projects, which is the model you want when job inputs, credentials, and outputs need to stay inside a pipeline.

A better contract for scraper jobs

For code-driven scraping, I want the job request to contain enough information to reproduce the run:

{
  "url": "https://example.com/products/widget-123",
  "javascript": true,
  "country": "US",
  "timeoutMs": 30000,
  "schema": {
    "title": "string",
    "price": "number",
    "currency": "string",
    "inStock": "boolean"
  }
}

And I want the response to separate transport success from extraction success:

{
  "status": "completed",
  "httpStatus": 200,
  "extractionStatus": "partial",
  "data": {
    "title": "Widget 123",
    "price": null,
    "currency": null,
    "inStock": true
  },
  "errors": [
    {
      "field": "price",
      "message": "No matching price text found"
    }
  ]
}

That gives your pipeline something useful to do. You can reject partial results, retry with a different region, alert on missing fields, or store the raw HTML for inspection.

You can also write tests against the contract:

import assert from "node:assert";

function validateProduct(row) {
  assert.equal(typeof row.title, "string");
  assert.ok(row.title.length > 0);
  assert.equal(typeof row.inStock, "boolean");

  if (row.inStock) {
    assert.equal(typeof row.price, "number");
    assert.ok(row.price > 0);
  }
}

This is much harder when the real extraction logic lives in a GUI project that your test suite cannot inspect.

Security is part of the API shape

Scraping often starts with public pages, then drifts into authenticated pages: admin dashboards, partner portals, internal tools, vendor accounts.

That is where visual scrapers can get uncomfortable. If the tool asks someone to type credentials into a recorded login step, you need to know exactly how those secrets are stored, who can view them, whether they are encrypted, and whether the API can ever return them.

At minimum, look for these properties:

• Secrets are stored outside the scraper definition
• The API can reference a credential by ID, not by raw password
• Secrets are never returned by read endpoints
• Runs produce audit logs
• Free or shared projects are not public by default

A safer request shape looks like this:

{
  "url": "https://portal.example.com/invoices",
  "auth": {
    "credentialId": "cred_vendor_portal_readonly"
  },
  "extract": {
    "invoiceNumber": "string",
    "total": "number",
    "dueDate": "date"
  }
}

The scraper gets access at run time. The pipeline never handles the password directly.

Wire is relevant here because its prebuilt actions can use vault-backed credentials without returning the secret through the API, which avoids mixing login material with extraction code.

When a visual scraper is still the right tool

Visual scrapers are not bad. They are often the right choice for a bounded task, especially when the person doing the work does not write code.

If someone needs to scrape one directory once a week, export a spreadsheet, and adjust the workflow manually when the site changes, a point-and-click builder may be faster than building an internal tool.

The tradeoff appears when the scraper becomes infrastructure. If the output feeds billing, search, analytics, alerts, model evaluation, or customer-facing features, you probably want the scrape definition, job inputs, validation, retries, and errors in code.

What to check before choosing

Before you depend on a scraping API, answer these questions with actual requests and responses:

• Can I define a new scrape entirely through the API?
• Can I pass a schema or extraction instructions at runtime?
• What does a partial extraction failure look like?
• Are JavaScript rendering, proxy region, timeout, and retries request parameters?
• Can I run the same job in CI with fixture URLs?
• Where are credentials stored, and can any endpoint read them back?
• Does the pricing model punish failed runs or repeated cached fetches?
• Are there page-per-run or worker caps that affect batch jobs?

The practical next step: take one real target page, write the API request you wish you could send, then compare tools against that contract instead of comparing feature lists.