Choosing a structured extraction API without getting trapped by the JSON

You point an extraction API at a product page and get clean JSON back. Great. Then the sixth site renders price in JavaScript, the seventh changes content by country, the eighth needs a login, and the ninth returns a valid response with the wrong fields. The hard part is not getting JSON from HTML once. The hard part is choosing an extraction model that keeps working when pages stop looking like your demo.

Start with the schema question

Most structured extraction tools fall into one of two buckets.

The first bucket gives you a fixed ontology. You send a URL, the system decides whether it is an article, product, person, organization, or something else, then returns fields from its own model. Diffbot is the clean example here. Its extraction APIs and Knowledge Graph are useful when your project fits those entity types and you want normalized public-web data.

The second bucket lets you define the fields you want per job. Anakin.io is an example of this approach: you scrape a URL and ask for structured JSON fields as part of that request.

Neither model is universally better. A fixed ontology helps when you want consistency across millions of pages. A custom schema helps when your data is specific:

{
  "productName": "string",
  "price": "number",
  "currency": "string",
  "shippingWindow": "string",
  "isMarketplaceSeller": "boolean",
  "warrantyText": "string"
}

That last field, warrantyText, is where fixed schemas often get awkward. If the extractor does not model the field, you either post-process raw text, store vendor-specific extensions, or accept missing data. Custom extraction flips the problem around: you get the field names you asked for, but you now own validation and quality checks.

Fetch behavior is part of extraction

Developers often evaluate extraction APIs by the response body. That misses half the integration.

Before an extractor can parse anything, it has to fetch the page correctly. That means handling JavaScript rendering, redirects, bot checks, cookies, language headers, and geography. A page from example-store.com might show one price from a US IP, another from Germany, and a cookie banner instead of content from a new session.

If the API gives you no control over those inputs, extraction quality becomes hard to debug. You see price: null, but the real failure happened earlier: the fetch got an interstitial, a login page, or a region-specific variant.

For authenticated scraping, Wire models login state as named identities with vault-backed credentials rather than making each scraper replay expiring cookies.

At minimum, check whether your provider exposes these fetch controls:

• JavaScript rendering on or off
• country or region selection
• custom headers
• cookie handling
• proxy behavior
• raw HTML or screenshot access for debugging

If you cannot inspect what the extractor actually saw, you will spend time guessing.

Watch how pricing behaves on failures

Monthly pricing looks simple until you add retries, failed fetches, and test runs. Diffbot's paid plans start at a higher monthly floor, which makes sense for teams buying into a Knowledge Graph or web-scale entity extraction. It makes less sense if you need structured data from a few hundred pages a day.

Per-call pricing has a different tradeoff. It usually starts cheaper and maps better to variable workloads, but you need to know exactly what counts as a billable request. Ask these questions before wiring it into a queue:

• Does a failed fetch consume credits?
• Does a timeout consume credits?
• Does JavaScript rendering cost extra?
• Does structured extraction cost extra?
• Are cached responses free or discounted?
• Are retries automatic, and are they billed separately?

A cheap request that fails 20 percent of the time may cost more than it looks. A more expensive request that only bills successful jobs may be easier to reason about. The only useful number is cost per valid row in your database.

Treat extraction as an async job

Many extraction APIs return a job ID first, then require polling. That is a good shape for real workloads because rendering and parsing can take longer than a normal HTTP request timeout.

Here is the pattern I usually want in application code: submit, poll with a deadline, validate the result, then store only typed data.

import { z } from "zod";

const Product = z.object({
  productName: z.string().min(1),
  price: z.number().nonnegative(),
  currency: z.string().length(3),
  shippingWindow: z.string().nullable(),
  isMarketplaceSeller: z.boolean().nullable(),
  warrantyText: z.string().nullable()
});

async function waitForExtraction(jobId, apiKey) {
  const deadline = Date.now() + 60_000;

  while (Date.now() < deadline) {
    const res = await fetch(`https://api.example.com/extract/${jobId}`, {
      headers: { Authorization: `Bearer ${apiKey}` }
    });

    if (!res.ok) {
      throw new Error(`poll failed: ${res.status} ${await res.text()}`);
    }

    const body = await res.json();

    if (body.status === "failed") {
      throw new Error(`extraction failed: ${body.error ?? "unknown error"}`);
    }

    if (body.status === "completed") {
      return Product.parse(body.data);
    }

    await new Promise(resolve => setTimeout(resolve, 1500));
  }

  throw new Error(`extraction timed out for job ${jobId}`);
}

The important part is not the endpoint name. It is the boundary. Do not let loosely structured AI output flow straight into your database. Validate it. Log the raw response somewhere short-lived. Track failures by domain so you can see whether one site is drifting.

Common failure modes are easy to miss:

• HTTP 200 with extracted fields set to null
• price returned as "$19.99" when your code expects a number
• currency inferred incorrectly from locale
• stale cached content after a product update
• login page extracted as if it were the target page

Schema validation catches some of this. Sampling catches the rest.

When a knowledge graph is the right tool

A Knowledge Graph is not just a scraper with a bigger database. If you need entity resolution, public-web facts, company profiles, people, relationships, and query language support across billions of entities, a graph product can save years of work.

That is the case where Diffbot makes sense. You are not paying just to parse one page. You are paying for a maintained model of the public web.

If your job is narrower, such as pulling custom fields from specific URLs, monitoring a set of pages, or feeding an internal enrichment pipeline, a page-level extraction API may fit better. You keep control over the schema and avoid buying infrastructure you do not use.

A practical evaluation checklist

Before choosing a provider, run the same test set through each option. Use at least:

• 10 normal pages
• 5 JavaScript-heavy pages
• 5 pages with regional differences
• 5 pages with missing or weird fields
• 3 pages that should fail

Measure valid rows, not successful HTTP responses. Record fetch status, extraction status, validation errors, latency, and billed units. Then calculate cost per valid row.

That small test will tell you more than a feature table. It will also show whether you need a fixed ontology, custom fields, better fetch control, or a full Knowledge Graph.