DEV Community: nyx464664816-stack

Same page, two outputs: Markdown vs structured JSON for AI agents

nyx464664816-stack — Mon, 26 Oct 2026 00:57:19 +0000

I work on WebExtract, a pay-per-call API that turns a URL into data. It has two main endpoints: GET /v1/markdown returns clean Markdown, and GET /v1/extract returns the same Markdown plus fixed-schema fields (product, tables, metadata). This post runs one product page and one table page through both and shows what an agent actually gets back. Every snippet below is copied from saved API responses in the examples/compare folder, not written by hand.

One note on where the files came from: they were captured through the Base Sepolia sandbox routes (/sandbox/v1/..., paid with test USDC) on 2026-10-25 (UTC). The sandbox runs the same extraction code as the live /v1/... routes.

Page 1: a product page

The page is https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html, a practice site built for scraping. The page itself says so: "Prices and ratings here were randomly assigned and have no real meaning."

The Markdown

/v1/markdown returned 1567 characters, 261 words. The end of it looks like this:

## Product Information

| UPC | a897fe39b1053632 |
| --- | --- |
| Product Type | Books |
| Price (excl. tax) | £51.77 |
| Price (incl. tax) | £51.77 |
| Tax | £0.00 |
| Availability | In stock (22 available) |
| Number of reviews | 0 |

This is good input for a model that only needs to read the page. It is short, and the price, stock and table are all there. But look at the table. The HTML table has no header row, and a Markdown table needs one, so the first data row (UPC) ends up in the header position. A model will probably read it correctly. A program that splits on pipes and treats row one as column names will not.

The structured JSON

/v1/extract returned the same Markdown (the two strings are identical) plus extra fields. Here is a jq selection of the product object:

jq '.product | {name, price, currency, availability, rating, source}' out/product.extract.json

{
  "name": "A Light in the Attic",
  "price": 51.77,
  "currency": "GBP",
  "availability": "InStock",
  "rating": {
    "value": 3,
    "count": null
  },
  "source": "heuristic"
}

price is a number, currency is a code, and availability is a fixed value. Your code can compare or store these without any regex.

The important field is source. This page has no JSON-LD, OpenGraph product tags or microdata, so the price came from page heuristics. The response says so instead of hiding it:

"evidence": "first [class*=price] element; same amount in table row \"Price (excl. tax)\""

[
  "heuristic_price_low_confidence",
  "currency_inferred_from_symbol"
]

That is product.price_detail.prices[0].evidence and product.price_detail.warnings. The evidence tells you where the number came from (and that the product table shows the same amount). The warnings tell you the price came from heuristics and the currency was inferred from the £ symbol. If your agent is about to act on a price, check price_detail.warnings first. We keep these warnings on purpose, even when the number turns out to be right, because heuristics can be wrong on other pages.

The table comes back like this:

["col_1","col_2"]
[["UPC","a897fe39b1053632"],["Product Type","Books"],["Price (excl. tax)","£51.77"]]
{"row_count":7,"column_count":2,"consistent_width":true}

Those are tables[0].headers, the first three rows, and checks. Since the page has no header row, the headers are placeholders (col_1, col_2) and UPC stays a data row. The checks object lets you confirm the shape before using it: 7 rows, 2 columns, every row the same width.

Page 2: a table page

The second page is https://webscraper.io/test-sites/tables, a practice page with two standard HTML tables. /v1/markdown returned 508 characters, 124 words, and here the Markdown tables look fine, because the page has real <thead> rows.

The difference is what /v1/extract adds. Here is tables[0], verbatim:

{
  "index": 0,
  "caption": null,
  "headers": [
    "#",
    "First Name",
    "Last Name",
    "Username"
  ],
  "rows": [
    [
      "1",
      "Mark",
      "Otto",
      "@mdo"
    ],
    [
      "2",
      "Jacob",
      "Thornton",
      "@fat"
    ],
    [
      "3",
      "Larry",
      "the Bird",
      "@twitter"
    ]
  ],
  "checks": {
    "row_count": 3,
    "column_count": 4,
    "consistent_width": true
  }
}

In the Markdown the two tables sit one after the other with nothing between them. In the JSON they are tables[0] and tables[1], each with its own headers and checks, so you don't have to work out where one ends and the next starts.

This is not a product page, and product is simply null. Nothing is guessed. The response also explains why:

{
  "reason": "no_product_markup",
  "detail": "no JSON-LD Product/ProductGroup, product:price meta, schema.org microdata, or price+title+cart heuristic match"
}

That is the product_unresolved field. A null with a reason is easier to handle than an empty-looking object.

Metadata

Both pages also get a metadata object. A selection for the table page:

{
  "title": "Table Playground | Web Scraper Test Sites",
  "language": "en",
  "canonical_url": "https://webscraper.io/test-sites/tables",
  "type": "website"
}

The full object also has description, authors, published_at, modified_at, site_name and image. Fields the page doesn't provide are null or empty, not filled in.

Which one should an agent use?

Use /v1/markdown when a model just needs to read the page: summarize it, answer a question about it, quote it. It costs $0.002 per call.
Use /v1/extract when your code needs the fields: compare prices, load a table into a dataframe, check whether something is in stock. It costs $0.005 per call. Because it also includes the Markdown, you don't need to call both.

Some limits to know about: product comes from JSON-LD, OpenGraph, microdata or, as a last resort, page heuristics. product.rating is filled only when the page has JSON-LD or star-rating markup. The image URL is kept as the page wrote it (../../media/... here), so you may need to resolve it against final_url.

Try it yourself

On REST, the free tier is opt-in: 20 free calls per day for each caller IP address, no signup. Add the header X-Billing: free:

curl -H "X-Billing: free" "https://api.webextract.dev/v1/extract?url=https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"

or the query parameter ?free=1 if your client can't set headers:

curl -G "https://api.webextract.dev/v1/markdown" --data-urlencode "free=1" \
  --data-urlencode "url=https://webscraper.io/test-sites/tables"

The free allowance is counted per caller IP address. If your outgoing IP changes between requests (a proxy, or a cloud host behind NAT), the calls are counted separately for each address, so X-Free-Remaining can go up and down. Failed REST fetches are not charged. Without the free opt-in, a key or a payment, the endpoint answers HTTP 402 with x402 payment requirements. You can then pay per call in USDC or use prepaid credits.

The folder also has four one-file clients: compare_rest.py (both endpoints side by side), langchain_tool.py, mcp_client.py (against https://api.webextract.dev/mcp, tools page_markdown and extract_url) and x402_client.py. The x402 client only runs against the Base Sepolia sandbox with test USDC and refuses mainnet. The LangChain tools were tested by calling them directly, not inside a model-driven agent.

If you run your own pages through both endpoints and the JSON gets something wrong, I'd like to hear about it. Wrong prices and broken tables are the bugs we most want to find.

Example folder: https://github.com/nyx464664816-stack/webextract/tree/main/examples/compare
Repository: https://github.com/nyx464664816-stack/webextract
Site: https://webextract.dev/r/devto

Disclosure: I'm on the WebExtract team, and WebExtract is our own product. This post was drafted with AI assistance. The JSON and Markdown excerpts are copied from saved API responses in the repository, and I checked them against those files before publishing.

Update (2026-10-26): I ran the same two endpoints over 10 public pages and put the raw files and scripts online. In the Base Sepolia sandbox, /markdown succeeded on 38 of 40 calls and /extract on 40 of 40, and the simple field checks passed 75/75 and 212/212. It is a small sample from one client, it covers easy pages only, and I did not run any other tool. Everything, including the two failures, is here: https://webextract.dev/benchmark (code: https://github.com/nyx464664816-stack/webextract/tree/main/examples/benchmark).

Let your x402 agent read web pages: from a free test call to a paid call

nyx464664816-stack — Sun, 25 Oct 2026 19:32:11 +0000

Disclosure: I'm on the WebExtract team. This post was drafted with AI assistance, and the commands below were run by hand before publishing. WebExtract is our own product.

If you're building an agent that has to read arbitrary web pages, there are three things to settle: how to try the API without an account, how the payment handshake works, and how to pay from code. This walkthrough covers all three using an x402-enabled endpoint, WebExtract, which turns a URL into clean Markdown (and, on the other endpoint, structured JSON). It costs $0.002 per call for Markdown and $0.005 per call for the structured extract.

1. Try it without a wallet

REST is opt-in for the free tier. Add one header and you get 20 free calls a day per IP, no signup:

curl -sS -G -H "X-Billing: free" "https://api.webextract.dev/v1/markdown" \
  --data-urlencode "url=https://example.com"

You get JSON back with title, markdown, word_count and the final URL. The x-free-remaining response header tells you how many free calls are left today. Failed fetches are not charged.

2. See what the server wants for payment

Call the endpoint without the header:

curl -sS -i "https://api.webextract.dev/v1/extract?url=https%3A%2F%2Fexample.com"

You get HTTP 402. The x402 v2 payment requirements are in the PAYMENT-REQUIRED response header, base64-encoded JSON. Decode it (the base64 -d is the whole trick):

curl -sS -D - -o /dev/null "https://api.webextract.dev/v1/extract?url=https%3A%2F%2Fexample.com" \
  | grep -i '^payment-required:' | cut -d' ' -f2- | tr -d '\r\n' | base64 -d | python3 -m json.tool

When I ran it, accepts listed seven USDC options, Base first, then Polygon, Arbitrum, Avalanche, Optimism, World Chain and Monad. Every one has amount: "5000", which is $0.005 because USDC has 6 decimals. The response also carries a Link: rel="payment" header and an X-Free-Tier-Hint header pointing back at the free option, so an agent that hits the 402 first can find its way.

3. Pay from code (testnet first)

The official Node packages handle the 402, signing and retry. Use a throwaway wallet and the sandbox, which only accepts Base Sepolia test USDC (free from Circle's faucet):

npm i @x402/fetch @x402/evm viem

import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";

const account = privateKeyToAccount(process.env.TESTNET_PRIVATE_KEY); // throwaway test wallet
const pay = wrapFetchWithPaymentFromConfig(fetch, {
  schemes: [{ network: "eip155:84532", client: new ExactEvmScheme(account) }],
});
const res = await pay("https://api.webextract.dev/sandbox/v1/markdown?url=https://example.com");
console.log(res.status, (await res.json()).markdown);

Our own run of this pattern returned HTTP 200 with a settled 0.002 USDC payment on Base Sepolia (transaction).

Two things we ran into with the SDK (version 2.28):

Registering eip155:* makes it pick the first option the server lists, which is Base, wherever your USDC actually is. If your USDC is only on Polygon, register just eip155:137 or pass your own payment-requirements selector.
Optimism (eip155:10) and World Chain (eip155:480) are rejected by the SDK's default spend controls because it has no default USDC asset for them. To pay there, add spendControls: { allowedAssets: [...] } for that network.

4. Go to mainnet

Point the same code at https://api.webextract.dev/v1/markdown (drop /sandbox) and register the mainnet network where you hold USDC, for example eip155:8453 for Base. Start with a wallet that holds a few dollars, not your main one.

5. If you'd rather not run a wallet

MCP: add https://api.webextract.dev/mcp as a remote server in Cursor, VS Code or Claude Desktop (via mcp-remote). The tools are extract_url and page_markdown, and the free tier works through MCP without any header. x402 payment is not available through MCP; after the free calls run out, the tool tells you to switch to REST or credits.
Prepaid credits: $1 = 200 credits by card, then send Authorization: Bearer <key>. Details at webextract.dev.

JS-heavy pages: use /v1/render/extract ($0.008 per call) or /v1/render/markdown ($0.004 per call); each IP gets 5 free renders a day.

What I'd like feedback on

If the 402 handshake is confusing for your agent framework, tell me where it breaks: the client library, the headers, or the docs. That's the part we're still improving.