DEV Community: AgentSearch

Turn Any URL into Clean Markdown for RAG (API Guide)

AgentSearch — Fri, 02 Oct 2026 23:58:53 +0000

Raw HTML makes poor context for an LLM. A typical page is mostly navigation, scripts, cookie banners and footer links. Putting all of that into an embedding model or a prompt wastes tokens and lowers retrieval quality. What you want is the content: headings, paragraphs, lists and links, as markdown, plus enough metadata to cite the source and to tell when it goes stale.

This guide uses AgentSearch Web Extract (agentsearch-web-extract-v1) to turn a URL into RAG-ready markdown. It covers the request, the response, chunking and error handling. Each call costs $0.005, paid in USDC over x402 or MPP on the Pocket Agentic Portal, with no account and no API key.

Why markdown is the right format for RAG

It keeps the structure. Headings (#, ##) tell you where sections begin, which gives you natural chunk boundaries and section titles to store as metadata.
It's token-efficient. Markdown carries the meaning of the HTML without the tags.
Links survive. [text](url) keeps the citation trail, so an agent can follow a link or quote its source.
Models read it well. LLMs see a lot of markdown and handle it reliably in prompts.

The request

The endpoint is POST /v1/extract, which takes a JSON body. Only url is required.

curl -X POST https://agent.pocket.network/v1/agentsearch-web-extract-v1/v1/extract \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com"}'
# → 402 Payment Required with the terms. An x402 or MPP client signs and retries.

These are the options from the extract OpenAPI spec:

Field	Type	Default	Notes
`url`	string	(required)	An absolute http(s) URL
`formats`	`["markdown"]`, `["text"]` or both	`["markdown"]`	Which renderings to return
`max_chars`	integer, 1–100000	50000	Caps the returned characters. `meta.truncated` tells you whether the cap was hit
`include_links`	boolean	true	Up to 100 absolute links from the page
`timeout_ms`	integer	4000	Fetch deadline. Values above 4000 are capped (a hard 4 s deadline)

The response

This is a real response captured on the portal service page (2026-09-25 UTC):

{
  "portal": {
    "provenance": "third-party-supplier",
    "serviceId": "agentsearch-web-extract-v1",
    "schemaCheck": "unchecked"
  },
  "data": {
    "request_id": "05f5a4ebe2684c79867aa9bf3515c33d",
    "url": "https://example.com/",
    "title": "Example Domain",
    "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n",
    "text": "",
    "links": [{ "href": "https://iana.org/domains/example", "text": "Learn more" }],
    "meta": {
      "status_code": 200,
      "content_type": "text/html",
      "fetched_at": "2026-09-25T15:53:07.843243+00:00",
      "chars": 167,
      "truncated": false
    },
    "error": null
  }
}

The fields that matter for RAG:

url is the final URL after redirects. Use it as your canonical document ID.
title is a ready-made citation label.
meta.fetched_at lets you re-extract stale documents on a schedule.
meta.truncated tells you whether you hit max_chars. If you did, raise the cap and fetch again.
error is null on success.

Step 1: Fetch markdown from code (TypeScript + x402)

Standard x402 v2 clients work with the portal unchanged. Here's @x402/fetch with @x402/evm, paying on Base:

import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";

const EXTRACT =
  "https://agent.pocket.network/v1/agentsearch-web-extract-v1/v1/extract";

// Use a dedicated wallet that holds only what you're willing to spend.
const account = privateKeyToAccount(process.env.AGENT_WALLET_KEY as `0x${string}`);
const pay = wrapFetchWithPaymentFromConfig(fetch, {
  schemes: [{ network: "eip155:8453", client: new ExactEvmScheme(account) }],
});

export async function extract(url: string, maxChars = 50000) {
  const res = await pay(EXTRACT, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ url, formats: ["markdown"], max_chars: maxChars }),
  });

  if (!res.ok) {
    // Portal errors aren't wrapped in the envelope: { error: { code, message, retryable } }
    const body = await res.json().catch(() => ({}));
    throw new Error(`portal ${res.status}: ${body?.error?.code ?? "unknown"}`);
  }

  const envelope = await res.json();
  if (envelope.portal?.provenance !== "third-party-supplier") {
    throw new Error("Unexpected response shape; refusing to use it.");
  }
  const doc = envelope.data;
  if (doc.error) {
    // The target site failed, but the HTTP status was still 200.
    throw Object.assign(new Error(doc.error.code), { retryable: doc.error.retryable });
  }
  return doc as {
    url: string; title: string | null; markdown: string;
    links: { href: string; text: string }[];
    meta: { fetched_at: string; chars: number; truncated: boolean };
  };
}

If delivery fails, the portal settles nothing, so you aren't charged for it. Payment details are in Pocket's integration guide.

Prefer no code? Use MCP

In Claude Desktop, Cursor or Claude Code, add npx -y @pocket-network/agentic-portal-mcp and ask the agent to call_service on agentsearch-web-extract-v1. The full setup, including spending limits, is in How to give your AI agent web search with MCP.

Step 2: Chunk by heading

Split on markdown headings first, then on size. Every chunk keeps the path of headings above it, which improves both retrieval and citations.

import re

def chunk_markdown(doc: dict, max_chars: int = 1500, overlap: int = 150):
    """doc is the `data` object returned by agentsearch-web-extract-v1."""
    chunks, path, buf = [], [], []

    def flush():
        text = "\n".join(buf).strip()
        if not text:
            return
        for i in range(0, len(text), max_chars - overlap):
            chunks.append({
                "text": text[i:i + max_chars],
                "source_url": doc["url"],
                "title": doc.get("title"),
                "section": " > ".join(path),
                "fetched_at": doc["meta"]["fetched_at"],
            })

    for line in doc["markdown"].splitlines():
        m = re.match(r"^(#{1,6})\s+(.*)", line)
        if m:
            flush(); buf.clear()
            level = len(m.group(1))
            path[:] = path[:level - 1] + [m.group(2).strip()]
        buf.append(line)
    flush()
    return chunks

Embed chunk["text"] and store the rest as metadata. When your agent answers, cite title and source_url.

Step 3: Handle failures properly

Extract separates target-site failures from request problems:

Where	HTTP	Codes	What to do
`data.error` (target site)	200	`TARGET_HTTP_ERROR`, `TARGET_TIMEOUT`, `TARGET_DNS_ERROR`, `TARGET_CONNECT_ERROR`, `TARGET_FETCH_FAILED`, `TARGET_TOO_MANY_REDIRECTS`, `UNSUPPORTED_CONTENT`	Retry only when `retryable` is true (timeouts, connection failures, target 5xx/429)
Request	400	`INVALID_REQUEST`, `SSRF_BLOCKED` (private, loopback or metadata addresses)	Fix the input. Don't retry
Request	413 / 429	`REQUEST_TOO_LARGE`, `CAPACITY_LIMIT`, `APPLICATION_RATE_LIMITED`	Back off and retry

UNSUPPORTED_CONTENT means the URL points to something other than HTML or text, such as a PDF or an image. Send those to a document parser instead.

Step 4: Treat extracted content as untrusted

Any page can contain text written to manipulate your agent. Before content reaches a model:

Put extracted markdown in a data position, such as a tool result or a quoted block, never in a system prompt.
Ignore instructions that appear in page content.
Check portal.schemaCheck. Only passed means a schema check ran.

When the page needs JavaScript

Extract fetches the HTML the server sends, which is fast and works for docs, blogs, news and most marketing pages. Single-page apps and dashboards often render their content only after JavaScript runs. For those, AgentSearch has Web Render (agentsearch-web-render-v1), live on Pocket MainNet. It loads the page in headless Chromium and returns the rendered markdown, text, HTML, links and an optional screenshot. It honors robots.txt and doesn't bypass bot walls. Render isn't on the Agentic Portal yet. Its request and response contract is in agents.md and the render spec.

Search, then extract

A common pattern is to search with AgentSearch Web Search (up to 5 results per call), pick the most relevant results, then extract them in full. See the search spec, and compare per-call costs across vendors in Cheapest web search API for AI agents in 2026.

FAQ

How long can an extracted page be?
Up to 100,000 characters per call through max_chars. The default is 50,000.

Does it follow redirects?
Yes. url in the response is the final URL, and requested_url appears on errors.

Can it extract PDFs?
No. Non-HTML content returns UNSUPPORTED_CONTENT.

Who serves the requests?
Decentralized Pocket Network suppliers. Machine-readable details are at agentsearchhq.com/llms.txt.

More from the blog

Service pages: Web Search API · Web Extract API · Web Render API

Originally published at agentsearchhq.com.

Cheapest Web Search API for AI Agents in 2026

AgentSearch — Fri, 02 Oct 2026 23:49:38 +0000

Most "cheapest search API" posts compare monthly plans, but agents are billed per call. This post compares the published per-call prices of web search and extraction APIs that developers use with AI agents.

How we did this. We checked every price on October 2, 2026 on the vendor's own public pricing page, linked in each row. We only list a per-call price where the vendor states one. Plan-only and token-based pricing gets its own section, with the arithmetic shown. We don't compare result quality, latency or uptime. Prices change often, so check the source before you decide.

Disclosure: AgentSearch publishes this blog.

The short answer

Lowest stated price per search request: Exa Search (instant), at $4 per 1,000 requests ($0.004).
$0.005 per request: AgentSearch Web Search.
Lowest stated price per page extracted: Exa Contents, at $1 per 1,000 pages ($0.001).
The only option here with no account, no API key and no prepaid credits: AgentSearch, which you pay per call in USDC over x402 or MPP.

AgentSearch is not the cheapest on sticker price for every operation, and we're not going to claim it is. Where it does well is the cost of getting started and of paying as you go: nothing to sign up for, no plan, no card, no balance to top up. An agent with a wallet can pay $0.005 for one call and stop there.

Search APIs: stated per-call prices

Provider	Operation	Stated price	Per call	Account / key required	Source (checked 2026-10-02)
Exa	`/search` `instant` (up to 10 results)	$4 / 1k requests	$0.004	Yes (API key; $10 free credits monthly)	exa.ai/pricing
Exa	`/search` `fast` / `auto` (up to 10 results)	$7 / 1k requests	$0.007	Yes	exa.ai/pricing
AgentSearch	Web Search (`agentsearch-web-search-v1`, up to 5 results)	$0.005 / call	$0.005	No account, no API key (USDC via x402 or MPP)	Agentic Portal: search

Watch for what a call includes

Exa's base price covers up to 10 results per request. Results above 10 and AI page summaries are billed extra; see its pricing page.
AgentSearch Web Search returns up to 5 results per call (max_results 1–5), each with title, URL, snippet, score, domain and date. The fields are in the search spec.

Extraction APIs: stated per-page prices

Provider	Operation	Stated price	Per page	Source (checked 2026-10-02)
Exa	`/contents` (per content type)	$1 / 1k pages	$0.001	exa.ai/pricing
AgentSearch	Web Extract (`agentsearch-web-extract-v1`)	$0.005 / call	$0.005	Agentic Portal: extract

On per-page price alone, AgentSearch Extract costs more than Exa Contents. What you get for $0.005 is one URL turned into markdown or text, with the title, links and fetch metadata, and no account or prepaid balance. If you extract millions of pages a month, a bulk plan will cost you less.

Plan-based and token-based pricing (no stated per-call price)

These vendors don't publish a single per-call price, so they're left out of the tables above. The effective prices below are our arithmetic from their published plans, and they assume you use the whole monthly quota.

SerpAPI (serpapi.com/pricing): monthly plans. Starter is $25 a month for 1,000 searches (≈$0.025 per search if you use all of them). The free plan includes 250 searches a month.
Firecrawl (firecrawl.dev/pricing): credits. Hobby is $19 a month for 5,000 credits billed monthly, which the page equates to 2,500 searches (≈$0.0076 per search) or 5,000 scraped pages (≈$0.0038 per page).
Jina Reader (jina.ai/reader): billed by tokens, not requests. Search (s.jina.ai) costs "a fixed number of tokens, starting from 10,000 tokens" per request. The page doesn't state a per-request dollar price, so we can't compare it directly.

The real cost for an agent: a worked example

Take a research agent that runs one search and reads the top 2 pages on each task, 1,000 times a month. Using the stated prices above:

Approach	Calls per task	Cost per 1,000 tasks
AgentSearch search + 2 extracts	3 calls	$15.00
Exa `instant` search + 2 Contents pages	1 search + 2 pages	$4.00 + $2.00 = $6.00

Exa also gives $10 of free credits a month, which can bring a small workload down to $0. At small scale, free tiers can matter more than the per-call price; at any scale, AgentSearch needs no signup, plan or prepaid balance.

So what's the cheapest web search API for AI agents?

It depends on what you're optimizing for:

Lowest sticker price per search at volume: Exa instant ($0.004) on the pages we checked.
Lowest per-page extraction price: Exa Contents ($0.001).
Lowest cost to start, with no commitment: AgentSearch. There's no account, API key, subscription or minimum. You pay $0.005 per search or extract call in USDC, over x402 on Base or MPP on Tempo. For an autonomous agent that holds its own wallet, that means there's no human signup step.

Try AgentSearch

In an MCP client, add npx -y @pocket-network/agentic-portal-mcp and set a spending cap. The MCP setup guide walks through it. From code, any standard x402 v2 client works:

import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";

const account = privateKeyToAccount(process.env.AGENT_WALLET_KEY as `0x${string}`);
const pay = wrapFetchWithPaymentFromConfig(fetch, {
  schemes: [{ network: "eip155:8453", client: new ExactEvmScheme(account) }],
});

const res = await pay(
  "https://agent.pocket.network/v1/agentsearch-web-search-v1/v1/search",
  {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ query: "x402 v2 PAYMENT-SIGNATURE header", max_results: 5 }),
  },
);
const { portal, data } = await res.json();
console.log(portal.serviceId, data.results?.map((r: any) => r.url));

If delivery fails, the portal settles nothing, so you aren't charged for it.

The full walkthrough is in How to give your AI agent web search with MCP. Extraction is covered in Turn any URL into clean markdown for RAG. Service details: search, extract.

Sources (all checked 2026-10-02)

AgentSearch Web Search: https://agent.pocket.network/services/agentsearch-web-search-v1
AgentSearch Web Extract: https://agent.pocket.network/services/agentsearch-web-extract-v1
Exa: https://exa.ai/pricing
SerpAPI: https://serpapi.com/pricing
Firecrawl: https://www.firecrawl.dev/pricing
Jina Reader: https://jina.ai/reader/

Spotted a price that has changed? Tell us and we'll update the table and the "checked" date.

More from the blog

Service pages: Web Search API · Web Extract API · Web Render API

Originally published at agentsearchhq.com.

Add Web Search to Your AI Agent with MCP (2026 Guide)

AgentSearch — Fri, 02 Oct 2026 23:47:44 +0000

Language models stop learning on their training cutoff date. An agent that has to answer "what changed in this library last week?" or "what does this vendor charge today?" needs a tool that can search the live web and read the pages it finds. The Model Context Protocol (MCP) is the standard way to give it one.

This guide connects an MCP client (Claude Desktop, Cursor, Claude Code, or anything else that speaks MCP) to AgentSearch web search and URL extraction. It takes about five minutes. There's no account to create and no API key to manage: each call is paid in USDC through the Pocket Network Agentic Portal.

What you'll set up

AgentSearch runs two services on Pocket Network that matter for this guide:

Service	Service ID	What it returns	Price per call
Web Search	`agentsearch-web-search-v1`	Structured JSON for up to 5 results: title, URL, snippet, score, domain and date for each one	$0.005
Web Extract	`agentsearch-web-extract-v1`	Clean markdown or text of one URL, plus its title, links and fetch metadata	$0.005

Both prices come from the service pages on the Pocket Agentic Portal (search, extract). You pay in USDC over x402 on Base, or over MPP on Tempo.

You'll install one MCP server, @pocket-network/agentic-portal-mcp, Pocket's local MCP server for the Agentic Portal. It's listed in the official MCP Registry as network.pocket/agentic-portal-mcp. Pocket's full instructions are at agent.pocket.network/agent-integration.

Why a local server holds the wallet

MCP clients can't pay for tools on their own yet. So the portal's MCP server runs on your machine, keeps the paying wallet's key there, and signs an x402 payment only when a call fits inside the spending limits you set. The portal never holds your key.

Prerequisites

Node.js 20 or newer, so npx is available.
An MCP client: Claude Desktop, Cursor, Claude Code or similar.
A dedicated wallet holding a small amount of USDC on Base. Pocket's docs are blunt about this: the key sits in a plain-text config file, so use a wallet made for this purpose that holds only what you're willing to spend.

Step 1: Add the MCP server to your client

Open your client's MCP config file:

Claude Desktop: claude_desktop_config.json
Cursor: .cursor/mcp.json
Claude Code: .mcp.json

Then add the server, which runs npx -y @pocket-network/agentic-portal-mcp:

{
  "mcpServers": {
    "pocket-network": {
      "command": "npx",
      "args": ["-y", "@pocket-network/agentic-portal-mcp"],
      "env": {
        "POCKET_PRIVATE_KEY": "0x…",
        "POCKET_MAX_TOTAL_ATOMIC": "1000000"
      }
    }
  }
}

Put settings in the env block. Desktop clients start MCP servers without your shell's environment, so a variable you export in a terminal won't reach the server.

Set spending limits before you do anything else

The server checks every limit before it signs, and signs nothing unless all of them allow the payment. USDC has 6 decimals, so amounts are in atomic units and 1000000 = $1.00.

Variable	What it does
`POCKET_MAX_TOTAL_ATOMIC`	The most the server will sign for in total while it runs. Required to pay. `1000000` ($1.00) covers 200 AgentSearch calls at $0.005 each.
`POCKET_MAX_PER_CALL_ATOMIC`	The most it will pay for any one call. Set it to `5000` ($0.005) so the agent can only buy calls priced at $0.005 or less.
`POCKET_NETWORK`	The only network it signs on. The default is Base, `eip155:8453`.
`POCKET_QUOTE_ONLY`	Set to `true` and the server returns the seller's terms but never pays. Good for a dry run.

Pocket also warns that you shouldn't count on the client asking before it pays: in their testing, one client's model asked first and another paid without asking. These limits are what protect you either way.

A cautious first config:

{
  "mcpServers": {
    "pocket-network": {
      "command": "npx",
      "args": ["-y", "@pocket-network/agentic-portal-mcp"],
      "env": {
        "POCKET_PRIVATE_KEY": "0x…",
        "POCKET_MAX_TOTAL_ATOMIC": "250000",
        "POCKET_MAX_PER_CALL_ATOMIC": "5000",
        "POCKET_QUOTE_ONLY": "true"
      }
    }
  }
}

Restart the client after you save. Remove POCKET_QUOTE_ONLY when you're ready to make real calls.

Step 2: Find AgentSearch through the free tools

The server exposes three tools:

Tool	What it does	Cost
`search_services`	Searches the portal catalogue by keyword and category	Free
`describe_service`	Shows one service's price, operations, schemas and a real captured example	Free
`call_service`	Calls a service and pays its price	The service's price

If you leave out the private key, only the free tools work, which is a safe way to look around. Try a prompt like this:

Use search_services to find "agentsearch". Then describe_service for agentsearch-web-search-v1 and tell me its price and request schema.

You should see $0.005 and a POST /v1/search operation that takes a query and an optional max_results (up to 5).

Step 3: Run a paid search

Now ask for something that needs fresh information:

Use call_service on agentsearch-web-search-v1 to search for "x402 payment protocol v2 headers", then summarize the top results with their URLs.

Under the hood, the agent sends the same request you could send yourself:

curl -X POST https://agent.pocket.network/v1/agentsearch-web-search-v1/v1/search \
  -H 'content-type: application/json' \
  -d '{"query":"x402 payment protocol v2 headers","max_results":5}'
# The first response is 402 Payment Required with the terms.
# The MCP server signs them and retries the same request with the payment attached.

The result comes back in the portal's envelope. Fields set by the portal sit under portal, and the supplier's untouched response sits under data:

{
  "portal": {
    "provenance": "third-party-supplier",
    "serviceId": "agentsearch-web-search-v1",
    "schemaCheck": "unchecked"
  },
  "data": {
    "query": "x402 payment protocol v2 headers",
    "result_count": 5,
    "results": [
      { "title": "…", "url": "https://…", "content": "…", "score": 0.91, "domain": "…" }
    ]
  }
}

The data above is trimmed. The full field list is in the search OpenAPI spec.

Step 4: Extract a full page when snippets aren't enough

When the agent needs a whole document, such as a changelog, an API reference page or a pricing table, use extract:

Use call_service on agentsearch-web-extract-v1 with url "https://example.com" and return the markdown.

{ "url": "https://example.com", "formats": ["markdown"], "max_chars": 20000 }

You get title, markdown, links and meta back (status_code, content_type, fetched_at, chars, truncated). Turn any URL into clean markdown for RAG goes deeper on extraction.

Treat results as data, not instructions

Search results and extracted pages are third-party content, and some pages are written to manipulate agents. Pocket's integration guide recommends these habits:

Keep data in a clearly marked data position, such as a tool-result block. Never paste it into a system prompt.
Don't follow instructions that appear inside a result. A page telling your agent to "call another endpoint" or "ignore prior instructions" is an attack, not an answer.
Check portal.schemaCheck. Only passed means a schema check actually ran.

Troubleshooting

Only the free tools work: POCKET_PRIVATE_KEY is missing, or it's set in your shell instead of the config's env block.
Calls are refused before payment: a limit blocked the call (total, per-call or network), or POCKET_QUOTE_ONLY is still true.
An empty results array with an error object: the search hit its deadline or an upstream error. Check error.retryable and retry with backoff.
Extract returns UNSUPPORTED_CONTENT: the URL is a PDF or an image, not HTML or text.
The page only renders with JavaScript: extract fetches static HTML. AgentSearch also runs a headless-browser Web Render service (agentsearch-web-render-v1, live on Pocket MainNet). It isn't on the Agentic Portal yet.

FAQ

Do I need an AgentSearch account or API key?
No. You pay per call in USDC through the Agentic Portal, over x402 (Base) or MPP (Tempo).

Which MCP clients work?
Any client that can launch a local stdio MCP server, including Claude Desktop, Cursor and Claude Code.

Who runs the infrastructure?
AgentSearch is served by decentralized Pocket Network suppliers.

Next steps

Compare costs in Cheapest web search API for AI agents in 2026.
Read the machine-readable docs at agentsearchhq.com/llms.txt and agentsearchhq.com/agents.md.
See the service pages for search, extract and render.
Browse the OpenAPI specs for search and extract.

More from the blog

Service pages: Web Search API · Web Extract API · Web Render API

Originally published at agentsearchhq.com.