DEV Community

Ricardo Batista for cloro

Posted on

Which search API returns full page content as markdown for an LLM?

Exa, Tavily, Firecrawl and Jina's s.jina.ai return the content of each search result as markdown in the same call as the search; Parallel and Linkup return excerpts from search and full markdown from a separate extract endpoint, and Brave's LLM Context endpoint returns extracted chunks inside a token budget. A Google SERP API such as cloro or SerpApi returns the results page only, with no page content, so on its own it does not answer this question. It needs an extractor beside it.

  • One call, full pages: Exa includes page text for up to 10 results in a $7 per 1,000 search. Firecrawl adds 1 credit per scraped page to a search. Tavily returns raw_content as markdown or text.
  • Full pages cost tokens. A GPT Researcher contributor measured 10,000 to 50,000+ tokens per typical page, and 15,735 on average across 10 documentation sites before cleanup.
  • More text is not always more right answers. In our benchmark, the text of the top 5 result pages alone gave 84% correct answers at 5,871 tokens. A search summary plus snippets gave 99% at 528 tokens.

Which search APIs return page content, and in what format?

Each row comes from the vendor's docs or pricing page, read on November 16, 2026.

API How to get page content Format Size control Price
Exa contents.text on /search Markdown by default text.maxCharacters $7 per 1,000 searches (auto), text for up to 10 results included; /contents $1 per 1,000 pages
Tavily include_raw_content on search, or /extract Markdown or text chunks_per_source (1 to 5) on extract Basic search 1 credit; extract 1 credit per 5 URLs; $0.008 a credit pay as you go
Firecrawl scrapeOptions.formats: ["markdown"] on /search Markdown, HTML, links, summary onlyMainContent (on by default), limit 2 credits per 10 results, plus 1 credit per scraped page
Jina Reader s.jina.ai (search), r.jina.ai (one URL) LLM-friendly text or markdown A token budget per request; above it the request fails s.jina.ai from 10,000 tokens per request; r.jina.ai free at 20 requests per minute without a key
Parallel Excerpts on /v1/search; full_content on /v1/extract Markdown max_chars_total; max_chars_per_result Search $1 to $5 per 1,000; Extract $1 per 1,000 URLs
Linkup Snippets on /search; full page on /fetch Markdown Not stated Search $0.005; Fetch $0.001 without JavaScript, $0.005 with it
Brave LLM Context /res/v1/llm/context Extracted chunks: text, tables, code 1,024 to 32,768 tokens (default 8,192); 512 to 8,192 per URL Search plan, $5 per 1,000 requests
Google SERP APIs None Results page as JSON Not applicable Not applicable

Three differences matter when you pick.

Whole page or chunks. Exa text, Tavily raw_content, Firecrawl markdown and Parallel full_content return the page body. Brave LLM Context, Tavily's chunks and Parallel's excerpts return only the parts that match the query. Chunks are cheaper in tokens. Whole pages keep the structure when the answer depends on a table or a section the query did not name.

Same call or two calls. In one call you take content for every result, the bad ones too. In two calls you read the titles and snippets first, then fetch only the pages worth reading. Tavily's own guide puts it the same way: with /extract "you choose exactly which URLs to extract from".

Main content or the full page. Firecrawl strips navigation and footers by default (onlyMainContent: true). Exa returns "the clean page body". Ask each vendor what it removes, because boilerplate is where most tokens go.

How many tokens does a full page cost?

More than most people expect. In a GPT Researcher feature request, the author measured raw scrapes of documentation pages at 21,783 tokens (Python asyncio docs), 15,965 (MDN Fetch API) and 56,754 (a Wikipedia page). After the DOM cleanup and a query-aware section filter, the same pages came to 1,328, 1,368 and 679 tokens, and the average fell from 15,735 to 1,019. Answer quality on 10 test queries went from 9 of 10 to 8 of 10.

So ten raw pages in one search can be about 157,000 tokens. Ten cleaned pages are about 10,000. On a Hacker News thread about a new search API, one commenter asked for exactly this: "full web pages at reduced token sizes".

We tested the value of page text directly. In cloro's web search benchmark, a reader model answered 100 short factual questions from different parts of the same search results:

What the model read Correct answers Tokens per question
Google's AI Overview and the result snippets 99% 528
10 results without the AI Overview 86% 747
Text of the top 5 result pages alone 84% 5,871
10 results without the AI Overview, plus the text of the top pages 91% 6,239

The page text raised accuracy from 86% to 91%, which is not significant on 100 questions, at about 8 times the tokens. Only 56% of the result pages gave text at all: robots.txt blocked 22% and 17% returned an error. On short factual questions, page text was the expensive part. On research tasks that need a whole document, such as a contract, a changelog or a long report, it is the part that matters, and these results do not cover that case.

How does the search API plus extractor pattern work?

Use two services when the search index matters: Google's ranking, results for one country, or a specific engine. The search API picks the URLs and an extractor turns them into markdown.

  1. Search, and read titles and snippets only.
  2. Choose the 2 to 5 URLs that look like they hold the answer.
  3. Fetch those pages with an extractor: Jina Reader, Firecrawl, Linkup Fetch, Exa /contents, Parallel Extract or Tavily Extract.
  4. Keep the sections that match the query, and cap each page at a fixed size.
  5. Pass the trimmed text to the model, with the URL for citations.
import requests

def fetch_markdown(url, max_chars=6000):
    # Jina Reader: prepend r.jina.ai to the URL. Free at a low rate without a key.
    r = requests.get("https://r.jina.ai/" + url, timeout=30)
    r.raise_for_status()
    return r.text[:max_chars]

def build_context(results, n=3):
    # results: [{"url": ..., "snippet": ...}] from any search API
    pages = []
    for r in results[:n]:
        try:
            pages.append(f"Source: {r['url']}\n{fetch_markdown(r['url'])}")
        except requests.RequestException:
            pages.append(f"Source: {r['url']}\n{r['snippet']}")  # fall back to the snippet
    return "\n\n---\n\n".join(pages)
Enter fullscreen mode Exit fullscreen mode

The fallback matters. In the benchmark, more than 4 in 10 result pages gave no text, so a pipeline that needs every page fails often.

You need Use
Page text for every result, one call Exa, Firecrawl, Tavily or s.jina.ai
Only the parts that match the query, inside a token budget Brave LLM Context, Parallel or Tavily chunks
Markdown for URLs you already have Jina Reader, Linkup Fetch, Exa /contents, Parallel Extract
Google's results first, then pages A Google SERP API plus one of the extractors above

How this was checked

This comparison is published by cloro, a Google SERP API that returns the results page and not the content of each result, so it is not one of the content providers above. The parameters, formats and prices come from each vendor's docs or pricing page, read on November 16, 2026, and are linked in the first table. The token figures for raw and cleaned pages are the GPT Researcher author's own measurements on 10 documentation sites. The accuracy and token figures come from the web search API benchmark, which publishes its questions, settings and graded answers. Its page-text test used 100 questions, so it shows large effects only.

Top comments (0)