DEV Community: SerpApi

DeepSeek Search and Fetch APIs vs. SerpApi: Features, Pricing, and Use Cases

Magenta Qin — Sun, 08 Nov 2026 20:16:15 +0000

DeepSeek has opened the web search infrastructure behind its models to developers. Its new Search and Fetch APIs give agents and retrieval-augmented generation (RAG) applications a way to find live web pages and turn them into model-ready context.

That inevitably raises a question for developers already using search and web extraction APIs: can DeepSeek Search and Fetch API replace SerpApi?

The short answer is that the two products have overlapping features, but they operate at different layers of the search stack. DeepSeek Search is an AI-native web retrieval API. It is designed to find relevant pages, rank them using signals such as relevance and authority, and return summaries that a language model can read directly. SerpApi provides structured search data. It gives applications access to results from Google, Bing, Baidu, Google Maps, Google Shopping, YouTube, Google Flights, and many other general and vertical search products.

Both can power AI agents, but they provide agents with different kinds of context. DeepSeek Search emphasizes ranked web pages and model-ready summaries, while SerpApi can provide source-specific, structured data such as flight options, local business details, and shopping results. The right choice depends on whether the agent needs general web evidence or data from a particular search product.

This blog will compare their features, pricing, rate limits, and best use cases.

What DeepSeek Released

DeepSeek introduced two endpoints for live web retrieval:

Search API: finds web pages for a query and returns a ranked list of structured results.
Fetch API: retrieves a specific public web page and returns its main content as Markdown.

A basic search request looks like this:

curl https://api.deepseek.com/v1/tools/search \
  -H "Authorization: Bearer $DEEPSEEK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "中国 AI 编程工具对比 2026",
    "count": 5,
    "need_content": true
  }'

The API returned the following result:

{
  "results": [
    {
      "title": "GLM、MiniMax、Kimi、豆包、千问，国产模型Coding Plan怎么选？一文全套短评",
      "site_name": "tencent.cn",
      "authority": 3,
      "rerank_score": 2.226604461669922,
      "date": "2026-03-02T16:00:00+00:00",
      "url": "https://cloud.tencent.cn/developer/article/2633452#1",
      "summary": "最近使用 Claude Code 编程，结合 Skills，完成一些新的项目，也趁此机会测试了国产模型编程方面的效果……"
    }
    ...
  ]
}

DeepSeek Search returns at most 50 results in one response, and its current documentation does not expose a pagination parameter.

By contrast, SerpApi supports pagination on supported search engines. For example, Google Search responses can includeserpapi_pagination.next, and the start parameter can be used to request subsequent result pages. See SerpApi’s pagination documentation for examples.

How SerpApi Approaches Search

SerpApi takes a different approach. Instead of maintaining one general-purpose search index, it turns existing search engines and vertical search products into structured APIs.

A Google Search request, for example, can return separate sections for:

Organic results
Answer boxes
Knowledge Graph data
Local results
News, images, and videos
Shopping results
Related questions and searches
Ads and other result-page features

The same pattern extends beyond general web search. SerpApi provides dedicated APIs for products such as Google Maps, Google Shopping, Google Flights, Google Hotels, YouTube, Amazon, Apple App Store, and many others. Each API exposes fields specific to its source.

That difference becomes important when an agent needs reliable fields it can act on, not only prose it can read. A local-search workflow may need a business's rating, review count, address, phone number, opening hours, and location on a map. A flight-search workflow may need airports, departure times, layovers, duration, airline, carbon emissions, and price. Those are structured business fields, not just webpage summaries.

SerpApi lets developers define where a search is performed, which country and language it targets, and whether it runs on desktop or mobile. Engine-specific parameters can further filter or sort the results. These controls make searches more targeted and reproducible, rather than relying mainly on the caller's IP address.

Many SerpApi engines also expose pagination. For Google Search, for example, the response can includeserpapi_pagination.next, while the start parameter can request subsequent result pages. The number of available pages still depends on what the underlying search engine serves, and each page is a separate SerpApi search. See the Google Search API documentationhttps://serpapi.com/search-api) for an example of the available parameters and response types.

The Core Difference: Web Retrieval vs. Search Product Data

The simplest way to distinguish the two products is to ask what the agent or application ultimately needs.

Both DeepSeek Search API and SerpApi can return relevant web pages that help a model answer a question. DeepSeek's distinction is that every result includes model-oriented retrieval signals: a long summary, optional page content, rerank_scoreand an authority tier. These fields can help an agent decide which sources to read first.

SerpApi's distinction is its breadth of sources and structured result types. An agent can search SerpApi's own web index or retrieve data from a particular engine or vertical product. Instead of flattening every result into generic text, SerpApi preserves source-specific fields. For example, a Maps result can include ratings and opening hours, while a Flights result can include prices, schedules, and layovers. SerpApi's Search Index documentation does not currently expose an equivalent to DeepSeek's authority field, but it provides pagination, localization, multiple content types, and source-specific controls.

SEO Use Case DeepSeek Does Not Cover: Cross-Border E-commerce Monitoring

A brand selling internationally needs data from the marketplaces and shopping engines where customers actually search, not just a list of relevant pages from the general web.

Consider a Chinese consumer-electronics brand expanding into the United States and Germany. It may need to track where its products appear for a keyword on Google Shopping, Amazon, and Walmart; compare prices and ratings with competitors; distinguish sponsored from organic listings; and repeat the same searches for different countries, languages, and devices.

SerpApi supports this type of workflow. Its dedicated Google Shopping, Amazon, and Walmart APIs contain source-specific fields such as product position, price, rating, review count, seller, and product identifiers. The application can store the structured responses and compare changes across markets and over time.

DeepSeek Search can still support market research by finding product reviews, news, or competitor websites. However, its results come from a general web index rather than a specified marketplace. Its rerank_score is not an Amazon product rank or a Google Shopping position, and it does not return a stable marketplace-specific product schema. DeepSeek Fetch API can read a known product page, but the application would still need to extract and normalize fields such as price, seller, and rating itself.

For marketplace rank tracking, international price monitoring, and cross-border competitor analysis, SerpApi therefore provides structured source data that DeepSeek Search and Fetch API cannot replace.

Developer Experience: Raw HTTP vs. Official Client Libraries

DeepSeek Search and Fetch are raw HTTP APIs, not language-specific SDKs. The documentation shows cURL and Pythonrequests, but they can be called from any language with an HTTP client. Without a dedicated SDK, teams may need to handle authentication, retries, errors, and response types themselves.

SerpApi supports raw HTTP and provides official client libraries for Ruby, Python, JavaScript/TypeScript, PHP, Go, Java, .NET, Swift, and C++. These libraries can reduce integration and maintenance work through language-native helpers for common API operations.

Search and Page Extraction

DeepSeek Fetch and SerpApi's Extract API have the most direct overlap in features. DeepSeek Fetch accepts a URL and returns its title and content as Markdown. Here is how both APIs can extract the same page:

# DeepSeek Fetch
curl https://api.deepseek.com/v1/tools/fetch \
  -H "Authorization: Bearer $DEEPSEEK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://cloud.tencent.cn/developer/article/2633452"}'

# SerpApi Extract with Markdown output
curl --get https://serpapi.com/search \
  --data-urlencode "engine=extract" \
  --data-urlencode "url=https://cloud.tencent.cn/developer/article/2633452" \
  --data-urlencode "output=md" \
  --data-urlencode "api_key=$SERPAPI_API_KEY"

DeepSeek Fetch returned a JSON object containing the page title and Markdown content. The response below is shortened:

{
  "title": "GLM、MiniMax、Kimi、豆包、千问，国产模型Coding Plan怎么选？一文全套短评",
  "content": "# GLM、MiniMax、Kimi、豆包、千问，国产模型Coding Plan怎么选？\n\n最近使用 Claude Code 编程，结合 Skills，完成一些新的项目……"
}

With the Extract API and output=mdSerpApi returned Markdown directly. This response is also shortened:

www.tangshuang.net

[社区首页](https://cloud.tencent.cn/developer) > [专栏](https://cloud.tencent.cn/developer/column)

# GLM、MiniMax、Kimi、豆包、千问，国产模型Coding Plan怎么选？一文全套短评

发布于 2026-03-03 16:09:25

Hello大家好呀！最近使用 Claude Code 编程，结合 Skills，完成一些新的项目，
也趁此机会测试了国产模型编程方面的效果……

From the responses above, we can see that DeepSeek focused more tightly on the article's main content. At the same time, SerpApi preserved more of the original page structure , including navigation links and page-level metadata. The difference here is the extraction scope, not just the output format: SerpApi can also return JSON, but in this test it retained a broader set of page elements than DeepSeek Fetch did.

SerpApi's Extract API also supports:

scrape, map, and crawl modes
JSON, Markdown, and HTML output
Desktop, tablet, and mobile browser profiles

It follows robots.txt and uses the appropriate SerpApi engine when the URL is a supported search page, such as Google/Amazon or Bing/Baidu.

JSON for Code, Markdown for Models

DeepSeek Search and Fetch API are both JSON APIs. Search returns a results array, while Fetch returns title and content. Fetch API stores Markdown in the content field, but the response itself is still JSON. The current DeepSeek documentationdoes not describe a direct Markdown response mode for either API.

SerpApi offers several response formats. JSON remains useful when application code needs exact fields and a strict schema. When search results are sent directly to an LLM, however, Markdown can represent the same evidence with less formatting overhead.

JSON repeats field names for every result and adds braces, quotes, commas, and nested objects. SerpApi's Markdown output turns result arrays into tables, uses native Markdown links, and removes internal tracking fields, opaque tokens, and duplicated values. SerpApi reports roughly 50% average token savings compared with its JSON responses, although the result varies by API and query. Complex nested responses can see savings above 90%; use JSON when the application needs fields omitted from the curated Markdown view.

A Google Shopping experiment explains the difference. Across 12 queries, Markdown used 89.55% fewer input tokens on average. With a fixed 1,550-token budget, the JSON representation fit 16 products, while Markdown fit all 30. Markdown did not improve the model itself; it allowed it to compare more of the original evidence within the same context window.

Pricing and Rate Limits

DeepSeek's pricing is relatively low for a web search API:

API	Price	Published rate limit
DeepSeek Search	$0.75 per 1,000 successful requests	100 QPS per account
DeepSeek Fetch	$0.75 per 1,000 successful requests	25 QPS per account(per-domain QPS limit is not publicly documented)

SerpApi pricing is based on monthly plans. Each plan includes a monthly search quota and an hourly throughput limit.

Plan	Monthly price	Searches per month	Throughput per hour
Free	$0	250	50
Starter	$25	1,000	200
Developer	$75	5,000	1,000
Production	$150	15,000	3,000
Big Data	$275	30,000	6,000

DeepSeek is much cheaper per request, but the cheapest request does not always produce the lowest total cost.

For general research, DeepSeek may return the sources and summaries an agent needs in one request. For product monitoring, SerpApi may cost more per request but return prices, sellers, and ratings directly, avoiding extra page fetches and model-based extraction.

The real cost therefore depends on the number of API calls, the model tokens consumed, and any additional processing required. We will compare these factors in the follow-up benchmark blog.

Choosing Between DeepSeek and SerpApi

Choose DeepSeek Search/Fetch API when :

You need to search the general web and read a few relevant pages, rather than relying on a specific search engine/product such as Google Maps or Google Shopping.
Built-in summaries, rerank_score, authorityand optional page content are more useful than source-specific fields.
You want a low-cost API and do not need more than 50 results or pagination.

Typical uses include research assistants, technical-documentation discovery, and RAG source collection.

When SerpApi Is the Better Fit

SerpApi is a stronger fit when:

You need results from a specific source, such as Google, Bing, Google Maps, Google Shopping, YouTube, or Google Flights.
Your code needs structured fields such as prices, ratings, opening hours, or flight schedules.
You need precise control over location, language, country, domain, or device.
You are building rank tracking, price monitoring, local search, travel search, or marketplace research.
You need features such as pagination, raw HTML, archived searches, asynchronous requests, or site crawling.

Use them together

A product does not have to use the same API for every search. It can route each request based on what the task requires:

Simple web research: Use DeepSeek Search when general web results are enough, and the task can be completed within its 50-result limit. Use DeepSeek Fetch API to read the most relevant pages.
Larger or more controlled searches: Use SerpApi when the product needs pagination, precise location or device settings, or results from a specific search engine.
Source-specific product features: Use SerpApi's structured JSON for current rankings, price monitoring, local search, travel search, or data from sources such as Google, Amazon, and Instagram.
AI analysis: DeepSeek's summaries and fetched content can be passed to a model. SerpApi results can also be requested as Markdown and sent directly to the model, without first converting a large JSON response into readable text. For example, a market-intelligence product could use DeepSeek to discover new competitors across the web, SerpApi to monitor their Amazon listings and Instagram profiles, and SerpApi's Markdown output to provide an AI agent with concise context for its final analysis.

At-a-glance comparison

Capability	DeepSeek Search and Fetch API	SerpApi
Main focus	General-purpose web search and page retrieval	Engine-specific search across 100+ search engines and verticals
Search source	DeepSeek's general web index	The caller selects an `engine`, such as `search_index`, `google`, `amazon`, `google_maps`, or `google_flights`
Result structure	Web results with summaries and optional page content	Structured fields tailored to each search engine or vertical
Ranking signals	`rerank_score` and `authority`	Result position and source-specific fields
Page retrieval	Search can include page content with `need_content`; Fetch reads one known URL and returns its title and main content; no documented site mapping or crawling	Extract can scrape one page, map a site's URLs, or crawl up to 100 pages; crawl mode uses one credit per successfully extracted page
Pagination	No documented pagination; up to 50 results	Available on supported engines
Search controls	Localization IP and query operators	Location, country, language, domain, device, and source-specific parameters
Response formats	JSON; extracted page content is stored as Markdown inside the JSON response	JSON, Markdown, and HTML
Client libraries	Raw HTTP; no dedicated Search/Fetch SDK suite	Official libraries for nine languages
Pricing	$0.75 per 1,000 successful Search requests; $0.75 per 1,000 successful Fetch requests	Monthly plans: $0 for 250, $25 for 1,000, $75 for 5,000, $150 for 15,000, or $275 for 30,000 searches
Rate limits	Search: 100 QPS per account; Fetch: 25 QPS per account plus an unpublished per-domain limit	50 to 6,000 searches per hour, depending on the plan; custom limits for Enterprise

Conclusion

DeepSeek Search and Fetch API provide a low-cost way to find information across the general web and read selected pages. SerpApi covers a broader range of sources and use cases. It combines its own web index with APIs for specific search engines and vertical products, preserving fields such as prices, ratings, opening hours, and flight details. Its JSON, Markdown, and HTML responses also support both application code and AI workflows.

So, can DeepSeek Search and Fetch replace SerpApi?

DeepSeek can be simpler and less expensive. But it is not a like-for-like replacement when a product needs results from a specific source, pagination, precise localization, or source-specific structured fields.

The more useful question is not which API is universally better. It is whether the task needs compact general web retrieval or source-specific, structured search data. Some products will need both.

In the next blog, we will test both APIs on breaking news, local business search, product research, and technical research. We will compare relevance, latency, output size, rate-limit behavior, and the actual cost of completing each task.

DeepSeek #Exract API

Markdown vs. JSON: Which Response Format Should You Use?

Sonika Arora — Sun, 08 Nov 2026 04:10:01 +0000

When using APIs to get search results, the format you choose can impact how easily your application uses that data. Markdown and JSON both let you work with search results, but they're designed for different needs. JSON organizes information into structured fields that your code can access directly. Markdown presents information in a readable, text-based format that's easy for people and language models to interpret.

The right choice depends on what happens after you retrieve your search results.

Markdown vs. JSON at a glance

💡 The short answer

➡️ Choose Markdown when the next step is primarily reading, displaying, or reasoning, especially when the output feeds an LLM and token cost matters.

➡️ Choose JSON when the next step is primarily processing or manipulating data, such as extracting fields, validating, or storing.

SerpApi's APIs offer responses in both formats so you can pick the best one that fits into your workflow. Let’s dive into how Markdown and JSON compare across real-world SerpApi integration scenarios.

Example use cases that benefit from markdown

Markdown is a lightweight text format that supports headings, lists, links, and other simple formatting. For search results, this makes Markdown particularly useful when you want to give an AI model or agent information it can understand and use to complete a task.

You can read more about the benefit of Markdown output from our APIs here:

SerpApi: Markdown Output for AI Agents

Use case 1: Building AI agents that research and answer questions

Imagine you're building an AI research agent that answers questions using live web search.

A user asks: "What are the latest developments in the electric vehicle market, and how are the leading manufacturers responding?"

Your agent needs to search the web, gather relevant results, compare sources, identify trends, and generate a coherent answer.

Markdown is a natural fit for this workflow. Search results can be presented as readable text with titles, snippets, links, and other available information.

Markdown works well here because the model can focus on understanding the information rather than interpreting a complex data structure, and search results can be passed into summarization, comparison, and question-answering workflows.

Here’s an example of the Markdown output for the query latest developments in the electric vehicle market using Google News API that gets the latest news on that:

a screenshot of the playground example

Try it in the playground:

SerpApi Playground - SerpApi

💡 If you see the JSON output in the playground and want to switch to Markdown, head over to the tools button in the top right of the output window and toggle over to Markdown 👇

Switch between JSON and Markdown

Use case 2: Retrieval-augmented generation (RAG)

Retrieval-augmented generation, or RAG, combines information retrieved from external sources with an LLM's ability to generate answers.

For example, you might build an assistant that retrieves current web information before answering questions about competitors, industry trends, or product specifications - Markdown works well in this workflow.

This is because the retrieved information is intended to be read and interpreted by a language model. Choose Markdown when search is one step in a larger reasoning workflow and the agent needs to interpret the results before taking action.

Here’s an example of the Markdown output for the query iPhone using Google Shopping API that gets all the shopping results in a simple, easy-to-analyze format:

a screenshot of the playground example

Try it in the playground:

SerpApi Playground - SerpApi

Use case 3: Creating readable research summaries and reports

You might be building a tool that generates:

Competitor research reports
Market research summaries
News briefings
Product comparisons
Research notes for analysts
Summaries of relevant web sources

If the end product is primarily readable text, Markdown is a practical intermediate format.

Here’s an example of the Markdown output for the query coffee using Google Maps API that gets all the coffee shops in a particular area. This can easily be compiled into a report by any AI agent.

a screenshot of the playground example

Try it in the playground:

SerpApi Playground - SerpApi

Example use cases that benefit from JSON

JSON, or JavaScript Object Notation, is a structured data format. It organizes information into objects and arrays containing named fields and values.

This works great when your application needs to identify specific pieces of information and do something with them programmatically.

Use case 1: Building applications that display search results

Say you're building a search interface that displays Google Shopping product comparisons inside your own application. You'll ideally want to display each result in a consistent layout with a title, link, snippet, image, or price, depending on the search engine and result type. JSON makes this straightforward.

Your application can read the relevant fields and use them to populate UI components. Each result can become a card, row, or other interface element without having to extract information from formatted text.

Here's an example of what the JSON output looks like from SerpApi's Google Shopping API for the query "laptop":

a screenshot of the playground example

Your application can access title, price, and source directly and render them wherever needed.

Try it in the playground:

SerpApi Playground - SerpApi

Use case 2: Using data in automated pipelines

Consider an application that tracks flight prices across multiple airlines. The app needs to retrieve flight information, extract prices, compare them over time, and store the results for further analysis, all of which is set up in a series of steps.

The results need to enter a repeatable data-processing pipeline, so JSON is well suited to this process. The application can access fields directly rather than trying to recover individual values from formatted text.

Here's an example of what the JSON output looks like from SerpApi's Google Flights API for flights from CDG to AUS:

a screenshot of the playground example

Try it in the playground:

SerpApi Playground - SerpApi

Use case 3: Monitoring, filtering, and triggering actions

Say you're building an application that needs to compare specific values across requests. In this case, JSON makes it easier to access the fields required for that comparison, apply business rules, and trigger the next action.

For example, if you're building a system that monitors Google Search rankings for a list of keywords, and you want to detect when a particular website enters the top five results, record its position, and send a Slack notification when its ranking changes. This logic is more straightforward when the application can access structured fields rather than interpret formatted text.

The same approach works for monitoring job listings, tracking product availability, identifying new search results, or detecting changes in other supported search-engine data.

Here's an example of the JSON output from SerpApi's Google Jobs API that you can use to track open jobs over time:

a screenshot of the playground example

Try it in the playground:

SerpApi Playground - SerpApi

Common Questions

Can you use JSON with LLMs?

JSON is not limited to traditional software applications. LLMs can interpret JSON, extract information from it, compare fields, and use it as context. In fact, JSON can be preferable when you need the model to reason about specific attributes, such as product prices, publication dates, or search positions.

For example, if an LLM needs to compare three products by price and rating, a structured JSON representation makes those attributes explicit. Markdown may be more convenient and less token-intensive if the task involves reading several articles and synthesizing their arguments.The distinction is about the shape of the information and the task being performed.

JSON doesn't guarantee that an LLM will produce correct answers, and Markdown doesn't guarantee better reasoning. Both formats still require appropriate context management and validation.

Our APIs provide an option to receive the results in either JSON or Markdown format, so you can use the one that works best for your use case.

Does Markdown use fewer tokens than JSON?

Generally, yes - if you use it for the right tasks. Markdown can avoid a lot of the structural overhead associated with nested JSON objects and repeated field names.

We've observed that using the Markdown version of the response from our APIs consumes ~50% fewer LLM tokens on average. All informational content is preserved: titles, snippets, links, ratings, prices, etc. Internal tracking fields (opaque tokens, redirect URLs, duplicate numeric extractions) are stripped to save tokens.

Markdown vs JSON: Token savings

That means lower LLM costs, faster inference, and more of the context window free for your actual task.

Can you switch between Markdown and JSON in the same application?

Yes, you can switch between using JSON and Markdown for any SerpApi endpoint.

JSON is the default output, so no additional parameters or settings are needed. Any search can return the Markdown output in three ways:

Ways to get Markdown output

If you're using code, this looks like:

# All three produce the same Markdown output:

curl "https://serpapi.com/search?engine=google&q=coffee&api_key=YOUR_KEY&output=md"
curl "https://serpapi.com/search.md?engine=google&q=coffee&api_key=YOUR_KEY"
curl -H "Accept: text/markdown" "https://serpapi.com/search?engine=google&q=coffee&api_key=YOUR_KEY"



> 
> _Ways to get Markdown output_

An application can use different response formats for different tasks. For example, a research agent could consume Markdown for a broad search-and-summarize step, while a separate data collection service uses JSON for structured extraction.

Markdown output works across all 100+ SerpApi APIs — Google, Bing, YouTube, Amazon, Walmart, DuckDuckGo, and every other API we support.

## Conclusion

With SerpApi, you can retrieve search-engine data to support a wide range of applications. Choosing the right response format helps you integrate that data into your workflow with less token consumption and processing time.

**Ready to build?** Explore the [API documentation](https://serpapi.com/) to find the relevant engine, supported parameters, and response fields for your application.

If you have any questions, feel free to reach out to us at [contact@serpapi.com](mailto:contact@serpapi.com).

## Related Posts

- [Introducing SerpApi's New Markdown Output](https://serpapi.com/blog/introducing-serpapi-markdown-output/)

- [Markdown Search Results: SerpApi vs. Exa, Tavily, Firecrawl](https://serpapi.com/blog/markdown-search-results-serpapi-vs-exa-tavily-firecrawl/)

- [How to Fit More Web Search Data into a Local LLM](https://serpapi.com/blog/local-llm-web-search-markdown-output/)

How to Scrape Amazon Promotions Coupons

Noraina Nordin — Fri, 06 Nov 2026 19:58:58 +0000

Amazon coupons live on their own promotion pages. Each page lists the discount, the expiry date, how to claim it, the terms, and every product the coupon applies to. That's useful data if you run a deal site, monitor a competitor's discounts, or want to know the real price of a product after the coupon.

Amazon promotion page

The problem is that Amazon has no public API endpoint that returns a coupon page's terms and qualifying products. Scraping the page yourself is also tricky because the product grid loads with JavaScript, making a DIY scraper harder to maintain.

In this tutorial, you'll use the Amazon Promotion API from SerpApi to retrieve promotion data as clean JSON or Markdown. You'll also export qualifying products to a CSV file and build a small script to check whether your coupons are still active.

What data does the Amazon Promotion API return?

Each request takes one promotion ID and returns a promotion_results object with:

title: the coupon headline, for example Save 25% on your Subscribe & Save subscription.
saving_info: what the discount is and how it's applied.
schedule: when the promotion runs, such as Limited-time offer or a date range.
expiration: the coupon's expiry date.
how_to_claim: the steps a shopper follows to use the coupon.
terms: the terms and conditions, grouped into sections with a title, text, and items.
link: the Amazon promotion page URL.

If you add include_products=true, you also get qualifying_items, the products the coupon applies to. Each item includes its asin, title, brand, price, extracted_price, rating, reviews, prime, merchant_id, and a serpapi_amazon_product_link, which is a ready-made Amazon Product API request for that item.

Here's a trimmed response for promotion ASV27IGU190N with include_products=true:

{
  "promotion_results": {
    "promotion_id": "ASV27IGU190N",
    "title": "Save 25% on your Subscribe & Save subscription",
    "saving_info": "25% extra savings on the first order, applied at checkout. Coupons may be subject to a maximum dollar value, which will be reflected at checkout if applicable.",
    "schedule": "Limited-time offer",
    "expiration": "Coupon Expiry Date January 1, 2027",
    "how_to_claim": [
      "Select the Clip Coupon button.",
      "Add 1 item from the products below to your Cart using the Add to Cart button or select See Options for specific product variations.",
      ...
    ],
    "terms": [
      {
        "title": "How to redeem a coupon:",
        "items": [...]
      },
      ...
    ],
    "link": "https://www.amazon.com/promotion/psp/ASV27IGU190N"
  },
  "qualifying_items": [
    {
      "position": 1,
      "asin": "B0CDHNX333",
      "serpapi_amazon_product_link": "https://serpapi.com/search.json?amazon_domain=amazon.com&asin=B0CDHNX333&device=desktop&engine=amazon_product",
      "title": "Orgain Organic Kids Dairy 8g Protein Shake, Fruity Cereal, 8.25 oz, 12pk | ...",
      "brand": "Orgain",
      "price": "$19.49",
      "extracted_price": 19.49,
      "rating": 4.5,
      "reviews": 644,
      "prime": true,
      "merchant_id": "ATVPDKIKX0DER"
    },
    ...
  ]
}

SerpApi returns Amazon promotion pages in a structured output

Getting started with SerpApi

You need a free SerpApi account before you can use the API. You can upgrade to a paid plan later if you need more searches, faster speeds, or additional features.

First, create an account and verify your email. Then grab your API key from your account dashboard and save it in an environment variable. Set it in your terminal, replacing the placeholder with your real key:

export SERPAPI_API_KEY="YOUR_SERPAPI_API_KEY"

The free plan includes 250 searches per month, and you can try any promotion live in the interactive playground before writing code.

🔐 You should store your API key in a safe location if you're sharing or publishing your code. If it's ever leaked, you can generate a new one from the dashboard.

Install the SerpApi library (optional)

SerpApi has official libraries for Python, JavaScript, Ruby, Java, and more. They're a thin wrapper around the API and aren't required. The API works just as well with plain GET requests, cURL, or fetch() in Node.js.

Review the Amazon Promotion API documentation

The engine is amazon_promotion, and the only required parameter is promotion_id. Other useful parameters:

include_products: set to true to also get the qualifying products. Responses take longer with this on.
amazon_domain: the Amazon marketplace to use, such as amazon.co.uk or amazon.de. Defaults to amazon.com. See the supported Amazon domains. amazon.cn isn't supported because Amazon no longer serves promotion pages there.
language: the page language, such as es_US on amazon.com. See the supported Amazon languages.
output: set to md for a token-efficient Markdown response.

For the full field reference and live examples, see the Amazon Promotion API documentation.

SerpApi's Amazon Promotion API documentation

Find the promotion ID

The promotion ID is in the URL of every Amazon coupon page. Open a coupon from a product page (the "Shop items" or "Terms" link next to the coupon badge), and copy the code after /promotion/psp/ or /promotion/details/popup/:

Promotion ID location

How to scrape Amazon coupons

The results are same across every library, so use whichever method fits your stack.

GET request

This gets the coupon details for a promotion ASV27IGU190N:

https://serpapi.com/search.json?engine=amazon_promotion&promotion_id=ASV27IGU190N&api_key=SERPAPI_API_KEY

By default, you get JSON, but you can request Markdown by adding output=md (or using the /search.md endpoint, or the Accept: text/markdown header). Markdown is handy when you feed coupon terms to an LLM or AI agent:

https://serpapi.com/search.json?engine=amazon_promotion&promotion_id=ASV27IGU190N&output=md&api_key=SERPAPI_API_KEY

Python

This gets the coupon details and prints the headline, expiry date, and claim steps, using the official SerpApi Python library:

Install the library:

pip install serpapi

import os
import serpapi

client = serpapi.Client(api_key=os.environ["SERPAPI_API_KEY"])

results = client.search({
    "engine": "amazon_promotion",
    "promotion_id": "ASV27IGU190N",
})

promotion = results.get("promotion_results", {})

print(promotion.get("title"))
print(promotion.get("expiration"))
for step in promotion.get("how_to_claim", []):
    print(f"- {step}")

Output:

Save 25% on your Subscribe & Save subscription
Coupon Expiry Date January 1, 2027
- Select the Clip Coupon button.
- Add 1 item from the products below to your Cart using the Add to Cart button or select See Options for specific product variations.
- When you're done shopping, select Go to Cart.
- The offer will automatically be applied at checkout, if eligible.

JavaScript and Node.js

This runs the same request with the SerpApi JavaScript library. Install it:

npm install serpapi

Create a new file index.js

import { getJson } from "serpapi";

const results = await getJson({
  engine: "amazon_promotion",
  api_key: process.env.SERPAPI_API_KEY,
  promotion_id: "ASV27IGU190N",
});

const promotion = results.promotion_results ?? {};

console.log(promotion.title);
console.log(promotion.expiration);
for (const step of promotion.how_to_claim ?? []) {
  console.log(`- ${step}`);
}

cURL

This gets the coupon details straight from the command line:

curl --get https://serpapi.com/search \
 -d api_key="SERPAPI_API_KEY" \
 -d engine="amazon_promotion" \
 -d promotion_id="ASV27IGU190N"

Get the products that qualify for a coupon

A coupon often covers dozens of products, and only some variants qualify. Add include_products=true to get the full list. This script reuses the client from above, prints each product, and saves them to qualifying_items.csv:

import os
import serpapi
import csv

client = serpapi.Client(api_key=os.environ["SERPAPI_API_KEY"])

PROMOTION_ID = "ASV27IGU190N"

results = client.search({
    "engine": "amazon_promotion",
    "promotion_id": PROMOTION_ID,
    "include_products": "true",
})

items = results.get("qualifying_items", [])

for item in items:
    print(f"{item.get('asin')} | {item.get('price')} | {item.get('title', '')[:60]}")

with open("qualifying_items.csv", "w", encoding="UTF-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["promotion_id", "asin", "title", "brand", "price", "rating", "reviews", "prime", "link"])
    for item in items:
        writer.writerow([
            PROMOTION_ID,
            item.get("asin"),
            item.get("title"),
            item.get("brand"),
            item.get("extracted_price"),
            item.get("rating"),
            item.get("reviews"),
            item.get("prime", False),
            item.get("link"),
        ])

print(f"Saved {len(items)} qualifying items to qualifying_items.csv")

if results.get("qualifying_items_truncated"):
   print("Note: this list is incomplete. Try again later with no_cache=true.")
elif not items:
   print("This promotion has no product list. It only shows terms.")

Output:

B0CDHNX333 | $19.49 | Orgain Organic Kids Dairy 8g Protein Shake, Fruity Cereal, 8
B00BYOW7VG | $21.21 | Orgain Organic Kids Dairy 8g Protein Shake, Vanilla, 8.25 oz,
...
Saved 95 qualifying items to qualifying_items.csv

Qualifying products exported to CSV

Three things to know about this list:

Not every coupon has a product list. Some coupons, often small ones such as "Apply $1 coupon", only have a terms page. For these, the response has no qualifying_items field, and the title is a generic View promotion details. The script prints a note when that happens.
qualifying_items_truncated is true when the list is incomplete, either because a very large promotion hit the safety cap or because Amazon didn't return the product grid. The second case can be temporary, so try the request again later with no_cache=true.
Prices are shelf prices. extracted_price is the price before the coupon. Use saving_info to calculate the discounted price.

Each item also has a serpapi_amazon_product_link. Call it to get the full product page, including variants, specifications, and reviews. For more on looking up products by ASIN, see our Amazon ASIN lookup guide.

Track Amazon coupons with Python

Coupons end early, run out, or get replaced. If you keep a list of promotion IDs, for example from your own brand, your competitors, or a deal feed, this script checks each one and records whether it's still active.

Here's what the script does:

Checks every promotion ID in PROMOTION_IDS, up to MAX_PROMOTIONS per run
Records the title, savings, and expiry date for active coupons
Marks a coupon as ended when the API says it has ended or wasn't found
Stops with an error for anything else, such as an invalid API key, so a bad key isn't recorded as an ended coupon
Saves the results to coupon_status.json with the date checked

import json
import os
from datetime import date

import serpapi

client = serpapi.Client(api_key=os.environ["SERPAPI_API_KEY"])

PROMOTION_IDS = [
    "ASV27IGU190N",
    "A1C1U2DDI9B2GB",
]
MAX_PROMOTIONS = 20  # each promotion uses one search
ENDED_MESSAGE = "Promotion has ended or was not found."

def check_promotion(promotion_id):
    try:
        results = client.search({
            "engine": "amazon_promotion",
            "promotion_id": promotion_id,
        })
        error = results.get("error")
    except serpapi.HTTPError as http_error:
        error = http_error.error

    if error == ENDED_MESSAGE:
        return {"promotion_id": promotion_id, "status": "ended"}
    if error:
        raise RuntimeError(f"{promotion_id}: {error}")  # e.g. invalid API key

    promotion = results.get("promotion_results", {})
    return {
        "promotion_id": promotion_id,
        "status": "active",
        "title": promotion.get("title"),
        "saving_info": promotion.get("saving_info"),
        "expiration": promotion.get("expiration"),
        "link": promotion.get("link"),
    }

report = {
    "checked_on": date.today().isoformat(),
    "promotions": [check_promotion(pid) for pid in PROMOTION_IDS[:MAX_PROMOTIONS]],
}

for promotion in report["promotions"]:
    print(f"{promotion['promotion_id']} | {promotion['status']} | {promotion.get('title', '')}")

with open("coupon_status.json", "w", encoding="UTF-8") as f:
    json.dump(report, f, indent=2)

active = sum(p["status"] == "active" for p in report["promotions"])
print(f"Checked {len(report['promotions'])} promotions, {active} active. Saved to coupon_status.json")

Output:

ASV27IGU190N | active | Save 25% on your Subscribe & Save subscription
A1C1U2DDI9B2GB | ended | 
Checked 2 promotions, 1 active. Saved to coupon_status.json

💡 MAX_PROMOTIONS caps the number of searches per run. Lower it while testing so you don't use up your monthly searches.

There are a few ways you can extend this script:

Run it on a schedule. Use cron or Task Scheduler to run it daily and compare each run with the last one.
Alert on changes. Send a message when a coupon ends or its saving_info changes.
Add the products. Set include_products to true to also track which products join or leave a promotion.

FAQ

Does Amazon have a coupon API?

Not a public one for coupon pages. Sellers can create and report on their own coupons in Seller Central, and affiliates get product and offer data through Amazon's Product Advertising API. Neither returns a public coupon page with its terms and qualifying products. This Amazon Promotion API fills that gap.

Where do I find an Amazon promotion ID?

In the coupon page URL. It's the code after /promotion/psp/ or /promotion/details/popup/, for example ASV27IGU190N.

Can I find every Amazon coupon with this API?

No. It returns one promotion per request, so you need the ID first. To spot products that have a coupon, use the Amazon Search API. Its organic results include a save_with_coupon field, such as Save 10% with coupon, but not the promotion ID.

What happens when a coupon ends?

The API returns an error: Promotion has ended or was not found. The tracking script above records these as ended.

Does it work on other Amazon marketplaces?

Yes. Set amazon_domain to the marketplace you need, such as amazon.co.uk or amazon.co.jp. Every supported Amazon domain works except amazon.cn.

Conclusion

The Amazon Promotion API turns any Amazon coupon page into structured JSON or Markdown, so you can check coupon terms, export qualifying products, and track which coupons are still active without maintaining your own scraper. You can then pair it with the Amazon Product API to get full details for each qualifying product, or use it alongside an Amazon price tracker to see the real price after a coupon.

If you need help getting started, contact us. We're happy to help.

Ecommerce Web Scraping: How to Track Competitor Prices, Products, and Stock

Srujana Maddula — Thu, 29 Oct 2026 11:01:45 +0000

Almost all major e-commerce players already scrape each other’s sites, and they feed that data straight into pricing intelligence, inventory stocking and supply chain optimization. If you don't do it, you’re behind.

In this guide, I’ll show you how ecommerce web scraping works and which data is worth scraping. Even if you can’t pull entire catalogs, you’ll learn the two signals everyone should collect at a minimum. You’ll also get a working script that scrapes a popular e-commerce site in Python. And if you have another marketplace like Amazon or Walmart in mind, I have a solution for that too.

What Ecommerce data is worth scraping?

You can scrape almost anything from an online store. The question is what’s worth the effort. In this section, the first two give you a rich dataset, and the last two are the minimum every store should be scraping.

Product detail pages

Scrape product pages

By scraping single product pages, you get title, brand, category, price, availability, customer reviews, product description, and much more about an item.

Start with how they sell their best sellers. That means compare the descriptions of their top products with their slow movers. Note which features they lead with and which claims they repeat. Then test those same moves on your own listings.

The product detail page also carries the customer reviews. Reviews tell you which features buyers care about, what breaks or disappoints them, and whether they felt the price was fair. If the same phrase keeps showing up, like “too expensive for what it is,” a better-priced alternative could win those buyers. If reviews keep praising a product as “great value for the price,” you learn what shoppers consider fair and can position your own products around it.

And here’s the clever part. You’ll never see a competitor’s real sales numbers. But review counts are a decent stand-in. When a product that usually gets a handful of reviews a week suddenly gets dozens, something has happened: a sales spike, a promotion or a product going viral on TikTok. Spotting it early tells you what’s about to be in demand.

You’ll also see which category/department each product belongs to. Compare them with your own catalog to fix misclassified products and to spot the categories and products you’re missing.

Search results listings

Scrape ecommerce search results

When shoppers search for a product on a marketplace, the results page shows what the platform thinks is relevant to their query. Those listings are worth scraping.

You can collect product titles, images, prices, rankings, and other listing details for a search query. This shows how competitors position their products for different search terms, so you can benchmark your own listings, sharpen your titles and descriptions, and spot gaps in your assortment.

For example, say Amazon's top results for "cordless drill 20V" include bestseller brands or models that Walmart doesn't have. That could be a buying opportunity: stock those products or consider a comparable alternative under your own brand.

Search results also tells you how a marketplace interprets a query. If the top results for "air fryer" mostly feature large-capacity models, that's a clue about what shoppers may expect to find for that search. If your site keeps surfacing irrelevant products for the same query, you identified to fix a search-relevance problem.

The above two broad categories: product pages and search results are the two richest datasets. But if you're a smaller retailer and don't need that much data, or don't have the budget to build a comprehensive competitor dataset yet? Start with the price and stock data fields that are useful to almost every e-commerce business.

Product prices

Price is the first thing a shopper weighs before buying, and it’s the most tracked field. In a June 2024 letter to Walmart and Amazon, Senator Sherrod Brown pointed out that some retailers using pricing algorithms change their prices more than 2,000 times a day.

So scrape their prices regularly, and decide in time whether to match a drop, offer a better deal, or hold your price because your margins matter more. Over time, that price history also shows you how they run promotions, so you can plan your own pricing instead of only reacting to theirs. If you only get to track one thing, track this.

Stock availability

When a competitor runs out of a product people want, those shoppers look somewhere else. If you’re watching, that somewhere else can be you. Restock it, feature it, or sharpen your price while they’re empty, and you catch the demand they couldn’t serve.

Why is Ecommerce web scraping difficult?

Ecommerce sites are some of the most heavily defended sites on the web. Basic defenses like IP rate limiting and User-Agent blocking are common across the web. But ecommerce sites often go further, using advanced browser fingerprints and other signals to spot automated traffic.

Big retailers rely on bot-protection services like Akamai, DataDome and Cloudflare, which look far beyond your IP. They check whether your connection really looks like Chrome, whether your browser has the quirks a normal one would, and whether you scroll, click and pause like a person. A script that loads page after page at perfect intervals stands out right away.

And even when you get that far, a CAPTCHA can stop you in your tracks. Sites use these challenges when they suspect a visitor is a bot, sometimes alongside other checks that run in the background.

So a generic automation script won't help pass such sophisticated defense systems. Even if you try to build the first class scraper yourself, you have to spend on proxies and IPs. To remove that hassle, you can try the below simple API call approach to get any e-commerce data regularly.

Scraping an Ecommerce website in Python

You can send any public URL to SerpApi's Extract API and get the page back as structured JSON or Markdown. In this example, you’ll start from a Best Buy search results page, find the products on it, and pull each product’s title, price and stock status into clean JSON.

Prerequisites

You’ll need:

Python 3.8 or later
A code editor like VS Code, or a Jupyter notebook
A SerpApi account, which comes with 250 free searches per month.

Set up SerpApi

Install the SerpApi Python client:

pip install serpapi

Then save your API key as an environment variable, so it never ends up in your code:

export SERPAPI_API_KEY="your_api_key"

Choose your target site and the data you want

In this case, it is Best Buy’s search results for the search term "samsung galaxy": https://www.bestbuy.com/site/searchpage.jsp?id=pcat17071&st=samsung%20galaxy

The search page shows a grid of products, but the details you want live on each product’s own page. So the script works in two passes:

Fetch the search page and collect the product links.
Visit each product page and pull out the data.

For each product, you’ll collect:

Title
Product URL
Price
Stock availability

Unlike CSS-selector based ecommerce data scraping, you don’t need to open DevTools and hunt for class names. The Extract API reads the page for you, including the structured product data most big stores embed for Google.

Build the Ecommerce web scraper

Create a file called best_buy_scraper.pyand add the below code we discuss in each step.

1. Import the packages and set your target

import html
import json
import os
import sys
from urllib.parse import urlsplit, urlunsplit

import serpapi


PRODUCT_LIMIT = 3
SEARCH_PAGE_URL = "https://www.bestbuy.com/site/searchpage.jsp?id=pcat17071&st=samsung%20galaxy"

PRODUCT_LIMIT sets how many products you want back. SEARCH_PAGE_URL is your search term result page of Best Buy. To track another keyword listings, change st=samsung%20galaxy to any search term.

2. Filter the product links

def bestbuy_product_links(data):
    links = []
    seen = set()
    headings = {
        item.get("position"): html.unescape(item["title"])
        for item in data.get("headings", [])
        if isinstance(item, dict)
        and isinstance(item.get("position"), int)
        and isinstance(item.get("title"), str)
    }
    for item in data.get("inline_links", []):
        if not isinstance(item, dict) or not isinstance(item.get("link"), str):
            continue
        parsed = urlsplit(item["link"])
        hostname = (parsed.hostname or "").lower()
        if parsed.scheme not in ("http", "https"):
            continue
        if hostname != "bestbuy.com" and not hostname.endswith(".bestbuy.com"):
            continue
        if not parsed.path.lower().startswith("/product/"):
            continue

        link = urlunsplit((parsed.scheme, parsed.netloc, parsed.path, parsed.query, ""))
        if link not in seen:
            seen.add(link)
            links.append({
                "link": link,
                "title": item.get("title") or headings.get(item.get("position")),
            })
    return links

Here’s what each check does:

parsed.scheme: drops anything that isn’t a regular web link, like mailto: or javascript: links.
hostname: keeps only links on bestbuy.com or its subdomains, so ads pointing to other sites are out.
parsed.path.startswith("/product/"): keeps only product pages. Every Best Buy product URL follows this pattern.
urlunsplit(...) and seen: strips the #fragment from each URL and skips duplicates. The same product often appears twice on a page, once on its image and once on its title.
headings: if a link has no title of its own, the function borrows the title from the nearest heading on the page.

If you adapt this script for another store, this is the function to change. Amazon product URLs, for example, contain /dp/ instead of /product/.

Since the Extract API is a generic API for all web pages, you need to parse Best Buy URLs to find product links and then parse the JSON output to extract the required fields.

However, if you want to scrape major retailers, SerpApi provides dedicated Walmart , Amazon , and eBay APIs, along with specialized APIs for other platforms. A single call to any of these retail APIs returns structured JSON with fields like price, title, product ID, and much more, making the data much easier to work with.

SerpApi's 250 free searches per month give you access to all these APIs. Check out these guides to scraping Walmart and Amazon data.

3. Pull out price and stock

This function takes the Extract API’s response for one product page and returns the output:

def extract_prices(data, result):
    if (
        data.get("error")
        or data.get("search_metadata", {}).get("status") == "Error"
    ):
        raise RuntimeError(
            data.get("error") or "SerpApi Extract request failed"
        )

    product = next(
        (
            item for item in data.get("json_ld", [])
            if isinstance(item, dict) and item.get("@type") == "Product"
        ),
        {},
    )
    offer = product.get("offers") or {}
    if isinstance(offer, list):
        offer = offer[0] if offer else {}
    if not isinstance(offer, dict):
        offer = {}

    page_offer = next(
        (
            item for item in data.get("price_offers", [])
            if isinstance(item, dict)
            and item.get("extracted_price") is not None
        ),
        {},
    )
    current_price = offer.get("price")
    if current_price is None:
        current_price = page_offer.get("extracted_price")

    currency = offer.get("priceCurrency") or page_offer.get("currency")
    if current_price is not None and currency:
        amount = str(current_price)
        if amount.endswith(".00"):
            amount = amount[:-3]
        symbol = "$" if currency == "USD" else f"{currency} "
        current_price = f"{symbol}{amount}"

    availability = offer.get("availability") or page_offer.get("availability")
    if availability:
        availability = str(availability).rsplit("/", 1)[-1]
        availability = {
            "instock": "In stock",
            "outofstock": "Out of stock",
        }.get(availability.lower(), availability)

    return {
        "title": (
            product.get("name")
            or (data.get("page_information") or {}).get("title")
            or result.get("title")
        ),
        "url": result["link"],
        "regular_price": current_price,
        "availability": availability,
    }

The function looks for the price in two places

json_ld: most large stores embed structured product data in their pages so Google can show prices in search results. The function finds the item marked "@type": "Product" and reads the price, currency and stock status from its offers. This is the most reliable source, because it’s built for machines to read.
price_offers: if the structured data has no price, the function falls back to the prices the Extract API found in the visible page content.

4. Handle failed pages and extract successful product pages

When you scrape many pages, a few may fail: a timeout, a removed product, a blocked request. Handle them like below:

def error_detail(error):
    if isinstance(error, serpapi.HTTPError):
        return f"HTTP {error.status_code}: {error.error}"
    return str(error) or "request timed out"


def extract_product(client, product):
    try:
        data = client.search({"engine": "extract", "url": product["link"]})
        return extract_prices(data, product)
    except (serpapi.HTTPError, serpapi.TimeoutError, RuntimeError) as error:
        print(f"Skipping {product['link']}: {error_detail(error)}", file=sys.stderr)
        return None

extract_product() fetches each product page with one Extract API call per product. If anything goes wrong, it prints which URL failed and returns None, so the main loop can skip it.

5. Run the ecommerce scraper

Run the scraper: python best_buy_scraper.py

Here’s the output:

You can each product's title, url, price, and availability, ready to save to a file, load into a spreadsheet or feed into a price tracker. Run it on a schedule and compare each day’s results with the last, and you have a basic competitor price and stock monitor.

Key takeaways

Ecommerce data scraping is now the cost of staying competitive. Dynamic pricing is a baseline, catalogs adapt to customer demand, and shoppers have plenty of alternatives, so working from stale competitor data costs you sales.

The hard part isn’t writing the scraper, it’s the infra to maintain. Ecommerce sites are among the hardest to scrape, handling proxies, blocks and parsers that break whenever a page changes cost you engineering overload and resource drains.

As the code in this guide shows, you send a simple request to SerpApi and get structured data back into your pipeline, with none of that infrastructure to maintain. It includes 250 free searches a month, so try it out on your competitor site.

How to Scrape Booking.com Hotel Listings and Prices (2026 Tutorial)

Noraina Nordin — Tue, 27 Oct 2026 02:42:49 +0000

Say you want to track what a three-night stay in New York costs, or compare how hotels and vacation rentals are priced across a city. On Booking.com, you'd type a destination, pick your dates, and scroll. Manually extracting this data is slow and difficult to scale, especially across multiple destinations and date ranges. Replicating this process in code usually requires building and maintaining your own Booking.com scraper: rendering JavaScript pages, getting past anti-bot checks, fixing your parser when the HTML changes, and dealing with results that Booking.com ranks differently for every visitor.

SerpApi’s Booking.com Search API handles that for you and returns the results as structured JSON or Markdown. The same way you would do on manual search, you can search for a destination based on your travel dates, number of guests, and preferred currency. You’ll receive a list of properties with prices, review scores, room options, policies, and the filters Booking.com offers for that search. This data can be used to build a price tracker, market research dashboard, or travel assistant.

Booking.com interface

In this tutorial, you'll make your first request with cURL, Python, and JavaScript, then save the results to CSV, page through more properties, and filter by price and amenities.

What data can you retrieve with the Booking.com Search API?

Each search returns a properties array with the places to stay that Booking.com shows for your destination and dates. The most useful data includes:

Property details: name, type (such as Hotel or Apartment), and a link to the listing on Booking.com.
Prices: price for the whole stay, price_per_night, taxes_and_fees that aren’t included in the price, and the currency they’re in.
Reviews: the review score in rating (out of 10) and the number of reviews.
Location: location (district and city) and gps_coordinates with latitude and longitude.
Policies: whether the property offers free_cancellation, plus the rooms recommended for your guests in rooms.
Filters: the filters array lists the filters Booking.com offers for your search, which you can reuse in your next request (more on that below).

Prices depend on dates. If you don't set check_in_date and check_out_date, Booking.com returns properties without prices.

SerpApi's Booking.com Search API documentation

For the full list of fields and parameters, see the Booking.com Search API documentation.

How to scrape Booking.com hotel listings

You can call the Booking.com Search API from the playground, a plain GET request, cURL, Python, or JavaScript. Every method returns the same results.

Try it in the playground

The quickest way to explore the API is the SerpApi interactive playground. This example searches New York for a three-night stay:

engine: booking
q: New York
check_in_date: 2026-11-22
check_out_date: 2026-11-25

SerpApi's Booking.com search playground

Run the search and look at the properties array: compare price with price_per_night and taxes_and_fees, open the rooms of a few properties, and scroll down to filters to see which neighborhoods, property types, and amenities you can filter by.

💡 Swap in your own destination and future dates as you experiment. The q parameter accepts a city, region, district, landmark, or airport (such as Paris, Tuscany, Manhattan, or Eiffel Tower). However, Booking.com doesn't return results for most street addresses.

Get your API key

To call the API from your own code, register for a SerpApi account and copy your private API key from the API key page in your dashboard.

The examples below read the key from an environment variable. Set it in your terminal, replacing the placeholder with your real key:

export SERPAPI_API_KEY="YOUR_SERPAPI_API_KEY"

You should store your API key in a safe location if you're sharing or publishing your code. If it's ever leaked, you can generate a new one from the dashboard.

Make a GET request

The API endpoint is https://serpapi.com/search with engine=booking. Here's the same New York search as a plain GET request. JSON is the default output:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&api_key=YOUR_SERPAPI_API_KEY

To get Markdown instead, add output=md:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&api_key=YOUR_SERPAPI_API_KEY&output=md

Markdown output is a token-efficient format with tables, Markdown links, and YAML frontmatter, which works well when you pass results to an LLM or AI agent. Use JSON when your code needs exact field access, such as reading extracted_price or gps_coordinates.latitude. The rest of this tutorial uses JSON.

cURL implementation

Here's the equivalent request in cURL, using the environment variable you set earlier:

curl --get https://serpapi.com/search \
 -d engine="booking" \
 -d q="New+York" \
 -d check_in_date="2026-11-22" \
 -d check_out_date="2026-11-25" \
 -d api_key="YOUR_SERPAPI_API_KEY"

Here's what each parameter does:

engine=booking selects the Booking.com Search API.
q is the destination to search. It's required.
check_in_date and check_out_date set the stay in YYYY-MM-DD format. They're optional, but if you use them, they must be used together to get prices. The check-out date can be up to 90 nights after check-in.
api_key is your private SerpApi key. It's required.

By default, the search is for 2 adults in 1 room. You can change that with adults, children, children_ages, and rooms, all documented in the Booking.com Search API documentation.

Python implementation

We'll use the requests library. Install it first:

pip install requests

Create a file named main.py:

import os
import sys

import requests

SERPAPI_API_KEY = os.environ.get("SERPAPI_API_KEY")

if not SERPAPI_API_KEY:
    sys.exit("Set the SERPAPI_API_KEY environment variable first.")

params = {
    "engine": "booking",
    "q": "New York",
    "check_in_date": "2026-11-22",
    "check_out_date": "2026-11-25",
    "api_key": SERPAPI_API_KEY,
}


def fetch_properties(params):
    try:
        response = requests.get("https://serpapi.com/search", params=params, timeout=60)
    except requests.RequestException as error:
        # Only print the error type: the full message can contain the request URL and API key.
        sys.exit(f"Request failed: {type(error).__name__}")

    try:
        data = response.json()
    except ValueError:
        sys.exit(f"Unexpected response (HTTP {response.status_code}).")

    if response.status_code != 200 or data.get("error"):
        sys.exit(f"SerpApi error (HTTP {response.status_code}): {data.get('error', 'Unknown error')}")

    return data


data = fetch_properties(params)
properties = data.get("properties", [])

if not properties:
    print("The search succeeded, but no properties were returned.")
    sys.exit()

info = data.get("search_information", {})
print(f"Destination: {info.get('destination')} ({info.get('total_results', 'unknown')} properties found)\n")

for prop in properties[:5]:
    print(f"{prop.get('position')}. {prop.get('name')} ({prop.get('type')})")
    print(f"   Location: {prop.get('location')}")
    print(f"   Review score: {prop.get('rating', 'n/a')} {prop.get('rating_text', '')} from {prop.get('reviews', 0)} reviews")
    print(f"   Stay price: {prop.get('price', 'n/a')} ({prop.get('price_per_night', 'n/a')} per night)")
    print(f"   Free cancellation: {prop.get('free_cancellation', False)}\n")

Run it:

python main.py

The script prints the destination and the first five properties. A few notes on what you're seeing:

price is the price of the whole stay. By default, it's in each property's currency, which is returned in currency.
price_per_night is Booking.com's average nightly rate. It can exclude charges included in price, so it isn't necessarily the stay price divided by the number of nights.
rating is the review score out of 10, and reviews is the number of reviews behind it.

First five properties found in New York

The script uses .get() for each field because optional fields such as price, rating, or deals aren't returned for every property. It also keeps three outcomes separate: a failed request or API error stops the script with a message, a successful search with no properties says so, and a successful search prints results. If a search fails, SerpApi returns a message in the error field.

Export results to CSV

Property listings fit nicely into rows. To save them, add import csv to the imports at the top of main.py, then add this function below fetch_properties:

def save_csv(properties, filename="booking_properties.csv"):
    fieldnames = [
        "position", "property_id", "name", "type", "rating", "reviews",
        "price", "extracted_price", "currency", "free_cancellation",
        "latitude", "longitude", "room_name", "deals", "link",
    ]
    with open(filename, "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()
        for prop in properties:
            gps = prop.get("gps_coordinates", {})
            rooms = prop.get("rooms", [])
            writer.writerow({
                "position": prop.get("position"),
                "property_id": prop.get("property_id"),
                "name": prop.get("name"),
                "type": prop.get("type"),
                "rating": prop.get("rating"),
                "reviews": prop.get("reviews"),
                "price": prop.get("price"),
                "extracted_price": prop.get("extracted_price"),
                "currency": prop.get("currency"),
                "free_cancellation": prop.get("free_cancellation"),
                "latitude": gps.get("latitude"),
                "longitude": gps.get("longitude"),
                # Keep only the first recommended room and join deals into one cell.
                "room_name": rooms[0].get("name") if rooms else None,
                "deals": "; ".join(prop.get("deals", [])),
                "link": prop.get("link"),
            })
    print(f"Saved {len(properties)} properties to {filename}")

Finally, add this line at the end of the file, after the for loop:

save_csv(properties)

Run python main.py again to create booking_properties.csv. The nested gps_coordinates object becomes two columns, the deals array is joined into one cell, and only the first room from the rooms array is kept. Use extracted_price instead of price when you want to sort or compare numbers in a spreadsheet.

Properties found in New York in CSV format

JavaScript implementation

For Node.js, we'll use SerpApi's official JavaScript library. Install it:

npm install serpapi

Create a file named main.cjs. The .cjs extension tells Node.js to treat it as a CommonJS module, so require works without extra configuration:

const { getJson } = require("serpapi");

const apiKey = process.env.SERPAPI_API_KEY;

if (!apiKey) {
  console.error("Set the SERPAPI_API_KEY environment variable first.");
  process.exit(1);
}

async function main() {
  let data;
  try {
    data = await getJson({
      engine: "booking",
      q: "New York",
      check_in_date: "2026-11-22",
      check_out_date: "2026-11-25",
      api_key: apiKey,
    });
  } catch (error) {
    // Redact the key in case the error message includes the request URL.
    console.error("Request failed:", String(error.message).replaceAll(apiKey, "[REDACTED]"));
    process.exit(1);
  }

  if (data.error) {
    console.error("SerpApi error:", data.error);
    process.exit(1);
  }

  const properties = data.properties || [];
  if (properties.length === 0) {
    console.log("The search succeeded, but no properties were returned.");
    return;
  }

  const info = data.search_information || {};
  console.log(`Destination: ${info.destination} (${info.total_results ?? "unknown"} properties found)\n`);

  for (const property of properties.slice(0, 5)) {
    console.log(`${property.position}. ${property.name} (${property.type})`);
    console.log(`   Location: ${property.location}`);
    console.log(`   Review score: ${property.rating ?? "n/a"} ${property.rating_text ?? ""} from ${property.reviews ?? 0} reviews`);
    console.log(`   Stay price: ${property.price ?? "n/a"} (${property.price_per_night ?? "n/a"} per night)`);
    console.log(`   Free cancellation: ${property.free_cancellation ?? false}\n`);
  }
}

main();

Run it:

node main.cjs

You'll see the same fields as the Python version. If you’d rather pass results straight to an LLM, use the output=md GET request shown earlier.

Other languages and no-code options

SerpApi has libraries for more programming languages, listed on the integrations page. Since the Booking.com Search API is a regular GET request with engine=booking, any HTTP client also works.

If you prefer no-code tools:

SerpApi's Make.com app has native modules for some APIs and a Universal "Make an API Call" module where you set the request path and each parameter yourself. Pagination isn't automated in the Universal module.
SerpApi's verified n8n node launched with 22 popular APIs. For APIs that aren't in the node, the announcement shows how to use n8n's HTTP Request node with the engine parameter, for example engine=booking.

How to paginate the results

A search returns 25 properties by default. Here's how to get more:

offset skips a number of results. With the default page size, offset=25 is page 2 and offset=50 is page 3. It can go up to 1,000; beyond that, Booking.com returns the same properties.
limit sets the number of properties per page, from 1 to 100. Booking.com sometimes returns a few extra properties, such as sponsored ones.
session_token keeps the same Booking.com visitor between requests. Booking.com ranks properties differently for each visitor, so without it, later pages can repeat properties from earlier ones. It's returned in serpapi_pagination.session_token.
serpapi_pagination.next is a ready-made link to the next page. It already includes the next offset, the session_token, and your original parameters.

For example, the second page of the New York search looks like this, where SESSION_TOKEN_FROM_PAGE_1 is the serpapi_pagination.session_token value from the first response:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&offset=25&session_token=SESSION_TOKEN_FROM_PAGE_1&api_key=YOUR_SERPAPI_API_KEY

In code, the simplest approach is to follow the next link and add your API key to it. This paginate.py script fetches up to three pages, stops early when there’s no next page or a page comes back empty, and saves all properties to a JSON file:

import json
import os
import sys

import requests

SERPAPI_API_KEY = os.environ.get("SERPAPI_API_KEY")

if not SERPAPI_API_KEY:
    sys.exit("Set the SERPAPI_API_KEY environment variable first.")

MAX_PAGES = 3  # Each page is a separate search request.

params = {
    "engine": "booking",
    "q": "New York",
    "check_in_date": "2026-11-22",
    "check_out_date": "2026-11-25",
    "api_key": SERPAPI_API_KEY,
}

url = "https://serpapi.com/search"
all_properties = []

for page in range(1, MAX_PAGES + 1):
    try:
        response = requests.get(url, params=params, timeout=60)
    except requests.RequestException as error:
        sys.exit(f"Request for page {page} failed: {type(error).__name__}")

    try:
        data = response.json()
    except ValueError:
        sys.exit(f"Unexpected response on page {page} (HTTP {response.status_code}).")

    if response.status_code != 200 or data.get("error"):
        sys.exit(f"SerpApi error on page {page} (HTTP {response.status_code}): {data.get('error', 'Unknown error')}")

    properties = data.get("properties", [])
    all_properties.extend(properties)
    print(f"Page {page}: {len(properties)} properties")

    next_link = data.get("serpapi_pagination", {}).get("next")
    if not properties or not next_link:
        break

    # The next link already carries offset, session_token, and the original filters.
    url = next_link
    params = {"api_key": SERPAPI_API_KEY}

with open("booking_properties.json", "w", encoding="utf-8") as file:
    json.dump(all_properties, file, indent=2, ensure_ascii=False)

print(f"Saved {len(all_properties)} properties to booking_properties.json")

Run it with python paginate.py. Each page is a separate search and can use a search credit, so keep MAX_PAGES small while you're testing. If you need more properties per request, raise limit (up to 100) instead of fetching many small pages.

Customize or filter the results

The Booking.com Search API supports the sorting and filters you'd use on Booking.com itself. Here are three useful ones:

sort_by: Changes the order of results. The default is popularity ("Our top picks"). Other options include price (lowest first), bayesian_review_score (top reviewed), review_score_and_price, and distance_from_search (distance from downtown).
min_price and max_price: Limit the price per night, in the currency of the results. They must be used together with a currency code, such as USD or EUR. See the Booking.com currencies page for supported codes.
nflt: Applies any other Booking.com filter, such as amenities, meals, neighborhoods, or brands. It's a semicolon-separated list of filter values. You don't need to guess them: take the value from filters[].options in a previous response. For example, the documentation uses di=1127 for London's Theatreland neighborhood.

Here's a New York search for stays between 100 and 300 USD per night, sorted by lowest price, limited to properties with a filter value taken from the filters array:

https://serpapi.com/search?engine=booking&q=New+York&check_in_date=2026-11-22&check_out_date=2026-11-25&currency=USD&min_price=100&max_price=300&sort_by=price&nflt=roomfacility%3D38&api_key=YOUR_SERPAPI_API_KEY

In this example, roomfacility=38 is the "Private bathroom" option from the documentation's New York sample. Note that nflt values need URL encoding (= becomes %3D and ; becomes %3B); requests and the JavaScript library handle that for you.

Other handy filters include free_cancellation=true, rating for a minimum review score (9, 8, 7, or 6), hotel_class for star ratings (for example, 4,5), and property_types (for example, 201,220 for apartments and vacation homes). You can also localize results with hl for the language and currency for prices, or set device=mobile to get Booking.com's mobile results, which can include mobile-only prices. See the Booking.com Search API documentation for the full list.

Conclusion

You can now scrape Booking.com hotel listings for any destination and dates, read their prices, review scores, rooms, and policies, save them to CSV, page through more results with session_token, and narrow searches with sorting and filters. You can also switch to Markdown output with output=md when the results are going to an LLM.

To compare Booking.com prices with other sources, see how to scrape Google Hotels data, scrape Airbnb data, and scrape Tripadvisor.

Frequently asked questions

Which parameters are required?

Only engine=booking, q (the destination), and api_key. Everything else is optional, but you'll usually want check_in_date and check_out_date, because Booking.com returns properties without prices when dates aren't set.

Why are prices in different currencies?

By default, currency is hotel_currency, so each property's price is in its own currency (for example, EUR in Paris). Set currency to a code such as USD to get all prices in one currency. The currency field on each property tells you which one was used.

Can I search flexible dates?

Yes. Set flex_window to 1, 2, 3, or 7 to let the check-in and check-out dates move up to that many days earlier or later for the best price of each property. Each property then includes the check_in_date and check_out_date of its price. flex_window must be used with both dates.

Is this the official Booking.com API?

No. The Booking.com Search API is a SerpApi API that returns Booking.com's public search results (the same properties, prices, and filters you see on Booking.com) as structured data. It's a separate product from the APIs Booking.com offers its own partners, and you only need a SerpApi API key to use it.

Does the Booking.com Search API support Markdown output?

Yes. Add output=md to get Markdown, which is useful for LLMs and AI agents. You can also use the /search.md endpoint or the Accept: text/markdown request header. See the Markdown output page for details.

From JSON to Markdown: Reducing AI SEO Analysis Token Costs by 90%

Magenta Qin — Sun, 25 Oct 2026 10:33:25 +0000

AI SEO analysis pipelines usually run into two problems quickly.

First, the more data you analyze, the more input tokens you send to the model. Adding more keywords, markets, competitors, pages, or result rows makes every prompt longer. That may not matter for a one-off analysis, but it matters when a pipeline runs thousands of times a day.

Second, cutting the input down can make the analysis less useful. When the full dataset does not fit into the model context, you might often split it into batches, summarize each batch, or rely on agent memory. Those techniques help, but they can not replace the original data. If positions 1-10 and positions 21-30 land in different batches, the model may never compare them side by side.

The core question is simple: Can we help the model see more SEO evidence in the same token budget?

In this post, I will compare JSON and Markdown as API response formats. I will first reproduce a token benchmark for Google Shopping results. Then I will run a practical SEO analysis demo to see whether the larger sample changes the model's recommendation. The demo code is available here.

Why build an AI SEO analysis pipeline?

Existing SEO tools already provide rich datasets, dashboards, reports, and AI features. For many workflows, that is enough.

But internal SEO questions often depend on business context that a general-purpose SEO platform does not have. For example:

Why did a competitor's product enter the top results for this query?
Which ranking changes should the team handle today?
Does this search result change matter for revenue, conversion rate, inventory, or a product launch?

When you build your own AI SEO analysis pipeline, you can choose the market, device, timestamp, query set, and data source. You can also combine search data with internal business data before asking the model to reason over it.

The goal is not to rebuild a full SEO platform. The goal is to add a decision layer that matches your company's workflow.

The hidden cost of API response format

Most APIs return JSON by default. JSON is excellent for code. It gives programs exact fields, predictable structure, and a reliable contract. But when the next consumer is an AI model, the response format also affects context usage. Consider a simplified Google Shopping result in JSON:

{
  "shopping_results": [
    {
      "position": 1,
      "title": "Trail Runner X",
      "price": "$89",
      "rating": 4.7
    },
    {
      "position": 2,
      "title": "Aero Sprint",
      "price": "$74",
      "rating": 4.5
    }
  ]
}

The same information can be represented as a Markdown table:

## Shopping Results

| Position | Title | Price | Rating |
|---:|---|---:|---:|
| 1 | Trail Runner X | $89 | 4.7 |
| 2 | Aero Sprint | $74 | 4.5 |

The JSON response repeats keys such as position, title, price, and rating for every item. It also includes quotes, commas, braces, and nesting. The Markdown table keeps the field names in the header and uses rows for the values. Both formats describe the same result set, but Markdown often uses fewer tokens. That means the same model context can fit more search results before batching becomes necessary.

This is why Markdown is moving beyond documentation. In April 2025, Cloudflare added a Browser Rendering API endpoint that converts webpages into Markdown. In February 2026, Notion documents page content access as Markdown in its data APIs. In August 2026, SerpApi supports Markdown output for search results by using output=md.

The shared pattern is clear: API responses are no longer consumed only by programs. They may go directly into an AI agent's context.

Benchmark: How many input tokens can Markdown save?

SerpApi published a Markdown output benchmark comparing JSON and Markdown responses across several APIs. The result depends on the API, query, returned fields, and tokenizer, but the largest savings appear in more complex result sets.

In that benchmark:

These numbers compare input tokens only. If a model provider charges the same price per input token, reducing input tokens reduces that part of the cost. The total cost of a full AI pipeline will still depend on model choice, output length, orchestration, retries, caching, and other API calls. I wanted to reproduce the Google Shopping result with my own code.

Reproducing the Google Shopping token benchmark

The reproduction changes only one variable: response format. I prepared 12 Google Shopping queries. For each query, I used the same location, language, and device settings. Then I requested the same Google Shopping results twice through SerpApi: once as JSON and once as Markdown.

I sent the raw API responses to the same DeepSeek model and read usage.prompt_tokens from the DeepSeek API response. That value reports the actual number of input tokens the model received. I disabled thinking mode because this experiment measures input representation, not reasoning quality. I did not ask DeepSeek to solve an SEO problem in this benchmark. I used it as the token accounting endpoint.

The calculation was:

saving = 100 * (1 - markdown_tokens / json_tokens)

Let's execute it:

0:00
/0:18
1×

Across the 12 queries, Markdown reduced input tokens by an average of 89.55%. Individual queries ranged from 86.59% to 91.20%. That closely matches the published Google Shopping benchmark of about 90%.

The run saves intermediate evidence so the result can be checked later:

Output	Purpose
`raw/.json` and `raw/.md`	Original API responses
`deepseek-usage/*.json`	DeepSeek token usage receipts
`results.csv` and `results.json`	Per-query comparison data
`summary.md`	Final summary table

For this sample, Markdown compressed the Google Shopping inputs to roughly one-tenth of the JSON token usage.

The next question is more important for AI SEO: if Markdown lets the model see more results in the same budget, can that change the analysis?

A practical SEO demo: Trail running shoes

For the second experiment, I used a practical Google Shopping question: I sell trail running shoes. To show up higher in Google Shopping, should I optimize titles, lower prices, or get more reviews?

I fixed the search setup:

Setting	Value
Query	`trail running shoes`
Market	New York, United States
Language	English
Device	Desktop
Fields kept	position, title, price, rating, reviews, source
Input budget	1,550 prompt tokens
Model	Same DeepSeek model and prompt for both formats

SerpApi returned the same Google Shopping snapshot with 30 products. Python then measured how many products could fit into the 1550-token budget for each format.

Let's run it:

0:00
/0:15
1×

The result:

Format	Products that fit	Prompt tokens used	Position coverage
JSON	16	1,548	1-16
Markdown	30	1,515	1-30

With the same token budget, Markdown let DeepSeek see 1.88x more products.

The larger sample changed the recommendation

The JSON sample only covered positions 1-16. In that sample, the average query-term coverage in product titles was:

Sample	Query-term coverage
Top positions 1-5	27%
Lower positions 12-16	20%

That difference was small. DeepSeek treated the title signal as weak and did not see titles as a clear explanation for position.

The Markdown sample covered all 30 products. With the full sample, the title pattern became clearer:

Sample	Query-term coverage
Top positions 1-10	47%
Lower positions 21-30	17%

With the additional 14 products, DeepSeek gave a more specific recommendation: test title changes for lower-position products by adding terms such as "trail running" and gender-specific modifiers. Price and review count did not show meaningful rank relationships in this snapshot:

Metric	JSON sample	Markdown sample
Spearman position vs. price	0.001	0.155
Spearman position vs. reviews	0.118	-0.031

So the supported next step was not to cut prices or chase more reviews for ranking. The better next step was to run a title A/B test.

This does not prove that title changes cause higher Google Shopping positions. A single search snapshot can only generate hypotheses. But it does show how a compact response format can expose a signal that a smaller sample hides.

Markdown did not make the model smarter. It gave the model more original evidence to compare in the same context budget.

JSON and Markdown both have a role

This is not a case for replacing JSON everywhere. JSON remains the better format when code needs a strict contract: tool arguments, IDs, status fields, saved state, validation, and orchestration logic. Markdown is useful when the next step is model understanding. If an AI agent needs to read, compare, summarize, or reason over search results, Markdown can preserve a clear structure while reducing token overhead.

A practical agent workflow can use both:

Role	Better fit
Control data	JSON
Tool arguments	JSON
IDs and state	JSON
Search result context for the model	Markdown
Documents, citations, and comparison tables	Markdown

What I learned

This experiment led to three takeaways.

First, building an AI SEO pipeline gives teams control over what the model sees and how recommendations get tested. The value comes from connecting search data with the business question, not from rebuilding a full SEO platform.

Second, Markdown can reduce input token usage substantially for search result payloads. In our Google Shopping benchmark reproduction, Markdown reduced input tokens by 89.55% on average.

Third, a more compact input format can change the model's answer by increasing evidence coverage. In the Shopping demo, JSON fit 16 products while Markdown fit all 30. The larger sample changed the title signal and led DeepSeek to recommend testing title relevance first.

If you are building an AI agent or AI SEO pipeline, test both formats in one real workflow. Compare:

Actual input tokens reported by the model API
How much source data fits into the prompt
Whether the larger context changes the final recommendation

File size alone is not enough. What matters is how much useful evidence the model can see when it makes the decision.

Try Markdown output with SerpApi

SerpApi supports Markdown responses for search APIs with output=md. If your workflow sends search results into an LLM or AI agent, this can reduce formatting overhead and help the model analyze more results in one context.

You can try SerpApi with a free account, which includes 250 searches per month. If you have questions or feedback about Markdown responses, contact us at contact@serpapi.com.

AI #SEO #markdown

RubyLLM 2.x: What's new for your agents

Josef Strzibny — Sat, 24 Oct 2026 10:24:10 +0000

RubyLLM 2.0 and 2.1 are out and come with new APIs for video, speech, OCR, reranking, batches, conversation controls, expanded list of supported providers and even better Rails integration. But that's not all. If you build chat-based agents in Rails, tools can now wait for a human to approve them, you can run the agent loop one step at a time, and the agents can live in the database and survive a restart. So let's have a look on how to upgrade our old agents and give them new capabilities.

💡 This post is mainly about RubyLLM 2.0 and 2.1 features for agents. For the full list, see also the official What's New in 2.0 and 2.1 guides.

A quick look on the changes

The short version of both releases include the following main additions:

Tool approvals. Mark a tool with requires_approval and the agent pauses until a human says yes or no.
Caller-controlled loop. ask_later, generate, run_tools, and step let you run the agent loop one piece at a time.
Durable agents. Agents can be backed by your Chat model and resumed across requests, jobs, and restarts.
Provider-hosted tools. Web search, code execution, and remote MCP servers through one provider_tools API, with citations.
Batches. Submit many chats at once at the provider's batch rates and collect results later.
Fallbacks, prompt caching, and cost tracking. with_fallbacks, with_caching, and normalized tokens and cost on every response.
MCP client. Connect agents to Linear, GitHub, Notion, or any MCP server with a class, the same way you describe a tool.
Tool progress. Tools report what they're doing while they run, and your UI can show it.
Tool search. Mark tools as deferred and the model searches for the ones it needs instead of receiving every definition.
Evaluations. Run your agent against a dataset of reference answers and get a graded report.
Cache lifetimes per boundary. cache_until_here(ttl: "1h") for the prefix that never changes.
Faster by default. Streaming parses incrementally, replies stop retaining requests, and connections can be reused.

New agent capabilities

Tool approvals

One of the biggest additions to RubyLLM is the ability to approve tool calls. The obvious problem is that nobody wants a language model repricing the catalog on its own. RubyLLM 2.0 solves this with requires_approval:

class UpdatePrice < RubyLLM::Tool
  description "Updates the price of a product in our catalog"
  requires_approval

  param :sku, desc: "Product SKU"
  param :price, desc: "New price in USD"

  def execute(sku:, price:)
    product = Product.find_by!(sku: sku)
    product.update!(price: price)
    "Updated #{product.name} to $#{price}"
  end
end

Add it to the agent's tools and ask for a change:

class PriceMonitorAgent < RubyLLM::Agent
  # ...
  tools LookupProduct, SearchGoogleShopping, UpdatePrice
end

agent = PriceMonitorAgent.new
agent.ask "Lower the Chemex 6-Cup to match the market average."
agent.awaiting_approval? # => true

The agent still runs LookupProduct and SearchGoogleShopping on its own. When it gets to UpdatePrice, it stops. The tool call stays pending and ask returns. You can inspect what the model wants to do before anything touches the database:

agent.pending_approvals.each do |call|
  puts "#{call.name} #{call.arguments.inspect}"
end
# update_price {"sku"=>"CHX-6", "price"=>44.97}

Then approve or deny and let the agent finish:

agent.approve(agent.pending_approvals.first)
agent.complete

deny works the same way. The model gets told its request was rejected and can respond accordingly, usually by explaining what it would have done.

This is the single most useful change in 2.0 for anyone running agents against production data. Read-only tools backed by SerpApi can run freely. Tools that adjust live prices might need a human in the loop, and now that's possible.

Controlling the agent loop

In RubyLLM 1.x, ask ran the whole thing: send the message, run tools, send results back, repeat until the model answers. That's convenient until you want to be more specific and choose the number of rounds, log every step, or run each turn in a separate background job.

RubyLLM 2.0 breaks the loop into 4 parts:

ask_later stages the question without sending it.
generate asks the model for one response.
run_tools executes any pending tool calls.
step advances by one generation or one tool execution, whichever is next.

So the previous loop can be run one step at the time:

agent = PriceMonitorAgent.new
agent.ask_later "Which of our coffee products are priced more than 15% above market?"

agent.generate  # model decides to call LookupProduct and SearchGoogleShopping
agent.run_tools # the SerpApi searches happen here
agent.generate  # model reads the results and answers

Nothing new happens compared to ask, but you can now put a counter, a timeout, or a job boundary between any two lines. For a pricing agent that might run dozens of Google Shopping searches, being able to say "stop after ten tool rounds" is a feature, not a limitation.

Rails durable agents

Previously we could persist chats with acts_as_chat. Now we can go further and lets the agent itself be the record. Point the agent at your chat model with chat_model:

class PriceMonitorAgent < RubyLLM::Agent
  chat_model Chat
  model "gpt-5.4-mini"

  instructions <<~PROMPT
    You are a pricing analyst for the merchandising team.
    Always show specific numbers when comparing prices.
  PROMPT

  tools LookupProduct, SearchGoogleShopping, UpdatePrice
end

Then use create! instead of new:

chat = PriceMonitorAgent.create!
chat.ask "Lower the Chemex 6-Cup to match the market average."
chat.awaiting_approval? # => true

This is where approvals get practical. The request that triggered the agent has long returned. The merchandiser sees the proposed price change in the UI, maybe hours later, and clicks approve. Your controller or job just reloads the agent and continues:

chat = PriceMonitorAgent.find(params[:chat_id])
chat.approve(params[:tool_call_id])
chat.complete

The transcript, the pending tool call, and the decision all live in your database. If a job dies halfway, you can resume from the last saved step.

💡 One caveat from the docs: an interrupted operation may run again, so tools should tolerate retries. UpdatePrice is fine since setting a price twice is harmless. A tool that sends an email should check first.

Your app still owns the chats and messages tables. RubyLLM 2.0 takes ownership of the supporting tables for the model registry, tool calls, usage, and batches, so it can evolve them without you maintaining the models.

Provider tools

The next useful addition is a shared API for tools hosted by the provider. Things like web search, code execution, and remote MCP servers. You enable them with with_provider_tools on a chat or provider_tools on an agent, and the results come back with citations:

class PriceMonitorAgent < RubyLLM::Agent
  model "claude-sonnet-5"
  provider_tools :web_search
  tools LookupProduct, SearchGoogleShopping, UpdatePrice
end

response = PriceMonitorAgent.new.ask "Any recent news about Chemex pricing?"
response.citations.each { |c| puts "#{c.title} - #{c.url}" }

Provider web search is great starting point for open-ended questions. The tool returns answers with links.

Provider web search vs. SerpApi tools

Previously, I showed you how to build RubyLLM agents with the examples of wrapping SerpApi SDK to give your agents access to 100+ search engines. While it's now easier than ever to just use the default search, a SerpApi-backed tool gives you structured data from a very specific engine.

For example, SerpApi's Google Shopping API always returns a title, a price, and a source every single time. You pick the engine, so swapping to Amazon, Walmart, or eBay is a one-line change. You control the parameters, so you can pin the location or currency. And you can see every query the agent ran in the search inspector.

This gives you accuracy and flexibility the default search cannot.

Batch processing

Checking one product price in a chat is one thing. But what about checking the whole catalog every night? RubyLLM now adds batch processing on top of ask_later which lets you use providers batch rates for more affordable queries.

An example of this could be submitting a chat question in bulk:

chats = Product.where(category: "coffee").map do |product|
  market = SerpApi.search(q: product.name, engine: "google_shopping")
                  .shopping_results.first(5)
                  .map { |r| "#{r[:source]}: #{r[:price]}" }
                  .join(", ")

  RubyLLM.chat(model: "gpt-5.4-mini")
    .with_instructions("You are a pricing analyst. Answer in one sentence.")
    .ask_later("Our #{product.name} is $#{product.price}. Market: #{market}. " \
               "Are we more than 15% above market?")
end

batch = RubyLLM.batch(chats)

Save batch.id somewhere and another background job can check on it later:

batch = RubyLLM::Batch.find(batch_id, provider: :openai)

if batch.complete?
  batch.messages.each { |message| puts message.content }
end

Results come back in submission order, so zipping them with the products list is trivial. The Batches guide covers failures, embedding batches, and how tool calls behave inside a batch if you'd rather let the model call the tool itself.

Fallbacks, caching, and cost

Finally let's go over three smaller (but neat) additions that matter once the agent runs on a schedule rather than in a console.

Fallbacks. When the primary model fails with a transient error, try another one:

chat = RubyLLM.chat(model: "gpt-5.4-mini")
             .with_tools(LookupProduct, SearchGoogleShopping)
             .with_fallbacks("claude-sonnet-5")

The conversation keeps its tools and settings, so pick a fallback that supports what you use. You need credentials for both providers.

Prompt caching. Our agent sends the same instructions on every request. Mark them as a cacheable prefix:

chat = RubyLLM.chat(model: "claude-sonnet-5").with_caching
chat.with_instructions(File.read("pricing-policy.txt")).cache_until_here

response = chat.ask "Is the Aeropress competitively priced?"
response.tokens.cache_read

Providers still decide the minimum prefix length and lifetime, and a cache hit isn't guaranteed. But for a long pricing policy that doesn't change, it adds up.

Cost tracking. Every response now reports normalized tokens and cost, including the failed attempts behind a fallback:

response.tokens.input
response.tokens.output
response.cost.total
chat.cost.total

Unknown pricing stays nil rather than showing as zero. In Rails, usage is recorded in its own ledger, so changing model prices later doesn't rewrite history. If you've ever been asked "how much does this agent cost us per day?", this is the answer.

Upgrading from 1.x

Install or upgrade the gem with the following Bundler call:

bundle add ruby_llm --version 2.0.0

Applications on 1.x go to 2.0 first, then 2.1. From 2.1 on, each release ships the upgrade from the release before it:

bundle update ruby_llm --version 2.1.0

The 2.0 step is mostly renames, with one name per concept across the library. For tools, param became parameter, desc: became description:, and params do ... end became parameters do ... end.

A 1.x tool:

class LookupProduct < RubyLLM::Tool
  description "Looks up a product in our catalog by name"
  param :name, desc: "Product name to search for"

  def execute(name:)
    # ...
  end
end

Becomes:

class LookupProduct < RubyLLM::Tool
  description "Looks up a product in our catalog by name"
  parameter :name, description: "Product name to search for"

  def execute(name:)
    # ...
  end
end

The agent class itself needs no changes. Other renames you'll hit first:

max_tokens is now max_output_tokens.
with_params and provider-specific options go through with_provider_options.
response.input_tokens and friends live under response.tokens.
Callbacks follow Rails naming: on_tool_call is before_tool_call, on_end_message is after_message.
Tools can no longer halt the loop; use approvals or the caller-controlled loop instead.
Tool results that return a Hash or Array are sent as JSON text instead of Ruby's inspect output.

In Rails, version 2.0 moves the model registry, tool calls, usage, and batches into RubyLLM-owned tables. bin/rails generate ruby_llm:upgrade creates the migrations in phases, with cleanup in a later deploy, and an optional --mode copy keeps a route back to 1.16 while you verify. Then run the generator again on 2.1 to add the MCP credentials table and the new columns.

The official upgrade guide has the full list.

Also new

Here are some other things I skipped because they don't change how agents work, but which you might want to start using too:

RubyLLM.animate is now available for requesting video
RubyLLM.speak for speech
RubyLLM.transcribe for transcription with speaker labels and timestamps
RubyLLM.ocr for PDFs and scans
RubyLLM.rerank for search results
RubyLLM.research for provider-hosted deep research
RubyLLM.workflow for grouping steps in your instrumentation

Plus there are now four new providers too: Cohere, Deepgram, ElevenLabs, and Ollama Cloud.

Conclusion

RubyLLM 2.x makes agents safer to run against real data. Approvals keep humans in charge of writes, loop control lets you bound what an agent does, and durable agents mean a pending decision can wait in your database until someone makes it.

SerpApi fits in the same place it did before: as an agent tool that gives your agent structured, inspectable search data from Google Shopping, Amazon, Walmart, and other engines. Give it a try with 250 free searches a month.

How to Fit More Web Search Data into a Local LLM

Tomás Murúa — Tue, 20 Oct 2026 01:52:22 +0000

A local model can only use the web search results that fit in its context window. When we built the app for How to Connect Your Local LLM with Web Search Data, a single SerpApi response was often bigger than the whole context window available. So we trimmed every response with the JSON Restrictor, which keeps only the fields the model needs and drops the rest on SerpApi’s servers before it reaches the LLM.

SerpApi's new Markdown output changes that budget. The same data takes fewer tokens in Markdown than in JSON, so the restrictor can keep more of it. For most searches, the full response now fits with no restrictor at all.

This blog will show you how to apply the JSON Restrictor to Markdown output, and when the full response is worth the wait. We'll use the same open-source app that gives a local model running in LM Studio internet access. The app is built with TanStack Start, and we upgraded it to Start 1.0 along the way.

GitHub - serpapi/local-llm-web-search: Give a local LLM real-time web search via SerpApi function calling.

What’s New Since the First Blog

SerpApi's Markdown output returns curated results as a Markdown document instead of JSON. You add output=md to a request, and the results come back as headings, lists and tables without the JSON keys, quotes and braces. It works across SerpApi's100+ APIs, and a Markdown search costs the same as a JSON search. The launch blog Introducing SerpApi’s New Markdown Output covers the format and its token savings across our engines.

The app now lets you choose what the model reads after each search. The benchmark compares four outputs, and a model only ever reads three of them.

Output	What the model reads	Parameters
JSON (Raw)	The full JSON response. The benchmark measures it, but no model reads it	No restrictor
JSON (Restrictor)	JSON trimmed to the fields the model needs, as in the first blog	`json_restrictor`
Markdown (Restrictor)	Markdown with more results and more fields, at about the same size. This is the app's default	`json_restrictor` and `output=md`
Markdown (Raw)	The full Markdown document	`output=md`

The setup changed too. The first blog ran Qwen3.5-9B on a MacBook Pro M4 Max with 36 GB of memory. This time I ran four of LM Studio's staff picks on a MacBook Pro M5 Pro with 48 GB, using LM Link to connect both MacBooks. They are:

How We Tested

Every model ran with a 32K context window, the same size as in the first blog. LM Studio loads a model with an 8K window by default, which is too small once a chat carries search results. A 32K window is enough for a long conversation, and it fits a 27B model in memory on a 36 GB Mac.

What the Benchmark Measures

The app's Benchmark page measures two things, and this blog uses both.

Payload size: The same seven searches run once for each of the four outputs, with no model involved. There's one search per tool, coveringSearch,Finance,News,Maps,Flights,Hotels andShopping. This shows what each output costs before a model reads anything. These sizes come from OpenAI's tiktoken library with its cl100k_base encoding, which splits text into tokens. Each local model splits text its own way, so treat these sizes as estimates for comparing formats.

End to end: Thirteen questions run through the whole app on each model, once for each of the three outputs a model can read. Seven need live data from SerpApi, one search per tool. Two need two searches at once, such as comparing AAPL with TSLA. The other four need no search, such as "What is 15% of 240?", so they show how the model behaves on its own. We used it to compare how each model handles each output, how long it takes, and what it answers.

How Much Does Markdown Save?

To see what the format alone saves, we kept the restrictor strings from the first blog and sent the same seven searches as JSON and as Markdown. The seven results took 4,668 tokens in JSON and 3,412 in Markdown, 27% fewer. Here's how each tool changed:

Tool	JSON	Markdown	Change
Flights	823	439	−47%
Finance	521	324	−38%
News	678	422	−38%
Search	785	586	−25%
Hotels	567	433	−24%
Maps	497	449	−10%
Shopping	797	759	−5%
Total	4,668	3,412	−27%

The saving depends on how deeply the data is nested. A flight option holds legs, and each leg holds airports and times, so JSON repeats keys, quotes and braces at every level. Markdown turns lists of results into tables and writes each column name once. Shopping results are flat, with a title, a price and a store per product, so there is little to remove.

So instead of keeping that saving, we spend it. In Markdown, the same token budget can carry more fields and more results, so the Markdown output gets a looser restrictor.

Apply the JSON Restrictor to Markdown Output

There's no separate restrictor for Markdown. You send the same json_restrictor parameter together with output=md, and SerpApi filters the sections first, then renders what's left as Markdown.

In the app, each tool has two restrictor strings. JSON sends the original one, and Markdown sends a looser one with more results and more fields. The Markdown request is the same search with output=md added:

// Pick the restrictor for the output mode. Markdown (Raw) sends none.
function restrictorFor(name: RestrictorTool, output: ToolOutput): string | null {
  if (output === "markdown-full") return null
  return output === "markdown" ? MARKDOWN_RESTRICTORS[name] : RESTRICTORS[name]
}

// Same parameters as the JSON request, plus output=md.
async function serpApiMarkdown(params: Record<string, unknown>): Promise<string> {
  const url = new URL("https://serpapi.com/search")
  for (const [key, value] of Object.entries(params)) {
    if (value !== undefined && value !== null) url.searchParams.set(key, String(value))
  }
  url.searchParams.set("output", "md")
  url.searchParams.set("api_key", process.env.SERPAPI_API_KEY ?? "")

  const res = await fetch(url)
  return res.text()
}

Each tool spends the saving on different extras:

Tool	JSON (Restrictor) keeps	Markdown (Restrictor) keeps
Search	Answer box, AI Overview (6 blocks), 3 related questions, knowledge graph, 5 web results	5 web results, answer box with its lists, AI Overview (8 blocks), knowledge graph, 3 top stories, 3 related questions
Finance	Price, key stats, 3 news items	Price, key stats, company description, 5 news items
News	5 stories, 1 related article each	7 stories, 2 related articles each
Maps	5 places	6 places, with opening hours and a description
Flights	3 best and 3 other options	4 best and 4 other options, with aircraft, legroom, overnight layovers and price insights
Hotels	5 hotels	7 hotels, with a description and a location rating
Shopping	5 products, plus 3 inline results	7 products, plus 3 inline results, with second-hand condition

For example, here are both restrictor strings for Maps, from restrictors.ts:

// JSON (Restrictor): five places.
export const RESTRICTORS = {
  google_maps_search: [
    "local_results[0:5].{title,rating,reviews,type,address,price,open_state,phone,website,gps_coordinates}",
    "place_results.{title,rating,reviews,type,address,price,open_state,phone,website}",
    "error",
    "search_metadata.google_maps_url",
  ].join(","),
  // ...one restrictor per tool
}

// Markdown (Restrictor): six places, with opening hours and a description.
export const MARKDOWN_RESTRICTORS = {
  google_maps_search: [
    "local_results[0:6].{title,rating,reviews,type,address,price,open_state,hours,phone,website,description}",
    "place_results.{title,rating,reviews,type,address,price,open_state,hours,phone,website,description}",
    "error",
    "search_metadata.google_maps_url",
  ].join(","),
  // ...one restrictor per tool
}

The extra data shows up in the answers. Asked for the best barbecue in Austin, Qwen 3.8 27B listed five places with JSON and six with Markdown. The new one, Good BBQ Company, was one of the two highest rated. The Markdown answer also described Terry Black's as "4th-generation pitmasters, old-school smoked meats," from the place descriptions only the Markdown restrictor keeps.

What Happens When You Skip the JSON Restrictor

The bigger change is the full response, which I called Raw in the app. In the first blog, sending a raw SerpApi response to a local model wasn't an option. The seven raw JSON responses added up to 326,949 tokens, almost 10 times the context window. The same seven responses in Markdown came to 71,244 tokens, 4.6 times smaller, and none of them was bigger than the window.

Fitting in the window doesn't make it fast. With Markdown (Raw), the models read two to three times as many tokens as with Markdown (Restrictor), and took up to 2.7 times as long to answer.

Two full pages in one question can still overflow the context window. On one of the four models, the prompt comparing AAPL and TSLA went over 32K tokens. LM Studio cut part of it without an error, and the answer got Apple’s daily change wrong.

Which Output Should You Use?

Most local chats should start with Markdown using the restrictor, and the other outputs fit narrower cases.

Markdown (Restrictor): The default. Use it whenever a model reads the results.
JSON (Restrictor): Use it when your code reads the results, for example to sort by price.
Markdown (Raw): Use it when an answer needs the whole page and can wait up to three times longer. Avoid it for questions that need two big searches, such as comparing two stocks.
JSON (Raw): Don't send it to a local model.

If your hardware has the memory, you can raise the 32K window with the app’s LMSTUDIO_CONTEXT_LENGTH setting.

How the Four Models Compare

All four models finished every question. The times below are the average per search question in Markdown (Restrictor):

Qwen 3.8 27B is the app's default model when it's installed. In long test conversations, it turned follow-ups like "around there" and "which of those" into the right searches. It took about 42 seconds per question.
Meta Muse Glimmer 30B splits text into fewer tokens than the other three, so the same search results take less of its window. It was the slowest, about 62 seconds per question.
Prism Bonsai 27B is a 27B model that LM Studio lists at about 4 GB, and it took about 24 seconds per question.
Gemma 4 12B QAT is the smallest model here, at 12B parameters, and the fastest, at about 16 seconds per question, or about 43 with Markdown (Raw). In long conversations it had the most problems, such as giving a price that wasn’t in the search results.

With less memory than the 48 GB used here, the app may load a smaller window, so less search data fits. New open models also come out almost every week, so the best choice changes fast. The staff picks inLM Studio's model search are a good place to start, because LM Studio matches them to your hardware.

When Search Results Contradict What the Model Knows

Every model stops learning at its training cutoff, so it doesn’t know what happened after that date. On September 1, 2026, John Ternus became Apple's CEO and Tim Cook became executive chairman, after 15 years as CEO. So "Who is the current CEO of Apple?" puts a model's training against the search results.

Here's what Qwen 3.8 27B thought before its first search:

The user is asking who the current CEO of Apple is. This is a factual question. As of my knowledge, Tim Cook has been Apple's CEO since 2011. However, since this is a question about the current situation, I should verify with a search to make sure there haven't been any changes. Let me search.

That part is right. The model knows its knowledge can be out of date, and it checks. The harder part comes when the results disagree with it. In an earlier run, the AI Overview in Qwen's first results already named John Ternus, and Qwen didn't accept it.

This is a significant claim - it suggests Tim Cook stepped down and John Ternus took over. Today's date is 2026-10-02, so this would be about a month ago. Let me verify this with a news search to make sure this is accurate and not a hallucination.

We can read all of this because the models run locally. LM Studio writes each model's reasoning to its server log, word for word. Hosted reasoning models usually show you a summary instead. When OpenAI launched o1, itchose not to show the raw chain of thought, and other AI labs have done the same recently. One competitive risk is knowledge distillation, where a smaller model learns to copy a bigger model's outputs, and full reasoning is valuable material for that.

Conclusion

In this blog we added SerpApi’s Markdown output to the app and benchmarked it on four local models. For the same fields, Markdown took 27% fewer tokens than JSON, so the restrictor could keep more results for about the same budget. Even without a restrictor, all seven full Markdown responses fit in the 32K window.

The same approach works for any SerpApi engine. Add output=md to the request, keep one restrictor per engine, and spend the tokens Markdown saves on the fields your questions need. To try it, clone the repo and add your SerpApi API key.A free account includes 250 searches a month.

When the next model you want to try shows up in LM Studio, run it through the app's Benchmark page and compare it with these four.

How to build a web scraper: A beginner's guide

Hilman Ramadhan — Sat, 17 Oct 2026 17:34:05 +0000

The simple definition of web scraping is the automated process of extracting data from websites. It also usually involves saving that data into a structured format, like a database, CSV or JSON file, or text document. That way, you can parse large collections of pages (or entire websites) in a much more efficient way than a person browsing the pages and manually copying the desired information.

A web scraper first needs to make requests to a web server, and then fetch the contents of the requested page, just like a person using a web browser normally would. Then, it needs to parse the page into whichever format is required. Both of those steps can be tricky at scale—many servers will throttle or block IP addresses making bulk requests, and parsers have to understand various page layouts and rendering methods.

In this guide, we'll go over how to build your own web scraper in Python, JavaScript, and Ruby, and how to use the easier option—the Extract API from SerpApi.

Building your own web scraper

Web scraping can be done using a variety of programming languages and tools. Whether you're comfortable with JavaScript, Ruby, Go, or another language, libraries and frameworks are available to help you scrape websites.

The first step in web scraping is to fetch the HTML content of the web page you want. This is similar to how your web browser works when you visit a website—it sends a request to the server, and the server responds with the HTML, CSS, JavaScript, and other resources needed to display the page.

In web scraping, you'll use a programming language or tool to send an HTTP request to the website's server. This can be done using various methods:

Programming Libraries : Languages like Python, JavaScript, Ruby, and others have libraries that make it easy to send HTTP requests.
Built-in Functions : Some languages have built-in functions for making web requests.

By requesting the HTML content, you're retrieving the raw data that makes up the web page, which you can then process and analyze.

Once you've obtained the HTML content, the next step is to parse it to extract the information you're interested in. Web pages are structured documents that use HTML tags to define elements like headings, paragraphs, links, images, and more.

Some common methods of navigating through HTML pages are CSS selectors or XPath. For example, if you wanted to find all links in a page with "product" in the destination URL, you could use a CSS selector like this:

a[href*="product"]

Libraries like Beautiful Soup for Python, jsdom or Cheerio for Node.js, Nokogiri or Nokolexbor for Ruby, and others can parse HTML and XML documents. That way, you can navigate the document elements using standard selectors. Another option is using an AI model to parse the HTML.

Once you've identified the correct elements, you can extract the text, attributes, or other relevant information. That gives you the data for analyzing trends, feeding into another application, building a dataset, or other tasks.

Practice scraping a real website

It's time to build a web scraper! We will use the Hacker News home page as the example site, with the goal of extracting the title, external link, and visible link label for each post.

First, we need to figure out where the links are in the page's HTML. Try right-clicking one of the links, then clicking the "Inspect Element" or "Inspect" menu option.

🧭 Using Safari on Mac? You may need to enable development features first. From the top menu bar, select Safari > Settings, click the Advanced tab, then check the box for "Show features for web developers."

This will open the HTML page's structure, and jump to where that link element is located.

As of the time of writing, each submission is a <tr> table row element with a athing CSS class. In each of those table rows, there's a .titleline element containing everything we need: a link element (a) with the external link and post title, and a .sitestr tag with the visible link label.

Now it's time to scrape that information!

Web scraping using Python

If you want to build a web scraper in Python, you can start with the Requests and Beautiful Soup libraries. For more complex web pages beyond this example, you can try Selenium or Scrapy.

First, install Python on your computer if you haven't already, then create the folder you want to use for this project. Next, open the Terminal (Mac/Linux) or PowerShell (Windows) application, and change the current directory to that project folder:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, create a Python virtual environment for the project, so all dependencies and other data stays within the folder:

python -m venv venv

💻 Python command not found? Try "python3" instead. The name of the executable depends on how it was installed.

Now, activate the Python virtual environment. On a Mac with the default bash/zsh shell, run this:

source venv/bin/activate

On a Windows PC in PowerShell, run this:

venv\Scripts\Activate.ps1

Next, install the Beautiful Soup and Requests libraries in the project:

pip install requests beautifulsoup4

Now it's time to start writing code! Create a new Python script called main.py in your project with the following contents, then try running it with the python main.py command:

import requests

URL = 'https://news.ycombinator.com/'
response = requests.get(URL)

html_content = response.text
print(html_content)

This imports the Requests library, sends a GET request to the URL, and stores the HTML content in a variable.

Now, let's parse the HTML to extract the data. Add the following import statement to the top of the script:

from bs4 import BeautifulSoup

Replace the print(html_content) statement with this line, which creates a Beautiful Soup object for the parsed HTML content:

soup = BeautifulSoup(html_content, 'html.parser')

Next, the script needs to get a list of all table rows with the athing class, and extract each row's link URL, inner text, and visible link. You can do this with a soup.find_all function and a for loop at the end of the script:

rows = soup.find_all('tr', class_='athing')

for row in rows:
    titleline = row.find('span', class_='titleline')
    link_tag = titleline.find('a', href=True)

    title = link_tag.text
    link = link_tag['href']

    visible_link_tag = row.find('span', class_='sitestr')
    # This avoids an error when the visible_link_tag is None
    visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

    print("Title:", title)
    print("Link:", link)
    print("Visible Link:", visible_link)
    print('---')

Running the entire script should give you an output like this:

Title: StreetComplete on iOS is now in public beta
Link: https://github.com/streetcomplete/StreetComplete/issues/5421
Visible Link: github.com/streetcomplete
---
Title: RIP, vector database
Link: https://turbopuffer.com/blog/rip-vector-database
Visible Link: turbopuffer.com
---
Title: How to speed up the Rust compiler in September 2026
Link: https://nnethercote.github.io/2026/09/30/how-to-speed-up-the-rust-compiler-in-september-2026.html
Visible Link: nnethercote.github.io
---

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

Page 1: https://news.ycombinator.com/?p=1
Page 2: https://news.ycombinator.com/?p=2
Page n: https://news.ycombinator.com/?p=n

You can check each page by wrapping the entire script in another for loop that counts from 1 to 3, and then runs the GET request with that number as the page count in the URL. The full script should now look like this:

import requests
from bs4 import BeautifulSoup

URL = 'https://news.ycombinator.com/'
max_pages = 3 # I gave the limit to only 3 pages

for i in range(1, max_pages+1):
    response = requests.get(f'{URL}?p={i}')
    html_content = response.text
    soup = BeautifulSoup(html_content, 'html.parser')

    rows = soup.find_all('tr', class_='athing')

    for row in rows:
        titleline = row.find('span', class_='titleline')
        link_tag = titleline.find('a', href=True)

        title = link_tag.text
        link = link_tag['href']

        visible_link_tag = row.find('span', class_='sitestr')
        visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

        print("Title:", title)
        print("Link:", link)
        print("Visible Link:", visible_link)
        print('---')

The output will now include results from the first three pages.

You probably want to save the data somewhere other than the terminal output. Python's built-in CSV library can be used to write a CSV spreadsheet file, which can then be opened by Google Sheets, Microsoft Excel, LibreOffice Calc, and many other applications.

Here's how you could write a CSV file with the title, link, and visible link as the columns, and each post as a row:

import requests
from bs4 import BeautifulSoup
import csv

URL = 'https://news.ycombinator.com/'
max_pages = 3

# Open a CSV file in write mode
with open('data.csv', mode='w', newline='') as file:
    writer = csv.writer(file)

    # Write the header row
    writer.writerow(['Title', 'Link', 'Visible Link'])

    for i in range(1, max_pages+1):
        response = requests.get(f'{URL}?p={i}')
        html_content = response.text
        soup = BeautifulSoup(html_content, 'html.parser')

        rows = soup.find_all('tr', class_='athing')

        for row in rows:
            titleline = row.find('span', class_='titleline')
            link_tag = titleline.find('a', href=True)

            title = link_tag.text
            link = link_tag['href']

            visible_link_tag = row.find('span', class_='sitestr')
            visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

            # Write the data row to the CSV file
            writer.writerow([title, link, visible_link])

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

You now have a basic web scraper running in Python! The below resources can help you take on more complex scraping tasks:

Web scraping with CSS selectors using Python : A guide on using CSS selectors to find and parse HTML elements.
Selenium web scraping : Web pages that load with JavaScript or other dynamic elements won't work with only GET requests and Beautiful Soup, but you can parse them with Selenium.
Web scraping in Python: A more in-depth guide on building scrapers with Python.
The documentation for Beautiful Soup, Requests, and Python's CSV library.
13 ways to scrape public data: More methods for parsing complex web pages in Python.

You can also skip to the Extract API for a simpler method of web scraping.

Web scraping using JavaScript

JavaScript is also an option for web scraping projects, though not in the same way you might use the language in front-end development. Web browsers will block access to most external domains with CORS restrictions, which is required for scraping, so you need to use Node.js or another non-browser runtime. Those runtimes typically don't have a built-in Document Object Model (DOM), so common functions for parsing HTML like querySelector or getElementById() are unavailable unless you add additional libraries.

For this web scraping project, we'll use the Node.js runtime with the jsdom library. Node.js has a built-in Fetch API for retrieving web content, so additional libraries for that purpose (like Axios) are no longer required.

First, you'll need Node.js and the NPM package manager installed on your computer. Then create a folder somewhere for your project, open a Terminal window (Mac/Linux) or PowerShell prompt (Windows), and navigate to that folder. For example, this will change the current directory to a "scraper" folder on your Desktop:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, we need to install the jsdom library in the project, with this command:

npm install jsdom

Next, create a new JavaScript file called main.js in the project folder, and start with this code that fetches the HTML for the home page and displays it in the console:

const response = await fetch("https://news.ycombinator.com");
const text = await response.text();
console.log(text);

You can run it with the node main.js command. Next, we can parse the HTML by importing the jsdom library, passing the text to a JSDOM constructor, and making a for loop that goes through every .athing table row:

import jsdom from "jsdom";

const response = await fetch("https://news.ycombinator.com");
const text = await response.text();
const dom = new jsdom.JSDOM(text);

// Get every table row representing a post
for (let el of dom.window.document.querySelectorAll("tr.athing")) {
    // Get the title, link, and visible link
    const title = el.querySelector(".titleline a").textContent;
    const link = el.querySelector(".titleline a").href;
    const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
    // Display the data
    console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`);
}

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

Page 1: https://news.ycombinator.com/?p=1
Page 2: https://news.ycombinator.com/?p=2
Page n: https://news.ycombinator.com/?p=n

This is a modified version of the script that fetches the first three pages, and then displays all links from all pages:

import jsdom from "jsdom";

const baseUrl = "https://news.ycombinator.com"

for (let i = 1; i <= 3; i++) {
    const response = await fetch(`${baseUrl}/?p=${i}`);
    const text = await response.text();
    const dom = new jsdom.JSDOM(text);
    // Get every table row representing a post
    for (let el of dom.window.document.querySelectorAll("tr.athing")) {
        // Get the title, link, and visible link
        const title = el.querySelector(".titleline a").textContent;
        const link = el.querySelector(".titleline a").href;
        const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
        // Display the data
        console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`);
    }
}

You might want to save the scraped data outside of the terminal output. We can write the results to a CSV file by importing Node's fs library and constructing the CSV format:

import jsdom from "jsdom";
import fs from "node:fs/promises";

const baseUrl = "https://news.ycombinator.com"
let csvText = `"Title","Link","Visible Link"`

for (let i = 1; i <= 3; i++) {
    const response = await fetch(`${baseUrl}/?p=${i}`);
    const text = await response.text();
    const dom = new jsdom.JSDOM(text);
    // Get every table row representing a post
    for (let el of dom.window.document.querySelectorAll("tr.athing")) {
        // Get the title, link, and visible link
        const title = el.querySelector(".titleline a").textContent;
        const link = el.querySelector(".titleline a").href;
        const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
        // Add the data as a new row
        csvText += `\n"${title}","${link}","${visibleLink}"`
    }
}

await fs.writeFile("data.csv", csvText);

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

You now have a basic web scraper running in Node.js with JavaScript! The below resources can help you take on more complex scraping tasks:

CSS selectors and combinators: Learn about using CSS selectors to find elements, which also work in jsdom and other HTML parsing methods.
Web scraping a dynamic website: How to use Puppeteer with Node.js to scrape a website that uses a dynamic layout instead of static HTML.
Web scraping with JavaScript and Node.js: A more detailed guide on web scraping techniques using JavaScript.

You can also skip to the Extract API for a simpler method of web scraping.

Web scraping using Ruby

Ruby is another language option for building a web scraping tool. You can use libraries like Net::HTTP or HTTParty for making GET requests to web pages, Nokolexbor or Nokogiri for parsing HTML and XML elements, and Selenium for scraping with an automated web browser.

First, you need Ruby installed on your computer. You can download it from the official website or use a version manager like rbenv or RVM. Then create a folder somewhere for your project, open a Terminal window (Mac/Linux) or PowerShell prompt (Windows), and navigate to that folder. For example, this will change the current directory to a "scraper" folder on your Desktop:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, create a plain text file called Gemfile in your project folder to keep track of dependencies, with source 'https://rubygems.org' as the only line. You can do that in a few different ways:

Use a plain text editor (make sure not to add a .txt extension to the file)
For Windows PowerShell, run this command in the project folder: Add-Content Gemfile "source 'https://rubygems.org'"
For Mac/Linux, run this command in the project folder: "source 'https://rubygems.org'" >> Gemfile

We will use HTTParty for fetching web pages, Nokolexbor for parsing HTML, and csv for writing the data to CSV files. The CSV gem is usually bundled with Ruby, but it should still be defined as a dependency. Make sure they are installed with this command:

bundle add httparty nokolexbor csv

Next, create a Ruby file called main.rb in your project folder. The first version fetches the home page for Hacker News using the httparty gem:

require 'httparty'

url = 'https://news.ycombinator.com/'
response = HTTParty.get(url)
html_content = response.body
puts html_content

You can run it with the ruby main.rb command. Next, we need to add a require line for nokolexbor and then use it to parse each .athing table row as a post:

require 'httparty'
require 'nokolexbor'

url = 'https://news.ycombinator.com/'
response = HTTParty.get(url)
html_content = response.body
parsed_page = Nokolexbor::HTML(html_content)

parsed_page.css('tr.athing').each do |el|
    # Get the title, link, and visible link
    title = el.at_css('.titleline a').text
    link = el.at_css('.titleline a')['href']
    visible_link = el.at_css('.sitestr') ? el.at_css('.sitestr').text : 'No visible link'
    # Display the data
    puts "Title: #{title}\nLink: #{link}\nVisible Link: #{visible_link}\n---"
end

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

Page 1: https://news.ycombinator.com/?p=1
Page 2: https://news.ycombinator.com/?p=2
Page n: https://news.ycombinator.com/?p=n

Modifying the script to parse the first three pages isn't too difficult. We just need a loop that adds the current iteration to the page URL, like this:

require 'httparty'
require 'nokolexbor'

url = 'https://news.ycombinator.com/'
max_pages = 3

(1..max_pages).each do |i|
    response = HTTParty.get("#{url}?p=#{i}")
    html_content = response.body
    parsed_page = Nokolexbor::HTML(html_content)

    parsed_page.css('tr.athing').each do |el|
        # Get the title, link, and visible link
        title = el.at_css('.titleline a').text
        link = el.at_css('.titleline a')['href']
        visible_link = el.at_css('.sitestr') ? el.at_css('.sitestr').text : 'No visible link'
        # Display the data
        puts "Title: #{title}\nLink: #{link}\nVisible Link: #{visible_link}\n---"
    end
end

You might want to save the scraped data for future use. We can write the results to a CSV file by adding a require line for csv, creating a header row and a row for each post, and then writing it to a file:

require 'csv'

URL = 'https://news.ycombinator.com/'
max_pages = 3

# Open a CSV file in write mode
CSV.open('data.csv', 'w') do |csv|
  # Write the header row
  csv << ['Title', 'Link', 'Visible Link']

  (1..max_pages).each do |i|
    response = HTTParty.get("#{URL}?p=#{i}")
    html_content = response.body
    parsed_page = Nokogiri::HTML(html_content)

    rows = parsed_page.css('tr.athing')

    rows.each do |row|
      titleline = row.at_css('span.titleline a')
      title = titleline.text
      link = titleline['href']

      visible_link_tag = row.at_css('span.sitestr')
      visible_link = visible_link_tag ? visible_link_tag.text : 'No visible link'

      # Write the data row to the CSV file
      csv << [title, link, visible_link]
    end
  end
end

puts 'Data successfully written to data.csv'

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

You now have a Ruby-powered web scraper! Check out these resources for help with more complex scraping tasks:

CSS selectors and combinators: Learn about using CSS selectors to find elements, which are used by Nokolexbor, Nokogiri, and other HTML parsing methods.
Selenium documentation: For sites that load dynamically instead of providing static HTML, you need to use Selenium or another browser automation tool.
Web scraping with Ruby: A more complete guide on web scraping methods while using Ruby.

You can also skip to the Extract API for a simpler method of web scraping.

Challenges with web scraping

Basic web scraping, with pages already rendered in HTML with consistent element names, isn't too complicated in many programming languages and environments. However, you will inevitably run into problems as you try parsing different websites, or try requesting many pages.

These are some of the common challenges with web scraping at scale:

Identifying data sources : Determining where and how data is stored can be complex. Websites may store data across different pages or make requests over APIs, making it difficult to pinpoint the exact source.
Handling CAPTCHA tests : Websites often use a CAPTCHA, short for "Completely Automated Public Turing test to tell Computers and Humans Apart," to block automated access. Bypassing these tests without violating terms of service requires sophisticated methods, such as using CAPTCHA-solving services, which adds cost and complexity.
Managing IP blocks and rate limits : Frequent requests from the same IP can lead to blocks or bans. Using proxies to rotate IPs and implementing smart throttling mechanisms can help mitigate this but require careful management and additional infrastructure.
Legal and ethical considerations : Scraping must be done in compliance with laws and website terms of service. Ignoring these can lead to legal actions and ethical issues.
Data structure changes : Websites often update their layout and structure, which can break scrapers. Regularly maintaining and updating scraping scripts is crucial to handle these changes.
Dynamically-rendered pages: Many websites will render the page with JavaScript, instead of sending a complete HTML structure over the initial request. This requires fetching the page with an automated web browser, like Selenium or Puppeteer, instead of a simple GET request.

The easy web scraping method

If you don't want to build and maintain your own web scraper, you can use our Extract API. It can parse any public web page and return its contents as structured JSON or Markdown, including automatic detection of headers, meta tags, inline links, and the complete page text. There's also a Map mode for finding all pages on a given website, and Crawl mode for finding and parsing all of a website's pages.

If you request a page on an engine or platform that has a dedicated SerpApi API, the results will be automatically routed to that API. For example, fetching a YouTube video URL with the Extract API will return all the data available through the YouTube Video API, like the video description and all recommended videos.

If the built-in parsing from the Extract API isn't enough, you can still use it to retrieve the fully-rendered HTML, and then build your own parser using that input.

Getting started with SerpApi

You need a free SerpApi account before you can use the Google Play Store APIs. You can upgrade to a paid account in the future if you need more search capacity, faster speeds, or other features.

First, create an account and verify your email and other details. After that, get the API key from your account dashboard.

You should store your API key in a safe location if you are sharing or publishing your code. If the key is leaked or stolen, you can make a new one from the SerpApi account dashboard.

Install the SerpApi library (optional)

You can use SerpApi's official libraries for Python, JavaScript, Ruby, Java, and other languages. They aren't required, but they provide a simple wrapper around the API requests.

The APIs can also be used with cURL, fetch() in Node.js, or anything else that can send GET requests and process a JSON response.

Extract API with a GET request

Here's how you could fetch the Hacker News home page with the Extract API, using a simple GET request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&api_key=YOUR_KEY_GOES_HERE

The default response is in JSON format, with basic page information in the page_information object, a list of all included links in the inline_links object, and so on.

If you want the response in Markdown format, instead of JSON, just add output=md to the request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&output=md&api_key=YOUR_KEY_GOES_HERE

If you need the page's rendered HTML, for building your own parser or some other use case, you can set output=html for the request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&output=html&api_key=YOUR_KEY_GOES_HERE

Extract API with JavaScript

Here's how you can use the Extract API with the SerpApi JavaScript library to display all posts from the first Hacker News home page:

import { getJson } from 'serpapi';

let search = await getJson({
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  url: "https://news.ycombinator.com/",
  mode: "scrape"
});

for (let link of (search?.inline_links || [])) {
  // Skip internal links, and the "API" link at the bottom of the page
  if (link.internal || link.link === "https://github.com/HackerNews/API") {
    continue;
  }
  // Display the title, link, and visible link text
  console.log(`Title: ${link.title}\nLink: ${link.link}\nVisible Link: ${link.displayed_link}\n----`)
}

The Extract API fetches the page contents and parses it, so all that is required is going through the list of extracted links (search.inline_links) and filtering the results to external URLs.

Here's a more advanced version that scans the first three pages, and saves the results to a CSV file:

import { getJson } from 'serpapi';
import fs from "node:fs/promises";

const baseUrl = "https://news.ycombinator.com"
let csvText = `"Title","Link","Visible Link"`

let settings = {
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  mode: "scrape"
}

for (let i = 1; i <= 3; i++) {
  settings.url = `${baseUrl}/?p=${i}`;
  let search = await getJson(settings);
  for (let link of (search?.inline_links || [])) {
    // Skip internal links, and the "API" link at the bottom of the page
    if (link.internal || link.link === "https://github.com/HackerNews/API") {
      continue;
    }
    // Add the data as a new row
    csvText += `\n"${link.title}","${link.link}","${link.displayed_link}"`;
  }
}

await fs.writeFile("data.csv", csvText);

The crawl mode is helpful for analyzing many pages in a group. This example will crawl the UniFi store, up to 50 pages, to find every product page:

// import { getJson } from 'serpapi';

let search = await getJson({
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  url: "https://store.ui.com/us/en",
  mode: "crawl",
  max_pages: 50
});

let productUrls = [];

// Iterate through all crawled pages
for (let page of search.pages) {
  // Iterate through all inline links
  for (let result of (page?.inline_links || [])) {
    // Filter out non-product links, and links already added to the list
    if (result.link.includes("/products/") && !productUrls.includes(result.link)) {
      productUrls.push(result.link);
    }
  }
}

// List all products
for (let product of productUrls) {
  console.log(product);
}

More information

You can learn more from the documentation pages for the Extract API, Map mode, and Crawl mode.

7 Best Rank Tracking API Options Compared

Noraina Nordin — Thu, 15 Oct 2026 09:11:51 +0000

Once you track more than a few dozen keywords, checking positions by hand stops working. You need data you can load into a dashboard, a client report or your own app. Picking the best rank tracking API starts with one decision. You can buy a rank tracker that has an API, like Semrush, AccuRanker or Ahrefs. Or you can build on a SERP API, which returns the raw results page and lets you work out positions yourself.

Rank trackers give you dashboards, history and alerts out of the box. SERP APIs give you more control over location, device and schedule, and usually cost less per keyword check. This guide compares 7 options across both camps on cost, accuracy, mobile support and AI Overview parsing.

🚦 Disclosure: this is SerpApi's blog, and SerpApi is one of the 7 options. We list its limitations alongside everyone else's.

Best rank tracking APIs at a glance

Prices and features below were checked on each provider's pricing and documentation pages in October 2026.

Provider

Type

Starting price

Free tier

Location & device targeting

Update frequency

AI Overview parsing

SerpApi

SERP API

$25/mo for 1,000 searches

250 searches/mo

City, coordinates or uule; desktop, tablet, mobile

Real time, on your schedule

Yes

|
|

DataForSEO

SERP API

$0.60 per 1,000 SERPs (standard queue)

$1 credit + sandbox

Location code, name or coordinates; desktop, mobile

About 5 min (standard) to about 6 sec (live)

Yes

|
|

Serpent API

SERP API

$0.60 per 1,000 calls

Free calls on sign-up

Country and language only

Real time, on your schedule

Yes (Deep Search)

|
|

cloro

SERP API

$30/mo for 37,500 credits

500 credits/mo

Country, city or uule; desktop, iOS, Android

Real time, on your schedule

Yes (+2 credits)

|
|

Semrush Position Tracking

Rank tracker API

$549/mo (Advanced) + API units

No API access on free plan

Down to postal code; desktop, phone, tablet

Daily

Filter by AI Overview

|
|

AccuRanker

Rank tracker API

$764/mo (Expert, for API)

Not stated

Country or local; device not stated

Daily + on-demand refresh

Not stated

|
|

Ahrefs Rank Tracker

Rank tracker API

$129/mo (Lite)

Not stated

Country, state or city; desktop, mobile

Weekly

Yes

Pick by what you need most:

SerpApi if you want real-time results, city-level and tablet targeting, and no charge for failed or cached searches.
DataForSEO if cost per keyword matters more than speed and you can work with a task queue.
Serpent API if you only track country-level desktop rankings and want the lowest entry cost.
cloro if AI Overview tracking is a core need and you want iOS and Android as separate devices.
Semrush if your team already works in Semrush and wants rankings flowing into its reports.
AccuRanker if you need on-demand refreshes and an unlimited read API at enterprise scale.
Ahrefs if weekly data is enough and you already pay for Ahrefs.

SERP API or rank tracker API?

Both return a position for a keyword. The difference is who does the work between the search and the number.

A rank tracker API sits on top of a finished product. You set up a project in Semrush, AccuRanker or Ahrefs, add keywords, and the tool checks them on its own schedule. The API then lets you read that stored history. You get clean data and no parsing. You also accept the tool's update frequency, its locations and its price per tracked keyword.

A SERP API returns the full results page as JSON every time you call it. You decide when to check, from where and on which device. You then find your domain in the results and store the position yourself. That takes some code, but it removes per-seat pricing. It also lets you track things a dashboard may not show, like who holds the featured snippet.

A rough rule of thumb:

Choose a rank tracker API if you need reports for non-technical stakeholders and your keyword list fits the plan.
Choose a SERP API if you track many locations or devices, need checks on your own schedule, or want rankings inside your own product.

The r/SEO_LLM discussion on SEO API providers shows the same split. Some people want finished metrics, while others want raw SERP data they can shape themselves.

There is also a middle path. Open source trackers give you a dashboard but run on a SERP API you pay for directly. SerpBear is the best known. It is MIT licensed, runs in Docker, and can use SerpApi or several other scrapers.

We built our own take on this idea. SerpTrail, our self-hosted rank tracker, runs in Docker with Rails and SQLite. It tracks Google rankings and AI Overviews across multiple locations on a daily, weekly, biweekly or monthly schedule. It can check up to five pages of Google results, and the source code is on GitHub. It runs on your SerpApi key, so the free plan's 250 searches are enough to try it.

SerpTrail self-hosted rank tracker dashboard

What makes an accurate rank tracker?

An accurate rank tracker is one that reports the results page your customers actually see. There is no single "true" position for a keyword. Google builds each page from the searcher's location, device, language and the moment of the search. A tracker is accurate when it controls all four and keeps them the same from one check to the next.

Location, device, language and time

Location matters most for anything with local intent. A national check and a check from a specific city can return different pages, so set the location at city level when it matters. If you target several countries, our guide to global SEO strategies covers how results shift by market and language. Match the language (hl) and country (gl) to your audience too.

Device is the next big factor. On October 6, 2026, we ran "best running shoes for women" through SerpApi from Austin, Texas, once on desktop and once on mobile. Desktop returned 10 organic results and mobile returned 9. Only 6 pages appeared in both. Outside Online ranked #3 on desktop but #7 on mobile, and GearJunkie's #5 desktop result didn't appear on mobile at all.

Desktop vs mobile Google organic results for the same keyword and location

Time of check matters because rankings move during the day. Check at the same hour each day so changes reflect Google, not your schedule.

Why tools and humans see different results

A common complaint is that a tool's ranking doesn't match a manual search. Usually the tool isn't wrong. This is caused by the person searching being logged in, having search history, or sitting a few miles from the location the tool uses. Accuracy comes from consistent parameters more than from any one provider.

The num=100 change

In September 2025, Google stopped supporting the num=100 parameter, which many tools used to fetch 100 results in one request. Tracking the top 100 now takes about 10 paginated requests per keyword. Ask any provider how it handles depth and how it bills for it.

Check against Search Console and pixel position

Google Search Console is a useful sanity check. Its average position blends every location and device, so it won't match a single check. But a large, lasting gap between the two is a sign to recheck your tracker's settings.

Position number alone also hides how visible a result is. A #1 organic result can sit below an AI Overview, ads and a local pack. Pixel ranking measures how far down the page a result appears, which tells you whether it is seen without scrolling. For more on how AI Overviews changed what a ranking means, read rank tracking in the age of AI Overviews.

Organic result #1 pushed below the fold by an AI Overview

Mobile rank tracking through an API

Google uses mobile-first indexing, which means it uses the mobile version of a page for indexing and ranking. That doesn't mean desktop and mobile results are the same. The mobile page has a narrower layout, shows SERP features in a different order, and often places local packs higher. As our Austin test showed, the set of pages can differ as well as their order.

For most sites, a large share of searches happen on phones. If you only track desktop, you may be looking at rankings that most of your visitors never see.

Mobile doubles the call count

Every device is a separate search. Tracking 500 keywords daily on desktop and mobile means 1,000 checks a day, not 500. With a SERP API, that doubles your bill. Rank trackers handle it differently. Some count a mobile keyword as a separate tracked keyword, while others include both devices. Ahrefs, for example, says mobile rankings are available on all Rank Tracker plans. Check this before comparing prices.

What to check before you pick a provider

A device parameter. The API should accept a mobile device value. Some APIs have separate options for iOS and Android tracking, and a tablet setting.
Location depth on mobile. City-level targeting matters even more on mobile, where local results take up more of the screen.
SERP features in the mobile response. Make sure local packs, AI Overviews and other features are parsed for mobile, not only desktop.
How mobile is billed. Find out whether a mobile check costs the same as desktop or counts against a separate limit.

One of the seven APIs in this guide, Serpent API, has no device parameter in its documentation. That rules it out for mobile rank tracking.

The 7 best rank tracking APIs

We start with the four SERP APIs, then cover the three rank tracker APIs. For each one, we cover targeting, data freshness, AI Overview support, cost and where it falls short.

1. SerpApi

SerpApi is a real-time SERP API. Each request runs a live Google search and returns the parsed results page as JSON or Markdown. It also covers Bing, YouTube, Google Maps and dozens of other engines through the same key.

Type: SERP API
Best for: teams that need precise location and device control, or that want rankings inside their own product
Location and device targeting: location at city level, uule, or lat and lon with a radius. Set gl, hl and google_domain for country and language. device accepts desktop, tablet or mobile.
Refresh and scheduling: real-time on every call, on whatever schedule you set. Results are cached for an hour, and cached searches are free. Add no_cache to force a fresh search. SerpTrail adds daily, weekly, biweekly or monthly scheduling if you don't want to write your own.
AI Overview parsing: returned in an ai_overview block. Sometimes Google loads it separately, and the response includes a page_token you pass to the Google AI Overview API.
Starting price and free tier: 250 free searches a month. Paid plans start at $25 for 1,000 searches, then $75 for 5,000 and $150 for 15,000. Only successful searches count, so errors and failed searches are free. See every plan on the SerpApi pricing page.

SerpApi Google Search Playground

A few extras matter for rank tracking. The Pixel Position API (output=json_with_pixel_position) returns the on-page coordinates of each result, so you can measure pixel rank as well as position. Production plans and above include the U.S. Legal Shield, which covers legal questions around scraping public search results. SerpApi also offers an MCP server, so AI assistants can run searches directly.

Limitation: SerpApi costs more per search than queue-based APIs. At 30,000 searches a month, it is $275 against $18 on DataForSEO's standard queue. There is also no way to fetch 100 results in one request. You page through with start, and each page is one search.

2. DataForSEO

DataForSEO sells SERP data by the request, with three speed tiers. You post a task and collect results later, or pay more to get them live.

Type: SERP API
Best for: high-volume tracking where cost per keyword matters more than speed
Location and device targeting: location_code, location_name or location_coordinate (latitude, longitude and radius). device is desktop or mobile, and os sets Windows, macOS, Android or iOS.
Refresh and scheduling: the standard queue returns in about 5 minutes. Live mode returns in up to about 6 seconds. depth goes up to 700 results, billed per page of 10.
AI Overview parsing: returned as an ai_overview item. Setting load_async_ai_overview fetches ones Google loads separately, for an extra $0.0006 per task.
Starting price and free tier: $0.0006 per SERP on the standard queue, $0.0012 on priority and $0.002 live. New accounts get $1 of credit and a free sandbox.

DataForSEO is the cheapest per search of the SERP APIs here. If you track tens of thousands of keywords a day and can wait a few minutes, that gap adds up quickly.

Limitation: the cheapest tier uses a post-then-fetch workflow, which means more code than a single synchronous call. There is no tablet device option.

3. Serpent API

Serpent API is a low-cost SERP API that covers Google, Bing, Yahoo, DuckDuckGo and Brave. It is pay as you go, with discounts for larger deposits.

Type: SERP API
Best for: country-level desktop tracking on a tight budget, or tracking several search engines at once
Location and device targeting: country and language only. The docs list no device or city parameter.
Refresh and scheduling: real time on each call. num accepts 1 to 100 and pages accepts 1 to 10.
AI Overview parsing: include_aio on Deep Search requests, plus include_ai_mode for AI Mode. A paid pixel_position option is also available.
Starting price and free tier: $0.60 per 1,000 calls pay as you go, with a $10 minimum deposit. A $100 deposit unlocks 10× lower pricing and $500 unlocks 20× lower pricing. New accounts get free calls.

Serpent's own blog puts 1,000 keywords tracked daily at about $18 a month on the default tier. Paid credits never expire, which suits irregular usage. Promotional credits expire after 30 days.

Limitation: with no device or city targeting, Serpent API can't do mobile rank tracking or local rank tracking. It fits national desktop tracking only.

4. cloro

cloro is a SERP API with a credit system and a focus on AI search results, including AI Overviews.

Type: SERP API
Best for: tracking AI Overview presence next to classic rankings at a mid-range price
Location and device targeting: country (or gl), plus location by canonical name or uule for city level. hl sets language. device accepts desktop, ios, android or mobile.
Refresh and scheduling: real time on each call. pages fetches 1 to 10 pages.
AI Overview parsing: set include.aioverview for an extra 2 credits per search.
Starting price and free tier: 500 free credits a month. Lite is $30 for 37,500 credits and Hobby is $100 for 250,000. Higher tiers bring the rate down to $0.31 per 1,000 credits.

The math for search credits is a bit complicated here. A Google search costs 3 credits, each extra page adds 2, and AI Overview adds 2. Synchronous requests add 2 more. A synchronous search with AI Overview therefore costs 7 credits.

Limitation: the headline price per 1,000 credits is not a price per search. Unused credits also don't roll over to the next month.

5. Semrush Position Tracking

Semrush Position Tracking is a full rank tracker inside the Semrush suite. Its API lets you create campaigns, add keywords and pull ranking reports into your own tools.

Type: rank tracker API
Best for: agencies and in-house teams already reporting from Semrush
Location and device targeting: locations down to postal code. Devices are desktop, phone and tablet.
Refresh and scheduling: daily updates, tracking up to the top 100 positions.
AI Overview parsing: you can filter tracked keywords by SERP features, including AI Overview.
Starting price and free tier: API access starts on the Advanced plan at $549 a month, billed monthly. There is no API access on the free plan.

API calls use Semrush API units on top of the subscription. Creating a campaign, adding a keyword and requesting a report each cost 100 units. Check how many units your plan includes before you plan a large integration.

Semrush's strength is everything around the rankings. Keyword volumes, competitor visibility and site audits live in the same account, so the ranking data connects to the rest of your SEO work.

Limitation: the price of entry is high if you only need ranking data. API units are a second cost to track, and you are tied to Semrush's daily schedule.

6. AccuRanker

AccuRanker is a dedicated rank tracker known for speed and on-demand refreshes. It is built for agencies and large in-house teams.

Type: rank tracker API
Best for: large keyword sets that need fresh data on demand
Location and device targeting: country or local level. The pages we checked don't state device options.
Refresh and scheduling: daily by default. You can also refresh one keyword or a whole domain on demand, every 2 hours.
AI Overview parsing: not stated on the pages we checked.
Starting price and free tier: an unlimited read API comes with the Expert plan at $764 a month. The write API, for adding keywords and domains, needs an Enterprise plan.

AccuRanker's features page describes full API access at no additional cost. That makes the read API good value if you already need Expert-level tracking. You can pull as much history as you like without counting units.

Limitation: the API sits behind a high-priced plan, and creating projects by API needs Enterprise. Confirm mobile tracking and AI Overview support with sales before you buy.

7. Ahrefs Rank Tracker

Ahrefs Rank Tracker is part of the Ahrefs toolset. Its API returns stored rankings for the projects you set up in Ahrefs.

Type: rank tracker API
Best for: teams already on Ahrefs who can work with weekly data
Location and device targeting: country, state or province, or city. Each request sets device to desktop or mobile.
Refresh and scheduling: weekly updates.
AI Overview parsing: responses list SERP features, including ai_overview. A best_position_kind field shows when your best position is inside an AI Overview.
Starting price and free tier: Lite costs $129 a month and includes 750 tracked keywords. It also includes API and MCP access with 1,000 credits per user.

The rank-tracker-overview endpoint needs a project ID, a device and a date. It returns position, URL, location and SERP features for each keyword. Ahrefs also offers an MCP server, so you can query rankings from an AI assistant.

Limitation: weekly data misses short-term swings, such as the days after an algorithm update. The 1,000 API credits on Lite also limit how often you can pull data.

What rank tracking costs per keyword

SERP APIs charge per search, so the cost of tracking comes down to one formula:

Monthly searches = keywords × devices × locations × checks per month

Take a common setup: 500 keywords, checked on desktop and mobile, from one location, every day. That is 500 × 2 × 1 × 30, or 30,000 searches a month. Here is what that costs with each provider.

Provider

What 30,000 checks needs

Monthly cost

Notes

SerpApi

30,000 searches

$275 (Big Data plan)

Real time; failed and cached searches are free

|
|

DataForSEO

30,000 SERPs

$18 standard, $36 priority, $60 live

Standard queue takes about 5 minutes

|
|

Serpent API

15,000 calls (desktop only)

About $9

No mobile option, so half the setup isn't possible

|
|

cloro

90,000 to 210,000 credits

$100 (Hobby plan)

Range depends on AI Overview and sync requests

|
|

Semrush

500 keywords in Position Tracking

$549 + API units

Advanced plan needed for API

|
|

AccuRanker

500 keywords

$764 (Expert plan)

Unlimited read API

|
|

Ahrefs

500 keywords

$129 (Lite plan)

Weekly updates only, so not a like-for-like match

Three things change the math. Each extra location adds another full set of searches. Tracking beyond the top 10 multiplies SERP API costs, since the end of num=100 means each extra page is a separate request. And rank trackers bill by tracked keyword, so their price stays flat however often they check.

For small keyword lists, a rank tracker can be the simpler buy. Once you track many locations or devices, a SERP API is usually cheaper per check.

FAQ

What is the best rank tracker tool?

It depends on how you'll use the data. For dashboards and client reports, a rank tracker such as Semrush, AccuRanker or Ahrefs is the simplest choice. For tracking inside your own app, or across many locations and devices, use a SERP API like SerpApi or DataForSEO. You get more control at a lower cost per check.

Which AI-powered rank tracker is the best?

Most tools now report AI Overviews in some form. Ahrefs and Semrush flag when one appears for a tracked keyword. SERP APIs like SerpApi, DataForSEO and cloro return the AI Overview text and its cited sources, so you can see whether your site is cited. To try this yourself, follow our guide to track AI Overview rankings in Python.

How do I check my ranking?

For a one-off check, search in a private browser window with location set to your target city. Then compare with the average position in Google Search Console. For ongoing tracking, use a rank tracker or a SERP API so every check uses the same location, device and language.

Start tracking your rankings

SerpApi gives you real-time Google results with city-level and device targeting, AI Overview parsing and pixel positions. Failed searches are never billed. Sign up for a free account to get 250 free searches a month, then try your first ranking check in minutes.

If you need help getting started, contact us. We're happy to help.

What Developers Built With SerpApi at API World 2026

Sonika Arora — Mon, 12 Oct 2026 21:43:46 +0000

At API World 2026, we wanted developers to do more than just hear about SerpApi. So we put our APIs in their hands and let them build.

At our booth, developers could experiment with SerpApi live, connect search data to their existing tools and workflows, and see what they could create with real-time web data. Many developers participated in our hackathon and used SerpApi to build real-time search capabilities, automated workflows, and AI-driven applications.

The result? A collection of 124 amazing projects that showed just how many ways developers can put search data to work.

Check out five of the many interesting projects built with SerpApi at API World.

1. AccessForm

Call. Talk. Your form is filled.

The people who most need help with official paperwork are the ones least able to do it online. A blind person with a hospital bill, a 70-year-old who can't walk to the bus, a student with ADHD facing a disability office form. They all have a phone. Most of them don't have a screen they can use, and some don't have internet at all.

How AccessForm helps

You call a number and say what's going on in your own words. AccessForm works out which official program fits, finds the real application form from the official source, asks you its questions one at a time in plain language, fills the actual PDF, and texts you the result with a list of what's still missing.

Everything happens in one phone call. It never submits, never signs, and never decides eligibility. If it can't verify an official form for the exact organization you named, it says so and stops.

https://www.youtube.com/embed/F-DBDH3wa48?feature=oembed

How SerpApi was used

Vapi runs the voice agent on a Twilio number. Every tool call hits a Next.js backend that uses SerpApi to find and verify the official form, OpenAI to read the form's fields and turn them into spoken questions, pdf-lib to fill the real PDF, Nutrient for accessibility tagging when credits allow, and Xano as the system of record for cases, answers, and completeness. A conversation page rebuilds the call live from Xano's event stream, so a laptop can watch a phone call as it happens.

The developers used SerpApi to search for and discover official application forms (PDFs) based on a user's specified need, organization, and location (e.g., executing live web queries to find official .gov, .edu, or official organizational site links).

Built with: SerpApi's Google Search API

👉 See project details

👉 Github

2. PantryProof

PantryProof turns fragmented recall evidence into cited, human-reviewed actions and a replayable closure packet for every partner pantry.

Food banks and partner pantries form the “true last mile” of a food recall. Products may arrive through central distribution, retail rescue, local purchasing, food drives, and other paths. Partner agencies may operate intermittently or rely on volunteers, so sending an alert does not prove that the affected product was found, contained, or communicated downstream.

PantryProof was built around a simple question: How can a food bank prove that every partner completed the correct recall response without letting an uncertain AI prediction make the final safety decision?

How PantryProof helps

PantryProof streamlines food safety responses across partner pantries by discovering official recall notices via SerpApi, extracting cited data, requiring human approval for extracted fields, and deterministically matching identifiers (UPC/GTIN and lot numbers) against inventory to trigger necessary quarantine, disposal, or notification actions before generating a verified, audit-ready PDF closure packet.

https://player.vimeo.com/video/1223743447?app_id=122963

The browser experience is built with TypeScript, JavaScript, semantic HTML, and responsive CSS. A deterministic TypeScript domain engine owns identifier normalization, matching policy, agency aggregation, human review state, response completeness, and closure readiness.

How SerpApi was used

Developers used SerpApi to get structured, current Google News recall results ordered newest first. They limit results to FDA and USDA domains. PantryProof validates every returned URL, determines evidence tier from the actual hostname rather than an untrusted label, and allows a coordinator to attach a reviewed result as canonical evidence.

This makes web search part of a traceable evidence workflow instead of treating the first search result as truth.

Built with: SerpApi's Google News API

👉 See project details

👉 Github

👉 See it live

3. ClinicalBrief AI (by LearnHive Labs)

Upload patient lab data - GPT-4o searches PubMed, retrieves clinical guidelines, and delivers a signed clinical brief to the clinician in under 60 seconds.

Clinicians spend 2+ hours per complex patient case searching literature, cross-referencing lab values against treatment protocols, and writing structured briefs. Most existing AI tools either hallucinate clinical recommendations or cost $50,000+ per year. There is no open, inspectable, evidence-grounded pipeline that any clinic can run.

How ClinicalBrief AI helps

ClinicalBrief AI was built based on a research publication on adaptive health orchestration to prove that a fully agentic, source-cited, confidence-gated clinical intelligence pipeline can be built on open infrastructure - in n8n, without a single line of server code.

https://www.youtube.com/embed/OMoF1RoaBQA?feature=oembed

A patient or pharmacy staff member fills a Google Sheet with lab values: HbA1c, LDL, BMI, blood pressure, medications, diagnoses. No app. No login. No technical knowledge required.

How SerpApi was used

When a new row is added, the n8n pipeline fires automatically - SerpApi queries Google Scholar in real-time using the patient's diagnoses. Three current PubMed publications are retrieved and injected into the agent prompt before any generation happens. GPT-4o generates clinical briefs using patient data, PubMed evidence, and clinical guidelines. Briefs with a confidence score of 80% or higher are electronically signed and delivered to clinicians, while those below are reviewed by humans. All data is sent to a physician.

Built with: SerpApi's Google Scholar API

👉 See project details

👉 Github

4. ApplyPilot

ApplyPilot finds real opportunities and proves why they match you - using grounded evidence, deterministic scoring, bounded AI, and explicit uncertainty instead of black-box recommendations.

Most AI job-matching systems give candidates a score. But a score can hide a dangerous assumption: when evidence is missing, an AI system may treat that missing information as if it were negative evidence. That idea became the foundation for an evidence-controlled opportunity intelligence system where AI can interpret evidence, but cannot silently manufacture it.

How ApplyPilot helps

ApplyPilot uses a multi-step process to match candidates with real job opportunities. It extracts evidence from résumés, verifies it, and then searches the web for relevant jobs. The final result is a ranked list of opportunities with detailed explanations of why they are a good fit.

The end-to-end workflow is:

Résumé → grounded evidence → human review → candidate profile → live web discovery → deterministic evaluation → bounded AI reasoning → self-audit → ranked opportunities → Why this job?

https://www.youtube.com/embed/X8vQZmbpb20?feature=oembed

How SerpApi was used

ApplyPilot uses SerpApi's Google Jobs API to discover real public opportunities during a Mission. The system preserves the provider query, raw listing evidence, canonical opportunity, and source lineage. Repostings can be deterministically deduplicated while preserving the underlying sources.

SerpApi helps ApplyPilot operate on changing real-world opportunities rather than a static demonstration dataset. Live web data changes the AI experience directly: the reasoning pipeline evaluates opportunities discovered during the run instead of answering from stale or fabricated listings.

Built with: SerpApi's Google Jobs API

👉 See project details

👉 Github

👉 See it live

5. DepositCheck

DepositCheck protects renters from listing scams by using SerpApi's Google Lens API to perform visual reverse image searches on listing photos, matching them against the live web to confirm if an apartment matches its claimed address or belongs to another property.

Rental scams work by theft, not invention. A scammer copies the photos and the description from a real listing, swaps the contact details and the address, and reposts it somewhere with weaker verification. Someone wires a deposit for a property the "landlord" has never owned.

The photos are the one thing the scammer cannot change. Swap them out and the listing stops looking like the flat that attracts victims in the first place. But a renter cannot check that. You would have to recognize one apartment interior among billions of published images, from memory, on a phone, while a "landlord" tells you three other people are viewing tonight.

How DepositCheck helps

You upload one photo from the listing and type the address you were given. SerpApi's google_lens engine with type=exact_matches runs Google's visual matching model over the live web index and returns every page carrying that same image. The claimed address is matched against what those pages say. The photo is deleted as soon as the lookup returns.

One of three verdicts is generated - CORROBORATED (a genuine syndicated listing), CONTRADICTED (the photos belong to one specific other address), or UNVERIFIED (not enough evidence either way).

https://player.vimeo.com/video/1223247077?app_id=122963

How SerpApi was used

DepositCheck uses SerpApi's Google Lens API with type=exact_matches to return every page carrying that same image. The claimed address is matched against what those pages say. The team aims to extend the solution in the future by using Google Maps API to verify the address is a residential property rather than a parking lot or a mailbox storefront and SerpApi's Google Search API on the landlord's phone number and email, which scam operations reuse across dozens of listings.

Built with: SerpApi's Google Lens API, Google Maps API (Future Extension), and Google Search API (Future Extension)

👉 See project details

👉 Github

👉 See it live

What Developers Are Building With Search Data

The projects built at API World were very different from one another, but they had something in common:

Developers were using search as a building block for something bigger

Some projects focused on AI and agents, using search results to give applications access to current information. Others used search data for automation, research, data collection, and discovery.

And because SerpApi returns structured search results in JSON and Markdown, developers can take that data and plug it directly into the applications and workflows they're already building.

Check out all the projects developers built with SerpApi at API World here:

DevNetwork [API + Cloud + AI] Hackathon 2026

The projects built at API World are just a small sample of what's possible.

Build with SerpApi

For us, the most exciting part wasn't simply seeing developers call an API and build a tool using it. It was seeing what they built once they had access to real search data.

Whether you're building an AI agent, research tool, automation, data pipeline, or something completely different, live search results can be an important source of fresh information for your application.

And you don't have to build the search infrastructure yourself.

With SerpApi, developers can access Google and other search engines through an easy and fast API and work with the results in a format that's ready to use.

Have an idea? Start building .

Building something useful with SerpApi? Share it with us.

Interested in exploring our use cases? Check out our use cases page .

Read Some Of Our Latest Posts

Introducing Review Audit: An open-source tool to check how real a Google Maps rating is

Cagdas Salur — Sun, 11 Oct 2026 21:37:48 +0000

Review Audit reads a Google Maps place's newest reviews, pulls the full Google history of the reviewers worth a closer look, and tells you how much of the rating the place's own reviewers can explain. What they can't explain gets set aside, and you see the rating without it. It is open source, runs on your machine with your own SerpApi key, and costs about 42 credits a place.

GitHub - serpapi/reviewaudit: Is this 4.8 real? Reads a Google Maps place’s newest reviews through SerpApi and shows the patterns worth a closer look: first-time reviewers, five-star bursts, reviews posted minutes apart. Runs locally on your own API key.

What an audit looks like

Each place you read gets one page: the verdict, the rating without the reviews it can't explain, and the evidence.

An audit of Gull Rock Dinner Cruise, an invented place from the demo

Every place in these screenshots is invented. Review Audit ships a demo that puts made-up businesses with made-up reviews on real Portland, Maine streets, so the images show the real tool without pointing at a real business. More on that at the end.

Why you can't spot a fake review by reading it

Every business with a rating has an incentive to improve it, and there is a market that will help. On the place page, a review that was paid for or asked for looks exactly like a real one: five stars, a name, a date.

They don't look the same in aggregate. Padded reviews tend to come from accounts that have never reviewed anything else. They arrive together, in office hours, minutes apart. They say the same thing in the same words. None of that is visible from one review, and all of it is visible from 200 of them plus the history of the people who wrote them. That is what Review Audit reads.

What it looks for

Each pattern is measured against the place itself, not against a global rule. A busy lunch spot that gets 30 reviews on a Saturday is not suspicious; a dentist that gets 30 in a week when its normal week is four is.

First-time reviewers

Reviewers with no record like it more than anyone else. Review Audit compares how often first-time accounts give five stars against accounts with 4 to 50 reviews, at the same place. The gap is the finding, not the count. A famous place gets plenty of first reviews; a place where first reviews are five stars far more often than everyone else's has something to explain.

A week with too many five-star reviews

A week carried far more five-star reviews than the place gets. Judged against the place's own median week, and flagged only when it is both at least 2.5 times normal and very unlikely by chance.

Five-star reviews minutes apart

Five-star reviews arrive in batches, minutes apart. More pairs within ten minutes than the hours people actually post at would predict, so a lunch rush is not mistaken for a batch upload.

The same waiter, over and over

Reviews that name the same member of staff, posted in pairs after dinner by people with the same surname, are a different kind of padding: reviews asked for at the table.

Repeated wording and shared reviewers

Reviews that say the same thing in the same words, and groups of accounts that keep reviewing the same places. Two accounts that share other places are usually a couple. Review Audit needs three.

Almost no four-star reviews

Nobody ever gives it four stars. Real places collect a four-star tail: the customer who enjoyed it but waited too long. A wall of fives with nothing beside it means someone is choosing who gets asked.

What it does with what it finds

None of these condemns a single review on its own. What gets set aside is only the excess over what the place's own reviewers explain, and every review set aside is listed with a link back to Google, so you can disagree.

The audits so far, most padded first

What the reviews say

Below the findings, each audit also reads the reviews for what they say: what people praise and what they complain about, food/service/atmosphere sub-ratings with and without the set-aside reviews, the hours people post at, where else the reviewers go, and how often the owner replies.

What the reviews say: praise, complaints, sub-ratings and the hours people post at

Each audit is a single self-contained HTML file, so it can be kept, compared and shared.

Try it in 5 minutes

To install Review Audit, you can use uv tool install command as follows:

uv tool install git+https://github.com/serpapi/reviewaudit
reviewaudit serve

That opens the app. Paste your key from serpapi.com/manage-api-key, search for a place, and press Read the reviews. The free plan's 250 searches a month read about five places.

Searching Google Maps from the app

The read runs in the background and shows each stage, so you can watch where the credits go:

A read in progress, one stage at a time

There is a CLI too:

export SERPAPI_KEY=...
reviewaudit "<place name> <city>"

How it uses SerpApi

Three SerpApi engines, about 42 calls:

step	engine	what it gives
1	Google Maps API	the place, its photo, hours and Google's lifetime star counts
2	Google Maps Reviews API ×10	the 200 newest reviews, each with the reviewer's review count, Local Guide status, photos, sub-ratings and an exact timestamp
3	Google Maps Contributor Reviews API ×30	everything the reviewers worth checking have ever reviewed

The cheap checks run first, so the 30 paid record lookups go to the reviewers they point at. Every response is cached, so re-reading a place or rebuilding its page costs nothing.

If you build on these APIs yourself

A few things worth knowing:

The contributor engine returns relative dates only ("a week ago"); the reviews engine has iso_date. So timing patterns come from the place's reviews, and a reviewer's record is used for what they reviewed, not when.
user.reviews on a review counts written reviews. The full record also has rating-only entries, so an account that says "1 review" can have 19 ratings behind it. Review Audit trusts the record.
The first page of reviews is 8, the rest 20.

What it can't tell you

It reports patterns worth a closer look, not verdicts on individual reviews, which is why every verdict reads "looks". It also has blind spots, and each audit prints them:

A place where everyone gives five stars looks the same whether they mean it or not. A flat 95% is not evidence; the gradient across account depth is.
A first review at a famous place is normal. A thin account alone never counts.
Hotels carry Tripadvisor and Trip.com reviews with no Google account behind them. Those are left out, and the page says how many.
200 reviews at a very busy place can be two weeks. The audit says so when the window is short, and --reviews 600 reads further back.

About those screenshots

Running a tool like this against real businesses and publishing the results is a different thing from building it. So the repo includes tools/demo.py: a fake Google Maps that answers the same three engines in the same shapes, for businesses that don't exist (every name was checked against Google Maps) on real Portland streets, with invented reviews and reviewers. The padding is written in on purpose, a burst here, a waiter named over and over there, and the real Review Audit finds it. Every image in the README file and in this post comes from it.

Get the code

The code is on GitHub. Issues and pull requests are welcome, especially new patterns with a rationale behind them, and staff-name and stop-word lists for languages beyond English and Turkish.

If you don't have a key yet, the free plan is 250 searches a month, enough for about five places.

DEV Community: SerpApi

DeepSeek Search and Fetch APIs vs. SerpApi: Features, Pricing, and Use Cases

What DeepSeek Released

How SerpApi Approaches Search

The Core Difference: Web Retrieval vs. Search Product Data

SEO Use Case DeepSeek Does Not Cover: Cross-Border E-commerce Monitoring

Developer Experience: Raw HTTP vs. Official Client Libraries

Search and Page Extraction

JSON for Code, Markdown for Models

Pricing and Rate Limits

Choosing Between DeepSeek and SerpApi

Choose DeepSeek Search/Fetch API when :

When SerpApi Is the Better Fit

Use them together

Conclusion

DeepSeek #Exract API

Markdown vs. JSON: Which Response Format Should You Use?

Markdown vs. JSON at a glance

Example use cases that benefit from markdown

Use case 1: Building AI agents that research and answer questions

Use case 2: Retrieval-augmented generation (RAG)

Use case 3: Creating readable research summaries and reports

Example use cases that benefit from JSON

Use case 1: Building applications that display search results

Use case 2: Using data in automated pipelines

Use case 3: Monitoring, filtering, and triggering actions

Common Questions

Can you use JSON with LLMs?

Does Markdown use fewer tokens than JSON?

Can you switch between Markdown and JSON in the same application?

How to Scrape Amazon Promotions Coupons

What data does the Amazon Promotion API return?

Getting started with SerpApi

Install the SerpApi library (optional)

Review the Amazon Promotion API documentation

Find the promotion ID

How to scrape Amazon coupons

GET request

Python

JavaScript and Node.js

cURL

Other languages and no-code solutions

Get the products that qualify for a coupon

Track Amazon coupons with Python

FAQ

Does Amazon have a coupon API?

Where do I find an Amazon promotion ID?

Can I find every Amazon coupon with this API?

What happens when a coupon ends?

Does it work on other Amazon marketplaces?

Conclusion

Related posts

Ecommerce Web Scraping: How to Track Competitor Prices, Products, and Stock

What Ecommerce data is worth scraping?

Product detail pages

Search results listings

Why is Ecommerce web scraping difficult?

Scraping an Ecommerce website in Python

Set up SerpApi

Choose your target site and the data you want

Build the Ecommerce web scraper

Key takeaways

How to Scrape Booking.com Hotel Listings and Prices (2026 Tutorial)

What data can you retrieve with the Booking.com Search API?

How to scrape Booking.com hotel listings

Try it in the playground

Get your API key

Make a GET request

cURL implementation

Python implementation

Export results to CSV

JavaScript implementation

Other languages and no-code options

How to paginate the results

Customize or filter the results

Conclusion

Frequently asked questions

Which parameters are required?

Why are prices in different currencies?

Can I search flexible dates?