DEV Community: SerpApi

How to Scrape Walmart Reviews Results

Noraina Nordin — Wed, 02 Sep 2026 04:55:27 +0000

Customer reviews are one of the richest signals Walmart provides. Ratings, review text, and helpfulness feedback tell you what buyers actually think about a product, where it delights, where it disappoints, and which features drive or sink a purchase. In this tutorial, you'll learn how to scrape Walmart review results using a simple API from SerpApi's Walmart Product Reviews API, page through every review for a product, and export the results to CSV for analysis.

This is a companion to our main guide, How to Scrape Walmart, which covers scraping search listings and product details. If you already have a product_id, you can jump straight in below.

Why scrape Walmart reviews?

A single product page can hold hundreds of reviews, and at scale that feedback becomes a dataset. By scraping Walmart reviews, you can:

Run sentiment analysis to understand how buyers feel about a product over time.
Monitor recurring complaints and feature requests to inform product or sourcing decisions.
Compare review sentiment and ratings across competing products before stocking or dropshipping.
Track how ratings shift after a price change, a new variant, or a seller switch.
Feed structured review text into downstream models for summarization or classification.

For e-commerce teams, analysts, and automation builders, review data turns anecdotal feedback into something you can measure and act on.

What can you extract from Walmart product reviews?

The Walmart Product Reviews API returns two things at once. A summary of a product's overall review profile and the individual reviews themselves. Here's what's available:

Product info: The product name, its Walmart URL, and the category path it sits in.
Overall rating: The product's average star rating across all reviews (for example, 4.6).
Total review count: The total number of reviews for the product. It is useful for gauging sample size or weighting sentiment.
Rating distribution: A breakdown of how many reviews fall under each star level, 1 through 5, so you can see the shape of sentiment at a glance rather than just the average.
Top positive review: The highest-rated review that other customers found most helpful.
Top negative review: The lowest-rated review that customers found most helpful. Often, the most actionable complaints surface.
Individual reviews: For every review on the page, you get the title, full review text, star rating, positive and negative feedback counts (how many users up- or down-voted it), submission date, reviewer nickname, and customer type (such as VerifiedPurchaser).

You can also filter by star rating (1 - 5) and sort the results by relevancy, helpfulness, newest or oldest submission, or highest or lowest rating to pull exactly the slice of feedback you need.

Why use an API?

Reviews are paginated, rendered dynamically, and protected by the same anti-bot measures as the rest of Walmart, so a DIY scraper means maintaining pagination logic, rotating proxies, and patching parsers every time the markup shifts.

With SerpApi, that overhead disappears. The Walmart Product Reviews API returns clean, structured review data. No browser automation, no HTML parsing, and get fast response times. You can see live response times and success rates on the SerpApi Status page. For the full picture on scraping Walmart search and product pages, see the complete Walmart scraping guide.

Getting the product_id

To pull reviews for a product, you need to pass the product_id parameter. You can get it from:

The Walmart product URL itself.

Walmart product_id from URL

The Walmart Search results. If you're working from search data, extract the product_id from each organic result. See the main Walmart guide for how to scrape those listings.

You can test any product_id on our interactive playground before writing a line of code.

Walmart product reviews API playground

To learn more about the parameters, visit the Walmart Reviews API documentation.

Walmart Reviews API documentation

How to scrape Walmart reviews

If you have your API key, you're ready to start pulling review data from Walmart. The results will be identical across every method below, so use whichever one you prefer.

Walmart product reviews page

For every review on the page, we'll extract the "title", "rating", "review text", "positive and negative feedback", "review submission time", "user nickname", and "customer type".

GET request

This fetches the first page of reviews for a specific product (product_id), straight from the search.json endpoint:

https://serpapi.com/search.json?engine=walmart_product_reviews&product_id=2205851521&page=1&api_key=SERPAPI_API_KEY

Increment the page parameter to move through additional pages of reviews. By default, each page returns 20 reviews.

Python Tutorial

The examples below use the official SerpApi Python library. This is the most thorough walkthrough. It also covers paging through every review and exporting the results to CSV.

Setup

After installing the serpapi-python package, import the libraries and load your API key.

import serpapi
import os, csv
from dotenv import load_dotenv

load_dotenv()

Note: Make sure you create a .env file to store your API key.

Define the parameters. The page parameter is optional. By default, one page returns 20 reviews.

params = {
    'api_key': os.getenv("SERPAPI_API_KEY"),
    'engine': 'walmart_product_reviews',
    'product_id': '2205851521',
    'page': 1
}

Initialize the SerpApi client:

client = serpapi.Client()

Send the Walmart product reviews request:

results = client.search(params)

Parse the reviews

Loop through the reviews on the page and print each field:

reviews = results.get('reviews', [])

print("Reviews:")
for review in reviews:
    title = review.get('title')
    review_text = review.get('text')
    rating = review.get('rating')
    positive_feedback = review.get('positive_feedback')
    negative_feedback = review.get('negative_feedback')
    review_submission_time = review.get('review_submission_time')
    user_nickname = review.get('user_nickname')
    customer_type = review.get('customer_type')

    print(f"Title: {title}")
    print(f"Review text: {review_text}")
    print(f"Rating: {rating}")
    print(f"Positive feedback: {positive_feedback}")
    print(f"Negative feedback: {negative_feedback}")
    print(f"Review submission time: {review_submission_time}")
    print(f"User nickname: {user_nickname}")
    print(f"Customer type: {customer_type}")
    print("-" * 50)

The output

Walmart product reviews results in terminal

Scrape every page of reviews

A single request returns one page (20 reviews). Popular products have many pages, so to collect all of them you can increment the page parameter until the API stops returning reviews. We add a max_pages cap so the loop always terminates:

client = serpapi.Client()

all_reviews = []
page = 1
max_pages = 50  # safety cap so the loop can't run away

while page <= max_pages:
    params = {
        'api_key': os.getenv("SERPAPI_API_KEY"),
        'engine': 'walmart_product_reviews',
        'product_id': '2205851521',
        'page': page
    }
    results = client.search(params)
    reviews = results.get('reviews', [])

    if not reviews:
        break

    all_reviews.extend(reviews)
    print(f"Page {page}: collected {len(reviews)} reviews (total: {len(all_reviews)})")
    page += 1

print(f"Done. Collected {len(all_reviews)} reviews in total.")

Export the reviews to CSV

Printing to the terminal is useful for debugging, but in real workflows you'll want the data saved for analysis. Here's how to write every collected review to a CSV file you can open in Excel or Google Sheets:

header = [
    'title', 'rating', 'text', 'positive_feedback', 'negative_feedback',
    'review_submission_time', 'user_nickname', 'customer_type'
]

with open('walmart_reviews.csv', 'w', encoding='UTF8', newline='') as f:
    writer = csv.writer(f)
    writer.writerow(header)

    for review in all_reviews:
        writer.writerow([
            review.get('title'),
            review.get('rating'),
            review.get('text'),
            review.get('positive_feedback'),
            review.get('negative_feedback'),
            review.get('review_submission_time'),
            review.get('user_nickname'),
            review.get('customer_type'),
        ])

You now have a clean, structured CSV containing every review for the product, ready for sentiment analysis, dashboards, or cross-product comparisons.

JavaScript and Node.js

This example uses the SerpApi JavaScript library to fetch the first page of reviews for a product and print the title, rating, and text of each one:

import { getJson } from 'serpapi';

const search = await getJson({
  engine: "walmart_product_reviews",
  api_key: SERPAPI_API_KEY,
  product_id: "2205851521",
  page: 1
});

for (let review of search?.reviews ?? []) {
  console.log(`${review.title} - ${review.rating}`);
  console.log(review.text);
  console.log("-".repeat(50));
}

To collect every review, increment the page value until the API stops returning results:

import { getJson } from 'serpapi';

const allReviews = [];
let page = 1;
const maxPages = 50; // safety cap so the loop can't run away

while (page <= maxPages) {
  const search = await getJson({
    engine: "walmart_product_reviews",
    api_key: SERPAPI_API_KEY,
    product_id: "2205851521",
    page
  });

  const reviews = search?.reviews ?? [];
  if (reviews.length === 0) break;

  allReviews.push(...reviews);
  console.log(`Page ${page}: collected ${reviews.length} reviews (total: ${allReviews.length})`);
  page++;
}

console.log(`Done. Collected ${allReviews.length} reviews in total.`);

cURL

This fetches the first page of reviews for a specific product:

curl --get https://serpapi.com/search \
 -d api_key="YOUR_KEY_GOES_HERE" \
 -d engine="walmart_product_reviews" \
 -d product_id="2205851521" \
 -d page="1"

Conclusion

Walmart reviews are a high-signal, high-volume source of customer feedback, but scraping them reliably means wrestling with pagination and anti-bot protections. In this tutorial, we used SerpApi's Walmart Product Reviews API to skip that overhead and pull structured review data directly. You learned how to:

Find the product_id you need to query reviews
Retrieve and parse reviews for a specific product
Page through every review with a bounded loop
Export the results to CSV for downstream analysis

Want the full workflow including search listings and product details as well as reviews? Read the complete guide to scraping Walmart.

Ready to start collecting Walmart review data without maintaining a scraper? Create your free SerpApi account today.

Understanding HTTP 429: Too Many Requests

Josef Strzibny — Mon, 31 Aug 2026 20:46:16 +0000

HTTP 429 Too Many Requests is the status code a server returns when a client exceeds a rate limit for a given URL. This often happens when exhausting API quotas or when scraping public web pages too aggressively. The error doesn't necessarily say that requesting the resource isn't allowed, but rather points out that the server cannot handle more requests like that at the moment. Let's go and understand what this code means exactly and how to make it go away.

HTTP status codes

HTTP status codes tell clients what happened with the request they sent. They are grouped by their first digit into 5 categories. The group tells a client whether a request succeeded, requires another step, or failed:

Range	Meaning	Common examples
`1xx`	Informational response while processing continues	`100 Continue`
`2xx`	The request succeeded	`200 OK`, `201 Created`, `204 No Content`
`3xx`	The client needs to follow a redirect or use cached content	`301 Moved Permanently`, `304 Not Modified`
`4xx`	The client must change something about the request or its behavior	`400 Bad Request`, `401 Unauthorized`, `404 Not Found`, `429 Too Many Requests`
`5xx`	The server failed to complete an otherwise valid request	`500 Internal Server Error`, `502 Bad Gateway`, `503 Service Unavailable`

HTTP 429 belongs to the 4xx group because the client is expected to change its behavior by sending fewer requests or waiting before trying again. It does not necessarily mean the request data is malformed or that the server is broken.

A request can be valid and still receive a 429 because it arrived after the client's allowed quota had been exhausted.

HTTP 429: Too Many Requests

You might encounter this error while calling an API, scraping a website, submitting forms, polling for updates, or running several background jobs at once. The request may be perfectly valid. The problem is how frequently the requests are being sent.

A typical response looks like this:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 60

{
  "error": "rate_limit_exceeded",
  "message": "Try again in 60 seconds"
}

The server is telling the client to slow down. Retrying the same request immediately usually extends the problem rather than fixing it.

HTTP 429 versus HTTP 503

Both status codes call for controlled retries. The distinction matters for monitoring as repeated 429 responses usually point to client behavior or plan limits, while widespread 503 responses usually indicate a server-side availability problem.

Here's the main difference between these two error status codes:

HTTP 429 means the client has exceeded a limit. The response is usually specific to an API key, account, IP address, or request pattern.
HTTP 503, known as Service Unavailable, generally means the service itself cannot handle the request because it is overloaded or undergoing maintenance.

Why servers return HTTP 429

Rate limits protect services from accidental overload, abusive traffic, and unexpectedly expensive workloads. They also help providers distribute limited capacity fairly among users.

Common causes include:

Sending too many requests per second or minute
Running more concurrent requests than an API plan permits
Sharing one API key across too many workers
Polling an endpoint more frequently than necessary
Retrying failed requests immediately and without a limit
Scraping pages faster than a website can reasonably serve them
Exceeding a daily or monthly account quota

Not every limit is based on an IP address. A service may limit requests by API key, account, endpoint, user, geographic region, or a combination of these factors.

Check the Retry-After header

A 429 response may include a Retry-After header telling the client when it can try again. The value can be a number of seconds:

Retry-After: 60

But it can also be an HTTP date:

Retry-After: Wed, 26 Aug 2026 14:30:00 GMT

Clients should support both formats when possible.

Some APIs also return headers describing the active limit, remaining requests, and reset time. Header names vary between providers, so check the API documentation instead of assuming one universal format.

A 503 response may also contain Retry-After, but reducing one client's request rate might not resolve the underlying outage.

How clients should handle HTTP 429

The correct response to receiving the Too Many Requests status code is to reduce pressure on the server immediately and be more mindful in the future.

Respect Retry-After

Do not retry before the server's requested delay has passed. If several workers receive a 429 together, add a small random delay so they do not all retry at exactly the same moment.

Use exponential backoff and jitter

When Retry-After is not specified, increase the delay after every failure. If that doesn't help, increase it again exponentially. You can also consider adding jitter which adds randomness to each delay. This prevents synchronized clients from creating another traffic spike when the waiting period ends.

Limit retries

A retry loop must have a maximum number of attempts. Permanent account quotas, invalid plans, and strict website limits will not be fixed by retrying forever.

Reduce concurrency

A scraper with many workers can exceed a limit even when each worker appears slow. Use a shared rate limiter or queue so all processes follow the same request budget.

Cache and batch requests

Avoid requesting the same resource repeatedly. Cache responses when freshness requirements allow it, combine requests when an API supports batching, and stop polling when the result is no longer needed.

Authenticate correctly

Anonymous requests often have lower limits than authenticated ones. Confirm that the API key is present, belongs to the expected account, and is not being shared unintentionally across environments.

Retrying a 429 response

Here's a simplified Ruby example that respects a numeric Retry-After value and otherwise falls back to exponential backoff with jitter:

require "faraday"

MAX_RETRIES = 5
attempt = 0

loop do
  response = Faraday.get("https://api.example.com/data")
  break unless response.status == 429

  raise "Rate limit exceeded" if attempt >= MAX_RETRIES

  retry_after = Integer(response.headers["retry-after"], exception: false)
  backoff = [2**attempt, 60].min
  delay = retry_after || backoff

  sleep(delay + rand)
  attempt += 1
end

A production implementation should also handle an HTTP-date Retry-After value, request timeouts, network errors, logging, cancellation, and the API's documented rate-limit headers.

Conclusion

If you are getting HTTP error 429 when using an external API or when scraping public web sites, it's important to stop the requests immediately and retry later. How much later can be determined from the Retry-After header if present. If the header is not present, try exponential backoff with jitter to not overwhelm the servers on the other side.

Introducing SerpApi's New Markdown Output

Tomás Murúa — Mon, 24 Aug 2026 19:59:59 +0000

We’re proud to announce the launch of Markdown Output, a new feature designed to deliver real-time search data in a format optimized for LLMs and AI agents. It is available on every search, on every API, and it comes back from the same structured search that JSON does. This is SerpApi’s effort to adapt and improve our product for LLMs and AI agents. Based on our benchmarks, the new output gives you roughly half the tokens of JSON on average and up to 90% for some APIs. If you already use our APIs, you can implement it right away; no configuration is needed, and it works automatically across our 100+ APIs.

SerpApi: Markdown Output for AI Agents

In this post, we’ll show what the new Markdown Output looks like on a real search and explain when to use it and when to stay on JSON.

Getting Markdown results does not require a new endpoint. Any existing search can return it in three ways:

# Query parameter
curl "https://serpapi.com/search?engine=google&q=coffee&api_key=YOUR_KEY&output=md"

# Route extension
curl "https://serpapi.com/search.md?engine=google&q=coffee&api_key=YOUR_KEY"

# Accept header
curl -H "Accept: text/markdown" "https://serpapi.com/search?engine=google&q=coffee&api_key=YOUR_KEY"

Why Markdown for LLMs

Markdown is a plain-text format with headers, tables, links, and lists that you can read raw or render as structure. In the last few years, it also became the default language of LLMs. Models are trained on enormous amounts of it (documentation, READMEs, forums); chat interfaces render their answers from it, and OpenAI's own prompt engineering guide recommends Markdown headers and lists to communicate hierarchy to the models.

We took that signal from our users and from the industry, everyone converging on Markdown as the common input and output format for LLMs and agents, and adapted our product to it. Every design choice in the new output serves one goal, to fit more useful data in your model's context window. Let's see each one with a real search for coffee on Google.

SerpApi Playground - SerpApi

Token Efficient

Take the first organic result from a coffee search on Google. This is everything JSON carries for it:

{
  "position": 1,
  "title": "Coffee",
  "link": "https://en.wikipedia.org/wiki/Coffee",
  "redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://en.wikipedia.org/wiki/Coffee&ved=2ahUKEwiOsKzK96uWAxXmQjABHecdCtsQFnoECCcQAQ",
  "displayed_link": "https://en.wikipedia.org › wiki › Coffee",
  "thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQs1E6FVWWD12JBVWlsho0fGc7GxAR50Pf030IPyxJNl5A9xYPmgt_1&usqp=CAE&s",
  "favicon": "https://serpapi.com/images/i/iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b-u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz_6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1-flswMu1EQZ3xlE7lK8eKDkjtwE-crBMV-wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk_9ryhFMAAAAASUVORK5CYII.png",
  "snippet": "Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content;",
  "snippet_highlighted_words": [
    "a beverage brewed from roasted ground coffee beans"
  ],
  "about_this_result": {
    "source": {
      "description": "Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content;",
      "source_info_link": "https://en.wikipedia.org/wiki/Coffee",
      "icon": "https://serpapi.com/images/i/iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b-u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz_6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1-flswMu1EQZ3xlE7lK8eKDkjtwE-crBMV-wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk_9ryhFMAAAAASUVORK5CYII.png"
    },
    "languages": [
      "en"
    ],
    "regions": [
      "US"
    ]
  },
  "about_page_link": "https://www.google.com/search?q=About+https://en.wikipedia.org/wiki/Coffee&tbm=ilp",
  "about_page_serpapi_link": "https://serpapi.com/search.json?engine=google_about_this_result&google_domain=google.com&q=About+https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FCoffee",
  "source": "Wikipedia"
}

And the same result as a Markdown table row:

## Organic Results

| Position | Title | Thumbnail | Snippet | About Page Link | Source | Read More Link | Date |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | [Coffee](https://en.wikipedia.org/wiki/Coffee) | ![Thumbnail](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQs1E6FVWWD12JBVWlsho0fGc7GxAR50Pf030IPyxJNl5A9xYPmgt_1&usqp=CAE&s) | Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content; | https://www.google.com/search?q=About+https://en.wikipedia.org/wiki/Coffee&tbm=ilp | Wikipedia |  |  |

Nothing the model needs is missing. The position, title, thumbnail, snippet, and source are all kept in the output, and the row costs a fraction of the tokens the object did. Across the full coffee response, the JSON weighs 24,723 tokens and the markdown 6,435, a 74% cut.

Here are some examples:

Token savings across SerpApi APIs

Native Markdown Links

In JSON a result splits its data across separate fields:

"title": "Coffee",
"link": "https://en.wikipedia.org/wiki/Coffee",
"displayed_link": "https://en.wikipedia.org › wiki › Coffee"

In Markdown that is one cell:

[Coffee](https://en.wikipedia.org/wiki/Coffee)

The model gets the title and the URL already bound together.

No Noise

The same JSON result also carries a redirect_link (a 150-character Google tracking URL), a 320-character favicon URL, and internal metadata objects:

"redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://en.wikipedia.org/wiki/Coffee&ved=2ahUKEwiOsKzK96uWAxXmQjABHecdCtsQFnoECCcQAQ",
"favicon": "https://serpapi.com/images/i/iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b-u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz_6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1-flswMu1EQZ3xlE7lK8eKDkjtwE-crBMV-wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk_9ryhFMAAAAASUVORK5CYII.png"

None of that helps a model answer a question about coffee. The Markdown Output strips them automatically, and you can check the table row above; none of these fields made it in. The title, link, snippet, and source remain.

Structured Output

Markdown does not flatten the response into prose. Each result array becomes a table, and the nine organic results of our coffee search arrive as nine rows under one header:

## Organic Results

| Position | Title | Thumbnail | Snippet | About Page Link | Source | Read More Link | Date |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | [Coffee](https://en.wikipedia.org/wiki/Coffee) | ![Thumbnail](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQs1E6FVWWD12JBVWlsho0fGc7GxAR50Pf030IPyxJNl5A9xYPmgt_1&usqp=CAE&s) | Coffee is a beverage brewed from roasted ground coffee beans. Dark-colored and bitter, coffee has a stimulating effect on humans due to its caffeine content; | https://www.google.com/search?q=About+https://en.wikipedia.org/wiki/Coffee&tbm=ilp | Wikipedia |  |  |
| 2 | [Fresh Roasted, Single Origin Coffee Beans](https://creaturecoffee.co/collections/coffee?srsltid=AfmBOoo5rAxqBgQmreq6MaHX-U_rw-P5UI-90YLvKI1oRGTosk_eXwMU) |  | Creature Coffee roasts single origin coffee beans from around the globe right here in Austin, Texas. | https://www.google.com/search?q=About+https://creaturecoffee.co/collections/coffee?srsltid=AfmBOoo5rAxqBgQmreq6MaHX-U_rw-P5UI-90YLvKI1oRGTosk_eXwMU&tbm=ilp | Creature Coffee Co |  |  |
| 3 | [Coffee Shops Austin, TX - Last Updated August 2026](https://www.yelp.com/search?find_desc=Coffee+Shops&find_loc=Austin%2C+TX) |  | Coffee Shops Austin, TX · 1. Mazúr Coffee · 2. Palomino Coffee · 3. Idlewild Coffee · 4. Desnudo Coffee · 5. Flora Coffee · 6. Daydreamer Coffee. 3.9 ... | https://www.google.com/search?q=About+https://www.yelp.com/search?find_desc=Coffee+Shops&find_loc=Austin%2C+TX&tbm=ilp | Yelp | https://www.yelp.com/search?find_desc=Coffee+Shops&find_loc=Austin,+TX#:~:text=Coffee%20Shops%20Austin%2C%20TX&text=1.%20Maz%C3%BAr%20Coffee&text=2.%20Palomino%20Coffee&text=3.%20Idlewild%20Coffee&text=4.%20Desnudo%20Coffee&text=5.%20Flora%20Coffee&text=6.%20Daydreamer%20Coffee,3.9%20(25%20reviews) |  |
| 4 | [r/Coffee](https://www.reddit.com/r/Coffee/) |  | This is a place to share and talk about your favorite coffee roasters or beans. recommendations, reviews, etc. | https://www.google.com/search?q=About+https://www.reddit.com/r/Coffee/&tbm=ilp | Reddit · r/Coffee |  |  |
| 5 | [Cuvée Coffee: Always Something Brewing. Austin, TX](https://cuveecoffee.com/?srsltid=AfmBOorjX3_f5tAt5zldI8Vqr4pNmmfh2FRWlZcVVMEGXhowAfzLZ_Jt) |  | At Cuvée, coffee isn't a job, it's a passion. Since 1998, we've become known for roasting exceptional, ethically sourced, sustainable coffee. | https://www.google.com/search?q=About+https://cuveecoffee.com/?srsltid=AfmBOorjX3_f5tAt5zldI8Vqr4pNmmfh2FRWlZcVVMEGXhowAfzLZ_Jt&tbm=ilp | Cuvée Coffee |  |  |
| 6 | [Coffee - The Nutrition Source](https://nutritionsource.hsph.harvard.edu/food-features/coffee/) |  | Various polyphenols in coffee have been shown to prevent cancer cell growth in animal studies. Coffee has also been associated with decreased estrogen levels, a ... | https://www.google.com/search?q=About+https://nutritionsource.hsph.harvard.edu/food-features/coffee/&tbm=ilp | The Nutrition Source | https://nutritionsource.hsph.harvard.edu/food-features/coffee/#:~:text=Type%202%20Diabetes |  |
| 7 | [Texas Coffee Traders](https://texascoffeetraders.com/) |  | No information is available for this page. | https://www.google.com/search?q=About+https://texascoffeetraders.com/&tbm=ilp | Texas Coffee Traders |  |  |
| 8 | [33 Best Coffee Shops In Austin, Sorted By Neighborhood](https://www.atasteofkoko.com/visit-austin/coffee-shops-in-austin) |  | Best Coffee Shops in Central Austin · Hank's · Civil Goat Coffee Roasters (Central) · Merit Coffee (Central) · Bennu Coffee · Terrible Love. | https://www.google.com/search?q=About+https://www.atasteofkoko.com/visit-austin/coffee-shops-in-austin&tbm=ilp | A Taste of Koko | https://www.atasteofkoko.com/visit-austin/coffee-shops-in-austin#:~:text=Best%20Coffee%20Shops%20in%20Central,Austin&text=Hank's&text=Civil%20Goat%20Coffee%20Roasters%20(Central)&text=Merit%20Coffee%20(Central)&text=Bennu%20Coffee&text=Terrible%20Love | Apr 22, 2026 |
| 9 | [Try the new gold standard of coffee at home!](https://icoffee.co/) |  | At Industry Coffee, we take great care in every step of our coffee making process. From selecting the best beans, to roasting, brewing, and blending our ... | https://www.google.com/search?q=About+https://icoffee.co/&tbm=ilp | Industry Coffee | https://icoffee.co/#:~:text=At%20Industry%20Coffee%2C%20we%20take,committed%20to%20quality%20and%20consistency. |  |

The same happens for shopping results, hotel listings, and any other result array.

The response also opens with a YAML frontmatter block carrying the search parameters and metadata:

---
search_metadata:
  id: 6a853ba34c437ff2e1aa1265
  status: Success
  json_endpoint: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.json"
  markdown_endpoint: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.md"
  pixel_position_endpoint: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.json_with_pixel_position"
  created_at: "2026-08-19T05:14:11.488Z"
  processed_at: "2026-08-19T05:14:11.497Z"
  google_url: "https://www.google.com/search?q=Coffee&oq=Coffee&uule=w+CAIQICIaQXVzdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM&hl=en&gl=us&sourceid=chrome&ie=UTF-8"
  raw_html_file: "https://serpapi.com/searches/cPnrW0xTHlgDT2yVVjFSmg/6a853ba34c437ff2e1aa1265.html"
  total_time_taken: 1.91
search_parameters:
  engine: google
  q: Coffee
  location_requested: Austin, Texas, United States
  location_used: Austin,Texas,United States
  google_domain: google.com
  hl: en
  gl: us
  device: desktop
---

So even in text form you still have the search id, status, and the endpoints to fetch the same search again in either format.

When to Choose Markdown or JSON?

Both formats come from the same search, so this is not a migration. It is a per-call choice.

Markdown fits when:

You feed results straight into an LLM prompt or a RAG context
An agent consumes search results as a tool response
You want a quick human-readable digest of a search
Tokens are the budget you optimize for

JSON fits when:

Your code extracts fields programmatically (prices as numbers, coordinates, ratings)
You handle pagination, filters, or metadata downstream
You store results in a database with a fixed schema
You need every field, including the ones Markdown strips

Simply put: if a model reads it, use Markdown. If code reads it, use JSON.

Using the JSON Restrictor

Before Markdown Output, the tool for cutting response size was the JSON Restrictor, and it is still the right tool when you stay on JSON. You pass a field selector with the request, and SerpApi filters the response server-side before sending it back. We have used it for exactly this purpose in earlier posts, like connecting a local LLM to web search data, where trimming the JSON was the difference between fitting the context window or not.

SerpApi: JSON Restrictor

Cutting Tokens While Staying on JSON

If your pipeline needs structured fields but not all of them, the restrictor alone gets you most of the way. We'll request only the two fields an agent might rank on, position and title:

curl "https://serpapi.com/search?engine=google&q=Coffee&location=Austin,+Texas,+United+States&api_key=YOUR_KEY&json_restrictor=organic_results[].position,organic_results[].title"

The response is the whole search reduced to this:

{
  "organic_results": [
    {
      "position": 1,
      "title": "Coffee"
    },
    {
      "position": 2,
      "title": "Cuvée Coffee: Always Something Brewing. Austin, TX"
    },
    {
      "position": 3,
      "title": "Coffee Shops Austin, TX - Last Updated August 2026"
    },
    {
      "position": 4,
      "title": "r/Coffee"
    },
    {
      "position": 5,
      "title": "Coffee - The Nutrition Source"
    },
    {
      "position": 6,
      "title": "Texas Coffee Traders"
    },
    {
      "position": 7,
      "title": "50 Best Austin Coffee Shops You Should Try"
    },
    {
      "position": 8,
      "title": "Buy Coffee, Tea, Powders Online | The Coffee Bean & Tea ..."
    },
    {
      "position": 9,
      "title": "History of coffee - NCA"
    }
  ]
}

You keep exact types and programmatic access, and drop every section your code never touches. The difference with Markdown Output is what each one removes. The restrictor cuts entire fields and sections you name, while Markdown keeps every section but re-renders it in a lighter format. One subtracts data, the other changes its shape.

Stacking Both

You don't have to pick one. Pass json_restrictor on a Markdown request, and SerpApi first filters the sections, then renders only what is left as Markdown:

curl "https://serpapi.com/search.md?engine=google&q=Coffee&location=Austin,+Texas,+United+States&api_key=YOUR_KEY&json_restrictor=organic_results"

The response is just the Organic Results table from earlier, nothing else. On our coffee search, the four levels look like this:

Request	Tokens
Full JSON	24,723
JSON + `json_restrictor=organic_results`	8,486
Full markdown	6,435
Markdown + `json_restrictor=organic_results`	1,298

That is a 95% reduction from the original JSON, and you chose exactly which part of the search enters the model's context. For an agent that only reasons over organic results, everything else was noise anyway.

Conclusion

Markdown Output is live today on every SerpApi plan, at no extra cost, across all 100+ APIs. Same searches, same credits, and around 50% fewer tokens on average and up to 90% for some APIs.

The fastest way to see what it does for you is to take an API you already use, add output=md to one request, and count the tokens before and after.

Other Resources

How a Lead Generation Company Scaled Outreach with SerpApi's Google Maps API

Sonika Arora — Sat, 22 Aug 2026 03:43:22 +0000

For businesses that rely on outbound sales, growth often comes down to one thing: a consistent pipeline of qualified leads.

For a lead generation company specializing in Google Business Profile optimization and client acquisition through email marketing, generating those leads at scale was becoming increasingly challenging. As demand grew, the team needed a faster and more reliable way to collect business data from Google Maps without the operational headaches that came with managing their own scraping infrastructure.

In their COO's words:

"SerpApi is the only platform that has really helped us expand rapidly."

By leveraging SerpApi's Google Maps API, the company was able to dramatically increase its lead generation capacity and support the rapid growth of its outreach efforts.

📌 Customer Snapshot

Industry: Lead Generation & Marketing

Primary APIs: Google Maps API, Google Maps Reviews API

Use Case: Business Lead Discovery

Scale: Millions of searches monthly

Key Benefit: Faster lead generation through high-concurrency data collection

About The Use Case

The company specializes in helping businesses improve their presence on Google Business Profile (formerly Google My Business) and acquire new customers through targeted email marketing campaigns.

The company's COO oversees the systems used to identify and engage potential customers. For this process to succeed, reliable access to accurate business data from Google Maps is essential.

Why Google Maps Data Matters For Businesses

Google Maps is more than a directory of businesses - it's a foundation of any company's lead generation process. By identifying businesses with specific characteristics, a company can build highly targeted outreach campaigns for any industry.

Additionally, reviews data from Google Maps can help businesses understand customer sentiment, identify strengths and weaknesses, and improve products or services based on real user feedback. It also supports competitor analysis, market research, reputation monitoring, and AI-powered recommendations.

The Challenge: Generating Leads at Scale

Before adopting SerpApi, the team relied on a combination of internal tools and alternative approaches to gather business information from Google Maps.

They even built their own Selenium-based scraping solution, which worked for smaller workloads. However, as the business grew, so did the volume of data required. They needed to scrape tens of thousands of business listings every day to support their lead generation workflows.

At that scale, several challenges emerged:

Internal scraping infrastructure wasn't fast enough.
High request volumes increased the risk of IP blocks.
Maintaining reliable data collection required significant effort.
Lead generation capacity was becoming a bottleneck for growth.

The team explored various options, and settled on using SerpApi.

In their words:

"Well, we always had issues generating fast enough leads - which obviously caused declines in revenue. With SerpApi, we fixed that."

Why SerpApi?

SerpApi streamlines the process of web-scraping, manages the intricacies of scraping and returns structured JSON results.

We do all the work to maintain all of our parsers and adapt them to respond to changes on Google's side. This is important, as Google is constantly experimenting with new layouts, new elements, and other changes. By taking care of this for you on our side, we eliminate a lot of time and complexity from your workflow.

According to their COO, one feature stood out above everything else: concurrency.

"We were looking for an API that scrapes Google Maps. Building our own Selenium tool worked, and we have used it, but we need to scrape over 40k places everyday, that would

cause our IP’s to be blocked and

It isn’t fast enough."

The ability to run large volumes of searches simultaneously allows the company to process data at a scale that would have been difficult to achieve with internally managed tools.

Building a Scalable Lead Generation Engine

Today, they use SerpApi's Google Maps API as a core component of their lead generation workflow. They have developed their own proprietary algorithm that identifies and evaluates business profiles from Google Maps, helping the team discover high-value prospects for outreach campaigns.

Empowered with the data that the company gets using SerpApi, it can focus on improving its lead qualification process and scaling outreach operations.

The Impact

The biggest benefit was removing a critical growth constraint. Before implementing SerpApi, they frequently struggled to generate enough leads quickly enough to support their sales efforts.

"Well, we always had issues generating leads fast enough, which obviously caused declines in revenue. With SerpApi, we fixed that."

With a more scalable and reliable data collection pipeline in place, they significantly increased their lead generation capacity, enabling the business to support more outreach campaigns and accelerate growth.

In addition to the product itself, the team highlighted the responsiveness of the SerpApi team as a key part of their experience.

"Very fast responses and very helpful."

Looking Ahead

As they continue expanding their lead generation efforts, scalable access to Google Maps data remains a critical part of their growth strategy.

By replacing the limitations of self-managed scraping infrastructure with SerpApi's Google Maps API, the company has been able to focus less on data collection challenges and more on what matters most: finding and connecting with new customers.

When asked whether they would recommend SerpApi to others, their COO's answer was jokingly reluctant:

"No, because I want to gatekeep it. "

Fortunately for everyone else, they're willing to share their success story. Behind the humor is a clear result. By replacing the limitations of self-managed scraping infrastructure with SerpApi's Google Maps API, they transformed lead generation from a bottleneck into a scalable growth engine.

About SerpApi

SerpApi provides real-time search engine data APIs, including Google Search API, Google Maps API, Google Shopping API, Amazon Search API, Yelp Search API, and many more (See all our APIs here). Developers and businesses use SerpApi to access structured search data in JSON format.

Relevant Links

Explore More Google Maps API Use Cases

How to Scrape Google Flights

Noraina Nordin — Fri, 14 Aug 2026 10:50:50 +0000

Google Flights is Google's free flight search service, part of the Google Travel platform. It aggregates flights from multiple airlines so travelers can search by departure and destination, compare prices across carriers, track fares, and book tickets in one place.

That aggregated data is exactly what makes Google Flights worth scraping. If you want to build or research flight prices, routes, and availability, scraping Google Flights gives you a rich, continuously updated data source, and SerpApi's Google Flights API lets you pull it as structured JSON in Python, JavaScript, or in any programming language of your choice without maintaining your own scraper.

Google Flights User Interface

Who Needs the Scraped Google Flights Data?

Travel agencies and online travel businesses use it for market analysis and competitive intelligence. They can track competitors' pricing strategies, route popularity, and demand trends. With that visibility, they can adjust their own fares and offers to stay competitive.

Developers use it to build flight comparison platforms, fare trackers, and AI travel assistants. All of these need live prices, routes, and availability. The API supplies that data on demand, so there's no fragile in-house flight scraper to build or maintain.

Available Data on the Google Flights API

Before scraping, it helps to understand the structure of the data returned.

Flight Results

A response is organized into two main arrays:

best_flights : The top options for your search criteria. However, note that these results are not always returned by Google Flights itself.
other_flights : The remaining options, in the same structure.

Each entry contains a list of individual flights with:

Departure and arrival airports and times
Duration, airplane model, airline, travel class, and flight number
Layover details (duration, and whether it's overnight)
The journey's total_duration, carbon emissions data, and price
An extensions array of flight features, and ticket_also_sold_by
A departure_token (used to fetch the return legs of a round trip)

Full field-by-field detail is in the Google Flights API documentation.

Price Insights

Each response also carries a price_insights object, so you can tell whether a fare is a good deal:

lowest_price : The cheapest ticket among the options
price_level : The affordability tier for that price
typical_price_range : A [low, high] array of expected prices for the route
price_history : Timestamped price points, ideal for a price-tracking chart

More detail in the price insights documentation. We'll read these fields in the Scrape flight prices section.

Basic Google Flights Data Scraping

You can get structured Google Flights data with a single GET request using SerpApi's Google Flights API.

Prefer to watch? Here's a quick walkthrough of scraping Google Flights with the API:

https://www.youtube.com/embed/zqZSvuj7JMs?feature=oembed

Setting Up a SerpApi Account

SerpApi offers a free plan for newly created accounts. Head to the sign-up page to register an account and complete your first search with our interactive playground.

When you want to do more searches with us, please visit the pricing page.

Once you are familiar with all the results, you can utilize SERP APIs using yourAPI Key.

SerpApi Google Flights API data from playground

Search by Airport Code

Search a route by IATA airport code (find codes on IATA's site or Google Flights).

Let's find the best flights to Berlin. The data contains: "total_duration", "price", "type", "number_of_flights", and more.

cURL implementation

The simplest way to test the API:

curl --get https://serpapi.com/search \
 -d api_key="SERPAPI_API_KEY" \
 -d engine="google_flights" \
 -d departure_id="AUS" \
 -d arrival_id="BER" \
 -d outbound_date="2026-09-15" \
 -d return_date="2026-09-22"

Python Implementation

First, install the SerpApi client library.

pip install serpapi

Get your API Key fromSerpApi credentials and run a basic search. Store your key as an environment variable (export SERPAPI_KEY=your_key) and read it with os.getenv:

import os
import serpapi

client = serpapi.Client(api_key=os.getenv("SERPAPI_API_KEY"))

params = {
    "engine": "google_flights",
    "departure_id": "AUS",
    "arrival_id": "BER",
    "outbound_date": "2026-09-15",
    "return_date": "2026-09-22",
    "currency": "USD",
    "hl": "en",
}

results = client.search(params)
print(results)

The results object behaves like a regular dictionary, so you can read fields straight off it.

Let's say you only want the price, total duration, type, and number of flights from each of the best options:

for item in results.get("best_flights", []):
    print("Price:", item.get("price"))
    print("Total duration (min):", item.get("total_duration"))
    print("Type:", item.get("type"))
    print("Number of flights:", len(item.get("flights", [])))
    print("-" * 20)

Finally, let's export the data to a CSV file for easier analysis.

import csv

header = ["total_duration", "price", "type", "number_of_flights"]

with open("google_flights.csv", "w", encoding="UTF8", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(header)
    for item in results.get("best_flights", []):
        writer.writerow([
            item.get("total_duration"),
            item.get("price"),
            item.get("type"),
            len(item.get("flights", [])),
        ])

print("Data exported to google_flights.csv")

The result:

Google Flight Result in CSV format

JavaScript Implementation

Install the serpapi package:

npm install serpapi

Run a basic query:

const { getJson } = require("serpapi");

getJson({
    api_key: "SERPAPI_API_KEY",
    engine: "google_flights",
    departure_id: "AUS",
    arrival_id: "BER",
    outbound_date: "2026-09-15",
    return_date: "2026-09-22",
    currency: "USD",
    hl: "en",
}, (results) => {
    console.log(results["best_flights"]);
});

More on the library: serpapi-javascript on GitHub.

You can call the API with a plain GET request in any language. Ready-made libraries for Ruby, PHP, Java, Go, and more are listed on SerpApi Integrations.

Search by city ID

To include every airport in a city, search by city ID instead of an airport code. City IDs can be retrieved with the Google Maps API:

params.update({"departure_id": "/m/04jpl", "arrival_id": "/m/07dfk"})
results = client.search(params)

Customize the Flight Search

The API exposes the same controls you'd use on Google Flights itself. Add any of these to the params dict from the basic search, then run client.search(params) again.

Round-trip vs. one-way

Set type to 1 for a round trip (default), 2 for one-way, or 3 for multi-city. A round trip needs a return_date:

params.update({
    "type": "1",
    "outbound_date": "2026-09-15",
    "return_date": "2026-09-22",
})

Retrieving the returning flights

A round-trip search returns only the outbound flights first. Each outbound flight carries its own departure_token, you need to pass it back to get that flight's return options:

params.update({
    "type": "1",
    "departure_token": "PASTE_DEPARTURE_TOKEN_FROM_PREVIOUS_RESPONSE",
})

(This is the Google Flights equivalent of pagination. You drill into a flight rather than page through a list.)

Scrape flight prices and price insights

Prices are usually the whole point of a flight price scraper. Every response carries the fare on each option plus the price_insights object described in the data structure.

import os
import serpapi

client = serpapi.Client(api_key=os.getenv("SERPAPI_API_KEY"))

results = client.search({
    "engine": "google_flights",
    "departure_id": "JFK",
    "arrival_id": "LHR",
    "outbound_date": "2026-09-15",
    "return_date": "2026-09-22",
    "currency": "USD",
})

insights = results.get("price_insights", {})
print("Lowest price:", insights.get("lowest_price"))
print("Price level:", insights.get("price_level"))            # e.g. "low", "typical", "high"
print("Typical range:", insights.get("typical_price_range"))  # [low, high]

# The cheapest fares from the results themselves
for item in results.get("best_flights", []):
    print(item.get("price"), "-", item.get("total_duration"), "min")

Because the response is already structured, scraping flight pricing is just reading fields without HTML parsing or broken selectors. To watch a route over time, run the same request on a schedule and store lowest_price and price_history on each run.

Cabin class

travel_class: 1 Economy (default), 2 Premium economy, 3 Business, 4 First.

params.update({"travel_class": "3"})   # Business

Filter by airline

exclude_airlines drops carriers while include_airlines keeps only the ones you name (comma-separated). The two can't be combined:

params.update({"exclude_airlines": "MH,EY"})   # or {"include_airlines": "QF"}

Stops, duration, and other filters

Filter by number of stops with stops: 0 any (default), 1 nonstop only, 2 one stop or fewer, 3 two stops or fewer.

params.update({"stops": "1"})   # nonstop only

Add any of these to narrow results further:

max_duration : The total trip-time cap in minutes (e.g. "1400")
layover_duration : Layover range in minutes, two-number string (e.g. "250,300")
exclude_conns : Exclude a connecting airport by ID (e.g. "AUH")
outbound_times / return_times : Departure hour ranges (e.g. "14,18" = 2–6 PM)
max_price : Upper price limit for the whole trip (e.g. "600")
sort_by : 1 Top flights (default), 2 Price, 3 Departure time, 4 Arrival time, 5 Duration, 6 Emissions

For full list check out the advanced filter parameters in the documentation.

Flexible dates

Google Flights has no native flexible-date search, but a few lines turn the API into a flexible-date flight scraper. Loop over a window and collect the cheapest fare per day:

import os
import serpapi
from datetime import date, timedelta

client = serpapi.Client(api_key=os.getenv("SERPAPI_API_KEY"))

def search_flexible_dates(origin, destination, flexible_days):
    start = date(2026, 9, 15)
    for i in range(flexible_days):
        outbound = start + timedelta(days=i)
        return_date = outbound + timedelta(days=7)   # adjust trip length as needed

        results = client.search({
            "engine": "google_flights",
            "departure_id": origin,
            "arrival_id": destination,
            "outbound_date": outbound.isoformat(),
            "return_date": return_date.isoformat(),
            "currency": "USD",
            "hl": "en",
            "type": "1",
        })
        print(f"\nOutbound: {outbound} | Return: {return_date}")
        for item in results.get("best_flights", []):
            for flight in item["flights"]:
                print(
                    f"{flight['flight_number']} | "
                    f"{flight['departure_airport']['id']} -> {flight['arrival_airport']['id']} | "
                    f"{flight['airline']} | {item['total_duration']} min | "
                    f"{item['price']} {results['search_parameters']['currency']}"
                )

search_flexible_dates("JFK", "LAX", 3)

Groups of passengers

Tailor results to a group with adults, children, infants_in_seat, and infants_on_lap:

params.update({
    "adults": "3",
    "children": "2",
    "infants_in_seat": "1",
    "infants_on_lap": "1",
})

Bonus: Deals and airport autocomplete

Hunting for cheap fares specifically? See the Google Flights Deals API and our tutorial: How to Scrape Google Flights Deals.
Building a search box? The Google Flights Autocomplete API returns airport and city suggestions.

Frequently Asked Questions (FAQs)

Is it legal to scrape Google Flights?

Collecting publicly available data is generally permitted, and using an API like SerpApi keeps you off Google's infrastructure. Always review the applicable terms of service for your use case.

How do I scrape flight prices?

Send a Google Flights API request for your route and read the price on each flight plus the price_insights object (lowest_price, price_level, typical_price_range, price_history). See the scrape flight prices section above for a working example.

How do I scrape Google Flights in Python?

Install the serpapi package (pip install serpapi), create a client with your API key, and call client.search() with engine="google_flights" and your route parameters. The results are returned in JSON format, so no HTML scraping is required. The full code is in Basic Google Flights Data Scraping.

How much does it cost?

Why scrape Google Flights data?

Live flight prices, routes, and availability power price-comparison tools, fare trackers, travel dashboards, and market analysis for travel businesses without maintaining a fragile scraper.

Wrapping Up

That's it! You can now scrape Google Flights end-to-end: set up the API, understand the response, run a basic search, and customize it by route, dates, prices, class, filters, and passengers. Experiment for free in the Google Flights playground.

Building a broader travel product? Pair this with our other travel APIs:

If you have any questions, feel free to contact our team at contact@serpapi.com.

Introducing SerpApi Search Tools: Real-Time Web Search for Python AI Agents

Adarsh D — Sun, 09 Aug 2026 10:17:07 +0000

serpapi-search-tools gives Python AI agents real-time access to the web, news, maps, images, shopping, videos, hotels, flights, and travel search, powered by SerpApi. It works with 14 popular agent SDKs, including LangChain, Agno, CrewAI, OpenAI Agents, Pydantic AI, and Google ADK.

AI agents are useful, but they cannot answer questions about current news, prices, places, or travel options unless they have a way to search. SerpApi lets applications search services such as Google, Bing, Google Maps, Google News, Google Shopping, YouTube, Google Hotels, and Google Flights, then returns organized data that the agent can use.

We released serpapi-search-tools to make that search data easy to use in Python agents. The package gives your agent ready-to-use search tools, so you can focus on what the agent should do instead of building a search integration from scratch.

The package is open source on GitHub, and the complete guides and examples are available in the documentation.

Add real-time web search to your first Python agent

This example uses the OpenAI Agents SDK and gives a simple agent access to web search.

Install

You need Python 3.10 or newer, a SerpApi account, and an API key for the model provider your agent uses. This example uses OpenAI.

For this tutorial, install serpapi-search-tools and the OpenAI Agents SDK together using pip:

pip install "serpapi-search-tools[openai-agents]"

If OpenAI Agents is already installed, you can install only the search tools:

pip install serpapi-search-tools

If you use uv, run:

uv add "serpapi-search-tools[openai-agents]"

For another agent SDK, choose its install option from the SDK examples.

Add your API keys

Your agent needs a SerpApi key for search and an OpenAI key for the model:

export SERPAPI_API_KEY="your-serpapi-key"
export OPENAI_API_KEY="your-openai-key"

You can create a SerpApi account and copy your private API key from the SerpApi dashboard.

Build your first agent

Save this as search_agent.py:

import asyncio

from agents import Agent, Runner
from serpapi_search_tools import web_search


async def main():
    agent = Agent(
        name="research-agent",
        model="gpt-5.6-luna",
        instructions=(
            "Use web search for current facts."
        ),
        tools=[web_search()],
    )

    result = await Runner.run(
        agent,
        "Find three new Python features and briefly explain them.",
    )

    for item in result.new_items:
        if item.type == "tool_call_item":
            print(f"Tool called: {item.tool_name}")
            print(f"Arguments: {item.raw_item.arguments}")

    print("Agent Response: ", result.final_output)


asyncio.run(main())

Run it:

python search_agent.py

The example prints each tool call and its arguments before the final answer. This makes the agent's search process visible, including the queries it created and the search engine it selected.

Here is the output from an actual run:

Tool called: web_search
Arguments: {"query":"Python latest release new features Python 3.14 official what's new","engine":"google_light"}

Tool called: web_search
Arguments: {"query":"site:python.org/downloads/release Python 3.14 new features","engine":"google_light"}

Agent Response:  According to the official Python 3.14 documentation, three notable new features are:

1. **Template string literals (t-strings)** — A flexible way to create customized string-processing templates, useful for safer formatting and domain-specific text handling.

2. **Deferred evaluation of annotations** — Type annotations are evaluated later rather than immediately, reducing import problems and improving compatibility with forward references.

3. **Standard-library subinterpreters** — Python now provides tools for running isolated interpreters within one process, enabling better parallelism and isolation.

Source: [Python 3.14 “What’s New”](https://docs.python.org/3/whatsnew/3.14.html)

The two printed calls show that the agent searched more than once to verify the answer. web_search() gave it the search capability, while the instruction told it to use that capability for current facts.

Nine search tools for common agent tasks

Different searches need different information. A hotel search needs stay dates, while a flight search needs airports and a travel date. serpapi-search-tools gives your agent a focused tool for each task so it knows what information to provide.

What you want your agent to do	Search tool	Search source
Research a current topic	`web_search`	Google Light by default, with Google, Bing, Yahoo, or DuckDuckGo available
Follow the latest news	`news_search`	Google News
Find places and local businesses	`maps_search`	Google Maps
Find images and visual references	`images_search`	Google Images
Compare products, prices, and sellers	`shopping_search`	Google Shopping, Amazon, Walmart, or eBay
Find videos on YouTube	`videos_search`	YouTube
Find hotel stays and prices	`hotels_search`	Google Hotels
Compare flights for a route	`flights_search`	Google Flights
Discover possible destinations	`travel_explore_search`	Google Travel Explore

Choose only the tools your agent needs. A shopping assistant might use web, shopping, and image search. A trip planner might use flights, hotels, maps, and travel exploration.

One package for 14 Python agent SDKs

An agent SDK is the Python library you use to build and run an agent. You can use serpapi-search-tools with the SDK you already know, and each link below opens a complete example.

Supported SDK	Start here
OpenAI Agents SDK	OpenAI Agents example
Pydantic AI	Pydantic AI example
LangChain	LangChain example
LangGraph	LangGraph example
CrewAI	CrewAI example
LlamaIndex	LlamaIndex example
Claude Agent SDK	Claude Agent SDK example
Microsoft Agent Framework	Microsoft Agent Framework example
AutoGen	AutoGen example
Haystack	Haystack example
Semantic Kernel	Semantic Kernel example
Agno	Agno example
smolagents	smolagents example
Google ADK	Google ADK example

The same search tools in every SDK

The search tool names stay the same across supported SDKs:

from serpapi_search_tools import maps_search, news_search, web_search

search_tools = [
    web_search(),
    news_search(),
    maps_search(),
]

Add these to your SDK the way you normally add tools. If you later switch SDKs, the surrounding agent code changes but your search setup stays familiar: web_search() is still web_search(). The SDK examples show the complete setup for every supported integration.

The package recognizes your SDK automatically

In a typical project, you install one supported agent SDK and call a search tool such as web_search(). The package recognizes the installed SDK and prepares the tool for it automatically.

You usually do not need any extra setup. If your project has more than one supported SDK installed, the agent SDK guide shows how to select the one you want.

Customize search for your agent

The defaults are a good place to start, so you can use a simple call such as web_search() in your first agent. When you need more control, these six recipes cover common search and response settings.

1. Use one fast web search engine

Google Light is the default web engine. You can make it the agent's only choice and set the language, country, result count, and timeout in your application:

from serpapi_search_tools import web_search

search = web_search(
    allowed_engines=["google_light"],
    default_params={"num": 3, "hl": "en", "gl": "us"},
    timeout=20.0,
)

The agent still chooses the search query. Your application keeps control of the search engine and settings.

2. Give the agent regional search choices

Create separately named tools when the agent needs to search different countries or languages:

from serpapi_search_tools import web_search

search_us = web_search(
    allowed_engines=["google_light"],
    default_params={"gl": "us", "hl": "en", "num": 3},
    name="web_search_us",
)
search_de = web_search(
    allowed_engines=["google_light"],
    default_params={"gl": "de", "hl": "de", "num": 3},
    name="web_search_de",
)

tools = [search_us, search_de]

The names help the agent choose the right regional search for the question.

3. Compare products across marketplaces

Create separate shopping tools when you want the agent to compare results from different marketplaces:

from serpapi_search_tools import shopping_search

google_products = shopping_search(
    allowed_engines=["google_shopping"],
    default_params={"gl": "us", "hl": "en", "num": 5},
    name="google_products",
)
amazon_products = shopping_search(
    allowed_engines=["amazon"],
    default_params={"num": 5},
    name="amazon_products",
)

tools = [google_products, amazon_products]

Google Shopping can compare products across merchants, while Amazon searches its own marketplace. You can create similar tools for Walmart and eBay.

4. Add safe, localized image search

Keep safe search, language, and country settings under your application's control:

from serpapi_search_tools import images_search

safe_images = images_search(
    default_params={"safe": "active", "hl": "en", "gl": "us"},
    name="safe_image_search",
)

The agent only needs to describe what images it wants to find; your application applies the search policy every time.

5. Keep travel prices in one currency

Use the same currency and locale across flight and hotel tools so their prices are easier to compare:

from serpapi_search_tools import flights_search, hotels_search

travel_defaults = {"currency": "USD", "hl": "en", "gl": "us"}

flight_search = flights_search(
    default_params=travel_defaults,
    name="us_flights",
)
hotel_search = hotels_search(
    default_params=travel_defaults,
    name="us_hotel_prices",
)

tools = [flight_search, hotel_search]

The agent still provides the route, destination, and dates. Your application keeps the display currency and locale consistent.

6. Choose compact or full results

Every search tool uses compact results by default. Compact mode keeps the response focused, which is usually the best choice for an agent:

from serpapi_search_tools import web_search

search = web_search()

If your application needs the complete SerpApi response, including additional sections and metadata, choose full mode:

from serpapi_search_tools import SearchResultMode, web_search

search = web_search(mode=SearchResultMode.FULL)

Use full results only when your application needs the extra data; compact results help avoid filling the model's context with information it may not use.

You can also configure locations, result limits, and other tool-specific settings. The configuration guide contains the complete set of options and examples. Visit the full documentation for installation, SDK guides, recipes, and API details.

Complete agent projects you can copy

The agent cookbook contains complete projects for every supported SDK. Each guide includes setup instructions, a prompt you can edit, runnable code, and an output you can inspect.

SDK	Cookbook agent	SerpApi capabilities used
LangChain	Deep market research brief	Web and news
LangGraph	Product-launch intelligence graph	Web, news, and shopping
CrewAI	Collaborative trip planner	Flights, hotels, and maps
LlamaIndex	Remote-work destination brief	Travel Explore, web, and maps
OpenAI Agents	Managed research report	Web and news
Claude Agent SDK	Source-verification memo	Web and news
Pydantic AI	Visual location scout	Images, maps, and web
Microsoft Agent Framework	Technology due-diligence memo	Web and news
AutoGen	Company intelligence memo	Web and news
Haystack	Weekly industry newsletter	Web and news
Semantic Kernel	Plan-and-execute competitor brief	Web and news
Agno	Market research report	Web, news, and shopping
smolagents	Purchase research assistant	Shopping, images, and videos
Google ADK	Retail location strategy	Maps, web, and news

Start with the cookbook project closest to your idea, then change the prompt and search tools to fit your use case.

Learn more about building agents

If you are new to AI agents or want a longer tutorial, continue with these guides:

Building an AI Agent in Python explains agents, prompts, context, memory, tools, MCP, and skills from the beginning.
Build Smarter Pydantic AI Agents with Real-Time Search shows how to add SerpApi search to a Pydantic AI agent step by step.
Build an AI Agent with the Claude Agent SDK shows how to build a Claude agent and connect it to custom tools.

Start building

Install the option for your SDK, add one or two search tools, and start from the cookbook project closest to your idea. The package is on PyPI, the source is on GitHub, and issues and pull requests are welcome.

If you are new to SerpApi, create a free account and give your Python agent real-time search data in a few minutes.

Measuring World Cup Attention: What the data says about Vozinha

Tomás Murúa — Sun, 09 Aug 2026 02:13:34 +0000

During the 2026 World Cup, Cape Verde played for the first time, and its goalkeeper Vozinha had the kind of attention that nobody could have imagined before the event started.

His Instagram following stood at roughly 27,000 before the tournament. Then Cape Verde held Spain to a goalless draw, with Vozinha stopping all eight shots on target from the 23 Spain produced. In the days after that match his following passed 14 million. By the end of the tournament it had grown past 29 million.

Vozinha's story raised a question. How many smaller versions of that story were buried across 48 teams and 104 matches?

So during the tournament, we tracked search interest, news coverage, and Instagram followers for a watchlist of players to answer a simple question. Beyond the trophy, who actually captured the world's attention?

This blog is about what the data showed, the player stories hiding in it, and what any of it can and cannot tell you.

The final attention leaderboard in Sports Pulse

Everything in this post comes from Sports Pulse, a live dashboard I built on SerpApi's APIs to track this, the same data teams use for sponsorship scouting, media monitoring, and market research. Every number below has a page there where you can see it for yourself.

Sports Pulse

The Question That Turned Into a Dashboard

The World Cup writes down everything that happens on the pitch. Every goal, every save, every minute played. What it does not write down is who the world went looking for while it happened. That record is scattered across search queries, news archives, and follower counts, three things measured in different units and stored in different shapes. The match report tells you Vozinha played four games. It does not tell you the world went looking for him afterwards.

Vozinha’s World Cup stats

So I built a pipeline that assembles that second record. For every player it produces a single attention score from 0 to 100, built from three signals. Search interest, news coverage, and follower growth, each compared against the rest of the field. The score answers one narrow question.

Relative to everyone else at this tournament, how much attention did this player draw during the event?

It is not a measure of how good a player is, and it is not a measure of fame. A player who was already massively popular before the tournament and stayed that way will score low, because the score is built around movement. And it never says why something happened. A spike after a match is a spike after a match.

I scored a watchlist of 39 hand-picked players rather than the 1,248 players spread across the 48 squads. Scoring all of them means measuring hundreds of players nobody searched for, which buries the signal in noise. So I picked the players most likely to draw attention, based on their profiles going in, minutes played, and how far their team went. Less data, better data.

The 39-player watchlist

Reading the Board

The board does not rank the best players at the World Cup. It ranks who moved the most while it was happening, and the players at the top got there in completely different ways.

The Top Six

The top six by attention score

Lamine Yamal

Yamal finishes first with a score of 73.6, and he is the only player on the board who is strong in all three signals at once. The highest search interest of the tournament, 727 articles, and 13.7 million new followers across Spain's eight-match run to the title.

Yamal's page in Sports Pulse

That is the sanity check. A measure of tournament attention that did not pull Yamal near the top would be measuring something else or completely wrong. What makes him unusual is not the total; it is that he has no weak column.

Vozinha

Vozinha finishes fourth with an attention score of 56.4, across four matches.

Vozinha's page Sports Pulse

He gained 29.5 million followers during the window, the largest movement in the field by a distance nobody else is close to. His whole score rests in one column, and that is the pattern a scoreboard cannot show you and a raw follower count cannot contextualize, because a follower count on its own has no field to compare against.

The per-match Instagram follower graph makes it concrete. The Spain match added 14.2 million followers; the Argentina match in the final added 11.8 million new followers to the Vozinha account.

Vozinha follower growth by match

Lionel Messi

Messi is the most useful player on this watchlist for understanding what the score actually measures.

He has the most news coverage of anyone at the tournament with 783 articles covering him, more than Yamal. His search interest is the second highest on the watchlist. And his follower growth scores exactly zero.

Messi page Sports Pulse

The zero is not missing data, and it is not because he stood still. Messi gained 8.9 million followers during the tournament, more than most players on the watchlist. But on a base of 515 million that is a 0.5% move, and in a field where Vozinha's following multiplied more than a thousand times over, a 0.5% move lands below the middle of the pack and scores nothing.

Messi's follower growth by match

That is the metric behaving exactly as designed and it still looks strange written down. Messi finishing sixth is not a claim that five players had a bigger tournament than him. It is a claim that five players moved more than he did, which is a different sentence, and the difference is the whole point of the exercise.

Haaland and Olise

Vozinha's pattern is not a one-off. Haaland finishes second after gaining 32.8 million followers across six matches.

Haaland page Sports Pulse

And Olise finishes fifth with 6 million gained from a far smaller base. Both maxed out the same follower signal Vozinha did.

Olise page Sports Pulse

Three of the top five got there mostly on follower growth, and that is a finding about tournaments, not about the metric. While the competition is running, the audience moves faster than the press does. Reporters filed a few hundred articles about Vozinha over four weeks. Tens of millions of people tapped follow. His story shows up in follower counts first and everywhere else later.

What the Attention Turned Into

The attention score stops at the tournament window. Vozinha's story did not.

Within weeks of the final he signed with Colo Colo, one of the biggest clubs in Chile, and reports linked him with offers from several other teams, including rumors of a move to Inter Miami to share a locker room with Messi. Sponsorship offers came after. And searches for Cape Verde itself climbed alongside its goalkeeper, a country of half a million people that most of the new audience could not have placed on a map in May.

Cape Verde search interest

All of that happened to a 40-year-old goalkeeper, at an age when most careers are already over. The data in this post cannot tell you what caused any of it. What it can show is the moment the audience arrived, tens of millions of people in two match windows, and everything above followed while the world was watching.

What This Cannot Tell You

Three honest limits, so the numbers above are read as what they are.

It does not explain why. The score measures attention during the tournament. A player whose following grew after a strong performance grew after a strong performance. What produced that growth is outside anything these APIs can see.
It only knows 39 players. Every score is relative to that hand-picked field and would change if the field changed.
It measures attention, not merit. A player can have an outstanding tournament in near silence. The score would show a low number, and it would be right about attention and say nothing about football.

How We Measured This

Everything above comes from one pipeline built on four SerpApi engines. The Google Sports API supplies the skeleton, the tournament's dates, the squads, and all 104 matches, so nobody maintains a match list by hand. On top of that skeleton, three signals are measured for every player on the watchlist over a window taken from the tournament's own dates. The Google Trends API gives search interest, the Google News API gives coverage volume, and the Instagram Profile API gives follower growth rather than raw totals, which is the difference between measuring who is already famous and measuring who is becoming famous.

The three signals arrive in three incompatible units, so each is converted to the same scale before combining into a single 0 to 100 attention score. A player with a gap in one signal is scored on the signals that exist rather than punished with a zero.

How the attention score is measured

How the conversion works and how each engine is queried in practice is a story of its own, and it did not fit here.

Pointing It at the Next Tournament

Nothing in the pipeline knows it is looking at football. The window comes from the event's dates, the player list from the event's structure, and the score from comparing whoever is on the watchlist. Swap the event and the watchlist, and the same three signals work for the Champions League, the NBA, the NFL, and other sports events.

Conclusion

Yamal topped the board by being strong everywhere. Messi put up two of the biggest signals at the tournament next to a zero. And a 40-year-old goalkeeper from a country of half a million people finished fourth on a single column because tens of millions of people went looking for him at once.

One blended number would have flattened all of that into a rank. Keeping the three signals visible is what lets you say why a player is on the board, and in Vozinha's case the why is the entire story.

The dashboard is live at Sports Pulse, with the full board, per player pages with the follower and search curves, per-match breakdowns, and a page that walks through the calculation on real data. I would start with Vozinha's page. The curve is the entire reason this exists.

How to Scrape Google Maps Autocomplete

Noraina Nordin — Thu, 06 Aug 2026 09:56:15 +0000

Google Maps Autocomplete creates a faster and smoother experience when users search for an address, place, or business location. The same as-you-type suggestions developers know from Google Places Autocomplete. Whether you're building a food delivery app, travel platform, store locator, CRM, or local search tool, the Google Maps Autocomplete helps users find the right location with fewer keystrokes.

Google Maps Autocomplete

What Does Google Maps Autocomplete API Do

The Google Maps Autocomplete API by SerpApi allows developers to retrieve autocomplete suggestions from Google Maps with a simple API request. Because each place suggestion comes back with its street address attached, it doubles as an address autocomplete API for any app where users need to find a place or street quickly.

Instead of managing browser automation or reverse engineering Google's autocomplete responses, you receive structured JSON that can be integrated directly into your application. No key juggling, no CAPTCHAs, no proxy pool. You send a query and a location, and you get back the same suggestions Google Maps would show, ready to drop into a search box, an address form, or a data pipeline.

SerpApi's Google Maps Autocomplete API documentation

In this tutorial, you'll learn how to scrape the autocomplete suggestions from Google Maps using a simple API with cURL, Python, and JavaScript, how to customize requests, and what data you can retrieve.

How Does the Google Maps Autocomplete API Work

The API takes a partial query (q) and a set of GPS coordinates (ll) and returns the list of suggestions Google Maps would show for that input from that location. Some suggestions are plain keyword completions, some are specific places, and those come enriched with an address, latitude and longitude, and a data_id.

That data_id is the part worth paying attention to. It's the same identifier the rest of the Google Maps family uses, so an autocomplete suggestion is an entry point, not a dead end. Pick a place from the suggestions, and you can pass its data_id straight to the Google Maps Reviews API or Google Maps Photos API to pull everything Google knows about it.

Try it out in the playground

What data you can extract

Each suggestion in the response can include:

value: the suggested keyword or place name
type: either keyword (a search completion) or place (a specific location)
subtext: for places, typically the street address
latitude/longitude: coordinates of the place
data_id: the place identifier, reusable across the Maps, Reviews, and Photos APIs
reviews_serpapi_link: a ready-made link to that place's reviews
photos_serpapi_link: a ready-made link to that place's photos
maps_serpapi_link : a ready-made link to that place's full Google Maps listing

How to use the Autocomplete API

Here's the quickest way to see the data before writing any code.

Step 1: Open the playground

Head to the Google Maps Autocomplete playground. No sign-up is needed to run a first search.

Step 2: Enter a query and coordinates

Set q to the partial term a user might type, let's say cafe and set ll to the location the search should originate from, in the format @latitude,longitude,zoom, for example @40.7455096,-74.0083012,14z. The zoom value ranges from 3z (fully zoomed out) to 21z (fully zoomed in). If you need to find coordinates for a place, our guide on how to find the GPS coordinates of any place walks through it.

SerpApi's Google Maps Autocomplete Playground

Step 3: Run the search

The response comes back as JSON, with keyword completions and nearby places interleaved the way Google Maps returns them. A trimmed example for q=cafe:

"suggestions":[
   {
      "value":"cafè",
      "serpapi_link":"https://serpapi.com/search.json?engine=google_maps_autocomplete&ll=%4040.7455096%2C-74.0083012%2C14z&q=caf%C3%A8",
      "maps_serpapi_link":"https://serpapi.com/search.json?engine=google_maps&google_domain=google.com&hl=en&ll=%4040.7455096%2C-74.0083012%2C14z&q=caf%C3%A8&type=search",
      "type":"keyword"
   },
   {
      "value":"Cafe Paradiso",
      "serpapi_link":"https://serpapi.com/search.json?engine=google_maps_autocomplete&ll=%4040.7455096%2C-74.0083012%2C14z&q=Cafe+Paradiso",
      "subtext":"West 65th Street, New York, NY",
      "type":"place",
      "latitude":40.7733834,
      "longitude":-73.9835585,
      "data_id":"0x89c2596625484969:0xdcf19d6c26e407c2",
      "reviews_serpapi_link":"https://serpapi.com/search.json?data_id=0x89c2596625484969%3A0xdcf19d6c26e407c2&engine=google_maps_reviews&hl=en",
      "photos_serpapi_link":"https://serpapi.com/search.json?data_id=0x89c2596625484969%3A0xdcf19d6c26e407c2&engine=google_maps_photos&hl=en",
      "maps_serpapi_link":"https://serpapi.com/search.json?data=%214m5%213m4%211s0x89c2596625484969%3A0xdcf19d6c26e407c2%218m2%213d40.7733834%214d-73.9835585&engine=google_maps&google_domain=google.com&hl=en&ll=%4040.7455096%2C-74.0083012%2C14z&q=Cafe+Paradiso&type=place"
   },

  ...
  ...
  ...

  }
]

cURL

The quickest way to call the API from the command line is a single GET request. Pass your query in q, the origin coordinates in ll, and your API key:

curl --get "https://serpapi.com/search" \
  -d engine="google_maps_autocomplete" \
  -d q="cafe" \
  -d ll="@40.7455096,-74.0083012,14z" \
  -d api_key="YOUR_SERPAPI_API_KEY"

The response is the same structured JSON shown above. From here, the Python and JavaScript examples below wrap the same request so you can parse the suggestions in your app.

Python tutorial

First install the SerpApi client library (GitHub):

pip install serpapi

Then set your API key and run the search:

import serpapi

client = serpapi.Client(api_key="YOUR_SERPAPI_API_KEY")
results = client.search({
    "engine": "google_maps_autocomplete",
    "q": "cafe",
    "ll": "@40.7455096,-74.0083012,14z",
    "gl": "us",
    "hl": "en"
})

print(results)

To get the suggestions and their type:

for suggestion in results.get("suggestions", []):
    print(suggestion["value"], "-", suggestion.get("type"))

To keep only the place suggestions and their data_id:

places = [s for s in results.get("suggestions", []) if s.get("type") == "place"]

for place in places:
    print(place["value"], place.get("subtext"), place.get("data_id"))

The result:

Results from the Python script

JavaScript tutorial

Install the package:

npm install serpapi

The example code:

import { getJson } from "serpapi";

const API_KEY = process.env["API_KEY"]; // https://serpapi.com/manage-api-key

async function getSuggestions(query) {
  const response = await getJson("google_maps_autocomplete", {
    api_key: API_KEY,
    q: query,
    ll: "@40.7455096,-74.0083012,14z",
    gl: "us",
    hl: "en"
  });
  return response["suggestions"];
}

console.log(await getSuggestions("cafe"));

To get the suggestions and their type:

const suggestions = await getSuggestions("cafe");

for (const suggestion of suggestions) {
  console.log(suggestion.value, "-", suggestion.type);
}

To keep only the place suggestions and their data_id:

const places = suggestions.filter((s) => s.type === "place");

for (const place of places) {
  console.log(place.value, place.subtext, place.data_id);
}

Since every place suggestion carries a data_id, you can pass it straight to the Reviews or Photos API to pull that place's reviews or photos, with no separate request to look it up.

Common use cases

Search-as-you-type place pickers. Feed the suggestions straight into an autocomplete dropdown so users can select a business or landmark as they type, with coordinates already attached.
Address and location entry. Speed up checkout, sign-up, and booking forms with location suggestions, without standing up and metering your own Places Autocomplete key.
Seeding data pipelines. Use autocomplete as the first step in a scrape. Resolve a partial name into a concrete data_id, then pull the full place, its reviews, and its photos from the rest of the Maps APIs.
Local keyword and place discovery. Vary the ll coordinates to see which places and completions Google surfaces from different locations, useful for local research and coverage checks.

FAQ

Is the Google Maps Autocomplete API free?

SerpApi's free plan includes 250 searches per month with no credit card required, which is enough to build and test an integration before you commit to a plan.

Do I need to pass coordinates?

Yes. The ll parameter is required, since autocomplete results depend on where the search originates. The gl and hl parameters for country and language are optional.

Can I use it as an address autocomplete API?

Yes. Because each place suggestion comes back with its street address attached, it works as an address autocomplete API for address forms, checkout pages, and any interface where users need address suggestions.

Can I get full place details from a suggestion?

Yes. Each place suggestion includes a data_id you can pass to the Google Maps API, Reviews API, or Photos API to retrieve ratings, hours, reviews, photos, and more.

How is this different from Google Places Autocomplete?

Google Places Autocomplete is Google's own service. The difference is operational: with SerpApi, there's no Google Cloud project, API key, or billing to set up, and no session tokens to manage. Every place suggestion chains directly into SerpApi's other Google Maps endpoints for reviews, photos, and full place details.

Documentation

For a broader look at pulling place data, ratings, and reviews from Google Maps, see our Google Maps Scraper guide. If you're after search keyword suggestions rather than places, our guide on how to scrape Google Autocomplete results covers the Google Search side.

You can reach us at contact@serpapi.com for any questions.

Build an AI Event Discovery Agent That Adds Events to Your Calendar in One Click

Sonika Arora — Tue, 04 Aug 2026 01:42:20 +0000

AI and software professionals often miss valuable conferences, meetups, and webinars because relevant events are scattered across dozens of websites. Instead of manually searching and comparing listings every week, an AI agent can continuously discover events that match your interests, location, and schedule, and then even add them directly to your calendar.

In this tutorial, we'll build an AI-powered Event Discovery Agent using SerpApi MCP and Claude that takes care of discovery and recommendation. You'll be able to give the agent instructions, using which it will discover events and produce a clean markdown report with recommended events.

Follow along and by the end, you'll have built a personalized AI event concierge.

What We'll Build

The goal is to be able to ask Claude:

Find upcoming AI conferences, meetups, and webinars.

My preferences:
- Located within an hour from San Francisco
- Interested in AI agents, MCP, RAG, and developer tools
- Budget under $500
- Prefer in-person events
- Within the next 90 days

Generate a Markdown report and provide Google Calendar links.

Claude will use SerpApi MCP to search Google events and gather structured event data. Then, it will rank events by relevance, create calendar links and generate a final markdown report.

Example output we aim to see:

Setup

Create a SerpApi Account

For this tutorial, we're going to use SerpApi's Google Events API.

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API key here: https://serpapi.com/manage-api-key

Install Claude Code

💡 You'll need to create an account with Claude in case you don't have one yet. You can do that here: https://claude.ai/login

To access Claude Code, you'll need a Claude Pro, Max, or Team subscription OR you can pay per usage via Claude API.

Install Claude Code:

npm install -g @anthropic-ai/claude-code

Verify the installation:

claude --version

You should see the installed version printed in your terminal.

Configure SerpApi MCP

SerpApi's MCP server allows AI assistants to perform live searches using your SerpApi subscription without requiring custom tool implementation. Claude Code can connect to external MCP Servers like SerpApi's to perform searches.

We'll add the SerpApi MCP server to Claude Code:

claude mcp add --transport http serpapi https://mcp.serpapi.com/YOUR_SERPAPI_API_KEY/mcp

Once configured, Claude Code can access SerpApi tools directly.

Verify the MCP Connection

Run the following command in your terminal:

claude mcp list

You should see output similar to:

Configured MCP servers:

✓ serpapi
  URL: https://serpapi.com/...
  Status: Connected

If the serpapi server appears in the list, Claude Code has successfully detected the MCP server and can use SerpApi tools during your session. If you don't see the server listed, double-check your MCP configuration and API key, then rerun the setup command.

💡 If you need to remove an incorrectly configured serpapi MCP server, use this command to remove it before rerunning the setup command: claude mcp remove serpapi

Create a Project Directory

Create a workspace for the project:

mkdir event-discovery-agent
cd event-discovery-agent

We'll store generated reports here.

For example:

event-discovery-agent/
├── events.md
└── prompts.md

The events.md file will be generated by Claude.

Define the Agent's Instructions

Here, we provide instructions describing the events we want. This step is important because we're not just searching. We're defining decision criteria.

Think of this as the equivalent of giving requirements to a human assistant.

Without requirements :

Find AI events.

With requirements :

Find AI events focused on developers,
exclude expensive conferences,
prioritize in-person networking opportunities.

The second prompt produces dramatically better results.

Let's create a file called prompts.md in the event-discovery-agent/ directory and add a detailed prompt there:

Use SerpApi MCP to find upcoming conferences, meetups, and webinars related to AI agents, MCP, RAG, and developer tooling.

Requirements:
- Within 100 miles of San Francisco
- Budget under $500
- Next 90 days
- Prefer in-person events

Create a file called events.md.

For each event include:
- Event title
- Date
- Location
- Event URL
- Why it matches my interests
- Google Calendar link

This prompt is the agent's operating instructions.

Run the Event Discovery Agent

Start Claude Code

Head over to the event-discovery-agent/ directory and run:

claude

The first time you run it, Claude Code will prompt you to log in. Once you're logged in, you'll see an interactive prompt similar to:

You can now chat with Claude Code directly. By default, Claude Code will have access to all files in the event-discovery-agent/ directory.

Send Claude the prompt file

Now simply give Claude Code the prompt:

Read prompts.md and complete the task.

Claude will follow the instructions in prompts.md and the final output file events.md (with a curated list of events) will be created in the same directory once Claude finishes.

💡 Claude may ask you to confirm if it can perform certain searches. In that case, you must respond with yes so it can complete the task successfully.

Behind the Scenes of the Agent

A software script works great to filter events by date or location. But the agent can evaluate relevance.

For example, suppose an event description contains:

Enterprise AI

Agent Frameworks

RAG Systems

MCP Integrations

Claude can recognize that the event strongly matches the user's interests.

Another event may contain:

Academic Research

AI neural network Theory

Graduate Student Workshop

Even if it technically matches the keyword "AI", Claude may determine that it is less relevant for a software engineer.

The agent thinks about:

Is this event relevant?
How strongly does it align with preferences?

This reasoning step is what makes the output valuable. Rather than simply listing events, Claude also explains why each one was selected. This is another place where AI adds value beyond simple search results..

Once events have been evaluated, Claude ranks them. Rather than dumping twenty search results into a report, the agent highlights the most useful opportunities first. This makes the final report feel curated rather than generated.

For each event, Claude generates a Google Calendar URL. These URLs contain all event information such as event title, date/time, location etc. When users click the link, Google Calendar opens with a pre-filled event draft. The event can be added to your schedule in one click.

Sample Response

Claude created an events.md file after multiple SerpApi requests.

Here's the final result in events.md at the end of Claude's research:

events.md

Upcoming AI / Developer Events near San Francisco

Search focus: AI agents, MCP, RAG, and developer tooling

Filters applied: within ~100 miles of San Francisco · budget under $500 · next 90 days (Jul 14 – Oct 12, 2026) · in-person preferred

Source: SerpApi Google Events engine (searched 2026-07-14), with ticket pricing spot-checked via web search.

Note on budget: meetups below are free or low-cost. For the multi-day conferences, ticket tiers vary — I've flagged which ones can stay under $500 (early-bird / expo tiers) and which typically exceed it. Always confirm the current price on the registration page before booking.

✅ Best matches (in-person, near SF, next 90 days, likely under $500)1. Data for AI: One AI, Every Data Model

Date: Wednesday, July 22, 2026, 5:30–8:30 PM

Location: AWS Builder Loft, 525 Market St, San Francisco, CA

Event URL: https://datastrato.ai/events/data-for-ai-one-ai-every-data-model

Why it matches: An evening builder session on constructing a single AI layer over every data model — directly relevant to RAG and the data-infrastructure side of agentic apps. Hosted at the AWS Builder Loft, hands-on developer audience, and free to attend (well under budget).

Google Calendar: https://calendar.google.com/calendar/render?action=TEMPLATE&text=Data for AI%3A One AI%2C Every Data Model&dates=20260722T173000/20260722T203000&ctz=America/Los_Angeles&location=AWS Builder Loft%2C 525 Market St%2C San Francisco%2C CA&details=https%3A%2F%2Fdatastrato.ai%2Fevents%2Fdata-for-ai-one-ai-every-data-model

19th San Jose Hardware Meetup — AI/ML in Lab Automation

Date: Tuesday, July 21, 2026, 6:00–9:00 PM

Location: BigHat Biosciences, 1900 Alameda de las Pulgas, Ste 400, San Mateo, CA

Event URL: https://somo.social/e/19th-san-jose-hardware-meetup-722

Why it matches: A free, in-person developer/ML community meetup exploring applied AI and ML pipelines (here, in lab automation). Good for the developer-tooling and applied-ML side of your interests, and a strong local-network opportunity in the Peninsula.

Google Calendar: https://calendar.google.com/calendar/render?action=TEMPLATE&text=19th San Jose Hardware Meetup%3A AI%2FML in Lab Automation&dates=20260721T180000/20260721T210000&ctz=America/Los_Angeles&location=BigHat Biosciences%2C 1900 Alameda de las Pulgas Ste 400%2C San Mateo%2C CA&details=https%3A%2F%2Fsomo.social%2Fe%2F19th-san-jose-hardware-meetup-722

API World 2026 (with AI TechWorld)

Date: Tuesday–Thursday, September 1–3, 2026

Location: Santa Clara Convention Center, 5001 Great America Pkwy, Santa Clara, CA (~45 mi from SF)

Event URL: https://apiworld.co/

Why it matches: Billed as the world's largest API conference & expo, co-located with AI TechWorld — squarely in the developer-tooling space (APIs, integrations, agent tooling). Choose the free Expo pass or the OPEN tier (early-bird ~$299–$399) to stay under $500; PRO/PREMIUM passes run higher, so avoid those to honor the budget.

Google Calendar: https://calendar.google.com/calendar/render?action=TEMPLATE&text=2026 API World %26 AI TechWorld&dates=20260901/20260904&ctz=America/Los_Angeles&location=Santa Clara Convention Center%2C 5001 Great America Pkwy%2C Santa Clara%2C CA&details=https%3A%2F%2Fapiworld.co%2F

⚠️ Strong topic match, but tickets typically exceed $500

These are excellent for AI agents / RAG / dev-tooling content and are in-person near SF within the window, but standard passes generally run above the $500 budget. Listed for awareness — check for free expo passes, community/scholarship tickets, or day passes that might fit.

GenAI Summit SF 2026 (co-located with AGI Summit — "The Rise of Agentic Intelligence")

Date: Saturday–Sunday, July 18–19, 2026

Location: Palace of Fine Arts, 3301 Lyon St, San Francisco, CA

Event URL: https://www.eventbrite.com/e/genai-summit-sf-2026-tickets-1985545163032

Why it matches: Large generative-AI gathering; the paired AGI Summit track focuses explicitly on agentic intelligence. Great for AI-agents content — but full passes typically exceed $500 (look for an early-bird code or expo/hackathon pass to fit budget).

Google Calendar: https://calendar.google.com/calendar/render?action=TEMPLATE&text=GenAI Summit SF 2026&dates=20260718/20260720&ctz=America/Los_Angeles&location=Palace of Fine Arts%2C 3301 Lyon St%2C San Francisco%2C CA&details=https%3A%2F%2Fwww.eventbrite.com%2Fe%2Fgenai-summit-sf-2026-tickets-1985545163032

The AI Conference 2026

Date: Tuesday–Thursday, September 29 – October 1, 2026

Location: Pier 48 (Sheds A & B), San Francisco, CA

Event URL: https://aiconference.com/

Why it matches: Applied-AI conference with a dedicated agentic AI track plus LLMs and infrastructure — 120+ speakers, builder-heavy. Highly on-topic; standard passes generally run ~$1,000+, so only fits budget via a discounted/early team ticket.

Google Calendar: https://calendar.google.com/calendar/render?action=TEMPLATE&text=The AI Conference 2026&dates=20260929/20261002&ctz=America/Los_Angeles&location=Pier 48%2C San Francisco%2C CA&details=https%3A%2F%2Faiconference.com%2F

Salesforce Dreamforce 2026

Date: Tuesday–Thursday, September 15–17, 2026

Location: Moscone Center, 747 Howard St, San Francisco, CA

Event URL: https://www.salesforce.com/dreamforce/

Why it matches: Heavy focus on Agentforce and enterprise AI agents. Relevant to the AI-agents theme, but the full pass (~$1,999) is well over budget — included only if a free/community or expo pass becomes available.

Google Calendar: https://calendar.google.com/calendar/render?action=TEMPLATE&text=Salesforce Dreamforce 2026&dates=20260915/20260918&ctz=America/Los_Angeles&location=Moscone Center%2C 747 Howard St%2C San Francisco%2C CA&details=https%3A%2F%2Fwww.salesforce.com%2Fdreamforce%2F

Notes & tips

Closest topic fit found:"Graph Exchange — Agentic AI Memory" (GraphRAG + AI agents, AWS Builder Loft SF) and "AI Loves Data SF: GenAI & Intelligent Agents" are near-perfect matches but fall on Nov 5 & Nov 18, 2026 — just outside the 90-day window. Worth setting a reminder to register when they open.

For free, on-topic meetups on short notice , watch the AWS Builder Loft (525 Market St) calendar and lu.ma / Meetup groups for "AI agents," "MCP," and "RAG" — these tend to be posted only 2–4 weeks ahead, so re-run this search monthly.

Google Calendar links are pre-filled with title, date, location, and the event URL. Times are set to America/Los_Angeles; multi-**day conferences are added as all-day spans.

Sources: SerpApi Google Events, API World, GenAI Summit (Eventbrite), The AI Conference.

Because Claude Code can create files directly, the output becomes a reusable artifact rather than a temporary chat response. Every week or month, you can rerun the workflow and generate an updated report.

Conclusion

In this project, we used Claude Code and SerpApi MCP to build an AI-powered Event Discovery Agent. Through a single prompt, Claude was able to discover conferences and meetups using SerpApi, evaluate each event against user-defined preferences, generate Google Calendar links, and compile the results into a clean Markdown report.

I hope you found this useful. If you have any questions, feel free to reach out at contact@serpapi.com.

Relevant Links

Google Search Parameters (2026 Guide)

Noraina Nordin — Mon, 03 Aug 2026 09:36:47 +0000

The Google Search URL parameters are important to understand whether you are maximizing the conversion rate in your ad groups and optimizing your cost per click(CPC) rates in Google Analytics for your ad campaigns, improving your SEO(Search Engine Optimization) metrics for your e-commerce business, or collecting data for your social media project. Using custom parameters for your search will affect the Search Engine Result Page (SERP) data significantly in your favor. SerpApi unlocks the power of SERP data to you with minimal effort, the fastest response times, and in the most complete form.

Register to claim free credits

In this blog post, we will explore some of the Google search parameters with their corresponding attributions in SerpApi Documentation. You can head to SerpApi Playground to test all of the information you will read below.

What are Google Search parameters?

Google Search parameters are like special codes you can add to the web address (URL) when you search on Google. They help you control and customize your search results. For example, you can use them to specify the language or location for your search.

Here's how they work: When you type a search query into Google, it creates a URL with your search terms. You can then add parameters to this URL using an ampersand (&) followed by the parameter name and value. These parameters, often called "params" for short, tell Google how you want your search results to be customized.

For instance, if you want to search for "cats" in Spanish, you could add "hl=es" to the end of the URL. This tells Google to show results in Spanish. Once you've added all the parameters you want, you have your final URL ready for tailoring your tracking template. When you open this URL in your web browser, Google will show you search results based on the parameters you've specified.

The `q` parameter

It represents the search query submitted to Google. For example, in "q=Coffee", "Coffee" is the query you're searching for. It can also be used with special commands for filtering specific file types, filtering keywords in URLs and bodies, etc. giving you more exact matches. Some of these advanced search operators include filetype, inurl, intitle, intext, inanchor, allinurl, allinanchor, allintitle, allintext. You can check out Ultimate Guide to Google Search Operators for detailed, and up-to-date information on how to use Google search operators.

Google search with q parameter

Reference in SerpApi Documentation

The 'oq' parameter of Google represents the query typed into the Google search box.

Google search with oq parameter

The `hl` parameter

It stands for a ' host language', and it determines the language used for the search. There are different versions of the 'hl' parameter you can pass to a Google Search. The two-letter version is for all languages supported by Google.

For example, "hl=en" searches in English. There are also two-letter versions followed by a hyphen and two-letter or three-digit versions representing dialects. Examples of this kind are "hl=en-gb" representing British English, or "hl=es-419" representing Latin American Spanish.

Google Search with hl parameter

Reference in SerpApi Documentation

The `gl` parameter

It represents ' geolocation', which can be used to personalize search results based on the region. It is a two-letter country code. For example, "gl=us" will serve results for the United States. It's recommended to be used with the 'location' parameter for more consistent country-specific filtering.

Google Search with gl parameter

Reference in SerpApi Documentation

The `lr` parameter

It stands for ' language restriction', it's used to limit your search results to webpages in a specific language or languages. It is a two-letter language code preceded by the word lang and an underscore. Multiple languages can be passed using a bar symbol.

For example "lr=lang_en|lang_fr" will restrict the results to websites with only English and French languages.

Google Search with lr parameter

Reference in SerpApi Documentation

The `cr` parameter

It represents ' country restriction', it refines results based on preferred region. It is a two-letter uppercase country code preceded by the word country. Multiple countries can be passed using a bar symbol.

For example, "cr=countryUS|countryFR" will restrict the results pages to the United States and France.

Google Search with lr parameter

Reference in SerpApi Documentation

The `tbs` parameter

It stands for ' to be searched', it can be used for various purposes (it's optional), for example, to filter results by a specific date or a date range (e.g., past week, past month).

For example, "tbs=qdr:h" will serve only the results from the last hour.

Google Search with tbs parameter

Reference in SerpApi Documentation

The `source` parameter

It indicates the source of your search , such as the Google homepage or a referral from another website.

For example, "source=hp" means that the page was accessed through the homepage.

Google Search with search parameter

The `ei` parameter

It possibly stands for engine identifier , a unique identifier assigned to each search, likely used for internal tracking by Google. Its decoded form consists of four numbers. The first two are a Unix timestamp and microseconds of that timestamp respectively. The last two are undisclosed and unknown.

Google Search with ei parameter

The `ved` parameter

It is used for internal tracking of links and elements. Apart from the link itself, the element index and type can be gathered by decoding the ved value.

Google Search with ved parameter

Open Source Repository for Decoding the ved values

The `gs_lp` parameter

Not much is disclosed about this parameter. It contains the search query and search client.

Google Search with gs_lp parameter

The `iflsig` parameter

This parameter isn't disclosed. It is speculated to be about internal redirects. Other assumptions include relations to parsing.

Google Search with iflsig parameter

The `ibp` parameter

It is responsible for rendering elements differently than their usual format. For example, when used with the ludocid parameter (unique id for a Google Business listing) in a mobile view, "ibp=gwp;0,7&ludocid=1663467585083384531", the 'ibp' parameter can give full page views of the listing.

Google Search with ibp parameter

Reference in SerpApi Documentation

The `kgmid` parameter

The parameter is responsible for serving elements saved by Google including knowledge graph. It can override the 'q' (query) parameter.

For example "kgmid=/m/07gyp7" will get results for McDonald's with a knowledge graph regardless of the query parameter. The parameter is likely acting as a cached version of the Google advanced search parameters.

Google Search with kgmid parameter

Reference in SerpApi Documentation

We write a detailed tutorial on how to find the Google Knowledge Graph ID (kgmid) for anything in the post below:

How to find the Google Knowledge Graph ID (kgmid) for anything

The `si` parameter

It is used to serve knowledge graph tabs alongside other tasks (it's an optional parameter).

For example, in the example below, the 'si' parameter is responsible for serving the Google search results with the 'Music Groups' tab of the knowledge graph for Jimi Hendrix.

Google Search with si parameter

Reference in SerpApi Documentation

💡 For some searches, 'si' may override all other parameters except 'start', and 'num' parameters.

The `sxsrf` parameter

It is possibly an internal token for security. The 'sxsrf' consists of an encoded token probably for security purposes, and a unix timestamp.

Google Search with sxsrf parameter

The `uule` parameter

It is used to specify a location for a search. The 'uule' parameter has two types. The first type is an encoded form of the canonical name of a place in Google Ads API Geo Targets. The second type is an encoded form of the GPS coordinates alongside other data.

Google Search with uule parameter

Reference in SerpApi Documentation

We write more details about the different types of UULE in the post below:

Real World Example of GPS Targeted Local SERP

❗ The 'uule' and location parameters can't be used together.

The `sclient` parameter

It stands for search client. For example "sclient=gws-wiz" means Google Web Search Wizard is used for the search.

Google Search with sclient parameter

The `tbm` parameter

It stands for " to be matched". This parameter is optional and it specifies the type of content that the user is searching for, allowing Google to tailor the search results accordingly.

For instance, "tbm=isch" indicates that the search is intended for image search, while "tbm=nws" directs Google to prioritize results from Google News. By using this parameter, users can fine-tune their searches to precisely start finding pages, sites, or specific content types using Google's sophisticated algorithm.

Additionally, users can specify whether they want to search within meta information, anchor text, or other elements through the appropriate parameters. This level of customization empowers users to efficiently navigate the vast expanse of information accessible through Google's search bar.

Google Search with tbm parameter

Reference in SerpApi Documentation

Use the `safe` parameter to filter explicit results

If your application is designed for general audiences, the safe parameter helps ensure family-friendly results.

Example:

safe=active — filters out explicit content.
safe=off — shows all results without filtering.

This is particularly helpful for educational or public-facing dashboards using SerpApi.

Google Search with safe parameter

Optimize for freshness with the `filter` and `tbs` combo

While tbs=qdr:h limits results to the last hour, pairing it with filter=0 ensures that similar pages are not clustered together—revealing the most recent, diverse results.

Example:

Google Search with filter and tbs combination

This combination is perfect for trend monitoring or social sentiment analysis.

Conclusion

I would like to thank the reader for their attention. I hope this writing could shed some light on how Google search parameters work. For more up-to-date information about web scraping, SERP data, and more related topics, you may visit the SerpApi Blog.

DataForSEO vs. SerpApi

Noraina Nordin — Sun, 02 Aug 2026 09:30:23 +0000

If you're comparing SEO platforms to choose where your search data comes from, whether it's for rank tracking, dashboards, or a Google search ranking API integrated into your product, both DataForSEO and SerpApi are strong options. At a quick glance, they seem similar, but they have important differences.

Live Search Data vs SEO Platform

Despite the overlap on Google SERP data, SerpApi and DataForSEO are built around two different priorities. Which option fits best depends more on your priorities than on any single line item. Price is part of it, and we'll be straight about it further down.

DataForSEO is a broad, pay-as-you-go SEO data platform. Beyond SERPs, it provides backlinks, keywords, on-page, domain analytics, and more, all billed per request from a prepaid balance. Its strongest selling point is low-cost data at scale. It's the raw layer that many SEO dashboards are built on.

SerpApi is a real-time web search API. You make a call, and it returns structured JSON of what a search engine shows for that query. The results include AI overviews, organic results, ads, featured snippets, People Also Ask, the local pack, Shopping, and more than 100 SERP feature types. Its best features are speed, coverage, and developer experience.

DataForSEO is optimized to be the most cost-effective way to store search data. SerpApi is optimized to be the fastest, most complete way to query it in real time. Picking between them is really about which of those two jobs is yours.

SerpApi vs DataForSEO illustration

Data Freshness

The most consequential difference between SerpApi and DataForSEO is how the results reach you.

Search Result Timeline Diagram

DataForSEO's default model is a queue. You submit a task, wait, and retrieve the result later. On the Standard Queue, turnaround is typically around 5 minutes (a Priority Queue cuts it to about 1 minute, and a synchronous Live Mode runs at a higher rate). For batch rank tracking that runs overnight, where five minutes is insignificant, the queue is perfectly suited.

But if your application is customer-facing or agent-driven, where a user types a query and waits, or an LLM that needs search results mid-response, five minutes is a non-starter, and even one minute is. For those workloads, the queue isn't an option. You're choosing between DataForSEO's Live Mode and SerpApi. And "live" doesn't mean equally fast: in our own reproducible benchmark, SerpApi averaged 0.73 seconds per query, compared to 3.57 seconds for DataForSEO's fastest option, and the worst-case gap was wider still, with a p99 of 1.45 seconds versus 13.31 seconds. DataForSEO's own comparison points in the same direction, measuring its live mode at around 6 seconds. For an AI agent making several search calls in a single chain, that difference compounds quickly.

SerpApi is synchronous by default. Every call returns live results in roughly a second, and with Ludicrous Speed / Ludicrous Speed Max you can push median response times well below that. We benchmarked response times across providers in who has the fastest Google Search API. There’s no queue to manage, no task-polling loop to construct. The optional Search Archive API is available if you want to re-fetch past searches; it's for convenience, not a required step. For real-time workloads, fast processing determines whether you can deliver the feature or not.

Do you actually need real-time data?

It's worth meeting the strongest version of the other side head-on because DataForSEO-leaning comparisons have highlighted that most SEO work is batch-based, so you're paying a premium for speed you'll never use. For pure scheduled SEO, such as overnight rank tracking or weekly content-gap reports, that's fair. If nothing is waiting on the result, the queue is the right call.

The argument breaks down on two points. First, the number of applications requiring real-time results is growing rapidly. Gartner estimates that 40% of enterprise applications will have task-specific AI agents integrated by the end of 2026, up from under 5% in 2025. Agents, RAG pipelines, in-product search, and live AI-Overview / GEO monitoring all need a result in a single request. The moment a human or an agent is waiting, the comparison becomes Live Mode at ~ $2.00/1K versus SerpApi. That's a much smaller gap than the headline numbers suggest, and part of it buys back real engineering time. With a synchronous API, there's no poll-and-retrieve layer to build, debug, and maintain.

Second, when DataForSEO-leaning comparisons concede that the queue can't serve real-time use cases, the workaround they tend to suggest is to pair it with Google's Custom Search JSON API for live queries. That pairing doesn't hold up. The Custom Search API is capped at low daily volume, returns a restricted custom-search index rather than the live public SERP, and omits most SERP features and verticals. Offering it as the real-time answer mostly confirms that it's the queue being defended, not your latency needs being met.

The real question is simpler than "batch or real-time": does anything wait on the result? If so, compare SerpApi against Live Mode, and count the poll-and-retrieve layer you won't have to build.

Let's take a look at two scenarios that show how this plays out.

For the SaaS team, the queue is fine for weekly rank reports that can be processed in batches. But AI Overview citations aren't. They appear, vanish, and swap sources far faster than bluelinks move, and being cited for the "best <your category> software" is the new position #1. Monitoring on day-old snapshots means learning you lost the citation days after a competitor's page replaced yours. Checking live means alerting the content team while the cause is still identifiable and answerable.

For an e-commerce team , the gap is larger because automation acts directly on the data. Google Shopping positions are reshuffled in response to competitor price changes, sometimes hourly. So a repricing engine or Minimum Advertised Price (MAP) violation monitor running on 5-minute-old queue data can act on prices that no longer exist. A human reading a stale report adds judgment before acting. An automated repricer doesn't get that chance.

DataForSEO pricing vs. SerpApi pricing

If you're sizing up SerpApi as a DataForSEO alternative, the prices only make sense once you know which speed tier you'd actually use.

DataForSEO is pure pay-as-you-go. No monthly subscription. You prepay a balance (a $50 minimum deposit, with a $1 free trial credit to test first) and draw it down per request. For the SERP API, the same query costs a different amount depending on how fast you need it:

DataForSEO SERP API tier	Price per 1,000 SERPs	Delivery
Standard Queue	~ $0.60	Queued, ~ 5 min turnaround
Priority Queue	~ $1.20	Queued, ~ 1 min turnaround
Live Mode	~ $2.00	Real-time, synchronous

(Base price covers the first 10 results. Since Google removed its num=100 parameter in late 2025, deeper result pages are billed per additional page budget for that if you paginate.)

SerpApi is subscription-based , priced on a monthly bucket of successful searches:

SerpApi Plan	Price	Searches/Month	Price per 1,000 Searches
Free	$0	250	-
Starter	$25	1,000	$25.00
Developer	$75	5,000	$15.00
Production	$150	15,000	$10.00
Big Data	$275	30,000	~ $9.17

Plans scale up to 1,000,000+ searches/month, and an Enterprise tier offers reserved and on-demand pricing for high volume. Only fully successful searches count. While cached, errored, blocked, and CAPTCHA requests are free, unused searches don't roll over.

Look at those two tables side by side, and the takeaway is simple: at the same volume, DataForSEO's price is lower, and it is the better fit for latency-tolerant batch work. SerpApi is the premium option. That's what you get for the real-time speed, one-call integration, wider search-surface coverage, human support, and legal cover , which is what the rest of this comparison walks through.

Ease of Integration

This one rarely makes the spec sheet but shows up on day one of building.

SerpApi

With one API key, you get synchronous request and structured JSON back. Plus, official libraries are available for Python, Ruby, JavaScript, Go, PHP, Java, .NET, Rust, and more, plus an MCP integration for AI assistants.

For quick testing, the playground lets you run a search query with zero code, and one thing I really like about it is that it shows a visual preview of the SERP on the same page you'd see in a regular browser, side by side with the JSON response. When I'm tracing where a field comes from ("which box on the page is this organic_results object?"), I can see the element and its parsed data side by side. DataForSEO's output is JSON only, so mapping a field back to what actually appeared on the page requires opening a browser and reconstructing the SERP yourself. It sounds small. But it stops being small the first time you debug why a value isn't where you expected it to be.

SerpApi Google Search Playground

DataForSEO

The authentication uses a login-plus-API-password credential pair, and the queued endpoints follow a task-based pattern. You POST a task, then either poll for results or wait for a callback to retrieve them. It's well-documented and fine once built, but it really involves more work than a single synchronous GET. (DataForSEO also ships an MCP server, so for AI-assistant workflows, both options exist.)

For solo developers or small teams trying to ship faster, integration friction creates real costs, even if it doesn't appear on the invoice.

Coverage

"Which one covers more?" depends on more of what.

On SERP engines and Google verticals, SerpApi is wider. DataForSEO's SERP API spans roughly 8 engines (Google, Bing, Yahoo, Baidu, Yandex, Naver, Seznam, YouTube). SerpApi supports 100+ APIs across Google Search, Maps, Shopping, Scholar, Trends, Flights, Hotels, Lens, Jobs, News, Patents, and more, as well as Bing, Baidu, Yandex, Naver, DuckDuckGo, Amazon, eBay, Walmart, YouTube, Yelp, Tripadvisor, and others. If your roadmap touches multiple search surfaces, that range matters.

In terms of SEO data types, DataForSEO offers a broader range. Backlinks, a historical keyword database, on-page audits, domain analytics, and its API list go well beyond SERPs. SerpApi doesn't support these features because its main focus is on the web search API rather than an SEO data suite. If you need backlink graphs or a keyword-volume database alongside SERPs, DataForSEO consolidates that in one account.

In short, SerpApi provides the depth and freshness of live search results parsing across many search engines, while DataForSEO provides a wider range of raw SEO datasets in one place.

SerpApi Endpoints That Affect SEO Decisions

Several endpoints return the exact signals that SEO teams call on. A few worth knowing, covered by both SerpApi and DataForSEO:

Pixel Position: the real-browser X/Y coordinates of each result. Nominal "rank #1" can sit well below the fold once ads, an AI Overview, and a featured snippet stack above it. Pixel Position reports actual above-the-fold visibility, per device, so you can tell which rankings are worth defending and which are wins only on paper.
Google AI Overview and AI Mode: whether your content is cited in Google's AI answers, and which domains are. This is the core AEO/GEO decision of 2026: track AI visibility the way you track bluelink rank. We've written about what AI Overviews change for rank tracking in depth.
Rank Tracking in the Age of AI Overviews: What’s Changed
Related Questions (People Also Ask): the questions Google attaches to a query, which map straight to content gaps, FAQ blocks, and subheadings worth writing.
Autocomplete: the long-tail variations real users actually type, for keyword discovery grounded in live suggestions rather than a static database.
Google Trends: demand direction and seasonality, so you prioritize topics that are rising rather than fading.
Google Local: local-pack results by location, for tracking visibility, where the map results, not the organic ones, win the click.

Some endpoints currently lack DataForSEO equivalents, such as the Google Forums API (covering the "Discussions and forums" SERP block where Reddit visibility appears), Google Short Videos, and Bing's vertical results beyond organic Copilot, Images, Maps, News, and Shopping, where DataForSEO only covers organic Bing results.

The key difference is that SerpApi returns the results in a single, real-time call, parsing them in a structured JSON format. This introduces the same real-time versus queued trade-off mentioned earlier, but specifically applied to the signals that inform SEO decisions.

Feature Comparison

Here's the full picture in one place with everything we covered so far, plus a few dimensions that haven't come up yet:

Dimension	SerpApi	DataForSEO
Core model	Real-time search API (subscription)	SEO data platform (pay-as-you-go)
Default delivery	Synchronous, ~1 sec	Queued (Standard/Priority); Live mode synchronous
Pricing	From $25/mo (1K searches); free tier 250/mo	From $0.60/1K (queued) to ~$2.00/1K (live); $50 min deposit
SERP engines	100+ APIs incl. many Google verticals	~8 SERP engines
Other SEO data	SERP-focused (no backlinks/keyword DB)	Backlinks, keyword DB, on-page, domain analytics
SERP feature parsing	100+ feature types as structured objects	Rich features via "Advanced" result type
Integration	Single API key, sync request	Login + API password, task-based queue workflow
Legal	U.S. Legal Shield assumes scraping liability for you	No Legal Shield
Transparency	Public status page + public roadmap	Status page; private roadmap
Best fit	Real-time apps, AI agents, multi-engine SEO tools	High-volume batch data, cost-sensitive pipelines, full SEO suites

Main differentiators

Legal posture. These aren't the same promise. DataForSEO states that scraping publicly available SERP data isn't a CFAA/DMCA violation. i.e., we believe this is legal. SerpApi's U.S. Legal Shield goes further: on recurring paid plans, SerpApi contractually assumes liability for scraping and parsing on your behalf (short of genuinely illegal use). If legal risk transfer is something your company needs in writing, that's a SerpApi-specific feature, not a line item in the price.

Transparency. SerpApi publishes a public status page with 30-day average response times filterable by engine, and a public roadmap where anyone can see bugs and file feature requests consistent with the company's "painful honesty" value. For the infrastructure you're betting a product on, being able to inspect reliability before you commit is worth weighing.

When to choose each

Choose DataForSEO when:

Unit cost is your binding constraint, and you're operating at high volume.
Your workload is batch and latency-tolerant, such as overnight rank tracking and bulk SERP snapshots.
Your usage is spiky, and you'd rather pay only for what you consume than fit into a monthly bucket.
You need backlinks, a keywords database, or on-page data in the same account as your SERPs.

Choose SerpApi when:

Your app is customer-facing, or agent-driven, and results need to be real-time.
You want the simplest possible integration: one key, one synchronous call.
You need a broad engine and Google-vertical coverage with high-fidelity SERP-feature parsing.
Legal liability transfer, public transparency, and responsive support are part of the decision.
You'd rather not build and maintain a task-queue retrieval layer.

FAQ

Is DataForSEO actually cheaper than SerpApi?

On headline per-query price, yes, especially on the Standard Queue (~$0.60/1K vs. SerpApi's $25/1K entry rate). The gap narrows when you need real-time results, since you'd use DataForSEO's Live Mode (~$2.00/1K). And it narrows further once you price in integration time, latency requirements, and whether you need legal cover. Cheapest-per-query isn't the same as cheapest-for-your-project.

Can DataForSEO do real-time?

Yes, via Live Mode, which is synchronous but priced higher than its queued tiers. SerpApi is real-time by default across all plans.

Which has more search engines?

SerpApi, by a wide margin on SERP such as Google, Bing, DuckDuckGo, Baidu, Yandex, Naver, Yahoo, YouTube, Yelp, Tripadvisor, Amazon, eBay, Walmart and more (100+ APIs). DataForSEO covers more SEO data types (backlinks, keyword DB, on-page) but fewer SERP engines.

Can I use both?

Plenty of teams do. DataForSEO for cheap, high-volume batch collection, SerpApi for the real-time and multi-engine paths. They're not mutually exclusive.

Which is the best SERP tracking tool, SerpApi or DataForSEO?

Neither is a turnkey rank tracker. Both are API layers you'd build a SERP-tracking tool on top of. For tracking that runs on a schedule, DataForSEO's queue is cost-efficient, but for tracking that updates on demand or inside a user-facing dashboard, SerpApi's real-time responses are the better fit. Pick the data layer that matches how your tool refreshes.

Is SerpApi a good DataForSEO alternative?

If your use case is real-time or spans multiple search engines, yes — SerpApi covers the SERP side with lower latency and simpler integration. But if you also rely on backlinks, a keyword database, or on-page audits, SerpApi won't replace those parts of DataForSEO. In that case, the answer to "what's the best DataForSEO alternative" is often SerpApi plus a dedicated tool for the rest, rather than a single swap.

Related comparisons

Weighing SerpApi against other providers? We've broken down the trade-offs in each of these:

SerpApi vs. Brave Search API
SerpApi vs. Exa AI
SerpApi vs. Firecrawl: search intelligence vs. website content
SerpApi vs. Serper vs. ValueSERP vs. SearchApi
Who has the fastest Google Search API?

The bottom line

DataForSEO and SerpApi appear to be competitors because both provide Google SERP data. But you're choosing between a pay-as-you-go data warehouse and a real-time search API, and the right answer falls out of one question: is your bottleneck unit cost, or latency, coverage, integration speed, or legal cover?

If you're warehousing search data at scale on a tight budget and can tolerate a queue, DataForSEO's prices are definitely lower. If you're building something where results need to arrive now, such as an AI agent, a customer-facing product, a multi-engine SEO tool, and you'd rather ship than build retrieval plumbing, that's exactly what SerpApi is for.

The good news: you don't have to take anyone's word for the trade-offs. SerpApi's status page and roadmap are public, and the free plan includes 250 searches per month to test real responses against your own queries.

Try a query in the playground and compare the JSON for yourself! If you have questions, reach us at contact@serpapi.com.

Why is Ruby on Rails a great fit for AI agents

Josef Strzibny — Sat, 01 Aug 2026 09:24:07 +0000

What makes a programming framework good for autonomous AI agents? It's how well the language and APIs communicates intent within the tight constraints that LLMs operate under. Agents need to read code, understand it, and generate more of it, all within a finite context window.

That means three things matter most: token efficiency (how much meaning fits in how few tokens), predictability (how consistently code is structured so the model doesn't have to guess), and ecosystem ergonomics (how pleasant and productive the tooling is for iterative agent workflows).

Ruby on Rails happens to excel at all three.

Ruby on Rails logo

What are AI agents?

Traditional AI usage like running a single ChatGPT prompt is mostly linear and reactive. You ask a question and receive an answer. An autonomous agent is dynamic by contrast. It can break a task into steps, follow them one by one, recover from errors, and adjust their behavior based on new information.

Agents are processes with access to tools and memory to orchestrate complex work. Tools let agents do things beyond the usual text APIs like running code or searching the web. Memory allows agensts to keep track of context across individual steps or even across sessions. Control logic defines how they decide what to do next.

Token efficiency for both humans and robots

When it comes to AI efficiency, most will bring up tokens. But what are tokens anyway? When it comes to large language models (LLMs), tokens are the basic units of text that model either reads or generates. They are processable text chunks that don't always map to words but which these models understand.

A token can be a whole word, part of a word, punctuation, or special characters. A word unbelievable might be split into un, believ, and able. Different models rely on different tokenization rules, so these chunks are not always the same. The process of turning raw text into these chunks is called tokenization.

Context windows

LLMs currently cannot process unlimited amount of text. The amount of data models can process is limited by computer memory which significantly increases with bigger context length. We call the maximum length a model can work with at a given time a context window and we measure it in tokens.

This memory constrain includes the entire prompt, system instructions, conversation history, and the model's response. Less tokens as input enables longer answers and vice versa. When it comes to programming, the usage compounds between source files being read, changes being applied, and diffs being reviewed.

Token windows significantly increased from the GPT-3.5 era of 4096 tokens and are now approching numbers higher than 1 million with models like Opus 4.6 and Sonnet 4.6. But even today the efficiency factor is significant when it comes to entire codebases. We are still spending energy, prolonging response times, and increasing cost.

Ruby token efficiency

Different programming languages means different token efficiency and context windows. This is where Ruby stands out. Martin Alderson, a co-founder of CatchMetrics, made a bit of a fuss lately with a post titled Which programming languages are most token-efficient?. You might not know Martin, but his writing was previously praised by OpenAI co-founder Andrej Karpathy and his posts put him to top 100 most highest-ranking personal blogs on Hacker News for 2025.

In his post, Alderson looks at tasks from the RosettaCode project that have implementation across all 19 most popular mainstream languages. He then ran them through the GPT-4 tokenizer to get the number of tokens required for the solution. The results showed up to 2.6x gap between the most efficient and least efficient languages from the list. That’s pretty significant difference as it's the result between one to almost three files in the context window. Have a look yourself:

Martin Alderson's graph of most token-efficient languages

The updated post also features a mention of the J language which could get efficiency around the half of Closure which dominates the graph. Looks like array languages can be extremely token-efficient. Ruby isn't first on this list, but it is number one when only the major languages are considered.

As the author of Rails, David Heinemeier Hansson (DHH), has said many times "Rails wouldn't exist without Ruby." Ruby was always putting the human experience of writing and reading above other metrics. No, not just the reading part like many others. Writing too. And it's interesting to realize that coming close to English language isn't just helpful for people, but also just as effective for AI.

Language models got increasingly better in working with larger context windows over time, but the numbers of exchanged tokens still play a vital role in their practicality. More tokens means smaller available context, slower responses, and higher cost to run. As DHH puts it:

The answer is (of course) agentic workflows. If you've used coding agents for any length of time, you'll quickly reach a point where you hit the dreaded 'compaction' stage. Compaction is the process most agents use when they reach the limit of the context window. It condenses earlier parts of the conversation - preserving recent context and key artifacts but losing a lot of the detail from earlier in the session.

Rails token efficiency

Of course, Rails is more than Ruby, and lots of people will point out that the framework is large with lots of project generated files and other ceremony. There are not wrong, but is that a problem? Well, a software engineer and researcher Felipe Maciel Ramos Vieira tried to answer this question in his comparative view called SyntaxTax that prioritizes the most context-efficient web stacks

SyntaxTax report of popular web frameworks

Felipe's report implement the same API features in different stacks and compare how many tokens the handwritten code consumes. The Rails framework won at lowest handwritten costs and it was second as lowest total cost. When we talk about big opinionated frameworks it was also first in total efficiency #1

primary handwritten efficiency.

While neither report is detailed and comprehensive enough to be considered scientific there are definitely indicators that being compact, terse, and expressive might as well mean token efficient.

Less ambiguity for the model

Ruby's brevity is a good start but can mean very little without the generated tokens being correct. Modern LLMs can write valid code in almost any language. Agents don't fail at syntax, but they may struggle with ambiguity. They have harder time with architectural decisions. They must ask questions like Where should this logic go?, What pattern should I follow?, and How does this connect to the rest of the system?

Very flexible frameworks leave these questions wide open, and every unanswered question is a chance for the agent to do guess work. Not in Rails. These questions tend to be already answered at the start because of the Rails' Convention over Configuration approach to building applications. An approach that had already lifted a lot of weight from people teams finds its new value in agentic workflows.

Convention over Configuration

Rails conventions act as implicit instructions. If there's a resource :posts route, there's a PostsController controller with CRUD action. If there's a PostsController, there is likely a Post model with user business logic and persistance conveniently provided by Active Record. If there's a has_many relationship, there is a foreign key of a certain name.

Naming things, structure of files, configuration (or lack of). These are building stones that an agent can rely on. Just a simple thing as agreeing on singular vs plural naming saves a lot of overhead. It also makes things consistent. Consistency was good for humans and is equally helpful now with robots.

Configuration

Convention over Configuration means less need for writing configuration files, although default configuration files exist in Rails. And funnily enough, they also follow Convention over Configuration by specifying predefined environment like development or production, and standardizing how Rails components or gems accept configuration.

Rails projects start with many files by default so it seems as if Rails cannot compete with micro frameworks on that minimalism front. However, that's deliberate choice with its own benefits and if we only really cared for the size, we can build a single file application as Greg Molnar shows us:

# frozen_string_literal: true
require 'bundler/inline'

gemfile(true) do
  source 'https://rubygems.org'

  gem 'rails', '~> 7.0.4'
  gem 'sqlite3'
end

require 'rails/all'
database = 'development.sqlite3'

ENV['DATABASE_URL'] = "sqlite3:#{database}"
ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: database)
ActiveRecord::Base.logger = Logger.new(STDOUT)
ActiveRecord::Schema.define do
  create_table :posts, force: true do |t|
  end

  create_table :comments, force: true do |t|
    t.integer :post_id
  end
end

class App < Rails::Application
  config.root = __dir__
  config.consider_all_requests_local = true
  config.secret_key_base = 'i_am_a_secret'
  config.active_storage.service_configurations = { 'local' => { 'service' => 'Disk', 'root' => './storage' } }

  routes.append do
    root to: 'welcome#index'
  end
end

class WelcomeController < ActionController::Base
  def index
    render inline: 'Hi!'
  end
end

App.initialize!

run App

But this kind of minimalism is only helpful in micro examples. Typical applications will inevitably grow, so this is less of a concern. Generating more files at the beginning shows where we want things to live. It complements all of the other important design choices in Rails.

Effortlessly RESTful

A big part of any web application, be it something user facing or just an API, is its RESTful interface, coming from Representational State Transfer (REST) coined by Roy Fielding in 2000. There is lot of plumbing in representing state and data, and it's something that's practically solved since the Rails early days.

When it comes to implementing Rails applications, you don't need to prompt "Create a RESTful controller with standard CRUD actions, following the naming convention…" — you just say "Add posts to this app", and the agent can fill in the rest correctly, because the conventions tell it everything it needs to know. These conventions fundamentally changes how agents behave. Instead of guessing, they can infer. Instead of generating plausible-but-wrong code, they land on the idiomatic solution from the start.

Less decision-making means better performance. Agents are not creative problem solvers in the human sense; they are pattern completion systems. Every extra decision you force them to make increases the chance of incorrect structure, inconsistent naming, or broken integration. Rails removes entire categories of these decisions this way. The result is more reliable code generation, faster iteration loops, and fewer corrections by humans.

Baked-in features

Rails packs a lot of functionality out of the box. This is by design as the framework has always believed that Convention over Configuration isn't just about file structure, it's about shipping defaults that are genuinely production-ready. Rails is extracted, not designed. There is no committee adding imaginary nice-to-have features. New features are extracted from real world applications.

As an example, I was thinking about using Django early in my career and followed its releases over the years. Perhaps it's not entirely fair given the smaller team and different commercial attention, but Django always missed things Rails already had. Many things were added over time to Django as well, but Rails continues to have opinionated answers to almost anything. Django still largely leaves these as "pick your own adventure."

Rails is a framework of frameworks.

When you create a Rails application, you're not starting with a router and a blank slate. You're getting Active Record, Action Mailer, Active Job, Action Cable, Active Storage, and more. All integrated, all following the same conventions, all tested together, and on top supported by Hotwire and Solid libraries, which expand on these core frameworks even further.

Some of these main frameworks are:

Active Record is the database layer. Models map to tables, associations are declared with one line (has_many, belongs_to), and queries are chainable Ruby instead of raw SQL. Migrations version your schema alongside your code.
Active Model handles model's validations, serialization, type casting, dirty tracking, and more even without full database-backed Active Record so you can use it in your own plain Ruby objects modeling your domain layer.
Action View is the templating layer. ERB templates, partials, layouts, helpers, and form builders. Form helpers handle CSRF tokens, URL generation, and type-appropriate inputs automatically.
Action Controller & Action Dispatch are the HTTP layer. Controllers handle requests, dispatch routes them. You declare routes in one file (routes.rb) and they map to controller actions. Action Dispatch also handles middleware and request/response objects.
Action Mailer brings email as controllers. Each mailer is a class with methods that correspond to email templates. You call UserMailer.welcome(user).deliver_later and get templated HTML/text email with the same layout system as your views.
Active Job is the common interface for background jobs. You define a job class with a perform method, call MyJob.perform_later(args), and Rails queues it. It works with many adapters from Sidekiq, to Solid Queue.
Action Cable is the Rails WebSockets integration. Channels work like controllers for persistent connections. You subscribe to channels and broadcast data from anywhere in your application.
Action Mailbox routes inbound emails to mailbox classes, similar to how Action Controller routes HTTP requests to controllers. Handles ingestion from Postmark, Sendgrid, Mailgun, or raw SMTP.
Active Storage handles upload, storage, URL generation, transformations, and variants of your model's file attachments. It comes with swappable backends and supports direct uploads to S3/GCS/Azure in production.
Action Text brings rich text content editing to your application backed by the editor of your choice (defaults to Lexxy). It's an abstraction for your rich text fields that integrates with Active Storage for embedded attachments. This tight integration is unique to Rails.

All and more is available to your agents in a single package. Take just the storage for example. It's not just that you can attach a file to your model, you can immediately use a form helper for that, make it a direct upload, create more efficient variants, use it later in a background job or WYSIWYG editor.

This kind of internal standardization can make complex features feel easy during agentic programming as it's the experience of our Engineering Director Illia Zub when it came to using Rails Engines:

I asked @GitHubCopilot to research the options and propose a plan. Then I kept reviewing it and writing, "Nah, I feel this can be simpler and more consistent." After several rounds of feedback to the plan, it landed on a Rails engine (...) The result is a whole new feature in only a handful of files. It reuses our existing controller directly instead of making a separate API call, which is super nice.

And that's not all.

Solid Trifecta

Another example of this philosophy is the so-called Solid Trifecta stack: Solid Cache , Solid Queue , and Solid Cable. These three libraries let you replace Redis and other external services with your primary database, either with MySQL, PostgreSQL or SQLite.

Solid Trifecta is designed to reduce complexity. Even if they are not technically part of the Rails code directly, Rails ships with all three as defaults in new applications. Most frameworks would tell you to figure out your own message queue and caching layer, but Rails has one.

They all plug into to a higher level APIs like Active Job which still lets you replace them with alternatives at any moment while keeping the same API interface. The new directions around SQLite is especially interesting given how much simplification you can do to your architecture.

Testing

Even the default test suite reflects this philosophy. Minitest , the Rails' default testing framework, supports parallel test execution leveraging multiple cores with a single configuration line. You don't need to reach for a separate gem or configure a CI matrix differently. It's there, it works, and it's fast. The assumption is that your tests should run quickly on the hardware you already have.

On the other hand, lots of applications (including SerpApi) still choose RSpec , mainly for its DSL syntax focused on specifying behavior. Since it's a more popular option, it's also a great alternative for agentic coding. Rails usually gives you something out of the box, but still makes choosing an alternative easy.

The alternatives in Rails are usually limited to a few of options, like here with RSpec being the only common alternative to Minitest, but that's a good thing. Coding agents do better with things that are well used and understood.

Training data

The Rails predictable nature and long list of features are great, but it's the real world use that makes the training possible. Rails is what I call the OG of all modern web frameworks; one that brought MVC and Active Record patterns to the masses, and inspired a whole generation of frameworks since that. Laravel author Taylor Otwell mentioned several times how he got inspired by Rails, namely by Active Record, to create Eloquent ORM.

This kind of history and pedigree helped Rails accumulate a list of famous open source projects with well-structured codebases. Some of the projects you might heard about include Spree (eCommerce), Redmine (project management), GitLab (DevOps), Diaspora (social network), Discourse (forums), Maybe (finance), and Chatwoot (customer support).

But it's not just the grand past. Recently, 37signals specifically made many of its applications source available. Campfire, Writebook, and Fizzy sources are now available for LLM training as well. The Rails foundation even started to publish this kind of applications as a small directory directly on the web as Reference Apps.

The new Rails apps showcase

The Rails documentation, namely the part called Rails Guides , also stands out as a source on how to write Rails code. Since Rails applications tend to look similar, models get better generalizing patterns they can follow for writing new code or refactoring. Rails Guides are also coming through a bit of an update now, with completely new guides being written like the one on Accessibility made by a blind programmer Bruno Prieto.

Ruby also had an agent-friendly documentation long before we heard about AI agents. Apart from RDoc, the usual documentation generator, Ruby has a tool called RI which is a command line interface documentation:

ri Hash             # Document for class Hash.
ri Array#sort       # Document for instance method sort in class Array.
ri read             # Documents for methods ::read and #read in all classes and modules.
ri ruby:dig_methods # Document for page dig_methods.

By passing the --no-pager option, we get results printed directly to the terminal.

Today's agents are powerful enough to fetch even HTML documentation, but it's nice to have this option, especially for completely local development.

World-class libraries

I think that originally lots of people thought that with all LLM training happening in other languages, Ruby could felt behind. That's no longer true. The thing is that Ruby doesn't need to have the best ecosystem for LLM training. Very few companies can afford to train a model from scratch anyways.

On the other hand, every developer wants to take advantage of calling LLM, embedding data, fine tune models, and run their own agents. Using agents to build AI features might look like a scene from Inception, but it's actually our reality. And that's where both Ruby and Rails have a lot to offer today.

The RubyLLM revolution

The things started to look better with an unofficial ruby-openai gem for integrating ChatGPT, today also with an option to use the official openai-ruby library from OpenAI. But a small revolution started with RubyLLM which brought the most beautiful API for Ruby chat interfaces:

chat = RubyLLM.chat model: "gpt-4o-mini"
response = chat.ask "What's the day today?"
response = response.content

A simple chat is just the beginning. RubyLLM's elegance really shows as you reach for more advanced features. Need to analyze an image, a PDF, or an audio file? It's the same interface:

chat.ask "What's in this image?", with: "ruby_conf.jpg"
chat.ask "Summarize this document", with: "contract.pdf"
chat.ask "Describe this meeting", with: "meeting.wav"

Want to stream a response in real time? Pass a block:

chat.ask "Tell me a story about Ruby" do |chunk|
  print chunk.content
end

RubyLLM doesn't stop with an interface to text models. Even image generation, transcribing audio, and embeddings follow the same pattern of radical simplicity:

# Generate images
RubyLLM.paint "a sunset over mountains in watercolor style"

# Transcribe audio to text
RubyLLM.transcribe "meeting.wav"

# Create embedding
RubyLLM.embed "Ruby is elegant and expressive"

These calls will use image, speech-to-text, and embedding models instead, but the DSL is similar.

The problem with simple model calls is that they are not complete systems. They understand intent well, but are not that good at executing arbitrary tasks. That's why early tools like LangChain came with a way to abstract tools which is something that's already offered by all leading LLM companies.

Tools enable models to interact with external tools by outputting JSON objects that specify functions and their arguments. This allows models to retrieve live data, something they otherwise lack as they are trained on existing data.

Imagine you need for a model to help you with Search Engine Optimization (SEO). You can now write a tool to get live Google search results using SerpApi and provide it to your chat context:

class SearchGoogle < RubyLLM::Tool
  description "Search Google and return top organic results for a query"

  params do
    string :query, description: "The search query"
    integer :num, description: "Number of results to return (max 100, default 10)"
  end

  def execute(query:, num: 10)
    client = SerpApi::Client.new(engine: "google", api_key: ENV.fetch("SERPAPI_API_KEY"))
    results = client.search(q: query, num: [num, 100].min)
    client.close

    (results[:organic_results] || []).map do |r|
      { position: r[:position], title: r[:title], url: r[:link], snippet: r[:snippet] }
    end
  end
end

chat = RubyLLM.chat(model: "gpt-4o-mini").with_tools(SearchGoogle)
chat.ask "What are the top 5 Google results for 'ruby on rails'?"

Want a reusable agent with its own personality and tools? Then you can define the above tool-augmented chat as its own agent:

class SeoAgent < RubyLLM::Agent
  model "gpt-4o-mini"

  instructions <<~PROMPT
    You are an SEO research assistant. You help users investigate Google search rankings,
    analyze competitors, and review historical position data for the sites and keywords.
    Use the tools available to you whenever a question requires live search data.
  PROMPT

  tools CheckKeywordPosition, SearchGoogle, FindCompetitors
end

agent = SeoAgent.new
response = agent.ask("What are the top 5 Google results for 'ruby on rails'?")

Now your application has agents with access to internal and external tools to do their best work. The DSL for that is small, compact, and beautiful. Yes, it's just an abstraction really. These agents aren't autonomous agents that writes your code.

Abstractions do matter though. Good abstractions will DRY your code and become the predictable patterns for your code writing agents. If you want to see another example, go read my previous post How to implement AI agents in Rails with RubyLLM.

RubyLLM and Rails

RubyLLM isn't just a Ruby library, it comes with a first-class ActiveRecord integration. A single acts_as_chat declaration gives you a persistent, database-backed chat complete with message history, tool calls, and model switching:

class SeoChat < ApplicationRecord
  acts_as_chat
end

chat = Chat.create! model: "claude-sonnet-4"
chat.ask "What's onpage SEO?"

Even better, we can now update our agent to also have persistence using this model by declaring chat_model:

class SeoAgent < RubyLLM::Agent
  chat_model SeoChat
  model "gpt-4o-mini"
  instructions "You are an SEO research assistant..."
  tools CheckKeywordPosition, SearchGoogle, FindCompetitors
end

Compare all of this to the usual experience where you configure clients, define message schemas, manage request formats and write case statements for provider-specific quirks.

MCPs

Model Context Protocol (MCP) is making AI-assisted Rails development far more practical by giving agents structured access to application context instead of forcing them to rely on massive prompts. Rather than pasting files into a chat window, developers can expose tools that let agents inspect routes, models, database schemas, controllers, jobs, and other parts of an app directly.

The rails-mcp-server project brings this idea to Rails by exposing application internals through MCP-compatible tooling. This allows AI agents such as Claude Code, Cursor, or terminal-based assistants to query the application in a much more targeted and reliable way, similarly as they would do for SerpApi API using the SerpApi MCP.

Another interesting project is rails-ai-context, which focuses on automatically generating structured Rails context for AI tools. Instead of manually curating prompts, the gem helps agents understand relationships between routes, models, Stimulus controllers, validations, and other parts of the stack.

More tools

The RubyLLM and MCP tooling are excellent additions to an already rich ecosystem of gems handling anything from agentic workflows (like

Active Agent) and authorization (like Action Policy), to analytics (like Ahoy) and search (like Searchkick).

Rails also comes with admin panels like Active Admin or Avo

Kamal, the recent addition to the default Rails application Gemfile, is especially interesting given the agents can now use it to deploy your application on your own terms. This simple deploy tool comes with built-in gapless deployments, local Docker registry, or SSL/TLS support. Your agents now only need something like a Digital Ocean or Hetzner token for handling everything.

Rails doesn't always come with many gem alternatives given the relatively smaller community, but the libraries it has are usually of a higher quality and the consolidation of choices once again benefit the training and output of coding agents.

Future

What about the future? Can Rails still keep up? Luckily, it seems that Rails is well positioned for a successful future thanks to its leadership and the Rails Foundation which now heavily invests in the documentation and ecosystem. We are now getting new guides and improvements every day.

37signals, the company DHH co-founded, and the one that's constantly giving back to Rails is on the trajectory to adopt agents in their apps, be it with agent skills or the new CLIs. This adoption is important, because DHH and 37signals still remain the stewardship of Rails. In one of the latest piece Promoting AI agents, DHH shares his optimistic view on the progress in agent coding:

But with these autonomous agents, the experience is very different. It's more like working on a team and less like working with an overly-zealous pair programmer who can't stop stealing the keyboard to complete the code you were in the middle of writing. With a team of agents, they're doing their work autonomously, and I just review the final outcome, offer guidance when asked, and marvel at how this is possible at all.

DHH seems to have a healthy view on AI, continuously adopting new ways of working with it and reevaluating the status quo.

Similarly, the new Rails documentation work aimed directly at AI agents is also another step forward. Better machine-readable documentation combined with MCP tooling gives agents access to authoritative framework knowledge and paves the way for agents to work on the Rails codebase itself.

Fin

It's not that Rails is the winner in every single category we could consider for agentic coding. There are more token-efficient languages than Ruby, they are frameworks that are more popular, more minimal, with a faster runtime, and with project leaders more bullish on AI. But the thing with Rails is that it ranks high enough in all these categories, from token efficiency to high quality guides.

Considering all of that, Rails is the most attractive framework for robots today, just as it once was for humans. At SerpApi we continue to bet on Rails to build the world-class SERP API and we invest in its thriving ecosystem and community. We joined the Rails foundation, we are sponsoring the upcoming Rails World in Austin, and we continue to work on our Open Source projects like SerpTrail and Nokolexbor.

Join us in our effort to make Rails better.

DEV Community: SerpApi

How to Scrape Walmart Reviews Results

Why scrape Walmart reviews?

What can you extract from Walmart product reviews?

Why use an API?

Getting the product_id

How to scrape Walmart reviews

GET request

Python Tutorial

Setup

Parse the reviews

The output

Scrape every page of reviews

Export the reviews to CSV

JavaScript and Node.js

cURL

Other languages and no-code solutions

Conclusion

Understanding HTTP 429: Too Many Requests

HTTP status codes

HTTP 429: Too Many Requests

HTTP 429 versus HTTP 503

Why servers return HTTP 429

Check the Retry-After header

How clients should handle HTTP 429

Respect Retry-After

Use exponential backoff and jitter

Limit retries

Reduce concurrency

Cache and batch requests

Authenticate correctly

Retrying a 429 response

Conclusion

Introducing SerpApi's New Markdown Output

Why Markdown for LLMs

Token Efficient

Native Markdown Links

No Noise

Structured Output

When to Choose Markdown or JSON?

Using the JSON Restrictor

Cutting Tokens While Staying on JSON

Stacking Both

Conclusion

Other Resources

How a Lead Generation Company Scaled Outreach with SerpApi's Google Maps API

About The Use Case

Why Google Maps Data Matters For Businesses

The Challenge: Generating Leads at Scale

Why SerpApi?

Building a Scalable Lead Generation Engine

The Impact

Looking Ahead

About SerpApi

Relevant Links

Explore More Google Maps API Use Cases

How to Scrape Google Flights

Who Needs the Scraped Google Flights Data?

Available Data on the Google Flights API

Flight Results

Price Insights

Basic Google Flights Data Scraping

Setting Up a SerpApi Account

Search by Airport Code

cURL implementation

Python Implementation

JavaScript Implementation

Search by city ID

Customize the Flight Search

Round-trip vs. one-way

Retrieving the returning flights

Scrape flight prices and price insights

Cabin class

Filter by airline

Stops, duration, and other filters

Flexible dates

Groups of passengers

Bonus: Deals and airport autocomplete

Frequently Asked Questions (FAQs)

Is it legal to scrape Google Flights?