DEV Community: Sonika Arora

Turn Bing News into an AI Briefing with Claude and SerpApi

Sonika Arora — Tue, 30 Jun 2026 07:06:28 +0000

Most news apps and search engines already present headlines well, so simply listing them is becoming less valuable. The real value has shifted to connecting the dots: grouping related news stories, identifying patterns across sources, and explaining what actually matters and why.

In this post, we’ll build a simple pipeline that turns raw Bing News results into a structured daily briefing which does exactly that using:

SerpApi's Bing News API for real-time news data
Claude for summarization and insight generation

Setting Up

LLMs work best when you give them clean, formatted context instead of raw scraped HTML, so we'll do exactly that.

Get results using SerpApi (Bing News API) → Normalize → Send to Claude with a relevant prompt → Get structured briefing

For this tutorial, we're going to use SerpApi's new Python library to get results from search.

Install SerpApi's new Python serpapi library, the python-dotenv library (for using environment variables) and anthropic library (for Claude) in your environment:

pip install serpapi python-dotenv anthropic

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API key from this page.
[Optional but Recommended] Set your API keys in an environment variable, instead of directly pasting them in the code. Refer here to understand more about using environment variables.

SERPAPI_API_KEY=
CLAUDE_API_KEY=

Get your SerpApi API key here: https://serpapi.com/manage-api-key

Get your Claude API key here: https://platform.claude.com/settings/keys

💡 You'll need to create an account with Claude in case you don't have one yet. You can do that here: https://claude.ai/login

Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import json
import anthropic
from dotenv import load_dotenv
load_dotenv()

SERPAPI_API_KEY = os.environ["SERPAPI_API_KEY"]
CLAUDE_API_KEY = os.environ["CLAUDE_API_KEY"]

Fetch News with Bing News API

We’ll write a function to use SerpApi’s Bing News API (bing_news engine) and get all the news results.

def fetch_news(query):
    client = serpapi.Client(api_key=SERPAPI_API_KEY)
    results = client.search({
        "engine": "bing_news",
        "q": query,
        "device": "desktop",
        "mkt": "en-au"
    })
    return results.get("organic_results", [])

We can call this function with any query such as "Self driving software" or "AI" and scrape results from Bing News.

You can test the Bing News API out in our playground with parameters of your choice:

SerpApi Playground - SerpApi

Prompting Claude

Good prompts can make all the difference when building a AI briefing based on the results.

A bad prompt would look like:

“Summarize these articles”

Better prompts often define structure, force grouping, and require reasoning.

def build_prompt(articles):
    return f"""
You are a technical news analyst.

Task: Convert raw news articles into a structured daily briefing.

Requirements:
- Group related articles into 3-5 themes
- Each theme must have:
  - A short title
  - A 2-3 sentence summary
  - A "Why it matters" explanation
- Be concise and non-generic
- Avoid repeating the same point across themes

Articles:
{articles}
"""

Generate the Briefing (Claude API)

Using Claude's API, we can turn the data we got from SerpApi into a briefing. Here is some simple code to use Anthropic API -

def generate_briefing(prompt):
    client = anthropic.Anthropic(api_key=CLAUDE_API_KEY)
    response = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=1500,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

Putting It Together

Now that we have all the pieces we'll use, let's write the code to call the different functions and get our news briefing:

def run():
    query = "AI"
    articles = fetch_news(query)
    prompt = build_prompt(articles)
    briefing = generate_briefing(prompt)
    print(briefing)

if __name__ == " __main__":
    run()

Now, we're ready to run this file. We can do so by typing python <filename> in the terminal.

Automated Delivery

A briefing is only useful if it reaches you regularly automatically. Add a delivery layer like automated daily email or a slack webhook.

Automated Daily Email

Simple automation run daily via cron or other scheduling methods
Email formatted briefing to yourself or a team

Slack Webhook

Post structured sections into a channel
Good for teams or shared awareness feeds
Enables discussion around themes

Read this blog post for more info on setting up a Slack webhook (The use case in the post is different, but the process remains the same). It also covers how to schedule a job every day using the Python schedule library:

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Generated AI Briefing Example Output

Here's an example run. In my SerpApi call, I set the query to "AI" to keep things simple. I used these parameters for Anthropic API:

model="claude-opus-4-7",
max_tokens=1500,
messages=[{"role": "user", "content": prompt}]

Here's the Claude generated briefing:

Daily AI Briefing

1. AI Reshapes the Workforce: Layoffs, Skill Demands, and Survival Rules

Cloudflare announced a 20% workforce cut (1,100 employees) citing agentic AI as a fundamental disruptor of its operations, sending its stock down 18%. Meanwhile, Airbnb CEO Brian Chesky warned that "pure people managers" and change-resistant workers won't survive the AI era, and CBS reports employers are increasingly demanding AI fluency from new hires.

Why it matters: This is one of the first major tech layoffs explicitly attributed to agentic AI rather than macro conditions—signaling a shift from AI-as-productivity-tool to AI-as-headcount-replacement. Workers and managers face mounting pressure to reskill or face displacement.

2. Conflicting Narratives on AI's Human Impact

Nvidia's Jensen Huang publicly framed AI as a net job creator, urging workers not to fear the technology. In contrast, a new study found that just 10 minutes of AI use measurably impaired cognitive performance on arithmetic and reasoning tasks.

Why it matters: The optimistic vendor narrative (Huang) is increasingly at odds with both labor-market evidence (Cloudflare) and emerging cognitive research. Expect this tension to drive policy debates and shape enterprise adoption guardrails.

3. AI Stocks Diverge as Earnings Season Tests the Hype

SoundHound AI beat Q1 expectations with 51.7% revenue growth but still dropped 12.4%, while Cloudflare cratered 18% post-earnings. Analysts are also reframing the Nvidia-vs-Palantir debate for 2026 as both stocks face renewed scrutiny.

Why it matters: Markets are no longer rewarding AI exposure alone—even strong top-line growth isn't enough. Investors are demanding margin discipline and clear AI monetization paths, marking a maturation point for the sector.

4. Military AI Moves From Theoretical to Operational

The UK's Royal Air Force chief confirmed AI-powered uncrewed fighter aircraft are already being integrated into operations, far ahead of prior timelines. Simultaneously, CNN reports the Pentagon faces growing scrutiny over the legal limits of AI in warfare, including its use in Iran-related operations.

Why it matters: Autonomous combat systems have crossed from R&D to deployment faster than legal and ethical frameworks can adapt. The gap between operational reality and governance is becoming a strategic vulnerability for Western militaries.

5. The Consumer AI Landscape Crystallizes

TechRepublic published a side-by-side comparison of the major chatbots—ChatGPT, Gemini, Copilot, Claude, Perplexity, Grok, DeepSeek, and Meta AI—mapping each to specific use cases and integrations.

Why it matters: With eight serious contenders now in the consumer/prosumer market, differentiation is shifting from raw model quality to ecosystem integration and vertical specialization. The "one assistant to rule them all" thesis is dead; multi-tool workflows are the new norm.

Building On Top Of This

Changing The Model

As of May 2026, here are the options Claude provides and what each is best for:

Opus (Most Intelligent): Best for complex tasks, deep reasoning, coding, and strategic analysis. It is the most powerful but slower and more expensive.
Sonnet (Balanced): It provides the best balance of speed, intelligence, and cost, making it ideal for daily coding, data analysis, and content generation.
Haiku (Fastest): Optimized for speed and cost-efficiency. It is perfect for high-volume tasks, quick responses, and simple, repetitive jobs.

You can test the script with different models and figure out the best one for your use case. I used Claude Opus 4.7 for this demo.

Persisting History

Instead of generating standalone briefings, store each run with a timestamp. This turns your system into a time-series of news summaries. Once you have history, you can:

Compare today vs yesterday (what’s new, what disappeared)
Track theme momentum (what’s increasing or fading)
Ask Claude for delta summaries instead of fresh-only summaries

Example prompt changes:

“Compare today’s briefing with yesterday’s. Highlight new, removed, and shifting themes.”

Conclusion

The key takeaway here is that LLMs work best when they’re grounded in high-quality external data. In this project, SerpApi provides structured, real-time Bing News data, while Claude turns that raw information into something readable and useful: a concise daily briefing with themes, summaries, and context.

Instead of asking a model to know everything, we can give it fresh information and let it focus on synthesis and reasoning.

I hope you found this useful. If you have any questions, feel free to reach out at contact@serpapi.com

Relevant Links

How To Scrape Home Depot Product Reviews

Sonika Arora — Mon, 29 Jun 2026 07:05:48 +0000

Customer reviews are one of the most valuable sources of product insight. They reveal real-world feedback, highlight product strengths and weaknesses, and help businesses understand customer sentiment.

In this post, we'll explore how to collect product reviews from Home Depot using SerpApi's Home Depot Product Reviews API. By the end, you'll be able to fetch product reviews data such as ratings, titles, and comments for analysis and monitoring.

Why Scrape Reviews?

Home Depot is one of the largest home improvement retailers in the United States. Product reviews on its site provide valuable insights for:

Competitive analysis — Understand how competing products are perceived.
Sentiment analysis — Identify common praise or complaints.
Market research — Learn what customers value most.
Product development — Discover improvement opportunities.

Instead of manually collecting reviews, SerpApi lets you retrieve them programmatically and with many no code options.

What Type of Data Can You Extract?

The Home Depot Product Reviews API scrapes reviews for products from Home Depot and provides them in JSON format. Here are some prominent fields from the review object:

id — Unique identifier for the review.
title — Short headline summarizing the reviewer’s opinion.
text — Full written content of the customer review.
rating — Numeric score given by the reviewer (typically on a 1–5 scale).
badges — Indicates badges on products like verifiedPurchaser.
reviewer.name — Display name of the person who wrote the review.
time — Timestamp showing when the review was published.
original_source.name — Website where the review originally appeared.
images.link — URLs of photos included in the review.
total_positive_feedback — Number of users who found the review helpful.
total_negative_feedback — Number of users who found the review unhelpful.

Read the documentation page here:

The Home Depot Product Reviews API - SerpApi

Here's an example in our playground:

SerpApi Playground - SerpApi

Setting up SerpApi

Are you new to SerpApi? We'll walk you through the steps from sign up to integrating with SerpApi.

Getting Started with SerpApi

To get started with SerpApi, I recommend following the steps in this blog post:

Getting started with SerpApi: The Web Search API

Home Depot Product Reviews API Documentation

We recommend familiarizing yourself with the available parameters and details in the documentation here:

The Home Depot Product Reviews API - SerpApi

Integrations / Libraries

Some examples below will include our SerpApi libraries. If you would like to use one of our SerpApi libraries, we offer instructions for each language, such as Python, Node.js, Ruby, PHP, Java, Go, and more:

SerpApi: Integrations

Scraping Home Depot Product Reviews

You can use any programming language in order to call our APIs. The parameters and results we reviewed above will not differ between the request methods, however the syntax used to call the APIs may differ.

You will see across the examples the engine is always set to home_depot_product_reviews, the api_key will be set to your API key, and we include the product_id in every query. In this example we are going to look for reviews for product with ID: 302752040

GET Request

The syntax for the actual GET request between languages will vary between programming languages. However, the URL used each time is consistent. When you are sending a GET request in any language you will use this URL to make the request:

https://serpapi.com/search.json?engine=home_depot_product_reviews&product_id=302752040

Feel free to explore more parameters here, and add any that are relevant to your use case.

An Example With Our Python Library

Install SerpApi's new Python serpapi library in your environment:

pip install serpapi

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

Now, let's take a look at the code:

import serpapi

client = serpapi.Client(api_key=<YOUR_API_KEY>)

results = client.search({
  "engine": "home_depot_product_reviews",
  "product_id": "302752040"
})

Replace the placeholder with your API key.

An Example With cURL

curl --get https://serpapi.com/search \
 -d api_key="YOUR_SECRET_API_KEY" \
 -d engine="home_depot_product_reviews" \
 -d product_id="302752040"

No Code Options

Google Sheets

To connect your code with Google Sheets first follow this blog post:

Connect SERP API with Google Sheets (No Code)

=SERPAPI_RESULT("engine=home_depot_product_reviews&product_id=302752040", "reviews.0.text")

Make.com

While not every API we offer is on Make.com, there is a Universal Module where you can use any API.

Review this blog post to get started:

Announcing SerpApi’s Make App

N8N

Similar to Make.com, not every API is currently included in N8N. However, you can use the HTTP Request node to still call any of our APIs.

To get started review this blog post:

Boost Your n8n Workflows with SerpApi’s Verified Node

Conclusion

Scraping product reviews can unlock valuable insights about customer behavior and product performance. With the Home Depot Product Reviews API, you can retrieve structured review data in just a few lines of code.

I hope you find this useful. If you have any questions, feel free to reach out at contact@serpapi.com

Relevant Links

Related Blogs

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Sonika Arora — Sun, 28 Jun 2026 07:05:02 +0000

Monitoring when competitors begin ranking for new keywords can reveal product launches and marketing strategy changes. Instead of manually checking search results every day, you can automate the process with a Slack bot that sends alerts whenever a new company appears for keywords that matter for your brand.

In this tutorial, we’ll build a simple system that:

Checks search results using the SerpApi's Google Search API.
Detects when a competitor domain appears for a tracked keyword and sends an alert to a Slack channel.
Run this process everyday using Python's schedule library.

Setting Up

This system will work in four steps:

Send in requests to SerpApi's Google Search API for a list of relevant queries
Extract all indexed pages
Compare them with previously stored URLs
Send alerts to Slack for new pages detected

We'll run this job every day using the Python schedule library. You can set the schedule as you like.

To start, you’ll need:

A SerpApi account
A Slack workspace (and a Slack Incoming Webhook URL)
Python installed

Create a Slack webhook

Go to Slack → Apps
Search for Incoming Webhooks
Create a webhook for your channel

You will get a Webhook URL like: https://hooks.slack.com/services/XXXX/XXXX/XXXX

Save this, as we'll need it later.

Code related setup steps

For this tutorial, we're going to use SerpApi's new Python library to get results from search.

Install SerpApi's new Python serpapi library, the python-dotenv library, tldextract library (for parsing the domains from URLs), and the schedule library in your environment:

pip install serpapi python-dotenv schedule tldextract

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.
[Optional but Recommended] Set up your Slack Webhook URL as an an environment variable as well. In your .env file, this will look like this:

SERPAPI_API_KEY=
SLACK_WEBHOOK_URL=
Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import requests
import json
import schedule
import time
import tldextract
from dotenv import load_dotenv
load_dotenv()

SERPAPI_KEY = os.environ["SERPAPI_API_KEY"]
queries = [] # List of queries for which you want to find the updated domains
DB_FILE = "known_domains.json"
SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK_URL"]

Provide the value of your slack webhook in the SLACK_WEBHOOK variable. It looks something like this: https://hooks.slack.com/services/XXXX/XXXX/XXXX.

Provide a list of queries you're looking to track competitors with. Example: ["ai images tool", "ai image generators", "image creation ai tools"]

Get all indexed domains for a list of queries

For the purpose of this post, we'll consider the first 30 organic results and use a simple list of 3 queries. You can increase or decrease this number as needed and change the queries as well depending on your use case.

A simple function to get the root domain from the URL using the tldextract library we installed earlier:

def get_root_domain(url):
    extracted = tldextract.extract(url)

    # Join the domain and suffix to get the root domain
    return f"{extracted.domain}.{extracted.suffix}"

Now, let's create a script com get_domains.py where we use SerpApi 's Google Search API to get the domains of the first 30 search results for a particular query.

def get_all_domains(query_list):
    domains_for_each_query = {}
    client = serpapi.Client(api_key=SERPAPI_KEY)

    for query in query_list:
        domains_for_each_query[query] = set()

        for start in range(0, 30, 10):
            results = client.search({
                "engine": "google",
                "google_domain": "google.com",
                "q": query,
                "start": start
            })

            if "organic_results" not in results:
                break

            for result in results["organic_results"]:
                link = result.get("link")
                if not link:
                    continue

                root_domain = get_root_domain(link)
                domains_for_each_query[query].add(root_domain)

    return {k: list(v) for k, v in domains_for_each_query.items()}

This function sends a request to SerpApi's Google Search API and collects the root domains from the organic result links, and stores them per query. It uses a set to keep only unique domains and fetches up to the first 30 results (pagination with start). It returns a dictionary mapping each query to a list of unique domains found in the search results.

💡 If you want to dive deeper into how to scrape Google search results with Python, read this post.

Save known domains

For the purpose of this example, let's assume the file known_domains.json stores all previously stored pages in JSON format. So, we now need to compare with the domains in this file to find newly added ones.

💡 If you are starting from scratch, using the scripts here will pull all the domains because the known_domains.json file will be empty. Next time, when you use the script, you'll only see the new ones added.

Let's write some code to load the URLs from the file and save the new URLs to the file which we can use later on:

def load_domains():
    try:
        with open(DB_FILE) as f:
            return dict(json.load(f))
    except Exception:
        return {}


def save_domains(domains):
    with open(DB_FILE, "w") as f:
        json.dump(domains, f)

Send slack alerts

Let's write a simple function to send slack alerts for any new domains seen for each query. We'll use this later.

def send_slack_alert(domains):

    # Filter out keys with no new values
    domains = {k: v for k, v in domains.items() if v}

    if not domains:
        message = {
            "text": "✅ No new companies detected for any query."
        }
        requests.post(SLACK_WEBHOOK, json=message)
        return
    for query in domains:
        if domains.get(query) != []:
            message = {
                "text": f"🚨 New company detected for '{query}':\n" +
                        "\n".join(domains[query])
            }
            print(message)
            requests.post(SLACK_WEBHOOK, json=message)

Once everything is set up, your Slack channel will receive alerts like this once a day if new domains detected for any query:

🚨 New company detected for 'image creation ai tools':

company1.com

company2.com

Detect new pages

Now let's work on code to specifically detect new pages by looking at already stored ones in known_domains.json.

def check_for_new_domains():
    print("Checking for new domains...")

    known_domains = load_domains()
    current_domains = get_all_domains(QUERY_LIST)
    new_domains = {}

    for query in current_domains:
        new_domains[query] = list(
            set(current_domains[query]) - set(known_domains.get(query, []))
        )

        known_domains[query] = list(
            set(known_domains.get(query, [])) | set(current_domains[query])
        )

    send_slack_alert(new_domains)
    save_domains(known_domains)

This function checks the search results for new domains appearing for each query compared to previously stored domains. It computes the difference between current domains and known domains, sends a Slack alert for any new ones, and then updates the stored list with the latest domains.

Run the bot on a schedule

Now the only thing left is to schedule this so it runs once everyday.

To do this, we'll use Python's schedule library.

# Schedule the bot to run every day at 9 AM
schedule.every().day.at("09:00").do(check_for_new_domains)
print("Query domain monitoring bot running...")

while True:
    schedule.run_pending()
    time.sleep(60)

To keep the tasks running even after you close your terminal or IDE, you can choose to run the script as a persistent background process. For linux/macOS, use nohup or disown to detach the script from the terminal, e.g., python3 script.py & disown.

What the alert looks like

I ran the code twice with QUERY_LIST = ["ai images tool", "ai image generators", "image creation ai tools"]

The first time, I got some new pages (because a few new companies started ranking for the queries since when I ran it last)

The next time, since all the webpages were already seen, I got a message letting me know that no new pages were detected.

These were the alerts I received:

Now your team will instantly know when new companies start ranking for important keywords that matter for your brand.

Possible Enhancements

Once the basic setup works, you can extend it for a couple more use cases:

Track ranking changes : Alert when a domain jumps from page 2 -> page 1
Monitor multiple regions : Use different location parameters
Track different features : Monitor if competitors appear in other parts of SERP like featured snippets, shopping results, and knowledge panels

Conclusion

By combining SerpApi, Slack, and a Python script, you can build a lightweight monitoring system that continuously tracks competitor activity in search.

With this, you’ll have a practical developer tool that saves hours of manual SERP monitoring and helps your team react faster to competitive changes. You can also use this tutorial to setup any other recurring tasks and send notifications to the slack channel.

You can find the entire code here:

GitHub - serpapi/slack-alerts-for-new-company-ranking: In this tutorial, we’ll build a simple system that detects when a new brand ranks for a tracked keyword and send an alert to a Slack channel.

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Build a Slack Bot That Alerts You When Your Competitor Launches a New Page

Sonika Arora — Sat, 27 Jun 2026 07:04:07 +0000

If you're tracking competitors, one of the most valuable signals is when they launch a new page and it shows up in search results. New pages that companies add often reveal:

New product launches
New feature pages
New landing pages targeting keywords
New content marketing strategies

Manually checking for changes can take quite some time, and repetitive work as pages are updated often. Instead of manually checking competitor sites, you can automate this process using:

SerpApi (Google Search API)
Slack Webhook
Python's schedule library

In this tutorial, we’ll build a simple system that checks search results using the SerpApi's Google Search API to detect when a new competitor domain URL is noticed in search results and then sends an alert to a Slack channel.

Setting Up

This system will work in four steps:

Send in a request to SerpApi's Google Search API with a relevant query
Extract all indexed pages
Compare them with previously stored URLs
Send alerts to Slack for new pages detected

We'll run this job every day using the Python schedule library. You can set the schedule as you like.

To start, you’ll need a SerpApi account, a Slack workspace (and a Slack Incoming Webhook URL), and Python installed.

Create a Slack Webhook

Go to Slack → Apps
Search for Incoming Webhooks
Create a webhook for your channel

You will get a Webhook URL like: https://hooks.slack.com/services/XXXX/XXXX/XXXX

Save this, as we'll need it later.

Code Related Setup Steps

For this tutorial, we're going to use SerpApi's new Python library to get results from Google Search.

Install SerpApi's new Python serpapi library, the python-dotenv library, and the schedule library in your environment:

pip install serpapi python-dotenv schedule

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.
[Optional but Recommended] Set up your Slack Webhook URL as an an environment variable as well. In your .env file, this will look like this:

SERPAPI_API_KEY=
SLACK_WEBHOOK_URL=
Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import requests
import json
import schedule
import time
from dotenv import load_dotenv
load_dotenv()

SERPAPI_KEY = os.environ["SERPAPI_API_KEY"]
DOMAIN = "company.com"
DB_FILE = "known_urls.json"
SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK_URL"]

Fetch All Indexed Pages For Competitor

Let's create a script competitor_page_monitor.py where we use SerpApi 's Google Search API to get all pages which show up in search results for a particular company.

def get_all_pages():
   urls = set()
   client = serpapi.Client(api_key=apikey)

   for start in range(0, 100, 10):

       results = client.search({
           'engine': 'google',
           "q": f"site:{DOMAIN} -site:forum.{DOMAIN} -site:{DOMAIN}/blog -site:{DOMAIN}/careers -site:{DOMAIN}/release-notes",
           "start": start,
       })

       if "organic_results" not in results:
           break

       for result in results["organic_results"]:
           urls.add(result["link"])

   return urls

💡 You may need to add more customized search operators to your query if you want to ignore specific types of pages. For this example, I'm using site:company.com -site:forum.company.com -site:company.com/blog -site:company.com/careers -site:company.com/release-notes" to ignore the blog, forum and career pages.

Conversely, you can also choose to target specific areas such as blogs and get notified what exactly your competitors are writing about publicly.

Feel free to change the maximum pagination as you'd like. I've set start to a max value of 100 which means we'll see data from up to 10 pages (since each page usually has 10 results).

Now that we have all the pages for that domain, we need to find the newly added ones since our last check.

💡 If you want to dive deeper into how to scrape Google search results with Python, refer to this post.

Save Known URLs

For the purpose of this example, let's assume the file known_urls.json stores all previously stored pages in JSON format. So, we now need to compare with the URLs in this file to find newly added ones.

💡 If you are starting from scratch, using the scripts here will pull all the URLs for the competitor because the known_urls.json file will be empty. Next time, when you use the script, you'll only see the new ones added.

Let's write some code to load the URLs from the file and save the new URLs to the file which we can use later on:

def load_urls():
    try:
        with open(DB_FILE) as f:
            return set(json.load(f))
    except:
        return set()

def save_urls(urls):
     with open(DB_FILE, "w") as f:
          json.dump(list(urls), f)

Send Slack Alerts

Let's write a simple function to send slack alerts. We'll use this later.

SLACK_WEBHOOK = "YOUR_SLACK_WEBHOOK"

def send_slack_alert(urls):
   if urls:
       message = {
           "text": f"🚨 New competitor pages detected: \n{'\n'.join(urls)}"
       }
   else:
       message = {
           "text": "✅ No new competitor pages detected."
       }
   requests.post(SLACK_WEBHOOK, json=message)

💡 Provide the value of your slack webhook in the SLACK_WEBHOOK variable. It looks something like this: https://hooks.slack.com/services/XXXX/XXXX/XXXX.

Detect New Pages

Now let's work on code to specifically detect new pages by looking at already stored ones in known_urls.json.

def check_new_pages():
    print("Checking for new pages...")

    known_urls = load_urls()
    current_urls = get_all_pages()
    new_pages = current_urls - known_urls

    send_slack_alert(new_pages)

    new_seen_set = known_urls.union(current_urls)
    save_urls(new_seen_set)

Run the Bot on a Schedule

Now the only thing left is to schedule this so it runs once everyday.

To do this, we'll use Python's schedule library.

💡 I'm using Python's schedule library instead of cron because it works reliably on macOS and local environments

schedule.every().day.at("09:00").do(check_new_pages)
print("Competitor monitoring bot running...")

while True:
    schedule.run_pending()
    time.sleep(60)

What The Alert Looks Like

Once this is set up, your Slack channel will receive alerts like this once a day:

🚨 New competitor pages detected:

https://competitor.com/products/ai-assistant

https://competitor.com/features/fast-mode

I ran the code twice for the domain serpapi.com. The first time, I got some new pages and the next time, since all the webpages were already seen, I got a message letting me know that no new pages were detected. These were the alerts I received:

This lets you instantly spot new product launches, new feature releases, and new landing pages which are showing up in search results for any company.

Conclusion

With about 50 lines of Python, you now have a bot that monitors all indexed pages of a competitor, detects new launches, and sends instant Slack alerts.

This is one of the simplest ways to build a competitive intelligence system using search data. You can also use this tutorial to setup any other recurring tasks and send notifications to the slack channel.

You can find the entire code here:

GitHub - serpapi/slack-alerts-for-new-competitor-pages: In this tutorial, we’ll build a simple system that uses SerpApi’s Google Search API to detect when a new competitor domain URL appears in search results and then sends an alert to a Slack channel.

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Get AI Generated Responses From Search Engines In Structured JSON

Sonika Arora — Fri, 26 Jun 2026 07:02:48 +0000

Search engine results are no longer just a list of links. Many search engines like Google and Bing surface AI generated answers directly in the results page - in summaries, conversational responses, and synthesized insights that sit above or inside traditional SERP features. For developers and data teams, this raises an interesting question: How can you programmatically collect and analyze these AI answers the same way you do classic SERP data?

With SerpApi you can scrape these generated answers without needing to write brittle scraping logic.

In this post we'll walk through how AI generated answers appear across major search engines and how SerpApi exposes them in structured JSON.

Why scrape AI generated responses?

AI generated answers have quickly become the first thing users often see. Scraping them enables:

Generative engine optimization (GEO) & content research - Understand what content AI models consider authoritative and trustworthy, and track how your pages appear in AI-generated answers.
Search experience monitoring - Track how AI summaries describe your brand or product.
Competitive analysis - See which competitors are mentioned or cited.
LLM grounding - Compare AI answers across engines for factual drift.
Product intelligence - Detect changes in how queries are answered over time.

Let's dive into where we can find these AI responses.

Google: AI Mode, AI Overviews, and AI in People Also Ask

Google AI Mode (Conversational Search)

Google AI Mode introduces a chat‑like search experience, where the response is generated dynamically rather than assembled from classic SERP blocks.

With SerpApi, you can capture these responses in structured form with our Google AI Mode API:

The full AI response text
Follow up questions at the end of the answer
All the references
Shopping results

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Google AI Overviews

Google AI Overviews appear at the top of the SERP results as a synthesized AI generated answer with citations.

With SerpApi’s Google Search API, AI Overviews are returned as a dedicated JSON object, typically including:

The generated summary text
Referenced sources / citations
Highlighted concepts

💡 Google may not always return the overview content, but instead return a page_token in the ai_overview object. When this happens, you can send an additional request to the AI Overview API to retrieve the full AI Overview.

You can do this using the page_token or by simply sending a GET request to the serpapi_link in the ai_overview object in the Google Search API response. The page_token is only valid for one minute after the initial request, so we recommend querying for the AI overview immediately after your initial search.

This allows you to extract the AI generated answer without parsing HTML or executing JavaScript.

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like when the ai_overview content is included:

Here is what the response looks like when the a page_token is returned instead:

After a request to the AI overview API with that page_token we get:

AI Answers in “People Also Ask” (PAA)

The People Also Ask section has evolved from expandable snippets with human written answers into AI‑style synthesized answers for many queries.

Using SerpApi, you can get each PAA question and answer using our Google Search API, which includes:

The associated question
The generated answer text

- Source references (when available)

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Bing Copilot Answers

Microsoft Bing has a Copilot page where you can get ask questions in a chat‑like search experience.

With SerpApi’s Bing Copilot API, Copilot responses are exposed as structured fields rather than embedded UI components.

You can extract the following from the Copilot generated answer:

Text and lists
Links wherever they appear in the answer
Reference indexes
Related follow‑up prompts

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

DuckDuckGo Search Assist API

DuckDuckGo exposes AI assisted summaries and responses through its Search Assist experience.

With SerpApi’s DuckDuckGo Search Assist API, by setting the search_assist parameter to true, you can programmatically retrieve:

AI generated instant answers
Referenced sources / citations

💡 Search assist results may contain answer, expanded_answer, sources, titles, abstracts, domains, urls, organic_word_ngrams, and organic_wiki_entities.

DuckDuckGo is the only engine to include organic word n-grams (sequences of n consecutive words) in their AI assisted summaries. These may be used to structure and optimize content based on actual search-driven data. They allow you to move beyond single-keyword targeting to understand user intent, phrase patterns, and topical depth.

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Naver AI Briefing

Naver, the dominant search engine in South Korea, has rolled out its own AI Briefing feature, tailored to local content and language preferences.

SerpApi supports Naver search results and surfaces the AI generated answer in structured JSON with the Naver AI Overview API, including:

AI generated answer (in markdown or broken down by paragraphs)
Linked sources from Naver properties and external sites
Reference indexes
Any displayed media

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Why Use SerpApi?

SerpApi streamlines the process of web-scraping. SerpApi manages the intricacies of scraping and returns structured JSON results. This allows you to save time and effort by avoiding the need to build your own scrapers or rely on other web scraping tools.

We do all the work to maintain all of our parsers and adapt them to respond to changes on search engines. This is important, as search engines are constantly experimenting with new layouts, new elements, and other changes. By taking care of this for you on our side, we eliminate a lot of time and complexity from your workflow.

Conclusion

As search continues to evolve from links to overviews and AI generated answers, having structured access to AI generated responses gives you a critical edge.

I hope this post enabled you in understanding how AI generated answers appear across major search engines and how you can use SerpApi to get them in structured JSON. Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Documentation

Get Hotel Details From Tripadvisor And Compare Hotels using Python

Sonika Arora — Thu, 25 Jun 2026 07:00:36 +0000

When people travel, one of the first things they do is compare hotels. Price alone isn’t enough - ratings, amenities, location, and recent reviews all factor into the decision. For developers building travel products, data analysts, or recommendation tools, this means turning rich but messy public data into structured, comparable insights.

Tripadvisor is one of the most valuable public sources for this kind of hotel intelligence, but extracting structured, comparable data at scale is difficult. In this post, we’ll walk through how to scrape hotel details from Tripadvisor using SerpApi’s Tripadvisor Place Results API with Python, and then compare hotels programmatically using the returned data.

What Are Tripadvisor Place Results?

The Tripadvisor Place Results page looks like this:

Here's an example URL: https://www.tripadvisor.com/1197076

A place on Tripadvisor represents a specific entity such as:

A hotel
A restaurant
An attraction

For hotels, place results typically include:

Hotel name and Tripadvisor URL
Rating and reviews
Pricing
Address and geographic location
Pictures and descriptions
Nearby destinations
Amenities and categories

This makes it a perfect input for hotel comparison workflows.

Get Hotel Results from Tripadvisor

Using SerpApi, you can request Tripadvisor place results by providing a hotel ID.

To get the hotel ID (place_id), you can make a request to our Tripadvisor Search API like this:

Set engine parameter to tripadvisor
Provide a search query as the location (example: "Rome") and specify the ssrc(Search Filter) parameter as h (for Hotels)
Specify your API key in the api_key parameter
Receive structured JSON with all hotels in that location
Grab the place_id

Here's an example in our Playground:

SerpApi Playground - SerpApi

Then, for each hotel, we can get the Place Results using the place_id and sending the request to the Tripadvisor Place API.

The request for getting hotel details looks like:

Set engine parameter to tripadvisor_place
Provide a place_id (example: 228406)
Specify your API key in the api_key parameter
Receive structured JSON with hotel details

Here's an example in our Playground:

SerpApi Playground - SerpApi

The response gets us Tripadvisor place data for each hotel that would otherwise require manual parsing.

Some Basic Setup Steps

Setup your environment

Ensure you have the google-search-results library installed.

pip install google-search-results

google-search-results_ _is our Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. However, serpapi is a new one, and all the examples you can find on our website are from the old one google-search-results. If you'd like to use our Python library with all the examples from our website, you should install the google-search-results module instead of serpapi.

For this blog post, I am using google-search-results because all of our documentation references this one.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

In addition to that, we'll need to install a few more libraries:

Pandas: Useful for organizing data. Install this with: pip install pandas
Matplotlib: Useful for building static plots. Install this with pip install matplotlib

Get your SerpApi API key

To begin scraping data, first, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API.

Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.

Import Libraries

Import all the necessary libraries for this project:

import csv
import os
from serpapi import GoogleSearch
from dotenv import load_dotenv
import pandas as pd
import matplotlib.pyplot as plt

load_dotenv()

Get Hotel Details using Python

For this tutorial, we'll look for 10 hotels in Rome. Once the setup is complete, here is a a simple Python script to get all hotels in Rome, and get hotels details for each one and add that to a list:

hotels = []
def get_hotel_results_from_tripadvisor(q):
    params = {
        "engine": "tripadvisor",
        "q": q,
        "api_key": os.environ["SERPAPI_API_KEY"],
    }
    results = GoogleSearch(params).get_dict()
    return results.get("places", [])

def get_hotel_details_page_from_tripadvisor(place_id):
    params = {
        "engine": "tripadvisor_place",
        "place_id": place_id,
        "api_key": os.environ["SERPAPI_API_KEY"],
    }
    results = GoogleSearch(params).get_dict()
    return results.get("place_result", {})

if __name__ == " __main__":
    q = "Rome"
    place_results = []
    hotels = get_hotel_results_from_tripadvisor(q)
    first_10_hotels = hotels[:10]
    for result in first_10_hotels:
        place_id = result.get("place_id", None)
        if place_id:
            hotel_details = get_hotel_details_page_from_tripadvisor(place_id)
            place_results.append(hotel_details)

Compare Hotels Programmatically

Let's first start by identifying fields that will be of importance to us.

Pick The Fields For Fair Comparison

To understand a place's identity:

place_result.name
place_result.rating
place_result.reviews
place_result.hotel_stars
place_result.num_rooms

Useful for competitive context:

place_result.ranking → parse numeric rank
place_result.reviews → popularity signal

For understanding pricing:

price_range.low
price_range.high
Lowest offers[].extracted_price

Sub rating across various dimensions:

These are already numeric and normalized by Tripadvisor, making them perfect for comparison:

Location
Rooms
Value
Cleanliness
Service
Sleep Quality

Review highlights to extract insights from Reviews

reviews_highlights.category
reviews_highlights.value

💡 You can ignore summaries & quotes for scoring to keep things quantitative. For qualitative data, the review summaries are a good place to get it.

Define a Scoring Strategy

As a simple comparison strategy, we can compute three layers:

Base Quality Score → from subratings
Sentiments → from reviews_highlights
Value-for-Money Score → rating ÷ price

We can also weight the three differently when computing the final score for a hotel. For this tutorial, we'll use:

60% quality score (most reliable)
25% sentiment score (guest opinions)
15% price efficiency score

Let's define the plan for how we will use this scoring strategy on the list of place results we have already:

Get place_results for each hotel we want to compare

↓

Extract hotel features we want to compare for each hotel

↓

Compute scores for each hotel based on the scoring strategy above

↓

Store the hotel results in a table/dataframe

↓

Sort / filter / plot / export results

Code Solution

def extract_hotel_features(place_result):
    HIGHLIGHT_SCORE_MAP = {
        "Immaculate": 5.0,
        "Abundant": 4.5,
        "Convenient": 4.5,
        "Good": 4.0,
        "Inconsistent": 3.0,
        "Average": 3.0,
        "Dated": 2.0,
        "Outdated": 2.0,
        "Pricey": 2.0,
    }

    hotel = {
        "name": place_result.get("name"),
        "rating": place_result.get("rating"),
        "review_count": place_result.get("reviews"),
        "hotel_stars": place_result.get("hotel_stars"),
        "num_rooms": place_result.get("num_rooms"),
    }

    # Ranking number
    ranking = place_result.get("ranking")
    if ranking:
        hotel["city_rank"] = int(ranking.split("#")[1].split(" ")[0])

    # Price range
    price_range = place_result.get("price_range", {})
    hotel["price_low"] = price_range.get("low")
    hotel["price_high"] = price_range.get("high")

    # Lowest observed price
    offers = place_result.get("prices", {}).get("offers", [])
    if offers:
        hotel["lowest_price"] = min(
            o["extracted_price"] for o in offers if "extracted_price" in o
        )

    # Subratings (primary scoring dimensions)
    for item in place_result.get("subratings", []):
        key = item["category"].lower().replace(" ", "_")
        hotel[key] = item["score"]

    # Review highlights (sentiment signals)
    for h in place_result.get("reviews_highlights", []):
        category = h["category"].lower()
        value = h["value"]
        hotel[f"{category}_sentiment"] = HIGHLIGHT_SCORE_MAP.get(value, 3.0)

    return hotel

def compute_scores(hotel):
    # 1. Base quality score (Tripadvisor subratings)
    quality_fields = [
        "location",
        "rooms",
        "value",
        "cleanliness",
        "service",
        "sleep_quality",
    ]

    quality_scores = [
        hotel[f] for f in quality_fields if f in hotel
    ]

    hotel["quality_score"] = round(
        sum(quality_scores) / len(quality_scores), 2
    )

    # 2. Value-for-money score
    if hotel.get("lowest_price") and hotel.get("rating"):
        hotel["value_for_money"] = round(
            hotel["rating"] / hotel["lowest_price"] * 100, 2
        )

    # 3. Sentiment score
    sentiment_keys = [k for k in hotel if k.endswith("_sentiment")]
    if sentiment_keys:
        hotel["sentiment_score"] = round(
            sum(hotel[k] for k in sentiment_keys) / len(sentiment_keys), 2
        )

    # 4. Final composite score
    hotel["overall_score"] = round(
        (
            hotel["quality_score"] * 0.6 +
            hotel.get("sentiment_score", 3) * 0.25 +
            hotel.get("value_for_money", 0) * 0.15
        ),
        2,
    )
    return hotel

With these basic functions in place, we can now compare hotels:

def compare_hotels(place_results):
    hotels = []

    for place_result in place_results:
        hotel = extract_hotel_features(place_result)
        hotel = compute_scores(hotel)
        hotels.append(hotel)

    hotels_sorted = sorted(
        hotels,
        key=lambda h: h.get("overall_score", 0),
        reverse=True
    )

    df = pd.DataFrame(hotels_sorted)

    # Select only the most useful comparison columns
    comparison_df = df[

[
"name",
"overall_score",
"quality_score",
"sentiment_score",
"value_for_money",
"rating",
"review_count",
"city_rank",
"price_low",
"price_high",
"lowest_price",
]
]

    return comparison_df

Plotting The Results

Best “at a glance” Ranking

Once we have the comparison_df, we can now plot the overall scores we computed earlier:

def plot_hotels_bar_chart(comparison_df):
    import matplotlib.pyplot as plt

    df = comparison_df.sort_values("overall_score", ascending=False)

    plt.figure()
    plt.barh(df["name"], df["overall_score"])
    plt.xlabel("Overall Score")
    plt.title("Overall Hotel Comparison")
    plt.gca().invert_yaxis()
    plt.tight_layout()
    plt.show()

What that looks like:

Price comparison (range chart)

Let's understand the price spread per hotel.

def plot_price_transparency(comparison_df):
    plt.figure()
    for i, row in enumerate(comparison_df.itertuples()):
        plt.plot(

[row.price_low, row.price_high],
[i, i],
marker="o"
)

    plt.yticks(range(len(comparison_df)), comparison_df["name"])
    plt.xlabel("Price (€)")
    plt.title("Hotel Price Ranges")
    plt.tight_layout()
    plt.show()

Calling this function plots a graph which makes pricing transparent:

Practical Use Cases

This workflow is especially useful for:

Travel startups building hotel comparison products
Data teams analyzing hotel trends
SEO teams creating location-based hotel content
Investors and analysts tracking hotel performance across regions

Instead of manually browsing Tripadvisor for each hotel, you get structured, comparable insights at scale.

Conclusion

I hope that this tutorial helps you use SerpApi's Tripadvisor Place Results API and reliably extract hotel details and compare hotels in any destinations with minimal effort.

You can find all the code used in this blog post here:

https://github.com/sonika-serpapi/get-hotel-details-from-trip-advisor-and-compare-hotels

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Documentation

Scrape News Results from Baidu to CSV with Python

Sonika Arora — Wed, 24 Jun 2026 07:09:36 +0000

Baidu is the dominant search engine in China, and Baidu News is a critical source for understanding media coverage, brand sentiment, and trending topics in the Chinese market. However, scraping Baidu directly is notoriously difficult due to aggressive bot detection, regional constraints, and frequent HTML changes.

In this post, we’ll walk through how to scrape Baidu News using SerpApi.

Why Use SerpApi To Scrape News From Baidu

If you’ve tried scraping Baidu directly, you’ve likely run into issues such as a dynamic HTML, frequent layout changes, javaScript-rendered elements, Chinese - specific quirks (encodings, redirects, tracking parameters). For production use cases, brand monitoring, market research, competitive intelligence, DIY scraping quickly becomes impractical.

That’s where SerpApi’s Baidu News API comes in. It's a structured API, which returns a clean JSON response. SerpApi manages the intricacies of scraping and returns structured JSON results. This allows you to save time and effort by avoiding the need to build your own Baidu scraper using generic web scraping frameworks like BeautifulSoup which require constant maintenance. We do all the work to maintain all of our parsers and adapt them to respond to changes on search engines. By taking care of this for you on our side, we eliminate a lot of time and complexity from your workflow.

About Baidu News API

Baidu News API returns structured news data from Baidu News pages. The API returns structured fields such as title, link, source, date, and snippet.

At a minimum, you need the following parameters:

engine - set to baidu_news
q (search query) - set to your query
api_key - set to your private API key

Here is an example of the information the API scrapes:

Get results using a simple GET request

https://serpapi.com/search.json?engine=baidu_news&q=Artificial+Intelligence&api_key=YOUR_API_KEY

This request searches the Baidu News search engine for Artificial Intelligence and provides JSON results. Try it out on our playground:

SerpApi Playground - SerpApi

An Example Use Case

Let's write some code to extract Baidu News results for Phones and write them to a CSV.

Setup your environment

Ensure you have the necessary libraries installed.

pip install google-search-results

google-search-results_ _is our Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. However, serpapi is a new one, and all the examples you can find on our website are from the old one google-search-results. If you'd like to use our Python library with all the examples from our website, you should install the google-search-results module instead of serpapi.

For this blog post, I am using google-search-results because all of our documentation references this one.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

Get your SerpApi API key

To begin scraping data, first, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API.

Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.

Using Baidu News API

First, we'll write a simple function to get news results from the Baidu News API:

def get_baidu_news(q):
    params = {
        "engine": "baidu_news",
        "q": q,
        "api_key": os.environ["SERPAPI_API_KEY"],
    }

    results = GoogleSearch(params).get_dict()
    return results.get("organic_results", [])

Writing Results To a CSV

Then, we can write a simple script to call this function and write the news results to CSV:

if __name__ == " __main__":
    q = "Phones"
    output_file = "baidu_news_results.csv"

    header = ["position", "title", "link", "date", "snippet", "source", "thumbnail"]

    with open(output_file, "w", encoding="UTF8", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(header)

        results = get_baidu_news(q)
        for r in results:
            writer.writerow([
                r.get("position", ""),
                r.get("title", ""),
                r.get("link", ""),
                r.get("date", ""),
                r.get("snippet", ""),
                r.get("source", ""),
                r.get("thumbnail", "")
            ])

You can run this script to fetch Baidu News results for the query “Phones.” Then, this creates a CSV file called baidu_news_results.csv and writes a header row that defines each column. You can then call get_baidu_news to retrieve the results and loop through them. For each result, it safely extracts the fields and writes them as rows in the CSV file.

Results

Here's what the results look like:

Adding Localization

Use the ct parameter to control the language of the results returned. Set this when you want to restrict results to a specific Chinese language variant.

1 — All languages (default)
2 — Simplified Chinese
3 — Traditional Chinese

Adding Pagination

Use the pn, rn parameters together to control paging and result count. Use pn and rn together to paginate through large result sets.

Result Offset

The pnparameter defines how many results to skip, and is used to move through pages of results.
Examples:
- pn=0 → first page (default)
- pn=10 → second page
- pn=20 → third page

Results per Page

Use the rn parameter defines the maximum number of results to return. (max: 50)
Examples:
- rn=10 → return 10 results (default)
- rn=30 → return 30 results
- rn=50 → return 50 results

A Couple of Advanced Parameters

Sorting the results

The rtt parameter controls how results are ordered. Set this to a particular value when freshness matters more than relevance or vice versa.

1 — Sort by attraction/relevance (default)
4 — Sort by time (most recent first)

Medium Filter

Filters results by content source using the medium parameter. Use this to limit results to specific publisher types or platforms.

0 — No filtering (default)
1 — Only medium-authority sites
2 — Only Baijiahao sources (baijiahao.baidu.com)

All of these parameters are optional, but combining localization, pagination, and filters gives you precise control over the Baidu News results you receive.

Conclusion

Scraping Baidu News directly is fragile and time-consuming. I hope this tutorial helped you understand how to use SerpApi’s Baidu News API to get clean, structured news data with production-ready reliability.

If you’re building anything that depends on Chinese news visibility, this is the fastest and easiest way to get started.

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Documentation

Scrape Reviews From OpenTable to CSV with Python

Sonika Arora — Tue, 23 Jun 2026 07:07:45 +0000

OpenTable is one of the most influential platforms for restaurant reservations and diner feedback. Whether you're doing sentiment analysis, building a restaurant discovery app, or tracking customer satisfaction, accessing review data can be incredibly valuable.

SerpApi’s OpenTable Reviews API allows you to scrape OpenTable reviews reliably and at scale.

In this post, we’ll walk through:

What the SerpApi OpenTable Reviews API is and what data you can extract
How to integrate it using Python
Pagination and Localization

Why Use SerpApi?

SerpApi streamlines the process of web-scraping. We take care of proxies and any CAPTCHAs that might be encountered, so that you don't have to worry about your searches being blocked. If you were to implement your own scraper using tools like BeautifulSoup and Requests, you'd need to determine your own solution for this. SerpApi manages the intricacies of scraping and returns structured JSON results. This allows you to save time and effort by avoiding the need to build your own OpenTable scraper or rely on other web scraping tools.

We also do all the work to maintain all of our parsers and adapt them to respond to changes on search engines. This is important, as search engines like OpenTable are constantly experimenting with new layouts, new elements, and other changes. By taking care of this for you on our side, we eliminate a lot of time and complexity from your workflow.

About the OpenTable Reviews API

OpenTable Reviews API returns structured review data from OpenTable restaurant pages.

The API returns structured review fields such as Reviewer name, Review date, Attached Images, Rating, Text content, Rating and AI summary, and Pagination tokens.

You can try it out in our playground, where you can visualize the response with various parameters:

SerpApi Playground - SerpApi

An Example Use Case

Let's write some code to extract OpenTable reviews for Nobu Palo Alto and write them to a CSV.

Setup your environment

Ensure you have the necessary libraries installed.

pip install google-search-results

google-search-results_ _is our Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. However, serpapi is a new one, and all the examples you can find on our website are from the old one google-search-results. If you'd like to use our Python library with all the examples from our website, you should install the google-search-results module instead of serpapi.

For this blog post, I am using google-search-results because all of our documentation references this one.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

Get your SerpApi API key

To begin scraping data, first, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API.

Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.

Using OpenTable Reviews API

First, we'll write a simple function to get reviews from the OpenTable Reviews API:

def get_opentable_reviews(place_id):
    params = {
        "engine": "open_table_reviews",
        "place_id": place_id,
        "api_key": os.environ["SERPAPI_API_KEY"],
    }

    results = GoogleSearch(params).get_dict()
    return results.get("reviews", [])

Writing Results To a CSV

First, we need to get the place_id for Nobu Palo Alto. This identifier can be found in the URL of the restaurant's page on OpenTable, immediately following /r/.

For Nobu Palo Alto, the URL is https://www.opentable.com/r/nobu-palo-alto , and so the ID is nobu-palo-alto.

Then, we can write a simple script to call this function and write the reviews to CSV:

if __name__ == " __main__":
    place_id = "nobu-palo-alto"
    output_file = "opentable_reviews.csv"

    header = ["submitted_at", "id", "user_name", "rating", "content"]

    with open(output_file, "w", encoding="UTF8", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(header)

        reviews = get_opentable_reviews(place_id)
        for r in reviews:
            writer.writerow([
                r.get("submitted_at", ""),
                r.get("id", ""),
                r.get("user", {}).get("name", ""),
                r.get("rating", {}).get("overall", ""),
                r.get("content", "")
            ])

This block runs only when the script is executed directly. We set the place_id and create a new CSV file with the appropriate header. We then fetch the OpenTable reviews using get_opentable_reviews(), and for each review, we write a row to the CSV. We safely extract fields like the submission date, review ID, username, rating, and content so the script won’t break if any data is missing.

Results

Here's what the results look like:

Pagination with `page`

OpenTable has more reviews than those served on the first page with our Reviews API. We can use the parameter page for further data retrieval and page navigation.

This parameter defines the page number. 1 (default) is the first page of 10 results, 2 is the 2nd page, etc. To get more results beyond the ones we just got, you can make the same request with page set to 2, 3 and so on depending on the number of reviews available and the number of reviews you want to retrieve.

Localization

The open_table_domain parameter defines the OpenTable domain to use. You can localize the request if you are looking for results from another domain for Opentable besides www.opentable.com.

You can view the OpenTable domains page for a full list of supported OpenTable domains.

Conclusion

Using the SerpApi OpenTable Reviews API simplifies the process of collecting high-quality, structured review data. Instead of spending time building and maintaining a fragile scraper, you get to focus on the valuable part: analyzing the customer sentiment and insights hidden within the reviews.

I hope this blog post was helpful in understanding how to you can write a simple script to scrape reviews from OpenTable. If you have any questions, don't hesitate to reach out to me at sonika@serpapi.com.

Relevant Links

Documentation

Scrape reviews from other search engines like Google Maps and Yelp:

DEV Community: Sonika Arora

Turn Bing News into an AI Briefing with Claude and SerpApi

Setting Up

More About Our Python Libraries

Fetch News with Bing News API

Prompting Claude

Generate the Briefing (Claude API)

Putting It Together

Automated Delivery

Generated AI Briefing Example Output

Building On Top Of This

Changing The Model

Persisting History

Conclusion

Relevant Links

Related Blog Posts

How To Scrape Home Depot Product Reviews

Why Scrape Reviews?

What Type of Data Can You Extract?

Setting up SerpApi

Getting Started with SerpApi

Home Depot Product Reviews API Documentation

Integrations / Libraries

Scraping Home Depot Product Reviews

GET Request

An Example With Our Python Library

More About Our Python Libraries

An Example With cURL

No Code Options

Google Sheets

Make.com

N8N

Conclusion

Relevant Links

Related Blogs

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Setting Up

Create a Slack webhook

Code related setup steps

More About Our Python Libraries

Get all indexed domains for a list of queries

Save known domains

Send slack alerts

Detect new pages

Run the bot on a schedule

What the alert looks like

Possible Enhancements

Conclusion

Relevant Links

Build a Slack Bot That Alerts You When Your Competitor Launches a New Page

Setting Up

Create a Slack Webhook

Code Related Setup Steps

More About Our Python Libraries

Fetch All Indexed Pages For Competitor

Save Known URLs

Send Slack Alerts

Detect New Pages

Run the Bot on a Schedule

What The Alert Looks Like

Conclusion

Relevant Links

Get AI Generated Responses From Search Engines In Structured JSON

Why scrape AI generated responses?

Google: AI Mode, AI Overviews, and AI in People Also Ask

Google AI Mode (Conversational Search)

Google AI Overviews

AI Answers in “People Also Ask” (PAA)

- Source references (when available)

Bing Copilot Answers

DuckDuckGo Search Assist API

Naver AI Briefing

Why Use SerpApi?

Conclusion

Relevant Links

Documentation

Related Posts

Get Hotel Details From Tripadvisor And Compare Hotels using Python

What Are Tripadvisor Place Results?

Get Hotel Results from Tripadvisor

Pagination with `page`