DEV Community: SerpApi

Building a Naver News Watchlist With Ruby on Rails

Braden — Wed, 01 Jul 2026 14:31:48 +0000

The problem with monitoring news feeds isn’t finding news; it’s wasting time chasing what’s actually relevant. Refreshing feeds, re-running the same searches, skimming overlapping headlines across numerous tabs and apps, clicking into the same article 10 times before realizing you’ve already read it — it’s exhausting.

Let’s flip that script: you decide exactly which topics and outlets you care about, how many items you want per topic, and how often you want them, and a small, API-only Rails service pulls fresh Naver News results via SerpApi and quietly delivers a single, fully customized digest straight to your inbox.

Sound intriguing? Let’s build it.

What is Naver?

Naver is South Korea’s flagship search portal, and is central to how millions of Koreans discover news, content, and services. Historically dominant in domestic search, Naver remains especially strong on mobile and tablet usage, even as Google has taken the lead on desktop.

For recent market-share context, see StatCounter’s Korea page.

Why SerpApi for Naver News?

Naver’s HTML can change frequently; scraping the page directly is difficult and costly to maintain. SerpApi handles the tricky parts for you:

Stable, normalized JSON: The news_results array returned by SerpApi remains consistent and predictable, even as Naver’s frontend changes.
Proxy and CAPTCHA handling: No need to manage IP rotation or deal with CAPTCHAs.
Smart caching: Identical searches return cached results for free. (Cache expires after 1 hour. You can disable caching by setting no_cache=true. Learn more)
First-class Ruby library: An official Ruby Library by SerpApi that is easy-to-use, well-documented, and consistent in behavior.

Using SerpApi lets you focus on what you want to build, rather than on keeping your scraper alive.

Try it First

Before writing any code, you can explore Naver News Results in the SerpApi Playground. Inspect the exact JSON structure returned by SerpApi, test advanced filtering parameters like period and sort_by, and confirm which fields you want to use.

Check out the Documentation for examples and additional details.

What We’ll Build

Follow along as we build an API-only Rails 8 app that:

Fetches Naver News Results via SerpApi for your keywords watchlist
Filters results in memory
Emails a single daily digest with your desired number of items
Leverages SerpApi’s free caching to minimize API credit usage

This tutorial demonstrates a concrete example of a lightweight news monitoring service. For a broader overview and other patterns, see SerpApi’s News Monitoring Services use case page.

End Result

Here’s an example of what the email digest will look like when we’re finished:

Prerequisites

Most recent versions of Ruby and Rails (as of the time of this writing):
- Ruby 4.0.0 (newly released December 25, 2025)
- Rails 8.1.1
Bundler 4.0.3
A SerpApi account and API key
- Sign up for free
- Get your API key
SerpApi’s official Ruby Library
Basic familiarity with Ruby on Rails

Sample JSON Result

To see what we’ll be working with, here’s a partial sample of a single item from the news_results array returned by SerpApi’s Naver News Results:

{
  "position": 1,
  "title": "[기고] 애그테크로 여는 전북농업의 대전환, 지금이 골든타임",
  "link": "https://www.jjan.kr/article/20260107500290",
  "thumbnail": "https://search.pstatic.net/common/?src=https%3A%2F%2Fimgnews.pstatic.net%2Fimage%2Forigin%2F5335%2F2026%2F01%2F07%2F311174.jpg&type=ofullfill208_208_dominantcolor&expire=2&refresh=true",
  "news_info": {
    "news_date": "11시간 전",
    "press_name": "전북일보",
    "press_link": "http://www.jjan.kr",
    "press_icon": "https://search.pstatic.net/common/?src=https%3A%2F%2Fmimgnews.pstatic.net%2Fimage%2Fupload%2Foffice_logo%2F5335%2F2024%2F05%2F30%2Flogo_5335_18_20240530182450.png&type=f54_54&expire=24&refresh=true"
  },
  "snippet": "“AI가 설계하고, 데이터가 실행하며, 사람이 완성한다.” 전북이 그 중심에 설 수 있는 시간은 바로 지금이다. 지금이 골든타임이다. 김창길 서울대 아시아연구소 방문학자 · 전 한국농촌경제연구원장 이준서 기자 /이준서 imhendsome@naver.com /이준서"
}

We’ll use the following:

position → the rank of the news article in the results
title → the headline of the news article
link → the URL for the news article
snippet → a brief excerpt from the article
news_info.news_date → human-readable time of when the article was published
news_info.press_name → the name of the outlet
news_info.press_link → URL to the outlet’s page on Naver News

Building the App

1. Create an API-only Rails app

We don’t need a frontend for this app, so we’ll use the API-only template, with no database:

rails new naver_news_watchlist --api --skip-active-record
cd naver_news_watchlist

Optional

If you want to include a database so you can persist news results, simply omit --skip-active-record.

2. Add the SerpApi gem

Next, add the SerpApi gem to your app and install it with:

bundle add serpapi

3. Store your API key

Retrieve your API key from your SerpApi dashboard and store it securely in Rails credentials:

rails credentials:edit

Add:

serpapi:
  api_key: YOUR_API_KEY

Close the file to save it. Now you can securely access your API key anywhere in your app with:

Rails.application.credentials.dig(:serpapi, :api_key)

4. Service: Naver News Fetcher

We’ll isolate all SerpApi interaction into a single service object. This keeps API concerns out of jobs and models, makes testing easier, and gives us a single place to tune default parameters (period, sort_by, etc.) later.

Create the file for the service object:

mkdir app/services
touch app/services/naver_news_fetcher.rb

Then paste this into the file you just created:

class NaverNewsFetcher
  DEFAULT_PARAMS = {
    engine: "naver",
    where: "news",
    sort_by: DailyWatchlistJob::SORT_MAP[:relevance], # Relevance
    period: "1d" # 1 day
  }.freeze

  attr_reader :base_params

  def initialize(query:, **options)
    @base_params = DEFAULT_PARAMS.merge(options).merge(query:)
  end

  def self.call(page: 1, **args)
    new(**args).call(page:)
  end

  def call(page: 1)
    client = SerpApi::Client.new(api_key: serpapi_key)
    client.search(page:, **base_params)
  end

  private

  def serpapi_key
    Rails.application.credentials.dig(:serpapi, :api_key)
  end
end

We’ll create the DailyWatchlistJob with SORT_MAP a bit later. By default, we’re sorting results by relevance, and we’re using period: "1d" to limit results to the last 1 day. You can override these defaults per watchlist item later, or by passing explicit keyword arguments when initializing NaverNewsFetcher. Any other supported SerpApi parameters (e.g., device, no_cache, etc.) can be passed via the options hash and will be merged into the request.

For a full list of supported values for sort_by and period, see the Advanced Filters Documentation for SerpApi’s Naver Search API.

5. Email digest with Action Mailer

Once the news results are fetched, we assemble a short digest and send it somewhere you’ll actually see it. For this example, we’ll send a simple email digest.

First, update your ApplicationMailer in app/mailers/application_mailer.rb:

class ApplicationMailer < ActionMailer::Base
  default from: "Naver News Watchlist <#{Rails.application.credentials.dig(:smtp, :user_name)}>"
  layout false
end

Note

The default from: address will be pulled from your encrypted Rails credentials, which we’ll configure in the next step.

Then generate the daily digest mailer:

rails g mailer daily_digest

Paste the following into the generated app/mailer/daily_digest_mailer.rb:

class DailyDigestMailer < ApplicationMailer
  def digest_email(recipient_email, sections)
    @sections = sections

    digest_date = Time.zone.today.strftime("%F")
    @title = "Naver News Watchlist Digest — #{digest_date}"

    mail(
      to: recipient_email,
      subject: @title,
      charset: "UTF-8", # ensures Korean text is rendered correctly
    )
  end
end

Create a plain-text template:

touch app/views/daily_digest_mailer/digest_email.text.erb

And add:

<%= @title %> (<%= pluralize(@sections.sum { it[:items].size }, "result") %>)

<% sections_list = @sections.map do |section|
  items         = section[:items]
  results_count = items.size

  section_title = "#{pluralize(results_count, 'result')} for \"#{section[:query]}\""

  device = section.fetch(:device, "desktop")
  sort   = DailyWatchlistJob::SORT_MAP.key(section.fetch(:sort_by, 0)).to_s.titleize

  period = section.fetch(:period, "1d")

  period = if (match = period.match(/from(\d{8})to(\d{8})/))
    begin
      from_date, to_date = match.captures.map(&:to_date)
      "from #{from_date} to #{to_date}"
    rescue Date::Error
      'unknown period'
    end
  else
    DailyWatchlistJob::PERIOD_MAP.fetch(period, 'unknown period')
  end

  filtered_by_press = section[:press_names]&.join(', ')

  meta_parts = { device:, sort:, period:, filtered_by_press: }.compact_blank.map { |key, value| "#{key.to_s.humanize}: #{value}" }
  section_details = " (#{meta_parts.join(", ")})"

  punctuation = results_count.zero? ? "." : ":"

  header = [ section_title, section_details, punctuation ].join

  body = items.map do |item|
    [
      "  #{item[:position]}) \"#{item[:title]}\"",
      "    Date:    #{item[:news_date]}",
      "    Link:    #{item[:link]}",
      (
        "    Source:  #{[
          item[:press_name].presence,
          item[:press_link].present? ? " (#{item[:press_link]})" : nil
        ].compact.join}" if item[:press_name].present? || item[:press_link].present?
      ),
      ("    Snippet: \"#{item[:snippet]}\"" if item[:snippet].present?)
    ].compact.join("\n")
  end.join("\n\n")

  [ header, body ].join("\n\n")
end %>

<%= sections_list.join("\n\n\n---\n\n\n") %>

It looks like there’s a lot going on here, but all we’re really doing is iterating through @sections (which are grouped by keyword, i.e., query), formatting each item so it can be printed neatly, and rendering each keyword as its own section. Take a look at the source code on GitHub to see how I extracted much of the formatting logic to a DailyDigestSectionFormatter class.

6. Configure an email delivery method

To keep it simple, we’ll use Gmail SMTP with our credentials stored in Rails encrypted credentials.

Step 1: Generate a Gmail App Password

Gmail does not allow SMTP access using your normal account password. Instead, you must create an App Password.

Go to your Google Account security page: https://myaccount.google.com/security
Ensure 2-Step Verification is enabled (App Passwords are unavailable without it)
Create an App Password: https://myaccount.google.com/apppasswords Assign it a memorable name, like “Rails Naver News Watchlist.”
Google will generate a 16-character password. Copy it or write it down now — you will not be able to see it again.

This password will be used by Rails to authenticate with Gmail’s SMTP server.

Step 2: Store SMTP credentials in Rails credentials

Run:

rails credentials:edit

Add the following:

smtp:
  user_name: YOUR_EMAIL@gmail.com
  password: YOUR_APP_PASSWORD

Replace YOUR_EMAIL@gmail.com with your gmail address and YOUR_APP_PASSWORD with the app password obtained in the previous step. The ApplicationMailer we modified above will automatically use smtp.user_name as the sender address.

Step 3: Configure Action Mailer

Configure Action Mailer to work in production by setting the following in config/environments/production.rb:

config.action_mailer.delivery_method = :smtp

config.action_mailer.smtp_settings = {
  user_name: Rails.application.credentials.dig(:smtp, :user_name),
  password: Rails.application.credentials.dig(:smtp, :password),
  address: "smtp.gmail.com",
  port: 587,
  authentication: :plain,
  enable_starttls_auto: true
}

config.action_mailer.perform_deliveries = true
config.action_mailer.raise_delivery_errors = true

To test our emails locally, we’ll use the letter_opener gem. Run this command to add the gem:

bundle add letter_opener --group development

Then edit config/environments/development.rb:

config.action_mailer.delivery_method = :letter_opener
config.action_mailer.perform_deliveries = true

Now when you try to send emails in development, letter_opener will now automatically open them in your default browser. Next, let’s create the job to actually send the emails.

7. Create the Daily Watchlist Job

Generate the job:

rails g job daily_watchlist

This creates a new job in app/jobs/daily_watchlist_job.rb. This job will coordinate the entire flow:

Fetch news for each watchlist item
Filter duplicates and unwanted press names
Group items by keyword into separate sections
Build a readable digest
Send the email

Paste the following into app/jobs/daily_watchlist_job.rb:

class DailyWatchlistJob < ApplicationJob
  queue_as :default

  RECIPIENT_EMAILS = ["you@example.com", "another@example.com"].freeze # replace with desired emails
  RESULTS_PER_PAGE = 10 # Naver News shows 10 results per page
  MAX_PAGES = 5

  SORT_MAP = {
    relevance: 0, # default
    latest: 1,
    oldest: 2
  }.freeze

  PERIOD_MAP = {
    "all" => "all time", # default
    "1h" => "1 hour",
    "2h" => "2 hours",
    "3h" => "3 hours",
    "4h" => "4 hours",
    "5h" => "5 hours",
    "6h" => "6 hours",
    "1d" => "1 day",
    "1w" => "1 week",
    "1m" => "1 month",
    "3m" => "3 months",
    "6m" => "6 months",
    "1y" => "1 year"
  }.freeze

  WATCHLIST = [
    {
      # Industry trends
      query: "coffee",
      press_names: [
        "매일경제", # Maeil Business Newspaper
        "한국경제", # Korea Economic Daily
        "조선비즈", # Chosun Biz
        "토큰포스트" # TokenPost
      ], # filter by specific press sources
      max: Float::INFINITY, # set max to Float::INFINITY to allow unlimited results up to MAX_PAGES
      sort_by: SORT_MAP[:latest], # sort by latest news
      period: "from20251201to20260105" # custom date range
    },
    {
      # Topical news
      query: "Naver AI",
      press_names: [], # set press_names to `[]` to allow all press sources
      max: RESULTS_PER_PAGE # set max to RESULTS_PER_PAGE to fetch only the first page
    },
    {
      # Competitor monitoring
      query: "Kakao Enterprise",
      press_names: nil, # set press_names to `nil` to allow all press sources
      max: RESULTS_PER_PAGE * 2, # set max greater than RESULTS_PER_PAGE to fetch multiple pages
      sort_by: SORT_MAP[:latest],
      period: "1m",
      device: "tablet"
    },
    {
      # Technology updates
      query: "AI semiconductor",
      # omit press_names to allow all press sources
      # omit max to allow unlimited results up to MAX_PAGES
      sort_by: SORT_MAP[:latest],
      period: "3m"
    },
    {
      # Fresh breaking news with a shorter time frame
      query: "Generative AI",
      max: nil, # set max to `nil` to allow unlimited results up to MAX_PAGES
      sort_by: SORT_MAP[:latest],
      period: "3h",
      device: "mobile" # simulate mobile device results
    }
  ].freeze

  def perform
    sections = sections_for_watchlist
    sections = empty_sections if sections.all? { |section| section[:items].empty? }

    post_digest sections
  end

  def sections_for_watchlist
    WATCHLIST.map { |item| build_section(item) }
  end

  private

  def build_section(item)
    {
      **item.without(:max),
      items: fetch_and_select(item)
    }
  end

  def empty_sections
    [
      {
        query: nil,
        items: []
      }
    ]
  end

  def fetch_and_select(item)
    seen = []
    selected = []

    limit = item[:max] || Float::INFINITY
    page = 1

    while selected.size < limit && page <= MAX_PAGES
      options = item.without(:max, :press_names).merge(page:)

      api_response = NaverNewsFetcher.call(**options)

      results = api_response[:news_results].to_a
      break if results.empty?

      results.each do |result|
        break if selected.size >= limit

        link = result[:link].to_s
        next if link.blank? || link.in?(seen)

        seen << link

        press_name = result.dig(:news_info, :press_name).to_s
        next unless item[:press_names].blank? || press_name.in?(item[:press_names])

        rank = result[:position].to_i + (page - 1) * RESULTS_PER_PAGE # calculate rank across pages

        selected << {
          position: rank,
          title: "result[:title].to_s,"
          snippet: result[:snippet].to_s,
          link:,
          news_date: result.dig(:news_info, :news_date).to_s,
          press_name:,
          press_link: result.dig(:news_info, :press_link).to_s
        }.compact
      end

      page += 1
    end

    selected
  end

  def post_digest(sections)
    DailyDigestMailer.digest_email(RECIPIENT_EMAILS, sections).deliver_later
  end
end

Scheduling

To run the watchlist automatically every day, you can use a scheduler like cron or the whenever gem to run the job at your desired time.

8. Error Handling & Reporting

To ensure you’re notified of any issues during the job execution, we’ll add some simple error handling and email reporting.

Step 1: Add an Error Email Method and Template

First, update app/mailers/daily_digest_mailer.rb to include an error email method:

class DailyDigestMailer < ApplicationMailer
  # existing digest_email method...

  def error_email(recipient_email, error_payload)
    @error_class = error_payload[:class]
    @error_message = error_payload[:message]
    @error_backtrace = Array(error_payload[:backtrace])

    @error_time = Time.zone.now.strftime("%F %R %Z")
    @title = "Naver News Watchlist Error — #{@error_time}"

    mail(
      to: recipient_email,
      subject: @title,
      charset: "UTF-8",
    )
  end
end

Create the corresponding view template app/views/daily_digest_mailer/error_email.text.erb:

<%= @title %>

<%
  lines = [
    "Time: #{@error_time}",
    "Error: #{@error_class} - #{@error_message}",
    "",
    "Backtrace:"
  ]

  backtrace_lines = @error_backtrace.map { " #{it}" }

  body = (lines + backtrace_lines).join("\n")
%>

<%= body %>

Step 2: Update Naver News Fetcher with Error Handling

Update the call method in app/services/naver_news_fetcher.rb to report errors when an error occurs:

class NaverNewsFetcher
  # existing code...

  def call(page: 1)
    client = SerpApi::Client.new(api_key: serpapi_key)
    client.search(page:, **base_params)
  rescue => error
    # Ignore "no results" errors
    return empty_payload if error.message.include? "Naver hasn't returned any results for this query"

    Rails.error.report(
      error,
      handled: false,
      severity: :error,
      context: {
        source: self.class.name,
        page:,
        **base_params.without(:engine, :where)
      }.compact
    )
    raise error
  end

  private

  # existing code...

  def empty_payload
    {
      search_metadata: { status: "Success" },
      news_results: []
    }
  end
end

Step 3: Update Daily Watchlist Job for Error Handling

Update the perform method in app/jobs/daily_watchlist_job.rb to handle exceptions and send error emails:

class DailyWatchlistJob < ApplicationJob
  # existing code...

  def perform
    sections = sections_for_watchlist
    sections = empty_sections if sections.all? { |section| section[:items].empty? }

    post_digest sections
  rescue => error
    Rails.error.report(
      error,
      handled: false,
      severity: :error,
      context: {
        job: self.class.name,
        watchlist_queries: WATCHLIST.pluck(:query),
        recipients: RECIPIENT_EMAILS
       }
    )

    error_payload = {
      class: error.class.name,
      message: error.message,
      backtrace: error.backtrace
    }

    DailyDigestMailer.error_email(RECIPIENT_EMAILS, error_payload).deliver_later
    raise error
  end

  # existing code...
end

Update the fetch_and_select method and add ensure_serpapi_success! in app/jobs/daily_watchlist_job.rb to raise errors when SerpApi responses indicate failure:

class DailyWatchlistJob < ApplicationJob
  # existing code...

  def fetch_and_select(item)
    # existing code...

    while selected.size < limit && page <= MAX_PAGES
      options = item.without(:max, :press_names).merge(page:)

      api_response = NaverNewsFetcher.call(**options)
      ensure_serpapi_success! api_response, options[:query], page # <-- add this line

      results = api_response[:news_results].to_a
      break if results.empty?

      results.each do |result|
        # existing code...
      end

      page += 1
    end

    selected
  end

  def ensure_serpapi_success!(api_response, query, page)
    status = api_response.dig(:search_metadata, :status)
    return if status == "Success"

    error_message = api_response[:error] || "Unknown SerpApi error"
    error = StandardError.new(
      "SerpApi Naver search failed (query=#{query.inspect}, page=#{page}): " \
      "status=#{status || 'unknown'} error=#{error_message}"
    )

    Rails.error.report(
      error,
      handled: false,
      severity: :error,
      context: {
        source: "SerpApi:NaverNews",
        query:,
        page:,
        serpapi_status: status,
        serpapi_error_message: error_message
      }
    )

    raise error
  end

  # existing code...
end

9. Run & Test

You can run the following commands in the Rails console to test each part of the system:

Fetch news results:

results = NaverNewsFetcher.call(query: "coffee")
news_results = results[:news_results]

Generate watchlist sections:

sections = DailyWatchlistJob.new.sections_for_watchlist

Run the full job:

DailyWatchlistJob.perform_now

letter_opener will automatically open the digest email in your default browser. It should look something like this:

Naver News Watchlist Digest — 2026-01-07 (25 results)


2 results for "coffee" (Device: desktop, Sort: Latest, Period: from 2025-12-01 to 2026-01-07, Filtered by press: 매일경제, 한국경제, 조선비즈, 토큰포스트):

  1) Vanadi Coffee, 비트코인 12개 추가 매입…총보유량 173개
    Date: 7시간 전
    Link: https://www.tokenpost.kr/news/breaking/321889
    Source: 토큰포스트 (https://www.tokenpost.kr)
    Snippet: 암호화폐 전문 매체 오데일리(Odaily)에 따르면, 커피 브랜드 Vanadi Coffee가 최근 비트코인(BTC) 12개를 추가로 매입했다. 현재까지 이 회사의 총 비트코인 보유량은 173개로 확대됐다. 이번 매입으로 Vanadi Coffee는 비트코인 보유량 기준 글로벌 기업 순위에서 91위로 올라섰다. Vanadi Coffee는... 

  3) [저녁 뉴스브리핑] 국회, 내달부터 암호화폐 현물 ETF 도입 관련 법안 논...
    Date: 13시간 전
    Link: https://www.tokenpost.kr/news/briefing/321785
    Source: 토큰포스트 (https://www.tokenpost.kr)
    Snippet: 스페인 커피 브랜드 바나디, 173 BTC 보유 중 스페인 커피 브랜드 바나디 커피(Vanadi Coffee)가 1월 6일(현지시간) 기준 173 BTC를 보유하고 있다고 공식 X를 통해 전했다. 바나디는 지난해 커피 사업에서 비트코인 중심 기업으로 전환하겠다고 밝히며, BTC 준비금 조성을 위해 총 11억 달러를 투자할... 


---


10 results for "Naver AI" (Device: desktop, Sort: Relevance, Period: 1 day, Filtered by press: ):

  1) 수자원공사, CES 2026서 글로벌 물관리 AI기술 선봬
    Date: 11시간 전
    Link: https://www.asiatoday.co.kr/view.php?key=20260107010003219
    Source: 아시아투데이 (http://www.asiatoday.co.kr)
    Snippet: 한국수자원공사가 CES 2026에서 국내 물기업들과 함께 물관리 AI기술혁신의 저력을 세계에 알린다.수자원공사는 9일까지 미국 라스베이거스에서... 함께 참가한 국내 물기업 21개 사 중 7개 기업이 CES 최고혁신상 및 혁신상을 수상하며 주목을 받고 있다. 이진희 william614@naver.com

  2) 미래도시 전환…익산시, ‘3+AI’ 전략 본격 추진
    Date: 12시간 전
    Link: https://www.jjan.kr/article/20260107500091
    Source: 전북일보 (http://www.jjan.kr)
    Snippet: 농업·식품·바이오에 AI 성장 엔진 더해 산업 구조 혁신 그동안 성과 바탕으로 미래 대응 및 도시 경쟁력 강화 정헌율 익산시장이 7일 신년 브리핑에서... “올해도 동심동덕(同心同德)의 자세로 시민과 함께 호흡하며 도시의 미래를 만들어 가겠다”고 다짐했다. 익산=송승욱 기자 /송승욱 ssw791221@naver.com /송승욱

  3) 업스테이지 이어 네이버까지…독자 AI 모델 불거진 표절 논란
    Date: 10시간 전
    Link: https://www.mt.co.kr/tech/2026/01/07/2026010718161644740
    Source: 머니투데이 (https://media.naver.com/press/008)
    Snippet: 앞서 업스테이지에 이어 이번에는 네이버(NAVER)다. 네이버가 중국산 모델을 베꼈다는 지적이다. 이와 관련 네이버는 "AI 모델에서 가장 핵심적인 기반은 자체 기술로 개발했다"는 입장을 내놓으며 '프롬 스크래치' 관련 "명확한 기준이 필요하다"고 했다. 7일 IT 업계에 따르면 최근 세계 최대 오픈소스 플랫폼인... 

...

Note

The two results for the “coffee” query are ranked 1 and 3 because other results were filtered out based on the specified press_names in the WATCHLIST.

Where to Go Next

From here, you can easily extend the system:

Set up a scheduler to run the DailyWatchlistJob automatically every day.
Translate the Korean text to another language automatically using a translation API.
Send digests to other destinations (e.g., Slack, Discord) via webhooks.
Enhance filtering with keywords or sentiment analysis to highlight the news that’s most relevant to you.

Even without any of that, this watchlist does exactly what it needs to do — it delivers relevant and timely news results, without distractions.

Build A No Code AI-Powered Local Lead Outreach System

Sonika Arora — Wed, 01 Jul 2026 07:07:22 +0000

Google Maps business listings contain valuable lead data. The challenge is finding the right businesses, qualifying them, and personalizing outreach at scale. What if you could automate the entire process including discovering businesses, generating personalized outreach messages, and sending leads directly to your CRM?

In this tutorial, we'll build a fully automated local lead outreach workflow without using any code with:

Workflow Overview

Example use case we'll work on: Find dentist offices in Miami whom you can sell your monthly subscription based appointment scheduling software to and draft personalized emails to them.

The automation will follow this process:

Extract businesses from Google Maps using SerpApi
Format and send information for each business to Claude to generate personalized outreach
Push lead data along with the personalized message into your CRM (Google Sheets in this example)

The entire workflow uses no code and runs in Make.

Step 1: Setup a Make scenario

If you haven't used SerpApi with Make, I recommend checking this blog post for setup steps:

Announcing SerpApi’s Make App

To begin, in Make, we'll create a new scenario. To make requests to SerpApi, we'll use SerpApi's Make integration.

💡 In case you don't see our Google modules, please use the Universal Module to construct API calls to any of our Google APIs.

Here's the flow we'll setup:

SerpApi Search Google Maps module (or the Universal Module) -> Iterator module -> Claude Simple Text Prompt module -> Google Sheets Add a Row module

The setup in Make

Here's the scenario for this blog post example so you can follow along:

Integrate SerpApi, Claude, Google Sheets - Make.com Automation Scenario

Step 2: Extract Google maps data with SerpApi

If you don't already have a SerpApi account, please register an account. You'll need your API key to set up the SerpApi integration in Make. Once you have an active account, you can find your API key here.

We’ll start by using the SerpApi Google Maps API to search for businesses.

Example query:

“dentists in Miami”
“restaurants in Austin”
“law firms in Chicago”

Example API request:

https://serpapi.com/search.json?engine=google_maps&q=dentists+in+miami&api_key=YOUR_API_KEY

The response includes:

Business names
Ratings
Reviews
Phone numbers
Websites
Coordinates
Categories
Hours
And more

Try out your request in our playground to visualize the result:

SerpApi Playground - SerpApi

With the Search Google Maps Module in Make, it looks like this:

Search Google Maps Module Setup in Make

💡 In case you don't see our Google modules, please use the Universal Module to construct API calls to any of our Google APIs.

With the Universal Module, it looks like this:

SerpApi's Universal Module Setup in Make

This gives you structured local business JSON data directly from Google Maps.

Step 3: Iterate through businesses

Google Maps results return multiple businesses matching the query. We'll use the Iterator module in Make to process each business individually.

To begin with, we'll focus on a few super useful fields found in the local results array like:

title
website
phone
rating
reviews
address

Iterator Module Setup in Make

To extend this solution, you can also filter leads by specific things like:

Only businesses with ratings below 4.0
Only businesses without websites
Only businesses with less than 50 reviews

This is especially useful for targeted prospecting.

Step 4: Generate personalized outreach with Claude

Now we connect Claude in Make. This step transforms raw lead data into personalized and actionable outreach.

Claude Module Setup in Make

💡 I chose the model Claude Opus 4.7. Feel free to change it according to your use case.

Here's the prompt I used:

Write a short cold outreach email for this business.

Details about the business from Google Maps: 
Title: {{title}}
Type of business:{{type}}
Rating: {{rating}}
Reviews: {{reviews}}
Website: {{website}}
Address: {{address}}
Phone: {{phone}}
User review from Google: {{user_review}}

For each entry, draft a personalized email to reach out and ask if they need an appointment scheduling software. I own a company called XYZ and we sell a monthly subscription based appoiintment scheduling software. Link my website called xyz.com.

Sample output for one of the dental offices:

Subject: Helping Relax and Smile Dental Care Save Time on Scheduling

Hi Relax and Smile Dental Care Team,

Congratulations on building such an outstanding reputation in Miami — a 4.9 rating with 883 reviews is no small feat! I noticed patients consistently rave about your efficiency, with one recent review highlighting "no wait time to see the hygienist" and a cleaning done in just 30 minutes. That kind of streamlined experience is exactly what keeps patients coming back.

My name is [Your Name], and I'm the founder of XYZ — a monthly subscription-based appointment scheduling software designed specifically for busy practices like yours. Our platform helps dental offices reduce no-shows, automate reminders, and let patients self-book online 24/7, freeing up your front desk to focus on in-person care.

Given how much your patients value your quick, gentle service, I think XYZ could help you maintain that pace as your practice continues to grow.

Would you be open to a quick 15-minute call this week to see if it's a fit? You can also learn more at xyz.com.

Looking forward to hearing from you,

[Your Name]

Founder, XYZ

xyz.com

Step 5: Push leads into a Google sheet

Finally, connect Google Sheets using the Google Sheet Integration in Make.

We'll note these fields in this blog post, but feel free to add any more you need to keep track of:

Business name
Website
Phone
Address
AI-generated outreach message

Create a sheet named "Dentist offices Make.com automation sheet" and then add the relevant details to the Google Sheets Add a Row module.

Google Sheets Add a Row Module Setup in Make

You can also do more with the data later like adding lifecycle stages in the sheet and/or triggering email sequences.

At this point, we have a fully automated AI-powered lead generation system.

Here's what the output looks like:

Output in Google Sheets

Conclusion

I hope it was useful to see how we can use SerpApi with Make to build a workflow that automatically discovers businesses, gets structured data about them, generates personalized outreach using Claude, and pushes qualified leads directly into your CRM (Google Sheets in this example).

Here's the Make scenario we created in this blog post:

Integrate SerpApi, Claude, Google Sheets - Make.com Automation Scenario

I hope you found this useful. If you have any questions, feel free to reach out at contact@serpapi.com.

Relevant Links

Turn Bing News into an AI Briefing with Claude and SerpApi

Sonika Arora — Tue, 30 Jun 2026 07:06:28 +0000

Most news apps and search engines already present headlines well, so simply listing them is becoming less valuable. The real value has shifted to connecting the dots: grouping related news stories, identifying patterns across sources, and explaining what actually matters and why.

In this post, we’ll build a simple pipeline that turns raw Bing News results into a structured daily briefing which does exactly that using:

SerpApi's Bing News API for real-time news data
Claude for summarization and insight generation

Setting Up

LLMs work best when you give them clean, formatted context instead of raw scraped HTML, so we'll do exactly that.

Get results using SerpApi (Bing News API) → Normalize → Send to Claude with a relevant prompt → Get structured briefing

For this tutorial, we're going to use SerpApi's new Python library to get results from search.

Install SerpApi's new Python serpapi library, the python-dotenv library (for using environment variables) and anthropic library (for Claude) in your environment:

pip install serpapi python-dotenv anthropic

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API key from this page.
[Optional but Recommended] Set your API keys in an environment variable, instead of directly pasting them in the code. Refer here to understand more about using environment variables.

SERPAPI_API_KEY=
CLAUDE_API_KEY=

Get your SerpApi API key here: https://serpapi.com/manage-api-key

Get your Claude API key here: https://platform.claude.com/settings/keys

💡 You'll need to create an account with Claude in case you don't have one yet. You can do that here: https://claude.ai/login

Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import json
import anthropic
from dotenv import load_dotenv
load_dotenv()

SERPAPI_API_KEY = os.environ["SERPAPI_API_KEY"]
CLAUDE_API_KEY = os.environ["CLAUDE_API_KEY"]

Fetch News with Bing News API

We’ll write a function to use SerpApi’s Bing News API (bing_news engine) and get all the news results.

def fetch_news(query):
    client = serpapi.Client(api_key=SERPAPI_API_KEY)
    results = client.search({
        "engine": "bing_news",
        "q": query,
        "device": "desktop",
        "mkt": "en-au"
    })
    return results.get("organic_results", [])

We can call this function with any query such as "Self driving software" or "AI" and scrape results from Bing News.

You can test the Bing News API out in our playground with parameters of your choice:

SerpApi Playground - SerpApi

Prompting Claude

Good prompts can make all the difference when building a AI briefing based on the results.

A bad prompt would look like:

“Summarize these articles”

Better prompts often define structure, force grouping, and require reasoning.

def build_prompt(articles):
    return f"""
You are a technical news analyst.

Task: Convert raw news articles into a structured daily briefing.

Requirements:
- Group related articles into 3-5 themes
- Each theme must have:
  - A short title
  - A 2-3 sentence summary
  - A "Why it matters" explanation
- Be concise and non-generic
- Avoid repeating the same point across themes

Articles:
{articles}
"""

Generate the Briefing (Claude API)

Using Claude's API, we can turn the data we got from SerpApi into a briefing. Here is some simple code to use Anthropic API -

def generate_briefing(prompt):
    client = anthropic.Anthropic(api_key=CLAUDE_API_KEY)
    response = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=1500,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

Putting It Together

Now that we have all the pieces we'll use, let's write the code to call the different functions and get our news briefing:

def run():
    query = "AI"
    articles = fetch_news(query)
    prompt = build_prompt(articles)
    briefing = generate_briefing(prompt)
    print(briefing)

if __name__ == " __main__":
    run()

Now, we're ready to run this file. We can do so by typing python <filename> in the terminal.

Automated Delivery

A briefing is only useful if it reaches you regularly automatically. Add a delivery layer like automated daily email or a slack webhook.

Automated Daily Email

Simple automation run daily via cron or other scheduling methods
Email formatted briefing to yourself or a team

Slack Webhook

Post structured sections into a channel
Good for teams or shared awareness feeds
Enables discussion around themes

Read this blog post for more info on setting up a Slack webhook (The use case in the post is different, but the process remains the same). It also covers how to schedule a job every day using the Python schedule library:

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Generated AI Briefing Example Output

Here's an example run. In my SerpApi call, I set the query to "AI" to keep things simple. I used these parameters for Anthropic API:

model="claude-opus-4-7",
max_tokens=1500,
messages=[{"role": "user", "content": prompt}]

Here's the Claude generated briefing:

Daily AI Briefing

1. AI Reshapes the Workforce: Layoffs, Skill Demands, and Survival Rules

Cloudflare announced a 20% workforce cut (1,100 employees) citing agentic AI as a fundamental disruptor of its operations, sending its stock down 18%. Meanwhile, Airbnb CEO Brian Chesky warned that "pure people managers" and change-resistant workers won't survive the AI era, and CBS reports employers are increasingly demanding AI fluency from new hires.

Why it matters: This is one of the first major tech layoffs explicitly attributed to agentic AI rather than macro conditions—signaling a shift from AI-as-productivity-tool to AI-as-headcount-replacement. Workers and managers face mounting pressure to reskill or face displacement.

2. Conflicting Narratives on AI's Human Impact

Nvidia's Jensen Huang publicly framed AI as a net job creator, urging workers not to fear the technology. In contrast, a new study found that just 10 minutes of AI use measurably impaired cognitive performance on arithmetic and reasoning tasks.

Why it matters: The optimistic vendor narrative (Huang) is increasingly at odds with both labor-market evidence (Cloudflare) and emerging cognitive research. Expect this tension to drive policy debates and shape enterprise adoption guardrails.

3. AI Stocks Diverge as Earnings Season Tests the Hype

SoundHound AI beat Q1 expectations with 51.7% revenue growth but still dropped 12.4%, while Cloudflare cratered 18% post-earnings. Analysts are also reframing the Nvidia-vs-Palantir debate for 2026 as both stocks face renewed scrutiny.

Why it matters: Markets are no longer rewarding AI exposure alone—even strong top-line growth isn't enough. Investors are demanding margin discipline and clear AI monetization paths, marking a maturation point for the sector.

4. Military AI Moves From Theoretical to Operational

The UK's Royal Air Force chief confirmed AI-powered uncrewed fighter aircraft are already being integrated into operations, far ahead of prior timelines. Simultaneously, CNN reports the Pentagon faces growing scrutiny over the legal limits of AI in warfare, including its use in Iran-related operations.

Why it matters: Autonomous combat systems have crossed from R&D to deployment faster than legal and ethical frameworks can adapt. The gap between operational reality and governance is becoming a strategic vulnerability for Western militaries.

5. The Consumer AI Landscape Crystallizes

TechRepublic published a side-by-side comparison of the major chatbots—ChatGPT, Gemini, Copilot, Claude, Perplexity, Grok, DeepSeek, and Meta AI—mapping each to specific use cases and integrations.

Why it matters: With eight serious contenders now in the consumer/prosumer market, differentiation is shifting from raw model quality to ecosystem integration and vertical specialization. The "one assistant to rule them all" thesis is dead; multi-tool workflows are the new norm.

Building On Top Of This

Changing The Model

As of May 2026, here are the options Claude provides and what each is best for:

Opus (Most Intelligent): Best for complex tasks, deep reasoning, coding, and strategic analysis. It is the most powerful but slower and more expensive.
Sonnet (Balanced): It provides the best balance of speed, intelligence, and cost, making it ideal for daily coding, data analysis, and content generation.
Haiku (Fastest): Optimized for speed and cost-efficiency. It is perfect for high-volume tasks, quick responses, and simple, repetitive jobs.

You can test the script with different models and figure out the best one for your use case. I used Claude Opus 4.7 for this demo.

Persisting History

Instead of generating standalone briefings, store each run with a timestamp. This turns your system into a time-series of news summaries. Once you have history, you can:

Compare today vs yesterday (what’s new, what disappeared)
Track theme momentum (what’s increasing or fading)
Ask Claude for delta summaries instead of fresh-only summaries

Example prompt changes:

“Compare today’s briefing with yesterday’s. Highlight new, removed, and shifting themes.”

Conclusion

The key takeaway here is that LLMs work best when they’re grounded in high-quality external data. In this project, SerpApi provides structured, real-time Bing News data, while Claude turns that raw information into something readable and useful: a concise daily briefing with themes, summaries, and context.

Instead of asking a model to know everything, we can give it fresh information and let it focus on synthesis and reasoning.

I hope you found this useful. If you have any questions, feel free to reach out at contact@serpapi.com

Relevant Links

How To Scrape Home Depot Product Reviews

Sonika Arora — Mon, 29 Jun 2026 07:05:48 +0000

Customer reviews are one of the most valuable sources of product insight. They reveal real-world feedback, highlight product strengths and weaknesses, and help businesses understand customer sentiment.

In this post, we'll explore how to collect product reviews from Home Depot using SerpApi's Home Depot Product Reviews API. By the end, you'll be able to fetch product reviews data such as ratings, titles, and comments for analysis and monitoring.

Why Scrape Reviews?

Home Depot is one of the largest home improvement retailers in the United States. Product reviews on its site provide valuable insights for:

Competitive analysis — Understand how competing products are perceived.
Sentiment analysis — Identify common praise or complaints.
Market research — Learn what customers value most.
Product development — Discover improvement opportunities.

Instead of manually collecting reviews, SerpApi lets you retrieve them programmatically and with many no code options.

What Type of Data Can You Extract?

The Home Depot Product Reviews API scrapes reviews for products from Home Depot and provides them in JSON format. Here are some prominent fields from the review object:

id — Unique identifier for the review.
title — Short headline summarizing the reviewer’s opinion.
text — Full written content of the customer review.
rating — Numeric score given by the reviewer (typically on a 1–5 scale).
badges — Indicates badges on products like verifiedPurchaser.
reviewer.name — Display name of the person who wrote the review.
time — Timestamp showing when the review was published.
original_source.name — Website where the review originally appeared.
images.link — URLs of photos included in the review.
total_positive_feedback — Number of users who found the review helpful.
total_negative_feedback — Number of users who found the review unhelpful.

Read the documentation page here:

The Home Depot Product Reviews API - SerpApi

Here's an example in our playground:

SerpApi Playground - SerpApi

Setting up SerpApi

Are you new to SerpApi? We'll walk you through the steps from sign up to integrating with SerpApi.

Getting Started with SerpApi

To get started with SerpApi, I recommend following the steps in this blog post:

Getting started with SerpApi: The Web Search API

Home Depot Product Reviews API Documentation

We recommend familiarizing yourself with the available parameters and details in the documentation here:

The Home Depot Product Reviews API - SerpApi

Integrations / Libraries

Some examples below will include our SerpApi libraries. If you would like to use one of our SerpApi libraries, we offer instructions for each language, such as Python, Node.js, Ruby, PHP, Java, Go, and more:

SerpApi: Integrations

Scraping Home Depot Product Reviews

You can use any programming language in order to call our APIs. The parameters and results we reviewed above will not differ between the request methods, however the syntax used to call the APIs may differ.

You will see across the examples the engine is always set to home_depot_product_reviews, the api_key will be set to your API key, and we include the product_id in every query. In this example we are going to look for reviews for product with ID: 302752040

GET Request

The syntax for the actual GET request between languages will vary between programming languages. However, the URL used each time is consistent. When you are sending a GET request in any language you will use this URL to make the request:

https://serpapi.com/search.json?engine=home_depot_product_reviews&product_id=302752040

Feel free to explore more parameters here, and add any that are relevant to your use case.

An Example With Our Python Library

Install SerpApi's new Python serpapi library in your environment:

pip install serpapi

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

Now, let's take a look at the code:

import serpapi

client = serpapi.Client(api_key=<YOUR_API_KEY>)

results = client.search({
  "engine": "home_depot_product_reviews",
  "product_id": "302752040"
})

Replace the placeholder with your API key.

An Example With cURL

curl --get https://serpapi.com/search \
 -d api_key="YOUR_SECRET_API_KEY" \
 -d engine="home_depot_product_reviews" \
 -d product_id="302752040"

No Code Options

Google Sheets

To connect your code with Google Sheets first follow this blog post:

Connect SERP API with Google Sheets (No Code)

=SERPAPI_RESULT("engine=home_depot_product_reviews&product_id=302752040", "reviews.0.text")

Make.com

While not every API we offer is on Make.com, there is a Universal Module where you can use any API.

Review this blog post to get started:

Announcing SerpApi’s Make App

N8N

Similar to Make.com, not every API is currently included in N8N. However, you can use the HTTP Request node to still call any of our APIs.

To get started review this blog post:

Boost Your n8n Workflows with SerpApi’s Verified Node

Conclusion

Scraping product reviews can unlock valuable insights about customer behavior and product performance. With the Home Depot Product Reviews API, you can retrieve structured review data in just a few lines of code.

I hope you find this useful. If you have any questions, feel free to reach out at contact@serpapi.com

Relevant Links

Related Blogs

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Sonika Arora — Sun, 28 Jun 2026 07:05:02 +0000

Monitoring when competitors begin ranking for new keywords can reveal product launches and marketing strategy changes. Instead of manually checking search results every day, you can automate the process with a Slack bot that sends alerts whenever a new company appears for keywords that matter for your brand.

In this tutorial, we’ll build a simple system that:

Checks search results using the SerpApi's Google Search API.
Detects when a competitor domain appears for a tracked keyword and sends an alert to a Slack channel.
Run this process everyday using Python's schedule library.

Setting Up

This system will work in four steps:

Send in requests to SerpApi's Google Search API for a list of relevant queries
Extract all indexed pages
Compare them with previously stored URLs
Send alerts to Slack for new pages detected

We'll run this job every day using the Python schedule library. You can set the schedule as you like.

To start, you’ll need:

A SerpApi account
A Slack workspace (and a Slack Incoming Webhook URL)
Python installed

Create a Slack webhook

Go to Slack → Apps
Search for Incoming Webhooks
Create a webhook for your channel

You will get a Webhook URL like: https://hooks.slack.com/services/XXXX/XXXX/XXXX

Save this, as we'll need it later.

Code related setup steps

For this tutorial, we're going to use SerpApi's new Python library to get results from search.

Install SerpApi's new Python serpapi library, the python-dotenv library, tldextract library (for parsing the domains from URLs), and the schedule library in your environment:

pip install serpapi python-dotenv schedule tldextract

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.
[Optional but Recommended] Set up your Slack Webhook URL as an an environment variable as well. In your .env file, this will look like this:

SERPAPI_API_KEY=
SLACK_WEBHOOK_URL=
Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import requests
import json
import schedule
import time
import tldextract
from dotenv import load_dotenv
load_dotenv()

SERPAPI_KEY = os.environ["SERPAPI_API_KEY"]
queries = [] # List of queries for which you want to find the updated domains
DB_FILE = "known_domains.json"
SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK_URL"]

Provide the value of your slack webhook in the SLACK_WEBHOOK variable. It looks something like this: https://hooks.slack.com/services/XXXX/XXXX/XXXX.

Provide a list of queries you're looking to track competitors with. Example: ["ai images tool", "ai image generators", "image creation ai tools"]

Get all indexed domains for a list of queries

For the purpose of this post, we'll consider the first 30 organic results and use a simple list of 3 queries. You can increase or decrease this number as needed and change the queries as well depending on your use case.

A simple function to get the root domain from the URL using the tldextract library we installed earlier:

def get_root_domain(url):
    extracted = tldextract.extract(url)

    # Join the domain and suffix to get the root domain
    return f"{extracted.domain}.{extracted.suffix}"

Now, let's create a script com get_domains.py where we use SerpApi 's Google Search API to get the domains of the first 30 search results for a particular query.

def get_all_domains(query_list):
    domains_for_each_query = {}
    client = serpapi.Client(api_key=SERPAPI_KEY)

    for query in query_list:
        domains_for_each_query[query] = set()

        for start in range(0, 30, 10):
            results = client.search({
                "engine": "google",
                "google_domain": "google.com",
                "q": query,
                "start": start
            })

            if "organic_results" not in results:
                break

            for result in results["organic_results"]:
                link = result.get("link")
                if not link:
                    continue

                root_domain = get_root_domain(link)
                domains_for_each_query[query].add(root_domain)

    return {k: list(v) for k, v in domains_for_each_query.items()}

This function sends a request to SerpApi's Google Search API and collects the root domains from the organic result links, and stores them per query. It uses a set to keep only unique domains and fetches up to the first 30 results (pagination with start). It returns a dictionary mapping each query to a list of unique domains found in the search results.

💡 If you want to dive deeper into how to scrape Google search results with Python, read this post.

Save known domains

For the purpose of this example, let's assume the file known_domains.json stores all previously stored pages in JSON format. So, we now need to compare with the domains in this file to find newly added ones.

💡 If you are starting from scratch, using the scripts here will pull all the domains because the known_domains.json file will be empty. Next time, when you use the script, you'll only see the new ones added.

Let's write some code to load the URLs from the file and save the new URLs to the file which we can use later on:

def load_domains():
    try:
        with open(DB_FILE) as f:
            return dict(json.load(f))
    except Exception:
        return {}


def save_domains(domains):
    with open(DB_FILE, "w") as f:
        json.dump(domains, f)

Send slack alerts

Let's write a simple function to send slack alerts for any new domains seen for each query. We'll use this later.

def send_slack_alert(domains):

    # Filter out keys with no new values
    domains = {k: v for k, v in domains.items() if v}

    if not domains:
        message = {
            "text": "✅ No new companies detected for any query."
        }
        requests.post(SLACK_WEBHOOK, json=message)
        return
    for query in domains:
        if domains.get(query) != []:
            message = {
                "text": f"🚨 New company detected for '{query}':\n" +
                        "\n".join(domains[query])
            }
            print(message)
            requests.post(SLACK_WEBHOOK, json=message)

Once everything is set up, your Slack channel will receive alerts like this once a day if new domains detected for any query:

🚨 New company detected for 'image creation ai tools':

company1.com

company2.com

Detect new pages

Now let's work on code to specifically detect new pages by looking at already stored ones in known_domains.json.

def check_for_new_domains():
    print("Checking for new domains...")

    known_domains = load_domains()
    current_domains = get_all_domains(QUERY_LIST)
    new_domains = {}

    for query in current_domains:
        new_domains[query] = list(
            set(current_domains[query]) - set(known_domains.get(query, []))
        )

        known_domains[query] = list(
            set(known_domains.get(query, [])) | set(current_domains[query])
        )

    send_slack_alert(new_domains)
    save_domains(known_domains)

This function checks the search results for new domains appearing for each query compared to previously stored domains. It computes the difference between current domains and known domains, sends a Slack alert for any new ones, and then updates the stored list with the latest domains.

Run the bot on a schedule

Now the only thing left is to schedule this so it runs once everyday.

To do this, we'll use Python's schedule library.

# Schedule the bot to run every day at 9 AM
schedule.every().day.at("09:00").do(check_for_new_domains)
print("Query domain monitoring bot running...")

while True:
    schedule.run_pending()
    time.sleep(60)

To keep the tasks running even after you close your terminal or IDE, you can choose to run the script as a persistent background process. For linux/macOS, use nohup or disown to detach the script from the terminal, e.g., python3 script.py & disown.

What the alert looks like

I ran the code twice with QUERY_LIST = ["ai images tool", "ai image generators", "image creation ai tools"]

The first time, I got some new pages (because a few new companies started ranking for the queries since when I ran it last)

The next time, since all the webpages were already seen, I got a message letting me know that no new pages were detected.

These were the alerts I received:

Now your team will instantly know when new companies start ranking for important keywords that matter for your brand.

Possible Enhancements

Once the basic setup works, you can extend it for a couple more use cases:

Track ranking changes : Alert when a domain jumps from page 2 -> page 1
Monitor multiple regions : Use different location parameters
Track different features : Monitor if competitors appear in other parts of SERP like featured snippets, shopping results, and knowledge panels

Conclusion

By combining SerpApi, Slack, and a Python script, you can build a lightweight monitoring system that continuously tracks competitor activity in search.

With this, you’ll have a practical developer tool that saves hours of manual SERP monitoring and helps your team react faster to competitive changes. You can also use this tutorial to setup any other recurring tasks and send notifications to the slack channel.

You can find the entire code here:

GitHub - serpapi/slack-alerts-for-new-company-ranking: In this tutorial, we’ll build a simple system that detects when a new brand ranks for a tracked keyword and send an alert to a Slack channel.

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Build a Slack Bot That Alerts You When Your Competitor Launches a New Page

Sonika Arora — Sat, 27 Jun 2026 07:04:07 +0000

If you're tracking competitors, one of the most valuable signals is when they launch a new page and it shows up in search results. New pages that companies add often reveal:

New product launches
New feature pages
New landing pages targeting keywords
New content marketing strategies

Manually checking for changes can take quite some time, and repetitive work as pages are updated often. Instead of manually checking competitor sites, you can automate this process using:

SerpApi (Google Search API)
Slack Webhook
Python's schedule library

In this tutorial, we’ll build a simple system that checks search results using the SerpApi's Google Search API to detect when a new competitor domain URL is noticed in search results and then sends an alert to a Slack channel.

Setting Up

This system will work in four steps:

Send in a request to SerpApi's Google Search API with a relevant query
Extract all indexed pages
Compare them with previously stored URLs
Send alerts to Slack for new pages detected

We'll run this job every day using the Python schedule library. You can set the schedule as you like.

To start, you’ll need a SerpApi account, a Slack workspace (and a Slack Incoming Webhook URL), and Python installed.

Create a Slack Webhook

Go to Slack → Apps
Search for Incoming Webhooks
Create a webhook for your channel

You will get a Webhook URL like: https://hooks.slack.com/services/XXXX/XXXX/XXXX

Save this, as we'll need it later.

Code Related Setup Steps

For this tutorial, we're going to use SerpApi's new Python library to get results from Google Search.

Install SerpApi's new Python serpapi library, the python-dotenv library, and the schedule library in your environment:

pip install serpapi python-dotenv schedule

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.
[Optional but Recommended] Set up your Slack Webhook URL as an an environment variable as well. In your .env file, this will look like this:

SERPAPI_API_KEY=
SLACK_WEBHOOK_URL=
Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import requests
import json
import schedule
import time
from dotenv import load_dotenv
load_dotenv()

SERPAPI_KEY = os.environ["SERPAPI_API_KEY"]
DOMAIN = "company.com"
DB_FILE = "known_urls.json"
SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK_URL"]

Fetch All Indexed Pages For Competitor

Let's create a script competitor_page_monitor.py where we use SerpApi 's Google Search API to get all pages which show up in search results for a particular company.

def get_all_pages():
   urls = set()
   client = serpapi.Client(api_key=apikey)

   for start in range(0, 100, 10):

       results = client.search({
           'engine': 'google',
           "q": f"site:{DOMAIN} -site:forum.{DOMAIN} -site:{DOMAIN}/blog -site:{DOMAIN}/careers -site:{DOMAIN}/release-notes",
           "start": start,
       })

       if "organic_results" not in results:
           break

       for result in results["organic_results"]:
           urls.add(result["link"])

   return urls

💡 You may need to add more customized search operators to your query if you want to ignore specific types of pages. For this example, I'm using site:company.com -site:forum.company.com -site:company.com/blog -site:company.com/careers -site:company.com/release-notes" to ignore the blog, forum and career pages.

Conversely, you can also choose to target specific areas such as blogs and get notified what exactly your competitors are writing about publicly.

Feel free to change the maximum pagination as you'd like. I've set start to a max value of 100 which means we'll see data from up to 10 pages (since each page usually has 10 results).

Now that we have all the pages for that domain, we need to find the newly added ones since our last check.

💡 If you want to dive deeper into how to scrape Google search results with Python, refer to this post.

Save Known URLs

For the purpose of this example, let's assume the file known_urls.json stores all previously stored pages in JSON format. So, we now need to compare with the URLs in this file to find newly added ones.

💡 If you are starting from scratch, using the scripts here will pull all the URLs for the competitor because the known_urls.json file will be empty. Next time, when you use the script, you'll only see the new ones added.

Let's write some code to load the URLs from the file and save the new URLs to the file which we can use later on:

def load_urls():
    try:
        with open(DB_FILE) as f:
            return set(json.load(f))
    except:
        return set()

def save_urls(urls):
     with open(DB_FILE, "w") as f:
          json.dump(list(urls), f)

Send Slack Alerts

Let's write a simple function to send slack alerts. We'll use this later.

SLACK_WEBHOOK = "YOUR_SLACK_WEBHOOK"

def send_slack_alert(urls):
   if urls:
       message = {
           "text": f"🚨 New competitor pages detected: \n{'\n'.join(urls)}"
       }
   else:
       message = {
           "text": "✅ No new competitor pages detected."
       }
   requests.post(SLACK_WEBHOOK, json=message)

💡 Provide the value of your slack webhook in the SLACK_WEBHOOK variable. It looks something like this: https://hooks.slack.com/services/XXXX/XXXX/XXXX.

Detect New Pages

Now let's work on code to specifically detect new pages by looking at already stored ones in known_urls.json.

def check_new_pages():
    print("Checking for new pages...")

    known_urls = load_urls()
    current_urls = get_all_pages()
    new_pages = current_urls - known_urls

    send_slack_alert(new_pages)

    new_seen_set = known_urls.union(current_urls)
    save_urls(new_seen_set)

Run the Bot on a Schedule

Now the only thing left is to schedule this so it runs once everyday.

To do this, we'll use Python's schedule library.

💡 I'm using Python's schedule library instead of cron because it works reliably on macOS and local environments

schedule.every().day.at("09:00").do(check_new_pages)
print("Competitor monitoring bot running...")

while True:
    schedule.run_pending()
    time.sleep(60)

What The Alert Looks Like

Once this is set up, your Slack channel will receive alerts like this once a day:

🚨 New competitor pages detected:

https://competitor.com/products/ai-assistant

https://competitor.com/features/fast-mode

I ran the code twice for the domain serpapi.com. The first time, I got some new pages and the next time, since all the webpages were already seen, I got a message letting me know that no new pages were detected. These were the alerts I received:

This lets you instantly spot new product launches, new feature releases, and new landing pages which are showing up in search results for any company.

Conclusion

With about 50 lines of Python, you now have a bot that monitors all indexed pages of a competitor, detects new launches, and sends instant Slack alerts.

This is one of the simplest ways to build a competitive intelligence system using search data. You can also use this tutorial to setup any other recurring tasks and send notifications to the slack channel.

You can find the entire code here:

GitHub - serpapi/slack-alerts-for-new-competitor-pages: In this tutorial, we’ll build a simple system that uses SerpApi’s Google Search API to detect when a new competitor domain URL appears in search results and then sends an alert to a Slack channel.

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Scrape and Summarize App Reviews from the Apple App Store

Catherine Li — Fri, 26 Jun 2026 07:02:59 +0000

AI coding tools have made building a mobile app more accessible than ever, but getting it into the App Store is only half the battle. The real challenge is finding product-market fit before you invest weeks of work and vibe-coding into the wrong idea.

That usually means hours of manual research: digging through App Store listings, scrolling review after review, trying to spot patterns in what users love, hate, and desperately wish existed. What gaps are worth pursuing? What language are real users actually using to describe their pain points?

Today, I want to show you a way to fetch all the reviews for a mobile app in the Apple App Store with only a few lines of JavaScript code and a couple of API calls to SerpApi. Since it’s JSON data that you can use anywhere, we’ll even pass along the reviews to OpenAI's large language model so we can get a summary of the reviews.

When building AI-powered applications that rely on recent web data as a review aggregator, you need a reliable data scraping solution. SerpApi leads the market in terms of both reliability and accuracy.

Scraping Apple App Store Search with Python

💡 Before we get started, make sure you’ve created a free account on SerpApi so you can get access to 250 free search credits. You may also need to sign up and create a developer account to receive an API key from OpenAI.

For this demo, we're also going to use OpenAI's latest large language model – gpt5.4 – to summarize our reviews.

Setting up your project

First, set up a basic Node.js project. You can do so by running npm init -y in your terminal at the root of your project folder. At the root of the project, create an index.js file.

Install required packages

For this demo, it's highly recommended you install and import into index.js the following packages:

dotenv: Node package that loads environment variables from your .env file directly into process.env so you can access your API keys
serpapi: SerpApi's open-source JavaScript SDK
openai: OpenAI's SDKto interface with their AI model

As per the OpenAI documentation online, you'll also need to create an instance of the OpenAI client to make the API calls:

import OpenAI from "openai";
import dotenv from "dotenv";
import { getJson } from "serpapi"
dotenv.config();

//OpenAI API setup
const openAiKey = process.env.OPEN_AI_KEY;
const client = new OpenAI({ apiKey: openAiKey });

//SerpApi setup
const serpApiKey = process.env.SERP_API_KEY;

Making the API call to SerpApi's Apple App Store API

Now, let’s make a call to SerpApi to get the app information, including its ID. We can just use fetch, but since we’ve imported the SDK, we'll use the getJson from the SerpApi library. And just like fetch, getJson is an asynchronous function that returns a JavaScript Promise. The API has three required parameters: API key , the search term which is the app you want to fetch data for, and the engine which in this case is apple_app_store since app data is what we want to scrape today.

const app = await getJson({
        api_key: serpApiKey,
        engine: "apple_app_store",
        term: "instagram",
        num: "5"
    });

Easily scrape Apple App Store and filter results by categories for better insights

Testing your API calls

If you console.log app, you should see JSON that resembles this in your terminal:

💡 For the sake of simplicity, we are not implementing validation, error handling or other guardrails that would be highly recommended should you launch this in production

Inside the organic_results array is where you'll find the search results for the query "instagram". As expected, the first item in this array pertains to the Instagram app owned by Meta. We simply need to pass in the id value into the second API call, which will be to SerpApi's Apple App Store Reviews API.

const appId = app?.organic_results[0]?.id;
    appReviews = await getJson({
        api_key: serpApiKey,
        engine: "apple_reviews",
        product_id: appId
    });

If you console.log(appId) and scroll down to the reviews array, you'll see the text of the reviews within the array objects' text property. Now, we just need to parse out the text reviews and send them to OpenAI to get the summary.

Making the API call to OpenAI

let response;
try {
    response = await client.responses.create({
        model: "gpt-5.4",
        input: `Given the list of comma-separated reviews for a mobile app, give me a summary of the main criticisms: ${textReviews.join(", ")}`,
    });
} catch (err) {
    console.error("Failed to get response from OpenAI:", err.message);
    process.exit(1);
}

The above example is a basic API call to OpenAI's Responses API to generate text from a prompt, similar to the way you'd generate text from OpenAI's ChatGPT.

If everything went according to plan, you should be able to see the summary in your terminal:

Next Steps

When running this in your local terminal, don't be alarmed if you don't see any response for a few seconds. As you may know, large language models tend to take some time to generate text. While I'm not doing so in this demo, there's a way to stream the response, which will allow the text to appear chunks at a time so your users are not left waiting until the entire response is complete. Make sure to give streaming a shot if you're building an AI-powered app that relies on timely responses. I hope you've found this post helpful! If you have any questions at all, leave me a comment below or contact us at contact@serpapi.com.

Apple App Store Reviews Scraper API - SerpApi

Get AI Generated Responses From Search Engines In Structured JSON

Sonika Arora — Fri, 26 Jun 2026 07:02:48 +0000

Search engine results are no longer just a list of links. Many search engines like Google and Bing surface AI generated answers directly in the results page - in summaries, conversational responses, and synthesized insights that sit above or inside traditional SERP features. For developers and data teams, this raises an interesting question: How can you programmatically collect and analyze these AI answers the same way you do classic SERP data?

With SerpApi you can scrape these generated answers without needing to write brittle scraping logic.

In this post we'll walk through how AI generated answers appear across major search engines and how SerpApi exposes them in structured JSON.

Why scrape AI generated responses?

AI generated answers have quickly become the first thing users often see. Scraping them enables:

Generative engine optimization (GEO) & content research - Understand what content AI models consider authoritative and trustworthy, and track how your pages appear in AI-generated answers.
Search experience monitoring - Track how AI summaries describe your brand or product.
Competitive analysis - See which competitors are mentioned or cited.
LLM grounding - Compare AI answers across engines for factual drift.
Product intelligence - Detect changes in how queries are answered over time.

Let's dive into where we can find these AI responses.

Google: AI Mode, AI Overviews, and AI in People Also Ask

Google AI Mode (Conversational Search)

Google AI Mode introduces a chat‑like search experience, where the response is generated dynamically rather than assembled from classic SERP blocks.

With SerpApi, you can capture these responses in structured form with our Google AI Mode API:

The full AI response text
Follow up questions at the end of the answer
All the references
Shopping results

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Google AI Overviews

Google AI Overviews appear at the top of the SERP results as a synthesized AI generated answer with citations.

With SerpApi’s Google Search API, AI Overviews are returned as a dedicated JSON object, typically including:

The generated summary text
Referenced sources / citations
Highlighted concepts

💡 Google may not always return the overview content, but instead return a page_token in the ai_overview object. When this happens, you can send an additional request to the AI Overview API to retrieve the full AI Overview.

You can do this using the page_token or by simply sending a GET request to the serpapi_link in the ai_overview object in the Google Search API response. The page_token is only valid for one minute after the initial request, so we recommend querying for the AI overview immediately after your initial search.

This allows you to extract the AI generated answer without parsing HTML or executing JavaScript.

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like when the ai_overview content is included:

Here is what the response looks like when the a page_token is returned instead:

After a request to the AI overview API with that page_token we get:

AI Answers in “People Also Ask” (PAA)

The People Also Ask section has evolved from expandable snippets with human written answers into AI‑style synthesized answers for many queries.

Using SerpApi, you can get each PAA question and answer using our Google Search API, which includes:

The associated question
The generated answer text

- Source references (when available)

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Bing Copilot Answers

Microsoft Bing has a Copilot page where you can get ask questions in a chat‑like search experience.

With SerpApi’s Bing Copilot API, Copilot responses are exposed as structured fields rather than embedded UI components.

You can extract the following from the Copilot generated answer:

Text and lists
Links wherever they appear in the answer
Reference indexes
Related follow‑up prompts

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

DuckDuckGo Search Assist API

DuckDuckGo exposes AI assisted summaries and responses through its Search Assist experience.

With SerpApi’s DuckDuckGo Search Assist API, by setting the search_assist parameter to true, you can programmatically retrieve:

AI generated instant answers
Referenced sources / citations

💡 Search assist results may contain answer, expanded_answer, sources, titles, abstracts, domains, urls, organic_word_ngrams, and organic_wiki_entities.

DuckDuckGo is the only engine to include organic word n-grams (sequences of n consecutive words) in their AI assisted summaries. These may be used to structure and optimize content based on actual search-driven data. They allow you to move beyond single-keyword targeting to understand user intent, phrase patterns, and topical depth.

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Naver AI Briefing

Naver, the dominant search engine in South Korea, has rolled out its own AI Briefing feature, tailored to local content and language preferences.

SerpApi supports Naver search results and surfaces the AI generated answer in structured JSON with the Naver AI Overview API, including:

AI generated answer (in markdown or broken down by paragraphs)
Linked sources from Naver properties and external sites
Reference indexes
Any displayed media

Try out an example here:

SerpApi Playground - SerpApi

Here is what the response looks like:

Why Use SerpApi?

SerpApi streamlines the process of web-scraping. SerpApi manages the intricacies of scraping and returns structured JSON results. This allows you to save time and effort by avoiding the need to build your own scrapers or rely on other web scraping tools.

We do all the work to maintain all of our parsers and adapt them to respond to changes on search engines. This is important, as search engines are constantly experimenting with new layouts, new elements, and other changes. By taking care of this for you on our side, we eliminate a lot of time and complexity from your workflow.

Conclusion

As search continues to evolve from links to overviews and AI generated answers, having structured access to AI generated responses gives you a critical edge.

I hope this post enabled you in understanding how AI generated answers appear across major search engines and how you can use SerpApi to get them in structured JSON. Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Documentation

Measuring Brand Presence Across AI Answer Engines

Matheus Braga — Thu, 25 Jun 2026 23:02:36 +0000

Most teams measure brand presence in AI answers with one number: a yes/no, or a citation count. Presence is actually four separate signals: mention, citation, linked source, and recommendation. Each one moves independently. Here's the model, and where each signal lives in the response from the five answer engines we work with.

The Four-Signal Model

We track brand presence in AI answers as four separately measurable signals, each tied to a different strategic question.

Mention: the brand name appears in the generated prose, with or without a link. Answers: do AI engines know we exist in this category at all?
Citation: the brand's domain appears in the formal references panel rendered alongside the answer, separate from any inline links inside the prose. Answers: do they treat our content as a source worth pointing readers at?
Linked source: the brand's URL is hyperlinked inline within the prose itself, not in the references panel. Answers: are our pages the click-through path users actually take?
Recommendation: the engine actively positions the brand as the choice, with verbs like "we recommend," "the best option is," or by ranking it first in a list. Answers: are we being endorsed, or just listed among others?

A brand can be cited without being mentioned, mentioned without being recommended, and recommended without ever being linked. Each combination tells you something different.

When to Track Each Signal

The four signals don't all matter to every brand. Here's a rough guide to which to prioritize.

Track Mention when:

You're early in a category and entity recognition is the goal
You're benchmarking share of voice against direct competitors

Track Citation when:

You publish content meant to be a reference: docs, research, long-form guides
You want to validate that AI engines treat your domain as authoritative on a topic

Track Linked source when:

You measure AI as a traffic channel and need a numerator that maps to landing pages
You optimize for click-through and care about which pages actually receive AI-driven visits

Track Recommendation when:

You sell a consideration-stage product that gets compared against alternatives
"Best X for Y" queries are a meaningful part of your keyword universe

Most brands end up tracking all four eventually. Starting with the one that maps to your current question is what keeps the four from collapsing back into a single number.

What "Brand Presence" Currently Means (and What It's Missing)

Two of those four signals, mention and citation, should sound familiar. They're the metrics most of the GEO writing published this year converges on. Both Ahrefs's AI Visibility Audit and Semrush's brand visibility post work from this two-part vocabulary, and they're the standard references for a reason. The vocabulary is useful.

It's also incomplete. A two-signal split lumps together things that behave very differently in practice, a brand quoted in passing versus a brand the engine actively endorses, a domain listed below the answer versus a domain hyperlinked inside the answer. Conflating them is what makes AI visibility feel like a black box: the metric moves, but you can't tell which part of the relationship moved with it.

Two signals isn't wrong. It's just not enough to act on.

How Query Style Shapes the Signals

The four don't move together in practice. Once you measure them separately, three patterns recur often enough to be worth naming.

The examples below come from running each query through SerpApi's Google AI Overview API, which returns Google's rendered SERP, including the AI Overview, as a structured JSON payload. The screenshots show what a user sees on Google; the JSON snippets show the underlying response shape.

Comparison-style queries

A query like "best email marketing platform for small business" shows how the references panel can miss the brands the prose features. In the AI Overview for that query, MailerLite, Klaviyo, Brevo, Kit, and Mailchimp are each named in the prose with an inline link to the vendor's own site. The references panel underneath cites Saltbox, PCMag, and Zapier, three sites writing about the category, none of the vendors themselves. The brands have mentions and linked sources, the formal citations belong to the publishers writing about them. A tracker reading only the references panel would log zero presence for the actual products.

Playground

Google AI Overview result for "best email marketing platform for small business"

"ai_overview": {
  "text_blocks": [
    {
      "type": "list",
      "list": [
        {
          "snippet": "MailerLite: Best for simplicity and affordability...",
          "snippet_links": [
            { "text": "MailerLite", "link": "https://www.mailerlite.com/" },
            { "link": "https://www.saltbox.com/blog/best-email-marketing-platforms-small-business" }
          ]
        }
        // Klaviyo, Brevo, Kit, Mailchimp follow the same shape:
        // each named in `text` with an inline link to the vendor site,
        // and a second snippet_link to the Saltbox article.
      ]
    }
  ],
  "references": [
    { "source": "Saltbox", "link": "https://www.saltbox.com/blog/best-email-marketing-platforms-small-business", "index": 0 },
    { "source": "PCMag", "link": "https://www.pcmag.com/picks/the-best-email-marketing-software", "index": 1 },
    { "source": "Zapier", "link": "https://zapier.com/blog/free-email-marketing-software/", "index": 2 }
  ]
}

Definitional queries

For example, "what is dwell time in seo" produces the inverse of the previous: extensive citation activity with no brand naming at all. The AI Overview for that query cites ten domains in its references panel: Semrush, Shopify, SpyFu, SEO Locale, AccuRanker, Rank Math, and four more, and links inline to half of them. None of the ten is named in the prose itself. A tracker reading only mentions would log zero presence for every site contributing to the answer.

Playground

Google AI Overview result for "what is dwell time in seo"

"ai_overview": {
  "text_blocks": [
    {
      "type": "paragraph",
      "snippet": "Dwell time is the amount of time a visitor spends on your webpage after clicking through from search engine results...",
      "snippet_links": [
        { "link": "https://seolocale.com/what-is-dwell-time-in-seo-and-why-does-it-matter/" }
      ]
    }
    // 4 more text_blocks (Key Concepts list, How Dwell Time Differs, How to Improve)
    // None of the 10 reference domains is named in any snippet text.
    // Of 7 inline snippet_links across the answer, 5 unique domains are linked.
  ],
  "references": [
    { "source": "Semrush", "link": "https://www.semrush.com/blog/dwell-time/", "index": 0 },
    { "source": "Shopify", "link": "https://www.shopify.com/blog/dwell-time-in-seo", "index": 1 },
    { "source": "SpyFu", "link": "https://www.spyfu.com/blog/dwell-time-seo/", "index": 2 },
    { "source": "Relentless-Digital","link": "https://www.relentless-digital.com/dwell-time-seo", "index": 3 },
    { "source": "SEO Locale", "link": "https://seolocale.com/what-is-dwell-time-in-seo-and-why-does-it-matter/", "index": 4 },
    { "source": "Ranked AI", "link": "https://www.ranked.ai/blog/post/optimizing-dwell-time-for-better-seo", "index": 5 },
    { "source": "LSEO", "link": "https://lseo.com/blog/content-marketing/what-is-dwell-time-why-does-it-matter-for-seo/", "index": 6 },
    { "source": "Rankability", "link": "https://www.rankability.com/ranking-factors/google/dwell-time/", "index": 7 },
    { "source": "AccuRanker", "link": "https://www.accuranker.com/blog/dwell-time-in-seo/", "index": 8 },
    { "source": "Rank Math", "link": "https://rankmath.com/fi/seo-glossary/dwell-time/", "index": 9 }
  ]
}

Replacement queries

Something like "alternatives to mailchimp" produces a stranger result: a mention that isn't a recommendation. The AI Overview opens with "If you're looking for a Mailchimp alternative...". Mailchimp is named, but the brand isn't in the references, and the rest of the answer routes users toward replacements like MailerLite, Klaviyo, Omnisend, and Brevo. To a brand-mention tracker, this counts as a win. To Mailchimp, it's a churn surface.

Playground

Google AI Overview result for "alternatives to mailchimp"

"ai_overview": {
  "text_blocks": [
    {
      "type": "paragraph",
      "snippet": "If you're looking for a Mailchimp alternative, your best options depend on your business model. For e-commerce, choose Omnisend... For simple, affordable newsletters, try MailerLite. If you want advanced automation, upgrade to ActiveCampaign.",
      "snippet_links": [
        { "link": "https://zapier.com/blog/mailchimp-alternatives/" }
      ]
    }
    // List items follow with inline links to Klaviyo, Omnisend, MailerLite,
    // Constant Contact, Brevo — every recommended replacement gets a vendor link.
    // Mailchimp is named only in the framing sentence above; no inline link, no reference entry.
  ],
  "references": [
    { "source": "Zapier", "link": "https://zapier.com/blog/mailchimp-alternatives/", "index": 0 },
    { "source": "MailerCheck", "link": "https://www.mailercheck.com/articles/mailchimp-competitors", "index": 1 },
    { "source": "YouTube", "link": "https://www.youtube.com/watch?t=8&v=Ny7uCVTjxPc", "index": 2 },
    { "source": "Omnisend", "link": "https://www.omnisend.com/blog/mailchimp-alternatives/", "index": 3 },
    { "source": "YouTube", "link": "https://www.youtube.com/watch?t=543&v=ZHbFq67qieI", "index": 4 },
    { "source": "YouTube", "link": "https://www.youtube.com/watch?v=Ny7uCVTjxPc&t=8", "index": 5 },
    { "source": "YouTube", "link": "https://www.youtube.com/watch?v=ZHbFq67qieI&t=543", "index": 6 }
  ]
}

The "AI citations" bucket in our post on rank tracking in the age of AI Overviews is exactly what this model decomposes. Rank tracking gave us a single citation metric to put on a dashboard. The four-signal model gives us the diagnostic underneath it.

Where Each Signal Lives, Engine by Engine

Across the five engines, the response shape is more unified than the marketing names suggest. Google AI Overview, Google AI Mode, Bing Copilot, and Naver AI Overview all expose the answer as a text_blocks[] array plus a references[] panel, same building blocks, same extraction logic. DuckDuckGo Search Assist is the structural outlier: a flat string answer plus a flat sources[] list, with no text_blocks and no inline links.

	Mention	Citation	Linked source	Recommendation
Google AI Overview	scan `text_blocks[].snippet`	`references[]`	`text_blocks[].snippet_links[].link`	text + list-position heuristic
Google AI Mode	scan `text_blocks[].snippet`	`references[]`	`text_blocks[].snippet_links[].link`	text + list-position heuristic
Bing Copilot	scan `text_blocks[].snippet`	`references[]`	`text_blocks[].snippet_links[].link`	text + list-position heuristic
Naver AI Overview	scan `text_blocks[].snippet` or top-level `markdown`	`references[]`	parse `[1]` footnote refs in `markdown`	text only
DuckDuckGo Search Assist	scan `search_assist.answer`	`search_assist.sources[]`	— (no inline links)	text only

ChatGPT and Claude aren't on this list. They're answer surfaces where brand presence matters, but they're conversational rather than search-shaped: no per-query SERP, no formal references panel, no structured response to map the four signals onto. The model translates in spirit, the measurement layer is different enough to warrant its own treatment.

Google AI Overview

All four signals come back in one payload from the Google AI Overview API: mention is a string match against the concatenated text_blocks[].snippet, citation is a domain match against references[], linked source walks text_blocks[].snippet_links[], and recommendation needs the prose plus the position of the brand inside any embedded list. The same shape returns from the standalone Google AI Overview API endpoint, useful when you want to fetch an AI Overview without the surrounding SERP.

Google AI Mode

Same text_blocks[] + references[] shape, returned at the top level by the Google AI Mode API. AI Mode answers are longer and more conversational than AI Overviews, which makes recommendation language easier to detect, and it makes responses noisier between runs.

Bing Copilot

Same text_blocks[] + references[] shape again, returned at the top level by the Bing Copilot API.

Naver AI Overview

Korean-language surface from the Naver AI Overview API. The text_blocks[] here don't carry snippet_links, instead, footnote references like [1] appear inline in the top-level markdown, and you map them back to references[] by index.

DuckDuckGo Search Assist

The minimal surface, returned by the DuckDuckGo Search Assist API: a flat search_assist.answer string and a flat search_assist.sources[] list, no text_blocks, no inline anchors. That makes Linked source structurally unmeasurable here, which is itself useful information, if click-through from AI is your primary KPI, this surface is currently a non-channel.

The engines themselves disagree on what presence can mean. Your measurement layer should make that disagreement legible, not paper over it with zeros, and any single composite score should respect that, not flatten it.

Conclusion

The single-number view of AI visibility is comfortable because it fits on a slide. The cost is that it tells you presence changed without telling you which signal moved.

The four-signal model adds instrumentation in exchange for diagnostic clarity. You pay for it in pipeline complexity, four extractions per engine, four columns on the dashboard instead of one, four conversations to have when something moves. You get back the ability to answer specific questions: the engines stopped recommending us this quarter, or our citations dropped but our mentions held steady, or DuckDuckGo isn't a click channel and never was. Those are the conversations that move strategy.

Brand presence in AI answers is not one thing. Treat it as four, weight them for what you're trying to learn, and the tracking conversation gets less ambiguous.

SerpApi's New Status Page Is Live!

Nathan Skiles — Thu, 25 Jun 2026 15:01:03 +0000

We’re excited to announce that SerpApi now has a new Status Page for service updates. This page is live and will serve as the central hub for all incident and outage communications. If there’s ever API downtime, service disruptions, or performance degradation, you’ll be able to find the latest updates there. This launch was driven by our commitment to transparency and direct customer feedback.

View the current status of all APIs and ensure you’re subscribed for notifications about any incidents. It’s live and ready to keep you informed.

Subscribe here!

Please note that we will continue using our Public Roadmap to track minor bugs and issues, and we’ll send direct updates once they’ve been resolved.

Subscribing to Updates

To subscribe to incident notifications, simply click the 'Subscribe' button at the top of the page. We offer several channels to receive notifications, including email, SMS, Slack, and Microsoft Teams.

After clicking Subscribe, you’ll receive a confirmation email and be redirected to a page where you can customize your notification preferences.

⚠️ If you do not confirm your subscription, you will not receive incident notifications.

If you do not receive a confirmation, please contact support at contact@serpapi.com.

Subscribe to the APIs You Use

Once you reach the component selection page, you can customize the notifications you receive based on the SerpApi services most relevant to your use case. By default, all APIs are selected, but you can easily narrow this down by selecting only the APIs you rely on (for example, just the Google Search API or AI Overview API).

I recommend at least subscribing to the API component to ensure you're at least notified about system-wide issues, along with your most used APIs.

💡 From the component selection screen (shown above), click "Select none" above the component table to deselect all APIs. This allows you to select only the APIs you care about, without manually deselecting irrelevant components.

Customizing your notifications ensures you only receive incident updates when the specific APIs you care about are impacted.

Final Thoughts

Our goal with this status page is to make service communications clearer, faster, and easier to follow.

If you haven’t already, we recommend subscribing now so you never miss an important update.

As always, please reach out if you have any questions or feedback!

Subscribe here!

Get Google Organic Search Results Instantly with Google Light Fast API

Catherine Li — Thu, 25 Jun 2026 07:00:52 +0000

The API documented here has been discontinued because Google now limits the underlying endpoint to only 3 organic results. We recommend using the Google Light API as an alternative.

Introduction

If you’re a marketer who wants to monitor keyword rankings for your business, or a developer wanting to pull the latest headlines for your news application, you know how much of a pain this can be. Most online tools are costly, bloated and can’t be customized to your problem. Today, we’ll use SerpApi’s Google Light Fast API to easily and quickly scrape Google search results. We’re going to retrieve the data programmatically, exactly as it appears on Google’s own search results page.

In this tutorial, we’ll write a simple server-side script that checks how a URL ranks for specific keywords on Google. With SerpApi, you don’t have to worry about the usual web scraping headaches, like updating your code for layout changes, dealing with IP blocks, or fighting CAPTCHAs. Instead, you can focus on building your app. Before we start, make sure you’ve signed up for a free account on SerpApi.

Prerequisites and Project Setup

💡 Important: This application is for demonstration purposes only. If you want to use it in production, you may wish to add more robust error handling and security features.

This tutorial assumes you have a basic understanding of JavaScript, Node, and npm. If not, no worries. We also have a no-code solution that can be found here. Alternatively, you can always come back to this post once you gain familiarity with basic JavaScript and Node. Feel free to jump directly to integrating with SerpApi if you already have a basic Node skeleton project set up.

To set up a brand new project, create a new folder, navigate to that folder in your terminal and run the following commands:

npm init -y

We need to install a few dependencies: dotenv to access environment variables, and serpapi for the public JavaScript library.

npm i serpapi dotenv

Next, create an index.js file in your root directory. We also need to store and access the API key that comes with the free SerpApi account. There are several different ways to do this, but for this tutorial let’s create an .env file and store the key there.

API_KEY=your_key_here

💡 Important: Make sure to keep your API key private and never commit it into version control.

Of course, we’ll need to import the libraries we’ve installed at the top of our file before using them:

import dotenv from "dotenv";
import { getJson } from "serpapi";

dotenv.config();
const apiKey = process.env.API_KEY;

Since we’ll be using ES6 modules in this tutorial, don’t forget to include “type”: "module" in your package.json.

"type": "module"

Scraping with JavaScript

Let’s imagine you own Starbucks, and you want to check your ranking for a list of popular coffee-related keywords. We’ll need to put those keywords we want to target into an array. We’ll also define the domain we want to be looking for as “starbucks.com”.

const keywords = [
  "coffee",
  "iced coffee",
  "oat latte",
  "best coffee",
  "americano",
  "latte",
  "matcha",
  "pumpkin spice latte",
  "oat milk latte",
];

const domain = "www.starbucks.com"

All we need to do is call getJson from the serpapi library. Note that getJson is an asynchronous function with callback and async/await support. Before calling an array of keywords, let’s try out the API with the first keyword in our array.

const data = await getJson({
  api_key: apiKey, //required
  engine: "google_light_fast",//required
  google_domain: "google.com",
  q: keywords[0],//required
  hl: "en",
  gl: "us",
});

console.log(data)

The only required parameters for the Google Light Fast API are api_key, engine, and q. For a full list of supported parameters, take a look at our documentation. Also, if you’d like to experiment more with this API, don’t forget to check out our public playground.

Now let's run this!

In your terminal, navigate to the root of this project and run:

node index.js

If everything went well, you should have received a response in your console within a second. The JSON response should resemble something like this:

{
  "search_metadata": {
    "id": "68d437484c437fdcf83f1897",
    "status": "Success",
    "json_endpoint": "https://serpapi.com/searches/967ec5090d430149/68d437484c437fdcf83f1897.json",
    "created_at": "2025-09-24 18:24:08 UTC",
    "processed_at": "2025-09-24 18:24:08 UTC",
    "google_light_fast_url": "https://www.google.com/search?q=Coffee&hl=en&gl=us&sourceid=chrome&ie=UTF-8",
    "raw_html_file": "https://serpapi.com/searches/967ec5090d430149/68d437484c437fdcf83f1897.html",
    "total_time_taken": 0.97
  },
  "search_parameters": {
    "engine": "google_light_fast",
    "q": "Coffee",
    "google_domain": "google.com",
    "hl": "en",
    "gl": "us"
  },
  "search_information": {
    "organic_results_state": "Results for exact spelling"
  },
  "organic_results": [
    {
      "position": 1,
      "title": "Coffee - Wikipedia",
      "link": "https://en.wikipedia.org/wiki/Coffee",
      "displayed_link": "https://en.wikipedia.org › wiki › Coffee",
      "favicon": "https://encrypted-tbn1.gstatic.com/favicon-tbn?q=tbn%3AANd9GcSVMU85vzsP5RTNub9IaAsl7rV0Nyf_xTINCBJZ1nlvBnxTDgf0HU9xmmPaba08IkD_mWLlL8LDthkMoG4tsyqVPp6VYs98tC-e5-E",
      "snippet": "Coffee is a beverage brewed from roasted, ground coffee beans. Darkly colored, bitter, and slightly acidic, coffee has a stimulating effect on humans"
    },
    {
      "position": 2,
      "title": "Starbucks Coffee Company",
      "link": "https://www.starbucks.com/",
      "displayed_link": "https://www.starbucks.com",
      "favicon": "https://encrypted-tbn3.gstatic.com/favicon-tbn?q=tbn%3AANd9GcRdvxQfPWiafCxH9HD0XtJerZ5FY7ctOVpxK4U9EldMD3SkXwh5oKlsGSS8A_GBPbNYj4_pdsmKya0-XXSG6XkkpircValazFMJsz9o",
      "snippet": "More than just great coffee. Explore the menu, sign up for Starbucks® Rewards, manage your gift card and more."
    },
    {
      "position": 3,
      "title": "Wonderstate Coffee: Sustainable Coffee Roasters | Specialty ...",
      "link": "https://wonderstate.com/?srsltid=AfmBOopAdGD571-IHoJTulurDRLUKETcgkzQfHKQNZPsH8Qq8eDsXRbw",
      "displayed_link": "https://wonderstate.com",
      "favicon": "https://encrypted-tbn1.gstatic.com/favicon-tbn?q=tbn%3AANd9GcRkkAe_Nr9r_KGO8rsBxg32ryD4lN3GUn5mZd4nByuqHpZ5QJQ1gyWDmiLC58xylF0G8jT5n_xxHic0KO6aCNAIe3UrOAZVr79s_w",
      "snippet": "Award-winning, organic coffee roasters in Wisconsin. 100% solar powered. Paying farmers more with a minimum price guarantee that's double the Fair Trade ..."
    },
    {
      "position": 4,
      "title": "r/Coffee - Reddit",
      "link": "https://www.reddit.com/r/Coffee/",
      "displayed_link": "https://www.reddit.com › Coffee",
      "favicon": "https://encrypted-tbn2.gstatic.com/favicon-tbn?q=tbn%3AANd9GcRt_ioW2I2eFmqJJLAWQSl7Fa22pXg9CBflQgcrSYXdMzIwWTa8eHdeZ7OKCeCySe9gqu2bcJ6_Fb4ZlTRQWnx9d5z3ide5sGU7",
      "snippet": "Welcome to the daily · /r/Coffee. question thread! There are no stupid questions here, ask a question and get an answer! We all have to start somewhere and ..."
    },
    {
      "position": 5,
      "title": "Blue Bottle Coffee | Fresh Roasted Specialty Coffee",
      "link": "https://bluebottlecoffee.com/?srsltid=AfmBOopxMGf5ASuuIzlP2zEnsmxVUaUpcVIL_Omb2XXZHLoMZIkDNYdf",
      "displayed_link": "https://bluebottlecoffee.com",
      "favicon": "https://encrypted-tbn3.gstatic.com/favicon-tbn?q=tbn%3AANd9GcQrS-d7qjN2SDnT3GkjE7o8RH7JrmSdLttp1UvAcoXpDs02e0_3g--k76_K93BYNdQb3qy6wIyxKfH1OSN6lW7q5kLmD77Q9DCuSs4Hnw7y",
      "snippet": "Blue Bottle Coffee is a specialty coffee roaster with cafes in LA, SF, NYC, & Japan. Shop our freshly roasted specialty coffee online & in-store."
    },
    {
      "position": 6,
      "title": "All Coffee - Verve Coffee Roasters",
      "link": "https://www.vervecoffee.com/collections/all-coffee?srsltid=AfmBOoqWVxbvNcxCo2cb5WPdO_8jZV2YDfbdabwn9OkQnbOQL3sw6hTo",
      "displayed_link": "https://www.vervecoffee.com › collections › all-...",
      "favicon": "https://encrypted-tbn0.gstatic.com/favicon-tbn?q=tbn%3AANd9GcQzMum2G8lsDS0YSPwMugOHqS0y9zN2MOXtp-QvDxJCqt9YeGdiZxUqkXmm0ta4_uG17l8u-HeDmVGkS0r2UksQOBS8Dt_9VLkqua0UNnk",
      "snippet": "Verve Coffee Roasters - Sermon - So easy to wake up to - Sweet, deep, & soulful - Medium Roast blend - Notes: Blueberry Pie, Cocoa, Candied Pecan"
    },
    {
      "position": 7,
      "title": "Buy Coffee, Tea, Powders Online | The Coffee Bean & Tea ...",
      "link": "https://www.coffeebean.com/",
      "displayed_link": "https://www.coffeebean.com",
      "favicon": "https://encrypted-tbn3.gstatic.com/favicon-tbn?q=tbn%3AANd9GcRc9wUBGZNChw_o9jALpdvp8DpUvmKJZBHoi6SjSbIAMxC3dKUta8lOmLQ93yAlgjciko0OlyYYnqxxYe3-er5G0jEMEGv8zpbOXPaZkA",
      "snippet": "Buy exceptional coffee, tea, powders, equipment and drinkware at The Coffee Bean & Tea Leaf® online store to enjoy our globally sourced products at home."
    },
    {
      "position": 8,
      "title": "Coffee - The Nutrition Source",
      "link": "https://nutritionsource.hsph.harvard.edu/food-features/coffee/",
      "displayed_link": "https://nutritionsource.hsph.harvard.edu › coffee",
      "favicon": "https://encrypted-tbn3.gstatic.com/favicon-tbn?q=tbn%3AANd9GcRobWpRagBKXMoDcfSN3X_img5d8Nq67TuwrAktRwrYtmdaP9mlboWwArcZiFjXnk8ebAjJTKbCjn9UliNZx9AZUMSxGoDrNKyi90c1AaIjO-4Dt8GOYJLeO1yy",
      "snippet": "A plain “black” cup of coffee is a very low calorie drink—8 ounces only contains 2 calories! However, adding sugar, cream, and milk can quickly bump up the ..."
    },
    {
      "position": 9,
      "title": "Black Rifle Coffee | Home | Veteran Founded",
      "link": "https://www.blackriflecoffee.com/?srsltid=AfmBOooqbrXMU1TgC5NfJI3S2N6OArBNRVxOQ0vn2ULhC_pDPuDCIOKh",
      "displayed_link": "https://www.blackriflecoffee.com",
      "favicon": "https://encrypted-tbn1.gstatic.com/favicon-tbn?q=tbn%3AANd9GcSQ0KPBZYN-E6_-TG395r9g5fduhEk-a1UgURz6gvCdO6_GfUaDSIJLD0ry6J8cxdeATqVju60LM3tsH6fiMPNFxVRiJvWfYd3CX3leJEHkU9Q5vw",
      "snippet": "Veteran-founded Black Rifle Coffee Company delivers premium small-batch coffee, gear, and unapologetic American values. Join the mission, fuel your day."
    },
    {
      "position": 10,
      "title": "Caribou Coffee® | Life Is Short. Stay Awake For It®",
      "link": "https://www.cariboucoffee.com/?srsltid=AfmBOopdVML0bJqbgvRe1RZV9qnofeFKw9oep5TWl_izz84n7_GMualB",
      "displayed_link": "https://www.cariboucoffee.com",
      "favicon": "https://encrypted-tbn3.gstatic.com/favicon-tbn?q=tbn%3AANd9GcRjjXlKFqUKDV8Z3T69L4YTdEL4lvE5SRtpTzdLjSAMvTDNEyJIgfeF7qemE3qaNhLBzA_lmcQ7ZUIoL-QapePqpBCttfqTcmJGU7iENd48sw",
      "snippet": "Caribou Coffee® is more than a premium coffeehouse featuring high-quality, handcrafted beverages and food. Explore our menu, sign up for Caribou Coffee® ..."
    }
  ]
}

The organic_results array stores the list of unpaid listings on Google’s results page and excludes paid ads, which are marked as “Sponsored”. The values in this array corresponds to the links you see on Google's search results page.

Therefore, the values in organic_results is what we will look at to find out in which position starbucks.com ranks for each of the keywords. As you can see, the Starbucks domain is the second search result on the page for the keyword “coffee”.

By default, the API will return the first ten search results which reflects what is displayed on the first page of Google's search results. For keyword rankings, oftentimes we'll need to pull search results beyond the first page. Due to recent changes in the number of results served by Google in a single search, many web scraping services on the market are unable to retrieve more than 10 results at a time. This is the case currently with our Google Light Search API, so you'd need to make 10 separate API calls if you wanted 100 results. However, with the new Light Fast API, all you need to do is assign the num parameter to 100 in our request:

const data = await getJson({
    api_key: apiKey, //required
    engine: "google_light_fast", //required
    google_domain: "google.com",
    q: keywords[0],//required 
    gl: "us",
    num: 100
})

As of now, SerpApi's Light Fast API is the only web scraping API on the market that lets you retrieve 100 results at a time.

Solution to scrape 100 Search Results on Google - Google Fast Light API

Getting the Keyword Ranking

We’ll write a helper function to find the ranking for starbucks.com within the organic search results. This function returns the first index of the search result whose displayed_link contains starbucks.com, and it will be called for each of our specified keywords.

const getRanking = (data) => {
    const index = data.organic_results.findIndex(result =>
        result.displayed_link.includes(domain)
    );

    return index !== -1 ? index + 1 : "N/A";
}

Since we have an array of keywords we want to check for and not just one, what we need to do now is call this API in a loop:

const responses = keywords.map(async (keyword) =>
    getJson({
        api_key: apiKey,
        engine: "google_light_fast",
        google_domain: "google.com",
        q: keyword,//required 
        gl: "us"
    })
);

const json = await Promise.all(responses);

Once all the API calls for the keywords are complete, we can then check for the ranking of starbucks.com for each of the results.

keywords.forEach((keyword, i) => {
    console.log(`The ranking for "${keyword}" is: ${getRanking(json[i])}`);
});

If all went well, you should see something like this:

This means that, with the given parameters, a link to starbucks.com appears as the second search result on Google if the term “coffee” is used to search. Of course, your exact results may be different since search results change frequently.

Track Google Search Rankings and Export to a .CSV file with Node.js

For the final version of this script, feel free to clone this GitHub repository. If you wanted to build on top of this, you could also add a feature to export the data into a csv. SerpApi makes it effortless to integrate up-to-date, relevant Google search results into your application. This month, we’ve boosted free plans from 100 to 250 credits. Make sure to give it a try by signing up for a free account and experimenting in ourpublic playground. If you have any questions, don't hesitate to contact us at SerpApi Contact.

Get Hotel Details From Tripadvisor And Compare Hotels using Python

Sonika Arora — Thu, 25 Jun 2026 07:00:36 +0000

When people travel, one of the first things they do is compare hotels. Price alone isn’t enough - ratings, amenities, location, and recent reviews all factor into the decision. For developers building travel products, data analysts, or recommendation tools, this means turning rich but messy public data into structured, comparable insights.

Tripadvisor is one of the most valuable public sources for this kind of hotel intelligence, but extracting structured, comparable data at scale is difficult. In this post, we’ll walk through how to scrape hotel details from Tripadvisor using SerpApi’s Tripadvisor Place Results API with Python, and then compare hotels programmatically using the returned data.

What Are Tripadvisor Place Results?

The Tripadvisor Place Results page looks like this:

Here's an example URL: https://www.tripadvisor.com/1197076

A place on Tripadvisor represents a specific entity such as:

A hotel
A restaurant
An attraction

For hotels, place results typically include:

Hotel name and Tripadvisor URL
Rating and reviews
Pricing
Address and geographic location
Pictures and descriptions
Nearby destinations
Amenities and categories

This makes it a perfect input for hotel comparison workflows.

Get Hotel Results from Tripadvisor

Using SerpApi, you can request Tripadvisor place results by providing a hotel ID.

To get the hotel ID (place_id), you can make a request to our Tripadvisor Search API like this:

Set engine parameter to tripadvisor
Provide a search query as the location (example: "Rome") and specify the ssrc(Search Filter) parameter as h (for Hotels)
Specify your API key in the api_key parameter
Receive structured JSON with all hotels in that location
Grab the place_id

Here's an example in our Playground:

SerpApi Playground - SerpApi

Then, for each hotel, we can get the Place Results using the place_id and sending the request to the Tripadvisor Place API.

The request for getting hotel details looks like:

Set engine parameter to tripadvisor_place
Provide a place_id (example: 228406)
Specify your API key in the api_key parameter
Receive structured JSON with hotel details

Here's an example in our Playground:

SerpApi Playground - SerpApi

The response gets us Tripadvisor place data for each hotel that would otherwise require manual parsing.

Some Basic Setup Steps

Setup your environment

Ensure you have the google-search-results library installed.

pip install google-search-results

google-search-results_ _is our Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. However, serpapi is a new one, and all the examples you can find on our website are from the old one google-search-results. If you'd like to use our Python library with all the examples from our website, you should install the google-search-results module instead of serpapi.

For this blog post, I am using google-search-results because all of our documentation references this one.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

In addition to that, we'll need to install a few more libraries:

Pandas: Useful for organizing data. Install this with: pip install pandas
Matplotlib: Useful for building static plots. Install this with pip install matplotlib

Get your SerpApi API key

To begin scraping data, first, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API.

Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.

Import Libraries

Import all the necessary libraries for this project:

import csv
import os
from serpapi import GoogleSearch
from dotenv import load_dotenv
import pandas as pd
import matplotlib.pyplot as plt

load_dotenv()

Get Hotel Details using Python

For this tutorial, we'll look for 10 hotels in Rome. Once the setup is complete, here is a a simple Python script to get all hotels in Rome, and get hotels details for each one and add that to a list:

hotels = []
def get_hotel_results_from_tripadvisor(q):
    params = {
        "engine": "tripadvisor",
        "q": q,
        "api_key": os.environ["SERPAPI_API_KEY"],
    }
    results = GoogleSearch(params).get_dict()
    return results.get("places", [])

def get_hotel_details_page_from_tripadvisor(place_id):
    params = {
        "engine": "tripadvisor_place",
        "place_id": place_id,
        "api_key": os.environ["SERPAPI_API_KEY"],
    }
    results = GoogleSearch(params).get_dict()
    return results.get("place_result", {})

if __name__ == " __main__":
    q = "Rome"
    place_results = []
    hotels = get_hotel_results_from_tripadvisor(q)
    first_10_hotels = hotels[:10]
    for result in first_10_hotels:
        place_id = result.get("place_id", None)
        if place_id:
            hotel_details = get_hotel_details_page_from_tripadvisor(place_id)
            place_results.append(hotel_details)

Compare Hotels Programmatically

Let's first start by identifying fields that will be of importance to us.

Pick The Fields For Fair Comparison

To understand a place's identity:

place_result.name
place_result.rating
place_result.reviews
place_result.hotel_stars
place_result.num_rooms

Useful for competitive context:

place_result.ranking → parse numeric rank
place_result.reviews → popularity signal

For understanding pricing:

price_range.low
price_range.high
Lowest offers[].extracted_price

Sub rating across various dimensions:

These are already numeric and normalized by Tripadvisor, making them perfect for comparison:

Location
Rooms
Value
Cleanliness
Service
Sleep Quality

Review highlights to extract insights from Reviews

reviews_highlights.category
reviews_highlights.value

💡 You can ignore summaries & quotes for scoring to keep things quantitative. For qualitative data, the review summaries are a good place to get it.

Define a Scoring Strategy

As a simple comparison strategy, we can compute three layers:

Base Quality Score → from subratings
Sentiments → from reviews_highlights
Value-for-Money Score → rating ÷ price

We can also weight the three differently when computing the final score for a hotel. For this tutorial, we'll use:

60% quality score (most reliable)
25% sentiment score (guest opinions)
15% price efficiency score

Let's define the plan for how we will use this scoring strategy on the list of place results we have already:

Get place_results for each hotel we want to compare

↓

Extract hotel features we want to compare for each hotel

↓

Compute scores for each hotel based on the scoring strategy above

↓

Store the hotel results in a table/dataframe

↓

Sort / filter / plot / export results

Code Solution

def extract_hotel_features(place_result):
    HIGHLIGHT_SCORE_MAP = {
        "Immaculate": 5.0,
        "Abundant": 4.5,
        "Convenient": 4.5,
        "Good": 4.0,
        "Inconsistent": 3.0,
        "Average": 3.0,
        "Dated": 2.0,
        "Outdated": 2.0,
        "Pricey": 2.0,
    }

    hotel = {
        "name": place_result.get("name"),
        "rating": place_result.get("rating"),
        "review_count": place_result.get("reviews"),
        "hotel_stars": place_result.get("hotel_stars"),
        "num_rooms": place_result.get("num_rooms"),
    }

    # Ranking number
    ranking = place_result.get("ranking")
    if ranking:
        hotel["city_rank"] = int(ranking.split("#")[1].split(" ")[0])

    # Price range
    price_range = place_result.get("price_range", {})
    hotel["price_low"] = price_range.get("low")
    hotel["price_high"] = price_range.get("high")

    # Lowest observed price
    offers = place_result.get("prices", {}).get("offers", [])
    if offers:
        hotel["lowest_price"] = min(
            o["extracted_price"] for o in offers if "extracted_price" in o
        )

    # Subratings (primary scoring dimensions)
    for item in place_result.get("subratings", []):
        key = item["category"].lower().replace(" ", "_")
        hotel[key] = item["score"]

    # Review highlights (sentiment signals)
    for h in place_result.get("reviews_highlights", []):
        category = h["category"].lower()
        value = h["value"]
        hotel[f"{category}_sentiment"] = HIGHLIGHT_SCORE_MAP.get(value, 3.0)

    return hotel

def compute_scores(hotel):
    # 1. Base quality score (Tripadvisor subratings)
    quality_fields = [
        "location",
        "rooms",
        "value",
        "cleanliness",
        "service",
        "sleep_quality",
    ]

    quality_scores = [
        hotel[f] for f in quality_fields if f in hotel
    ]

    hotel["quality_score"] = round(
        sum(quality_scores) / len(quality_scores), 2
    )

    # 2. Value-for-money score
    if hotel.get("lowest_price") and hotel.get("rating"):
        hotel["value_for_money"] = round(
            hotel["rating"] / hotel["lowest_price"] * 100, 2
        )

    # 3. Sentiment score
    sentiment_keys = [k for k in hotel if k.endswith("_sentiment")]
    if sentiment_keys:
        hotel["sentiment_score"] = round(
            sum(hotel[k] for k in sentiment_keys) / len(sentiment_keys), 2
        )

    # 4. Final composite score
    hotel["overall_score"] = round(
        (
            hotel["quality_score"] * 0.6 +
            hotel.get("sentiment_score", 3) * 0.25 +
            hotel.get("value_for_money", 0) * 0.15
        ),
        2,
    )
    return hotel

With these basic functions in place, we can now compare hotels:

def compare_hotels(place_results):
    hotels = []

    for place_result in place_results:
        hotel = extract_hotel_features(place_result)
        hotel = compute_scores(hotel)
        hotels.append(hotel)

    hotels_sorted = sorted(
        hotels,
        key=lambda h: h.get("overall_score", 0),
        reverse=True
    )

    df = pd.DataFrame(hotels_sorted)

    # Select only the most useful comparison columns
    comparison_df = df[

[
"name",
"overall_score",
"quality_score",
"sentiment_score",
"value_for_money",
"rating",
"review_count",
"city_rank",
"price_low",
"price_high",
"lowest_price",
]
]

    return comparison_df

Plotting The Results

Best “at a glance” Ranking

Once we have the comparison_df, we can now plot the overall scores we computed earlier:

def plot_hotels_bar_chart(comparison_df):
    import matplotlib.pyplot as plt

    df = comparison_df.sort_values("overall_score", ascending=False)

    plt.figure()
    plt.barh(df["name"], df["overall_score"])
    plt.xlabel("Overall Score")
    plt.title("Overall Hotel Comparison")
    plt.gca().invert_yaxis()
    plt.tight_layout()
    plt.show()

What that looks like:

Price comparison (range chart)

Let's understand the price spread per hotel.

def plot_price_transparency(comparison_df):
    plt.figure()
    for i, row in enumerate(comparison_df.itertuples()):
        plt.plot(

[row.price_low, row.price_high],
[i, i],
marker="o"
)

    plt.yticks(range(len(comparison_df)), comparison_df["name"])
    plt.xlabel("Price (€)")
    plt.title("Hotel Price Ranges")
    plt.tight_layout()
    plt.show()

Calling this function plots a graph which makes pricing transparent:

Practical Use Cases

This workflow is especially useful for:

Travel startups building hotel comparison products
Data teams analyzing hotel trends
SEO teams creating location-based hotel content
Investors and analysts tracking hotel performance across regions

Instead of manually browsing Tripadvisor for each hotel, you get structured, comparable insights at scale.

Conclusion

I hope that this tutorial helps you use SerpApi's Tripadvisor Place Results API and reliably extract hotel details and compare hotels in any destinations with minimal effort.

You can find all the code used in this blog post here:

https://github.com/sonika-serpapi/get-hotel-details-from-trip-advisor-and-compare-hotels

Feel free to reach out to us at contact@serpapi.com for any questions.

DEV Community: SerpApi

Building a Naver News Watchlist With Ruby on Rails

What is Naver?

Why SerpApi for Naver News?

What We’ll Build

End Result

Prerequisites

Sample JSON Result

Building the App

1. Create an API-only Rails app

2. Add the SerpApi gem

3. Store your API key

4. Service: Naver News Fetcher

5. Email digest with Action Mailer

6. Configure an email delivery method

Step 1: Generate a Gmail App Password

Step 2: Store SMTP credentials in Rails credentials

Step 3: Configure Action Mailer

7. Create the Daily Watchlist Job

8. Error Handling & Reporting

Step 1: Add an Error Email Method and Template

Step 2: Update Naver News Fetcher with Error Handling

Step 3: Update Daily Watchlist Job for Error Handling

9. Run & Test

Where to Go Next

Links

Related Blog Posts

Build A No Code AI-Powered Local Lead Outreach System

Workflow Overview

Step 1: Setup a Make scenario

Step 2: Extract Google maps data with SerpApi

Step 3: Iterate through businesses

Step 4: Generate personalized outreach with Claude

Step 5: Push leads into a Google sheet

Conclusion

Relevant Links

Related Blog Posts

Turn Bing News into an AI Briefing with Claude and SerpApi

Setting Up

More About Our Python Libraries

Fetch News with Bing News API

Prompting Claude

Generate the Briefing (Claude API)

Putting It Together

Automated Delivery

Generated AI Briefing Example Output

Building On Top Of This

Changing The Model

Persisting History

Conclusion

Relevant Links

Related Blog Posts

How To Scrape Home Depot Product Reviews

Why Scrape Reviews?

What Type of Data Can You Extract?

Setting up SerpApi

Getting Started with SerpApi

Home Depot Product Reviews API Documentation

Integrations / Libraries

Scraping Home Depot Product Reviews

GET Request

An Example With Our Python Library

More About Our Python Libraries

An Example With cURL

No Code Options

Google Sheets

Make.com

N8N

Conclusion

Relevant Links

Related Blogs

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Setting Up

Create a Slack webhook

Code related setup steps

More About Our Python Libraries

Get all indexed domains for a list of queries

Save known domains

Send slack alerts

Detect new pages