DEV Community: SerpApi

How to Scrape Google Flights Deals with a simple API

Hilman Ramadhan — Tue, 14 Jul 2026 06:34:07 +0000

Google Flights Deals helps travelers discover discounted flights without a specific destination. Instead of searching one route at a time, you can start with a departure airport or city and explore discounted destinations.

This data can be useful for building flight-deal trackers, travel discovery tools, price-comparison dashboards, or destination-recommendation apps. In this tutorial, we'll use SerpApi's Google Flights Deals API to retrieve flight deals as structured JSON with cURL, Python, and JavaScript.

Scrape Google Flights Deals website

How to use the Google Flights Deals API

A basic request uses these parameters:

Parameter	Requirement	Description
`engine`	Required	Set to `google_flights_deals`
`api_key`	Required	Your private SerpApi API key
`departure_id`	Optional	Departure airport code or location ID
`currency`	Optional	Currency used for returned prices

An airport ID is normally an uppercase three-letter IATA code such as:

CDG for Paris Charles de Gaulle Airport
AUS for Austin-Bergstrom International Airport
MAD for Adolfo Suárez Madrid-Barajas Airport

You can also provide a Google location ID beginning with /m/ or /g/. Multiple departure locations can be separated with commas.

Get your API key

Create a SerpApi account and retrieve your private key from the API key page.

Replace the following placeholder in each example:

YOUR_SERPAPI_API_KEY

Never expose your API key in public source code or commit it to a public repository.

The complete parameter reference is available in the official Google Flights Deals API documentation.

cURL implementation

Let's retrieve flight deals departing from Paris Charles de Gaulle Airport:

curl --get https://serpapi.com/search \
 -d engine="google_flights_deals" \
 -d departure_id="CDG" \
 -d currency="USD" \
 -d api_key="YOUR_SERPAPI_API_KEY"

The parameters in this request are:

engine=google_flights_deals selects the Google Flights Deals engine.
departure_id=CDG searches for deals departing from Paris Charles de Gaulle Airport.
currency=USD returns prices in US dollars.

The API returns JSON by default.

Python implementation

Create a new Python file:

main.py

Install the requests package:

pip install requests

Add the following code:

import requests


SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "engine": "google_flights_deals",
    "departure_id": "CDG",
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

response = requests.get(
    "https://serpapi.com/search",
    params=params,
    timeout=30,
)
response.raise_for_status()

data = response.json()

if "error" in data:
    raise RuntimeError(data["error"])

departure = data.get("departure_informations", {})

departure_name = (
    departure.get("airport_name")
    or departure.get("city")
    or "Unknown departure"
)

print(f"Flight deals from: {departure_name}")
print()

for deal in data.get("deals", []):
    duration = deal.get("flight_duration")

    if duration is not None:
        hours, minutes = divmod(duration, 60)
        formatted_duration = f"{hours}h {minutes}m"
    else:
        formatted_duration = "Not available"

    print("Destination:", deal.get("name", "Unknown"))
    print("Country:", deal.get("country", "Unknown"))
    print("Price:", deal.get("price", "Not available"))
    print(
        "Average price:",
        deal.get("average_price", "Not available"),
    )
    print(
        "Discount:",
        deal.get("discount_percentage", "Not available"),
    )
    print("Outbound date:", deal.get("outbound_date", "Not available"))
    print("Return date:", deal.get("return_date", "One-way flight"))
    print(
        "Route:",
        f"{deal.get('departure_airport_code', '?')} → "
        f"{deal.get('arrival_airport_code', '?')}",
    )
    print("Duration:", formatted_duration)
    print("Stops:", deal.get("stops", "Not available"))
    print("Airline:", deal.get("airline", "Not available"))
    print("Google Flights:", deal.get("flight_link", "Not available"))
    print("-" * 80)

The code prints selected fields from every item in the deals array. It also converts the documented flight_duration value from minutes into hours and minutes.

Here is the result

Flights Deals API scraper

Fields are accessed with .get() because some values are not guaranteed to appear in every deal.

Export the flight deals to CSV

You can also save the results to a CSV file for further analysis:

import csv


rows = []

for deal in data.get("deals", []):
    rows.append({
        "destination": deal.get("name", ""),
        "country": deal.get("country", ""),
        "destination_id": deal.get("destination_id", ""),
        "price": deal.get("price", ""),
        "average_price": deal.get("average_price", ""),
        "discount_percentage": deal.get(
            "discount_percentage",
            "",
        ),
        "outbound_date": deal.get("outbound_date", ""),
        "return_date": deal.get("return_date", ""),
        "departure_airport": deal.get(
            "departure_airport_code",
            "",
        ),
        "arrival_airport": deal.get(
            "arrival_airport_code",
            "",
        ),
        "flight_duration_minutes": deal.get(
            "flight_duration",
            "",
        ),
        "stops": deal.get("stops", ""),
        "airline": deal.get("airline", ""),
        "airline_code": deal.get("airline_code", ""),
        "google_flights_link": deal.get("flight_link", ""),
        "serpapi_flight_link": deal.get(
            "serpapi_flight_link",
            "",
        ),
    })

fieldnames = [
    "destination",
    "country",
    "destination_id",
    "price",
    "average_price",
    "discount_percentage",
    "outbound_date",
    "return_date",
    "departure_airport",
    "arrival_airport",
    "flight_duration_minutes",
    "stops",
    "airline",
    "airline_code",
    "google_flights_link",
    "serpapi_flight_link",
]

with open(
    "google_flights_deals.csv",
    "w",
    newline="",
    encoding="utf-8",
) as csv_file:
    writer = csv.DictWriter(
        csv_file,
        fieldnames=fieldnames,
    )
    writer.writeheader()
    writer.writerows(rows)

print(
    f"Saved {len(rows)} deals to "
    "google_flights_deals.csv"
)

This creates a file named google_flights_deals.csv with one row for each deal.

Exported result in CSV file

JavaScript implementation

SerpApi provides an official JavaScript package.

Install it with npm:

npm install serpapi

Create an index.js file:

const { getJson } = require("serpapi");

const SERPAPI_API_KEY = "YOUR_API_KEY";

getJson(
  {
    engine: "google_flights_deals",
    departure_id: "CGK",
    currency: "USD",
    api_key: SERPAPI_API_KEY,
  },
  (json) => {
    if (json.error) {
      console.error(json.error);
      return;
    }

    const departure = json.departure_informations || {};
    const deals = json.deals || [];

    console.log(
      `Flight deals from: ${
        departure.airport_name ||
        departure.city ||
        "Unknown departure"
      }\n`
    );

    deals.forEach((deal) => {
      console.log({
        destination: deal.name || "Unknown",
        country: deal.country || "Unknown",
        price: deal.price ?? "Not available",
        averagePrice:
          deal.average_price ?? "Not available",
        discountPercentage:
          deal.discount_percentage ?? "Not available",
        outboundDate:
          deal.outbound_date || "Not available",
        returnDate:
          deal.return_date || "One-way flight",
        route:
          `${deal.departure_airport_code || "?"} → ` +
          `${deal.arrival_airport_code || "?"}`,
        duration: deal.flight_duration ?? "Not available",
        stops: deal.stops ?? "Not available",
        airline: deal.airline || "Not available",
        flightLink: deal.flight_link || "Not available",
      });
    });
  }
);

Run the script:

node index.js

The official SerpApi JavaScript package exposes the getJson function and supports both callbacks and promises.

Customize Google Flights Deals results

The API provides several ways to customize the returned deals. Here are three useful examples.

Search using exact or flexible dates

For an exact round trip, provide one outbound_date and one return_date in YYYY-MM-DD format:

params = {
    "engine": "google_flights_deals",
    "departure_id": "FCO",
    "outbound_date": "2026-08-10",
    "return_date": "2026-08-17",
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

For flexible dates, use two comma-separated dates as the departure window:

params = {
    "engine": "google_flights_deals",
    "departure_id": "BCN",
    "outbound_date": "2026-09-01,2026-09-30",
    "travel_duration": 3,
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

Possible travel_duration values are:

Value	Duration
`1`	One week
`2`	Weekend
`3`	Two weeks

You can alternatively use trip_length for a custom duration:

params["trip_length"] = "5,10"

This requests trips lasting between 5 and 10 days.

travel_duration, trip_length, and return_date cannot be used together. For a round trip with one exact outbound date, one exact return date is required.

Filter by stops and maximum price

Use stops to control how many connections are allowed:

Value	Meaning
`0`	Any number of stops
`1`	Nonstop only
`2`	One stop or fewer
`3`	Two stops or fewer

You can combine it with max_price:

params = {
    "engine": "google_flights_deals",
    "departure_id": "CDG",
    "stops": 1,
    "max_price": 500,
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

This searches for nonstop deals costing no more than 500 in the selected currency.

Include or exclude airlines

Use include_airlines to return deals from selected airlines:

params = {
    "engine": "google_flights_deals",
    "departure_id": "/m/0947l",
    "include_airlines": "W4,FR",
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

The example includes:

W4 — Wizz Air
FR — Ryanair

Multiple airline codes are separated with commas.

You can use exclude_airlines instead when you want to remove specific airlines from the results:

params["exclude_airlines"] = "UA"

The include_airlines and exclude_airlines parameters cannot be used together.

The API also accepts the following alliance values:

STAR_ALLIANCE
SKYTEAM
ONEWORLD

Searching for one-way flight deals

Round trip is the default flight type. Set type to 2 to search for one-way deals:

params = {
    "engine": "google_flights_deals",
    "departure_id": "MAD",
    "type": 2,
    "currency": "USD",
    "api_key": SERPAPI_API_KEY,
}

The supported values are:

Value	Flight type
`1`	Round trip
`2`	One way

For a one-way search with a specific date, provide only outbound_date:

params["outbound_date"] = "2026-09-15"

Do not use return_date, travel_duration, or trip_length with one-way flights. One-way deal results do not contain an return_date.

What data can be retrieved from the Google Flights Deals API?

The API returns information about the departure location and a list of available flight deals.

The departure_informations object can include:

Departure airport name and IATA code
Departure city and city ID
Departure country
Airport latitude and longitude

The deals array can include:

Destination name and country
Destination Google Knowledge Graph ID
Deal price
Average price for the route
Discount percentage
Outbound and return dates
Departure and arrival airport codes
Flight duration in minutes
Number of stops
Airline name and code
Destination description and highlights
Destination thumbnail
Direct Google Flights link
SerpApi link for retrieving detailed flight information

Some fields, including average_price, discount_percentage, airline, and description, are not always present. The return_date field is also omitted for one-way flights.

Here is a simplified example of a deal:

{
  "destination_id": "/m/0k02q",
  "name": "São Vicente",
  "country": "Cape Verde",
  "price": 468,
  "average_price": 2023,
  "discount_percentage": 77,
  "outbound_date": "2026-09-21",
  "return_date": "2026-09-27",
  "departure_airport_code": "CDG",
  "arrival_airport_code": "VXE",
  "flight_duration": 350,
  "stops": 0,
  "airline": "Cabo Verde Airlines",
  "airline_code": "VR",
  "flight_link": "https://www.google.com/travel/flights?..."
}

The main results are available inside the deals array.

Scraping Google Flights

If you're interested in scraping the regular Google Flights website, we also have an API for that: Google Flights API

Here is the blog tutorial on how to scrape Google Flights.

Here is a video tutorial if you prefer to watch

https://www.youtube.com/embed/zqZSvuj7JMs?feature=oembed

Conclusion

The SerpApi Google Flights Deals API makes it possible to retrieve Google Flights deal discovery results as structured JSON.

With one request, you can collect destinations, deal prices, average prices, discount percentages, travel dates, airport codes, airlines, flight durations, stops, and links to Google Flights.

You can also customize the search using exact dates, flexible date ranges, trip durations, maximum prices, stop limits, travel classes, and airline filters.

For the complete and current parameter list, review the official Google Flights Deals API documentation.

Monitoring FIFA World Cup scores with a simple API

Josef Strzibny — Sun, 12 Jul 2026 14:33:47 +0000

World Cup data constantly changes. Matches kick off, goals arrive, red cards shift momentum, some teams are eliminated, and the bracket moves on. Instead of refreshing the tournament scores page by hand, we can monitor it with simple API calls. The Google Sports API from SerpApi returns live structured data for all kinds of sport events including the World Cup. This way we can learn about games, teams, scores, kickoff times, venues, goal summaries, and more.

Getting live structured World Cup data

We can use live data from the new Google Sports API to answer many questions during the World Cup. We can track individual match scores as well all answer all kinds of other things during the tournament:

How many matches are left in the tournament?
How much time is left until the final?
Which host nations are still alive?
What were the biggest knockout-stage upsets so far?
Which teams advanced despite red cards?
Who scored the most recent brace?
Who are the players on one's team?

This is because each of these questions maps to JSON fields returned by SerpApi's Google Sports API:

Signal	API data we need
Scoreboard status	match `status`, scores, latest finished game, next upcoming game
Matches left	match `status`, round names, games list
Tournament time remaining	`start_time` for upcoming games, plus the bracket tab for the final
Host nations still alive	team names, winners, knockout progression
Knockout-stage upsets	scores, winners, round names
Teams advancing despite red cards	team-level `win` and `red_card` fields
Recent braces	`goal_summary`, scorers, goal counts, match time

Let's go and have a closer look on how to run relevant queries for this kind of data.

Example of the game result from the SerpApi Google Sports API documentation

Using Google Sports API from SerpApi

SerpApi's Google Sports API retrieves structured data directly from Google Sports pages on every request. The only real requirement to use this API is to sign up for free. The API is quite simple with a single endpoint that needs to specify the google_sports engine:

GET https://serpapi.com/search.json?engine=google_sports

The important idea is that Google Sports data is organized around sports entities. In this post we use three of them:

League : the FIFA World Cup as a tournament, useful for schedules, groups, brackets, and many games at once.
Game : a single match, useful when we need more detail, such as goal summaries.
Team : a national team, useful for team-specific games and player lists.

To get the tournament data, we start with the FIFA World Cup league entity:

GET https://serpapi.com/search.json
  ?engine=google_sports
  &kgmid=/m/030q7
  &sp=ft
  &type=league
  &tab=gm
  &hl=en
  &gl=us
  &api_key=YOUR_API_KEY

Here are the parameters we will use most often:

Parameter	Meaning
`engine=google_sports`	Use SerpApi's Google Sports API
`kgmid=/m/030q7`	Google Knowledge Graph ID for the FIFA World Cup
`sp=ft`	Football. The API also supports other sports, such as basketball (`bs`), baseball (`bb`), cricket (`cr`), American football (`af`), ice hockey (`ih`), and rugby (`rg`)
`type=league`	Query a league or tournament entity
`tab=gm`	Return the games tab
`tab=br`	Return the bracket tab, useful for the full knockout tree
`type=game`	Query a specific match by its game `kgmid`
`type=team`	Query a specific team by its team `kgmid`
`hl=en`	Return English results
`gl=us`	Use United States location context

A shortened league response for an upcoming match looks like this:

{
  "league_results": {
    "game_groups": [
      {
        "title": "Quarter-finals",
        "games": [
          {
            "status": "upcoming",
            "start_time": "2026-07-09T20:00:00Z",
            "teams": [
              {
                "short_name": "France",
                "short_code": "FRA"
              },
              {
                "short_name": "Morocco",
                "short_code": "MAR"
              }
            ],
            "venue": {
              "name": "Boston Stadium",
              "location": "Foxborough"
            },
            "extensions": ["Quarter-finals"]
          }
        ]
      }
    ]
  }
}

A finished match includes scores and a winning team marker:

{
  "status": "finished",
  "status_original": "FT",
  "start_time": "2026-07-07T00:00:00Z",
  "teams": [
    {
      "name": "United States men's national soccer team",
      "short_code": "USA",
      "score": 1
    },
    {
      "short_name": "Belgium",
      "short_code": "BEL",
      "score": 4,
      "win": true
    }
  ],
  "extensions": ["Round of 16"]
}

💡 You can experiment with these parameters interactively in the SerpApi playground.

SerpApi provides official clients for many languages including JavaScript, Python, Ruby, and PHP that makes integration easy. We'll use the JavaScript package with Node.js but feel free to use your favourite language.

Setting up a simple Node.js monitor

Every request for the google_sports engine follows the same pattern. We choose the sports entity type, pass its kgmid, and set the sport with sp. For this post on current World Cup, the defaults are:

{
  engine: "google_sports",
  kgmid: "/m/030q7",
  sp: "ft",
  type: "league",
  tab: "gm",
  hl: "en",
  gl: "us"
}

This means the following:

use Google Sports
fetch the FIFA World Cup entity
treat it as football
query it as a league
and return the games tab

Using SerpApi's official JavaScript package, a minimal request looks like this:

const { getJson } = require("serpapi");

const data = await getJson({
  engine: "google_sports",
  kgmid: "/m/030q7",
  sp: "ft",
  type: "league",
  tab: "gm",
  hl: "en",
  gl: "us",
  api_key: process.env.SERPAPI_API_KEY
});

If you prefer not to use the package, the same request can be made as a regular GET request to https://serpapi.com/search.json.

The most important parameter to understand is type:

// Tournament-level data: games, bracket, standings, etc.
await getJson({
  engine: "google_sports",
  kgmid: "/m/030q7",
  sp: "ft",
  type: "league",
  tab: "gm",
  api_key: process.env.SERPAPI_API_KEY
});

// Single-match details, using a game kgmid from a previous response.
await getJson({
  engine: "google_sports",
  kgmid: "/g/11xmtfjhq2",
  sp: "ft",
  type: "game",
  api_key: process.env.SERPAPI_API_KEY
});

// Team-level data, using a team kgmid.
await getJson({
  engine: "google_sports",
  kgmid: "/m/0329gm",
  sp: "ft",
  type: "team",
  tab: "pl",
  api_key: process.env.SERPAPI_API_KEY
});

The tab parameter depends on the entity type:

Entity type	Useful tabs	What we use it for
`league`	`gm`, `br`, `sn`	Games, bracket, standings
`team`	`gm`, `pl`, `sn`	Team games, players, standings
`game`	usually no tab for football	Match details such as teams, score, goals, and metadata

For example, the World Cup games tab returns grouped games under league_results.game_groups. To work with those games, we usually flatten the groups first:

function getGames(data) {
  const groups = data.league_results?.game_groups || [];

  return groups.flatMap((group) =>
    (group.games || []).map((game) => ({
      ...game,
      group_title: group.title
    }))
  );
}

From there, each game object has the fields we need for most monitoring tasks: status, start_time, teams, venue, extensions, and often kgmid for fetching more detailed game data later.

One small cleanup helper is also useful. Some teams include short_name, while others only include a longer official name, such as "United States men's national soccer team". For display output, we can normalize that:

function teamName(team) {
  const name = team.short_name || team.name || team.short_code || "TBD";

  return name.replace(/\s+(?:men's\s+)?national\s+(?:football|soccer)\s+team$/i, "");
}

With those pieces in place, we can start building monitors answering questions we want from the JSON response.

Monitoring FIFA World Cup 2026

We'll use SerpApi's new Google Sports API to build a few small Node.js scripts for answering practical, real-time questions like What are the current scores?, How many matches are left?, Which host nations are still alive?, Who scored a brace?, and Which knockout results stand out?.

💡 The outputs shown below come from running the scripts against the live API while writing this post, so the exact numbers will naturally change as the tournament progresses.

What's the current scoreboard?

The most important thing we need to start with is the scoreboard. The following code will give you a quick summary for the World Cup and list of some latest finished games. Here's a full code that builds on top of the API call we discussed in the previous section but also filters through the relevant information:

#!/usr/bin/env node

const { getJson } = require("serpapi");

const WORLD_CUP_KGMID = "/m/030q7";

function getApiKey() {
  const apiKey = process.env.SERPAPI_API_KEY;

  if (!apiKey) {
    throw new Error("Missing SERPAPI_API_KEY. Set it with: export SERPAPI_API_KEY=your_key");
  }

  return apiKey;
}

async function serpapiSearch(params = {}) {
  const searchParams = Object.fromEntries(
    Object.entries({
      engine: "google_sports",
      kgmid: WORLD_CUP_KGMID,
      sp: "ft",
      type: "league",
      tab: "gm",
      hl: "en",
      gl: "us",
      api_key: getApiKey(),
      ...params
    }).filter(([, value]) => value !== undefined && value !== null)
  );

  const data = await getJson(searchParams);

  if (data.error) {
    throw new Error(`SerpApi error: ${data.error}`);
  }

  return data;
}

function getGames(data) {
  const groups = data.league_results?.game_groups || [];

  return groups.flatMap((group) =>
    (group.games || []).map((game) => ({
      ...game,
      group_title: group.title
    }))
  );
}

function teamName(team) {
  const name = team.short_name || team.name || team.short_code || "TBD";

  return name.replace(/\s+(?:men's\s+)?national\s+(?:football|soccer)\s+team$/i, "");
}

function scoreLine(game) {
  const teams = game.teams || [];

  return teams
    .map((team) => {
      const score = team.score_original ?? team.score;
      return score === undefined ? teamName(team) : `${teamName(team)} ${score}`;
    })
    .join(" - ");
}

function roundLabel(game) {
  const extension = (game.extensions || []).find((item) =>
    /Group Stage|Round of 32|Round of 16|Quarter-finals|Semi-finals|Third place play-off|Final/.test(item)
  );

  return extension || game.group_title || "Unknown round";
}

function formatUtc(isoString) {
  if (!isoString) return "unknown time";

  return `${isoString.replace("T", " ").replace("Z", "")} UTC`;
}

function summarizeStatuses(games) {
  const counts = new Map();

  for (const game of games) {
    counts.set(game.status, (counts.get(game.status) || 0) + 1);
  }

  return counts;
}

function printGames(title, games) {
  console.log(`\n${title}`);
  console.log("-".repeat(title.length));

  if (games.length === 0) {
    console.log("No games found.");
    return;
  }

  for (const game of games) {
    console.log(`- ${roundLabel(game)} — ${scoreLine(game)} — ${game.status} — ${formatUtc(game.start_time)}`);
  }
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data).sort(
    (a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0)
  );

  const finished = games.filter((game) => game.status === "finished");
  const liveOrOther = games.filter((game) => !["finished", "upcoming"].includes(game.status));
  console.log("World Cup score monitor");
  console.log(`Fetched ${games.length} games from SerpApi Google Sports API.`);

  console.log("\nStatus summary");
  console.log("--------------");
  for (const [status, count] of summarizeStatuses(games)) {
    console.log(`- ${status}: ${count}`);
  }

  printGames("Live or in-progress matches", liveOrOther);
  printGames("Latest finished scores", finished.slice(-8));
}

main().catch((error) => {
  console.error(error.message);
  process.exit(1);
});

It looks long but most of the code are just helpers for formatting the information.

To run it, saved it as a file, and type:

node score-monitor.js

You should get a similar results back, giving you a quick glance on the whole live tournament:

World Cup score monitor
Fetched 40 games from SerpApi Google Sports API.

Status summary
--------------
- finished: 32
- upcoming: 8

Live or in-progress matches
---------------------------
No games found.

Latest finished scores
----------------------
- Round of 16 — Canada 0 - Morocco 3 — finished — 2026-07-04 17:00:00 UTC
- Round of 16 — Paraguay 0 - France 1 — finished — 2026-07-04 21:00:00 UTC
- Round of 16 — Brazil 1 - Norway 2 — finished — 2026-07-05 20:00:00 UTC
- Round of 16 — Mexico 2 - England 3 — finished — 2026-07-06 01:00:00 UTC
- Round of 16 — Portugal 0 - Spain 1 — finished — 2026-07-06 19:00:00 UTC
- Round of 16 — United States 1 - Belgium 4 — finished — 2026-07-07 00:00:00 UTC
- Round of 16 — Argentina 3 - Egypt 2 — finished — 2026-07-07 16:00:00 UTC
- Round of 16 — Switzerland 0 (4) - Colombia 0 (3) — finished — 2026-07-07 20:00:00 UTC

This way we can build nice dashboards, especially if we would later format the data into HTML.

How many matches are left?

We already printed how many games are left, but we don't know which matches will take place. For that we can define a few additional functions:

const KNOCKOUT_ROUND_PATTERN = /Round of 32|Round of 16|Quarter-finals|Semi-finals|Third place play-off|Final/;

function getBracketGames(data) {
  const brackets = data.league_results?.brackets || [];

  return brackets.flatMap((bracket) =>
    (bracket.stages || []).flatMap((stage) =>
      (stage.games || []).map((game) => ({
        ...game,
        group_title: stage.title
      }))
    )
  );
}

function matchup(game) {
  const teams = game.teams || [];

  if (teams.length === 0) return "teams to be determined";

  return teams.map(teamName).join(" vs ");
}

function roundLabel(game) {
  const extension = (game.extensions || []).find((item) => KNOCKOUT_ROUND_PATTERN.test(item));

  return extension || game.group_title || "Unknown round";
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  let upcomingGames = games
    .filter((game) => game.status === "upcoming")
    .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));

  // The games tab returns a rolling window of relevant games, which may not
  // reach the Final. The bracket tab always covers the full knockout tree.
  let usedBracket = false;

  if (!upcomingGames.some((game) => roundLabel(game) === "Final")) {
    const bracketData = await serpapiSearch({ tab: "br" });
    const knownKgmids = new Set(upcomingGames.map((game) => game.kgmid));
    const missingGames = getBracketGames(bracketData).filter(
      (game) => game.status === "upcoming" && !knownKgmids.has(game.kgmid)
    );

    if (missingGames.length > 0) {
      usedBracket = true;
      upcomingGames = upcomingGames
        .concat(missingGames)
        .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));
    }
  }

  const breakdown = new Map();

  for (const game of upcomingGames) {
    const round = roundLabel(game);
    breakdown.set(round, (breakdown.get(round) || 0) + 1);
  }

  console.log(`Matches left: ${upcomingGames.length}`);

  if (usedBracket) {
    console.log("(games tab window did not include the Final; completed from the bracket tab)");
  }

  console.log("\nBreakdown by round:");

  for (const [round, count] of breakdown) {
    console.log(`- ${round}: ${count}`);
  }

  console.log("\nUpcoming matches:");

  for (const game of upcomingGames) {
    console.log(`- ${formatUtc(game.start_time)} — ${roundLabel(game)} — ${matchup(game)}`);
  }

  const thirdPlaceGames = upcomingGames.filter((game) => roundLabel(game) === "Third place play-off");
  const championPathCount = upcomingGames.length - thirdPlaceGames.length;

  console.log(`\nMatches on the path to crowning the champion: ${championPathCount}`);
}

There is one practical detail to handle. The games tab returns a rolling window of currently relevant games. In this run the window covered everything up to the Final, but a day earlier it ended at the third-place play-off. The bracket tab (tab=br) always covers the full knockout tree, so the script completes the list from there whenever the Final is missing:

if (!upcomingGames.some((game) => roundLabel(game) === "Final")) {
  const bracketData = await serpapiSearch({ tab: "br" });
  const knownKgmids = new Set(upcomingGames.map((game) => game.kgmid));

  const missingGames = getBracketGames(bracketData).filter(
    (game) => game.status === "upcoming" && !knownKgmids.has(game.kgmid)
  );

  upcomingGames = upcomingGames.concat(missingGames);
}

Save the new code and run it:

node matches-left.js

Note that I haven't repeated the same functions from before. You can, however, run the full script from GitHub.

And here's the output:

Matches left: 8

Breakdown by round:
- Quarter-finals: 4
- Semi-finals: 2
- Third place play-off: 1
- Final: 1

Upcoming matches:
- 2026-07-09 20:00:00 UTC — Quarter-finals — France vs Morocco
- 2026-07-10 19:00:00 UTC — Quarter-finals — Spain vs Belgium
- 2026-07-11 21:00:00 UTC — Quarter-finals — Norway vs England
- 2026-07-12 01:00:00 UTC — Quarter-finals — Argentina vs Switzerland
- 2026-07-14 19:00:00 UTC — Semi-finals — teams to be determined
- 2026-07-15 19:00:00 UTC — Semi-finals — teams to be determined
- 2026-07-18 21:00:00 UTC — Third place play-off — teams to be determined
- 2026-07-19 19:00:00 UTC — Final — teams to be determined

Matches on the path to crowning the champion: 7

How much time is left in the tournament?

If we are interested how far are we in the tournament in terms the total time left, we can define a function for the duration and try to calculate it:

function pluralize(count, singular, plural = `${singular}s`) {
  return `${count} ${count === 1 ? singular : plural}`;
}

function formatDuration(milliseconds) {
  const totalHours = Math.floor(milliseconds / 1000 / 60 / 60);
  const days = Math.floor(totalHours / 24);
  const hours = totalHours % 24;

  return `${pluralize(days, "day")} and ${pluralize(hours, "hour")}`;
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  const upcomingGames = games
    .filter((game) => game.status === "upcoming" && game.start_time)
    .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));

  if (upcomingGames.length === 0) {
    console.log("No upcoming matches found.");
    return;
  }

  const nextMatch = upcomingGames[0];
  let final = upcomingGames.find((game) => roundLabel(game) === "Final");

  // Fall back to the bracket tab when the games window ends before the Final.
  if (!final) {
    const bracketData = await serpapiSearch({ tab: "br" });
    final = getBracketGames(bracketData).find(
      (game) => roundLabel(game) === "Final" && game.status === "upcoming" && game.start_time
    );
  }

  console.log(`Next match: ${matchup(nextMatch)}`);
  console.log(`Next kickoff: ${formatUtc(nextMatch.start_time)}`);

  if (final) {
    const untilFinal = new Date(final.start_time) - Date.now();
    const span = new Date(final.start_time) - new Date(nextMatch.start_time);

    console.log(`Final kickoff: ${formatUtc(final.start_time)}`);
    console.log(`Time left until the final kicks off: ${formatDuration(untilFinal)}`);
    console.log(`Calendar span from next kickoff to final kickoff: ${formatDuration(span)}`);
  } else {
    console.log("Final kickoff: not found in the games or bracket responses");
  }

  const regulationMinutes = upcomingGames.length * 90;
  const withHalftimeMinutes = regulationMinutes + upcomingGames.length * 15;

  console.log("\nEstimated watching time for the remaining matches:");
  console.log(`- Regulation only: ${regulationMinutes} minutes (${(regulationMinutes / 60).toFixed(1)} hours)`);
  console.log(`- With halftime: ${withHalftimeMinutes} minutes (${(withHalftimeMinutes / 60).toFixed(1)} hours)`);
  console.log("- With stoppage time, extra time, and penalties: add more time depending on match events");
}

The same response includes start_time for upcoming games. We sort upcoming matches by kickoff time and look for the Final. We fall back to the bracket tab if the games window ends before it:

const upcomingGames = games
  .filter((game) => game.status === "upcoming" && game.start_time)
  .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));

const nextMatch = upcomingGames[0];
let final = upcomingGames.find((game) => roundLabel(game) === "Final");

if (!final) {
  const bracketData = await serpapiSearch({ tab: "br" });
  final = getBracketGames(bracketData).find(
    (game) => roundLabel(game) === "Final" && game.status === "upcoming"
  );
}

To run it, save it as a file, and type:

node time-left.js

And here's the output:

Next match: France vs Morocco
Next kickoff: 2026-07-09 20:00:00 UTC
Final kickoff: 2026-07-19 19:00:00 UTC
Time left until the final kicks off: 10 days and 9 hours
Calendar span from next kickoff to final kickoff: 9 days and 23 hours

Estimated watching time for the remaining matches:
- Regulation only: 720 minutes (12.0 hours)
- With halftime: 840 minutes (14.0 hours)
- With stoppage time, extra time, and penalties: add more time depending on match events

So the tournament wraps up with the Final at New York New Jersey Stadium on July 19, and a fan committed to every remaining match is looking at roughly 14 hours of football.

Which host nations are still alive?

It's often said that host nations have the advantage of local fans. If we want to track the results of host nation teams we could filter them based on their codes like in this following example:

const HOST_CODES = ["CAN", "MEX", "USA"];

function scoreLine(game) {
  const teams = game.teams || [];

  return teams
    .map((team) => {
      const score = team.score_original ?? team.score;
      return score === undefined ? teamName(team) : `${teamName(team)} ${score}`;
    })
    .join(" - ");
}

function roundLabel(game) {
  const extension = (game.extensions || []).find((item) => KNOCKOUT_ROUND_PATTERN.test(item));

  return extension || game.group_title || "Unknown round";
}

function isKnockout(game) {
  return KNOCKOUT_ROUND_PATTERN.test(
    `${game.group_title || ""} ${(game.extensions || []).join(" ")}`
  );
}

function hostStatus(games, hostCode) {
  const hostGames = games
    .filter((game) => (game.teams || []).some((team) => team.short_code === hostCode))
    .sort((a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0));

  if (hostGames.length === 0) {
    return "not present in the current games window";
  }

  for (const game of hostGames) {
    if (game.status !== "finished" || !isKnockout(game)) continue;

    const host = (game.teams || []).find((team) => team.short_code === hostCode);
    const winner = (game.teams || []).find((team) => team.win);

    if (winner && host && !host.win) {
      return `eliminated by ${teamName(winner)} — ${scoreLine(game)} (${roundLabel(game)})`;
    }
  }

  const nextGame = hostGames.find((game) => game.status === "upcoming");

  if (nextGame) {
    return `still alive — next match: ${roundLabel(nextGame)}`;
  }

  return "still alive";
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);

  console.log("Host nation status:\n");

  for (const hostCode of HOST_CODES) {
    const hostGame = games.find((game) =>
      (game.teams || []).some((team) => team.short_code === hostCode)
    );
    const hostTeam = (hostGame?.teams || []).find((team) => team.short_code === hostCode);
    const name = hostTeam ? teamName(hostTeam) : hostCode;

    console.log(`- ${name}: ${hostStatus(games, hostCode)}`);
  }
}

For this tournament, the host nations are:

Canada
Mexico
United States

To monitor their status, we can inspect knockout-stage games involving those teams and check whether they won or were eliminated.

const hostCodes = new Set(["CAN", "MEX", "USA"]);

const hostKnockoutGames = games.filter((game) => {
  const hasHost = (game.teams || []).some((team) => hostCodes.has(team.short_code));

  return isKnockout(game) && hasHost;
});

The isKnockout helper matches the round name against the knockout rounds (Round of 32 through Final) found in each game's extensions.

Again, I have shorten it to not repeat previous functions. Save and run as:

node host-nations.js

And here's the output:

Host nation status:

- Canada: eliminated by Morocco — Canada 0 - Morocco 3 (Round of 16)
- Mexico: eliminated by England — Mexico 2 - England 3 (Round of 16)
- United States: eliminated by Belgium — United States 1 - Belgium 4 (Round of 16)

So the answer for the current 2026 World Cup is straightforward:

None. All three host nations were eliminated before the quarter-finals.

What were the biggest knockout-stage upsets so far?

The next question is a fun one. We want to find out when a heavily favored team is defeated by a significantly weaker opponent (the "underdog").

// Editorial choice: traditional favorites losing to anyone outside this list
// counts as a possible upset.
const FAVORITES = new Set(["BRA", "ARG", "FRA", "ESP", "ENG", "GER", "POR"]);

(...)

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  const knockoutGames = games.filter(
    (game) => isKnockout(game) && game.status === "finished"
  );

  const possibleUpsets = knockoutGames.filter((game) => {
    const teams = game.teams || [];
    const winner = teams.find((team) => team.win);
    const loser = teams.find((team) => !team.win);

    return (
      winner &&
      loser &&
      FAVORITES.has(loser.short_code) &&
      !FAVORITES.has(winner.short_code)
    );
  });

  console.log("Possible knockout-stage upsets:\n");

  if (possibleUpsets.length === 0) {
    console.log("No upsets found by this definition.");
    return;
  }

  for (const game of possibleUpsets) {
    const winner = (game.teams || []).find((team) => team.win);
    const loser = (game.teams || []).find((team) => !team.win);

    console.log(
      `- ${teamName(winner)} beat ${teamName(loser)} — ${scoreLine(game)} (${roundLabel(game)})`
    );
  }
}

This monitor is a little more interpretive. We then add our own definition of what counts as an upset. One simple approach to find them is to keep a list of traditional favorite teams and see where one of them loses to a team outside that list:

const favorites = new Set(["BRA", "ARG", "FRA", "ESP", "ENG", "GER", "POR"]);

const possibleUpsets = knockoutGames.filter((game) => {
  const teams = game.teams || [];
  const winner = teams.find((team) => team.win);
  const loser = teams.find((team) => !team.win);

  return winner && loser && favorites.has(loser.short_code) && !favorites.has(winner.short_code);
});

We can again save this and run as:

node upsets.js

And here's a possible output:

Possible knockout-stage upsets:

- Paraguay beat Germany — Germany 1 (3) - Paraguay 1 (4) (Round of 32)
- Norway beat Brazil — Brazil 1 - Norway 2 (Round of 16)

Finding out these upsets might be a little bit personal, but hopefully you don't mind the logic here.

Which teams ended up with red cards?

Red cards mean the opposite of fair play. If you want to know which teams ended up with a red card during their games, you can ask on team.read_card:

function hasRedCard(game) {
  return (game.teams || []).some((team) => team.red_card);
}

function gameKey(game) {
  return game.kgmid || `${game.start_time || "unknown"}|${matchup(game)}|${roundLabel(game)}`;
}

function mergeGames(games) {
  const gamesByKey = new Map();

  for (const game of games) {
    const key = gameKey(game);
    const existing = gamesByKey.get(key);

    if (!existing || (!hasRedCard(existing) && hasRedCard(game))) {
      gamesByKey.set(key, game);
    }
  }

  return [...gamesByKey.values()].sort(
    (a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0)
  );
}

async function main() {
  const [gamesData, bracketData] = await Promise.all([
    serpapiSearch({ tab: "gm" }),
    serpapiSearch({ tab: "br" })
  ]);
  const games = mergeGames([
    ...getGames(gamesData),
    ...getBracketGames(bracketData)
  ]);

  const redCards = games.flatMap((game) =>
    (game.teams || [])
      .filter((team) => team.red_card)
      .map((team) => ({ team, game }))
  );

  console.log("Red cards in the tournament:\n");

  if (redCards.length === 0) {
    console.log("No red cards found in the returned tournament data.");
    return;
  }

  for (const { team, game } of redCards) {
    console.log(
      `- ${teamName(team)} — ${roundLabel(game)} — ${scoreLine(game)} — ${matchup(game)} — ${formatUtc(game.start_time)}`
    );
  }

  console.log(`\nTotal red-card team entries: ${redCards.length}`);
}

Save and run as:

node red-cards.js

And here's the output:

Red cards in the tournament:

- Ecuador — Round of 32 — Mexico 2 - Ecuador 0 — Mexico vs Ecuador — 2026-07-01 02:00:00 UTC
- United States — Round of 32 — United States 2 - Bosnia and Herzegovina 0 — United States vs Bosnia and Herzegovina — 2026-07-02 00:00:00 UTC
- England — Round of 16 — Mexico 2 - England 3 — Mexico vs England — 2026-07-06 01:00:00 UTC

Total red-card team entries: 3

Looks like there were 3 red cards so far.

Who scored the most recent brace?

You might know hat-tricks. They are amazing but not that common. There is also a term that sits below it. Braces are where a single player scores two goals in a single match. Using the structured data from the Google Sports API, we can try to find the last player who successfully scored a brace:

async function fetchGameDetails(game) {
  if (!game.kgmid) return game;

  const data = await serpapiSearch({
    kgmid: game.kgmid,
    type: "game",
    tab: undefined
  });

  return {
    ...game,
    ...(data.game_results?.info || {})
  };
}

function formatGoalMinute(goal) {
  const time = goal.in_game_time || {};
  const minute = time.minute;
  const stoppage = time.stoppage_minute;

  if (minute === undefined) return "unknown minute";
  if (stoppage !== undefined) return `${minute}+${stoppage}'`;

  return `${minute}'`;
}

function findBracesInGame(game) {
  const braces = [];

  for (const team of game.teams || []) {
    for (const summary of team.goal_summary || []) {
      const goals = summary.goals || [];

      if (goals.length >= 2) {
        const opponents = (game.teams || []).filter((otherTeam) => otherTeam !== team);

        braces.push({
          player: summary.player?.name || summary.player?.short_name || "Unknown player",
          team,
          goals,
          opponents,
          game
        });
      }
    }
  }

  return braces;
}

async function fetchGameDetails(game) {
  if (!game.kgmid) return game;

  const data = await serpapiSearch({
    kgmid: game.kgmid,
    type: "game",
    tab: undefined
  });

  return {
    ...game,
    ...(data.game_results?.info || {})
  };
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  const finishedGames = games
    .filter((item) => item.status === "finished")
    .sort((a, b) => new Date(b.end_time || b.start_time || 0) - new Date(a.end_time || a.start_time || 0));

  let mostRecentBrace;

  for (const game of finishedGames) {
    const detailedGame = findBracesInGame(game).length > 0 ? game : await fetchGameDetails(game);
    const braces = findBracesInGame(detailedGame);

    if (braces.length > 0) {
      mostRecentBrace = braces[0];
      break;
    }
  }

  if (!mostRecentBrace) {
    console.log("No braces found in the returned API data.");
    return;
  }

  const { player, team, goals, opponents, game } = mostRecentBrace;
  const goalMinutes = goals.map(formatGoalMinute).join(", ");

  console.log("Most recent brace found:\n");
  console.log(`Player: ${player}`);
  console.log(`Team: ${teamName(team)}`);
  console.log(`Opponent: ${opponents.map(teamName).join(", ")}`);
  console.log(`Match: ${matchup(game)}`);
  console.log(`Score: ${scoreLine(game)}`);
  console.log(`Round: ${roundLabel(game)}`);
  console.log(`Match time: ${formatUtc(game.start_time)}`);
  console.log(`Goal minutes: ${goalMinutes}`);
}

For this signal, we need game-level goal data. The league games response usually does not include goal_summary, so the script fetches game details for recent finished games via the game entity type. Passing tab: undefined removes the league-level tab=gm default from the shared helper, since game entities have no tabs:

const data = await serpapiSearch({
  kgmid: game.kgmid,
  type: "game",
  tab: undefined
});

const detailedGame = {
  ...game,
  ...(data.game_results?.info || {})
};

The detailed response includes a goal_summary per team:

{
  "player": {
    "name": "Charles De Ketelaere",
    "position": "Striker",
    "jersey_number": "17"
  },
  "goals": [
    { "in_game_time": { "minute": 9 } },
    { "in_game_time": { "minute": 33 } }
  ]
}

The function findBracesInGame is currently looking for 2 or more goals, so it will also find hat-tricks (3x), hauls (4x), and double hat-tricks (5x).

Save and run as:

node most-recent-brace.js

And see the output:

Most recent brace found:

Player: Charles De Ketelaere
Team: Belgium
Opponent: United States
Match: United States vs Belgium
Score: United States 1 - Belgium 4
Round: Round of 16
Match time: 2026-07-07 00:00:00 UTC
Goal minutes: 9', 33'

We found Charles De Ketelaere from Belgium who scored twice in the game against the United States.

What is the team journey in the tournament?

Finally, let's find out more about the teams themselves. The SerpApi's new Google Sports API supports type=team, so we can fetch team-specific data with the team's kgmid as well. There are plenty of things we might be interested in. Let's look at World Cup games in the current league, the last match, and the team structure:

const TEAM = {
  code: process.env.TEAM_CODE || "BEL",
  kgmid: process.env.TEAM_KGMID || "/m/0329gm",
  fallbackName: process.env.TEAM_NAME || "Belgium"
};

async function main() {
  const [leagueData, playersData] = await Promise.all([
    serpapiSearch({ tab: "gm" }),
    serpapiSearch({ kgmid: TEAM.kgmid, type: "team", tab: "pl" })
  ]);

  const games = getGames(leagueData);
  const teamGames = games
    .filter((game) => findTeamInGame(game))
    .sort((a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0));

  const players = playersData.team_results?.players || [];
  const inferredTeam = teamGames.map(findTeamInGame).find(Boolean);
  const displayName = inferredTeam ? teamName(inferredTeam) : TEAM.fallbackName;
  const nextGame = teamGames.find((game) => game.status === "upcoming");
  const latestFinished = [...teamGames].reverse().find((game) => game.status === "finished");

  console.log(`Team monitor: ${displayName}`);
  console.log(`Team code: ${TEAM.code}`);
  console.log(`Team kgmid: ${TEAM.kgmid}`);

  console.log("\nWorld Cup games in the current league response:");
  if (teamGames.length === 0) {
    console.log("- No games found for this team in the current league response.");
  } else {
    for (const game of teamGames) {
      const label = game.status === "upcoming" ? matchup(game) : scoreLine(game);
      console.log(`- ${roundLabel(game)} — ${label} — ${game.status} — ${formatUtc(game.start_time)}`);
    }
  }

  if (latestFinished) {
    console.log(`\nLatest finished World Cup match: ${scoreLine(latestFinished)} (${roundLabel(latestFinished)})`);
  }

  if (nextGame) {
    console.log(`Next World Cup match: ${matchup(nextGame)} (${roundLabel(nextGame)}) at ${formatUtc(nextGame.start_time)}`);
  }

  console.log("\nTeam players from Google Sports team endpoint:");
  console.log(`Players returned: ${players.length}`);

  for (const [position, count] of positionBreakdown(players)) {
    console.log(`- ${position}: ${count}`);
  }

  console.log("\nSample players:");
  for (const player of players.slice(0, 10)) {
    console.log(`- ${player.name}${player.position ? ` — ${player.position}` : ""}`);
  }
}

The example defaults to Belgium because Belgium is still alive in the current response (at the time of the writing) and even has a recent match with detailed goal data:

const playersData = await serpapiSearch({
  kgmid: "/m/0329gm",
  type: "team",
  tab: "pl"
});

const players = playersData.team_results?.players || [];

The script also reuses the World Cup league response to list Belgium's World Cup games in the current tournament window.

Save and run as usual:

node team-info.js

And here's the output for the provided Belgian team:

Team monitor: Belgium
Team code: BEL
Team kgmid: /m/0329gm

World Cup games in the current league response:
- Group Stage · Fri, Jun 26 — New Zealand 1 - Belgium 5 — finished — 2026-06-27 03:00:00 UTC
- Round of 32 — Belgium 3 - Senegal 2 — finished — 2026-07-01 20:00:00 UTC
- Round of 16 — United States 1 - Belgium 4 — finished — 2026-07-07 00:00:00 UTC
- Quarter-finals — Spain vs Belgium — upcoming — 2026-07-10 19:00:00 UTC

Latest finished World Cup match: United States 1 - Belgium 4 (Round of 16)
Next World Cup match: Spain vs Belgium (Quarter-finals) at 2026-07-10 19:00:00 UTC

Team players from Google Sports team endpoint:
Players returned: 26
- Defender: 9
- Forward: 7
- Goalkeeper: 3
- Midfielder: 7

Sample players:
- Jérémy Doku — Forward
- Kevin De Bruyne — Midfielder
- Romelu Lukaku — Forward
- Youri Tielemans — Midfielder
- Thibaut Courtois — Goalkeeper
- Senne Lammens — Goalkeeper
- Leandro Trossard — Forward
- Charles De Ketelaere — Forward
- Axel Witsel — Midfielder
- Amadou Onana — Midfielder

This is a useful complement to score monitoring. Once a team appears in the next fixture or a standout result, we can fetch its player list and recent team context with the same API. You can of course point the same script at another team by passing a different team code and kgmid.

Google Sports API examples

You can find all these World Cup API examples on our GitHub page. Checkout the repository and provide a SerpApi API key to run them individually with Node:

export SERPAPI_API_KEY=your_serpapi_api_key

node google-sports-api-world-cup/00-score-monitor.js
node google-sports-api-world-cup/01-matches-left.js
node google-sports-api-world-cup/02-time-left.js
node google-sports-api-world-cup/03-host-nations.js
node google-sports-api-world-cup/04-upsets.js
node google-sports-api-world-cup/05-red-cards.js
node google-sports-api-world-cup/06-most-recent-brace.js
node google-sports-api-world-cup/07-team-info.js

If you want SerpApi to fetch fresh results instead of using a cached response, set:

export SERPAPI_NO_CACHE=true

Wrapping up

SerpApi's Google Sports API is useful not only for keeping an eye on World Cup scores, but also for monitoring other parts of the tournament. With a few small scripts fetching the structured data live, we can track match status, remaining games, host nation eliminations, red cards, and more.

All of that makes the API a good fit for live dashboards, alerts, editorial tools, sports newsletters, and lightweight automation around major tournaments. See the Google Sports API documentation for yourself or try your own queries in the playground that lets you see the results before any implementation.

A complete guide to web scraping with Ruby (2026)

Josef Strzibny — Fri, 10 Jul 2026 14:33:26 +0000

Web scraping is the process of pulling out structured data from websites. If you're familiar with Ruby, you're in luck_._ Ruby has excellent tools you can use to scrape the web. From HTTP libraries to fast HTML parsers, Ruby tools cover all concerns of scraping. Paired with its developer-friendly syntax and productive frameworks, it's a great option for small scripts as well as complex applications.

Ruby as a language for web scraping

Ruby is a practical language for web scraping because it is good at the unglamorous parts of it all. The small scripts, readable objects, HTTP requests, background jobs, parsing JSON, databases, writing CSV files and glueing everything together. Some other languages might be more common for web scraping, but Ruby is an amazing choice too.

If you already work with Sinatra, Hanami, Ruby on Rails, Sidekiq, Active Job, or plain Ruby scripts, staying with Ruby for your scraping needs makes the most sense. You can scrape a page, parse the data, store it in your database, schedule a job, and expose the result in your web application without crossing language boundaries or adding a separate data pipeline just for web scraping.

Ruby is designed as a programmer's best friend

What will we learn

When it comes to web scraping, it's important to choose the right approach for the page in front of you. A static HTML page, a JavaScript-rendered interface, a form with cookies, a JSON endpoint, and a full website crawl are all different problems. Treating them as the same problem is where most scraping projects get messy.

This guide will take you through it all. We will map the main ways to scrape websites with Ruby and choose the right tools for each job. Then we will build up from HTML scraping to JavaScript-rendered pages, browser automation, crawling, JSON APIs, and data export. Finally, we touch production concerns like reliability and performance.

💡 Don't use Ruby? Have a look at our guides for web scraping in JavaScript and web scraping in Python.

The main ways to scrape websites with Ruby

Not every scraping is the same. Sometimes the data is already in the HTML, and a simple Faraday and Nokogiri script is enough. Sometimes the page is rendered by JavaScript and you need a browser tool like Selenium, Ferrum, Capybara, or Watir. Other times the approach is not to scrape HTML at all, but to call the same JSON endpoint the frontend uses.

Before picking a gem, figure out what kind of page you are dealing with. The best Ruby web scraping tool depends on where the data lives, how the page loads, and how much of the site you need to collect. A static product page, a React search interface, a login form, a JSON endpoint, and a full-site crawl can all fall under "web scraping," but they should not all be solved the same way.

1. Fetching a single static HTML page

This is the simplest case where you want to request and process one individual server rendered page. You send an HTTP request, receive HTML, parse the response, and extract the fields you need.

Use this when:

the data is already present in the initial HTML
you only need one page or a known list of URLs
you do not need to click, scroll, log in, or run JavaScript

Good Ruby tools for this job:

httparty for quick simple requests
Faraday for more complex HTTP requests
Nokogiri for parsing HTML

This is the best starting point for most Ruby scrapers. It is fast, simple, and easy to debug.

Browser's View Source can reveal if we see a fully formed HTML

2. Scraping JavaScript-rendered pages

Some pages return very little useful HTML at first. The browser loads JavaScript, calls APIs, updates the DOM, and only then displays the data you want. In that case, a plain HTTP request may not be enough. Nokogiri can only parse the HTML you give it to it as it cannot run JavaScript.

Use this when:

the data is missing from browser's "View Source"
the site uses React, Vue, Svelte, Angular, or another frontend framework
content appears only after the browser loads, waits, scrolls, or clicks

Good Ruby tools for this job:

Selenium for full browser automation driver
Ferrum for rendering pages in headless Chrome
Cuprite for using headless Chrome through Capybara

Before reaching for a browser, check the Network tab. Many JavaScript-heavy pages get their data from JSON endpoints you can request directly.

3. Using a headless or visible browser

A headless browser is a real browser without a visible window. It runs JavaScript, stores cookies, follows redirects, loads dynamic content, and behaves much more like a human browsing session than a plain HTTP client.

Use a headless browser when:

JavaScript rendering is required
you need to wait for elements to appear
you need browser APIs, cookies, local storage, or client-side navigation
the page behaves differently outside a browser

Use a visible browser when:

you are developing or debugging a scraper
you need to watch what the page is doing
popups, consent banners, or complex interactions are getting in the way

Good Ruby tools for this job:

Selenium when you want direct control over Chrome or another browser
Ferrum when you want direct control over headless Chrome through the Chrome DevTools Protocol
Capybara when you want a higher-level DSL for visits, clicks, forms, waits, and screenshots
Watir when you want a readable browser automation API built around WebDriver

Browser automation is powerful, but it is also slower and heavier. If a simple Faraday request can get the same data, do not use a browser.

💡 A little later we'll see how we can request JSON directly with some website which is a good alternative to full browser scraping.

4. Submitting forms, keep cookies, and work with sessions

Some sites require state. You might need to submit a search form, keep cookies, follow redirects, or use an authenticated session you are allowed to access.

Use this when:

the site has regular HTML forms
you need cookies or session persistence
you need to follow links like a browser
you need login-style behavior without JavaScript rendering

Good Ruby tools for this job:

Mechanize for forms, links, cookies, and redirects
Faraday with cookie support for custom HTTP flows
Selenium , Capybara , or Watir when JavaScript is part of the interaction

Mechanize is often the right tool for older, form-based sites. It is much lighter than running a browser, but it will not execute JavaScript.

5. Calling JSON endpoints instead of scraping HTML

Sometimes the page is mostly a frontend for an API. If you open the Network tab and inspect Fetch/XHR requests, you may find a clean JSON response containing the data you need.

Use this when:

the frontend loads data from a JSON endpoint
the HTML is hard to parse but the JSON is structured
the endpoint is public and permitted for your use case

Good Ruby tools for this job:

httparty for quick JSON requests in small scripts
Faraday for structured API requests with headers, timeouts, and retries
Ruby's built-in JSON library for parsing responses

This can be faster and more reliable than parsing HTML. But be careful with private, undocumented, or authenticated endpoints. Just because your browser can see a request does not automatically mean you can reuse it freely.

6. Crawling a website instead of scraping one page

Scraping means extracting data. Crawling means discovering pages. If you already know the URLs, you probably do not need a crawler. But if the job is to find pages, follow links, avoid duplicates, and keep going until a limit is reached, you are building one.

Example of crawling flow

Use this when:

you need to scrape many pages
you do not know every URL upfront
you need to follow internal links
you need depth limits, deduplication, queues, or sitemap discovery

Good Ruby tools for this job:

Spidr for quick site crawling
Anemone for spidering workflows
Addressabe for safer URL handling
a custom queue when you need production-level control

Small crawlers can live in memory. Larger crawlers usually need persistence, retries, rate limits, and a database or queue so they can resume after a failure.

Choosing the right Ruby web scraping tool

The rule of thumb is to use the lightest tool that can reliably get the data. Start with HTTP and Nokogiri. Move to Mechanize if you need forms, cookies, and sessions without JavaScript. Move to Selenium, Ferrum, Capybara, or Watir when you need JavaScript rendering or real browser interactions. Use a crawler when you need to discover pages, not just scrape known URLs.

Situation	Best approach	Ruby tools
Data is in the initial HTML	HTTP + parser	`Faraday` + `Nokogiri`
You need a quick request script	Simple HTTP helper	`HTTParty`
You need forms, cookies, redirects, and links without JavaScript	Browser-like HTTP automation	`Mechanize`
You need JavaScript rendering	Headless browser	`Ferrum`, `Selenium`
You need clicks, waits, screenshots, or testing-style flows	Browser automation DSL	`Capybara`, `Watir`
Data comes from a JSON endpoint	API request	`Faraday`, `JSON`
You need to discover many pages	Crawler	`Spidr`, `Anemone`, custom queue
You are scraping inside a Rails app	Background job	`ActiveJob`, `Sidekiq`, `Solid Queue`

This decision matters because every step up adds cost. Browsers are slower than HTTP requests. Crawlers are more complex than one-page scrapers. Authenticated sessions are more fragile than public pages. The best scraper is usually the simplest one that still works reliably.

Scraping static HTML pages with Ruby

Static scraping is the simplest way to scrape a website. It works when the data you need is already present in the HTML returned by the server. If you fetch the page with an HTTP client, the product names, prices, links, article text, or other fields are already sitting in a response body.

If the page loads an empty shell and fills it in later with React, Vue, Svelte, or a browser-side API request, parsers like Nokogiri will not see the rendered content. Nokogiri parses HTML and it does not run JavaScript. That makes static scraping way faster but not always possible as opposed to JavaScript-rendered scraping.

A quick way to check: open "View Source" in your browser and search for the text you want to scrape. If the text is there, a static scraper is probably enough. If it only appears in the Elements panel after JavaScript runs, you may need a browser or a JSON API endpoint instead.

For static pages, you usually need two pieces:

an HTTP client to download the page
an HTML parser to extract data from the response

In Ruby, the most practical default stack is:

Faraday Faraday for HTTP requests
Nokogiri for parsing HTML

Ruby ships with Net::HTTP, but Faraday usually lead to clearer scraper code. There is also the HTTParty gem, which is a good fit for quick scripts. I prefer Faraday for long-lived scrapers because its middleware model makes retries, adapters, headers, logging, and timeouts easier to manage as the scraper grows.

Set up a Ruby scraping project

Install the required libraries with gem install or write a Gemfile for better maintenance of the scraper:

# Gemfile
source "https://rubygems.org"

ruby ">= 3.2"

gem "faraday"
gem "faraday-retry"
gem "nokogiri"
gem "csv"

Then install dependencies:

bundle install

Fetch HTML with Faraday

Here is a minimal HTTP example:

require "faraday"

response = Faraday.get("https://books.toscrape.com/")

puts response.status
puts response.body[0, 200]

Before parsing a page, check that the request succeeded:

require "faraday"

response = Faraday.get("https://books.toscrape.com/")

unless response.success?
  abort "Request failed with status #{response.status}"
end

html = response.body

For anything more than a throwaway script, create a reusable connection with timeouts and headers:

require "faraday"

conn = Faraday.new(url: "https://books.toscrape.com") do |f|
  f.options.timeout = 10
  f.options.open_timeout = 5
end

response = conn.get("/") do |req|
  req.headers["User-Agent"] = "RubyStaticScraper/1.0"
  req.headers["Accept"] = "text/html"
end

unless response.success?
  abort "Request failed with status #{response.status}"
end

html = response.body

Parse HTML with Nokogiri

Once you have the HTML response, parse it with Nokogiri:

require "nokogiri"

doc = Nokogiri::HTML(html)

puts doc.at_css("title")&.text&.strip

at_css returns the first matching element. css returns all matching elements.

Extract text, links, and attributes

Now let's scrape book titles, prices, and URLs from books.toscrape.com:

require "faraday"
require "nokogiri"
require "uri"

BASE_URL = "https://books.toscrape.com/"

response = Faraday.get(BASE_URL)
raise "Request failed: #{response.status}" unless response.success?

doc = Nokogiri::HTML(response.body)

books = doc.css(".product_pod").filter_map do |book|
  link = book.at_css("h3 a")
  href = link&.[]("href")

  next unless link && href

  {
    title: link["title"],
    price: book.at_css(".price_color")&.text&.delete("^0-9."),
    url: URI.join(BASE_URL, href).to_s
  }
end


puts books

Notice a few details:

link["title"] extracts an attribute.
link&.[]("href") avoids crashing if the link is missing.
filter_map skips incomplete records and returns only complete book hashes.
URI.join turns relative links into absolute URLs.
delete("^0-9.") strips the currency symbol from the price.

Save the scraped data to CSV

For static pages, CSV is often the simplest useful output:

require "csv"

CSV.open("books.csv", "w", write_headers: true, headers: %w[title price url]) do |csv|
  books.each do |book|
    csv << [book[:title], book[:price], book[:url]]
  end
end

This gives you a spreadsheet-friendly file while keeping the scraper small and easy to run.

Faraday vs HTTParty

Both Faraday and HTTParty gems are good enough to build a scraper. Pick one based on how much structure the scraper needs:

Tool	Best for	Tradeoff
`Faraday`	Long-lived scrapers, retries, middleware, custom adapters	Slightly more setup
`HTTParty`	Quick scripts and simple requests	Less flexible for advanced HTTP workflows

A quick HTTParty version looks like this:

require "httparty"
require "nokogiri"

response = HTTParty.get("https://books.toscrape.com/")
doc = Nokogiri::HTML(response.body)

puts doc.css(".product_pod h3 a").map { |a| a["title"] }

For a one-off script, that is hard to beat. For a scraper that will run every day, use the structure and middleware you get from Faraday.

CSS selectors and XPath with Nokogiri

Nokogiri is the default HTML and XML parser in the Ruby ecosystem. If you search for web scraping with Ruby and Nokogiri, this is the gem you will see everywhere. As mentioned before, Nokogiri is not a browser and it cannot fetch web pages or execute JavaScript. It parses HTML or XML you already have.

CSS selectors in Nokogiri

CSS selectors are usually the easiest way to find elements:

doc.css(".product_pod").each do |product|
  title = product.at_css("h3 a")&.[]("title")
  price = product.at_css(".price_color")&.text&.strip

  puts "#{title}: #{price}"
end

Common selector patterns:

doc.at_css("h1") # first h1
doc.css("a") # all links
doc.css(".price") # elements with class price
doc.css("#content") # element with id content
doc.css("article.product a") # links inside product articles

XPath in Nokogiri

XPath is more verbose, but it can express queries that are awkward with CSS:

doc.xpath("//a").each do |link|
  puts link["href"]
end

For example, XPath is useful when you need text matching or more complex conditions:

# Find links containing specific text
links = doc.xpath("//a[contains(text(), 'Next')]")

# Find elements with a class fragment
prices = doc.xpath("//*[contains(@class, 'price')]")

In most Ruby web scrapers, CSS selectors are enough. Reach for XPath when CSS gets too clumsy.

Nokogiri vs Nokolexbor

Nokogiri is the safe default. It is mature, well documented, and widely used library that you most likely already know. For most projects, choosing Nokogiri is enough. But if you need to parse a large number of elements by CSS or XPath, nokolexbor is worth learning about.

The Nokolexbor gem from SerpApi is a parser based on Lexbor which focuses on fast HTML 5 parsing using at_css and at_xpath selectors.

Tool	Best for	Notes
`Nokogiri`	Most Ruby scraping projects	Mature, CSS/XPath support, widely used ecosystem
`Nokolexbor`	Performance-heavy HTML5 parsing	Consider when parsing speed matters

You do not have to optimize for this early on, but Nokolexbor can save you when you'll need it. Network requests and browser automation are usually much slower than parsing the response.

Static scraping is the fastest and most reliable option when it works. If your selectors return nothing, check whether the page is rendering data with JavaScript or fetching it from a JSON endpoint before assuming Nokogiri is broken.

Choosing the right HTTP client for scraping

Faraday and httparty are not the only Ruby HTTP clients. They are just convenient choices for a web scraping tutorial: Faraday gives you structure, middleware, retries, and adapters; HTTParty gives you a very small API for quick scripts.

Ruby has several other HTTP libraries worth knowing about:

Ruby ships with Net::HTTP, which avoids an extra dependency but is unnecessarily verbose.
HTTPX is a modern option worth considering for high-throughput scrapers, especially if you care about HTTP/2, persistent connections, and concurrent requests.
http.rb has a pleasant request-chaining API
Excon is a solid lower-level client used in parts of the Ruby ecosystem.

For most static scraping tasks, the HTTP client is not the hard part. Fetch the page reliably, then spend most of your effort on parsing, retries, rate limits, and maintenance. Use Faraday for the examples in this guide unless you have a reason not to. Use HTTParty for tiny scripts. Look at HTTPX when performance, concurrency, or HTTP/2 become important.

Scraping JavaScript-rendered pages with Ruby

If a Nokogiri scraper returns empty results, do not assume your selector is wrong. The data may not be in the original HTML at all. That is the classic example of Nokogiri JavaScript not working. Nokogiri parses the HTML response, but it does not run JavaScript. Let's see what we can do about it.

First check whether you really need JavaScript

Before using Selenium or Ferrum, inspect the page.

Open the page in your browser.
Open Developer Tools.
Go to the Network tab.
Filter by Fetch/XHR.
Reload the page.
Look for JSON responses containing the data.

If you find a public JSON endpoint, you may be able to call it directly with Faraday. That is usually faster and more reliable than running a browser.

Use a headless browser only when:

The data is generated client-side and no usable API is available.
The page requires clicks, scrolling, or browser state.
You need to test exactly what a browser sees.

Scrape a dynamic page with Selenium

Selenium is the most widely known browser automation tool. It can control Chrome, Firefox, and other browsers.

Add it to your Gemfile as:

gem "selenium-webdriver"

Here's a minimal headless Selenium example using Chrome:

require "selenium-webdriver"
require "nokogiri"

options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")

browser = Selenium::WebDriver.for(:chrome, options: options)

begin
  browser.navigate.to("https://quotes.toscrape.com/js/")

  wait = Selenium::WebDriver::Wait.new(timeout: 10)
  wait.until { browser.find_elements(css: ".quote").any? }

  doc = Nokogiri::HTML(browser.page_source)

  quotes = doc.css(".quote").map do |quote|
    {
      text: quote.at_css(".text")&.text&.strip,
      author: quote.at_css(".author")&.text&.strip
    }
  end

  puts quotes
ensure
  browser.quit
end

The important part is the wait. Avoid blind sleep calls when you can wait for a specific element.

Scrape a dynamic page with Ferrum

Ferrum controls Chrome or Chromium through the Chrome DevTools Protocol. It is a good Ruby-native choice for headless Chrome scraping.

Add it to your Gemfile:

gem "ferrum"

Example of using Ferrum to load a website and parsing the body with Nokogiri:

require "ferrum"
require "nokogiri"

browser = Ferrum::Browser.new(timeout: 10)

begin
  browser.go_to("https://quotes.toscrape.com/js/")
  browser.network.wait_for_idle

  doc = Nokogiri::HTML(browser.body)

  quotes = doc.css(".quote").map do |quote|
    {
      text: quote.at_css(".text")&.text&.strip,
      author: quote.at_css(".author")&.text&.strip
    }
  end

  puts quotes
ensure
  browser.quit
end

Ferrum is especially nice when you want headless Chrome without the full Selenium stack.

Selenium vs Ferrum

Both Selenium and Ferrum will get a job done. Here's a small comparison on how they differ:

Tool	Best for	Tradeoff
`Selenium`	Cross-browser automation, widely supported, familiar ecosystem	Heavier and needs browser/driver setup
`Ferrum`	Headless Chrome via Chrome DevTools Protocol	Chrome/Chromium-focused

If you need cross-browser behavior or already know Selenium, use Selenium. If you want a Ruby-friendly headless Chrome tool for scraping, try Ferrum.

Browser automation with Capybara and Watir

Selenium and Ferrum are lower-level browser control tools. Capybara and Watir provide nicer APIs for browser-like workflows. Think of them as higher-level tools closer to your application code.

Use Capybara and Watir when your scraper needs to behave like a user:

fill in fields
click buttons
wait for elements
navigate flows
save screenshots
work with sessions

Use Capybara for scraping workflows

Capybara is best known for Rails system tests, but it can also automate external websites in general. Here's such an example:

require "capybara"
require "capybara/dsl"
require "selenium-webdriver"

Capybara.register_driver :selenium_chrome_headless do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_argument("--headless=new")

  Capybara::Selenium::Driver.new(app, browser: :chrome, options: options)
end

Capybara.default_driver = :selenium_chrome_headless
Capybara.run_server = false
Capybara.default_max_wait_time = 10

include Capybara::DSL

visit "https://quotes.toscrape.com/js/"

all(".quote").each do |quote|
  puts quote.find(".text").text
end

page.save_screenshot("quotes.png")

Capybara's waiting behavior is one of its biggest advantages. find and has_css? wait for elements instead of failing immediately. Note the include Capybara::DSL like which makes this DSL possible.

Take screenshots with Capybara

Screenshots are useful when debugging browser scrapers. If an element is missing, a cookie banner blocks the page, or the page looks different in headless mode, saving a screenshot shows you what the browser actually saw.

With Capybara, screenshots are straightforward:

page.save_screenshot("page.png")

You can also save screenshots with dynamic names:

page.save_screenshot("screenshots/product-#{product_id}.png")

Just make sure the target directory exists before saving:

require "fileutils"

FileUtils.mkdir_p("screenshots")
page.save_screenshot("screenshots/page.png")

Screenshots work in headless mode too which makes them especially helpful in background jobs, CI, or production debugging.

This is not too different to what a typical Rails system test setup looks like.

Screenshots for JavaScript websites

However, if you make a screenshot too quickly, it might like dynamic elements, like in this case for the Ruby official website.

Capybara screenshot of ruby-lang.org without waiting for JavaScript

To fix it, you'll need to wait using the setTimeout function. For this cases, run execute_script (again from the Capybara DSL):

visit "https://www.ruby-lang.org/en/"

# ruby-lang.org is mostly static, so this creates a small visible marker
# after a delay to demonstrate waiting for JavaScript-rendered content.
execute_script(<<~JS)
  window.setTimeout(function () {
    var marker = document.createElement("div");
    marker.id = "capybara-js-ready";
    marker.textContent = "JavaScript content loaded — screenshot taken after Capybara waited";
    marker.style.position = "fixed";
    marker.style.left = "24px";
    marker.style.bottom = "24px";
    marker.style.zIndex = "999999";
    marker.style.padding = "14px 18px";
    marker.style.background = "#cc342d";
    marker.style.color = "white";
    marker.style.font = "16px sans-serif";
    marker.style.borderRadius = "8px";
    marker.style.boxShadow = "0 4px 16px rgba(0,0,0,0.25)";
    document.body.appendChild(marker);
  }, 2000);
JS

find "#capybara-js-ready", wait: 10
save_screenshot "ruby.png"

Now you should be able to see the site in all its glory:

Ruby official website screenshot

Use Watir for browser-like automation

Watir is another readable browser automation library built around WebDriver.

require "watir"

browser = Watir::Browser.new(:chrome, headless: true)

begin
  browser.goto("https://quotes.toscrape.com/js/")

  browser.divs(class: "quote").each do |quote|
    puts quote.span(class: "text").text
  end
ensure
  browser.close
end

Watir is pleasant when you want your scraper code to read like browser instructions.

Capybara vs Watir

Tool	Best for	Tradeoff
`Capybara`	Rails-style flows, waiting DSL, screenshots, system-test familiarity	Originally designed for testing
`Watir`	Human-readable browser automation	Another abstraction over WebDriver

If you already write Rails system tests, Capybara will feel natural. If you want a direct browser automation API, Watir is easy to read.

Scraping forms, cookies, and sessions with Mechanize

Mechanize is a classic Ruby gem for automated web interaction. It can follow links, submit forms, store cookies, handle redirects, and keep history. It does not execute JavaScript which is the main limitation. Use Mechanize when you need browser-like HTTP behavior without a browser.

When Mechanize is better than Selenium

Mechanize is often better than Selenium when:

The site is mostly static.
You need to submit normal HTML forms.
You need cookies and sessions.
You do not need JavaScript rendering.
You want speed and lower resource usage.

Selenium is better when JavaScript is required.

Submit a form with Mechanize

Here's a short example of submitting a search form:

require "mechanize"

agent = Mechanize.new
agent.user_agent_alias = "Mac Safari"

page = agent.get("https://example.com/search")

form = page.form_with(action: "/search")
form["q"] = "ruby"

results = agent.submit(form)

puts results.search("h2").map(&:text)

Mechanize pages can be searched with Nokogiri-style methods because Mechanize uses Nokogiri internally.

Mechanize vs Selenium

Both Mechanize and Selenium are excellent tools but they both have their ideal use-case as shown in the table below:

Tool	Best for	Tradeoff
`Mechanize`	Forms, links, cookies, sessions without JavaScript	Does not execute JavaScript
`Selenium`	JavaScript, real browser interactions, complex UI flows	Slower and heavier

If you search for Mechanize Ruby login, you will find many examples because this is exactly where Mechanize shines; handling old-school form-based websites.

Crawling multiple pages with Ruby

If you already know every URL, you do not need a crawler. Just loop through the URLs and scrape each page. However, you need crawling when you want to discover URLs by following links you find along the way.

Scraping one page vs crawling a site

A scraper extracts data from a page. A crawler finds pages. And most production systems are both. Here's an example of a scraping workflow involving crawling:

Start with one or more seed URLs.
Fetch a page.
Extract data.
Extract links.
Normalize links.
Skip links already visited.
Repeat until a depth, count, or domain limit is reached.

We already talked plenty about scraping itself, now let's talk about crawling.

Crawl links with Spidr

Spidr is a very popular Ruby web spidering library. It comes with a small DSL that gives us an option to work with every individual page like this:

require "spidr"

Spidr.site("https://example.com/") do |spider|
  spider.every_page do |page|
    puts page.url
  end
end

This is useful for quick site crawling and link discovery.

Crawl links with Anemone

Anemone is another Ruby crawling library with similar DSL:

require "anemone"

Anemone.crawl("https://example.com/") do |anemone|
  anemone.on_every_page do |page|
    puts page.url
  end
end

Both libraries are useful, but for serious scraping you may eventually want a custom queue so you control retries, persistence, rate limits, priorities, and deduplication.

Spidr vs Anemone vs custom queue

Here's a short comparison of Spidr vs Anemone libraries:

Approach	Best for	Tradeoff
`Spidr`	Quick site crawling	Less custom architecture
`Anemone`	Spidering workflows	Older ecosystem feel
Custom queue	Production control	More code

For a small Ruby web crawler, a gem is fine. For a production crawler, a custom queue backed by a database or Redis can be better.

Scraping JSON APIs instead of HTML

Getting data directly from JSON requests is insanely easier as the data already comes in the format you likely want. Before writing a complex browser scraper, check whether the page gets its data from JSON.

XHR requests on the Chrome Network tab

This is how you can do so in Chrome or Firefox:

Open Developer Tools.
Go to Network.
Filter by Fetch/XHR.
Reload the page.
Click requests and inspect responses.

If you find the data in JSON, you can often skip HTML parsing entirely.

Request JSON with Faraday

If you can find such requests, you can then fetch the respective JSON using Faraday (or alternative):

require "faraday"
require "json"

response = Faraday.get("https://api.example.com/products") do |req|
  req.headers["Accept"] = "application/json"
end

raise "Request failed: #{response.status}" unless response.success?

data = JSON.parse(response.body)

puts data

API scraping vs HTML scraping

Most scraping is done with fetching full HTML pages, but if you can avoid it consider scraping the API calls:

Approach	Pros	Cons
JSON/API	Structured, easier to parse, often more stable	May require auth, may be private, may be unsupported
HTML	Works when no API exists	Selectors break more often

Be careful with private or undocumented endpoints. Just because your browser can see a request does not automatically mean you can use it freely for commercial scraping. Check terms and permissions.

Reliability: Timeouts, retries, errors, and logging

The difference between a demo scraper and a scraper you'll run for a long time is reliability. Websites go down. Networks fail. HTML changes. Requests time out. Selectors break. Your code should expect that.

Set connection timeouts

Never let a scraper hang on a connection forever. Set connection times like these for Faraday:

conn = Faraday.new(url: "https://books.toscrape.com") do |f|
  f.options.timeout = 10
  f.options.open_timeout = 5
end

Retry temporary failures

You should try retried temporary failures. For Faraday, you can use faraday-retry for transient problems:

require "faraday"
require "faraday/retry"

conn = Faraday.new(url: "https://books.toscrape.com") do |f|
  f.request :retry,
    max: 3,
    interval: 0.5,
    backoff_factor: 2,
    retry_statuses: [429, 500, 502, 503, 504]

  f.options.timeout = 10
  f.options.open_timeout = 5
end

Retry carefully. Retrying 429 Too Many Requests immediately and aggressively is a good way to make the problem worse. Consider adding backoff and reducing request rate.

Handle HTTP status codes

Don't forget that requests don't just end in 200 status codes. You might want to fetch another page on redirect, stop at forbidden and not found pages, retry later for rate limiting and investigate on 500 server errors. Here's the reminder of the main status codes:

200: success
301 / 302: redirect
403: forbidden
404: not found
429: rate limited
500+: server error

An example handler could look like this example:

response = conn.get("/catalogue/page-1.html")

case response.status
when 200
  parse(response.body)
when 404
  logger.warn("Page not found")
when 429
  logger.warn("Rate limited")
else
  logger.error("Unexpected status: #{response.status}")
end

Avoid crashing on missing elements

Selectors break and optional elements disappear. Always parse defensively using the &. operator:

def text_at(node, selector)
  node.at_css(selector)&.text&.strip
end

price = text_at(product, ".price_color")

If a field is required, you can raise a clear error, but your scraper should likely be able to continue.

Add logging

Ruby's standard Logger is enough for many scrapers:

require "logger"

logger = Logger.new($stdout)
logger.info("Starting scrape")
logger.warn("Missing price for product")
logger.error("Request failed")

A scheduled scraper with no logs is painful to debug.

Cache pages during development

When building selectors, save sample HTML locally. This avoids hitting the target site repeatedly while you tune parsing code.

You can use:

plain saved .html fixtures
VCR gem
WebMock gem
a local cache directory

Parser tests against saved HTML fixtures are one of the best ways to keep scrapers maintainable.

💡 When things get difficult, you can rely on a service that does the hard parts for you. SerpApi provides a clean API for all kinds of search data, so you don't have to scrape search engine results yourself.

Performance: Concurrency, batching, and rate limits

Scraping performance is not just about making requests as fast as possible. A good scraper should finish in a reasonable time without overwhelming the target site, wasting local resources, or triggering rate limits.

The goal is controlled throughput when enough concurrency avoids waiting on the network all day, but not so much that you create failures for yourself or for the site you are scraping.

Use concurrency for HTTP, not everything

Static HTTP scraping is usually I/O-bound. Most of the time is spent waiting for the network, so Ruby threads can help even if the Ruby code itself is not CPU-heavy.

Browser scraping is different. A headless browser uses much more CPU and memory than a plain HTTP request. Running many browser sessions at once can overload your machine or server.

Parallel scraping with Ruby threads

For static pages, a small thread pool is often enough. Put URLs into a queue, let a few workers fetch and parse them, and collect the results in another queue:

require "thread"

urls = [
  "https://books.toscrape.com/catalogue/page-1.html",
  "https://books.toscrape.com/catalogue/page-2.html",
  "https://books.toscrape.com/catalogue/page-3.html"
]

queue = Queue.new
urls.each { |url| queue << url }

results = Queue.new
workers = 3.times.map do
  Thread.new do
    loop do
      url = queue.pop(true)
      results << scrape_page(url)
    rescue ThreadError
      break
    end
  end
end

workers.each(&:join)

This pattern gives you controlled concurrency. You can increase or decrease the number of workers based on the target site, your machine, and how much rate limiting you see.

In production, combine this with timeouts, retries, rate limits, and logging. Do not spawn 100 threads just because you can.

Add rate limits and delays

A small delay between requests can make a scraper much more polite:

pages.each do |url|
  scrape_page(url)
  sleep 1
end

For more advanced setups, use adaptive throttling. Slow down when responses take longer, error rates increase, or you start seeing 429 Too Many Requests.

Avoid browser automation unless needed

A headless browser can take seconds to start and hundreds of megabytes of memory to run. A plain HTTP request may take milliseconds and use almost no memory. Use browsers for browser-only problems; JavaScript rendering, clicks, scrolling, screenshots, or workflows that require real browser state.

Parse efficiently

Most scraper performance issues come from the network or browser automation, but parsing can still matter on large pages or large batches.

A few practical tips:

Scope selectors to a container instead of searching the whole document repeatedly.
Avoid parsing the same HTML more than once.
Write output incrementally for large scrapes instead of keeping everything in memory.

Ruby web scraping FAQ

What is the best Ruby gem for web scraping?

There is no single best gem. For most static pages, use Faraday and Nokogiri. For simple scripts, HTTParty is fine. For forms and cookies without JavaScript, use Mechanize. For JavaScript-rendered pages, use Selenium, Ferrum, Capybara, or Watir.

Can Nokogiri scrape JavaScript?

No. Nokogiri parses HTML and XML. It does not execute JavaScript. If the content is rendered by JavaScript, use a headless browser or find the JSON endpoint the page uses.

Is Ruby better than Python for web scraping?

Python has a larger scraping and data ecosystem, especially around Scrapy, Playwright, Pandas, and machine learning workflows. Ruby is excellent if you already work in Ruby or Rails, want readable automation scripts, or need scraping inside a Ruby application.

Should I use Selenium or Ferrum?

Use Selenium if you need cross-browser automation or want the most widely documented browser tool. Use Ferrum if you want a Ruby-friendly headless Chrome tool using the Chrome DevTools Protocol.

Should I use Mechanize or Nokogiri?

They solve different problems. Nokogiri parses HTML. Mechanize fetches pages, follows links, submits forms, and manages cookies, using Nokogiri under the hood. For simple parsing, use Nokogiri. For non-JavaScript form workflows, consider Mechanize.

How do I scrape multiple pages in Ruby?

If you know the URLs, loop through them with Faraday and parse each response with Nokogiri. If you need to discover URLs, use a crawler like Spidr, Anemone, or a custom queue that extracts and deduplicates links.

How do I avoid getting blocked while scraping with Ruby?

Scrape responsibly. Respect terms, robots.txt, and rate limits. Add delays. Set timeouts. Avoid unnecessary requests. Do not scrape private data or bypass access controls. If you receive 429, slow down.

Can I scrape websites from Rails?

Yes, but do it in background jobs, not controller actions. Put scraping logic in service objects, run it with ActiveJob, Sidekiq, or Solid Queue, and store results with ActiveRecord.

Conclusion

Web scraping with Ruby can be as simple as using Faraday with Nokogiri, but real-world scraping is broader than fetching one page and parsing a few tags.

The best Ruby web scraper starts with the right approach:

Use HTTP and Nokogiri for static HTML.
Use JSON endpoints when the site already exposes structured data.
Use Mechanize for forms, cookies, and non-JavaScript sessions.
Use Selenium, Ferrum, Capybara, or Watir when you need a real browser.
Use a crawler or queue when you need many pages.
Add retries, timeouts, logging, tests, and rate limits before calling it production-ready.

Ruby gives you all the pieces: clean HTTP clients, mature parsers, browser automation, background jobs, CSV/JSON support, and Rails integration. The trick is not finding a magic scraping gem. The trick is choosing the smallest reliable tool for the page in front of you.

Semantic Image Search with Elasticsearch

Tomás Murúa — Mon, 06 Jul 2026 22:32:52 +0000

Searching Instagram by hashtag or caption text misses the point when the content is visual. A photo of a Samoyed in the snow doesn't always say "Samoyed" in the caption, but you'd still want to find it by typing "a white fluffy dog in the snow." This tutorial builds a pipeline that searches images by what they look like, not what the caption says.

We'll fetch a public profile's feed with SerpApi's Instagram Profile API, embed every cover image into a shared text-image vector space with Jina AI, store those vectors in Elasticsearch, and search with plain language. By the end, you'll have a working search interface where typing "a dog wearing sunglasses" returns the matching image ranked by visual similarity, no caption needed.

Results for the search "a dog wearing sunglasses"

The complete project, notebook and Streamlit app, is on GitHub:

tutorials/python_projects/instagram-semantic-image-search at master · serpapi/tutorials

Prerequisites

Python 3.9+
SerpApi API key (free tier, 250 searches/month)
Jina AI API key (free tier, 10 million tokens)
Elasticsearch cluster (9.0 or higher)
~2 GB disk space for downloaded images

Structured Image Search at Scale

The hard part of image search has never been the vector math. It's getting the images in the first place. Public Instagram profiles show thousands of posts, but there's no "download all" button and no official bulk API from Instagram.

SerpApi Instagram Profile API solves the access problem. One call with engine="instagram_profile" and a username returns the profile's posts along with captions, like counts, comment counts, and a direct URL to each cover image.

That means a few dozen API calls give us hundreds of high-quality images with their metadata, ready to embed.

We'll use the @apple profile as our example. The feed spans landscapes, portraits, animals, and architectural photos, which gives the search plenty of visual variety to work with.

How It Works

Four steps take us from a username to searchable images.

Fetch: Page through the profile's feed with SerpApi, collecting cover images (images + video thumbnails).
Embed: Convert each image into a vector that captures its visual meaning, using Jina AI.
Index: Store the vector plus the metadata in Elasticsearch dense_vector field.
Search: Type a query and get the ranked images by similarity.

The four-step search pipeline

What Are Embeddings?

An embedding is a list of numbers (a vector) that captures what an image means, its visual features, objects, colors, and composition. Similar images produce vectors close together in this high-dimensional space; different images produce vectors far apart.

An image and a sentence become nearby vectors

The key that makes text-to-image search possible is that models like Jina V5 Omni are trained on image-text pairs, so they map both images and text into the same space. A photo of a sunset and the sentence "a person at sunset" end up near each other, not because of keywords but because the model learned their meaning is similar.

Results for the search "a person at sunset"

How Vector Search Works

Once all images are embedded and stored, the search flow is straightforward.

Jina embeds your text query "a dog wearing sunglasses" into a vector using the same model.
Elasticsearch compares that query vector against every stored image vector using cosine similarity, a measure of how close two vectors point in the same direction.
The k nearest neighbors are returned, ranked by similarity score. Scores closer to 1.0 mean more similar; scores closer to 0 mean unrelated.

The query never touches the captions. The ranking is purely visual.

The query vector finds its nearest neighbors

Why These Three Tools?

Component	Role	Why this one
SerpApi	Data access	Structured JSON from public Instagram profiles, paginated, no scraping infrastructure to maintain
Jina v5 Omni	Embedding model	Open-weight, shared text-image space in one model, free API tier
Elasticsearch	Vector store + search	Native `dense_vector` field with built-in kNN, exact and approximate search out of the box, scales from hundreds to millions of vectors

One model for everything: Jina v5 Omni handles both images and text in a single pipeline. If you later want to add semantic search over captions or post metadata, the same model and the same index work. No second model, no separate pipeline.
No raw images stored in the database: We store only the vector, a local file path, and metadata (caption, post URL, likes). Images live in a local images/ folder, just long enough to embed and display results.

What Is Elasticsearch?

If you haven't used Elasticsearch before, it's a search engine (the same technology behind site search on large websites) that also handles vector search natively. You store documents; each document has fields, and you query against those fields. For our use case, one of those fields is a vector, a long list of numbers that represents the visual content of an image.

The relevant feature is vector search via the knn retriever. Given a query vector, it finds the closest stored vectors and returns them ranked by similarity. That's our image search.

Two Ways to Run It

Two options for the Elasticsearch cluster. The notebook code is identical either way. Only ES_URL and ES_API_KEY in .env change.

Local Docker

We start a local single-node cluster using the Elastic start-local script:

curl -fsSL https://elastic.co/start-local | sh

This prints an ES_LOCAL_API_KEY. Put it in your .env as ES_API_KEY. The endpoint is http://localhost:9200.

Elastic Cloud

Create a free project on Elastic Cloud (14-day free trial). Copy the endpoint URL and API key into .env. Good if your machine is light on RAM or you want persistence without managing Docker.

The Pipeline

Step 1. Fetch the Profile

We'll page through the profile feed using the SerpApi Python client:

import serpapi

serp_client = serpapi.Client(api_key=os.getenv("SERPAPI_API_KEY"), timeout=30)

base = {"engine": "instagram_profile", "profile_id": "apple"}
params = dict(base)

for page in range(1, max_pages + 1):
    results = serp_client.search(params)
    profile = results.get("profile_results", {})
    posts = profile.get("posts", [])

    next_token = results.get("serpapi_pagination", {}).get("next_page_token")
    if not next_token:
        break
    params = {**base, "next_page_token": next_token}

Each request returns about 12 posts along with a next_page_token, which we send in the next call to get the next batch. When no token comes back, we've reached the end of the feed. So a few dozen calls are enough to pull hundreds of images.

Step 2. Embed the Images

Here we are sending the images downloaded from the Instagram API to the Jina Embedding API as base64, receiving one vector per image:

JINA_URL = "https://api.jina.ai/v1/embeddings"
JINA_MODEL = "jina-embeddings-v5-omni-small"

def embed_images(images_b64):
    inputs = [{"image": f"data:image/jpeg;base64,{b}"} for b in images_b64]
    payload = {"model": JINA_MODEL, "task": "retrieval.passage", "dimensions": 1024, "input": inputs}
    r = requests.post(JINA_URL, headers=jina_headers, json=payload, timeout=120)
    r.raise_for_status()
    data = sorted(r.json()["data"], key=lambda d: d["index"])
    return [d["embedding"] for d in data]

The one detail worth calling out is task="retrieval.passage". Jina uses different task modes for the vectors you store (retrieval.passage) and the vectors you search with (retrieval.query). Pairing them this way is what Jina recommends for retrieval. Step 4 handles the query side.

Step 3. Create the Index

We create an Elasticsearch index with a vector field to store the embeddings:

es.indices.create(
    index="instagram_photos",
    mappings={"properties": {
        "embedding": {"type": "dense_vector", "dims": 1024, "similarity": "cosine"},
        "caption": {"type": "text"},
        "shortcode": {"type": "keyword"},
        "username": {"type": "keyword"},
        # ... post_url, image_url, is_video, liked_by_count, comments_count
    }}
)

We use cosine similarity because it's the standard match for text and image embeddings, and 1024 dimensions matches Jina's output. See the notebook for the full mapping.

Step 4. Search

We embed the query with Jina (this time with task="retrieval.query") and pass the vector to Elasticsearch's kNN retriever:

def search(query, username, k=6):
    query_vector = embed_query(query)
    resp = es.search(
        index="instagram_photos",
        retriever={"knn": {
            "field": "embedding",
            "query_vector": query_vector,
            "k": k,
            "filter": {"term": {"username": username}},
        }},
        size=k,
        source_excludes=["embedding"],
    )
    return resp["hits"]["hits"]

The filter scopes results in a single profile even when the index holds many. source_excludes keeps the response lean by dropping the 1024-float vector we don't need back.

Results

With 577 images indexed from the Apple profile, the search delivers exactly what you'd expect.

The top result is a video thumbnail (poster frame) from Apple's feed, a dog in sunglasses, scored at 0.737 cosine similarity. No caption matching. No keyword overlap. The pipeline matched the visual content of the image to the meaning of the text query.

A dog wearing sunglasses

Results for the search "a dog wearing sunglasses"

More examples from the same index:

Snow-capped mountains

Results for the search "snow-capped mountains"

Horseback Riding

Results for the search "horseback riding"

Someone holding a phone

Results for the search "someone holding a phone"

A close-up portrait

Results for the search "a close-up portrait"

Beyond Instagram

The embedding and search code doesn't change. Only the SerpApi engine and the fields you extract differ. The same architecture applies to any visual data source SerpApi can reach.

Source	SerpApi engine	Image field	What you'd search
YouTube Search	`youtube`	`thumbnail.static`	Video thumbnails by visual content
Google Images	`google_images`	`original`	Web images matching a topic
Google Shopping	`google_shopping`	`thumbnail`	Product photos
Google Lens	`google_lens`	`thumbnail`	Visual matches for a source image
Amazon Search	`amazon`	`thumbnail`	Product gallery images

Where to Go from Here

Concrete next steps to extend this pipeline:

Hybrid search combining captions and embeddings: Add a text query over the caption field alongside the vector search using Elasticsearch's RRF (Reciprocal Rank Fusion). Posts where both the image and the caption match will rank higher.
Index multiple profiles: The username field already supports multi-profile indexing. Fetch a second profile, embed its images, and use the filter to search within or across profiles.
Try other SerpApi engines from the table above: Swap the fetch step for YouTube thumbnails or Google Shopping product photos. The embedding and search code stays identical.

Conclusion

This pipeline reads images the way people describe them, not the way someone happened to describe or tag them. That single capability changes what you can do with an image catalog.

The Instagram example is just a demonstration. An e-commerce store can surface products by what they look like, not by category tags. A media archive can retrieve photos by scene, mood, or subject. An LLM agent can query the same index in plain language and reason over the matches, no captions or manual tagging required.

Both sides of the search now carry meaning. The query is interpreted by what it means, and the images are stored by what they show. That opens a second layer of interaction between your users, or an LLM, and your content.

How to Scrape Naver AI Briefings with SerpApi

Braden — Fri, 03 Jul 2026 22:32:21 +0000

Naver is South Korea's dominant search engine. Like Google's AI Overviews, Naver generates AI-powered answers with cited sources called “AI Briefings.” If you're tracking your brand's visibility in Korea, monitoring competitors, or comparing how different engines answer the same query, you need structured access to these results. SerpApi's Naver AI Overview API returns AI Briefings as JSON, making them easy to extract and analyze programmatically.

Why Scrape Naver AI Briefings?

Naver's AI Briefing uses generative AI to deliver concise, well-structured summaries with cited sources, prioritizing key insights to reduce the time users spend finding relevant information. SerpApi's Naver AI Overview API lets you capture these responses as structured JSON. Use cases include:

Generative Engine Optimization (GEO): Track whether your content is cited in Naver's AI-generated answers and optimize for visibility in the Korean search market.
Cross-Engine AI Comparison: Compare how Naver, Google, Brave, and other AI search engines answer the same queries to identify differences in sourcing and factual accuracy.
Market Intelligence: Monitor which competitors or sources are cited by Naver's AI Briefings for queries in your industry.
Content Research: Extract synthesized answers and their source citations to understand what Naver considers authoritative for a given topic.
Brand Monitoring: Track how your company or products are described in Naver's AI-generated search results.

Understanding the Response Structure

You can easily test queries using SerpApi's Playground. Here's a condensed example of the response for the query “apple juice vs orange juice”. If you'd rather skip straight to the code, jump ahead to How to Scrape Naver AI Briefings using SerpApi.

{
  "markdown": "사과 주스와 오렌지 주스는 칼로리·탄수화물 기준으로 비슷하지만, 비타민 ... [1] 다만 ...",

  "text_blocks": [
    {
      "type": "paragraph",
      "snippet": "사과 주스와 오렌지 주스는 칼로리·탄수화물 기준으로 비슷하지만, 비타민 ...",
      "reference_indexes": [1, 2]
    },
    // headers, lists, tables ...
  ],

  "references": [
    {
      "index": 1,
      "title": "Orange and Apple Juice",
      "snippet": "Orange and apple juice Discover Pinterest’s best ideas and inspiration for Orange and apple juice.",
      "link": "https://www.pinterest.com/ideas/orange-and-apple-juice/940763462091/",
      "source": "Pinterest",
      "source_icon": "https://s.pinimg.com/webapp/favicon_48x48-7470a30d.png"
    },
    // ...
  ],

  "media": [
    {
      "title": "Apple Juice vs. Orange Juice: Which Has More Vitamin C for Immune Support?",
      "link": "https://www.verywellhealth.com/apple-juice-vs-orange-juice-11872120",
      "thumbnail": "https://www.verywellhealth.com/thmb/11UaGLH66Drx2TTuxJjh8q3fMz8...",
      "platform": "Verywell Health"
    },
    // ...
  ],

  "related_questions": [
    {
      "question": "사과 주스와 오렌지 주스 비타민C 함량 비교해 주세요",
      "serpapi_link": "https://serpapi.com/search.json?engine=naver_ai_overview&query=..."
    },
    // ...
  ]
}

Text Blocks

The text_blocks array is the core of the response. Each block has a type that determines its structure. Here's how each type works:

Paragraphs

This is the most common block type. Contains a snippet with the text, optional snippet_highlighted_words, and reference_indexes:

{
  "type": "paragraph",
  "snippet": "사과 주스와 오렌지 주스는 칼로리·탄수화물 기준으로 비슷하지만, 비타민 ...",
  "reference_indexes": [1, 2]
}

reference_indexes is an array of integers pointing to the top-level references array. "reference_indexes": [1, 2] means this paragraph cites the references where index equals 1 and 2. The snippet_highlighted_words field shows which phrases Naver considered most important in the answer.

Headers

A section title. Contains a snippet with the text and reference_indexes:

{
  "type": "header",
  "snippet": "영양 비교",
  "reference_indexes": [1]
}

Lists

A bulleted list. Uses a list array where each item has a snippet, optional snippet_highlighted_words, and reference_indexes:

{
  "type": "list",
  "list": [
    {
      "snippet": "비타민 C 섭취가 목적이면 오렌지 주스가 유리합니다.",
      "reference_indexes": [1]
    },
    {
      "snippet": "라벨에 “비타민 C 강화” 또는 “ascorbic acid”가 있으면 사과 주스도 비타민 C가 높을 수 있습니다.",
      "reference_indexes": [1, 2]
    }
  ]
}

Tables

Structured comparison data. Includes table, a two-dimensional array where the first row contains the table's headers, and a formatted version, an array of objects with named keys derived from the headers:

{
  "type": "table",
  "table": [

[
"구분",
"오렌지 주스",
"사과 주스",
"강화 사과 주스"
],
[
"칼로리",
"117 kcal",
"114 kcal",
"114 kcal"
],
// ...
],
"formatted": [
{
"구분": "칼로리",
"오렌지주스": "117 kcal",
"사과주스": "114 kcal",
"강화사과주스": "114 kcal"
},
// ...
]
}

References

The top-level references array contains every source Naver cited. Each reference has an index, title, snippet (a short excerpt from the source), link (the source URL), source (the name of the source), and source_icon (a favicon URL). The index is what reference_indexes inside text_blocks points to.

"references": [
  {
    "index": 1,
    "title": "Orange and Apple Juice",
    "snippet": "Orange and apple juice Discover Pinterest’s best ideas and inspiration for Orange and apple juice.",
    "link": "https://www.pinterest.com/ideas/orange-and-apple-juice/940763462091/",
    "source": "Pinterest",
    "source_icon": "https://s.pinimg.com/webapp/favicon_48x48-7470a30d.png"
  }
]

Markdown

The top-level markdown field contains the entire AI Briefing response—both text_blocks and references—flattened into a single Markdown-formatted string:

"markdown": "사과 주스와 오렌지 주스는 칼로리·탄수화물 기준으로 비슷하지만, 비타민 ... [1] 다만 ..."

Use markdown if you only need the full text; use text_blocks when you need to filter by type, work with inline references, or process the structure programmatically.

Media

The top-level media array contains images displayed alongside the AI Briefing, which Naver sourced from the cited pages. Each entry includes a title, link, thumbnail (image URL), and platform:

"media": [
  {
    "title": "Apple Juice vs. Orange Juice: Which Has More Vitamin C for Immune Support?",
    "link": "https://www.verywellhealth.com/apple-juice-vs-orange-juice-11872120",
    "thumbnail": "https://www.verywellhealth.com/thmb/11UaGLH66Drx2TTuxJjh8q3fMz8...",
    "platform": "Verywell Health"
  }
]

How to Scrape Naver AI Briefings using SerpApi

With the response structure covered, here's how to actually start retrieving Naver AI Briefing results with SerpApi.

If you're new to SerpApi, follow the steps in Getting started with SerpApi to get set up. You can also check out the Naver AI Overview API Documentation for more examples.

GET Request

Using SerpApi's Naver AI Overview API is as simple as sending a GET request to https://serpapi.com/search. Set the engine parameter to naver_ai_overview, include your API key in api_key, and your query in query:

https://serpapi.com/search?engine=naver_ai_overview&api_key=YOUR_API_KEY&query=YOUR_QUERY

Here's an example with cURL:

curl -G https://serpapi.com/search \
  -d engine=naver_ai_overview \
  -d api_key=$SERPAPI_KEY \
  --data-urlencode query="apple juice vs orange juice"

Ruby Example

All examples below use SerpApi's Ruby library. SerpApi also provides integrations for Python, JavaScript, PHP, Go, Java, and more, so you can use whatever language suits your stack.

Install the gem and set your API key:

gem install serpapi
export SERPAPI_KEY=your_secret_api_key

Then you can query the Naver AI Overview API and process the results:

require "serpapi"

client = SerpApi::Client.new(
  engine: "naver_ai_overview",
  api_key: ENV["SERPAPI_KEY"]
)

results = client.search(query: "apple juice vs orange juice")

# The full AI answer as Markdown
puts results[:markdown]

# Walk structured text blocks with source references
results[:text_blocks].each do |block|
  case block[:type]
  when "header"
    puts "\n## #{block[:snippet]}"
  when "paragraph"
    puts block[:snippet]

    block[:reference_indexes]&.each do |i|
      ref = results[:references]&.find { it[:index] == i }
      puts " -> #{ref[:title]} (#{ref[:link]})" if ref
    end
  when "list"
    block[:list]&.each { |item| puts "- #{item[:snippet]}" }
  end
end

Example: Tracking Citations in Naver AI Briefings

Here's a concrete use case: tracking citations. Suppose you want to know whether Naver cites serpapi.com when someone searches “how to get Google search results as an API?”, and if so, what it says.

DOMAIN = "serpapi.com"
QUERY = "how to get Google search results as an API?"

# 0. Perform the desired search
results = client.search(query: QUERY)

# 1. Find our domain in references
references = results[:references] || []
refs = references.select { it[:link].include? DOMAIN }

if refs.empty?
  puts "#{DOMAIN} was not cited."
  return
end

puts "#{DOMAIN} was cited #{refs.length} time(s):"
refs.each { |ref| puts "- [#{ref[:index]}] #{ref[:title]} (#{ref[:link]})" }

# 2. Find every paragraph that cites our domain
ref_indexes = refs.map { it[:index] }

results[:text_blocks].each do |block|
  next unless block[:type] == "paragraph"
  next unless block[:reference_indexes]&.intersect?(ref_indexes)

  puts "Paragraph: #{block[:snippet]}"
  puts " Sources: #{block[:reference_indexes].inspect}"
end

This scans references for your domain, then walks text_blocks to find which paragraphs cite you via reference_indexes. Here's what it looks like:

serpapi.com was cited 1 time(s):
- [4] Google Search Engine Results API - SerpApi (https://serpapi.com/search-api)

Paragraph: Google 검색 결과를 API처럼 가져오려면, 공식적으로는 Google Search Console API를 통해 사이트의 검색 결과(검색 애널리틱스) 데이터를 프로그래매틱하게 조회할 수 있습니다. 반면, 일반 사용자 검색 결과(웹 검색 결과 페이지)를 그대로 가져오려면 Google에서 공식 API를 제공하지 않으므로, 서드파티 SERP API나 스크래퍼를 이용하는 방식이 안내됩니다.
  Sources: [1, 2, 3, 4]

Paragraph: 또는 Apify의 Google Search Results Scraper처럼 검색 결과를 스크래핑하는 방식도 안내되지만, 비용 구조(이벤트 기반)와 결과가 국가, IP, 브라우저 지문 등에 따라 달라질 수 있다는 점이 함께 언급됩니다.
  Sources: [4]

Paragraph: serpapi.com, https://serpapi.com/search-api
  Sources: [4]

Run this across target queries on a schedule, and you have a basic GEO tracker for the Korean search market.

Conclusion

The citation tracker above is just a starting point. Try swapping in your own domain and target queries, pair the Naver AI Overview API with the Naver Search API to compare AI-generated answers against organic results, or run queries on a schedule to track how Naver's citations shift over time.

New to SerpApi? Sign up and receive 250 free credits per month.

Building a Naver News Watchlist With Ruby on Rails

Braden — Wed, 01 Jul 2026 14:31:48 +0000

The problem with monitoring news feeds isn’t finding news; it’s wasting time chasing what’s actually relevant. Refreshing feeds, re-running the same searches, skimming overlapping headlines across numerous tabs and apps, clicking into the same article 10 times before realizing you’ve already read it — it’s exhausting.

Let’s flip that script: you decide exactly which topics and outlets you care about, how many items you want per topic, and how often you want them, and a small, API-only Rails service pulls fresh Naver News results via SerpApi and quietly delivers a single, fully customized digest straight to your inbox.

Sound intriguing? Let’s build it.

What is Naver?

Naver is South Korea’s flagship search portal, and is central to how millions of Koreans discover news, content, and services. Historically dominant in domestic search, Naver remains especially strong on mobile and tablet usage, even as Google has taken the lead on desktop.

For recent market-share context, see StatCounter’s Korea page.

Why SerpApi for Naver News?

Naver’s HTML can change frequently; scraping the page directly is difficult and costly to maintain. SerpApi handles the tricky parts for you:

Stable, normalized JSON: The news_results array returned by SerpApi remains consistent and predictable, even as Naver’s frontend changes.
Proxy and CAPTCHA handling: No need to manage IP rotation or deal with CAPTCHAs.
Smart caching: Identical searches return cached results for free. (Cache expires after 1 hour. You can disable caching by setting no_cache=true. Learn more)
First-class Ruby library: An official Ruby Library by SerpApi that is easy-to-use, well-documented, and consistent in behavior.

Using SerpApi lets you focus on what you want to build, rather than on keeping your scraper alive.

Try it First

Before writing any code, you can explore Naver News Results in the SerpApi Playground. Inspect the exact JSON structure returned by SerpApi, test advanced filtering parameters like period and sort_by, and confirm which fields you want to use.

Check out the Documentation for examples and additional details.

What We’ll Build

Follow along as we build an API-only Rails 8 app that:

Fetches Naver News Results via SerpApi for your keywords watchlist
Filters results in memory
Emails a single daily digest with your desired number of items
Leverages SerpApi’s free caching to minimize API credit usage

This tutorial demonstrates a concrete example of a lightweight news monitoring service. For a broader overview and other patterns, see SerpApi’s News Monitoring Services use case page.

End Result

Here’s an example of what the email digest will look like when we’re finished:

Prerequisites

Most recent versions of Ruby and Rails (as of the time of this writing):
- Ruby 4.0.0 (newly released December 25, 2025)
- Rails 8.1.1
Bundler 4.0.3
A SerpApi account and API key
- Sign up for free
- Get your API key
SerpApi’s official Ruby Library
Basic familiarity with Ruby on Rails

Sample JSON Result

To see what we’ll be working with, here’s a partial sample of a single item from the news_results array returned by SerpApi’s Naver News Results:

{
  "position": 1,
  "title": "[기고] 애그테크로 여는 전북농업의 대전환, 지금이 골든타임",
  "link": "https://www.jjan.kr/article/20260107500290",
  "thumbnail": "https://search.pstatic.net/common/?src=https%3A%2F%2Fimgnews.pstatic.net%2Fimage%2Forigin%2F5335%2F2026%2F01%2F07%2F311174.jpg&type=ofullfill208_208_dominantcolor&expire=2&refresh=true",
  "news_info": {
    "news_date": "11시간 전",
    "press_name": "전북일보",
    "press_link": "http://www.jjan.kr",
    "press_icon": "https://search.pstatic.net/common/?src=https%3A%2F%2Fmimgnews.pstatic.net%2Fimage%2Fupload%2Foffice_logo%2F5335%2F2024%2F05%2F30%2Flogo_5335_18_20240530182450.png&type=f54_54&expire=24&refresh=true"
  },
  "snippet": "“AI가 설계하고, 데이터가 실행하며, 사람이 완성한다.” 전북이 그 중심에 설 수 있는 시간은 바로 지금이다. 지금이 골든타임이다. 김창길 서울대 아시아연구소 방문학자 · 전 한국농촌경제연구원장 이준서 기자 /이준서 imhendsome@naver.com /이준서"
}

We’ll use the following:

position → the rank of the news article in the results
title → the headline of the news article
link → the URL for the news article
snippet → a brief excerpt from the article
news_info.news_date → human-readable time of when the article was published
news_info.press_name → the name of the outlet
news_info.press_link → URL to the outlet’s page on Naver News

Building the App

1. Create an API-only Rails app

We don’t need a frontend for this app, so we’ll use the API-only template, with no database:

rails new naver_news_watchlist --api --skip-active-record
cd naver_news_watchlist

Optional

If you want to include a database so you can persist news results, simply omit --skip-active-record.

2. Add the SerpApi gem

Next, add the SerpApi gem to your app and install it with:

bundle add serpapi

3. Store your API key

Retrieve your API key from your SerpApi dashboard and store it securely in Rails credentials:

rails credentials:edit

Add:

serpapi:
  api_key: YOUR_API_KEY

Close the file to save it. Now you can securely access your API key anywhere in your app with:

Rails.application.credentials.dig(:serpapi, :api_key)

4. Service: Naver News Fetcher

We’ll isolate all SerpApi interaction into a single service object. This keeps API concerns out of jobs and models, makes testing easier, and gives us a single place to tune default parameters (period, sort_by, etc.) later.

Create the file for the service object:

mkdir app/services
touch app/services/naver_news_fetcher.rb

Then paste this into the file you just created:

class NaverNewsFetcher
  DEFAULT_PARAMS = {
    engine: "naver",
    where: "news",
    sort_by: DailyWatchlistJob::SORT_MAP[:relevance], # Relevance
    period: "1d" # 1 day
  }.freeze

  attr_reader :base_params

  def initialize(query:, **options)
    @base_params = DEFAULT_PARAMS.merge(options).merge(query:)
  end

  def self.call(page: 1, **args)
    new(**args).call(page:)
  end

  def call(page: 1)
    client = SerpApi::Client.new(api_key: serpapi_key)
    client.search(page:, **base_params)
  end

  private

  def serpapi_key
    Rails.application.credentials.dig(:serpapi, :api_key)
  end
end

We’ll create the DailyWatchlistJob with SORT_MAP a bit later. By default, we’re sorting results by relevance, and we’re using period: "1d" to limit results to the last 1 day. You can override these defaults per watchlist item later, or by passing explicit keyword arguments when initializing NaverNewsFetcher. Any other supported SerpApi parameters (e.g., device, no_cache, etc.) can be passed via the options hash and will be merged into the request.

For a full list of supported values for sort_by and period, see the Advanced Filters Documentation for SerpApi’s Naver Search API.

5. Email digest with Action Mailer

Once the news results are fetched, we assemble a short digest and send it somewhere you’ll actually see it. For this example, we’ll send a simple email digest.

First, update your ApplicationMailer in app/mailers/application_mailer.rb:

class ApplicationMailer < ActionMailer::Base
  default from: "Naver News Watchlist <#{Rails.application.credentials.dig(:smtp, :user_name)}>"
  layout false
end

Note

The default from: address will be pulled from your encrypted Rails credentials, which we’ll configure in the next step.

Then generate the daily digest mailer:

rails g mailer daily_digest

Paste the following into the generated app/mailer/daily_digest_mailer.rb:

class DailyDigestMailer < ApplicationMailer
  def digest_email(recipient_email, sections)
    @sections = sections

    digest_date = Time.zone.today.strftime("%F")
    @title = "Naver News Watchlist Digest — #{digest_date}"

    mail(
      to: recipient_email,
      subject: @title,
      charset: "UTF-8", # ensures Korean text is rendered correctly
    )
  end
end

Create a plain-text template:

touch app/views/daily_digest_mailer/digest_email.text.erb

And add:

<%= @title %> (<%= pluralize(@sections.sum { it[:items].size }, "result") %>)

<% sections_list = @sections.map do |section|
  items         = section[:items]
  results_count = items.size

  section_title = "#{pluralize(results_count, 'result')} for \"#{section[:query]}\""

  device = section.fetch(:device, "desktop")
  sort   = DailyWatchlistJob::SORT_MAP.key(section.fetch(:sort_by, 0)).to_s.titleize

  period = section.fetch(:period, "1d")

  period = if (match = period.match(/from(\d{8})to(\d{8})/))
    begin
      from_date, to_date = match.captures.map(&:to_date)
      "from #{from_date} to #{to_date}"
    rescue Date::Error
      'unknown period'
    end
  else
    DailyWatchlistJob::PERIOD_MAP.fetch(period, 'unknown period')
  end

  filtered_by_press = section[:press_names]&.join(', ')

  meta_parts = { device:, sort:, period:, filtered_by_press: }.compact_blank.map { |key, value| "#{key.to_s.humanize}: #{value}" }
  section_details = " (#{meta_parts.join(", ")})"

  punctuation = results_count.zero? ? "." : ":"

  header = [ section_title, section_details, punctuation ].join

  body = items.map do |item|
    [
      "  #{item[:position]}) \"#{item[:title]}\"",
      "    Date:    #{item[:news_date]}",
      "    Link:    #{item[:link]}",
      (
        "    Source:  #{[
          item[:press_name].presence,
          item[:press_link].present? ? " (#{item[:press_link]})" : nil
        ].compact.join}" if item[:press_name].present? || item[:press_link].present?
      ),
      ("    Snippet: \"#{item[:snippet]}\"" if item[:snippet].present?)
    ].compact.join("\n")
  end.join("\n\n")

  [ header, body ].join("\n\n")
end %>

<%= sections_list.join("\n\n\n---\n\n\n") %>

It looks like there’s a lot going on here, but all we’re really doing is iterating through @sections (which are grouped by keyword, i.e., query), formatting each item so it can be printed neatly, and rendering each keyword as its own section. Take a look at the source code on GitHub to see how I extracted much of the formatting logic to a DailyDigestSectionFormatter class.

6. Configure an email delivery method

To keep it simple, we’ll use Gmail SMTP with our credentials stored in Rails encrypted credentials.

Step 1: Generate a Gmail App Password

Gmail does not allow SMTP access using your normal account password. Instead, you must create an App Password.

Go to your Google Account security page: https://myaccount.google.com/security
Ensure 2-Step Verification is enabled (App Passwords are unavailable without it)
Create an App Password: https://myaccount.google.com/apppasswords Assign it a memorable name, like “Rails Naver News Watchlist.”
Google will generate a 16-character password. Copy it or write it down now — you will not be able to see it again.

This password will be used by Rails to authenticate with Gmail’s SMTP server.

Step 2: Store SMTP credentials in Rails credentials

Run:

rails credentials:edit

Add the following:

smtp:
  user_name: YOUR_EMAIL@gmail.com
  password: YOUR_APP_PASSWORD

Replace YOUR_EMAIL@gmail.com with your gmail address and YOUR_APP_PASSWORD with the app password obtained in the previous step. The ApplicationMailer we modified above will automatically use smtp.user_name as the sender address.

Step 3: Configure Action Mailer

Configure Action Mailer to work in production by setting the following in config/environments/production.rb:

config.action_mailer.delivery_method = :smtp

config.action_mailer.smtp_settings = {
  user_name: Rails.application.credentials.dig(:smtp, :user_name),
  password: Rails.application.credentials.dig(:smtp, :password),
  address: "smtp.gmail.com",
  port: 587,
  authentication: :plain,
  enable_starttls_auto: true
}

config.action_mailer.perform_deliveries = true
config.action_mailer.raise_delivery_errors = true

To test our emails locally, we’ll use the letter_opener gem. Run this command to add the gem:

bundle add letter_opener --group development

Then edit config/environments/development.rb:

config.action_mailer.delivery_method = :letter_opener
config.action_mailer.perform_deliveries = true

Now when you try to send emails in development, letter_opener will now automatically open them in your default browser. Next, let’s create the job to actually send the emails.

7. Create the Daily Watchlist Job

Generate the job:

rails g job daily_watchlist

This creates a new job in app/jobs/daily_watchlist_job.rb. This job will coordinate the entire flow:

Fetch news for each watchlist item
Filter duplicates and unwanted press names
Group items by keyword into separate sections
Build a readable digest
Send the email

Paste the following into app/jobs/daily_watchlist_job.rb:

class DailyWatchlistJob < ApplicationJob
  queue_as :default

  RECIPIENT_EMAILS = ["you@example.com", "another@example.com"].freeze # replace with desired emails
  RESULTS_PER_PAGE = 10 # Naver News shows 10 results per page
  MAX_PAGES = 5

  SORT_MAP = {
    relevance: 0, # default
    latest: 1,
    oldest: 2
  }.freeze

  PERIOD_MAP = {
    "all" => "all time", # default
    "1h" => "1 hour",
    "2h" => "2 hours",
    "3h" => "3 hours",
    "4h" => "4 hours",
    "5h" => "5 hours",
    "6h" => "6 hours",
    "1d" => "1 day",
    "1w" => "1 week",
    "1m" => "1 month",
    "3m" => "3 months",
    "6m" => "6 months",
    "1y" => "1 year"
  }.freeze

  WATCHLIST = [
    {
      # Industry trends
      query: "coffee",
      press_names: [
        "매일경제", # Maeil Business Newspaper
        "한국경제", # Korea Economic Daily
        "조선비즈", # Chosun Biz
        "토큰포스트" # TokenPost
      ], # filter by specific press sources
      max: Float::INFINITY, # set max to Float::INFINITY to allow unlimited results up to MAX_PAGES
      sort_by: SORT_MAP[:latest], # sort by latest news
      period: "from20251201to20260105" # custom date range
    },
    {
      # Topical news
      query: "Naver AI",
      press_names: [], # set press_names to `[]` to allow all press sources
      max: RESULTS_PER_PAGE # set max to RESULTS_PER_PAGE to fetch only the first page
    },
    {
      # Competitor monitoring
      query: "Kakao Enterprise",
      press_names: nil, # set press_names to `nil` to allow all press sources
      max: RESULTS_PER_PAGE * 2, # set max greater than RESULTS_PER_PAGE to fetch multiple pages
      sort_by: SORT_MAP[:latest],
      period: "1m",
      device: "tablet"
    },
    {
      # Technology updates
      query: "AI semiconductor",
      # omit press_names to allow all press sources
      # omit max to allow unlimited results up to MAX_PAGES
      sort_by: SORT_MAP[:latest],
      period: "3m"
    },
    {
      # Fresh breaking news with a shorter time frame
      query: "Generative AI",
      max: nil, # set max to `nil` to allow unlimited results up to MAX_PAGES
      sort_by: SORT_MAP[:latest],
      period: "3h",
      device: "mobile" # simulate mobile device results
    }
  ].freeze

  def perform
    sections = sections_for_watchlist
    sections = empty_sections if sections.all? { |section| section[:items].empty? }

    post_digest sections
  end

  def sections_for_watchlist
    WATCHLIST.map { |item| build_section(item) }
  end

  private

  def build_section(item)
    {
      **item.without(:max),
      items: fetch_and_select(item)
    }
  end

  def empty_sections
    [
      {
        query: nil,
        items: []
      }
    ]
  end

  def fetch_and_select(item)
    seen = []
    selected = []

    limit = item[:max] || Float::INFINITY
    page = 1

    while selected.size < limit && page <= MAX_PAGES
      options = item.without(:max, :press_names).merge(page:)

      api_response = NaverNewsFetcher.call(**options)

      results = api_response[:news_results].to_a
      break if results.empty?

      results.each do |result|
        break if selected.size >= limit

        link = result[:link].to_s
        next if link.blank? || link.in?(seen)

        seen << link

        press_name = result.dig(:news_info, :press_name).to_s
        next unless item[:press_names].blank? || press_name.in?(item[:press_names])

        rank = result[:position].to_i + (page - 1) * RESULTS_PER_PAGE # calculate rank across pages

        selected << {
          position: rank,
          title: "result[:title].to_s,"
          snippet: result[:snippet].to_s,
          link:,
          news_date: result.dig(:news_info, :news_date).to_s,
          press_name:,
          press_link: result.dig(:news_info, :press_link).to_s
        }.compact
      end

      page += 1
    end

    selected
  end

  def post_digest(sections)
    DailyDigestMailer.digest_email(RECIPIENT_EMAILS, sections).deliver_later
  end
end

Scheduling

To run the watchlist automatically every day, you can use a scheduler like cron or the whenever gem to run the job at your desired time.

8. Error Handling & Reporting

To ensure you’re notified of any issues during the job execution, we’ll add some simple error handling and email reporting.

Step 1: Add an Error Email Method and Template

First, update app/mailers/daily_digest_mailer.rb to include an error email method:

class DailyDigestMailer < ApplicationMailer
  # existing digest_email method...

  def error_email(recipient_email, error_payload)
    @error_class = error_payload[:class]
    @error_message = error_payload[:message]
    @error_backtrace = Array(error_payload[:backtrace])

    @error_time = Time.zone.now.strftime("%F %R %Z")
    @title = "Naver News Watchlist Error — #{@error_time}"

    mail(
      to: recipient_email,
      subject: @title,
      charset: "UTF-8",
    )
  end
end

Create the corresponding view template app/views/daily_digest_mailer/error_email.text.erb:

<%= @title %>

<%
  lines = [
    "Time: #{@error_time}",
    "Error: #{@error_class} - #{@error_message}",
    "",
    "Backtrace:"
  ]

  backtrace_lines = @error_backtrace.map { " #{it}" }

  body = (lines + backtrace_lines).join("\n")
%>

<%= body %>

Step 2: Update Naver News Fetcher with Error Handling

Update the call method in app/services/naver_news_fetcher.rb to report errors when an error occurs:

class NaverNewsFetcher
  # existing code...

  def call(page: 1)
    client = SerpApi::Client.new(api_key: serpapi_key)
    client.search(page:, **base_params)
  rescue => error
    # Ignore "no results" errors
    return empty_payload if error.message.include? "Naver hasn't returned any results for this query"

    Rails.error.report(
      error,
      handled: false,
      severity: :error,
      context: {
        source: self.class.name,
        page:,
        **base_params.without(:engine, :where)
      }.compact
    )
    raise error
  end

  private

  # existing code...

  def empty_payload
    {
      search_metadata: { status: "Success" },
      news_results: []
    }
  end
end

Step 3: Update Daily Watchlist Job for Error Handling

Update the perform method in app/jobs/daily_watchlist_job.rb to handle exceptions and send error emails:

class DailyWatchlistJob < ApplicationJob
  # existing code...

  def perform
    sections = sections_for_watchlist
    sections = empty_sections if sections.all? { |section| section[:items].empty? }

    post_digest sections
  rescue => error
    Rails.error.report(
      error,
      handled: false,
      severity: :error,
      context: {
        job: self.class.name,
        watchlist_queries: WATCHLIST.pluck(:query),
        recipients: RECIPIENT_EMAILS
       }
    )

    error_payload = {
      class: error.class.name,
      message: error.message,
      backtrace: error.backtrace
    }

    DailyDigestMailer.error_email(RECIPIENT_EMAILS, error_payload).deliver_later
    raise error
  end

  # existing code...
end

Update the fetch_and_select method and add ensure_serpapi_success! in app/jobs/daily_watchlist_job.rb to raise errors when SerpApi responses indicate failure:

class DailyWatchlistJob < ApplicationJob
  # existing code...

  def fetch_and_select(item)
    # existing code...

    while selected.size < limit && page <= MAX_PAGES
      options = item.without(:max, :press_names).merge(page:)

      api_response = NaverNewsFetcher.call(**options)
      ensure_serpapi_success! api_response, options[:query], page # <-- add this line

      results = api_response[:news_results].to_a
      break if results.empty?

      results.each do |result|
        # existing code...
      end

      page += 1
    end

    selected
  end

  def ensure_serpapi_success!(api_response, query, page)
    status = api_response.dig(:search_metadata, :status)
    return if status == "Success"

    error_message = api_response[:error] || "Unknown SerpApi error"
    error = StandardError.new(
      "SerpApi Naver search failed (query=#{query.inspect}, page=#{page}): " \
      "status=#{status || 'unknown'} error=#{error_message}"
    )

    Rails.error.report(
      error,
      handled: false,
      severity: :error,
      context: {
        source: "SerpApi:NaverNews",
        query:,
        page:,
        serpapi_status: status,
        serpapi_error_message: error_message
      }
    )

    raise error
  end

  # existing code...
end

9. Run & Test

You can run the following commands in the Rails console to test each part of the system:

Fetch news results:

results = NaverNewsFetcher.call(query: "coffee")
news_results = results[:news_results]

Generate watchlist sections:

sections = DailyWatchlistJob.new.sections_for_watchlist

Run the full job:

DailyWatchlistJob.perform_now

letter_opener will automatically open the digest email in your default browser. It should look something like this:

Naver News Watchlist Digest — 2026-01-07 (25 results)


2 results for "coffee" (Device: desktop, Sort: Latest, Period: from 2025-12-01 to 2026-01-07, Filtered by press: 매일경제, 한국경제, 조선비즈, 토큰포스트):

  1) Vanadi Coffee, 비트코인 12개 추가 매입…총보유량 173개
    Date: 7시간 전
    Link: https://www.tokenpost.kr/news/breaking/321889
    Source: 토큰포스트 (https://www.tokenpost.kr)
    Snippet: 암호화폐 전문 매체 오데일리(Odaily)에 따르면, 커피 브랜드 Vanadi Coffee가 최근 비트코인(BTC) 12개를 추가로 매입했다. 현재까지 이 회사의 총 비트코인 보유량은 173개로 확대됐다. 이번 매입으로 Vanadi Coffee는 비트코인 보유량 기준 글로벌 기업 순위에서 91위로 올라섰다. Vanadi Coffee는... 

  3) [저녁 뉴스브리핑] 국회, 내달부터 암호화폐 현물 ETF 도입 관련 법안 논...
    Date: 13시간 전
    Link: https://www.tokenpost.kr/news/briefing/321785
    Source: 토큰포스트 (https://www.tokenpost.kr)
    Snippet: 스페인 커피 브랜드 바나디, 173 BTC 보유 중 스페인 커피 브랜드 바나디 커피(Vanadi Coffee)가 1월 6일(현지시간) 기준 173 BTC를 보유하고 있다고 공식 X를 통해 전했다. 바나디는 지난해 커피 사업에서 비트코인 중심 기업으로 전환하겠다고 밝히며, BTC 준비금 조성을 위해 총 11억 달러를 투자할... 


---


10 results for "Naver AI" (Device: desktop, Sort: Relevance, Period: 1 day, Filtered by press: ):

  1) 수자원공사, CES 2026서 글로벌 물관리 AI기술 선봬
    Date: 11시간 전
    Link: https://www.asiatoday.co.kr/view.php?key=20260107010003219
    Source: 아시아투데이 (http://www.asiatoday.co.kr)
    Snippet: 한국수자원공사가 CES 2026에서 국내 물기업들과 함께 물관리 AI기술혁신의 저력을 세계에 알린다.수자원공사는 9일까지 미국 라스베이거스에서... 함께 참가한 국내 물기업 21개 사 중 7개 기업이 CES 최고혁신상 및 혁신상을 수상하며 주목을 받고 있다. 이진희 william614@naver.com

  2) 미래도시 전환…익산시, ‘3+AI’ 전략 본격 추진
    Date: 12시간 전
    Link: https://www.jjan.kr/article/20260107500091
    Source: 전북일보 (http://www.jjan.kr)
    Snippet: 농업·식품·바이오에 AI 성장 엔진 더해 산업 구조 혁신 그동안 성과 바탕으로 미래 대응 및 도시 경쟁력 강화 정헌율 익산시장이 7일 신년 브리핑에서... “올해도 동심동덕(同心同德)의 자세로 시민과 함께 호흡하며 도시의 미래를 만들어 가겠다”고 다짐했다. 익산=송승욱 기자 /송승욱 ssw791221@naver.com /송승욱

  3) 업스테이지 이어 네이버까지…독자 AI 모델 불거진 표절 논란
    Date: 10시간 전
    Link: https://www.mt.co.kr/tech/2026/01/07/2026010718161644740
    Source: 머니투데이 (https://media.naver.com/press/008)
    Snippet: 앞서 업스테이지에 이어 이번에는 네이버(NAVER)다. 네이버가 중국산 모델을 베꼈다는 지적이다. 이와 관련 네이버는 "AI 모델에서 가장 핵심적인 기반은 자체 기술로 개발했다"는 입장을 내놓으며 '프롬 스크래치' 관련 "명확한 기준이 필요하다"고 했다. 7일 IT 업계에 따르면 최근 세계 최대 오픈소스 플랫폼인... 

...

Note

The two results for the “coffee” query are ranked 1 and 3 because other results were filtered out based on the specified press_names in the WATCHLIST.

Where to Go Next

From here, you can easily extend the system:

Set up a scheduler to run the DailyWatchlistJob automatically every day.
Translate the Korean text to another language automatically using a translation API.
Send digests to other destinations (e.g., Slack, Discord) via webhooks.
Enhance filtering with keywords or sentiment analysis to highlight the news that’s most relevant to you.

Even without any of that, this watchlist does exactly what it needs to do — it delivers relevant and timely news results, without distractions.

Build A No Code AI-Powered Local Lead Outreach System

Sonika Arora — Wed, 01 Jul 2026 07:07:22 +0000

Google Maps business listings contain valuable lead data. The challenge is finding the right businesses, qualifying them, and personalizing outreach at scale. What if you could automate the entire process including discovering businesses, generating personalized outreach messages, and sending leads directly to your CRM?

In this tutorial, we'll build a fully automated local lead outreach workflow without using any code with:

Workflow Overview

Example use case we'll work on: Find dentist offices in Miami whom you can sell your monthly subscription based appointment scheduling software to and draft personalized emails to them.

The automation will follow this process:

Extract businesses from Google Maps using SerpApi
Format and send information for each business to Claude to generate personalized outreach
Push lead data along with the personalized message into your CRM (Google Sheets in this example)

The entire workflow uses no code and runs in Make.

Step 1: Setup a Make scenario

If you haven't used SerpApi with Make, I recommend checking this blog post for setup steps:

Announcing SerpApi’s Make App

To begin, in Make, we'll create a new scenario. To make requests to SerpApi, we'll use SerpApi's Make integration.

💡 In case you don't see our Google modules, please use the Universal Module to construct API calls to any of our Google APIs.

Here's the flow we'll setup:

SerpApi Search Google Maps module (or the Universal Module) -> Iterator module -> Claude Simple Text Prompt module -> Google Sheets Add a Row module

The setup in Make

Here's the scenario for this blog post example so you can follow along:

Integrate SerpApi, Claude, Google Sheets - Make.com Automation Scenario

Step 2: Extract Google maps data with SerpApi

If you don't already have a SerpApi account, please register an account. You'll need your API key to set up the SerpApi integration in Make. Once you have an active account, you can find your API key here.

We’ll start by using the SerpApi Google Maps API to search for businesses.

Example query:

“dentists in Miami”
“restaurants in Austin”
“law firms in Chicago”

Example API request:

https://serpapi.com/search.json?engine=google_maps&q=dentists+in+miami&api_key=YOUR_API_KEY

The response includes:

Business names
Ratings
Reviews
Phone numbers
Websites
Coordinates
Categories
Hours
And more

Try out your request in our playground to visualize the result:

SerpApi Playground - SerpApi

With the Search Google Maps Module in Make, it looks like this:

Search Google Maps Module Setup in Make

💡 In case you don't see our Google modules, please use the Universal Module to construct API calls to any of our Google APIs.

With the Universal Module, it looks like this:

SerpApi's Universal Module Setup in Make

This gives you structured local business JSON data directly from Google Maps.

Step 3: Iterate through businesses

Google Maps results return multiple businesses matching the query. We'll use the Iterator module in Make to process each business individually.

To begin with, we'll focus on a few super useful fields found in the local results array like:

title
website
phone
rating
reviews
address

Iterator Module Setup in Make

To extend this solution, you can also filter leads by specific things like:

Only businesses with ratings below 4.0
Only businesses without websites
Only businesses with less than 50 reviews

This is especially useful for targeted prospecting.

Step 4: Generate personalized outreach with Claude

Now we connect Claude in Make. This step transforms raw lead data into personalized and actionable outreach.

Claude Module Setup in Make

💡 I chose the model Claude Opus 4.7. Feel free to change it according to your use case.

Here's the prompt I used:

Write a short cold outreach email for this business.

Details about the business from Google Maps: 
Title: {{title}}
Type of business:{{type}}
Rating: {{rating}}
Reviews: {{reviews}}
Website: {{website}}
Address: {{address}}
Phone: {{phone}}
User review from Google: {{user_review}}

For each entry, draft a personalized email to reach out and ask if they need an appointment scheduling software. I own a company called XYZ and we sell a monthly subscription based appoiintment scheduling software. Link my website called xyz.com.

Sample output for one of the dental offices:

Subject: Helping Relax and Smile Dental Care Save Time on Scheduling

Hi Relax and Smile Dental Care Team,

Congratulations on building such an outstanding reputation in Miami — a 4.9 rating with 883 reviews is no small feat! I noticed patients consistently rave about your efficiency, with one recent review highlighting "no wait time to see the hygienist" and a cleaning done in just 30 minutes. That kind of streamlined experience is exactly what keeps patients coming back.

My name is [Your Name], and I'm the founder of XYZ — a monthly subscription-based appointment scheduling software designed specifically for busy practices like yours. Our platform helps dental offices reduce no-shows, automate reminders, and let patients self-book online 24/7, freeing up your front desk to focus on in-person care.

Given how much your patients value your quick, gentle service, I think XYZ could help you maintain that pace as your practice continues to grow.

Would you be open to a quick 15-minute call this week to see if it's a fit? You can also learn more at xyz.com.

Looking forward to hearing from you,

[Your Name]

Founder, XYZ

xyz.com

Step 5: Push leads into a Google sheet

Finally, connect Google Sheets using the Google Sheet Integration in Make.

We'll note these fields in this blog post, but feel free to add any more you need to keep track of:

Business name
Website
Phone
Address
AI-generated outreach message

Create a sheet named "Dentist offices Make.com automation sheet" and then add the relevant details to the Google Sheets Add a Row module.

Google Sheets Add a Row Module Setup in Make

You can also do more with the data later like adding lifecycle stages in the sheet and/or triggering email sequences.

At this point, we have a fully automated AI-powered lead generation system.

Here's what the output looks like:

Output in Google Sheets

Conclusion

I hope it was useful to see how we can use SerpApi with Make to build a workflow that automatically discovers businesses, gets structured data about them, generates personalized outreach using Claude, and pushes qualified leads directly into your CRM (Google Sheets in this example).

Here's the Make scenario we created in this blog post:

Integrate SerpApi, Claude, Google Sheets - Make.com Automation Scenario

I hope you found this useful. If you have any questions, feel free to reach out at contact@serpapi.com.

Relevant Links

Turn Bing News into an AI Briefing with Claude and SerpApi

Sonika Arora — Tue, 30 Jun 2026 07:06:28 +0000

Most news apps and search engines already present headlines well, so simply listing them is becoming less valuable. The real value has shifted to connecting the dots: grouping related news stories, identifying patterns across sources, and explaining what actually matters and why.

In this post, we’ll build a simple pipeline that turns raw Bing News results into a structured daily briefing which does exactly that using:

SerpApi's Bing News API for real-time news data
Claude for summarization and insight generation

Setting Up

LLMs work best when you give them clean, formatted context instead of raw scraped HTML, so we'll do exactly that.

Get results using SerpApi (Bing News API) → Normalize → Send to Claude with a relevant prompt → Get structured briefing

For this tutorial, we're going to use SerpApi's new Python library to get results from search.

Install SerpApi's new Python serpapi library, the python-dotenv library (for using environment variables) and anthropic library (for Claude) in your environment:

pip install serpapi python-dotenv anthropic

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API key from this page.
[Optional but Recommended] Set your API keys in an environment variable, instead of directly pasting them in the code. Refer here to understand more about using environment variables.

SERPAPI_API_KEY=
CLAUDE_API_KEY=

Get your SerpApi API key here: https://serpapi.com/manage-api-key

Get your Claude API key here: https://platform.claude.com/settings/keys

💡 You'll need to create an account with Claude in case you don't have one yet. You can do that here: https://claude.ai/login

Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import json
import anthropic
from dotenv import load_dotenv
load_dotenv()

SERPAPI_API_KEY = os.environ["SERPAPI_API_KEY"]
CLAUDE_API_KEY = os.environ["CLAUDE_API_KEY"]

Fetch News with Bing News API

We’ll write a function to use SerpApi’s Bing News API (bing_news engine) and get all the news results.

def fetch_news(query):
    client = serpapi.Client(api_key=SERPAPI_API_KEY)
    results = client.search({
        "engine": "bing_news",
        "q": query,
        "device": "desktop",
        "mkt": "en-au"
    })
    return results.get("organic_results", [])

We can call this function with any query such as "Self driving software" or "AI" and scrape results from Bing News.

You can test the Bing News API out in our playground with parameters of your choice:

SerpApi Playground - SerpApi

Prompting Claude

Good prompts can make all the difference when building a AI briefing based on the results.

A bad prompt would look like:

“Summarize these articles”

Better prompts often define structure, force grouping, and require reasoning.

def build_prompt(articles):
    return f"""
You are a technical news analyst.

Task: Convert raw news articles into a structured daily briefing.

Requirements:
- Group related articles into 3-5 themes
- Each theme must have:
  - A short title
  - A 2-3 sentence summary
  - A "Why it matters" explanation
- Be concise and non-generic
- Avoid repeating the same point across themes

Articles:
{articles}
"""

Generate the Briefing (Claude API)

Using Claude's API, we can turn the data we got from SerpApi into a briefing. Here is some simple code to use Anthropic API -

def generate_briefing(prompt):
    client = anthropic.Anthropic(api_key=CLAUDE_API_KEY)
    response = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=1500,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text

Putting It Together

Now that we have all the pieces we'll use, let's write the code to call the different functions and get our news briefing:

def run():
    query = "AI"
    articles = fetch_news(query)
    prompt = build_prompt(articles)
    briefing = generate_briefing(prompt)
    print(briefing)

if __name__ == " __main__":
    run()

Now, we're ready to run this file. We can do so by typing python <filename> in the terminal.

Automated Delivery

A briefing is only useful if it reaches you regularly automatically. Add a delivery layer like automated daily email or a slack webhook.

Automated Daily Email

Simple automation run daily via cron or other scheduling methods
Email formatted briefing to yourself or a team

Slack Webhook

Post structured sections into a channel
Good for teams or shared awareness feeds
Enables discussion around themes

Read this blog post for more info on setting up a Slack webhook (The use case in the post is different, but the process remains the same). It also covers how to schedule a job every day using the Python schedule library:

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Generated AI Briefing Example Output

Here's an example run. In my SerpApi call, I set the query to "AI" to keep things simple. I used these parameters for Anthropic API:

model="claude-opus-4-7",
max_tokens=1500,
messages=[{"role": "user", "content": prompt}]

Here's the Claude generated briefing:

Daily AI Briefing

1. AI Reshapes the Workforce: Layoffs, Skill Demands, and Survival Rules

Cloudflare announced a 20% workforce cut (1,100 employees) citing agentic AI as a fundamental disruptor of its operations, sending its stock down 18%. Meanwhile, Airbnb CEO Brian Chesky warned that "pure people managers" and change-resistant workers won't survive the AI era, and CBS reports employers are increasingly demanding AI fluency from new hires.

Why it matters: This is one of the first major tech layoffs explicitly attributed to agentic AI rather than macro conditions—signaling a shift from AI-as-productivity-tool to AI-as-headcount-replacement. Workers and managers face mounting pressure to reskill or face displacement.

2. Conflicting Narratives on AI's Human Impact

Nvidia's Jensen Huang publicly framed AI as a net job creator, urging workers not to fear the technology. In contrast, a new study found that just 10 minutes of AI use measurably impaired cognitive performance on arithmetic and reasoning tasks.

Why it matters: The optimistic vendor narrative (Huang) is increasingly at odds with both labor-market evidence (Cloudflare) and emerging cognitive research. Expect this tension to drive policy debates and shape enterprise adoption guardrails.

3. AI Stocks Diverge as Earnings Season Tests the Hype

SoundHound AI beat Q1 expectations with 51.7% revenue growth but still dropped 12.4%, while Cloudflare cratered 18% post-earnings. Analysts are also reframing the Nvidia-vs-Palantir debate for 2026 as both stocks face renewed scrutiny.

Why it matters: Markets are no longer rewarding AI exposure alone—even strong top-line growth isn't enough. Investors are demanding margin discipline and clear AI monetization paths, marking a maturation point for the sector.

4. Military AI Moves From Theoretical to Operational

The UK's Royal Air Force chief confirmed AI-powered uncrewed fighter aircraft are already being integrated into operations, far ahead of prior timelines. Simultaneously, CNN reports the Pentagon faces growing scrutiny over the legal limits of AI in warfare, including its use in Iran-related operations.

Why it matters: Autonomous combat systems have crossed from R&D to deployment faster than legal and ethical frameworks can adapt. The gap between operational reality and governance is becoming a strategic vulnerability for Western militaries.

5. The Consumer AI Landscape Crystallizes

TechRepublic published a side-by-side comparison of the major chatbots—ChatGPT, Gemini, Copilot, Claude, Perplexity, Grok, DeepSeek, and Meta AI—mapping each to specific use cases and integrations.

Why it matters: With eight serious contenders now in the consumer/prosumer market, differentiation is shifting from raw model quality to ecosystem integration and vertical specialization. The "one assistant to rule them all" thesis is dead; multi-tool workflows are the new norm.

Building On Top Of This

Changing The Model

As of May 2026, here are the options Claude provides and what each is best for:

Opus (Most Intelligent): Best for complex tasks, deep reasoning, coding, and strategic analysis. It is the most powerful but slower and more expensive.
Sonnet (Balanced): It provides the best balance of speed, intelligence, and cost, making it ideal for daily coding, data analysis, and content generation.
Haiku (Fastest): Optimized for speed and cost-efficiency. It is perfect for high-volume tasks, quick responses, and simple, repetitive jobs.

You can test the script with different models and figure out the best one for your use case. I used Claude Opus 4.7 for this demo.

Persisting History

Instead of generating standalone briefings, store each run with a timestamp. This turns your system into a time-series of news summaries. Once you have history, you can:

Compare today vs yesterday (what’s new, what disappeared)
Track theme momentum (what’s increasing or fading)
Ask Claude for delta summaries instead of fresh-only summaries

Example prompt changes:

“Compare today’s briefing with yesterday’s. Highlight new, removed, and shifting themes.”

Conclusion

The key takeaway here is that LLMs work best when they’re grounded in high-quality external data. In this project, SerpApi provides structured, real-time Bing News data, while Claude turns that raw information into something readable and useful: a concise daily briefing with themes, summaries, and context.

Instead of asking a model to know everything, we can give it fresh information and let it focus on synthesis and reasoning.

I hope you found this useful. If you have any questions, feel free to reach out at contact@serpapi.com

Relevant Links

How To Scrape Home Depot Product Reviews

Sonika Arora — Mon, 29 Jun 2026 07:05:48 +0000

Customer reviews are one of the most valuable sources of product insight. They reveal real-world feedback, highlight product strengths and weaknesses, and help businesses understand customer sentiment.

In this post, we'll explore how to collect product reviews from Home Depot using SerpApi's Home Depot Product Reviews API. By the end, you'll be able to fetch product reviews data such as ratings, titles, and comments for analysis and monitoring.

Why Scrape Reviews?

Home Depot is one of the largest home improvement retailers in the United States. Product reviews on its site provide valuable insights for:

Competitive analysis — Understand how competing products are perceived.
Sentiment analysis — Identify common praise or complaints.
Market research — Learn what customers value most.
Product development — Discover improvement opportunities.

Instead of manually collecting reviews, SerpApi lets you retrieve them programmatically and with many no code options.

What Type of Data Can You Extract?

The Home Depot Product Reviews API scrapes reviews for products from Home Depot and provides them in JSON format. Here are some prominent fields from the review object:

id — Unique identifier for the review.
title — Short headline summarizing the reviewer’s opinion.
text — Full written content of the customer review.
rating — Numeric score given by the reviewer (typically on a 1–5 scale).
badges — Indicates badges on products like verifiedPurchaser.
reviewer.name — Display name of the person who wrote the review.
time — Timestamp showing when the review was published.
original_source.name — Website where the review originally appeared.
images.link — URLs of photos included in the review.
total_positive_feedback — Number of users who found the review helpful.
total_negative_feedback — Number of users who found the review unhelpful.

Read the documentation page here:

The Home Depot Product Reviews API - SerpApi

Here's an example in our playground:

SerpApi Playground - SerpApi

Setting up SerpApi

Are you new to SerpApi? We'll walk you through the steps from sign up to integrating with SerpApi.

Getting Started with SerpApi

To get started with SerpApi, I recommend following the steps in this blog post:

Getting started with SerpApi: The Web Search API

Home Depot Product Reviews API Documentation

We recommend familiarizing yourself with the available parameters and details in the documentation here:

The Home Depot Product Reviews API - SerpApi

Integrations / Libraries

Some examples below will include our SerpApi libraries. If you would like to use one of our SerpApi libraries, we offer instructions for each language, such as Python, Node.js, Ruby, PHP, Java, Go, and more:

SerpApi: Integrations

Scraping Home Depot Product Reviews

You can use any programming language in order to call our APIs. The parameters and results we reviewed above will not differ between the request methods, however the syntax used to call the APIs may differ.

You will see across the examples the engine is always set to home_depot_product_reviews, the api_key will be set to your API key, and we include the product_id in every query. In this example we are going to look for reviews for product with ID: 302752040

GET Request

The syntax for the actual GET request between languages will vary between programming languages. However, the URL used each time is consistent. When you are sending a GET request in any language you will use this URL to make the request:

https://serpapi.com/search.json?engine=home_depot_product_reviews&product_id=302752040

Feel free to explore more parameters here, and add any that are relevant to your use case.

An Example With Our Python Library

Install SerpApi's new Python serpapi library in your environment:

pip install serpapi

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

Now, let's take a look at the code:

import serpapi

client = serpapi.Client(api_key=<YOUR_API_KEY>)

results = client.search({
  "engine": "home_depot_product_reviews",
  "product_id": "302752040"
})

Replace the placeholder with your API key.

An Example With cURL

curl --get https://serpapi.com/search \
 -d api_key="YOUR_SECRET_API_KEY" \
 -d engine="home_depot_product_reviews" \
 -d product_id="302752040"

No Code Options

Google Sheets

To connect your code with Google Sheets first follow this blog post:

Connect SERP API with Google Sheets (No Code)

=SERPAPI_RESULT("engine=home_depot_product_reviews&product_id=302752040", "reviews.0.text")

Make.com

While not every API we offer is on Make.com, there is a Universal Module where you can use any API.

Review this blog post to get started:

Announcing SerpApi’s Make App

N8N

Similar to Make.com, not every API is currently included in N8N. However, you can use the HTTP Request node to still call any of our APIs.

To get started review this blog post:

Boost Your n8n Workflows with SerpApi’s Verified Node

Conclusion

Scraping product reviews can unlock valuable insights about customer behavior and product performance. With the Home Depot Product Reviews API, you can retrieve structured review data in just a few lines of code.

I hope you find this useful. If you have any questions, feel free to reach out at contact@serpapi.com

Relevant Links

Related Blogs

Build a Slack Bot That Alerts You When a New Company Ranks for Keywords

Sonika Arora — Sun, 28 Jun 2026 07:05:02 +0000

Monitoring when competitors begin ranking for new keywords can reveal product launches and marketing strategy changes. Instead of manually checking search results every day, you can automate the process with a Slack bot that sends alerts whenever a new company appears for keywords that matter for your brand.

In this tutorial, we’ll build a simple system that:

Checks search results using the SerpApi's Google Search API.
Detects when a competitor domain appears for a tracked keyword and sends an alert to a Slack channel.
Run this process everyday using Python's schedule library.

Setting Up

This system will work in four steps:

Send in requests to SerpApi's Google Search API for a list of relevant queries
Extract all indexed pages
Compare them with previously stored URLs
Send alerts to Slack for new pages detected

We'll run this job every day using the Python schedule library. You can set the schedule as you like.

To start, you’ll need:

A SerpApi account
A Slack workspace (and a Slack Incoming Webhook URL)
Python installed

Create a Slack webhook

Go to Slack → Apps
Search for Incoming Webhooks
Create a webhook for your channel

You will get a Webhook URL like: https://hooks.slack.com/services/XXXX/XXXX/XXXX

Save this, as we'll need it later.

Code related setup steps

For this tutorial, we're going to use SerpApi's new Python library to get results from search.

Install SerpApi's new Python serpapi library, the python-dotenv library, tldextract library (for parsing the domains from URLs), and the schedule library in your environment:

pip install serpapi python-dotenv schedule tldextract

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.
[Optional but Recommended] Set up your Slack Webhook URL as an an environment variable as well. In your .env file, this will look like this:

SERPAPI_API_KEY=
SLACK_WEBHOOK_URL=
Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import requests
import json
import schedule
import time
import tldextract
from dotenv import load_dotenv
load_dotenv()

SERPAPI_KEY = os.environ["SERPAPI_API_KEY"]
queries = [] # List of queries for which you want to find the updated domains
DB_FILE = "known_domains.json"
SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK_URL"]

Provide the value of your slack webhook in the SLACK_WEBHOOK variable. It looks something like this: https://hooks.slack.com/services/XXXX/XXXX/XXXX.

Provide a list of queries you're looking to track competitors with. Example: ["ai images tool", "ai image generators", "image creation ai tools"]

Get all indexed domains for a list of queries

For the purpose of this post, we'll consider the first 30 organic results and use a simple list of 3 queries. You can increase or decrease this number as needed and change the queries as well depending on your use case.

A simple function to get the root domain from the URL using the tldextract library we installed earlier:

def get_root_domain(url):
    extracted = tldextract.extract(url)

    # Join the domain and suffix to get the root domain
    return f"{extracted.domain}.{extracted.suffix}"

Now, let's create a script com get_domains.py where we use SerpApi 's Google Search API to get the domains of the first 30 search results for a particular query.

def get_all_domains(query_list):
    domains_for_each_query = {}
    client = serpapi.Client(api_key=SERPAPI_KEY)

    for query in query_list:
        domains_for_each_query[query] = set()

        for start in range(0, 30, 10):
            results = client.search({
                "engine": "google",
                "google_domain": "google.com",
                "q": query,
                "start": start
            })

            if "organic_results" not in results:
                break

            for result in results["organic_results"]:
                link = result.get("link")
                if not link:
                    continue

                root_domain = get_root_domain(link)
                domains_for_each_query[query].add(root_domain)

    return {k: list(v) for k, v in domains_for_each_query.items()}

This function sends a request to SerpApi's Google Search API and collects the root domains from the organic result links, and stores them per query. It uses a set to keep only unique domains and fetches up to the first 30 results (pagination with start). It returns a dictionary mapping each query to a list of unique domains found in the search results.

💡 If you want to dive deeper into how to scrape Google search results with Python, read this post.

Save known domains

For the purpose of this example, let's assume the file known_domains.json stores all previously stored pages in JSON format. So, we now need to compare with the domains in this file to find newly added ones.

💡 If you are starting from scratch, using the scripts here will pull all the domains because the known_domains.json file will be empty. Next time, when you use the script, you'll only see the new ones added.

Let's write some code to load the URLs from the file and save the new URLs to the file which we can use later on:

def load_domains():
    try:
        with open(DB_FILE) as f:
            return dict(json.load(f))
    except Exception:
        return {}


def save_domains(domains):
    with open(DB_FILE, "w") as f:
        json.dump(domains, f)

Send slack alerts

Let's write a simple function to send slack alerts for any new domains seen for each query. We'll use this later.

def send_slack_alert(domains):

    # Filter out keys with no new values
    domains = {k: v for k, v in domains.items() if v}

    if not domains:
        message = {
            "text": "✅ No new companies detected for any query."
        }
        requests.post(SLACK_WEBHOOK, json=message)
        return
    for query in domains:
        if domains.get(query) != []:
            message = {
                "text": f"🚨 New company detected for '{query}':\n" +
                        "\n".join(domains[query])
            }
            print(message)
            requests.post(SLACK_WEBHOOK, json=message)

Once everything is set up, your Slack channel will receive alerts like this once a day if new domains detected for any query:

🚨 New company detected for 'image creation ai tools':

company1.com

company2.com

Detect new pages

Now let's work on code to specifically detect new pages by looking at already stored ones in known_domains.json.

def check_for_new_domains():
    print("Checking for new domains...")

    known_domains = load_domains()
    current_domains = get_all_domains(QUERY_LIST)
    new_domains = {}

    for query in current_domains:
        new_domains[query] = list(
            set(current_domains[query]) - set(known_domains.get(query, []))
        )

        known_domains[query] = list(
            set(known_domains.get(query, [])) | set(current_domains[query])
        )

    send_slack_alert(new_domains)
    save_domains(known_domains)

This function checks the search results for new domains appearing for each query compared to previously stored domains. It computes the difference between current domains and known domains, sends a Slack alert for any new ones, and then updates the stored list with the latest domains.

Run the bot on a schedule

Now the only thing left is to schedule this so it runs once everyday.

To do this, we'll use Python's schedule library.

# Schedule the bot to run every day at 9 AM
schedule.every().day.at("09:00").do(check_for_new_domains)
print("Query domain monitoring bot running...")

while True:
    schedule.run_pending()
    time.sleep(60)

To keep the tasks running even after you close your terminal or IDE, you can choose to run the script as a persistent background process. For linux/macOS, use nohup or disown to detach the script from the terminal, e.g., python3 script.py & disown.

What the alert looks like

I ran the code twice with QUERY_LIST = ["ai images tool", "ai image generators", "image creation ai tools"]

The first time, I got some new pages (because a few new companies started ranking for the queries since when I ran it last)

The next time, since all the webpages were already seen, I got a message letting me know that no new pages were detected.

These were the alerts I received:

Now your team will instantly know when new companies start ranking for important keywords that matter for your brand.

Possible Enhancements

Once the basic setup works, you can extend it for a couple more use cases:

Track ranking changes : Alert when a domain jumps from page 2 -> page 1
Monitor multiple regions : Use different location parameters
Track different features : Monitor if competitors appear in other parts of SERP like featured snippets, shopping results, and knowledge panels

Conclusion

By combining SerpApi, Slack, and a Python script, you can build a lightweight monitoring system that continuously tracks competitor activity in search.

With this, you’ll have a practical developer tool that saves hours of manual SERP monitoring and helps your team react faster to competitive changes. You can also use this tutorial to setup any other recurring tasks and send notifications to the slack channel.

You can find the entire code here:

GitHub - serpapi/slack-alerts-for-new-company-ranking: In this tutorial, we’ll build a simple system that detects when a new brand ranks for a tracked keyword and send an alert to a Slack channel.

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Build a Slack Bot That Alerts You When Your Competitor Launches a New Page

Sonika Arora — Sat, 27 Jun 2026 07:04:07 +0000

If you're tracking competitors, one of the most valuable signals is when they launch a new page and it shows up in search results. New pages that companies add often reveal:

New product launches
New feature pages
New landing pages targeting keywords
New content marketing strategies

Manually checking for changes can take quite some time, and repetitive work as pages are updated often. Instead of manually checking competitor sites, you can automate this process using:

SerpApi (Google Search API)
Slack Webhook
Python's schedule library

In this tutorial, we’ll build a simple system that checks search results using the SerpApi's Google Search API to detect when a new competitor domain URL is noticed in search results and then sends an alert to a Slack channel.

Setting Up

This system will work in four steps:

Send in a request to SerpApi's Google Search API with a relevant query
Extract all indexed pages
Compare them with previously stored URLs
Send alerts to Slack for new pages detected

We'll run this job every day using the Python schedule library. You can set the schedule as you like.

To start, you’ll need a SerpApi account, a Slack workspace (and a Slack Incoming Webhook URL), and Python installed.

Create a Slack Webhook

Go to Slack → Apps
Search for Incoming Webhooks
Create a webhook for your channel

You will get a Webhook URL like: https://hooks.slack.com/services/XXXX/XXXX/XXXX

Save this, as we'll need it later.

Code Related Setup Steps

For this tutorial, we're going to use SerpApi's new Python library to get results from Google Search.

Install SerpApi's new Python serpapi library, the python-dotenv library, and the schedule library in your environment:

pip install serpapi python-dotenv schedule

serpapi_ _is our new Python library. You can use this library to scrape search results from any of SerpApi's APIs.

More About Our Python Libraries

We have two separate Python libraries serpapi and google-search-results, and both work perfectly fine. We are in the process of deprecating google-search-results, so I recommend using serpapi as that is our latest python library.

You may encounter issues if you have both libraries installed at the same time. If you have the old library installed and want to proceed with using our new library, please follow these steps:

Uninstall google-search-results module from your environment.

Make sure that neither serpapi nor google-search-results are installed at that stage.

Install serpapi module, for example with the following command if you're using pip: pip install serpapi

To begin scraping data, create a free account on serpapi.com. You'll receive 250 free search credits each month to explore the API. Get your SerpApi API Key from this page.
[Optional but Recommended] Set your API key in an environment variable, instead of directly pasting it in the code. Refer here to understand more about using environment variables. For this tutorial, I have saved the API key in an environment variable named "SERPAPI_API_KEY" in my .env file.
[Optional but Recommended] Set up your Slack Webhook URL as an an environment variable as well. In your .env file, this will look like this:

SERPAPI_API_KEY=
SLACK_WEBHOOK_URL=
Import classes needed for this project at the beginning of your code file and set up some constants we'll use later on:

import os
import serpapi
import requests
import json
import schedule
import time
from dotenv import load_dotenv
load_dotenv()

SERPAPI_KEY = os.environ["SERPAPI_API_KEY"]
DOMAIN = "company.com"
DB_FILE = "known_urls.json"
SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK_URL"]

Fetch All Indexed Pages For Competitor

Let's create a script competitor_page_monitor.py where we use SerpApi 's Google Search API to get all pages which show up in search results for a particular company.

def get_all_pages():
   urls = set()
   client = serpapi.Client(api_key=apikey)

   for start in range(0, 100, 10):

       results = client.search({
           'engine': 'google',
           "q": f"site:{DOMAIN} -site:forum.{DOMAIN} -site:{DOMAIN}/blog -site:{DOMAIN}/careers -site:{DOMAIN}/release-notes",
           "start": start,
       })

       if "organic_results" not in results:
           break

       for result in results["organic_results"]:
           urls.add(result["link"])

   return urls

💡 You may need to add more customized search operators to your query if you want to ignore specific types of pages. For this example, I'm using site:company.com -site:forum.company.com -site:company.com/blog -site:company.com/careers -site:company.com/release-notes" to ignore the blog, forum and career pages.

Conversely, you can also choose to target specific areas such as blogs and get notified what exactly your competitors are writing about publicly.

Feel free to change the maximum pagination as you'd like. I've set start to a max value of 100 which means we'll see data from up to 10 pages (since each page usually has 10 results).

Now that we have all the pages for that domain, we need to find the newly added ones since our last check.

💡 If you want to dive deeper into how to scrape Google search results with Python, refer to this post.

Save Known URLs

For the purpose of this example, let's assume the file known_urls.json stores all previously stored pages in JSON format. So, we now need to compare with the URLs in this file to find newly added ones.

💡 If you are starting from scratch, using the scripts here will pull all the URLs for the competitor because the known_urls.json file will be empty. Next time, when you use the script, you'll only see the new ones added.

Let's write some code to load the URLs from the file and save the new URLs to the file which we can use later on:

def load_urls():
    try:
        with open(DB_FILE) as f:
            return set(json.load(f))
    except:
        return set()

def save_urls(urls):
     with open(DB_FILE, "w") as f:
          json.dump(list(urls), f)

Send Slack Alerts

Let's write a simple function to send slack alerts. We'll use this later.

SLACK_WEBHOOK = "YOUR_SLACK_WEBHOOK"

def send_slack_alert(urls):
   if urls:
       message = {
           "text": f"🚨 New competitor pages detected: \n{'\n'.join(urls)}"
       }
   else:
       message = {
           "text": "✅ No new competitor pages detected."
       }
   requests.post(SLACK_WEBHOOK, json=message)

💡 Provide the value of your slack webhook in the SLACK_WEBHOOK variable. It looks something like this: https://hooks.slack.com/services/XXXX/XXXX/XXXX.

Detect New Pages

Now let's work on code to specifically detect new pages by looking at already stored ones in known_urls.json.

def check_new_pages():
    print("Checking for new pages...")

    known_urls = load_urls()
    current_urls = get_all_pages()
    new_pages = current_urls - known_urls

    send_slack_alert(new_pages)

    new_seen_set = known_urls.union(current_urls)
    save_urls(new_seen_set)

Run the Bot on a Schedule

Now the only thing left is to schedule this so it runs once everyday.

To do this, we'll use Python's schedule library.

💡 I'm using Python's schedule library instead of cron because it works reliably on macOS and local environments

schedule.every().day.at("09:00").do(check_new_pages)
print("Competitor monitoring bot running...")

while True:
    schedule.run_pending()
    time.sleep(60)

What The Alert Looks Like

Once this is set up, your Slack channel will receive alerts like this once a day:

🚨 New competitor pages detected:

https://competitor.com/products/ai-assistant

https://competitor.com/features/fast-mode

I ran the code twice for the domain serpapi.com. The first time, I got some new pages and the next time, since all the webpages were already seen, I got a message letting me know that no new pages were detected. These were the alerts I received:

This lets you instantly spot new product launches, new feature releases, and new landing pages which are showing up in search results for any company.

Conclusion

With about 50 lines of Python, you now have a bot that monitors all indexed pages of a competitor, detects new launches, and sends instant Slack alerts.

This is one of the simplest ways to build a competitive intelligence system using search data. You can also use this tutorial to setup any other recurring tasks and send notifications to the slack channel.

You can find the entire code here:

GitHub - serpapi/slack-alerts-for-new-competitor-pages: In this tutorial, we’ll build a simple system that uses SerpApi’s Google Search API to detect when a new competitor domain URL appears in search results and then sends an alert to a Slack channel.

Feel free to reach out to us at contact@serpapi.com for any questions.

Relevant Links

Scrape and Summarize App Reviews from the Apple App Store

Catherine Li — Fri, 26 Jun 2026 07:02:59 +0000

AI coding tools have made building a mobile app more accessible than ever, but getting it into the App Store is only half the battle. The real challenge is finding product-market fit before you invest weeks of work and vibe-coding into the wrong idea.

That usually means hours of manual research: digging through App Store listings, scrolling review after review, trying to spot patterns in what users love, hate, and desperately wish existed. What gaps are worth pursuing? What language are real users actually using to describe their pain points?

Today, I want to show you a way to fetch all the reviews for a mobile app in the Apple App Store with only a few lines of JavaScript code and a couple of API calls to SerpApi. Since it’s JSON data that you can use anywhere, we’ll even pass along the reviews to OpenAI's large language model so we can get a summary of the reviews.

When building AI-powered applications that rely on recent web data as a review aggregator, you need a reliable data scraping solution. SerpApi leads the market in terms of both reliability and accuracy.

Scraping Apple App Store Search with Python

💡 Before we get started, make sure you’ve created a free account on SerpApi so you can get access to 250 free search credits. You may also need to sign up and create a developer account to receive an API key from OpenAI.

For this demo, we're also going to use OpenAI's latest large language model – gpt5.4 – to summarize our reviews.

Setting up your project

First, set up a basic Node.js project. You can do so by running npm init -y in your terminal at the root of your project folder. At the root of the project, create an index.js file.

Install required packages

For this demo, it's highly recommended you install and import into index.js the following packages:

dotenv: Node package that loads environment variables from your .env file directly into process.env so you can access your API keys
serpapi: SerpApi's open-source JavaScript SDK
openai: OpenAI's SDKto interface with their AI model

As per the OpenAI documentation online, you'll also need to create an instance of the OpenAI client to make the API calls:

import OpenAI from "openai";
import dotenv from "dotenv";
import { getJson } from "serpapi"
dotenv.config();

//OpenAI API setup
const openAiKey = process.env.OPEN_AI_KEY;
const client = new OpenAI({ apiKey: openAiKey });

//SerpApi setup
const serpApiKey = process.env.SERP_API_KEY;

Making the API call to SerpApi's Apple App Store API

Now, let’s make a call to SerpApi to get the app information, including its ID. We can just use fetch, but since we’ve imported the SDK, we'll use the getJson from the SerpApi library. And just like fetch, getJson is an asynchronous function that returns a JavaScript Promise. The API has three required parameters: API key , the search term which is the app you want to fetch data for, and the engine which in this case is apple_app_store since app data is what we want to scrape today.

const app = await getJson({
        api_key: serpApiKey,
        engine: "apple_app_store",
        term: "instagram",
        num: "5"
    });

Easily scrape Apple App Store and filter results by categories for better insights

Testing your API calls

If you console.log app, you should see JSON that resembles this in your terminal:

💡 For the sake of simplicity, we are not implementing validation, error handling or other guardrails that would be highly recommended should you launch this in production

Inside the organic_results array is where you'll find the search results for the query "instagram". As expected, the first item in this array pertains to the Instagram app owned by Meta. We simply need to pass in the id value into the second API call, which will be to SerpApi's Apple App Store Reviews API.

const appId = app?.organic_results[0]?.id;
    appReviews = await getJson({
        api_key: serpApiKey,
        engine: "apple_reviews",
        product_id: appId
    });

If you console.log(appId) and scroll down to the reviews array, you'll see the text of the reviews within the array objects' text property. Now, we just need to parse out the text reviews and send them to OpenAI to get the summary.

Making the API call to OpenAI

let response;
try {
    response = await client.responses.create({
        model: "gpt-5.4",
        input: `Given the list of comma-separated reviews for a mobile app, give me a summary of the main criticisms: ${textReviews.join(", ")}`,
    });
} catch (err) {
    console.error("Failed to get response from OpenAI:", err.message);
    process.exit(1);
}

The above example is a basic API call to OpenAI's Responses API to generate text from a prompt, similar to the way you'd generate text from OpenAI's ChatGPT.

If everything went according to plan, you should be able to see the summary in your terminal:

Next Steps

When running this in your local terminal, don't be alarmed if you don't see any response for a few seconds. As you may know, large language models tend to take some time to generate text. While I'm not doing so in this demo, there's a way to stream the response, which will allow the text to appear chunks at a time so your users are not left waiting until the entire response is complete. Make sure to give streaming a shot if you're building an AI-powered app that relies on timely responses. I hope you've found this post helpful! If you have any questions at all, leave me a comment below or contact us at contact@serpapi.com.

Apple App Store Reviews Scraper API - SerpApi