DEV Community: Josef Strzibny

Monitoring FIFA World Cup scores with a simple API

Josef Strzibny — Sun, 12 Jul 2026 14:33:47 +0000

World Cup data constantly changes. Matches kick off, goals arrive, red cards shift momentum, some teams are eliminated, and the bracket moves on. Instead of refreshing the tournament scores page by hand, we can monitor it with simple API calls. The Google Sports API from SerpApi returns live structured data for all kinds of sport events including the World Cup. This way we can learn about games, teams, scores, kickoff times, venues, goal summaries, and more.

Getting live structured World Cup data

We can use live data from the new Google Sports API to answer many questions during the World Cup. We can track individual match scores as well all answer all kinds of other things during the tournament:

How many matches are left in the tournament?
How much time is left until the final?
Which host nations are still alive?
What were the biggest knockout-stage upsets so far?
Which teams advanced despite red cards?
Who scored the most recent brace?
Who are the players on one's team?

This is because each of these questions maps to JSON fields returned by SerpApi's Google Sports API:

Signal	API data we need
Scoreboard status	match `status`, scores, latest finished game, next upcoming game
Matches left	match `status`, round names, games list
Tournament time remaining	`start_time` for upcoming games, plus the bracket tab for the final
Host nations still alive	team names, winners, knockout progression
Knockout-stage upsets	scores, winners, round names
Teams advancing despite red cards	team-level `win` and `red_card` fields
Recent braces	`goal_summary`, scorers, goal counts, match time

Let's go and have a closer look on how to run relevant queries for this kind of data.

Example of the game result from the SerpApi Google Sports API documentation

Using Google Sports API from SerpApi

SerpApi's Google Sports API retrieves structured data directly from Google Sports pages on every request. The only real requirement to use this API is to sign up for free. The API is quite simple with a single endpoint that needs to specify the google_sports engine:

GET https://serpapi.com/search.json?engine=google_sports

The important idea is that Google Sports data is organized around sports entities. In this post we use three of them:

League : the FIFA World Cup as a tournament, useful for schedules, groups, brackets, and many games at once.
Game : a single match, useful when we need more detail, such as goal summaries.
Team : a national team, useful for team-specific games and player lists.

To get the tournament data, we start with the FIFA World Cup league entity:

GET https://serpapi.com/search.json
  ?engine=google_sports
  &kgmid=/m/030q7
  &sp=ft
  &type=league
  &tab=gm
  &hl=en
  &gl=us
  &api_key=YOUR_API_KEY

Here are the parameters we will use most often:

Parameter	Meaning
`engine=google_sports`	Use SerpApi's Google Sports API
`kgmid=/m/030q7`	Google Knowledge Graph ID for the FIFA World Cup
`sp=ft`	Football. The API also supports other sports, such as basketball (`bs`), baseball (`bb`), cricket (`cr`), American football (`af`), ice hockey (`ih`), and rugby (`rg`)
`type=league`	Query a league or tournament entity
`tab=gm`	Return the games tab
`tab=br`	Return the bracket tab, useful for the full knockout tree
`type=game`	Query a specific match by its game `kgmid`
`type=team`	Query a specific team by its team `kgmid`
`hl=en`	Return English results
`gl=us`	Use United States location context

A shortened league response for an upcoming match looks like this:

{
  "league_results": {
    "game_groups": [
      {
        "title": "Quarter-finals",
        "games": [
          {
            "status": "upcoming",
            "start_time": "2026-07-09T20:00:00Z",
            "teams": [
              {
                "short_name": "France",
                "short_code": "FRA"
              },
              {
                "short_name": "Morocco",
                "short_code": "MAR"
              }
            ],
            "venue": {
              "name": "Boston Stadium",
              "location": "Foxborough"
            },
            "extensions": ["Quarter-finals"]
          }
        ]
      }
    ]
  }
}

A finished match includes scores and a winning team marker:

{
  "status": "finished",
  "status_original": "FT",
  "start_time": "2026-07-07T00:00:00Z",
  "teams": [
    {
      "name": "United States men's national soccer team",
      "short_code": "USA",
      "score": 1
    },
    {
      "short_name": "Belgium",
      "short_code": "BEL",
      "score": 4,
      "win": true
    }
  ],
  "extensions": ["Round of 16"]
}

💡 You can experiment with these parameters interactively in the SerpApi playground.

SerpApi provides official clients for many languages including JavaScript, Python, Ruby, and PHP that makes integration easy. We'll use the JavaScript package with Node.js but feel free to use your favourite language.

Setting up a simple Node.js monitor

Every request for the google_sports engine follows the same pattern. We choose the sports entity type, pass its kgmid, and set the sport with sp. For this post on current World Cup, the defaults are:

{
  engine: "google_sports",
  kgmid: "/m/030q7",
  sp: "ft",
  type: "league",
  tab: "gm",
  hl: "en",
  gl: "us"
}

This means the following:

use Google Sports
fetch the FIFA World Cup entity
treat it as football
query it as a league
and return the games tab

Using SerpApi's official JavaScript package, a minimal request looks like this:

const { getJson } = require("serpapi");

const data = await getJson({
  engine: "google_sports",
  kgmid: "/m/030q7",
  sp: "ft",
  type: "league",
  tab: "gm",
  hl: "en",
  gl: "us",
  api_key: process.env.SERPAPI_API_KEY
});

If you prefer not to use the package, the same request can be made as a regular GET request to https://serpapi.com/search.json.

The most important parameter to understand is type:

// Tournament-level data: games, bracket, standings, etc.
await getJson({
  engine: "google_sports",
  kgmid: "/m/030q7",
  sp: "ft",
  type: "league",
  tab: "gm",
  api_key: process.env.SERPAPI_API_KEY
});

// Single-match details, using a game kgmid from a previous response.
await getJson({
  engine: "google_sports",
  kgmid: "/g/11xmtfjhq2",
  sp: "ft",
  type: "game",
  api_key: process.env.SERPAPI_API_KEY
});

// Team-level data, using a team kgmid.
await getJson({
  engine: "google_sports",
  kgmid: "/m/0329gm",
  sp: "ft",
  type: "team",
  tab: "pl",
  api_key: process.env.SERPAPI_API_KEY
});

The tab parameter depends on the entity type:

Entity type	Useful tabs	What we use it for
`league`	`gm`, `br`, `sn`	Games, bracket, standings
`team`	`gm`, `pl`, `sn`	Team games, players, standings
`game`	usually no tab for football	Match details such as teams, score, goals, and metadata

For example, the World Cup games tab returns grouped games under league_results.game_groups. To work with those games, we usually flatten the groups first:

function getGames(data) {
  const groups = data.league_results?.game_groups || [];

  return groups.flatMap((group) =>
    (group.games || []).map((game) => ({
      ...game,
      group_title: group.title
    }))
  );
}

From there, each game object has the fields we need for most monitoring tasks: status, start_time, teams, venue, extensions, and often kgmid for fetching more detailed game data later.

One small cleanup helper is also useful. Some teams include short_name, while others only include a longer official name, such as "United States men's national soccer team". For display output, we can normalize that:

function teamName(team) {
  const name = team.short_name || team.name || team.short_code || "TBD";

  return name.replace(/\s+(?:men's\s+)?national\s+(?:football|soccer)\s+team$/i, "");
}

With those pieces in place, we can start building monitors answering questions we want from the JSON response.

Monitoring FIFA World Cup 2026

We'll use SerpApi's new Google Sports API to build a few small Node.js scripts for answering practical, real-time questions like What are the current scores?, How many matches are left?, Which host nations are still alive?, Who scored a brace?, and Which knockout results stand out?.

💡 The outputs shown below come from running the scripts against the live API while writing this post, so the exact numbers will naturally change as the tournament progresses.

What's the current scoreboard?

The most important thing we need to start with is the scoreboard. The following code will give you a quick summary for the World Cup and list of some latest finished games. Here's a full code that builds on top of the API call we discussed in the previous section but also filters through the relevant information:

#!/usr/bin/env node

const { getJson } = require("serpapi");

const WORLD_CUP_KGMID = "/m/030q7";

function getApiKey() {
  const apiKey = process.env.SERPAPI_API_KEY;

  if (!apiKey) {
    throw new Error("Missing SERPAPI_API_KEY. Set it with: export SERPAPI_API_KEY=your_key");
  }

  return apiKey;
}

async function serpapiSearch(params = {}) {
  const searchParams = Object.fromEntries(
    Object.entries({
      engine: "google_sports",
      kgmid: WORLD_CUP_KGMID,
      sp: "ft",
      type: "league",
      tab: "gm",
      hl: "en",
      gl: "us",
      api_key: getApiKey(),
      ...params
    }).filter(([, value]) => value !== undefined && value !== null)
  );

  const data = await getJson(searchParams);

  if (data.error) {
    throw new Error(`SerpApi error: ${data.error}`);
  }

  return data;
}

function getGames(data) {
  const groups = data.league_results?.game_groups || [];

  return groups.flatMap((group) =>
    (group.games || []).map((game) => ({
      ...game,
      group_title: group.title
    }))
  );
}

function teamName(team) {
  const name = team.short_name || team.name || team.short_code || "TBD";

  return name.replace(/\s+(?:men's\s+)?national\s+(?:football|soccer)\s+team$/i, "");
}

function scoreLine(game) {
  const teams = game.teams || [];

  return teams
    .map((team) => {
      const score = team.score_original ?? team.score;
      return score === undefined ? teamName(team) : `${teamName(team)} ${score}`;
    })
    .join(" - ");
}

function roundLabel(game) {
  const extension = (game.extensions || []).find((item) =>
    /Group Stage|Round of 32|Round of 16|Quarter-finals|Semi-finals|Third place play-off|Final/.test(item)
  );

  return extension || game.group_title || "Unknown round";
}

function formatUtc(isoString) {
  if (!isoString) return "unknown time";

  return `${isoString.replace("T", " ").replace("Z", "")} UTC`;
}

function summarizeStatuses(games) {
  const counts = new Map();

  for (const game of games) {
    counts.set(game.status, (counts.get(game.status) || 0) + 1);
  }

  return counts;
}

function printGames(title, games) {
  console.log(`\n${title}`);
  console.log("-".repeat(title.length));

  if (games.length === 0) {
    console.log("No games found.");
    return;
  }

  for (const game of games) {
    console.log(`- ${roundLabel(game)} — ${scoreLine(game)} — ${game.status} — ${formatUtc(game.start_time)}`);
  }
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data).sort(
    (a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0)
  );

  const finished = games.filter((game) => game.status === "finished");
  const liveOrOther = games.filter((game) => !["finished", "upcoming"].includes(game.status));
  console.log("World Cup score monitor");
  console.log(`Fetched ${games.length} games from SerpApi Google Sports API.`);

  console.log("\nStatus summary");
  console.log("--------------");
  for (const [status, count] of summarizeStatuses(games)) {
    console.log(`- ${status}: ${count}`);
  }

  printGames("Live or in-progress matches", liveOrOther);
  printGames("Latest finished scores", finished.slice(-8));
}

main().catch((error) => {
  console.error(error.message);
  process.exit(1);
});

It looks long but most of the code are just helpers for formatting the information.

To run it, saved it as a file, and type:

node score-monitor.js

You should get a similar results back, giving you a quick glance on the whole live tournament:

World Cup score monitor
Fetched 40 games from SerpApi Google Sports API.

Status summary
--------------
- finished: 32
- upcoming: 8

Live or in-progress matches
---------------------------
No games found.

Latest finished scores
----------------------
- Round of 16 — Canada 0 - Morocco 3 — finished — 2026-07-04 17:00:00 UTC
- Round of 16 — Paraguay 0 - France 1 — finished — 2026-07-04 21:00:00 UTC
- Round of 16 — Brazil 1 - Norway 2 — finished — 2026-07-05 20:00:00 UTC
- Round of 16 — Mexico 2 - England 3 — finished — 2026-07-06 01:00:00 UTC
- Round of 16 — Portugal 0 - Spain 1 — finished — 2026-07-06 19:00:00 UTC
- Round of 16 — United States 1 - Belgium 4 — finished — 2026-07-07 00:00:00 UTC
- Round of 16 — Argentina 3 - Egypt 2 — finished — 2026-07-07 16:00:00 UTC
- Round of 16 — Switzerland 0 (4) - Colombia 0 (3) — finished — 2026-07-07 20:00:00 UTC

This way we can build nice dashboards, especially if we would later format the data into HTML.

How many matches are left?

We already printed how many games are left, but we don't know which matches will take place. For that we can define a few additional functions:

const KNOCKOUT_ROUND_PATTERN = /Round of 32|Round of 16|Quarter-finals|Semi-finals|Third place play-off|Final/;

function getBracketGames(data) {
  const brackets = data.league_results?.brackets || [];

  return brackets.flatMap((bracket) =>
    (bracket.stages || []).flatMap((stage) =>
      (stage.games || []).map((game) => ({
        ...game,
        group_title: stage.title
      }))
    )
  );
}

function matchup(game) {
  const teams = game.teams || [];

  if (teams.length === 0) return "teams to be determined";

  return teams.map(teamName).join(" vs ");
}

function roundLabel(game) {
  const extension = (game.extensions || []).find((item) => KNOCKOUT_ROUND_PATTERN.test(item));

  return extension || game.group_title || "Unknown round";
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  let upcomingGames = games
    .filter((game) => game.status === "upcoming")
    .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));

  // The games tab returns a rolling window of relevant games, which may not
  // reach the Final. The bracket tab always covers the full knockout tree.
  let usedBracket = false;

  if (!upcomingGames.some((game) => roundLabel(game) === "Final")) {
    const bracketData = await serpapiSearch({ tab: "br" });
    const knownKgmids = new Set(upcomingGames.map((game) => game.kgmid));
    const missingGames = getBracketGames(bracketData).filter(
      (game) => game.status === "upcoming" && !knownKgmids.has(game.kgmid)
    );

    if (missingGames.length > 0) {
      usedBracket = true;
      upcomingGames = upcomingGames
        .concat(missingGames)
        .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));
    }
  }

  const breakdown = new Map();

  for (const game of upcomingGames) {
    const round = roundLabel(game);
    breakdown.set(round, (breakdown.get(round) || 0) + 1);
  }

  console.log(`Matches left: ${upcomingGames.length}`);

  if (usedBracket) {
    console.log("(games tab window did not include the Final; completed from the bracket tab)");
  }

  console.log("\nBreakdown by round:");

  for (const [round, count] of breakdown) {
    console.log(`- ${round}: ${count}`);
  }

  console.log("\nUpcoming matches:");

  for (const game of upcomingGames) {
    console.log(`- ${formatUtc(game.start_time)} — ${roundLabel(game)} — ${matchup(game)}`);
  }

  const thirdPlaceGames = upcomingGames.filter((game) => roundLabel(game) === "Third place play-off");
  const championPathCount = upcomingGames.length - thirdPlaceGames.length;

  console.log(`\nMatches on the path to crowning the champion: ${championPathCount}`);
}

There is one practical detail to handle. The games tab returns a rolling window of currently relevant games. In this run the window covered everything up to the Final, but a day earlier it ended at the third-place play-off. The bracket tab (tab=br) always covers the full knockout tree, so the script completes the list from there whenever the Final is missing:

if (!upcomingGames.some((game) => roundLabel(game) === "Final")) {
  const bracketData = await serpapiSearch({ tab: "br" });
  const knownKgmids = new Set(upcomingGames.map((game) => game.kgmid));

  const missingGames = getBracketGames(bracketData).filter(
    (game) => game.status === "upcoming" && !knownKgmids.has(game.kgmid)
  );

  upcomingGames = upcomingGames.concat(missingGames);
}

Save the new code and run it:

node matches-left.js

Note that I haven't repeated the same functions from before. You can, however, run the full script from GitHub.

And here's the output:

Matches left: 8

Breakdown by round:
- Quarter-finals: 4
- Semi-finals: 2
- Third place play-off: 1
- Final: 1

Upcoming matches:
- 2026-07-09 20:00:00 UTC — Quarter-finals — France vs Morocco
- 2026-07-10 19:00:00 UTC — Quarter-finals — Spain vs Belgium
- 2026-07-11 21:00:00 UTC — Quarter-finals — Norway vs England
- 2026-07-12 01:00:00 UTC — Quarter-finals — Argentina vs Switzerland
- 2026-07-14 19:00:00 UTC — Semi-finals — teams to be determined
- 2026-07-15 19:00:00 UTC — Semi-finals — teams to be determined
- 2026-07-18 21:00:00 UTC — Third place play-off — teams to be determined
- 2026-07-19 19:00:00 UTC — Final — teams to be determined

Matches on the path to crowning the champion: 7

How much time is left in the tournament?

If we are interested how far are we in the tournament in terms the total time left, we can define a function for the duration and try to calculate it:

function pluralize(count, singular, plural = `${singular}s`) {
  return `${count} ${count === 1 ? singular : plural}`;
}

function formatDuration(milliseconds) {
  const totalHours = Math.floor(milliseconds / 1000 / 60 / 60);
  const days = Math.floor(totalHours / 24);
  const hours = totalHours % 24;

  return `${pluralize(days, "day")} and ${pluralize(hours, "hour")}`;
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  const upcomingGames = games
    .filter((game) => game.status === "upcoming" && game.start_time)
    .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));

  if (upcomingGames.length === 0) {
    console.log("No upcoming matches found.");
    return;
  }

  const nextMatch = upcomingGames[0];
  let final = upcomingGames.find((game) => roundLabel(game) === "Final");

  // Fall back to the bracket tab when the games window ends before the Final.
  if (!final) {
    const bracketData = await serpapiSearch({ tab: "br" });
    final = getBracketGames(bracketData).find(
      (game) => roundLabel(game) === "Final" && game.status === "upcoming" && game.start_time
    );
  }

  console.log(`Next match: ${matchup(nextMatch)}`);
  console.log(`Next kickoff: ${formatUtc(nextMatch.start_time)}`);

  if (final) {
    const untilFinal = new Date(final.start_time) - Date.now();
    const span = new Date(final.start_time) - new Date(nextMatch.start_time);

    console.log(`Final kickoff: ${formatUtc(final.start_time)}`);
    console.log(`Time left until the final kicks off: ${formatDuration(untilFinal)}`);
    console.log(`Calendar span from next kickoff to final kickoff: ${formatDuration(span)}`);
  } else {
    console.log("Final kickoff: not found in the games or bracket responses");
  }

  const regulationMinutes = upcomingGames.length * 90;
  const withHalftimeMinutes = regulationMinutes + upcomingGames.length * 15;

  console.log("\nEstimated watching time for the remaining matches:");
  console.log(`- Regulation only: ${regulationMinutes} minutes (${(regulationMinutes / 60).toFixed(1)} hours)`);
  console.log(`- With halftime: ${withHalftimeMinutes} minutes (${(withHalftimeMinutes / 60).toFixed(1)} hours)`);
  console.log("- With stoppage time, extra time, and penalties: add more time depending on match events");
}

The same response includes start_time for upcoming games. We sort upcoming matches by kickoff time and look for the Final. We fall back to the bracket tab if the games window ends before it:

const upcomingGames = games
  .filter((game) => game.status === "upcoming" && game.start_time)
  .sort((a, b) => new Date(a.start_time) - new Date(b.start_time));

const nextMatch = upcomingGames[0];
let final = upcomingGames.find((game) => roundLabel(game) === "Final");

if (!final) {
  const bracketData = await serpapiSearch({ tab: "br" });
  final = getBracketGames(bracketData).find(
    (game) => roundLabel(game) === "Final" && game.status === "upcoming"
  );
}

To run it, save it as a file, and type:

node time-left.js

And here's the output:

Next match: France vs Morocco
Next kickoff: 2026-07-09 20:00:00 UTC
Final kickoff: 2026-07-19 19:00:00 UTC
Time left until the final kicks off: 10 days and 9 hours
Calendar span from next kickoff to final kickoff: 9 days and 23 hours

Estimated watching time for the remaining matches:
- Regulation only: 720 minutes (12.0 hours)
- With halftime: 840 minutes (14.0 hours)
- With stoppage time, extra time, and penalties: add more time depending on match events

So the tournament wraps up with the Final at New York New Jersey Stadium on July 19, and a fan committed to every remaining match is looking at roughly 14 hours of football.

Which host nations are still alive?

It's often said that host nations have the advantage of local fans. If we want to track the results of host nation teams we could filter them based on their codes like in this following example:

const HOST_CODES = ["CAN", "MEX", "USA"];

function scoreLine(game) {
  const teams = game.teams || [];

  return teams
    .map((team) => {
      const score = team.score_original ?? team.score;
      return score === undefined ? teamName(team) : `${teamName(team)} ${score}`;
    })
    .join(" - ");
}

function roundLabel(game) {
  const extension = (game.extensions || []).find((item) => KNOCKOUT_ROUND_PATTERN.test(item));

  return extension || game.group_title || "Unknown round";
}

function isKnockout(game) {
  return KNOCKOUT_ROUND_PATTERN.test(
    `${game.group_title || ""} ${(game.extensions || []).join(" ")}`
  );
}

function hostStatus(games, hostCode) {
  const hostGames = games
    .filter((game) => (game.teams || []).some((team) => team.short_code === hostCode))
    .sort((a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0));

  if (hostGames.length === 0) {
    return "not present in the current games window";
  }

  for (const game of hostGames) {
    if (game.status !== "finished" || !isKnockout(game)) continue;

    const host = (game.teams || []).find((team) => team.short_code === hostCode);
    const winner = (game.teams || []).find((team) => team.win);

    if (winner && host && !host.win) {
      return `eliminated by ${teamName(winner)} — ${scoreLine(game)} (${roundLabel(game)})`;
    }
  }

  const nextGame = hostGames.find((game) => game.status === "upcoming");

  if (nextGame) {
    return `still alive — next match: ${roundLabel(nextGame)}`;
  }

  return "still alive";
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);

  console.log("Host nation status:\n");

  for (const hostCode of HOST_CODES) {
    const hostGame = games.find((game) =>
      (game.teams || []).some((team) => team.short_code === hostCode)
    );
    const hostTeam = (hostGame?.teams || []).find((team) => team.short_code === hostCode);
    const name = hostTeam ? teamName(hostTeam) : hostCode;

    console.log(`- ${name}: ${hostStatus(games, hostCode)}`);
  }
}

For this tournament, the host nations are:

Canada
Mexico
United States

To monitor their status, we can inspect knockout-stage games involving those teams and check whether they won or were eliminated.

const hostCodes = new Set(["CAN", "MEX", "USA"]);

const hostKnockoutGames = games.filter((game) => {
  const hasHost = (game.teams || []).some((team) => hostCodes.has(team.short_code));

  return isKnockout(game) && hasHost;
});

The isKnockout helper matches the round name against the knockout rounds (Round of 32 through Final) found in each game's extensions.

Again, I have shorten it to not repeat previous functions. Save and run as:

node host-nations.js

And here's the output:

Host nation status:

- Canada: eliminated by Morocco — Canada 0 - Morocco 3 (Round of 16)
- Mexico: eliminated by England — Mexico 2 - England 3 (Round of 16)
- United States: eliminated by Belgium — United States 1 - Belgium 4 (Round of 16)

So the answer for the current 2026 World Cup is straightforward:

None. All three host nations were eliminated before the quarter-finals.

What were the biggest knockout-stage upsets so far?

The next question is a fun one. We want to find out when a heavily favored team is defeated by a significantly weaker opponent (the "underdog").

// Editorial choice: traditional favorites losing to anyone outside this list
// counts as a possible upset.
const FAVORITES = new Set(["BRA", "ARG", "FRA", "ESP", "ENG", "GER", "POR"]);

(...)

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  const knockoutGames = games.filter(
    (game) => isKnockout(game) && game.status === "finished"
  );

  const possibleUpsets = knockoutGames.filter((game) => {
    const teams = game.teams || [];
    const winner = teams.find((team) => team.win);
    const loser = teams.find((team) => !team.win);

    return (
      winner &&
      loser &&
      FAVORITES.has(loser.short_code) &&
      !FAVORITES.has(winner.short_code)
    );
  });

  console.log("Possible knockout-stage upsets:\n");

  if (possibleUpsets.length === 0) {
    console.log("No upsets found by this definition.");
    return;
  }

  for (const game of possibleUpsets) {
    const winner = (game.teams || []).find((team) => team.win);
    const loser = (game.teams || []).find((team) => !team.win);

    console.log(
      `- ${teamName(winner)} beat ${teamName(loser)} — ${scoreLine(game)} (${roundLabel(game)})`
    );
  }
}

This monitor is a little more interpretive. We then add our own definition of what counts as an upset. One simple approach to find them is to keep a list of traditional favorite teams and see where one of them loses to a team outside that list:

const favorites = new Set(["BRA", "ARG", "FRA", "ESP", "ENG", "GER", "POR"]);

const possibleUpsets = knockoutGames.filter((game) => {
  const teams = game.teams || [];
  const winner = teams.find((team) => team.win);
  const loser = teams.find((team) => !team.win);

  return winner && loser && favorites.has(loser.short_code) && !favorites.has(winner.short_code);
});

We can again save this and run as:

node upsets.js

And here's a possible output:

Possible knockout-stage upsets:

- Paraguay beat Germany — Germany 1 (3) - Paraguay 1 (4) (Round of 32)
- Norway beat Brazil — Brazil 1 - Norway 2 (Round of 16)

Finding out these upsets might be a little bit personal, but hopefully you don't mind the logic here.

Which teams ended up with red cards?

Red cards mean the opposite of fair play. If you want to know which teams ended up with a red card during their games, you can ask on team.read_card:

function hasRedCard(game) {
  return (game.teams || []).some((team) => team.red_card);
}

function gameKey(game) {
  return game.kgmid || `${game.start_time || "unknown"}|${matchup(game)}|${roundLabel(game)}`;
}

function mergeGames(games) {
  const gamesByKey = new Map();

  for (const game of games) {
    const key = gameKey(game);
    const existing = gamesByKey.get(key);

    if (!existing || (!hasRedCard(existing) && hasRedCard(game))) {
      gamesByKey.set(key, game);
    }
  }

  return [...gamesByKey.values()].sort(
    (a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0)
  );
}

async function main() {
  const [gamesData, bracketData] = await Promise.all([
    serpapiSearch({ tab: "gm" }),
    serpapiSearch({ tab: "br" })
  ]);
  const games = mergeGames([
    ...getGames(gamesData),
    ...getBracketGames(bracketData)
  ]);

  const redCards = games.flatMap((game) =>
    (game.teams || [])
      .filter((team) => team.red_card)
      .map((team) => ({ team, game }))
  );

  console.log("Red cards in the tournament:\n");

  if (redCards.length === 0) {
    console.log("No red cards found in the returned tournament data.");
    return;
  }

  for (const { team, game } of redCards) {
    console.log(
      `- ${teamName(team)} — ${roundLabel(game)} — ${scoreLine(game)} — ${matchup(game)} — ${formatUtc(game.start_time)}`
    );
  }

  console.log(`\nTotal red-card team entries: ${redCards.length}`);
}

Save and run as:

node red-cards.js

And here's the output:

Red cards in the tournament:

- Ecuador — Round of 32 — Mexico 2 - Ecuador 0 — Mexico vs Ecuador — 2026-07-01 02:00:00 UTC
- United States — Round of 32 — United States 2 - Bosnia and Herzegovina 0 — United States vs Bosnia and Herzegovina — 2026-07-02 00:00:00 UTC
- England — Round of 16 — Mexico 2 - England 3 — Mexico vs England — 2026-07-06 01:00:00 UTC

Total red-card team entries: 3

Looks like there were 3 red cards so far.

Who scored the most recent brace?

You might know hat-tricks. They are amazing but not that common. There is also a term that sits below it. Braces are where a single player scores two goals in a single match. Using the structured data from the Google Sports API, we can try to find the last player who successfully scored a brace:

async function fetchGameDetails(game) {
  if (!game.kgmid) return game;

  const data = await serpapiSearch({
    kgmid: game.kgmid,
    type: "game",
    tab: undefined
  });

  return {
    ...game,
    ...(data.game_results?.info || {})
  };
}

function formatGoalMinute(goal) {
  const time = goal.in_game_time || {};
  const minute = time.minute;
  const stoppage = time.stoppage_minute;

  if (minute === undefined) return "unknown minute";
  if (stoppage !== undefined) return `${minute}+${stoppage}'`;

  return `${minute}'`;
}

function findBracesInGame(game) {
  const braces = [];

  for (const team of game.teams || []) {
    for (const summary of team.goal_summary || []) {
      const goals = summary.goals || [];

      if (goals.length >= 2) {
        const opponents = (game.teams || []).filter((otherTeam) => otherTeam !== team);

        braces.push({
          player: summary.player?.name || summary.player?.short_name || "Unknown player",
          team,
          goals,
          opponents,
          game
        });
      }
    }
  }

  return braces;
}

async function fetchGameDetails(game) {
  if (!game.kgmid) return game;

  const data = await serpapiSearch({
    kgmid: game.kgmid,
    type: "game",
    tab: undefined
  });

  return {
    ...game,
    ...(data.game_results?.info || {})
  };
}

async function main() {
  const data = await serpapiSearch({ tab: "gm" });
  const games = getGames(data);
  const finishedGames = games
    .filter((item) => item.status === "finished")
    .sort((a, b) => new Date(b.end_time || b.start_time || 0) - new Date(a.end_time || a.start_time || 0));

  let mostRecentBrace;

  for (const game of finishedGames) {
    const detailedGame = findBracesInGame(game).length > 0 ? game : await fetchGameDetails(game);
    const braces = findBracesInGame(detailedGame);

    if (braces.length > 0) {
      mostRecentBrace = braces[0];
      break;
    }
  }

  if (!mostRecentBrace) {
    console.log("No braces found in the returned API data.");
    return;
  }

  const { player, team, goals, opponents, game } = mostRecentBrace;
  const goalMinutes = goals.map(formatGoalMinute).join(", ");

  console.log("Most recent brace found:\n");
  console.log(`Player: ${player}`);
  console.log(`Team: ${teamName(team)}`);
  console.log(`Opponent: ${opponents.map(teamName).join(", ")}`);
  console.log(`Match: ${matchup(game)}`);
  console.log(`Score: ${scoreLine(game)}`);
  console.log(`Round: ${roundLabel(game)}`);
  console.log(`Match time: ${formatUtc(game.start_time)}`);
  console.log(`Goal minutes: ${goalMinutes}`);
}

For this signal, we need game-level goal data. The league games response usually does not include goal_summary, so the script fetches game details for recent finished games via the game entity type. Passing tab: undefined removes the league-level tab=gm default from the shared helper, since game entities have no tabs:

const data = await serpapiSearch({
  kgmid: game.kgmid,
  type: "game",
  tab: undefined
});

const detailedGame = {
  ...game,
  ...(data.game_results?.info || {})
};

The detailed response includes a goal_summary per team:

{
  "player": {
    "name": "Charles De Ketelaere",
    "position": "Striker",
    "jersey_number": "17"
  },
  "goals": [
    { "in_game_time": { "minute": 9 } },
    { "in_game_time": { "minute": 33 } }
  ]
}

The function findBracesInGame is currently looking for 2 or more goals, so it will also find hat-tricks (3x), hauls (4x), and double hat-tricks (5x).

Save and run as:

node most-recent-brace.js

And see the output:

Most recent brace found:

Player: Charles De Ketelaere
Team: Belgium
Opponent: United States
Match: United States vs Belgium
Score: United States 1 - Belgium 4
Round: Round of 16
Match time: 2026-07-07 00:00:00 UTC
Goal minutes: 9', 33'

We found Charles De Ketelaere from Belgium who scored twice in the game against the United States.

What is the team journey in the tournament?

Finally, let's find out more about the teams themselves. The SerpApi's new Google Sports API supports type=team, so we can fetch team-specific data with the team's kgmid as well. There are plenty of things we might be interested in. Let's look at World Cup games in the current league, the last match, and the team structure:

const TEAM = {
  code: process.env.TEAM_CODE || "BEL",
  kgmid: process.env.TEAM_KGMID || "/m/0329gm",
  fallbackName: process.env.TEAM_NAME || "Belgium"
};

async function main() {
  const [leagueData, playersData] = await Promise.all([
    serpapiSearch({ tab: "gm" }),
    serpapiSearch({ kgmid: TEAM.kgmid, type: "team", tab: "pl" })
  ]);

  const games = getGames(leagueData);
  const teamGames = games
    .filter((game) => findTeamInGame(game))
    .sort((a, b) => new Date(a.start_time || 0) - new Date(b.start_time || 0));

  const players = playersData.team_results?.players || [];
  const inferredTeam = teamGames.map(findTeamInGame).find(Boolean);
  const displayName = inferredTeam ? teamName(inferredTeam) : TEAM.fallbackName;
  const nextGame = teamGames.find((game) => game.status === "upcoming");
  const latestFinished = [...teamGames].reverse().find((game) => game.status === "finished");

  console.log(`Team monitor: ${displayName}`);
  console.log(`Team code: ${TEAM.code}`);
  console.log(`Team kgmid: ${TEAM.kgmid}`);

  console.log("\nWorld Cup games in the current league response:");
  if (teamGames.length === 0) {
    console.log("- No games found for this team in the current league response.");
  } else {
    for (const game of teamGames) {
      const label = game.status === "upcoming" ? matchup(game) : scoreLine(game);
      console.log(`- ${roundLabel(game)} — ${label} — ${game.status} — ${formatUtc(game.start_time)}`);
    }
  }

  if (latestFinished) {
    console.log(`\nLatest finished World Cup match: ${scoreLine(latestFinished)} (${roundLabel(latestFinished)})`);
  }

  if (nextGame) {
    console.log(`Next World Cup match: ${matchup(nextGame)} (${roundLabel(nextGame)}) at ${formatUtc(nextGame.start_time)}`);
  }

  console.log("\nTeam players from Google Sports team endpoint:");
  console.log(`Players returned: ${players.length}`);

  for (const [position, count] of positionBreakdown(players)) {
    console.log(`- ${position}: ${count}`);
  }

  console.log("\nSample players:");
  for (const player of players.slice(0, 10)) {
    console.log(`- ${player.name}${player.position ? ` — ${player.position}` : ""}`);
  }
}

The example defaults to Belgium because Belgium is still alive in the current response (at the time of the writing) and even has a recent match with detailed goal data:

const playersData = await serpapiSearch({
  kgmid: "/m/0329gm",
  type: "team",
  tab: "pl"
});

const players = playersData.team_results?.players || [];

The script also reuses the World Cup league response to list Belgium's World Cup games in the current tournament window.

Save and run as usual:

node team-info.js

And here's the output for the provided Belgian team:

Team monitor: Belgium
Team code: BEL
Team kgmid: /m/0329gm

World Cup games in the current league response:
- Group Stage · Fri, Jun 26 — New Zealand 1 - Belgium 5 — finished — 2026-06-27 03:00:00 UTC
- Round of 32 — Belgium 3 - Senegal 2 — finished — 2026-07-01 20:00:00 UTC
- Round of 16 — United States 1 - Belgium 4 — finished — 2026-07-07 00:00:00 UTC
- Quarter-finals — Spain vs Belgium — upcoming — 2026-07-10 19:00:00 UTC

Latest finished World Cup match: United States 1 - Belgium 4 (Round of 16)
Next World Cup match: Spain vs Belgium (Quarter-finals) at 2026-07-10 19:00:00 UTC

Team players from Google Sports team endpoint:
Players returned: 26
- Defender: 9
- Forward: 7
- Goalkeeper: 3
- Midfielder: 7

Sample players:
- Jérémy Doku — Forward
- Kevin De Bruyne — Midfielder
- Romelu Lukaku — Forward
- Youri Tielemans — Midfielder
- Thibaut Courtois — Goalkeeper
- Senne Lammens — Goalkeeper
- Leandro Trossard — Forward
- Charles De Ketelaere — Forward
- Axel Witsel — Midfielder
- Amadou Onana — Midfielder

This is a useful complement to score monitoring. Once a team appears in the next fixture or a standout result, we can fetch its player list and recent team context with the same API. You can of course point the same script at another team by passing a different team code and kgmid.

Google Sports API examples

You can find all these World Cup API examples on our GitHub page. Checkout the repository and provide a SerpApi API key to run them individually with Node:

export SERPAPI_API_KEY=your_serpapi_api_key

node google-sports-api-world-cup/00-score-monitor.js
node google-sports-api-world-cup/01-matches-left.js
node google-sports-api-world-cup/02-time-left.js
node google-sports-api-world-cup/03-host-nations.js
node google-sports-api-world-cup/04-upsets.js
node google-sports-api-world-cup/05-red-cards.js
node google-sports-api-world-cup/06-most-recent-brace.js
node google-sports-api-world-cup/07-team-info.js

If you want SerpApi to fetch fresh results instead of using a cached response, set:

export SERPAPI_NO_CACHE=true

Wrapping up

SerpApi's Google Sports API is useful not only for keeping an eye on World Cup scores, but also for monitoring other parts of the tournament. With a few small scripts fetching the structured data live, we can track match status, remaining games, host nation eliminations, red cards, and more.

All of that makes the API a good fit for live dashboards, alerts, editorial tools, sports newsletters, and lightweight automation around major tournaments. See the Google Sports API documentation for yourself or try your own queries in the playground that lets you see the results before any implementation.

A complete guide to web scraping with Ruby (2026)

Josef Strzibny — Fri, 10 Jul 2026 14:33:26 +0000

Web scraping is the process of pulling out structured data from websites. If you're familiar with Ruby, you're in luck_._ Ruby has excellent tools you can use to scrape the web. From HTTP libraries to fast HTML parsers, Ruby tools cover all concerns of scraping. Paired with its developer-friendly syntax and productive frameworks, it's a great option for small scripts as well as complex applications.

Ruby as a language for web scraping

Ruby is a practical language for web scraping because it is good at the unglamorous parts of it all. The small scripts, readable objects, HTTP requests, background jobs, parsing JSON, databases, writing CSV files and glueing everything together. Some other languages might be more common for web scraping, but Ruby is an amazing choice too.

If you already work with Sinatra, Hanami, Ruby on Rails, Sidekiq, Active Job, or plain Ruby scripts, staying with Ruby for your scraping needs makes the most sense. You can scrape a page, parse the data, store it in your database, schedule a job, and expose the result in your web application without crossing language boundaries or adding a separate data pipeline just for web scraping.

Ruby is designed as a programmer's best friend

What will we learn

When it comes to web scraping, it's important to choose the right approach for the page in front of you. A static HTML page, a JavaScript-rendered interface, a form with cookies, a JSON endpoint, and a full website crawl are all different problems. Treating them as the same problem is where most scraping projects get messy.

This guide will take you through it all. We will map the main ways to scrape websites with Ruby and choose the right tools for each job. Then we will build up from HTML scraping to JavaScript-rendered pages, browser automation, crawling, JSON APIs, and data export. Finally, we touch production concerns like reliability and performance.

💡 Don't use Ruby? Have a look at our guides for web scraping in JavaScript and web scraping in Python.

The main ways to scrape websites with Ruby

Not every scraping is the same. Sometimes the data is already in the HTML, and a simple Faraday and Nokogiri script is enough. Sometimes the page is rendered by JavaScript and you need a browser tool like Selenium, Ferrum, Capybara, or Watir. Other times the approach is not to scrape HTML at all, but to call the same JSON endpoint the frontend uses.

Before picking a gem, figure out what kind of page you are dealing with. The best Ruby web scraping tool depends on where the data lives, how the page loads, and how much of the site you need to collect. A static product page, a React search interface, a login form, a JSON endpoint, and a full-site crawl can all fall under "web scraping," but they should not all be solved the same way.

1. Fetching a single static HTML page

This is the simplest case where you want to request and process one individual server rendered page. You send an HTTP request, receive HTML, parse the response, and extract the fields you need.

Use this when:

the data is already present in the initial HTML
you only need one page or a known list of URLs
you do not need to click, scroll, log in, or run JavaScript

Good Ruby tools for this job:

httparty for quick simple requests
Faraday for more complex HTTP requests
Nokogiri for parsing HTML

This is the best starting point for most Ruby scrapers. It is fast, simple, and easy to debug.

Browser's View Source can reveal if we see a fully formed HTML

2. Scraping JavaScript-rendered pages

Some pages return very little useful HTML at first. The browser loads JavaScript, calls APIs, updates the DOM, and only then displays the data you want. In that case, a plain HTTP request may not be enough. Nokogiri can only parse the HTML you give it to it as it cannot run JavaScript.

Use this when:

the data is missing from browser's "View Source"
the site uses React, Vue, Svelte, Angular, or another frontend framework
content appears only after the browser loads, waits, scrolls, or clicks

Good Ruby tools for this job:

Selenium for full browser automation driver
Ferrum for rendering pages in headless Chrome
Cuprite for using headless Chrome through Capybara

Before reaching for a browser, check the Network tab. Many JavaScript-heavy pages get their data from JSON endpoints you can request directly.

3. Using a headless or visible browser

A headless browser is a real browser without a visible window. It runs JavaScript, stores cookies, follows redirects, loads dynamic content, and behaves much more like a human browsing session than a plain HTTP client.

Use a headless browser when:

JavaScript rendering is required
you need to wait for elements to appear
you need browser APIs, cookies, local storage, or client-side navigation
the page behaves differently outside a browser

Use a visible browser when:

you are developing or debugging a scraper
you need to watch what the page is doing
popups, consent banners, or complex interactions are getting in the way

Good Ruby tools for this job:

Selenium when you want direct control over Chrome or another browser
Ferrum when you want direct control over headless Chrome through the Chrome DevTools Protocol
Capybara when you want a higher-level DSL for visits, clicks, forms, waits, and screenshots
Watir when you want a readable browser automation API built around WebDriver

Browser automation is powerful, but it is also slower and heavier. If a simple Faraday request can get the same data, do not use a browser.

💡 A little later we'll see how we can request JSON directly with some website which is a good alternative to full browser scraping.

4. Submitting forms, keep cookies, and work with sessions

Some sites require state. You might need to submit a search form, keep cookies, follow redirects, or use an authenticated session you are allowed to access.

Use this when:

the site has regular HTML forms
you need cookies or session persistence
you need to follow links like a browser
you need login-style behavior without JavaScript rendering

Good Ruby tools for this job:

Mechanize for forms, links, cookies, and redirects
Faraday with cookie support for custom HTTP flows
Selenium , Capybara , or Watir when JavaScript is part of the interaction

Mechanize is often the right tool for older, form-based sites. It is much lighter than running a browser, but it will not execute JavaScript.

5. Calling JSON endpoints instead of scraping HTML

Sometimes the page is mostly a frontend for an API. If you open the Network tab and inspect Fetch/XHR requests, you may find a clean JSON response containing the data you need.

Use this when:

the frontend loads data from a JSON endpoint
the HTML is hard to parse but the JSON is structured
the endpoint is public and permitted for your use case

Good Ruby tools for this job:

httparty for quick JSON requests in small scripts
Faraday for structured API requests with headers, timeouts, and retries
Ruby's built-in JSON library for parsing responses

This can be faster and more reliable than parsing HTML. But be careful with private, undocumented, or authenticated endpoints. Just because your browser can see a request does not automatically mean you can reuse it freely.

6. Crawling a website instead of scraping one page

Scraping means extracting data. Crawling means discovering pages. If you already know the URLs, you probably do not need a crawler. But if the job is to find pages, follow links, avoid duplicates, and keep going until a limit is reached, you are building one.

Example of crawling flow

Use this when:

you need to scrape many pages
you do not know every URL upfront
you need to follow internal links
you need depth limits, deduplication, queues, or sitemap discovery

Good Ruby tools for this job:

Spidr for quick site crawling
Anemone for spidering workflows
Addressabe for safer URL handling
a custom queue when you need production-level control

Small crawlers can live in memory. Larger crawlers usually need persistence, retries, rate limits, and a database or queue so they can resume after a failure.

Choosing the right Ruby web scraping tool

The rule of thumb is to use the lightest tool that can reliably get the data. Start with HTTP and Nokogiri. Move to Mechanize if you need forms, cookies, and sessions without JavaScript. Move to Selenium, Ferrum, Capybara, or Watir when you need JavaScript rendering or real browser interactions. Use a crawler when you need to discover pages, not just scrape known URLs.

Situation	Best approach	Ruby tools
Data is in the initial HTML	HTTP + parser	`Faraday` + `Nokogiri`
You need a quick request script	Simple HTTP helper	`HTTParty`
You need forms, cookies, redirects, and links without JavaScript	Browser-like HTTP automation	`Mechanize`
You need JavaScript rendering	Headless browser	`Ferrum`, `Selenium`
You need clicks, waits, screenshots, or testing-style flows	Browser automation DSL	`Capybara`, `Watir`
Data comes from a JSON endpoint	API request	`Faraday`, `JSON`
You need to discover many pages	Crawler	`Spidr`, `Anemone`, custom queue
You are scraping inside a Rails app	Background job	`ActiveJob`, `Sidekiq`, `Solid Queue`

This decision matters because every step up adds cost. Browsers are slower than HTTP requests. Crawlers are more complex than one-page scrapers. Authenticated sessions are more fragile than public pages. The best scraper is usually the simplest one that still works reliably.

Scraping static HTML pages with Ruby

Static scraping is the simplest way to scrape a website. It works when the data you need is already present in the HTML returned by the server. If you fetch the page with an HTTP client, the product names, prices, links, article text, or other fields are already sitting in a response body.

If the page loads an empty shell and fills it in later with React, Vue, Svelte, or a browser-side API request, parsers like Nokogiri will not see the rendered content. Nokogiri parses HTML and it does not run JavaScript. That makes static scraping way faster but not always possible as opposed to JavaScript-rendered scraping.

A quick way to check: open "View Source" in your browser and search for the text you want to scrape. If the text is there, a static scraper is probably enough. If it only appears in the Elements panel after JavaScript runs, you may need a browser or a JSON API endpoint instead.

For static pages, you usually need two pieces:

an HTTP client to download the page
an HTML parser to extract data from the response

In Ruby, the most practical default stack is:

Faraday Faraday for HTTP requests
Nokogiri for parsing HTML

Ruby ships with Net::HTTP, but Faraday usually lead to clearer scraper code. There is also the HTTParty gem, which is a good fit for quick scripts. I prefer Faraday for long-lived scrapers because its middleware model makes retries, adapters, headers, logging, and timeouts easier to manage as the scraper grows.

Set up a Ruby scraping project

Install the required libraries with gem install or write a Gemfile for better maintenance of the scraper:

# Gemfile
source "https://rubygems.org"

ruby ">= 3.2"

gem "faraday"
gem "faraday-retry"
gem "nokogiri"
gem "csv"

Then install dependencies:

bundle install

Fetch HTML with Faraday

Here is a minimal HTTP example:

require "faraday"

response = Faraday.get("https://books.toscrape.com/")

puts response.status
puts response.body[0, 200]

Before parsing a page, check that the request succeeded:

require "faraday"

response = Faraday.get("https://books.toscrape.com/")

unless response.success?
  abort "Request failed with status #{response.status}"
end

html = response.body

For anything more than a throwaway script, create a reusable connection with timeouts and headers:

require "faraday"

conn = Faraday.new(url: "https://books.toscrape.com") do |f|
  f.options.timeout = 10
  f.options.open_timeout = 5
end

response = conn.get("/") do |req|
  req.headers["User-Agent"] = "RubyStaticScraper/1.0"
  req.headers["Accept"] = "text/html"
end

unless response.success?
  abort "Request failed with status #{response.status}"
end

html = response.body

Parse HTML with Nokogiri

Once you have the HTML response, parse it with Nokogiri:

require "nokogiri"

doc = Nokogiri::HTML(html)

puts doc.at_css("title")&.text&.strip

at_css returns the first matching element. css returns all matching elements.

Extract text, links, and attributes

Now let's scrape book titles, prices, and URLs from books.toscrape.com:

require "faraday"
require "nokogiri"
require "uri"

BASE_URL = "https://books.toscrape.com/"

response = Faraday.get(BASE_URL)
raise "Request failed: #{response.status}" unless response.success?

doc = Nokogiri::HTML(response.body)

books = doc.css(".product_pod").filter_map do |book|
  link = book.at_css("h3 a")
  href = link&.[]("href")

  next unless link && href

  {
    title: link["title"],
    price: book.at_css(".price_color")&.text&.delete("^0-9."),
    url: URI.join(BASE_URL, href).to_s
  }
end


puts books

Notice a few details:

link["title"] extracts an attribute.
link&.[]("href") avoids crashing if the link is missing.
filter_map skips incomplete records and returns only complete book hashes.
URI.join turns relative links into absolute URLs.
delete("^0-9.") strips the currency symbol from the price.

Save the scraped data to CSV

For static pages, CSV is often the simplest useful output:

require "csv"

CSV.open("books.csv", "w", write_headers: true, headers: %w[title price url]) do |csv|
  books.each do |book|
    csv << [book[:title], book[:price], book[:url]]
  end
end

This gives you a spreadsheet-friendly file while keeping the scraper small and easy to run.

Faraday vs HTTParty

Both Faraday and HTTParty gems are good enough to build a scraper. Pick one based on how much structure the scraper needs:

Tool	Best for	Tradeoff
`Faraday`	Long-lived scrapers, retries, middleware, custom adapters	Slightly more setup
`HTTParty`	Quick scripts and simple requests	Less flexible for advanced HTTP workflows

A quick HTTParty version looks like this:

require "httparty"
require "nokogiri"

response = HTTParty.get("https://books.toscrape.com/")
doc = Nokogiri::HTML(response.body)

puts doc.css(".product_pod h3 a").map { |a| a["title"] }

For a one-off script, that is hard to beat. For a scraper that will run every day, use the structure and middleware you get from Faraday.

CSS selectors and XPath with Nokogiri

Nokogiri is the default HTML and XML parser in the Ruby ecosystem. If you search for web scraping with Ruby and Nokogiri, this is the gem you will see everywhere. As mentioned before, Nokogiri is not a browser and it cannot fetch web pages or execute JavaScript. It parses HTML or XML you already have.

CSS selectors in Nokogiri

CSS selectors are usually the easiest way to find elements:

doc.css(".product_pod").each do |product|
  title = product.at_css("h3 a")&.[]("title")
  price = product.at_css(".price_color")&.text&.strip

  puts "#{title}: #{price}"
end

Common selector patterns:

doc.at_css("h1") # first h1
doc.css("a") # all links
doc.css(".price") # elements with class price
doc.css("#content") # element with id content
doc.css("article.product a") # links inside product articles

XPath in Nokogiri

XPath is more verbose, but it can express queries that are awkward with CSS:

doc.xpath("//a").each do |link|
  puts link["href"]
end

For example, XPath is useful when you need text matching or more complex conditions:

# Find links containing specific text
links = doc.xpath("//a[contains(text(), 'Next')]")

# Find elements with a class fragment
prices = doc.xpath("//*[contains(@class, 'price')]")

In most Ruby web scrapers, CSS selectors are enough. Reach for XPath when CSS gets too clumsy.

Nokogiri vs Nokolexbor

Nokogiri is the safe default. It is mature, well documented, and widely used library that you most likely already know. For most projects, choosing Nokogiri is enough. But if you need to parse a large number of elements by CSS or XPath, nokolexbor is worth learning about.

The Nokolexbor gem from SerpApi is a parser based on Lexbor which focuses on fast HTML 5 parsing using at_css and at_xpath selectors.

Tool	Best for	Notes
`Nokogiri`	Most Ruby scraping projects	Mature, CSS/XPath support, widely used ecosystem
`Nokolexbor`	Performance-heavy HTML5 parsing	Consider when parsing speed matters

You do not have to optimize for this early on, but Nokolexbor can save you when you'll need it. Network requests and browser automation are usually much slower than parsing the response.

Static scraping is the fastest and most reliable option when it works. If your selectors return nothing, check whether the page is rendering data with JavaScript or fetching it from a JSON endpoint before assuming Nokogiri is broken.

Choosing the right HTTP client for scraping

Faraday and httparty are not the only Ruby HTTP clients. They are just convenient choices for a web scraping tutorial: Faraday gives you structure, middleware, retries, and adapters; HTTParty gives you a very small API for quick scripts.

Ruby has several other HTTP libraries worth knowing about:

Ruby ships with Net::HTTP, which avoids an extra dependency but is unnecessarily verbose.
HTTPX is a modern option worth considering for high-throughput scrapers, especially if you care about HTTP/2, persistent connections, and concurrent requests.
http.rb has a pleasant request-chaining API
Excon is a solid lower-level client used in parts of the Ruby ecosystem.

For most static scraping tasks, the HTTP client is not the hard part. Fetch the page reliably, then spend most of your effort on parsing, retries, rate limits, and maintenance. Use Faraday for the examples in this guide unless you have a reason not to. Use HTTParty for tiny scripts. Look at HTTPX when performance, concurrency, or HTTP/2 become important.

Scraping JavaScript-rendered pages with Ruby

If a Nokogiri scraper returns empty results, do not assume your selector is wrong. The data may not be in the original HTML at all. That is the classic example of Nokogiri JavaScript not working. Nokogiri parses the HTML response, but it does not run JavaScript. Let's see what we can do about it.

First check whether you really need JavaScript

Before using Selenium or Ferrum, inspect the page.

Open the page in your browser.
Open Developer Tools.
Go to the Network tab.
Filter by Fetch/XHR.
Reload the page.
Look for JSON responses containing the data.

If you find a public JSON endpoint, you may be able to call it directly with Faraday. That is usually faster and more reliable than running a browser.

Use a headless browser only when:

The data is generated client-side and no usable API is available.
The page requires clicks, scrolling, or browser state.
You need to test exactly what a browser sees.

Scrape a dynamic page with Selenium

Selenium is the most widely known browser automation tool. It can control Chrome, Firefox, and other browsers.

Add it to your Gemfile as:

gem "selenium-webdriver"

Here's a minimal headless Selenium example using Chrome:

require "selenium-webdriver"
require "nokogiri"

options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")

browser = Selenium::WebDriver.for(:chrome, options: options)

begin
  browser.navigate.to("https://quotes.toscrape.com/js/")

  wait = Selenium::WebDriver::Wait.new(timeout: 10)
  wait.until { browser.find_elements(css: ".quote").any? }

  doc = Nokogiri::HTML(browser.page_source)

  quotes = doc.css(".quote").map do |quote|
    {
      text: quote.at_css(".text")&.text&.strip,
      author: quote.at_css(".author")&.text&.strip
    }
  end

  puts quotes
ensure
  browser.quit
end

The important part is the wait. Avoid blind sleep calls when you can wait for a specific element.

Scrape a dynamic page with Ferrum

Ferrum controls Chrome or Chromium through the Chrome DevTools Protocol. It is a good Ruby-native choice for headless Chrome scraping.

Add it to your Gemfile:

gem "ferrum"

Example of using Ferrum to load a website and parsing the body with Nokogiri:

require "ferrum"
require "nokogiri"

browser = Ferrum::Browser.new(timeout: 10)

begin
  browser.go_to("https://quotes.toscrape.com/js/")
  browser.network.wait_for_idle

  doc = Nokogiri::HTML(browser.body)

  quotes = doc.css(".quote").map do |quote|
    {
      text: quote.at_css(".text")&.text&.strip,
      author: quote.at_css(".author")&.text&.strip
    }
  end

  puts quotes
ensure
  browser.quit
end

Ferrum is especially nice when you want headless Chrome without the full Selenium stack.

Selenium vs Ferrum

Both Selenium and Ferrum will get a job done. Here's a small comparison on how they differ:

Tool	Best for	Tradeoff
`Selenium`	Cross-browser automation, widely supported, familiar ecosystem	Heavier and needs browser/driver setup
`Ferrum`	Headless Chrome via Chrome DevTools Protocol	Chrome/Chromium-focused

If you need cross-browser behavior or already know Selenium, use Selenium. If you want a Ruby-friendly headless Chrome tool for scraping, try Ferrum.

Browser automation with Capybara and Watir

Selenium and Ferrum are lower-level browser control tools. Capybara and Watir provide nicer APIs for browser-like workflows. Think of them as higher-level tools closer to your application code.

Use Capybara and Watir when your scraper needs to behave like a user:

fill in fields
click buttons
wait for elements
navigate flows
save screenshots
work with sessions

Use Capybara for scraping workflows

Capybara is best known for Rails system tests, but it can also automate external websites in general. Here's such an example:

require "capybara"
require "capybara/dsl"
require "selenium-webdriver"

Capybara.register_driver :selenium_chrome_headless do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_argument("--headless=new")

  Capybara::Selenium::Driver.new(app, browser: :chrome, options: options)
end

Capybara.default_driver = :selenium_chrome_headless
Capybara.run_server = false
Capybara.default_max_wait_time = 10

include Capybara::DSL

visit "https://quotes.toscrape.com/js/"

all(".quote").each do |quote|
  puts quote.find(".text").text
end

page.save_screenshot("quotes.png")

Capybara's waiting behavior is one of its biggest advantages. find and has_css? wait for elements instead of failing immediately. Note the include Capybara::DSL like which makes this DSL possible.

Take screenshots with Capybara

Screenshots are useful when debugging browser scrapers. If an element is missing, a cookie banner blocks the page, or the page looks different in headless mode, saving a screenshot shows you what the browser actually saw.

With Capybara, screenshots are straightforward:

page.save_screenshot("page.png")

You can also save screenshots with dynamic names:

page.save_screenshot("screenshots/product-#{product_id}.png")

Just make sure the target directory exists before saving:

require "fileutils"

FileUtils.mkdir_p("screenshots")
page.save_screenshot("screenshots/page.png")

Screenshots work in headless mode too which makes them especially helpful in background jobs, CI, or production debugging.

This is not too different to what a typical Rails system test setup looks like.

Screenshots for JavaScript websites

However, if you make a screenshot too quickly, it might like dynamic elements, like in this case for the Ruby official website.

Capybara screenshot of ruby-lang.org without waiting for JavaScript

To fix it, you'll need to wait using the setTimeout function. For this cases, run execute_script (again from the Capybara DSL):

visit "https://www.ruby-lang.org/en/"

# ruby-lang.org is mostly static, so this creates a small visible marker
# after a delay to demonstrate waiting for JavaScript-rendered content.
execute_script(<<~JS)
  window.setTimeout(function () {
    var marker = document.createElement("div");
    marker.id = "capybara-js-ready";
    marker.textContent = "JavaScript content loaded — screenshot taken after Capybara waited";
    marker.style.position = "fixed";
    marker.style.left = "24px";
    marker.style.bottom = "24px";
    marker.style.zIndex = "999999";
    marker.style.padding = "14px 18px";
    marker.style.background = "#cc342d";
    marker.style.color = "white";
    marker.style.font = "16px sans-serif";
    marker.style.borderRadius = "8px";
    marker.style.boxShadow = "0 4px 16px rgba(0,0,0,0.25)";
    document.body.appendChild(marker);
  }, 2000);
JS

find "#capybara-js-ready", wait: 10
save_screenshot "ruby.png"

Now you should be able to see the site in all its glory:

Ruby official website screenshot

Use Watir for browser-like automation

Watir is another readable browser automation library built around WebDriver.

require "watir"

browser = Watir::Browser.new(:chrome, headless: true)

begin
  browser.goto("https://quotes.toscrape.com/js/")

  browser.divs(class: "quote").each do |quote|
    puts quote.span(class: "text").text
  end
ensure
  browser.close
end

Watir is pleasant when you want your scraper code to read like browser instructions.

Capybara vs Watir

Tool	Best for	Tradeoff
`Capybara`	Rails-style flows, waiting DSL, screenshots, system-test familiarity	Originally designed for testing
`Watir`	Human-readable browser automation	Another abstraction over WebDriver

If you already write Rails system tests, Capybara will feel natural. If you want a direct browser automation API, Watir is easy to read.

Scraping forms, cookies, and sessions with Mechanize

Mechanize is a classic Ruby gem for automated web interaction. It can follow links, submit forms, store cookies, handle redirects, and keep history. It does not execute JavaScript which is the main limitation. Use Mechanize when you need browser-like HTTP behavior without a browser.

When Mechanize is better than Selenium

Mechanize is often better than Selenium when:

The site is mostly static.
You need to submit normal HTML forms.
You need cookies and sessions.
You do not need JavaScript rendering.
You want speed and lower resource usage.

Selenium is better when JavaScript is required.

Submit a form with Mechanize

Here's a short example of submitting a search form:

require "mechanize"

agent = Mechanize.new
agent.user_agent_alias = "Mac Safari"

page = agent.get("https://example.com/search")

form = page.form_with(action: "/search")
form["q"] = "ruby"

results = agent.submit(form)

puts results.search("h2").map(&:text)

Mechanize pages can be searched with Nokogiri-style methods because Mechanize uses Nokogiri internally.

Mechanize vs Selenium

Both Mechanize and Selenium are excellent tools but they both have their ideal use-case as shown in the table below:

Tool	Best for	Tradeoff
`Mechanize`	Forms, links, cookies, sessions without JavaScript	Does not execute JavaScript
`Selenium`	JavaScript, real browser interactions, complex UI flows	Slower and heavier

If you search for Mechanize Ruby login, you will find many examples because this is exactly where Mechanize shines; handling old-school form-based websites.

Crawling multiple pages with Ruby

If you already know every URL, you do not need a crawler. Just loop through the URLs and scrape each page. However, you need crawling when you want to discover URLs by following links you find along the way.

Scraping one page vs crawling a site

A scraper extracts data from a page. A crawler finds pages. And most production systems are both. Here's an example of a scraping workflow involving crawling:

Start with one or more seed URLs.
Fetch a page.
Extract data.
Extract links.
Normalize links.
Skip links already visited.
Repeat until a depth, count, or domain limit is reached.

We already talked plenty about scraping itself, now let's talk about crawling.

Crawl links with Spidr

Spidr is a very popular Ruby web spidering library. It comes with a small DSL that gives us an option to work with every individual page like this:

require "spidr"

Spidr.site("https://example.com/") do |spider|
  spider.every_page do |page|
    puts page.url
  end
end

This is useful for quick site crawling and link discovery.

Crawl links with Anemone

Anemone is another Ruby crawling library with similar DSL:

require "anemone"

Anemone.crawl("https://example.com/") do |anemone|
  anemone.on_every_page do |page|
    puts page.url
  end
end

Both libraries are useful, but for serious scraping you may eventually want a custom queue so you control retries, persistence, rate limits, priorities, and deduplication.

Spidr vs Anemone vs custom queue

Here's a short comparison of Spidr vs Anemone libraries:

Approach	Best for	Tradeoff
`Spidr`	Quick site crawling	Less custom architecture
`Anemone`	Spidering workflows	Older ecosystem feel
Custom queue	Production control	More code

For a small Ruby web crawler, a gem is fine. For a production crawler, a custom queue backed by a database or Redis can be better.

Scraping JSON APIs instead of HTML

Getting data directly from JSON requests is insanely easier as the data already comes in the format you likely want. Before writing a complex browser scraper, check whether the page gets its data from JSON.

XHR requests on the Chrome Network tab

This is how you can do so in Chrome or Firefox:

Open Developer Tools.
Go to Network.
Filter by Fetch/XHR.
Reload the page.
Click requests and inspect responses.

If you find the data in JSON, you can often skip HTML parsing entirely.

Request JSON with Faraday

If you can find such requests, you can then fetch the respective JSON using Faraday (or alternative):

require "faraday"
require "json"

response = Faraday.get("https://api.example.com/products") do |req|
  req.headers["Accept"] = "application/json"
end

raise "Request failed: #{response.status}" unless response.success?

data = JSON.parse(response.body)

puts data

API scraping vs HTML scraping

Most scraping is done with fetching full HTML pages, but if you can avoid it consider scraping the API calls:

Approach	Pros	Cons
JSON/API	Structured, easier to parse, often more stable	May require auth, may be private, may be unsupported
HTML	Works when no API exists	Selectors break more often

Be careful with private or undocumented endpoints. Just because your browser can see a request does not automatically mean you can use it freely for commercial scraping. Check terms and permissions.

Reliability: Timeouts, retries, errors, and logging

The difference between a demo scraper and a scraper you'll run for a long time is reliability. Websites go down. Networks fail. HTML changes. Requests time out. Selectors break. Your code should expect that.

Set connection timeouts

Never let a scraper hang on a connection forever. Set connection times like these for Faraday:

conn = Faraday.new(url: "https://books.toscrape.com") do |f|
  f.options.timeout = 10
  f.options.open_timeout = 5
end

Retry temporary failures

You should try retried temporary failures. For Faraday, you can use faraday-retry for transient problems:

require "faraday"
require "faraday/retry"

conn = Faraday.new(url: "https://books.toscrape.com") do |f|
  f.request :retry,
    max: 3,
    interval: 0.5,
    backoff_factor: 2,
    retry_statuses: [429, 500, 502, 503, 504]

  f.options.timeout = 10
  f.options.open_timeout = 5
end

Retry carefully. Retrying 429 Too Many Requests immediately and aggressively is a good way to make the problem worse. Consider adding backoff and reducing request rate.

Handle HTTP status codes

Don't forget that requests don't just end in 200 status codes. You might want to fetch another page on redirect, stop at forbidden and not found pages, retry later for rate limiting and investigate on 500 server errors. Here's the reminder of the main status codes:

200: success
301 / 302: redirect
403: forbidden
404: not found
429: rate limited
500+: server error

An example handler could look like this example:

response = conn.get("/catalogue/page-1.html")

case response.status
when 200
  parse(response.body)
when 404
  logger.warn("Page not found")
when 429
  logger.warn("Rate limited")
else
  logger.error("Unexpected status: #{response.status}")
end

Avoid crashing on missing elements

Selectors break and optional elements disappear. Always parse defensively using the &. operator:

def text_at(node, selector)
  node.at_css(selector)&.text&.strip
end

price = text_at(product, ".price_color")

If a field is required, you can raise a clear error, but your scraper should likely be able to continue.

Add logging

Ruby's standard Logger is enough for many scrapers:

require "logger"

logger = Logger.new($stdout)
logger.info("Starting scrape")
logger.warn("Missing price for product")
logger.error("Request failed")

A scheduled scraper with no logs is painful to debug.

Cache pages during development

When building selectors, save sample HTML locally. This avoids hitting the target site repeatedly while you tune parsing code.

You can use:

plain saved .html fixtures
VCR gem
WebMock gem
a local cache directory

Parser tests against saved HTML fixtures are one of the best ways to keep scrapers maintainable.

💡 When things get difficult, you can rely on a service that does the hard parts for you. SerpApi provides a clean API for all kinds of search data, so you don't have to scrape search engine results yourself.

Performance: Concurrency, batching, and rate limits

Scraping performance is not just about making requests as fast as possible. A good scraper should finish in a reasonable time without overwhelming the target site, wasting local resources, or triggering rate limits.

The goal is controlled throughput when enough concurrency avoids waiting on the network all day, but not so much that you create failures for yourself or for the site you are scraping.

Use concurrency for HTTP, not everything

Static HTTP scraping is usually I/O-bound. Most of the time is spent waiting for the network, so Ruby threads can help even if the Ruby code itself is not CPU-heavy.

Browser scraping is different. A headless browser uses much more CPU and memory than a plain HTTP request. Running many browser sessions at once can overload your machine or server.

Parallel scraping with Ruby threads

For static pages, a small thread pool is often enough. Put URLs into a queue, let a few workers fetch and parse them, and collect the results in another queue:

require "thread"

urls = [
  "https://books.toscrape.com/catalogue/page-1.html",
  "https://books.toscrape.com/catalogue/page-2.html",
  "https://books.toscrape.com/catalogue/page-3.html"
]

queue = Queue.new
urls.each { |url| queue << url }

results = Queue.new
workers = 3.times.map do
  Thread.new do
    loop do
      url = queue.pop(true)
      results << scrape_page(url)
    rescue ThreadError
      break
    end
  end
end

workers.each(&:join)

This pattern gives you controlled concurrency. You can increase or decrease the number of workers based on the target site, your machine, and how much rate limiting you see.

In production, combine this with timeouts, retries, rate limits, and logging. Do not spawn 100 threads just because you can.

Add rate limits and delays

A small delay between requests can make a scraper much more polite:

pages.each do |url|
  scrape_page(url)
  sleep 1
end

For more advanced setups, use adaptive throttling. Slow down when responses take longer, error rates increase, or you start seeing 429 Too Many Requests.

Avoid browser automation unless needed

A headless browser can take seconds to start and hundreds of megabytes of memory to run. A plain HTTP request may take milliseconds and use almost no memory. Use browsers for browser-only problems; JavaScript rendering, clicks, scrolling, screenshots, or workflows that require real browser state.

Parse efficiently

Most scraper performance issues come from the network or browser automation, but parsing can still matter on large pages or large batches.

A few practical tips:

Scope selectors to a container instead of searching the whole document repeatedly.
Avoid parsing the same HTML more than once.
Write output incrementally for large scrapes instead of keeping everything in memory.

Ruby web scraping FAQ

What is the best Ruby gem for web scraping?

There is no single best gem. For most static pages, use Faraday and Nokogiri. For simple scripts, HTTParty is fine. For forms and cookies without JavaScript, use Mechanize. For JavaScript-rendered pages, use Selenium, Ferrum, Capybara, or Watir.

Can Nokogiri scrape JavaScript?

No. Nokogiri parses HTML and XML. It does not execute JavaScript. If the content is rendered by JavaScript, use a headless browser or find the JSON endpoint the page uses.

Is Ruby better than Python for web scraping?

Python has a larger scraping and data ecosystem, especially around Scrapy, Playwright, Pandas, and machine learning workflows. Ruby is excellent if you already work in Ruby or Rails, want readable automation scripts, or need scraping inside a Ruby application.

Should I use Selenium or Ferrum?

Use Selenium if you need cross-browser automation or want the most widely documented browser tool. Use Ferrum if you want a Ruby-friendly headless Chrome tool using the Chrome DevTools Protocol.

Should I use Mechanize or Nokogiri?

They solve different problems. Nokogiri parses HTML. Mechanize fetches pages, follows links, submits forms, and manages cookies, using Nokogiri under the hood. For simple parsing, use Nokogiri. For non-JavaScript form workflows, consider Mechanize.

How do I scrape multiple pages in Ruby?

If you know the URLs, loop through them with Faraday and parse each response with Nokogiri. If you need to discover URLs, use a crawler like Spidr, Anemone, or a custom queue that extracts and deduplicates links.

How do I avoid getting blocked while scraping with Ruby?

Scrape responsibly. Respect terms, robots.txt, and rate limits. Add delays. Set timeouts. Avoid unnecessary requests. Do not scrape private data or bypass access controls. If you receive 429, slow down.

Can I scrape websites from Rails?

Yes, but do it in background jobs, not controller actions. Put scraping logic in service objects, run it with ActiveJob, Sidekiq, or Solid Queue, and store results with ActiveRecord.

Conclusion

Web scraping with Ruby can be as simple as using Faraday with Nokogiri, but real-world scraping is broader than fetching one page and parsing a few tags.

The best Ruby web scraper starts with the right approach:

Use HTTP and Nokogiri for static HTML.
Use JSON endpoints when the site already exposes structured data.
Use Mechanize for forms, cookies, and non-JavaScript sessions.
Use Selenium, Ferrum, Capybara, or Watir when you need a real browser.
Use a crawler or queue when you need many pages.
Add retries, timeouts, logging, tests, and rate limits before calling it production-ready.

Ruby gives you all the pieces: clean HTTP clients, mature parsers, browser automation, background jobs, CSV/JSON support, and Rails integration. The trick is not finding a magic scraping gem. The trick is choosing the smallest reliable tool for the page in front of you.

Categorizing hotels using Google Hotels images

Josef Strzibny — Fri, 19 Jun 2026 08:34:11 +0000

Hotel listings often contain dozens of images, but the details travelers care about are buried inside them. Do rooms have a proper desk, a bathtub, a balcony, or enough natural light? By automatically categorizing hotel images with vision models, we can turn messy photo galleries into searchable attributes, verify claims from listing descriptions, or detect misleading photos. In this post, we'll look how to find the right hotel photos using SerpApi hotel APIs and determine if they feature a coffee machine or not.

Finding the right things in hotel images

The hard part is not just asking “what is in this image?” but deciding what counts for a specific use case. A coffee machine in a hotel lobby might be useful, but it is not the same as a coffee machine in a private room. A desk in a restaurant does not mean the room is suitable for remote work. A beautiful exterior photo tells us nothing about whether the room has a kitchenette, balcony, or bathtub. Good image categorization therefore needs both visual detection and context classification; what object is visible, where it appears, and whether that scene is relevant to the traveler’s intent.

Once images are categorized this way, hotel search can become much more practical and transparent. Instead of relying only on amenities text or user reviews, we can attach evidence to every claim: "coffee machine visible on cabinet," "work desk next to bed," "balcony attached to guest room," or "kitchenette visible in suite." This makes it possible to build filters that are based on what is actually shown, highlight the best supporting photos, ignore misleading common-area images, and give travelers more confidence that the hotel has the exact room features they care about.

Building a Hotel Coffee Analyzer

To demonstrate this, we'll build a Hotel Coffee Analyzer that can find coffee machines in selected hotels. Here's a rough workflow:

Find hotels from SerpApi's Google Hotels API.
Fetch room-section photos via SerpApi's Google Hotels Photos API.
Analyze each image URL with an LLM model using structured JSON response
Accept coffee pictures if:
- Coffee machine is found
- If the scene is guest_room, kitchenette, or suite
- Evidence fields are non-empty
- Model confidence is at least 0.80
Display results

Prerequisites

We'll need to integrate two API providers to make our idea a reality. For hotel and hotel pictures' searches we'll use SerpApi , while for analyzing the images we'll use OpenAI and its models:

For the SerpApi API key, head over to the SerpApi dashboard to copy your key.
For the OpenAI API key, log into the OpenAI Platform and create a new key under Manage > API keys.

Note down both keys in an .env file in the project's root directory as we'll load these keys from there:

$ cat .env
SERPAPI_API_KEY=a9142c...
OPENAI_API_KEY=sk-proj-g0...

💡 SerpApi is free for up to 250/searches a month. See full pricing.

How to find hotels using Google Hotels

SerpApi has a Google Hotels Properties API which we can use for our base hotel search. We have to specify the location of the hotel as well as check-in and check-out dates in the YYYY-MM-DD format. Optionally we can ask for certain number of stars, for how many adults are we booking, or even if the property is eco-certified.

As a result, the API returns a list of properties matching our criteria including their name, rate_per_night, reviews, amenities and many more. One important thing we'll have to note down is the property token which is important to fetch the images in the next step. You can try this search first in the playground.

Using the SerpApi client

Once you see the hotels you would expect, you can translate this call into code. I'll use Ruby in this post, but you can pick one of the many other available SDKs like the one for JavaScript or Python.

First, we install the official Ruby library from RubyGems (or add it to Gemfile):

gem install serpapi

Then we can require it and initialize the client with our API key.

require "serpapi" 

client = SerpApi::Client.new(
  api_key: "..."
)

Once done, we are ready to call @client.search() with the parameters we want.

For generic hotel search, we specify the google_hotels engine, provide the destination as a query with q and narrow it down using check_in_date and check_out_date dates:

@client.search(
  engine: "google_hotels",
  q: "#{destination} hotels",
  check_in_date: check_in_date,
  check_out_date: check_out_date,
  adults: adults,
  hotel_class: hotel_class,
  gl: country,
  hl: "en"
)

This returns a list of hotels that may contain these top-level fields:

properties: an array of the primary hotel results
ads: an array of the sponsored hotel results
non_matching_properties: an array of additional hotel candidates Google returned even though they may not match the requested filters exactly

We can extract candidates from all three, with priority given to primary exact matches. I'll talk about all of these in more detail in the next section.

Using pagination

A single API request gives you one page of hotel results. If more results are available, SerpApi may return a pagination token that you can use to request the next page. You can find it under serpapi_pagination as next_page_token:

{
  "serpapi_pagination": {
    "next_page_token": "..."
  }
}

Here's how to grab it in Ruby:

next_page_token = results.dig("serpapi_pagination", "next_page_token")

The follow-up search will have the exact same parameters except the newly provided next_page_token:

second_page = client.search(
  engine: "google_hotels",
  q: "Dresden hotels",
  check_in_date: "2026-06-10",
  check_out_date: "2026-06-11",
  adults: 2,
  hotel_class: 4,
  gl: "de",
  hl: "en",
  next_page_token: next_page_token
)

This way you can always get as many properties as needed or restart the search from a specific point.

How to find relevant hotels

Let's look again at the responses we are getting from the google_hotels engine to find the right candidate hotels from the results involving three different groups.

Primary properties

The output JSON comes with three different sections. The properties array is the main Google Hotels results. Since we are interested in hotels, we should check that we are getting objects of type: "hotel".

We can also be interested in the following fields regarding the hotel's stars:

hotel_class is a text like 4-star hotel or 4 star hotel
extracted_hotel_class as an assumed star number

The ads array is containing sponsored hotel listing. It's included only if they match the requested star class. This avoids mixing unrelated sponsored entries into the result list.

Non-matching properties

Finally, we also get the non_matching_properties array as a fallback for other hotel candidates after primary properties and matching ads. This is important because SerpApi's Google Hotels API can return only a small number of exact properties results for a star-class query while placing many useful hotel candidates in non_matching_properties.

Deduplication

Once you have your candidate hotels from these searches it's useful to deduplicate them using their property_token. The property_token is the only stable hotel identifier and is also needed later for google_hotels_photos.

Finding relevant hotels pictures

The next step is to download hotel images. Once we know which hotels we are interested in, we can fetch their pictures from the photo gallery using SerpApi's Google Hotels Photos API.

To run the search we can continue running a new search on the same @client we already initialized with our API key, just with a different engine parameter:

@client.search(
  engine: "google_hotels_photos",
  property_token: property_token
)

The API call only requires the hotel's property_token from the previous step. The response will looks like this:

{
   "search_metadata":{
      "id":"6a204a9eb24c5df05effc392",
      "status":"Success",
      "json_endpoint":"https://serpapi.com/searches/iI1l03cWiCA9ZJ028kaBdA/6a204a9eb24c5df05effc392.json",
      "created_at":"2026-06-03 15:39:10 UTC",
      "processed_at":"2026-06-03 15:39:10 UTC",
      "google_hotels_photos_url":"https://www.google.com/travel/hotels/entity/ChcI15Wcx823svUQGgsvZy8xdm50eHpmMhAB",
      "raw_html_file":"https://serpapi.com/searches/iI1l03cWiCA9ZJ028kaBdA/6a204a9eb24c5df05effc392.html",
      "prettify_html_file":"https://serpapi.com/searches/iI1l03cWiCA9ZJ028kaBdA/6a204a9eb24c5df05effc392.prettify",
      "total_time_taken":0.90
   },
   "search_parameters":{
      "engine":"google_hotels_photos",
      "property_token":"ChcI15Wcx823svUQGgsvZy8xdm50eHpmMhAB"
   },
   "sections":[
      {
         "title":"At a glance",
         "total":201,
         "next_page_token":"w57_wXicdVJLasMwEKXkFr1Dd6GQ5ehJlElwqVMScLYlnaafuNA03vcupXfKaSrJnyiSNeBnGL3PeKyf30P9tt2fJtfmKHpRlbvp7Ob-fVWwbsiWsU9BcfEAaaF7l-PHXQmRwmWHJVB30ZeJATnvrlw2fAIhy3KWUlsAIfMhgWPPMOnhIEZCUS1KnhG5eJSYgIR3NpE-hagaNZ97XEZd04uT3mgH8RhDqJ_pvL9BIS0ppEVx-bR4kkB3p16wxofh74fn181a3zpuO5j2WIqpanbRBT2iISzs_j8hxJWxREj7s7Gxm4S7J3DXuwQzpLHTGlbKxYu22HSpf1_bp8Ou3l_9A8jD2jE",
         "photos":[
            {
               "width":5616,
               "height":3744,
               "thumbnail_url":"https://lh3.googleusercontent.com/gps-cs-s/APNQkAHj65LO5BtfQGZwM_DuAG2ybjGRi3ZMc-uuByDtoMm941_3TX1RqL1MeTctAfalQyHaGVJABOQmcSnuWKHP-uiYxIr22t9ZsyaASieCzz92gtEotb787DzGis4ssgqDaxYnIkre0hLJbrBU=w253-h168-k-no",
               "source":"Owner Submitted",
               "photo_url":"https://lh3.googleusercontent.com/gps-cs-s/APNQkAHj65LO5BtfQGZwM_DuAG2ybjGRi3ZMc-uuByDtoMm941_3TX1RqL1MeTctAfalQyHaGVJABOQmcSnuWKHP-uiYxIr22t9ZsyaASieCzz92gtEotb787DzGis4ssgqDaxYnIkre0hLJbrBU=s5616",
               "posted_on":"2025-04-07",
               "source_url":"https://maps.google.com/maps/contrib/103108887507597475893"
            }
         ]
      }
   ]
}

Let's see next how to find the relevant data in there.

Gallery sections

The important part for us is sections. Depending on what you are trying to find, you can locale specific sections that categorize the hotel images into groups like:

rooms
guest rooms
suites
accommodation

Apart from the section title, you should see a number of total pictures, next_page_token to fetch more photos and the photos array with the pictures themselves. Here's a simplified representation of that structure:

{
  "sections": [
    {
      "title": "Rooms",
      "total": 201,
      "next_page_token": "...",
      "photos": []
    },
    {
      "title": "Exterior",
      "total": 201,
      "next_page_token": "...",
      "photos": []
    },
    {
      "title": "Amenities",
      "total": 201,
      "next_page_token": "...",
      "photos": []
    }
  ]
}

Each image object then may contain one or more URL fields:

{
  "thumbnail": "https://...",
  "image": "https://...",
  "original_image": "https://..."
}

The best-quality image is usually under original_image, but we can also fallback to image or thumbnail.

Deduplication

Hotel photo galleries can contain repeated URLs or many near-duplicates, so it is useful to deduplicate provided URLs before processing.

Categorizing the images

Once we know how to find the right hotel images using SerpApi, we need a way to categorize them. For that, we can use a capable LLM model such as GPT-4.1 or GPT-4.1-mini. I will use GPT-4.1-mini in the examples as a faster and cheaper model, but you can try GPT-4.1 if you need better accuracy.

💡 Choose a model that support image input. If the model does not support vision, it may ignore the image, error, or treat the URL as plain text depending on the provider behavior.

RubyLLM is a Ruby interface for LLM models with a beautiful APIs that also lets you build chat agents with custom tools. To set it up, we'll have to provide an OpenAI access key as part of its configuration:

RubyLLM.configure do |config|
  config.openai_api_key = ENV.fetch("OPENAI_API_KEY")
end

chat = RubyLLM.chat(model: "gpt-4.1-mini")

We'll also need classification system instructions that would determine what we are looking for. If we are looking for a coffee machine, we can write that directly inside the system prompt:

chat = chat.with_instructions(<<~PROMPT)
  Analyze the attached hotel image.

  Determine whether there is a real coffee machine visibly present.

  Do not infer from hotel name, amenities, or context.
  Only mark coffee_machine true if the machine is clearly visible.
  Reject kettles, cups, menus, minibars, lobbies, restaurants, bathrooms, and exterior photos.

  Also classify the scene type:
  guest_room, kitchenette, suite, reception, lobby, restaurant, exterior, bathroom, hallway, or other.
PROMPT

These instructions tell the model:

what visual feature to look for
what false positives to avoid
which scene categories are allowed
what kind of evidence is required

This simple prompt should already work, but if we want machine-readable results that would fill in things like the coffee_machine boolean above, we need to tell this to the model as well. Luckily, RubyLLM can request structured JSON output using with_schema which accepts a schema of the output:

schema = {
  name: "coffee_machine_detection",
  strict: true,
  schema: {
    type: "object",
    properties: {
      scene_type: {
        type: "string",
        enum: [
          "guest_room",
          "kitchenette",
          "suite",
          "reception",
          "lobby",
          "restaurant",
          "exterior",
          "bathroom",
          "hallway",
          "other"
        ]
      },
      suitable_room_photo: { type: "boolean" },
      room_evidence: { type: "string" },
      coffee_machine: { type: "boolean" },
      coffee_machine_evidence: { type: "string" },
      confidence: { type: "number" },
      reason: { type: "string" }
    },
    required: [
      "scene_type",
      "suitable_room_photo",
      "room_evidence",
      "coffee_machine",
      "coffee_machine_evidence",
      "confidence",
      "reason"
    ],
    additionalProperties: false
  }
}

chat = chat.with_schema(schema)

This step asks the model to return a predictable object instead of free-form prose. As you can see, we can define a return object, its properties, property types, and if a field is required. All of that is useful, if we want to work with that data later.

💡 You can read more about structured output in the OpenAI Developers documentation.

A final step for the categorization is asking RubyLLM to attach the hotel image and run the query which we can do by providing image URL in the with parameter:

response = chat.ask(
  "Analyze the attached image and return the structured detection result.",
  with: image_url
)

Building the Hotel Coffee Analyzer

Now that we understand how all of these APIs should work together, we can create a proper analyzer. Today we are analyzing rooms for coffee machines, so I am calling the class HotelCoffeeAnalyzer. A user is able to pass the arguments we talked about and get some interesting analysis back.

I am wrapping the analyzer in a runnable script so that we can see it in action immediately. The flow stays the same as in the beginning; finding the right hotels, getting the property ID, fetching the photos, and finally analyzing them. You can also grab the code on GitHub.

At the beginning, I am simply defining all the possible parameters, the extended prompt, the LLM response schema, and configuring clients. The main code is split into two classes. HotelCoffeeAnalyzer searches for the right hotel pictures and internally pass them to CoffeeMachineDetector which can run the LLM call.

require "json"
require "date"
require "fileutils"
require "dotenv/load"
require "serpapi"
require "ruby_llm"

# ------------------------------------------------------------
# Configuration
# ------------------------------------------------------------

SERPAPI_API_KEY = ENV.fetch("SERPAPI_API_KEY")
OPENAI_API_KEY = ENV.fetch("OPENAI_API_KEY")

CHECK_IN_DATE = ENV.fetch("HOTEL_CHECK_IN_DATE") { (Date.today + 1).iso8601 }
CHECK_OUT_DATE = ENV.fetch("HOTEL_CHECK_OUT_DATE") { (Date.today + 2).iso8601 }
HOTEL_DESTINATION = ENV.fetch("HOTEL_DESTINATION", "Dresden")
HOTEL_COUNTRY = ENV.fetch("HOTEL_COUNTRY", "de")
HOTEL_ADULTS = ENV.fetch("HOTEL_ADULTS", "2")
HOTEL_CLASS = ENV.fetch("HOTEL_CLASS", "4")
HOTEL_RESULT_LIMIT = ENV.fetch("HOTEL_RESULT_LIMIT", "10").to_i
HOTEL_SEARCH_PAGES = ENV.fetch("HOTEL_SEARCH_PAGES", "1").to_i
HOTEL_PHOTO_LIMIT = ENV.fetch("HOTEL_PHOTO_LIMIT", "1").to_i
FETCH_HOTEL_PHOTOS = ENV.fetch("FETCH_HOTEL_PHOTOS", "1") == "1"
SERPAPI_DEBUG = ENV.fetch("SERPAPI_DEBUG", "1") != "0"
SERPAPI_DEBUG_STDOUT = ENV.fetch("SERPAPI_DEBUG_STDOUT", "0") == "1"
VISION_DEBUG = ENV.fetch("VISION_DEBUG", "1") != "0"
VISION_DEBUG_STDOUT = ENV.fetch("VISION_DEBUG_STDOUT", "0") == "1"
VISION_DEBUG_DIR = ENV.fetch("VISION_DEBUG_DIR", "vision_debug")

RubyLLM.configure do |config|
  config.openai_api_key = OPENAI_API_KEY
end

PROMPT = <<~PROMPT
  Analyze the attached image from a hotel listing.

  Determine whether there is a REAL coffee machine visibly present in the image.

  Important:
  - Do not infer from the hotel name, amenities, listing context, or likelihood.
  - If the image is an exterior/building facade, street view, reception area, front desk,
    lobby, restaurant, breakfast room, bar, bathroom, hallway, generic city photo, map,
    logo, or any image where no coffee machine is clearly visible, return coffee_machine: false.
  - Do not classify reception/front-desk/common-area photos as guest_room.
  - suitable_room_photo must be true only for a private guest room, suite, or in-room kitchenette.
    A suitable room photo should show private-room evidence such as a bed, bedside table,
    wardrobe, desk in a guest room, private kitchenette, or suite living area. If this evidence
    is missing, suitable_room_photo must be false.
  - Only return coffee_machine: true when a coffee machine itself is clearly visible inside a
    suitable private guest-room/suite/kitchenette context.

  Accept as coffee machines:
  - Nespresso machines
  - Espresso machines
  - Pod coffee makers
  - Drip coffee machines
  - Built-in coffee stations

  Reject:
  - Electric kettles
  - Tea kettles
  - Water boilers
  - Tea trays
  - Mini bars
  - Beverage menus
  - Cups only
  - Coffee served in a cup
  - Exterior hotel photos
  - Reception desks
  - Front desks
  - Lobby/common-area coffee service

  Return ONLY valid JSON:

  {
    "scene_type": "guest_room | kitchenette | suite | reception | lobby | restaurant | exterior | bathroom | hallway | other",
    "suitable_room_photo": true,
    "room_evidence": "Bed and bedside table are visible",
    "coffee_machine": true,
    "coffee_machine_evidence": "Black Nespresso machine visible on cabinet",
    "confidence": 0.95,
    "reason": "Black Nespresso machine visible on cabinet in a private guest room"
  }
PROMPT

COFFEE_MACHINE_SCHEMA = {
  name: "coffee_machine_detection",
  strict: true,
  schema: {
    type: "object",
    properties: {
      scene_type: {
        type: "string",
        enum: %w[guest_room kitchenette suite reception lobby restaurant exterior bathroom hallway other]
      },
      suitable_room_photo: { type: "boolean" },
      room_evidence: { type: "string" },
      coffee_machine: { type: "boolean" },
      coffee_machine_evidence: { type: "string" },
      confidence: { type: "number" },
      reason: { type: "string" }
    },
    required: %w[
      scene_type
      suitable_room_photo
      room_evidence
      coffee_machine
      coffee_machine_evidence
      confidence
      reason
    ],
    additionalProperties: false
  }
}.freeze

ROOM_SECTION_NAMES = [
  "rooms",
  "room",
  "guest rooms",
  "guest room",
  "suites",
  "suite",
  "accommodation"
].freeze

# ------------------------------------------------------------
# Coffee detector
# ------------------------------------------------------------

class CoffeeMachineDetector
  def self.detect(image_url)
    puts "Vision request image: #{image_url}"

    response =
      RubyLLM
        .chat(model: "gpt-4.1-mini")
        .with_instructions(PROMPT)
        .with_schema(COFFEE_MACHINE_SCHEMA)
        .ask(
          "Analyze the attached image and return the structured JSON detection result.",
          with: image_url
        )

    raw_response = response.content
    parsed_response = parse_response(raw_response)

    debug(image_url, raw_response, parsed_response)

    puts "Vision result: scene=#{parsed_response["scene_type"] || "unknown"}, " \
      "suitable_room_photo=#{parsed_response["suitable_room_photo"]}, " \
      "room_evidence=#{parsed_response["room_evidence"]}, " \
      "coffee_machine=#{parsed_response["coffee_machine"]}, " \
      "coffee_machine_evidence=#{parsed_response["coffee_machine_evidence"]}, " \
      "confidence=#{parsed_response["confidence"]}, " \
      "reason=#{parsed_response["reason"]}"

    parsed_response
  rescue JSON::ParserError => e
    puts "JSON parse failed: #{e.message}"
    debug(image_url, raw_response, nil, error: e)
    nil
  rescue => e
    puts "Vision request failed: #{e.class}: #{e.message}"
    debug(image_url, defined?(raw_response) ? raw_response : nil, nil, error: e)
    nil
  end

  def self.parse_response(response_content)
    parsed_response =
      case response_content
      when Hash
        response_content
      else
        JSON.parse(response_content.to_s)
      end

    stringify_keys(parsed_response)
  end

  def self.stringify_keys(value)
    case value
    when Hash
      value.each_with_object({}) do |(key, item), result|
        result[key.to_s] = stringify_keys(item)
      end
    when Array
      value.map { |item| stringify_keys(item) }
    else
      value
    end
  end

  def self.debug(image_url, raw_response, parsed_response, error: nil)
    return unless VISION_DEBUG

    FileUtils.mkdir_p(VISION_DEBUG_DIR)

    payload = {
      image_url: image_url,
      raw_response: raw_response,
      parsed_response: parsed_response,
      error_class: error&.class&.name,
      error_message: error&.message
    }

    debug_path = File.join(VISION_DEBUG_DIR, "vision_response_#{next_debug_index}.json")
    debug_json = JSON.pretty_generate(payload)

    File.write(debug_path, debug_json)
    puts "Vision debug saved to #{debug_path}"
    puts debug_json if VISION_DEBUG_STDOUT
  end

  def self.next_debug_index
    @debug_index ||= existing_debug_indices.max.to_i
    @debug_index += 1
  end

  def self.existing_debug_indices
    Dir.glob(File.join(VISION_DEBUG_DIR, "vision_response_*.json")).filter_map do |path|
      File.basename(path).match(/vision_response_(\d+)\.json/)&.[](1)&.to_i
    end
  end
end

# ------------------------------------------------------------
# Google Hotels analyzer
# ------------------------------------------------------------

class HotelCoffeeAnalyzer
  def self.analyze(api_key:, **params)
    new(api_key: api_key).analyze(**params)
  end

  def initialize(api_key:)
    @client = SerpApi::Client.new(api_key: api_key)
  end

  def analyze(
    destination:,
    country:,
    check_in_date:,
    check_out_date:,
    adults:,
    hotel_class:,
    result_limit:,
    search_pages:,
    photo_limit:,
    fetch_hotel_photos:,
    serpapi_debug:,
    serpapi_debug_stdout:,
    results_path: "coffee_machine_results.json"
  )
    result_limit = Integer(result_limit)
    search_pages = Integer(search_pages)
    photo_limit = Integer(photo_limit)

    validate_analysis_params(
      check_in_date: check_in_date,
      check_out_date: check_out_date,
      result_limit: result_limit,
      search_pages: search_pages,
      photo_limit: photo_limit
    )

    hotels =
      fetch_star_hotels(
        destination: destination,
        country: country,
        check_in_date: check_in_date,
        check_out_date: check_out_date,
        adults: adults,
        hotel_class: hotel_class,
        result_limit: result_limit,
        search_pages: search_pages,
        serpapi_debug: serpapi_debug,
        serpapi_debug_stdout: serpapi_debug_stdout
      )

    puts
    puts "Found #{hotels.size} hotel candidate(s) for #{hotel_class}-star search"
    puts

    results =
      hotels.map do |hotel|
        analyze_hotel(
          hotel,
          photo_limit: photo_limit,
          fetch_hotel_photos: fetch_hotel_photos
        )
      end

    puts
    puts "=" * 80
    puts "FINAL RESULTS"
    puts "=" * 80

    results_json = JSON.pretty_generate(results)
    puts results_json

    File.write(results_path, results_json)

    puts
    puts "Saved to #{results_path}"

    results
  end

  private

  def validate_analysis_params(check_in_date:, check_out_date:, result_limit:, search_pages:, photo_limit:)
    if Date.iso8601(check_out_date) <= Date.iso8601(check_in_date)
      raise ArgumentError, "check_out_date must be after check_in_date"
    end

    raise ArgumentError, "result_limit must be greater than or equal to 0" if result_limit.negative?
    raise ArgumentError, "search_pages must be greater than or equal to 0" if search_pages.negative?
    raise ArgumentError, "photo_limit must be greater than or equal to 0" if photo_limit.negative?
  end

  def fetch_star_hotels(
    destination:,
    country:,
    check_in_date:,
    check_out_date:,
    adults:,
    hotel_class:,
    result_limit:,
    search_pages:,
    serpapi_debug:,
    serpapi_debug_stdout:
  )
    puts "Fetching #{destination} #{hotel_class}-star hotels from #{check_in_date} to #{check_out_date}..."

    return [] if result_limit.zero?

    params = {
      engine: "google_hotels",
      q: "#{destination} hotels",
      check_in_date: check_in_date,
      check_out_date: check_out_date,
      adults: adults,
      hotel_class: hotel_class,
      gl: country,
      hl: "en"
    }

    hotels = []

    search_pages.times do |page|
      results = @client.search(params)
      debug_serpapi_response(
        results,
        page,
        serpapi_debug: serpapi_debug,
        serpapi_debug_stdout: serpapi_debug_stdout
      )

      matching_hotels = extract_hotels(results, hotel_class: hotel_class)

      puts "Page #{page + 1}: properties: #{values_array(results, :properties).size}, ads: #{values_array(results, :ads).size}, non-matching: #{values_array(results, :non_matching_properties).size}, extracted hotels: #{matching_hotels.size}"

      hotels.concat(matching_hotels)
      hotels = deduplicate_hotels(hotels)

      break if hotels.size >= result_limit

      next_page_token =
        dig_value(results, :serpapi_pagination, :next_page_token) ||
        value(results, :next_page_token)

      break if next_page_token.to_s.empty?

      params = params.merge(next_page_token: next_page_token)
    end

    selected_hotels = hotels.first(result_limit)

    puts "Collected #{hotels.size} hotel candidate(s); selected #{selected_hotels.size}."

    selected_hotels
  end

  def deduplicate_hotels(hotels)
    hotels.uniq do |hotel|
      value(hotel, :property_token) || value(hotel, :name)
    end
  end

  def extract_hotels(results, hotel_class:)
    properties = values_array(results, :properties)
    ads = values_array(results, :ads)
    non_matching_properties = values_array(results, :non_matching_properties)

    property_hotels = properties.select { |hotel| value(hotel, :type).to_s == "hotel" }
    exact_property_hotels = property_hotels.select { |hotel| matching_star_hotel?(hotel, hotel_class: hotel_class) }

    primary_hotels = exact_property_hotels.empty? ? property_hotels : exact_property_hotels
    ad_hotels = ads.select { |hotel| matching_star_hotel?(hotel, hotel_class: hotel_class) }
    fallback_hotels = non_matching_properties.select { |hotel| value(hotel, :type).to_s == "hotel" }

    deduplicate_hotels(primary_hotels + ad_hotels + fallback_hotels)
  end

  def value(hash, key)
    return nil unless hash.respond_to?(:key?)

    string_key = key.to_s
    symbol_key = key.to_sym

    if hash.key?(symbol_key)
      hash[symbol_key]
    elsif hash.key?(string_key)
      hash[string_key]
    end
  end

  def values_array(hash, key)
    Array(value(hash, key))
  end

  def dig_value(hash, *keys)
    keys.reduce(hash) do |current, key|
      value(current, key)
    end
  end

  def debug_serpapi_response(results, page, serpapi_debug:, serpapi_debug_stdout:)
    return unless serpapi_debug

    debug_path = "serpapi_response_page_#{page + 1}.json"
    debug_json = JSON.pretty_generate(results)

    File.write(debug_path, debug_json)
    puts "Full SerpApi response saved to #{debug_path}"
    puts debug_json if serpapi_debug_stdout
  end

  def matching_star_hotel?(hotel, hotel_class:)
    value(hotel, :extracted_hotel_class).to_i == hotel_class.to_i ||
      value(hotel, :hotel_class).to_i == hotel_class.to_i ||
      value(hotel, :hotel_class).to_s.match?(/\b#{Regexp.escape(hotel_class.to_s)}[-]?star\b/i)
  end

  def analyze_hotel(hotel, photo_limit:, fetch_hotel_photos:)
    puts
    puts "-" * 80
    puts value(hotel, :name)
    puts "-" * 80

    photos = fetch_hotel_photos(hotel, photo_limit: photo_limit, fetch_hotel_photos: fetch_hotel_photos)

    puts "Photos selected for analysis: #{photos.size}"

    detections = []
    photos_scanned = 0

    photos.each do |photo_url|
      puts "Analyzing #{photo_url}"
      photos_scanned += 1

      detection = CoffeeMachineDetector.detect(photo_url)
      next if detection.nil?

      detection = detection.merge("image_url" => photo_url)
      detections << detection

      if positive_detection?(detection)
        puts "Coffee machine found with sufficient confidence; stopping photo analysis for this hotel."
        break
      end
    end

    positives = detections.select { |d| positive_detection?(d) }

    {
      hotel: value(hotel, :name),
      coffee_machine_present: positives.any?,
      highest_confidence: positives.map { |d| d["confidence"] }.max,
      positive_photos: positives.count,
      photos_scanned: photos_scanned,
      detections: detections,
      evidence: positives.first(3)
    }
  rescue => e
    {
      hotel: value(hotel, :name),
      error: e.message
    }
  end

  def positive_detection?(detection)
    detection["coffee_machine"] == true &&
      detection["suitable_room_photo"] == true &&
      suitable_scene_type?(detection["scene_type"]) &&
      detection["room_evidence"].to_s.strip != "" &&
      detection["coffee_machine_evidence"].to_s.strip != "" &&
      detection["confidence"].to_f >= 0.80
  end

  def suitable_scene_type?(scene_type)
    %w[guest_room kitchenette suite].include?(scene_type.to_s)
  end

  def fetch_hotel_photos(hotel, photo_limit:, fetch_hotel_photos:)
    photos = []

    if fetch_hotel_photos
      photos.concat(fetch_room_section_photos(value(hotel, :property_token)))
    end

    photos.concat(image_urls_from(value(hotel, :images)))
    photos.uniq.first(photo_limit)
  end

  def fetch_room_section_photos(property_token)
    return [] if property_token.nil?

    result =
      @client.search(
        engine: "google_hotels_photos",
        property_token: property_token
      )

    sections = values_array(result, :sections)

    room_sections =
      sections.select do |section|
        title = value(section, :title).to_s.downcase

        ROOM_SECTION_NAMES.any? do |name|
          title.include?(name)
        end
      end

    room_sections.flat_map do |section|
      image_urls_from(value(section, :images))
    end
  end

  def image_urls_from(images)
    Array(images).filter_map do |image|
      value(image, :original_image) ||
        value(image, :image) ||
        value(image, :thumbnail)
    end
  end
end

HotelCoffeeAnalyzer.analyze(
  api_key: SERPAPI_API_KEY,
  destination: HOTEL_DESTINATION,
  country: HOTEL_COUNTRY,
  check_in_date: CHECK_IN_DATE,
  check_out_date: CHECK_OUT_DATE,
  adults: HOTEL_ADULTS,
  hotel_class: HOTEL_CLASS,
  result_limit: HOTEL_RESULT_LIMIT,
  search_pages: HOTEL_SEARCH_PAGES,
  photo_limit: HOTEL_PHOTO_LIMIT,
  fetch_hotel_photos: FETCH_HOTEL_PHOTOS,
  serpapi_debug: SERPAPI_DEBUG,
  serpapi_debug_stdout: SERPAPI_DEBUG_STDOUT
)

The entry point is calling HotelCoffeeAnalyzer.analyze() with all the possible parameters. We validate them, run hotel and image searches with some deduplication and pass each photo for image detection. The script also comes with some debugging that comes handy when things go wrong.

Analyzing the results

An unsuccessful find will come with coffee_machine_present as false. We'll get the number of positive photos which will be 0 and the number of photos we considered as photos_scanned. We'll also get an array explaining each analyzed photo with its scene, confidence, and reasoning:

{
    "hotel": "Hotel Praga 1",
    "coffee_machine_present": false,
    "highest_confidence": null,
    "positive_photos": 0,
    "photos_scanned": 1,
    "detections": [
      {
        "scene_type": "restaurant",
        "suitable_room_photo": false,
        "room_evidence": "No private room elements like bed or kitchenette visible",
        "coffee_machine": false,
        "coffee_machine_evidence": "No coffee machine is visible in the image, only cups of coffee and other breakfast items",
        "confidence": 1,
        "reason": "Image shows a breakfast table with food and drinks but no coffee machine is visible; setting is a restaurant or breakfast room, not a guest room",
        "image_url": "https://lh3.googleusercontent.com/gps-cs-s/APNQkAFIsweeBnY-BfjyDnLLCnjjecnZw33k40xzQ9--UurxZMSiYNTCAN3_FzCJPSGkCmja9ANOSbPjipi2OgCiih_lv7Y1S-VO4CU098hhiToUQ8r6rQNs87WxJKelO9_r6BPw-IEKe9RGXKgC=s10000"
      }
    ],
    "evidence": []
  },

Here we only got one picture from the restaurant which we have to disqualify.

A successful find will come with coffee_machine_present as true as well as detections and evidence entries which will have scene_type set to guest_room (since we aren't interested in shared areas) and coffee_machine as true with high enough confidence score:

{
    "hotel": "Occidental Praha",
    "coffee_machine_present": true,
    "highest_confidence": 0.95,
    "positive_photos": 1,
    "photos_scanned": 1,
    "detections": [
      {
        "scene_type": "guest_room",
        "suitable_room_photo": true,
        "room_evidence": "Bed with pillows and bedside table visible",
        "coffee_machine": true,
        "coffee_machine_evidence": "Purple pod coffee maker visible on the desk",
        "confidence": 0.95,
        "reason": "A purple pod coffee maker is clearly visible on the desk in a private guest room with a bed",
        "image_url": "https://lh3.googleusercontent.com/gps-cs-s/APNQkAEjlZ7oYYUjR1y-VAYYPz0VuftzxVS4aeqFN1hKphaUgs8Up5nT83-km8ksQgR27Q0-ywkr9IKQYgzmVozuSDe76keTb8dw91SzrrmtgsdmiqlDTUo9Re-TI7s6kp9XeWFLufZZ=s10000"
      }
    ],
    "evidence": [
      {
        "scene_type": "guest_room",
        "suitable_room_photo": true,
        "room_evidence": "Bed with pillows and bedside table visible",
        "coffee_machine": true,
        "coffee_machine_evidence": "Purple pod coffee maker visible on the desk",
        "confidence": 0.95,
        "reason": "A purple pod coffee maker is clearly visible on the desk in a private guest room with a bed",
        "image_url": "https://lh3.googleusercontent.com/gps-cs-s/APNQkAEjlZ7oYYUjR1y-VAYYPz0VuftzxVS4aeqFN1hKphaUgs8Up5nT83-km8ksQgR27Q0-ywkr9IKQYgzmVozuSDe76keTb8dw91SzrrmtgsdmiqlDTUo9Re-TI7s6kp9XeWFLufZZ=s10000"
      }
    ]
  }

We'll also get a text-based reasoning as reason. The analyzed image itself is referenced in image_url and so it's easy to be opened in a browser and rechecked manually. In this case, we can confirm we have a match:

Conclusion

SerpApi's Google Hotels and Google Hotels Images APIs are a great way to find property images to analyze. They let you find the hotels, room availability, and their images for your applications with a simple API call. Today we went through analyzing hotel images to hunt for coffee machines, but thanks to RubyLLM and OpenAI models you can easily categorize hotels based on any visual metric.

Do more with hotel APIs

Analyzing images isn't the only thing you can do with SerpApi. Have a look at our other posts on Google Hotels APIs for inspiration on what you can build:

How to find your next product idea with Google Trends and SerpApi

Josef Strzibny — Tue, 02 Jun 2026 08:20:38 +0000

Lot of product ideas fail due to bad timing, not execution. A product that would have been a hit two years ago could be just a commodity now. Similarly, hardly anybody knows for sure what will sell well next year. But with Google Trends we can find what people are searching for right now and make an educated guess for picking the right product to market. By using SerpApi's API for Google Trends we can even completely automate it.

What even is Google Trends?

Google Trends is a free tool from Google that shows how search interest changes over time. It does not tell you exact search volume like an SEO tool would. Instead, it normalizes search activity on a 0–100 scale so you can compare whether interest in a topic is rising, falling, seasonal, or suddenly spiking. That kind of insight makes it especially useful for product research. You are not trying to prove that a keyword has exactly 18,000 searches per month, but you want to notice when attention when a new trend is forming.

Trending now vs historic data

Google Trends has two distinct modes that we need to understand. We'll actually use both of them, just for a different purpose. The first is the historical interest graph called Explore where you type a term and see its relative search volume over time. That's useful for validating an existing idea, but not as useful if you don't know what to validate in the first place. The second view is called Trending now and shows search queries that have seen a dramatic spike in the last 24 hours to a week for a given category.

The idea here is to spot trends on the Trending now page first, like me seeing a spike of interest in ice hockey match of Czech Republic against Canada since the country is participating in Ice Hockey World Championships and eurojackpot because someone scored the biggest win in the history of this lottery recently (lucky guy!):

Ideally we want to see something that we could sell. The search would indicate that we should be selling merchandise related to Czechs playing against Canadians as both countries are very proud of their teams and successes in the past. To know if it's worth our effort we might want to cross check this search term with historical data on the Explore page.

This page clearly shows that a spike in the past happened only once in 2022. It's probably too niche of a thing for merchandise unless we expand this to the whole championship. But it could still be a very good idea for a YouTube video that covers some details about the match.

💡 Tip: You can still do a lot of product research with Google Search too.

Finding good product ideas

Now let's return to Trending now and try to be a bit more specific. We'll choose United States as a country with a bigger potential and Technology as our category. Here's what the Trending now page showed me:

Some obvious picks from the page are fitbit air and iphone 18. Those are exact product names, except for iphone 18 not being released yet. People also search for that iPhone with other terms like latest iphone rumors. When I ask on the Explore page for the fitbit air searches of past year I see this:

Seems like we clearly see something that it is now becoming trendy. Something we might consider selling as a product. If yes, we can also cross check the data with marketplaces using SerpApi's Google Shopping API or Amazon Search API.

This kind of opportunity search can be time consuming, though. So why not automate it? What if we just want to see a nice report every day with suggestions for what's worth digging deeper?

Finding trends with SerpApi

SerpApi's Google Trends API and Google Trends Trending Now API wrap Google Trends search results into clean JSON you can built upon. These APIs let you query the current trending searches programmatically without manual scraping, browser automation or rate limits.

An example HTTP request for a trending queries looks like this:

GET https://serpapi.com/search.json
  ?engine=google_trends_trending_now
  &geo=US
  &hours=168
  &api_key=YOUR_API_KEY

The key parameters are:

geo: the country code (US, GB, DE, etc.). Trending searches vary dramatically by market.
hours: the lookback window: 4, 24, 48, or 168 (one week). Using the maximum gives you a wider pool of emerging trends to rank by volume.
category_id: category like Games or Health (in form of their ID).

Here's a simplified shape of what you get back:

{
  "trending_searches": [
    {
      "query": "cordless leaf blower",
      "search_volume": "200K+",
      "increase_percentage": 450,
      "trend_breakdown": [
        "best cordless leaf blower",
        "ryobi leaf blower",
        "battery powered leaf blower",
        "cheap cordless leaf blower"
      ]
    }
  ]
}

Note that search_volume is a rough bucket (50K+, 200K+, 500K+), not an exact number. The increase_percentage field tells you how sharp the spike is. The trend_breakdown field shows the full cloud of queries surrounding the trend which often reveals buying intent that the headline query alone does not.

The final JSON you'll get is quite a bit longer, though. You can always visit our API playground to see the what kind of JSON response you'll get, the full list of supported parameters and the correct input like the right category ID:

Automating Google Trends insights

Now let's put everything together with some code. I'll use Ruby, but SerpApi also comes with official SDKs for Python, JavaScript, and many more languages. You can now also add SerpApi MCP to Claude Code and let most of the code be written for you with AI agents.

The script below takes the full week of trending searches, sorts by volume to find the biggest spikes, then enriches each result with two additional checks to see if interest is historically growing and whether the trend has appeared in other markets like GB, AU, or CA. We collect the data first, then printed them as structured output a short summary for each keyword:

#!/usr/bin/env ruby
# Usage: gem install serpapi
# SERPAPI_KEY=your_key ruby rails_ai_agents/trending_products.rb
require "serpapi"

API_KEY = ENV["SERPAPI_KEY"] || abort("Set SERPAPI_KEY environment variable")
LOCATION = "US"
OTHER_LOCATIONS = %w[GB AU CA]

# Maximum supported by Trending Now
TRENDING_HOURS = 168

BUYER_INTENT_WORDS = %w[best buy review cheap deal alternative better simpler cheaper]
REVIEW_SOURCES = %w[wirecutter tomsguide goodhousekeeping consumerreports
                        cnet techradar rtings thewirecutter nytimes forbes]

@serpapi_client = SerpApi::Client.new(api_key: API_KEY)

# SerpApi call for trending now
def fetch_trending(geo:, hours:)
  @serpapi_client.search(
    engine: "google_trends_trending_now",
    geo: geo,
    hours: hours,
    symbolize_names: false
  )
end

# SerpApi call for historical trends
def fetch_historical(query, geo:)
  @serpapi_client.search(
    engine: "google_trends",
    q: query,
    geo: geo,
    date: "today 3-m",
    data_type: "TIMESERIES",
    symbolize_names: false
  )
end

def parse_volume(str)
  return 0 if str.nil?
  multiplier = str.to_s.include?("M") ? 1_000_000 : 1_000
  str.to_f * multiplier
end

def growth_direction(query, geo:)
  result = fetch_historical(query, geo: geo)
  points = result.dig("interest_over_time", "timeline_data") || []
  return :unknown if points.size < 6

  values = points.map { |p| p.dig("values", 0, "extracted_value").to_i }
  third = values.size / 3
  first_avg = values.first(third).sum.to_f / third
  last_avg = values.last(third).sum.to_f / third

  if last_avg > first_avg * 1.1
    :growing
  elsif last_avg < first_avg * 0.9
    :declining
  else
    :stable
  end
end

def markets_trending_in(query)
  OTHER_LOCATIONS.select do |geo|
    result = fetch_trending(geo: geo, hours: TRENDING_HOURS)
    (result["trending_searches"] || []).any? { |t| t["query"].downcase == query.downcase }
  end
end

def buyer_intent?(text)
  lower = text.downcase
  BUYER_INTENT_WORDS.any? { |word| lower.include?(word) }
end

def review_source?(source)
  lower = source.to_s.downcase.gsub(/\s+/, "")
  REVIEW_SOURCES.any? { |s| lower.include?(s) }
end

def intent_signals(trend)
  breakdown = trend["trend_breakdown"] || []
  signals = []
  signals << :query if buyer_intent?(trend["query"])
  signals << :breakdown if breakdown.any? { |q| buyer_intent?(q) }
  signals
end

def summarize(info)
  parts = []

  case info[:growth]
  when :growing then parts << "Interest has been growing steadily over the past 3 months"
  when :declining then parts << "Interest has been declining over the past 3 months"
  when :stable then parts << "Interest has been relatively stable over the past 3 months"
  when :unknown then parts << "Not enough historical data to judge the trend"
  end

  if info[:markets].any?
    parts << "also trending in #{info[:markets].join(", ")}"
  else
    parts << "not yet picked up in other tracked markets"
  end

  if info[:intent_signals].any?
    signal_text = {
      query: "the query itself contains buyer-intent language",
      breakdown: "related searches include buying intent"
    }
    matched = info[:intent_signals].map { |s| signal_text[s] }.join(", ")
    parts << matched
  else
    parts << "no direct buyer-intent signals detected — worth manual review"
  end

  parts.join("; ") + "."
end

puts "Fetching top 10 trends in #{LOCATION} over the last #{TRENDING_HOURS}h...\n"

result = fetch_trending(geo: LOCATION, hours: TRENDING_HOURS)
top_ten = (result["trending_searches"] || [])
            .sort_by { |t| -parse_volume(t["search_volume"]) }
            .first(10)

collected = top_ten.map do |trend|
  query = trend["query"]
  growth = growth_direction(query, geo: LOCATION)
  markets = growth == :growing ? markets_trending_in(query) : []
  signals = intent_signals(trend)

  {
    query: query,
    search_volume: trend["search_volume"],
    increase_pct: trend["increase_percentage"],
    breakdown: trend["trend_breakdown"] || [],
    growth: growth,
    markets: markets,
    intent_signals: signals
  }
end

collected.each do |info|
  puts "QUERY: #{info[:query]}"
  puts "VOLUME: #{info[:search_volume]} (+#{info[:increase_pct]}%)"
  puts "GROWTH: #{info[:growth]}"
  puts "MARKETS: #{info[:markets].any? ? info[:markets].join(", ") : "none"}"
  puts "SIGNALS: #{info[:intent_signals].any? ? info[:intent_signals].join(", ") : "none"}"
  puts "BREAKDOWN: #{info[:breakdown].first(5).join(", ")}" if info[:breakdown].any?
  puts "SUMMARY: #{summarize(info)}"
  puts
end

The script starts by loading the official SerpApi Ruby client and expects a SERPAPI_KEY environment variable which will hold the API key you can get from serpapi.com/manage-api-key page:

require "serpapi"
API_KEY = ENV["SERPAPI_KEY"] || abort("Set SERPAPI_KEY environment variable")
@serpapi_client = SerpApi::Client.new(api_key: API_KEY)

They are two methods using SerpApi, fetch_trending(...) and fetch_historical(...) which we use to fetch the data themselves:

def fetch_trending(geo:, hours:)
  @serpapi_client.search(
    engine: "google_trends_trending_now",
    geo: geo,
    hours: hours,
    symbolize_names: false
  )
end

def fetch_historical(query, geo:)
  @serpapi_client.search(
    engine: "google_trends",
    q: query,
    geo: geo,
    date: "today 3-m",
    data_type: "TIMESERIES",
    symbolize_names: false
  )
end

The beginning of the script looks at the results from Google Trends “Trending Now” page and tries to understand if the trend is worth checking out further. If the trend has a positive growh trajectory we also check other relevant markets. At last we are trying to understand buyer intent. We check for keywords like best, buy, review, deal, and alternative.

Once everything is processed we print a short summary with the relevant details:

the search query
search volume
increase percentage
growth direction
other markets where the query is trending
buyer-intent signals
related searches
a generated summary

Understanding trends' insights

A result with a high volume spike, a growing historical curve, signals in other markets, and buyer-intent words in the breakdown is the strongest combination you can find. Most results will have only one or two of those which is expected.

Here is what the output looks like for a product opportunity:

QUERY: cordless leaf blower
VOLUME: 200K+ (+450%)
GROWTH: growing
MARKETS: GB, CA
SIGNALS: breakdown
BREAKDOWN: best cordless leaf blower, ryobi leaf blower, cheap cordless leaf blower, battery leaf blower, cordless leaf blower review
SUMMARY: Interest has been growing steadily over the past 3 months; also trending in GB, CA; related searches include buying intent.

And here is what a non-opportunity looks like; a celebrity death that drives enormous search volume but has no product angle:

QUERY: kyle busch
VOLUME: 500K+ (+9200%)
GROWTH: unknown
MARKETS: none
SIGNALS: none
BREAKDOWN: kyle busch cause of death, how did kyle busch die, sepsis, kurt busch, nascar...
SUMMARY: Not enough historical data to judge the trend; not yet picked up in other tracked markets; no direct buyer-intent signals detected — worth manual review.

The breakdown tells you everything. A wall of "how did X die" queries is immediately recognizable as a news event, not a product category.

Conclusion

Google Trends comes with two ways of seeing data. The Trending now page shows trend spikes in the moment and the Explore page let us dive deeper into past searches. Using them together allows us to see the full picture behind a trend.

SerpApi provides programmatic access to this kind of search data and let us scrape Google Trends with a simple API call. We saw how we can find interesting new products with just a few lines of code. You can also find this code on GitHub.

How to setup your first Ubuntu VPS for self hosting

Josef Strzibny — Mon, 11 May 2026 08:33:20 +0000

Running your own server on the internet looks challenging, especially if you have never done that. It comes with security risks, performance considerations, and other challenges. It's not surprising that most people therefore opt for hosted solutions or managed platforms to run their applications without thinking too much about the underlying infrastructure.

So what does it take to start self hosting on your own with a Virtual Private Server (VPS)? This guide will walk you through a simple Ubuntu 26.04 LTS setup you can start with. We'll look on the most important considerations when it comes to server configuration whether you are going to deploy Docker Compose, Kamal, Coolify, ONCE, or anything else you'll run.

Server configuration software

There are a few common ways to set up a server. The right approach will depend on your needs. One of the biggest concerns is how many machines are you going to configure and how often.

The most straightforward way to configure a server is to SSH into the server and configure it by hand. That's fine for learning, and it's often how people start. The downside is that manual work is easy to forget and hard to reproduce. Three months later you may not remember which packages you installed, which config files you changed, or why one server behaves differently from another.

You might have seen sysadmins use one of the many configuration management tools. Some of the most famous ones are:

Ansible — an agentless tool that connects over SSH and uses YAML playbooks to describe what should be installed and configured.
Chef — a Ruby-based system built around recipes and cookbooks, usually with an agent running on each managed server.
Puppet — another long-running configuration management tool, built around a declarative language for describing system state.
SaltStack — a flexible tool that can run with or without agents and is known for managing large fleets quickly.

These tools are excellent when you manage many servers and need consistent, repeatable state everywhere. They help with idempotency, drift detection, orchestration, and auditing. But for a first VPS, they can be more complexity than you need.

It's good to know what you are doing before adopting a higher level tool to automate it. And as we'll see soon enough, you can start with relatively small number of changes that don't even warrant using a complex tool.

So we'll use basic Bash commands, but stack them together in a shell script. A script is easy to read, easy to run, and requires no extra infrastructure. Even better, VPS providers offer you to kickstart your server with something they often call user data and which is technically cloud-init underneath.

Server provisioning with cloud-init

Cloud-init is the standard way to configure a cloud server on first boot. When you create a VPS at providers like DigitalOcean, Hetzner, Linode, or most larger clouds, you usually get a field for user data. Whatever you put there is passed to the new instance and executed during startup.

Cloud-init can process YAML configuration, but it can also run a regular shell script. For this guide, a shell script is the easiest format as you can read it from top to bottom, paste it into a provider's user-data field, or run it manually as root on a fresh server. It's good to design it for idempotency, though, so it can be rerun later.

Main considerations

Let's talk about the main things to consider when starting out as the number of configuration options on a Linux box is limitless. The most important set of changes are security-oriented:

Disabling password-based authentication to protect unwanted access to the server, ideally paired with limiting access from private network.
Protecting SSH port from brute-force attacks to slow down repeated SSH login attempts.
Setting up automatic security updates so important patches are applied without you logging in.
Configuring firewall to expose only the ports you need on for the networks you trust, ideally paired with cloud (provider's) firewall.

Optionally, you can set up a non-root user for day-to-day administration, disable root SSH login altogether, use SSH keys with passwords , or allow only SSH private network connections in your firewall.

If you are willing to use a 3rd-party service, Tailscale is a popular choice today to secure the access to your servers. Tailscale creates a mesh network between your computer and your servers for direct communication letting you close public access to your SSH port.

Other non-security related changes you might consider:

Adding Swap space to make small VPS plans more forgiving under memory pressure.
Installing Docker and other essential software and utilities.

Remember there is way more you tune from more advance SSH configuration to AppArmor or file descriptors. But this is a good start.

Setup script

We'll need to translate these considerations into a reusable script we can pass as user data during the VPS setup. To start we will prepend it with shebang pointing to env bash which is a how to run the right Bash binary (remember that Ubuntu doesn't use Bash as default for users, but we'll use Bash as it's the most common shell). We'll follow it up with running the set command to set certain modes to run the script with.

The script starts would then look like this:

#!/usr/bin/env bash
set -euo pipefail

The set -euo pipefail line makes the script more strict:

set -e stops the script when a command fails.
set -u stops the script when it references an unset variable.
set -o pipefail makes a pipeline fail if any command in it fails.

It is better to stop early than to continue with a half-configured server. Then before running any specific commands, we'll update the system to latest packages:

apt update;
DEBIAN_FRONTEND=noninteractive apt upgrade -y

The apt update step matters on a fresh cloud image because package indexes can be stale. Without it, package installation may fail even though the package exists.

Your first server configuration

Now let's go through the actual configuration we are going to run in more detail.

SSH Keys

Secure Shell (SSH) is like regular shell on your system, but used to access other systems on the network as if you are the local user. Similarly to a local login, you need to authenticate when accessing remote server. They are two ways to authenticate with SSH; by using passwords or SSH keys.

The password-based authentication is not the secure option today, so the first step for a new box would be to set up password-less authentication using SSH keys. This is usually very simple with modern VPS providers as they offer this option in their provisioning setup when creating a new server.

If you don't have any keys yet, run the following ssh-keygen command and follow its instructions (choosing a password is better but some system admins prefer password-less keys):

ssh-keygen -t ed25519 -C “admin@example.com"

In the end you should end up with two keys. The public key will end with .pub and is the one you'll upload on the server. The private one should always stay safe with you and shouldn't be shared.

To completely disable password authentication on the system we need to set PasswordAuthentication to no. This can be done by changing the /etc/ssh/sshd_config config file directly or by appending a file into /etc/ssh/sshd_config.d/.

The following disables password logins entirely:

cat > /etc/ssh/sshd_config.d/91-require-ssh-keys.conf <<'EOF'
PasswordAuthentication no
EOF
sshd -t
systemctl restart ssh

The sshd -t command validates the SSH configuration before restarting the service. If you make a typo, the validation fails and you wan't end up with broken configuration.

fail2ban

Using SSH keys instead of passwords is likely the most important thing to do for a new Linux box, but since SSH ports are often a target, it make sense to monitor who's trying to log in and ban them from trying again.

fail2ban watches authentication logs and temporarily bans IP addresses after too many failed attempts. On modern Ubuntu systems, fail2ban should read SSH logs from journald. What I like the most, is the setup. By just installing the package and running the service, you already get a decent default configuration.

apt install fail2ban
systemctl enable fail2ban.service
systemctl restart fail2ban.service

If we want to make changes to the configuration we can do so by editing the fail2ban's sshd.local file:

# /etc/fail2ban/jail.d/sshd.local

[sshd]
enabled = true
backend = systemd
maxretry = 5
findtime = 10m
bantime = 10m

The enabled = true line makes the SSH jail explicit instead of relying on distribution defaults. This configuration bans an IP for ten minutes after five failed attempts within ten minutes. That is a reasonable starting point for a first server.

💡 Remember that (re)starting matters since enabling service will only start it on next reboot.

Automatic Updates

Running package and software updates is a daily bread of system management. We can make our life a bit easier with automated updates to patch vulnerabilities automatically.

To enable unattended upgrades, we have to first install the unattended-upgrades package:

apt install -y unattended-upgrades

Ubuntu's unattended-upgrades package can install security updates automatically when configured to do so:

printf 'APT::Periodic::Update-Package-Lists "1";\nAPT::Periodic::Unattended-Upgrade "1";\n' > /etc/apt/apt.conf.d/20auto-upgrades
systemctl restart unattended-upgrades

The two configuration lines tell Ubuntu to refresh package lists daily and apply unattended upgrades. By default, Ubuntu limits this mainly to security updates from official repositories, so this is not the same as blindly upgrading every package to a new major version overnight.

Remember that new package versions will only be running after the process restarts and so some key systems components might need a full reboot occasionally.

You can check if you system needs rebooting by running the following reboot-required program:

/var/run/reboot-required

Schedule a reboot during a maintenance window when required.

Automatic updates do not replace long-term maintenance as you will still need to upgrade the OS before its support window ends. But they cover the dangerous gap between the day a security update is released and the next time you remember to log in to run updates by hand.

Firewall

A firewall is a security system that monitors and controls network traffic. It allows or denies connections based on predefined rules. Firewall rules are mostly about narrowing connections to allowed ports and protocols. Each cloud instance can come with its own firewall and each Linux box will also come with a system firewall.

Creating firewall rules is easier than it looks once you understand what you need to do. Cloud firewalls also often make it easy by letting you assign resources you have already without remembering their exact IP address. You should also set up a system firewall , even if less strict (it's easier to add rules in your provider's interface and not lock yourself up).

Here's an example of firewall rules for a single server Kamal deployment:

A good rule of thumb is to deny all connections by default and allow connections we expect to have. Instead of exposing every service you are running and hoping each one is safe, you deny incoming traffic by default and open only the ports you actually need.

So what do you usually need? The ports 80 and 443 are for public HTTP and HTTPS traffic. The port 22 is for SSH. Those need to be reachable from outside if you want to serve web traffic and administer the machine remotely.

Then you'll need to allow incoming connections for your services. Since they usually only need to be accessed internally, you should limit them to your private subnet (IP addresses that are accessed only from your network of servers).

Some popular ports include:

3000 for an application server
3306 for MySQL or MariaDB
5432 for PostgreSQL
6379 for Redis

If you aren't exactly sure about your private subnet address space, it's usually save to default to 10.0.0.0/16 although you should always recheck with your provider.

Cloud firewall are set directly in the provider's admin area:

The exact location depends on the provider, could be under VM settings or separately under Security or Firewalls. You can sometimes design the rules based on descriptive names instead of raw data (HTTPS instead of 443, All IPv4 addresses instead of 0.0.0.0/0).

For, Ubuntu's UFW (the system firewall) we can set the rules with ufw commands:

ufw default deny incoming
ufw default allow outgoing
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 22/tcp
ufw allow from "10.0.0.0/16" to any port 3000 proto tcp
ufw allow from "10.0.0.0/16" to any port 3306 proto tcp
ufw allow from "10.0.0.0/16" to any port 5432 proto tcp
ufw allow from "10.0.0.0/16" to any port 6379 proto tcp
ufw logging on
ufw --force enable

This is the model you usually want: public web traffic comes in through HTTP/HTTPS, while databases and internal services stay on a private network. Your database should not be directly reachable from the public internet. You need to update this to your exact setup.

UFW uses nftables under the hood on current Ubuntu releases, but the user-facing commands are the same. You can keep using simple ufw allow, ufw deny, and ufw status commands without learning nftables directly. The ufw logging on line is useful when something cannot connect and you need to see whether the firewall is blocking it.

Opening SSH to the whole internet is simple, but it is not the only option. If you have a stable home IP, you can restrict SSH to that address. If you use Tailscale, you can eventually move administration onto the private network. For a first server, global SSH plus key-only login and fail2ban is a practical starting point.

💡 Note that Docker will override UFW on Ubuntu by default, so be careful what you are exposing when running Docker-based services.

Non-root user

It's better not to use the root user for everyday work as it's a privileged user that can do anything on the system. It's also the obvious attack vector for any malicious actor stealing your SSH private key.

We can create a non-root user with sudo access which gives the user the superpowers it needs to manage everything like root, but would require another information for the attacker to know.

USER=[USERNAME]
useradd --create-home $USER
usermod -s /bin/bash $USER
su - $USER -c 'mkdir -p ~/.ssh'
su - $USER -c 'touch ~/.ssh/authorized_keys'
cat /root/.ssh/authorized_keys >> /home/$USER/.ssh/authorized_keys
chmod 700 /home/$USER/.ssh
chmod 600 /home/$USER/.ssh/authorized_keys
echo "$USER ALL=(ALL:ALL) NOPASSWD: ALL" | tee /etc/sudoers.d/$USER
chmod 0440 /etc/sudoers.d/$USER
visudo -c -f /etc/sudoers.d/$USER
usermod -aG docker $USER

This creates a home directory, sets bash as the login shell, prepares ~/.ssh, and installs an SSH key for the new user. We are recreating root as a different user in a way.

💡 Sometimes paths and locations may vary from system to system or provider to provider. You might need to replace the paths in that case.

The NOPASSWD: settings let our user to run privileged commands without any additional password. You can set a password here, but it can make it difficult for running scripts. The visudo -c check is important because it validates the sudoers file before you rely on it. A broken sudoers file can leave you unable to use sudo.

Although not completely necessary (a lot of system admins keep the root login today depending on all the other protections), adding a non-root user can make your configuration quite a bit more secure.

Disabling root access

Once your non-root user works, you can close the login access for root. Similarly to previous SSH settings we can either edit /etc/ssh/sshd_config or drop a new config file to /etc/ssh/sshd_config.d/:

cat > /etc/ssh/sshd_config.d/92-disable-root-login.conf <<'EOF'
PermitRootLogin no
EOF
sshd -t
systemctl restart ssh

The settings we need this time is PermitRootLogin no.

Swap

Small VPS plans often come with 1GB or 2GB of RAM. That may be enough most of the time, but a web server, database, background jobs, deployments, and package upgrades can occasionally push memory usage higher than expected. If you use all the available RAM, processes start to crash.

Swap space lets the Linux kernel move less-used memory pages from RAM to disk so active processes can keep running. Accessing disk space is way slower than RAM, but it can prevent an out-of-memory crash in a traffic spike.

To allocate swap space we'll use the fallocate program, then make it the swap space with mkswap and swapon. Last, we can configure the swappiness parameter in /etc/sysctl.conf:

fallocate -l 2GB /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "\\n/swapfile swap swap defaults 0 0\\n" >> /etc/fstab
sysctl vm.swappiness=20;
echo "\\nvm.swappiness=20\\n" >> /etc/sysctl.conf

The vm.swappiness=20 setting tells the kernel to prefer RAM and use swap only when needed. A value around 10 to 20 is a good starting point for a VPS. The /etc/fstab line makes the swapfile survive reboots. Without it, swap would work only until the next restart.

Docker

Now if you are going to deploy things with Docker, be it using Docker Compose, Coolify, Kamal, or anything else, you might preinstall Docker in this provisioning step as well. Some tools, like Kamal, can install Docker automatically, but it might be useful to preinstall it the way you want.

You have several options:

Ubuntu's docker.io system package
The Docker Snap package
Docker's official docker-ce packages

They differ in their update policies and Snap also differ in ways it works (which is the reason it's often not compatible with other software like Kamal).

To install it, just add another apt install command:

apt install -y docker.io

If you are using a non-root user for your operations, you might want to add it to a docker group which will let it run privileged Docker commands:

usermod -aG docker [USERNAME]

A full example

Now let's see how such a cloud-init script can look like in full with most of the options from this article:

#!/usr/bin/env bash
set -euo pipefail

# Provide these variables to use as a standalone script
USER=${USER}
NAME=${NAME}

echo "Configuring the system $NAME for user $USER"

# Do a system update
apt update;
DEBIAN_FRONTEND=noninteractive apt upgrade -y

# Install essential packages
apt install -y docker.io curl unattended-upgrades fail2ban

# Set up unattented updates
echo -e "APT::Periodic::Update-Package-Lists \"1\";\nAPT::Periodic::Unattended-Upgrade \"1\";\n" > /etc/apt/apt.conf.d/20auto-upgrades
/etc/init.d/unattended-upgrades restart

# Install fail2ban
systemctl start fail2ban.service
systemctl enable fail2ban.service

# Configure firewall
ufw logging on;
ufw default deny incoming;
ufw default allow outgoing;

ufw allow 80;
ufw allow 443;
ufw allow 22;

ufw allow from 10.0.0.0/16 to any port 3000;
ufw allow from 10.0.0.0/16 to any port 3100;
ufw allow from 10.0.0.0/16 to any port 3306;
ufw allow from 10.0.0.0/16 to any port 5432;
ufw allow from 10.0.0.0/16 to any port 6379;
ufw allow from 10.0.0.0/16 to any port 9090;
ufw allow from 10.0.0.0/16 to any port 9100;

ufw --force enable;
systemctl restart ufw.service

# Add swap space
fallocate -l 2GB /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "\\n/swapfile swap swap defaults 0 0\\n" >> /etc/fstab
sysctl vm.swappiness=20;
echo "\\nvm.swappiness=20\\n" >> /etc/sysctl.conf

# Add non-root user
useradd --create-home $USER
usermod -s /bin/bash $USER
su - $USER -c 'mkdir -p ~/.ssh'
su - $USER -c 'touch ~/.ssh/authorized_keys'
cat /root/.ssh/authorized_keys >> /home/$USER/.ssh/authorized_keys
chmod 700 /home/$USER/.ssh
chmod 600 /home/$USER/.ssh/authorized_keys
echo "$USER ALL=(ALL:ALL) NOPASSWD: ALL" | tee /etc/sudoers.d/$USER
chmod 0440 /etc/sudoers.d/$USER
visudo -c -f /etc/sudoers.d/$USER
usermod -aG docker $USER

# Disable root access
sed -i 's@PasswordAuthentication yes@PasswordAuthentication no@g' /etc/ssh/sshd_config
sed -i 's@PermitRootLogin yes@PermitRootLogin no@g' /etc/ssh/sshd_config

systemctl restart ssh.service

After your system is fully provisioned, make sure to update the cloud's firewall and optionally configure some system monitoring. Some providers come with built-in monitoring that will work automatically if you opt-in into installing their package during a setup step.

You might also want to enable full virtual server backups as most providers support them. They will snapshot the entire virtual server which can come in handy when things go wrong (note they usually cost extra).

Troubleshooting

If your custom configuration didn't work for some unknown reason and you need to debug what went wrong with your script, log in to the machine using your SSH key and have a look at cloud-init log:

sudo cat /var/log/cloud-init-output.log

They are also other locations to check.

💡 Note that it's better not to close access for root before you are confident that your configuration will work.

Next steps

Once you have your server up and running, you can continue to improve the configuration, install a PaaS like Coofily, or start self-hosting with Kamal.

Have a look at some of our other posts on self hosting to get you started with that:

How to start self-hosting with Coolify 4 on a VPS

Self-host SerpBear with Coolify

How to implement AI agents in Rails with RubyLLM

Josef Strzibny — Fri, 24 Apr 2026 07:41:13 +0000

Chat-based agents are augmented LLM interfaces with access to a list of predefined tools. RubyLLM Agents are reusable AI assistants implemented as models with their configuration, runtime context, and prompt conventions. Let's see how we can start implementing custom OpenAI chat agents with access to SERP tools with the help of the RubyLLM gem.

💡 Note the difference between fully autonomous agents like Claude Code or Codex, and chat-based agents that still react to user input. This post is about the latter.

Simple chats vs agents

RubyLLM is a Ruby gem and an AI interface for GPT, Claude, and Gemini to give us an easy way to run LLM chat inside a Ruby application. It allows us to avoid writing JSON and lets us work with AI using beautiful Ruby DSL.

A regular RubyLLM chat is a conversation. A user sends a message, the model responds, and the exchange continues back and forth. It works but it's limited to what the model can do. Today's models can do way more than before as they often search the web and find up-to-date information. However, they still do a lot of guessing and cannot access your internal data. This means we need to be very specific and provide a lot of context for the LLM to understand our request.

Imagine we ask an LLM something more complex with a simple sentence:

# Choose any model you like, just make sure to set up
# the access token in the initializator
chat = RubyLLM.chat(model: "gpt-5.4-mini")
chat.ask "Is our 'Aeropress Coffee Maker' at $39.99 competitive?"

How could this LLM model know the answer?

Perhaps the agent could search our pages as well as our competitors' to gain a full understanding of our request. This usually requires some back and forth to ensure the LLM has all the context needed to give an accurate response. If we want more precision, access to application data, and better results, we need to give the chat agent access to the tools we control.

Now imagine what would happen if the chat agent has access to a local database and Google Shopping:

chat = RubyLLM.chat(model: "gpt-5.4-mini")
chat.with_tools(LookupProduct, SearchGoogleShopping)
chat.ask "Is our 'Aeropress Coffee Maker' competitively priced?"

# => "Your Aeropress Coffee Maker is listed at $39.99. Here are current
# Google Shopping prices:
# 1. Amazon — $34.95
# 2. Target — $37.99
# 3. Walmart — $33.49
# 4. Williams Sonoma — $41.95
# 5. REI — $39.95
#
# You're on the higher end. Three out of five retailers are under $38.
# Consider adjusting to ~$36-37 to stay competitive."

Suddenly it has everything it needs to answer such an ambiguous question. It can search for competitors products on Google Shopping and compare it with data we have in our product catalog. Not bad at all.

The concept of using tools is simple. We describe a set of tools to the model, each with a name, parameters, and what it does. When the model determines it needs specific information or wants to perform an action, it returns a structured tool call

instead of plain text which looks like JSON below:

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_abc123",
      "type": "function",
      "function": {
        "name": "lookup_product",
        "arguments": "{\"name\":\"Aeropress Coffee Maker\"}"
      }
    }
  ]
}

Our code now needs to execute this function call and feed the result back which is all handled by RubyLLM in the background. The model then continues reasoning with the new information.

This loop which reasons, acts, and observes is what turns a language model into something that can actually get work done. And even better, we can wrap it all in a reusable agent class thanks to the RubyLLM Agents support. But first, let's implement the tools.

Tools

RubyLLM tools are interfaces for runnable code we control in an LLM chat. Both LookupProduct and SearchGoogleShopping tools would be implemented as Ruby classes inherited from RubyLLM::Tool. We name them using description, provide a set of acceptable params using param, and implement the execute method.

Here's an example of how this could look when searching a local product catalog for LookupProduct:

class LookupProduct < RubyLLM::Tool
  description "Looks up a product in our catalog by name"
  param :name, desc: "Product name to search for"

  def execute(name:)
     Product.where("name ILIKE ?", "%#{name}%")
            .select(:name, :price, :sku, :category)
            .map(&:attributes)
  end
end

And here's the code for SearchGoogleShopping that works with Google Shopping using SerpApi:

class SearchGoogleShopping < RubyLLM::Tool
  description "Searches Google Shopping for current market prices of a product"
  param :query, desc: "The product to search for"

  def execute(query:)
    search = SerpApi.search(q: query, engine: "google_shopping")
    search.shopping_results.first(5).map do |r|
      { title: r[:title], price: r[:price], source: r[:source] }
    end
  end
end

You can see that RubyLLM does all the heavy lifting, allowing us to write code as usual.

Agents

Once we have our tools ready, we can introduce them to the chat using the chat.with_tools call. We can also go one step further and wrap this up as a reusable class in an agent thanks to the RubyLLM::Agent interface.

An agent in this context is a class that couples instructions with a set of tools:

class PriceMonitorAgent < RubyLLM::Agent
  model "gpt-5.4-mini"

  instructions <<~PROMPT
      You are a pricing analyst. You help the merchandising team keep our product
      catalog competitively priced. You can look up our products, check current
      Google Shopping prices, and find products where we are significantly overpriced.
      Always show specific numbers when comparing prices.
    PROMPT

  tools LookupProduct, SearchGoogleShopping, FindUndercut
end

A PriceMonitorAgent like the one above could help the shop's merchandising team find products that are overpriced on the current market:

agent = PriceMonitorAgent.new
agent.ask "Which of our coffee products are priced more than 15% above market?"

# => "I found 3 coffee products above the 15% threshold:
#
# 1. Baratza Encore Grinder — Our price: $179.99, Market avg: $149.80 (20.2% over)
# 2. Fellow Stagg Kettle — Our price: $94.99, Market avg: $79.60 (19.3% over)
# 3. Chemex 6-Cup — Our price: $54.99, Market avg: $44.97 (22.3% over)
#
# The Aeropress ($39.99 vs $37.48 avg) and Hario V60 ($11.99 vs $11.20 avg)
# are within range and look fine."

Depending on the question, the agent can pick one tool to generate a response or use an additional tool from the list to find the right answer. Combining internal data with live SERP data is incredibly powerful.

Debugging

Sometimes we might not be sure if the agents are using our tools in the way we expected. Luckily, chats in RubyLLM let us see all the tool calls that were done:

chat.messages.each do |m|
  m.tool_calls&.each do |tc|
    puts "#{tc.name} #{tc.arguments.inspect}"
  end
end

If we want to specifically recheck what search queries were run using SerpApi, we can head to serpapi.com/searches after logging in, and find the searches that were done. We'll get a full Search Inspector including the returned page and JSON:

Conclusion

SerpApi makes it super easy to integrate search data from Google, Amazon, Bing, and other search engines into your application. And RubyLLM makes it easy to expose this API as a tool for your agents. Give SerpApi a try with 250 free searches/month.

Self-host SerpBear with Coolify

Josef Strzibny — Thu, 02 Apr 2026 08:47:23 +0000

Tracking organic positions on search engines is one of the main concerns of SEO. SerpBear is a SERP position tracking tool that will help you track how well you are doing in Google search. This post will take you through all the steps to start tracking your keywords with your own SerpBear instance on Coolify.

What is SerpBear?

SerpBear is an Open Source search engine position tracker. It allows you to track your or your competitors' keyword positions in Google and to also get notified of these changes. Some of the features include:

Unlimited keywords: You can add as many domains and keywords to track as you want. SerpBear will use a 3rd-party API such as SerpApi for actual tracking behind the scenes.
Flexible scraping strategy: Choose your scraping strategy to control how many Google pages are checked for each domain. It's built to handle the Google's 2025 removal of the num=100parameter.
Email notification: Get notified of all your keyword position changes via email with at the frequency of your choosing.
Keyword research: You have the option to do keyword research with the integration of Google Ads.
Google Search Console: You can see the actual visit count and impressions for each keyword. Discover new keywords thanks to direct integration with Google Search Console.
Exports: Export your domain keyword data in CSV files.

If you want to track your keywords, but don't want to use SerpBear, have a look how to do your custom SERP tracking in JavaScript.

What is Coolify?

Coolify is an Open Source Platform as a Service (PaaS) that will help you self-host SerpBear or any other self-hostable software out there. You can also deploy applications, databases, and one-click services.

I am assuming you already have Coolify installed on your server. If you don't, you can learn how to deploy Coolify on any VPS or take advantage of Coolify Cloud version where you don't have to host Coolify yourself.

💡 We are going to deploy SerpBear 3.0 on Coolify 4.0. Coolify 4.0 is still beta software for the time being.

Signing up for SerpApi

SerpApi is a service providing access to various SERP results with a simple API. SerpBear is free and open source software, but it uses SerpApi to do the heavy lifting of querying and parsing Google results. SerpBear cannot do this on its own due to the nature of managing proxies and solving CAPTCHAs, but you can still start with free searches to set everything up and see it working before committing to a paid plan.

So first of all, sign up for SerpApi and then note your private API key from serpapi.com/manage-api-key page:

We are going to use this API key from SerpBear, but you can also use it to integrating one the many SerpApi SDKs to build anything needing SERP API.

Configuring DNS

Head over to your domain registrar console and set the DNS A records for domains or subdomains you want to use for SerpBear:

coolify.example.com -> 178.104.25.112
serpbear.example.com -> 178.104.25.112

Choose "A" type records to directly connect a domain name or subdomain to the IP address (replace the values, the above is just an example). If you'll run SerpBear on the same host as Coolify, the IP address remains the same for both.

Adding SerpBear

If your Coolify instance is fully set up, it's time to add SerpBear. To run a new Coolify project head over to Projects (from the left menu) and click on + Add next to the Projects headline. Give it a name and description.

From Projects, click on + Add Resource next to your project name or select your project first and click on + New next to Resources. From here, we can select Docker Compose Empty under Docker Based on the right:

SerpBear provides Docker Compose configuration which Coolify supports. Since SerpBear uses SQLite to save SERP data, it's not necessary to spin up process-based databases and the Compose file is relatively simple:

services:
  app:
    image: towfiqi/serpbear:latest
    # Build from source instead of pulling the image:
    # build: .
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"
    environment:
      - USER_NAME=${USER:-admin}
      - PASSWORD=${PASSWORD}
      - SECRET=${SECRET}
      - APIKEY=${APIKEY}
      - SESSION_DURATION=${SESSION_DURATION:-24}
      - NEXT_PUBLIC_APP_URL=${NEXT_PUBLIC_APP_URL:-http://localhost:3000}
      # Optional: Google Search Console integration
      - SEARCH_CONSOLE_CLIENT_EMAIL=${SEARCH_CONSOLE_CLIENT_EMAIL:-}
      - SEARCH_CONSOLE_PRIVATE_KEY=${SEARCH_CONSOLE_PRIVATE_KEY:-}
    volumes:
      - serpbear_data:/app/data
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:3000 || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s

volumes:
  serpbear_data:

The SerpBear service definition includes the official Docker image, container restart policy, ports, environment variables, volumes, and health check. The official guide suggests using the latest image (towfiqi/serpbear:latest) which will let you update the application with Redeploy at any later point, but you can choose a stable release to lock-in a particular version.

The required environment variables reference variables like {USER:-admin} and {SECRET} which let us manage the environment directly from the Coolify admin interface. Notice that SerpBear also supports Google Search Console, so if you can obtain these credentials as well. Finally notice the serpbear_data volume which is the future location of your SerpBear database.

Paste the configuration above. Click Save.

Adding the new resource should take you to the resource configuration. Under General we can change the resource name, under Persistent Storages we can check our persistent volume, and finally under Environment Variables we can go fill up the variables from the Compose file:

Here's the list of environment variables we should fill in:

USER_NAME: The username you want to use to login to the app.
PASSWORD: The password you want to use to log in to the app.
SECRET: A secret key that will be used for encrypting 3rd party API keys & passwords.
APIKEY: API key that will be used to access the app's API. This is not SerpApi account key!
SESSION_DURATION: The duration(in hours) of the user's logged-in session.
NEXT_PUBLIC_APP_URL: The URL where your app is hosted and can be accessed like https://serpbear.example.com

Note that every environment entry has its own Update button and they won't be saved all at once. You can add the Google Search Console credentials if you have them, but they aren't necessary to start tracking your keywords with SerpApi.

Restart the service.

Tracking keywords with SerpBear

If you carefully followed all the steps, you should be now able to access the SerpBear instance on the (sub)domain of your choosing. Log in using the chosen credentials from the previous step. You should see a red prompt at the top to configure your Scrapper/Proxy. Click on it to open settings, choose SerpApi.com as the Scraping Method and insert your SerpApi API token under Scraper API Key Or Token:

Click Update Settings to close the right sidebar and you'll be ready to add your first keywords to track. Start by adding a domain name. After entering your domain name you should end up on the Tracking tab for the domain.

Click Add Keyword and start adding keywords and phrases you care about. They should appear in a nice table with their last position, best position, history graph, volume, and URL. Here's an example of tracking "serp api" for serpapi.com in the Czech Republic:

And that's it! You can now add as many domains and keywords as you want to. Remember that SEO today is not just about tracking organic results in Google, so have a look at AI overviews as well.

Next steps

As next steps you can consider adding your Google Search Console data to SerpBear, handle external backups in Coolify, or exploring more of SerpApi. The Free tier comes with 250 free searches per month.

How to start self-hosting with Coolify 4 on a VPS

Josef Strzibny — Wed, 01 Apr 2026 11:57:51 +0000

Coolify is an open source Platform as a Service (PaaS) that can help you self-host a lot of different software from content management tools like Ghost to SEO tracking software like SerpBear. The only thing you'll need is a virtual private server (VPS) from a hosting provider of your choice and a little bit of set up which is exactly what we'll go through in this post.

💡

Coolify v4 is still beta software as of the time of writing of this post. If you need a little bit more stability, wait for the final release.

What is Coolify?

Coolify is a self-hostable alternative to Heroku, Vercel, or Netlify. It's an open source PaaS that allows developers to deploy their applications as well as manage 3rd-party services and databases.

Some of the biggest advantages include:

Self-hosted control: You own your data on your own servers which saves money and avoids vendor lock-in.
Hosting versatility: Hosts web applications, static websites, databases (PostgreSQL, MySQL, MongoDB, etc.), and services
Automated DevOps: Automatically installs dependencies, sets up databases, and handles deployments from GitHub, GitLab, or Bitbucket.
Web console: Offers a web-based dashboard for managing multiple servers, viewing logs, and monitoring resource usage like CPU and RAM.
Built-in tools: Features automatic S3-compatible backups, auto provisioning of SSL certificates, webhooks, or preview deployments.

You can run Coolify in various ways. You can use a single server for everything you host or do a separate deployment for the services you'll run.

Applications, databases and services

Coolify supports deploying custom applications and 3rd-party services. It generally distinguish between 3 different types of resources:

Applications come with a git source or are built using Docker from Dockerfile. You can deploy anything from PHP to Java.
Databases are preconfigured Docker images for running databases, like MySQL or PostgreSQL.
Services are deployments based on Docker Compose files that are stored directly on the server. You deploy well known software like Ghost.

You can mix and match any of these on a single Coolify instance.

Setting up the virtual server

There are two variants to self-hosting. One with physical servers that are usually dedicated to you and one with virtual servers where you share the underlying physical servers with others.

We'll set up a simple virtual server on Hetzner which is a provider praised for its affordable costs. You are free to use any other provider you like or already have. Some other popular providers include Digital Ocean, Vultr, or OVHcloud.

💡

Coolify currently offers $20 credit for deploying to Hetzner by following the https://coolify.io/hetzner link.

To create a virtual private server (VPS), you'll need an SSH key pair which you'll authenticate with in the future. You can optionally set a password for the private key as well. To generate a new one, run ssh-keygen:

ssh-keygen

The command will interactively ask you about your key details. If you name it coolify you should end up with two files inside the ~/.ssh directory, one for private key and one for public key (coolify.pub).

💡

Protect your SSH private key! You always provide others with your public key, not the private key. The private key should stay with you on your computer.

Once you have your key pair, it's fairly straightforward to spin up a new server with the provider of your choice.

On Hetzner, create a new project and on the project overview click Add Server.

Select a location close to you, Ubuntu 24.04 or your favorite operating system (Debian, SUSE, and Fedora based systems are supported), the size you want, and optionally enable full virtual machine backups. Under SSH Keys upload your public SSH key which should disable password-based access on most providers and let you use your private key instead.

As for the size of the box, Coolify recommends at least 2 vCPUs, 2 GB of RAM, and 30 GB of storage space. If you are going to self-host projects on the same instance, you generally need to increase the box size accordingly. For only running one or two extra applications, consider increasing RAM.

This is all that's needed for things to work, but we can also provide a custom cloud-init script under Cloud config. For example, we can set up automatic system updates and install fail2ban for extra protection of our SSH port:

# Do a system update
apt update;
DEBIAN_FRONTEND=noninteractive apt upgrade -y

# Install essential packages
apt install -y curl unattended-upgrades fail2ban

# Set up unattented updates
echo -e "APT::Periodic::Update-Package-Lists \"1\";\nAPT::Periodic::Unattended-Upgrade \"1\";\n" > /etc/apt/apt.conf.d/20auto-upgrades
/etc/init.d/unattended-upgrades restart

# Install fail2ban
systemctl start fail2ban.service
systemctl enable fail2ban.service

You can optionally install Docker at this step from the system package manager but do not install Docker from Snap as that's not supported by Coolify.

Finally, give the server a name (you can call it coolify, but do not use a domain name) and click Create & Buy now. Provisioning the server will take some minutes and once done you should be able to find the public server IPv4 address next to its name on the server page:

You should now be able to log in from your terminal as:

ssh-add ~/.ssh/path-to-your-key
ssh root@[IP_ADDRESS]

There is quite a bit more to running and securing servers. You should ideally choose a different user than root and/or disable public SSH access altogether in the cloud's firewall. If you want to do that, have a look at WireGuard or Tailscale.

Configuring DNS

Now that you have a public IP address, you can create DNS records to get nice URLs for accessing Coolify, SerpBear, and whatever else you decide to self host.

Head over to your domain registrar console and set the DNS A records for domains or subdomains you want to use for Coolify and the applications you want to host:

# For accesing Coolify web console (admin area)
coolify.example.com -> 178.104.25.112

# Example application
serpbear.example.com -> 178.104.25.112

Choose "A" type records to directly connect a domain name or subdomain to the IP address (replace the values, the above is just an example). You can use Porkbun or GoDaddy to register your first domain name if you don't have one yet.

Since we'll run everything on a single host, the IP address remains the same for all instances.

Installing Coolify

Now that we have our Linux box up and running, we can SSH into it and run the official installer script:

curl -fsSL https://cdn.coollabs.io/coolify/install.sh | sudo bash

You can change some installation details with environment variables, so go and review this list before running the script.

For example, you might want to choose the admin username, email, and password:

env ROOT_USERNAME=admin \
ROOT_USER_EMAIL=admin@example.com \
ROOT_USER_PASSWORD=SecurePassword123 \
bash -c 'curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash'

Once run, you'll see a Congratulations! screen with the list of the IP addresses, version as well as a log file:

____ _ _ _ _ _
  / ___|___ _ ____ _ _ ____ _| |_ _ _| | ___| |_(_)___ _ _____ | |
 | | / _ \| '_ \ / _` | ' __/ _` |__ | | | | |/ _` | __| |/ _ \| '_ \/__ | |
 | |__| (_) | | | | (_| | | | (_| | |_| |_| | | (_| | |_| | (_) | | | \__ \_|
  \ ____\___ /|_| |_|\ __, |_| \__ ,_|\ __|\__ ,_|_|\ __,_|\__ |_|\ ___/|_| |_|___ (_)
                   |___/


Your instance is ready to use!

You can access Coolify through your Public IPV4: http://178.104.25.112:8000
You can access Coolify through your Public IPv6: http://[2a01:4f8:1c19:f81a::1]:8000

If your Public IP is not accessible, you can use the following Private IPs:

http://10.0.0.1:8000
http://10.0.1.1:8000
http://2a01:4f8:1c19:f81a::1:8000
http://fdb0:c330:3639::1:8000

WARNING: It is highly recommended to backup your Environment variables file (/data/coolify/source/.env) to a safe location, outside of this server (e.g. into a Password Manager).


============================================================

[2026-03-19 11:02:44] Installation Complete
============================================================
[2026-03-19 11:02:44] Coolify installation completed successfully
[2026-03-19 11:02:44] Version: 4.0.0-beta.468
[2026-03-19 11:02:44] Log file: /data/coolify/source/installation-20260319-110120.log

After installation, you should find your self-hosted Coolify instance at http://[IP_ADDRESS]:8000:

You should see a Welcome to Coolify screen. Click Let's go!

Create your root account and on the next screen choose This Machine as your server type (we'll run everything on a single instance):

After the initial wizard, let's go to Settings (from the left menu) and input the instance URL with our (sub)domain we prepared for Coolify under General (make sure to include full URL including the leading https://).

Optionally you can also explore the Backup and Transactional Email tabs for setting up Coolify backups and transactional emails (for things like forgotten password).

Then head over to Servers (from the left menu) where you should see our localhost instance (running Coolify). Here you'll find settings for the Coolify components. Open it and go to the Proxy tab:

Checking that the proxy is running is important as it will route the traffic to the applications you'll self host. Even if Coolify is running and you are logged in inside the console, the coolify-proxy might not be.

In case the proxy is not running, open the logs and see what's wrong. In my case, the proxy wasn't running and I found the following inside the logs:

Creating required Docker Compose file.
Pulling docker image.
 Image traefik:v3.6 Pulling 
 Image traefik:v3.6 Pulled 
Ensuring network coolify exists...
Ensuring network havzqyfu212ybs9n5lg48gzy exists...
Starting coolify-proxy.
ParseAddr("fdb0:c330:3639::1/64"): unexpected character, want colon (at "/64")

There seemed to be an issue with parsing an IPv6 address. To resolve it I decided to turn off IPv6 support on the system. I opened an SSH connection to the server again to edit Docker's daemon.json file:

vi /etc/docker/daemon.json

I added an entry with "ipv6": false:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    {"base":"10.0.0.0/8","size":24}
  ],
  "ipv6": false
}

Then I restarted Docker:

systemctl restart docker

After that I could restart the proxy and see coolify-proxy container running on the system.

And that was it! If you can log in and see that the proxy is running, you are ready to start adding individual applications and services. You can still also use a hosted version of Coolify if you never set up a server before and all of this looks daunting.

Deploying resources

Resources in Coolify needs a project, so open Projects from the left side menu, and click + Add next to Projects. Then click + New next to the Resources headline. You should arrive on a New Resource page which lets you choose what you want to run. You can deploy directly from a git repository, Dockerfile or custom Docker Compose.

One-click services

One-click services are pre-configured Docker Compose templates provided directly by Coolify, skipping the complexity of manual setup and configuration. They are the easiest to start with.

Scroll a bit down and you should see them:

Coolify also maintains a directory for these services on the web. Some of services you can add are:

Internal tools: Appsmith, Budibase, NocoDB.
Chat & messaging: Matrix, Mattermost, Rocket.Chat.
Development & CI/CD: VS Code Server, Gitea, Jenkins.
Monitoring & analytics: Bugsink, CloudBeaver.
CMS: WordPress, Ghost.

They are all pretested, optimized, and coming with sensible defaults.

Custom services

Custom services in Coolify are those that you deploy with your own Docker Compose file. They require a little bit more work, but allow you to run almost anything. Have a look at how to deploy SerpBear, a SEO tool for keyword tracking as an example of a custom service.

Next steps

If you are new to self-hosting make sure to learn a little bit more about SSH, DNS, Linux, Docker, and related topics. You should also have a look at the Coolify documentation for even more information and tips.

I wrote a handbook for Kamal

Josef Strzibny — Fri, 05 Apr 2024 08:49:28 +0000

Kamal is an imperative deployment tool. It's basically a successor to Capistrano, but for a container era. It's a simple wrapper around Docker and that's the whole beauty of it. 37signals created Kamal to self-host Basecamp and Hey as part of their pull out of the cloud (running managed K8s).

I am always for simplicity when it comes to deployment and the truth is that lots of projects out there don't need the fully-featured Kubernetes to run. When Kamal was released I got intrigued and slowly adopted the tool. Nowadays I deploy all new projects with it.

Since the documentation is a little sparse at the moment and some people trying Kamal abandoned the effort when they faced their first issues, I decided to do something about it and wrote 'the missing manual' to Kamal called Kamal Handbook.

The book goes through all the important aspects of deploying with Kamal, describes the design choices and explains what's happening under the hood with illustrations. It's by design a small book you can read in a weekend. Hope you'll like it.

I just released InvoicePrinter 2.0

Josef Strzibny — Wed, 02 Oct 2019 11:50:08 +0000

InvoicePrinter is a Ruby library, server, and command-line client to make beautiful PDF invoices without browsers in no time. It's pure Ruby, but can be also used outside the Ruby world as a Docker container with a simple JSON API.

New features in 2.0:

New modern look & feel
More flexible buyer/seller boxes
Server is decoupled to a separate gem invoice_printer_server
New bundled fonts in a separate gem invoice_printer_fonts
Official Docker image
Prawn 2.2

Here is the project GitHub page, in the examples/ directory you can find a lot of code examples and how they look as PDFs. Follow docs/ to learn more about using it as a library, server or a command-line client.

Here is one example (download the final PDF):

And Ruby code to generate it:

#!/usr/bin/env ruby
# This is an example of a international invoice with Czech labels and English translation.

require 'invoice_printer'

labels = {
  name: 'Faktura',
  provider: 'Prodejce',
  purchaser: 'Kupující',
  tax_id: 'IČ',
  tax_id2: 'DIČ',
  payment: 'Forma úhrady',
  payment_by_transfer: 'Platba na následující účet:',
  account_number: 'Číslo účtu',
  issue_date: 'Datum vydání',
  due_date: 'Datum splatnosti',
  item: 'Položka',
  quantity: 'Počet',
  unit: 'MJ',
  price_per_item: 'Cena za položku',
  amount: 'Celkem bez daně',
  subtotal: 'Cena bez daně',
  tax: 'DPH 21 %',
  total: 'Celkem'
}

# Default English labels as sublabels
sublabels = InvoicePrinter::PDFDocument::DEFAULT_LABELS
labels.merge!({ sublabels: sublabels })

first_item = InvoicePrinter::Document::Item.new(
  name: 'Konzultace',
  quantity: '2',
  unit: 'hod',
  price: 'Kč 500',
  amount: 'Kč 1.000'
)

second_item = InvoicePrinter::Document::Item.new(
  name: 'Programování',
  quantity: '10',
  unit: 'hod',
  price: 'Kč 900',
  amount: 'Kč 9.000'
)

provider_address = <<ADDRESS
Rolnická 1
747 05  Opava
Kateřinky
ADDRESS

purchaser_address = <<ADDRESS
Ostravská 1
747 70  Opava
ADDRESS

invoice = InvoicePrinter::Document.new(
  number: 'č. 198900000001',
  provider_name: 'Petr Nový',
  provider_lines:  provider_address,
  provider_tax_id: '56565656',
  purchaser_name: 'Adam Černý',
  purchaser_lines: purchaser_address,
  issue_date: '05/03/2016',
  due_date: '19/03/2016',
  subtotal: 'Kč 10.000',
  tax: 'Kč 2.100',
  total: 'Kč 12.100,-',
  bank_account_number: '156546546465',
  account_iban: 'IBAN464545645',
  account_swift: 'SWIFT5456',
  items: [first_item, second_item],
  note: 'Osoba je zapsána v živnostenském rejstříku.'
)

InvoicePrinter.print(
  document: invoice,
  labels: labels,
  font: File.expand_path('../../assets/fonts/overpass/Overpass-Regular.ttf', __FILE__),
  logo: File.expand_path('../logo.png', __FILE__),
  file_name: 'promo.pdf',
  page_size: :a4
)

Before the final release I wrote a little background post on my blog.

Try it out and let me know what you think!

Django 2.2 polls app tutorial source code commit by commit

Josef Strzibny — Tue, 09 Apr 2019 22:47:01 +0000

TL;DR

I went through the famous Django "polls" app tutorial, and made it into a git repository which you can follow commit by commit:

GitHub link: deployment-from-scratch/django-2.2-polls

Tutorial link: Writing your first Django app

It's for Django 2.2.

Full Story:

Since the beginning of time (me wanting to leave PHP development) I was intrigued by Ruby on Rails and Django frameworks. I liked both for different reasons, but ended up doing mainly Rails because of Ruby the language (and living with a syndrome that I might made a mistake ever since).

Fast forward to 2019, and I am writing a book on deploying web application where I am trying to teach deployment of both Ruby and Python apps. There are pretty similar in that regards actually, but I needed some Python examples.

For this reason, I went to the Django site after many years (it's a lie, I keep watching Django space haha) and did the famous introductory tutorial of building "polls" app. I did it "commit by commit" so people can follow it easily while going through the tutorial.