DEV Community: Nathan Skiles

SerpApi's New Status Page Is Live!

Nathan Skiles — Thu, 25 Jun 2026 15:01:03 +0000

We’re excited to announce that SerpApi now has a new Status Page for service updates. This page is live and will serve as the central hub for all incident and outage communications. If there’s ever API downtime, service disruptions, or performance degradation, you’ll be able to find the latest updates there. This launch was driven by our commitment to transparency and direct customer feedback.

View the current status of all APIs and ensure you’re subscribed for notifications about any incidents. It’s live and ready to keep you informed.

Subscribe here!

Please note that we will continue using our Public Roadmap to track minor bugs and issues, and we’ll send direct updates once they’ve been resolved.

Subscribing to Updates

To subscribe to incident notifications, simply click the 'Subscribe' button at the top of the page. We offer several channels to receive notifications, including email, SMS, Slack, and Microsoft Teams.

After clicking Subscribe, you’ll receive a confirmation email and be redirected to a page where you can customize your notification preferences.

⚠️ If you do not confirm your subscription, you will not receive incident notifications.

If you do not receive a confirmation, please contact support at contact@serpapi.com.

Subscribe to the APIs You Use

Once you reach the component selection page, you can customize the notifications you receive based on the SerpApi services most relevant to your use case. By default, all APIs are selected, but you can easily narrow this down by selecting only the APIs you rely on (for example, just the Google Search API or AI Overview API).

I recommend at least subscribing to the API component to ensure you're at least notified about system-wide issues, along with your most used APIs.

💡 From the component selection screen (shown above), click "Select none" above the component table to deselect all APIs. This allows you to select only the APIs you care about, without manually deselecting irrelevant components.

Customizing your notifications ensures you only receive incident updates when the specific APIs you care about are impacted.

Final Thoughts

Our goal with this status page is to make service communications clearer, faster, and easier to follow.

If you haven’t already, we recommend subscribing now so you never miss an important update.

As always, please reach out if you have any questions or feedback!

Subscribe here!

Compressing API Responses

Nathan Skiles — Wed, 24 Jun 2026 15:09:58 +0000

When retrieving search engine results via SerpApi at scale, individual payloads may be small, but across hundreds of thousands or millions of searches, the total data transferred can add up quickly. One effective optimization is to enable gzip compression for API responses. This significantly reduces payload sizes and lowers overall bandwidth usage.

In this post, we’ll briefly cover why compressed responses are helpful, and how to request and handle compressed results in cURL, JavaScript, and Python. We’ll also show how to decompress the data on the client side.

Why Compress SerpApi API Responses?

SerpApi is a real-time API that scrapes search engine results for you and returns them as structured JSON, so you don’t have to manage your own scraping infrastructure.

Enabling compression on API responses reduces the size of the data transmitted over the network. This can lower bandwidth costs or help you stay within usage limits when bandwidth is a factor. Since JSON is text-based, it compresses extremely well with gzip. Gzip is optimized for text and JSON data, providing fast compression and decompression, which makes it ideal for API network transfers.

If your project makes many SerpApi calls, compression helps ensure you’re not transferring unnecessary data. It significantly reduces bandwidth usage while adding minimal overhead on both the client and server.

JSON Restrictor

While response compression is the ideal option when working with full payloads, you can also reduce the size of the data returned by using SerpApi’s JSON Restrictor feature. This lets you specify exactly which attributes you want to receive, and SerpApi will only scrape and return those fields. Doing so improves response times and reduces overall payload size.

You can read more about the JSON Restrictor here:

SerpApi: JSON Restrictor

How to Request Compressed Responses (and Decompress Them)

HTTP compression is a client-server agreement: the client tells the server it can handle compressed data, and the server replies with the data compressed. This happens through HTTP headers:

The client adds an Accept-Encoding: gzip header in the request, indicating it supports gzip compression.
If the server (SerpApi) supports it, it will compress the JSON response using gzip and include a Content-Encoding: gzip header to indicate that the data is compressed.

SerpApi’s servers do honor gzip compression when requested. However, the current official JavaScript and Python client libraries don’t expose any built-in option to enable it. This means that when using the official SerpApi packages, responses will typically be returned uncompressed by default. You can work around this by using standard HTTP tools, such as cURL or your language’s built-in HTTP libraries, to request compressed responses and decompress them yourself.

cURL Example (Command Line)

The simplest way to test gzip compression is with cURL. cURL has a convenient --compressed option that will do two things: send an Accept-Encoding: gzip header and automatically decompress the response for you. For example:

curl --compressed "https://serpapi.com/search.json?engine=google&q=coffee&api_key=YOUR_API_KEY"

In this command:

--compressed tells cURL to include Accept-Encoding: gzip, deflate in the request, and cURL will handle decompressing the response.
The URL is a SerpApi query (Google Search for “coffee”), including your api_key (replace YOUR_API_KEY with your actual key).

Without --compressed:

➜ Developer curl \
  -w "\nUncompressed download size: %{size_download} bytes\n" \
  -o /dev/null \
  "https://serpapi.com/search.json?engine=google&q=coffee&api_key=API_KEY"
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 210k 100 210k 0 0 1724k 0 --:--:-- --:--:-- --:--:-- 1726k

Uncompressed download size: 215728 bytes

With --compressed:

➜ Developer curl --compressed \
  -w "\nCompressed download size: %{size_download} bytes\n" \
  -o /dev/null \
  "https://serpapi.com/search.json?engine=google&q=coffee&api_key=API_KEY"
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 45529 0 45529 0 0 479k 0 --:--:-- --:--:-- --:--:-- 483k

Compressed download size: 45529 bytes

As you can see, we went from a payload size of ~210.79 KB to ~44.48 KB, roughly a 4.7x reduction.

cURL will output the JSON result directly to your terminal as if it were uncompressed (since it automatically unzips it). The transfer, however, was compressed behind the scenes; if you use the -v (verbose) flag, you can see the Content-Encoding: gzip in the response headers, indicating compression was used.

➜ Developer curl --compressed -v \
  -o /dev/null \
  "https://serpapi.com/search.json?engine=google&q=coffee&api_key=API_KEY"
...
> Host: serpapi.com
> User-Agent: curl/8.7.1
> Accept: */*
> Accept-Encoding: deflate, gzip
>
* Request completely sent off
< HTTP/2 200
< content-type: application/json; charset=utf-8
< content-encoding: gzip
< x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-download-options: noopen
< x-permitted-cross-domain-policies: none
< referrer-policy: strict-origin-when-cross-origin
< x-robots-tag: noindex, nofollow
< serpapi-search-id: 6915208fbf04d5f8cab7cdd0
< cache-control: max-age=3600, public
< etag: W/"937c88f1192291d10fa8ce4663bf03b5"
< x-request-id: 82d17608-eecc-45ae-a99c-61c44c321037
< x-runtime: 2.819488
< cf-cache-status: HIT
< vary: Accept-Encoding
< server: cloudflare
< cf-ray: 99da1bf63d0d4e7c-PDX
< alt-svc: h3=":443"; ma=86400

Alternatively, if you want to see the raw compressed data size or handle decompression yourself, you can do:

curl -H "Accept-Encoding: gzip" -o result.json.gz "https://serpapi.com/search.json?engine=google&q=coffee&api_key=YOUR_API_KEY"
# The response is now saved as result.json.gz in gzip format.
gunzip result.json.gz # This will decompress the file to result.json

After running gunzip, the file result.json will contain the original JSON response. You can compare the file sizes of result.json.gz vs result.json to appreciate the difference. In summary, using cURL with compression is straightforward and demonstrates the concept: ask for gzip in the request, get a gzip-compressed response, then unzip it to use the data.

JavaScript Example (Node.js)

With Node.js, we can achieve the same using common libraries. We’ll use axios for the HTTP request and Node’s built-in zlib module to decompress. (You could also use node-fetch or the native https module; the approach is similar.)

const axios = require('axios');
const zlib = require('zlib');

async function fetchSerpApiCompressed(query) {
  const url = `https://serpapi.com/search.json?engine=google&q=${query}&api_key=${API_KEY}`;
  // Send request with Accept-Encoding: gzip
  const response = await axios.get(url, {
    headers: { "Accept-Encoding": "gzip" },
    responseType: "arraybuffer", // get raw bytes (Buffer)
    decompress: false, // disable automatic decompression
  });
  console.log("Content-Encoding header:", response.headers["content-encoding"]); // should be "gzip" if compressed

  // The data is in gzip format (bytes)
  const outputBuffer = zlib.gunzipSync(Buffer.from(response.data));
  const jsonString = outputBuffer.toString("utf-8");
  const result = JSON.parse(jsonString);
  console.log("Total result count:", result.search_information.total_results);
}

fetchSerpApiCompressed("coffee");

Let’s break down what this does:

We set the Accept-Encoding: gzip header so SerpApi knows we want a compressed response.
We use responseType: 'arraybuffer' so that Axios gives us the raw binary data (instead of trying to convert it to a string or JSON automatically). The Content-Encoding from SerpApi should be gzip (you can check response.headers as shown).
We then use zlib.gunzipSync to unzip the response data. This function takes the compressed buffer and returns a new buffer with the decompressed data.
Finally, we convert the decompressed data to a string (UTF-8 text) and parse it as JSON to get the result object. At this point, the result is the same JavaScript object you would normally get if you had fetched uncompressed JSON.

💡 In Node, many HTTP libraries (Axios included) will automatically handle gzip if they see the Content-Encoding: gzip header. Axios sets decompress: true by default, so it may already return decompressed data if Accept-Encoding is honored. Here, we explicitly performed the decompression to demonstrate the process.

The key takeaway is that you must include the Accept-Encoding: gzip header in your request; without it, SerpApi will send plain JSON (most servers default to no compression unless asked ). With the header, the response shrinks dramatically, and you need a one-liner to unzip it in your code.

Python Example

For Python, we’ll use the requests library. Python’s requests can also handle gzip transparently, but we’ll show the manual steps for clarity:

import requests, gzip, json

url = "https://serpapi.com/search.json?engine=google&q=coffee&api_key=YOUR_API_KEY"
headers = { "Accept-Encoding": "gzip" }
response = requests.get(url, headers=headers)

print("Content-Encoding:", response.headers.get("Content-Encoding")) # should be "gzip" if compressed

compressed_data = response.content # raw bytes of the response
if response.headers.get("Content-Encoding") == "gzip":
    # Decompress the gzip data
    decompressed_bytes = gzip.decompress(compressed_data)
    data = json.loads(decompressed_bytes.decode('utf-8'))
else:
    # If by chance it's not compressed (Content-Encoding not gzip), parse directly
    data = response.json()

print("Total result count:", data["search_information"]["total_results"])

Explanation:

We send a GET request with Accept-Encoding: gzip in the headers.
We then check the Content-Encoding of the response - if SerpApi compressed it, it will be "gzip".
We take response.content, which is the raw bytes received. If it’s gzip-compressed, we use Python’s built-in gzip.decompress() to unzip the data. This returns the original JSON as bytes, which we decode to a string and then load into a Python dict using json.loads.
If the response wasn’t compressed (just in case), we fall back to response.json() to parse it normally.
Finally, we print something from the data (e.g., the total results count from search_information) to verify it worked.

Under the hood, requests will actually handle gzip for you (it sets Accept-Encoding: gzip, deflate, br by default, and decompresses the response when you access .text or .json() in many cases). So you could simplify the above by just doing data = response.json(). But explicitly calling gzip.decompress as shown leaves no doubt that we are handling the compressed data. Either way, the result data will be the same JSON content as usual, obtained in a more efficient way over the network.

Conclusion

Enabling gzip compression for SerpApi responses is a handy trick to decrease payload sizes. By adding a simple header to your request, you allow SerpApi to significantly reduce the JSON payload without losing any data. This leads to lower bandwidth consumption. Once the compressed data is received, you just need to decompress (unzip) it in your client code.

In practice, this optimization is usually straightforward: HTTP clients negotiate compression via Accept-Encoding and Content-Encoding headers, and modern tools/libraries often handle the heavy lifting. As a developer, it’s good to understand how it works so you can enable it when appropriate and ensure you handle the compressed data correctly. Given that compression can save a lot of bandwidth at minimal cost, it’s often worth using when dealing with large API responses or high volumes of requests.

By compressing SerpApi responses, you make your data transfer leaner, which is a win-win for both your application performance and resource usage.

Building a UI for Google AI Overviews using SerpApi: A Practical Guide

Nathan Skiles — Tue, 23 Jun 2026 15:08:09 +0000

AI Overviews have become the first thing many people read on Google, summarizing instructions, comparisons, and even simple calculations while citing sources at the top of the results page. In previous articles, we learned how to scrape Google AI Overviews with SerpApi, and specifically how to do this with Node JS.

In this post, we walk through the AI Overview data returned by SerpApi and show how to use it in a small, real-world example. We focus on the response structure—text blocks, highlights, and references—and explain when and why you may need to perform an additional request so you can render the overview cleanly and reliably in your UI.

SerpApi

SerpApi is a web scraping API that returns structured JSON from Google and other search engines. It handles the complex parts for you, including proxies, CAPTCHAs, headless browsing, and localization, so a single HTTP request yields clean, consistent results. In this post, we’ll use its Google Search and AI Overview endpoints to fetch and render the overview content.

Prerequisites

SerpApi API Key - If you don’t have an account yet, you can sign up for a free plan with 250 searches per month. If you already have an account, your API key is available in your dashboard.
Node.js 7.10.1 or newer (optional) - Used later in our real-world example.

AI Overview Flow

Before we dive into the data, it helps to understand how an AI Overview is obtained.

Initial Request

For many queries, the Google Search API returns an AI Overview directly in the initial response. This is the simplest path because you can parse and render the ai_overview object immediately with no extra calls.

Follow-up request (when a page_token is present)

For less common or more complex queries, Google may lazy-load the AI Overview while its model finishes constructing the response. In these cases, the initial Search API response includes a short-lived page_token. Use this token with SerpApi’s AI Overview API to fetch the full overview.

💡 Note: because this involves two API calls, it will count as two SerpApi searches.

Best Practices

Use no_cache: true to avoid stale results and expired tokens.
If a page_token is present, call the google_ai_overview engine immediately.
Process each query sequentially rather than in parallel to reduce the chance of token expiration.

For more information, check out my blog post on Fetching AI Overviews:

Fetching AI Overviews with Node.js

Anatomy of an AI Overview

At a glance, we'll be working with two top-level objects:

text_blocks: ordered content blocks to be shown in sequence (paragraphs, headings, lists).
references: a flat array of citations you can map to via numeric indices.

Here's a simple mental model for how AI Overviews are structured:

ai_overview
  ├─ text_blocks[] ← blocks to render in order
  │ ├─ type ← "paragraph" | "heading" | "list" | ...
  │ ├─ snippet ← content text
  │ ├─ snippet_highlighted_words[] (optional)
  │ ├─ reference_indexes[] (optional)
  │ └─ list[] ← when type === "list": array of { title, snippet, reference_indexes? }
  └─ references[] ← [{ title, link, snippet, source, index }]

Common block types

This is not an exhaustive list, since Google can introduce new types, but these are the ones you will see most often.

Paragraph : plain text in snippet. May include snippet_highlighted_words for emphasis and reference_indexes for inline citations.
Heading : section titles that segment the overview (for example, “Key features and functionality”, “Additional considerations”).
List : list is an array of items. Each item has a title, a snippet, and optional reference_indexes. Typically in the form of unordered lists.

Highlights and citations

Some blocks include highlighted words for emphasis or references that link to sources.

snippet_highlighted_words: an array of terms or phrases you can style in the UI for quick scanning.
reference_indexes → references: each number in reference_indexes maps to a record in references. Use these for superscripts or footnotes that link to the source list.

For example:

A paragraph with reference_indexes: [0, 2] should show two citations, which resolve to references[0] and references[2].
A list item can do the same. Treat each item’s reference_indexes independently, so you only show the citations that support that item.

💡 I recommend using switch statements on block.type, render snippet or list accordingly, then pass reference_indexes and the full references array to a small Citations helper.

This keeps the mapping logic in one place and makes it easy to add future block types without touching the rest of your UI.

Real World Example

Below is a simple Node.js + React setup that proxies requests from the frontend (to avoid CORS), performs the AI Overview follow-up when a page_token is present, and renders the common block types.

If you'd like to run or reference the example, you can find the repo here:

GitHub - NateSkiles/using_ai_overviews

Server

Keep your API key server-side, avoid CORS, and conditionally call the AI Overview API only when a page_token is present. Below is the code I used to handle requests from my frontend and pass them along to SerpApi.

Some key points to note:

json_restrictor=ai_overview in the initial request, ensures that we only return the AI Overview for this example. This can decrease response times and make it easier to focus on relevant data in the response, but it can be removed if you require additional data from the search results.
no_cache=true reduces token-expiration headaches.
A page_token follow-up is a second SerpApi call.

// ./server/index.js
app.get("/api/search", async (req, res) => {
  try {
    const q = req.query.q || "";
    if (!q) return res.status(400).json({ error: "Missing q" });

    // 1) Initial search: ask only for the AI Overview
    const params = new URLSearchParams({
      engine: "google",
      q,
      api_key: process.env.SERPAPI_KEY,
      json_restrictor: "ai_overview",
      no_cache: "true", // avoid stale results and expired tokens
    });

    const request = await fetch(`https://serpapi.com/search.json?${params}`);
    const data = await request.json();

    // 2) If Google did not include ai_overview at all, return what we got
    if (!data.ai_overview) {
      return res.json(data);
    }

    // 3) If a page_token is present, fetch the full AI Overview
    const token = data.ai_overview.page_token;
    if (token) {
      const aio_params = new URLSearchParams({
        engine: "google_ai_overview",
        page_token: token,
        api_key: process.env.SERPAPI_KEY,
        no_cache: "true",
      });
      const aio_request = await fetch(`https://serpapi.com/search.json?${aio_params}`);
      const aio_data = await aio_request.json();
      return res.json(aio_data); // note: counts as a second SerpApi search
    }

    // 4) Otherwise, we already have the overview
    return res.json(data);
  } catch (e) {
    res.status(500).json({ error: String(e) });
  }
});

using_ai_overviews/server/index.js at main · NateSkiles/using_ai_overviews

Client

When the form is submitted, we clear any previous state, ask our server for results, and stash the returned ai_overview in state. The server already handles the follow-up call when a page_token appears, so the client only needs to check whether an overview exists.

// ./client/src/App.jsx
async function onSearch(e) {
  e.preventDefault();
  setError(null);
  setOverview(null);
  setLoading(true);

  try {
    const r = await fetch(`/api/search?q=${encodeURIComponent(q)}`);
    const data = await r.json();

    if (!r.ok) throw new Error(data?.error || "Request failed");
    if (!data.ai_overview) {
      setError("No AI Overview in this result.");
    } else {
      setOverview(data.ai_overview);
    }
  } catch (err) {
    setError(String(err.message || err));
  } finally {
    setLoading(false);
  }
}

Once the overview is set, render each block in order and finish with the references section. Keeping this mapping in one place makes it easier to work with other block types later.

// ./client/src/App.jsx
{overview && (
  <div className="space-y-4">
    <article className="prose prose-neutral">
      {overview.text_blocks.map((b, i) => (
        <BlockRenderer key={i} block={b} />
      ))}
    </article>

    {overview.references?.length > 0 && (
      <References references={overview.references} />
    )}
  </div>
)}

We will pass each of the text_blocks in the overview to the BlockRenderer component to handle the more complex aspects of rendering the overview.

Initial search page

Rendering Blocks

BlockRenderer uses a switch statement on block.type and returns the appropriate element.

// ./client/src/components/BlockRenderer.jsx
export default function BlockRenderer({ block }) {
  switch (block.type) {
    case "heading":
      return <h2 className="text-lg font-semibold">{block.snippet}</h2>;
    case "paragraph":
      return (
        <div>
          <p>
            {block.snippet}
            <Citations indexes={block.reference_indexes} />
          </p>
          {block.video && <VideoEmbed video={block.video} />}
        </div>
      );
    case "list":
      return <ListItems items={block.list} />;
    default:
      return null;
  }
}

💡 Note: if you later decide to highlight snippet_highlighted_words, you can add that inside the paragraph branch without affecting other blocks.

Headings

Heading blocks are used to segment sections of the overview. We simply return them semantically using an H2 element :

<h2 className="text-lg font-semibold">{block.snippet}</h2>

Overview highlighting the heading block type

Paragraphs

While headings break up the content, most of the AI Overview will come in the form of paragraphs.

<div>
  <p>
    {block.snippet}
    <Citations indexes={block.reference_indexes} />
  </p>
  {block.video && <VideoEmbed video={block.video} />}
</div>

As you can see, I've also included a component to handle any videos that may be returned in the block. In my experience, videos are typically only returned in paragraph blocks, and are not a block type, meaning we can handle them here if they exist.

Overview highlighting the paragraph block type

Lists

While lists are simple on the surface, just an unordered list with each list item being a line item, Google will often return nested lists, which could break some of our logic.

Below is a component I used to handle nested lists, being sure to include any citations or deeper lists. This component only handles lists of a depth up to 4, but can be expanded if needed.

// ./client/src/components/BlockRenderer.jsx
function ListItems({ items, depth = 0 }) {
  // Recursive list renderer for nested lists
  if (!items || items.length === 0) return null;
  const styleByDepth = [
    "list-disc pl-6",
    "list-circle pl-6",
    "list-[square] pl-6",
    "list-disc pl-6",
  ];
  const ulClass = styleByDepth[depth] || styleByDepth[styleByDepth.length - 1];
  return (
    <ul className={ulClass}>
      {items.map((item, idx) => {
        const hasNested = item.list && item.list.length > 0;
        return (
          <li key={idx} className="mt-1">
            {item.title ? <strong>{item.title} </strong> : null}
            {item.snippet ? (
              <span>
                {item.snippet}
                <Citations indexes={item.reference_indexes} />
              </span>
            ) : null}
            {hasNested ? (
              <div className="mt-1">
                <ListItems items={item.list} depth={depth + 1} />
              </div>
            ) : null}
          </li>
        );
      })}
    </ul>
  );
}

Overview highlighting the list block type

References

Finially, we need to handle our references section, which we do using a component that we pass our references array to.

// ./client/src/components/References.jsx
export default function References({ references }) {
  if (!references || references.length === 0) return null;

  return (
    <details className="mt-8">
      <summary id="references" className="text-xl font-semibold mb-2">
        References
      </summary>
      <ol className="list-decimal pl-6 space-y-2">
        {references
          .sort((a, b) => a.index - b.index)
          .map((ref) => (
            <li id={`ref-${ref.index + 1}`} key={ref.index}>
              <a
                className="underline"
                href={ref.link}
                target="_blank"
                rel="noreferrer"
              >
                {ref.title || ref.link}
              </a>
              {ref.source ? ` — ${ref.source}` : ""}
            </li>
          ))}
      </ol>
    </details>
  );
}

Closing

That’s the whole flow: request the overview with the Search API, follow up with the AI Overview API only when a short-lived page_token is returned, then render the structured result using text_blocks and references. With a thin server proxy to avoid CORS and a simple, table-driven block renderer, you get predictable output, clean citations, and an easy path to add new block types as Google evolves the format. Keep no_cache: true, make the follow-up call immediately when a token appears, process sequentially if you are batch testing, and fall back gracefully when no overview exists.

If you want to try this yourself, clone the example, plug in your SerpApi key, and test with a few real queries. From there, tweak the renderer for your preferences, add styling for snippet_highlighted_words, and extend list handling or media as you need.

Also, read about how AI Overview is changing the rank tracking industry.

Additional Resources

Uploading Images and Searching with Google Lens via SerpApi

Nathan Skiles — Mon, 22 Jun 2026 15:41:25 +0000

Intro

Selling collectibles—like trading cards, vinyl records, and action figures—online can be challenging, especially when you’re unsure exactly what you have. In this tutorial, we’ll build an automation that identifies an item using visual search. By leveraging Google Lens (via SerpApi) for image recognition, you can quickly determine the exact name of a card or collectible. This approach is incredibly useful: instead of manually guessing keywords, you use image searches combined with AI to pinpoint an item’s title directly from a photo, saving you time and improving your listings.

What You’ll Need

SerpApi account & API key: Sign up for an API key (they offer a free tier of 100 searches/month).
n8n: an automation workflow tool (self-hosted or cloud).
Amazon Web Services (AWS): an AWS account with an S3 bucket to store images. We’ll use this to host the image for Google Lens to analyze.

Workflow Overview

In the workflow below, we’ll connect several powerful tools to automate identifying products from images. We’ll begin by receiving an image through a webhook, processing and uploading it to AWS S3, performing a visual search using Google Lens (via SerpApi), and finally extracting potential titles (or names) for the product.

Step-by-Step Setup

Below, we’ll walk through each component of the workflow in detail, starting from triggering the workflow all the way through extracting potential item titles. Follow along to build your automation step-by-step:

Workflow Triggers

n8n provides several methods for starting a workflow. Below, I’ve highlighted some additional trigger options you might consider depending on your specific needs:

Webhook Trigger (used in this tutorial): Send images directly to your workflow using an HTTP webhook, suitable for automation from external scripts or apps.
Manual Trigger : Ideal if you want full control to initiate your workflow manually, especially when retrieving images from specific sources at specific times.
Application Triggers : Automatically start your workflow based on events from integrated apps, such as:
- Google Sheets : Trigger workflows when a new row is added or updated.
- Email Trigger : Initiate workflows from emails containing images sent to a designated inbox.
- Google Drive : Start workflows based on file changes within a specific folder.

💡 Explore n8n’s trigger library for additional integration ideas.

Scheduled Trigger : Automate workflows at specified intervals. This is useful for regularly processing images collected over a set period.
Local File Trigger : For self-hosted n8n instances, monitor a local directory for new or updated image files to automatically kick off workflows.

In this guide, we’ll use the Webhook Trigger for simplicity and flexibility. Let’s continue with setting it up.

Create a workflow in your n8n instance and add the first step (trigger), as mentioned above, we’ll select “On webhook call”:

Once the webhook trigger is set, we can configure the HTTP Method we will use when calling the webhook, the Path we want for the webhook, the authentication you would like to use, and when the webhook should respond. We’ll set the following parameters:

HTTP Method: POST
- We will be POSTing image data to the webhook.
Path: upload
- You can set this to your preference, but make sure to keep this in mind later when we start sending image data to the webhook.
Authentication: None
- While I won’t be using authentication for my tutorial, refer to Webhook credentials for more information on setting up each credential type.
Response: Immediately
- As most of our processing will occur independently of our upload script, we want the workflow to simply confirm receipt of the request. However, you can adjust this to your liking if integrating more closely with existing systems.

While n8n offers additional configuration options, these will be all we should need for the time being. Once complete, your webhook configuration should look like this:

Upload Script

Now that our webhook is set up, we need to be able to send images to the webhook. While there are many ways to do this, for testing and my simple use case, we’ll use a Node.js script to send local images to the webhook as Base64 strings.

I’ve created a repository for the script I will be using for the rest of this post, which I recommend you clone and use to follow along. I won’t walk through setting up the upload script; please refer to the README to install it locally. I will, however, review how the script works in case you want to write your own or make adjustments to mine.

GitHub - NateSkiles/n8n-price-blog

processImages()

async function processImages() {
  try {
    console.log(`Looking for images in ${IMAGES_DIR}`);

    try {
      await fs.access(IMAGES_DIR);
    } catch (error) {
      console.error(
        `Error: Directory ${IMAGES_DIR} does not exist or is not accessible.`
      );
      return;
    }

    const files = await fs.readdir(IMAGES_DIR);

    const imageFiles = files.filter((file) =>
      IMAGE_EXTENSIONS.includes(path.extname(file).toLowerCase())
    );

    console.log(`Found ${imageFiles.length} image(s) in the images directory`);

    // Process each image file
    for (const imageFile of imageFiles) {
      const imagePath = path.join(IMAGES_DIR, imageFile);
      await postImageToWebhook(imagePath);
    }

    console.log("All images have been processed");
  } catch (error) {
    console.error("Error processing images:", error);
  }
}

This is the main driver function of the script. It checks for an images directory, finds all supported image files within it, and sends them one by one to the webhook. It includes basic validation (ensuring the directory exists), filters out non-image files, and passes each valid image to a helper function for uploading. It also logs progress throughout, which is helpful during testing or debugging.

postImageToWebhook()

async function postImageToWebhook(imagePath) {
  try {
    const imageBuffer = await fs.readFile(imagePath);
    const filename = path.basename(imagePath);

    console.log(`Posting ${filename} to webhook...`);

    const payload = {
      filename: filename,
      size: imageBuffer.length,
      imageData: imageBuffer.toString("base64"),
    };

    const response = await fetch(WEBHOOK_URL, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(payload),
    });

    if (response.ok) {
      console.log(`Successfully posted ${filename}`);
    } else {
      console.error(
        `Failed to post ${filename}: ${response.status} ${response.statusText}`
      );
    }
  } catch (error) {
    console.error(`Error posting ${path.basename(imagePath)}:`, error);
  }
}

This helper function handles the actual upload of an image. It reads the file into memory, converts it into a base64 string, and builds a JSON payload that includes the filename and file size. It then sends the payload as a POST request to the webhook URL. If the request fails or throws an error, it also logs that information.

By splitting the logic into these two async functions, the script stays modular and easy to extend, for example, to add rate limiting, error retries, or metadata tagging. Feel free to adapt this structure to fit your use case.

Test

Once you're ready to send images as Base64 strings to the webhook, click the "Listen for test event" button, and the webhook will start listening.

In the screenshot above, I've successfully sent a request to the webhook.

Convert Base64 to File

Because our images are being sent to the webhook as Base64 strings, we’ll need to convert that string back into a binary file before uploading to AWS S3. Fortunately, n8n provides a built-in “Convert to File” node that handles this for us.

Start by adding a new node, then search for and select “Convert to File.”

From the list of operations, choose “Move base64 string to file.”

Next, drag the imageData field from the body of the webhook request into the Base64 input of this node.

💡 If you’re using your own custom script, be sure to match whatever key name you used for the Base64 string in your payload. In the script I shared, it’s imageData.

Once added, execute the node. You should now see the resulting image file in the output preview.

Upload to AWS S3

Congratulations! You’ve completed the hard part; images are now successfully sent and received by your n8n instance. The next step is to upload those images to cloud storage so they can be publicly accessed via a URL, which we’ll need when using SerpApi’s Google Lens API in the next section.

For this tutorial, we’ll use AWS S3, a widely adopted storage service that integrates seamlessly with n8n via the AWS S3 node.

If you don’t already have an S3 bucket set up, refer to AWS’s documentation to get started:

Getting started with Amazon S3 - Amazon Simple Storage Service

⚠️ Make sure your bucket policy allows public read access, we’ll need this so the image can be accessed by SerpApi.

Before using the AWS S3 node, you’ll need to connect your AWS account to n8n. You can follow the official guide here:

AWS credentials | n8n Docs

Once your credentials are added:

Add the AWS S3 node to your workflow.
Set Resource to File, and Operation to Upload.
Enter the bucket name where your images will be stored.

By default, the Input Binary Field should already point to the output of the previous “Convert to File” step. You’ll also need to specify a name for the uploaded file.

You can either:

Drag the filename field from the Webhook step, or
Use an expression like this:

{{ $('Webhook').item.json.body.filename }}

Execute the step and ensure the file is uploaded correctly to S3.

Google Lens Search via SerpApi

SerpApi recently released an official n8n node that simplifies integration with our APIs. While you can still use the generic HTTP Request node to interact with SerpApi, this tutorial will use the SerpApi Official node to streamline configuration and reduce setup time.

For instructions on how to install the SerpApi node in your n8n instance, refer to our announcement post:

Boost Your n8n Workflows with SerpApi’s Verified Node

Set Up the SerpApi Node

Add a new node and search for “SerpApi Official.”
Add your SerpApi API key to the node’s credentials. You can find your key here: https://serpapi.com/manage-api-key
Select the Google Lens operation.
In the Image URL field, drag the Location attribute from the AWS S3 step (this is the public URL of the image you uploaded).

In the example below, I’ve also set the language parameter to English and the country parameter to United States :

Once configured, execute the node and confirm that SerpApi returns data based on the image URL.

Extract Matching Titles

At this point, you can begin branching out based on your specific use case. In my case, I want to determine the name of the product I uploaded, so I’ll extract a list of possible product names from the visual matches returned by SerpApi’s Google Lens API.

To do this, we’ll use the Code node in n8n to process and format the results with JavaScript.

Search for and add the Code node to your workflow.
Set the Mode to “Run Once for Each Item.”
Paste the following JavaScript into the code editor:

const matches = $json.visual_matches;
const topMatches = matches.slice(0, 5);
// Map titles, trim them, and enumerate
const titlesList = topMatches.map((m, index) => {
  return `${index + 1}. ${m.title}`;
});

return { json: { titlesList: titlesList.join("\n") } };

Finally, execute the node and confirm that the titles of the top 5 visual match results are returned in a clean, readable list.

Wrap-Up

In this first part, we built the foundation of our visual search workflow. Starting with a Base64-encoded image, we uploaded it to generate a public URL and used the Google Lens API (via SerpApi) to identify what the item could be. This gave us a shortlist of possible product names, no manual searching required.

Now that we have these possible titles, what’s next? In Part 2, we will bring in AI to decide which title is the best match for our item and then automate an eBay search for that exact product. With that, we’ll be able to fetch the current prices of the item on eBay and even prepare the data for tracking or listing.

Searching eBay with AI and Automation using n8n

How to Extract Full Opinion Text from Google Scholar Case Law with SerpApi

Nathan Skiles — Wed, 10 Jun 2026 21:47:52 +0000

SerpApi’s Google Scholar Case Law API returns structured case law data from Google Scholar, including case details and related metadata. For many workflows, that structured response is enough.

However, some use cases require the full opinion body. You may want to store the opinion text locally, make it searchable, review it in a cleaner format, or pass it into another internal workflow.

The full opinion body is available in the raw HTML response. In this tutorial, we’ll extract that opinion body from the HTML, convert it to Markdown, and save it locally using both JavaScript and Python.

If you’re new to the Google Scholar Case Law API, my colleague’s blog on returning the structured response is a helpful starting point, but it’s not required for following this tutorial:

👉 How to Scrape Google Case Law API for Legal Research, Analytics, AI, and More

Why use SerpApi?

Google Scholar Case Law pages can be scraped manually, but maintaining that workflow reliably can become difficult. At scale, you need to manage proxy infrastructure, CAPTCHA handling, retries, parsing changes, and monitoring for page structure updates.

SerpApi handles the search engine scraping layer and returns results through an API. For Google Scholar Case Law, that means you can retrieve structured case law data from the JSON response, while still having access to the underlying page HTML when you need the full opinion body.

Why the opinion body is in the raw HTML

The Google Scholar Case Law pages include the full legal opinion, but opinion length varies dramatically. Some opinions are relatively short, while others can be prohibitively long.

For that reason, the full opinion body is not returned directly in every JSON response. Including it by default would increase response size and overhead for users who only need metadata like the case title, court, decision date, citations, or related case information.

The full opinion is still available through the raw HTML. In the HTML, the opinion body is contained in the #gs_opinion element:

<div id="gs_opinion">

The sample HTML in the companion repo shows the case opinion inside #gs_opinion, including headings, body text, links, page references, blockquotes, and footnotes.

What we’re building

In these examples, we’ll request the Google Scholar Case Law page directly as HTML. If your workflow also needs structured metadata, you can request the JSON response first and then retrieve the raw HTML file linked in that response. The extraction logic is the same once you have the HTML.

The overall workflow is the same in both examples. We’ll walk through it in JavaScript first, then show the equivalent Python version:

Request the Google Scholar Case Law page from SerpApi as raw HTML.
Parse the HTML.
Select the #gs_opinion element.
Convert the opinion HTML to Markdown.
Save the Markdown file locally.

Requirements

To follow along, you’ll need:

A SerpApi account
A SerpApi API key
Node.js for the JavaScript example
Python 3 for the Python example

If this is your first time using SerpApi, you can sign up for a free account and use the included 250 monthly searches to test the examples in this tutorial.

JavaScript example: HTML to Markdown

Let’s start with the JavaScript version. This example requests the Google Scholar Case Law page as HTML, extracts the opinion body, converts it to Markdown, and saves the result locally.

The full JavaScript example is available in the /javascript directory of the companion repository.

Install dependencies

From the javascript directory, install the required packages:

npm install

The dependencies are already listed in the example project’s package.json. This example uses:

serpapi to fetch the raw HTML response from SerpApi.
cheerio to parse the HTML and select the opinion body with familiar CSS-style selectors.
turndown to convert the opinion HTML into Markdown.
dotenv to load your SerpApi API key from a local .env file.
Node.js fs and path modules to write the Markdown file locally.

Fetch the raw HTML

We import getHtml from the serpapi package and define the Google Scholar Case Law case_id we want to retrieve:

const { getHtml } = require("serpapi");
require("dotenv").config();

const caseId = "9174924986185145879";

Now we can request the page HTML. The parameters are fairly straightforward:

engine - Set to google_scholar_case_law, the API we are requesting data from.
api_key - Your SerpApi API key, loaded from the environment variable.
case_id - The Google Scholar Case Law case ID for the opinion we want to extract.

getHtml(
  {
    api_key: process.env.SERPAPI_KEY,
    engine: "google_scholar_case_law",
    case_id: caseId,
  },
  (html) => {
    // We'll parse the HTML in the next step.
  }
);

This returns the Google Scholar Case Law page as HTML, which we can then parse and extract the opinion body from.

Parse the opinion body

Once the HTML is returned, load it with Cheerio:

const $ = cheerio.load(html);

Cheerio lets us query the HTML with CSS-style selectors. Since the opinion body is contained in the #gs_opinion element, we can select that element and get its inner HTML:

const opinionHtml = $("#gs_opinion").html();

It’s also worth handling the case where the opinion body is not found:

if (!opinionHtml) {
  console.error("Could not find case opinion in the search results.");
  return;
}

At this point, opinionHtml contains the HTML for the opinion body, including paragraphs, headings, links, blockquotes, page references, and footnotes.

Convert the opinion to Markdown

Next, create a new Turndown service and pass the opinion HTML to turndown():

const turndownService = new TurndownService({
  headingStyle: "atx",
  codeBlockStyle: "fenced",
  strongDelimiter: "**",
  emDelimiter: "*",
  linkStyle: "inlined",
});

const markdown = turndownService.turndown(opinionHtml);

Turndown is a good fit here because we are not trying to manually scrape each paragraph, heading, link, or blockquote. At this point, we already have the opinion body as HTML. We just want to preserve the readable structure in a more portable text format.

Markdown works well for that because it keeps the output readable while still preserving useful formatting like headings, links, paragraphs, blockquotes, bold text, and italics.

Save the Markdown file

Finally, create an output filename and write the Markdown to the shared output directory:

const isoDate = new Date().toISOString().split("T")[0];
const outputPath = path.join(outputDir, `js_${caseId}_${isoDate}.md`);

fs.mkdirSync(outputDir, { recursive: true });
fs.writeFileSync(outputPath, markdown, "utf8");

console.log(`Saved case opinion to ${outputPath}`);

You can certainly tune Turndown further based on your use case and formatting needs. Here's some example output from the JavaScript example using Turndown:

**937 S.W.2d 444 (1996)**

### CONTINENTAL COFFEE PRODUCTS CO. and Allen D. Duff, Petitioners,  
v.  
Juanita CAZAREZ, Respondent.

[No. 95-0827.](/scholar?scidkt=15876565480693424878&as_sdt=2&hl=en)

**Supreme Court of Texas.**

Argued February 14, 1996.

Decided December 13, 1996.

Rehearing Overruled February 21, 1997.

[445](#p445)[\*445](#p445) A. Martin Wickliff, Jr., Barbara L. Johnson, Paul E. Hash, Houston, for Respondent.
...

Python example: HTML to Markdown

The Python version follows the same overall workflow: request the page as HTML, parse the returned HTML, select the #gs_opinion element, convert it to Markdown, and save the result locally.

The full Python example is available in the /python directory of the companion repository.

Install dependencies

From the python directory, install the required packages:

pip install -r requirements.txt

This example uses:

serpapi to fetch the raw HTML response from SerpApi.
beautifulsoup4 to parse the HTML and select the opinion body.
markdownify to convert the opinion HTML into Markdown.
python-dotenv to load your SerpApi API key from a local .env file.

Fetch the raw HTML

First, load the API key from your .env file and make sure it exists:

load_dotenv()

api_key = os.getenv("SERPAPI_KEY")

if not api_key:
    raise RuntimeError("SERPAPI_KEY is required.")

Then request the Google Scholar Case Law page as HTML:

html = serpapi.search(
    api_key=api_key,
    engine="google_scholar_case_law",
    case_id=CASE_ID,
    output="html",
)

The output="html" parameter tells SerpApi to return the Google Scholar Case Law page as HTML instead of JSON.

Parse the opinion body

Next, parse the HTML with Beautiful Soup and select the #gs_opinion element:

soup = BeautifulSoup(html, "html.parser")
opinion = soup.select_one("#gs_opinion")

As in the JavaScript example, it’s worth handling the case where the opinion body is not found:

if not opinion:
    raise ValueError("Could not find case opinion in the search results.")

Convert and save as Markdown

Once we have the opinion body, we can convert it to Markdown with markdownify:

markdown = markdownify(
    str(opinion),
    heading_style="ATX",
    bullets="-",
).strip()

Then create the output directory, generate a filename using the case ID and current date, and write the Markdown file:

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

iso_date = date.today().isoformat()
output_path = OUTPUT_DIR / f"py_{CASE_ID}_{iso_date}.md"

output_path.write_text(markdown, encoding="utf-8")

print(f"Saved case opinion to {output_path}")

The Python output follows the same structure as the JavaScript example, with Markdown headings, links, page references, and opinion text preserved.

Caveats and edge cases

This workflow is intentionally simple, but there are a few things to keep in mind.

First, the script should fail clearly if the #gs_opinion element is not found. If Google Scholar changes the page structure, or if a specific result does not include an opinion body, you do not want the script to silently save an empty file.

Second, HTML-to-Markdown libraries may handle links, anchors, spacing, and nested elements differently. The output should be reviewed before using it in production workflows, especially if you need to preserve legal citations or page references exactly.

Finally, Google Scholar opinions may include page markers, footnotes, blockquotes, citation links, and other formatting from the original opinion page. Depending on your use case, you may want to preserve those elements, clean them from the final Markdown, or customize the conversion rules further.

View the full code on GitHub

The full JavaScript and Python examples are available in the companion repository:

👉 https://github.com/NateSkiles/scraping_case_law_body

Each folder includes its own setup instructions and writes generated Markdown files to the repo’s root-level output directory.

Conclusion

SerpApi’s Google Scholar Case Law API gives you structured case law data, while the raw HTML provides access to the full opinion body when needed. By selecting the #gs_opinion element, you can extract the complete opinion and convert it into Markdown for storage, analysis, search indexing, or internal research workflows.

You can use the companion repository to run the JavaScript or Python example locally, then adapt the parsing and Markdown conversion steps for your own case law workflows.

For more, you can also check out:

Searching eBay with AI and Automation using n8n

Nathan Skiles — Mon, 04 Aug 2025 23:33:55 +0000

Welcome to Part 2 of our tutorial series. Previously (Part 1), we created an automated workflow that takes an image and returns potential item names using Google’s visual search.

In case you missed it, you can find Part 1 here:

Uploading Images and Searching with Google Lens via SerpApi

In this tutorial, we’ll build an automation that identifies an item using visual search. By leveraging Google Lens (via SerpApi) for image recognition, you can quickly determine the exact name of a card or collectible.

serpapi.com

Now, we’ll close the loop by integrating AI and price lookup. The goal for this second part is to automatically figure out which of those candidate titles is the correct one for our item, and then find how much it’s selling for on eBay.

We will use AI to pick the best title, and then use SerpApi’s eBay Search API to fetch live pricing data. We’ll also touch on ways to log or store this info for future use.

What You’ll Need

n8n: the same n8n setup from Part 1. The workflow will be extended, so have that ready.
SerpApi API key: continue using your SerpApi account for the eBay search step. (No new setup if you have it from Part 1.)
OpenAI API Key: an OpenAI account (or another supported LLM in n8n). You’ll input this in n8n’s credentials so the AI Agent node can access the model.
(optional) Google Sheet or Database: If you want to store the results (item names and prices) for record-keeping or price tracking, you can prepare a Google Sheet or database. This is optional, but we’ll discuss how you could send data to Google Sheets as an example.

Workflow Continuation

Now we will extend the workflow from Part 1. After extracting the list of possible titles, we’ll add steps for AI processing and eBay lookup. Referring to the workflow diagram (from Part 1) for context, our new nodes will come after the “Extract Visual Match Titles” step.

Your workflow should currently look like this:

Using the AI Agent to Refine Results

n8n offers a variety of AI nodes, with capabilities ranging from text summarization to classification. For our use case, we’ll use the AI Agent node, which lets you attach multiple chat models and tools to the agent.

docs.n8n.io

We’ll use the AI Agent to determine the most likely product name based on the title list we generated in the previous blog post.

Add the AI Agent node to your workflow and set the Source for Prompt option to Define below. This allows us to pass our title list to the chatbot and provide it with context.

For the prompt (user message), I’ve used the following:

Here are the top visual matches from a Google Lens search for a product image: {{ $json.titlesList }}

Return the most likely product title from these matches that best describes the specific item in the image.

Do not include bundle listings, condition, or unrelated items.

Next, add a System Message (not shown by default) to give the chatbot additional instructions:

You are an AI assistant that specializes in identifying and generating accurate, search-optimized product titles from visual search data.

Given a list of titles extracted from a reverse image search, return the most likely full product title for the specific item in the image.

Your output should be a clean, concise string that includes the key identifiers (brand, model, year, type, edition, etc.) — formatted like a product listing title someone might search for on a marketplace (e.g., eBay or Amazon).

Only return the title — no explanations or extra words.

Do not include specific model numbers unless they are relevant to the product. For example:

Sony PlayStation 5 Digital Edition Console (Model CFI-1218B)

Can just be:

Sony PlayStation 5 Digital Edition Console

Your AI Agent parameters should now look like this:

We’re not quite done configuring the AI Agent. We still need to add a Chat Model, along with any tools that might help the AI Agent process the data.

To add a Chat Model, click the plus (+) button shown at the bottom of the previous screenshot. I’ll be using an OpenAI chat model, but feel free to experiment with others to see which best suits your use case.

Configuration will vary depending on the model you choose. In most cases, you’ll need to provide an API key. Refer to n8n’s documentation for setup instructions specific to your model.

Once configured, execute the step to ensure the model returns a clear and relevant product title for the input provided:

Searching eBay via SerpApi

Next, we'll use eBay to get a list of recent sales for our product to determine a fair price.

From the SerpApi node, add the Search eBay action to the workflow. From the parameters menu, we'll only need to set the following:

Search Query: {{ $json.output }} - This will pass the output from our AI Agent step as the query parameter.
Results filter (additional field): Sold - It is important that this parameter is capitalized. This parameter ensures that we only get results back for recently sold items.
Disable Caching (optional): true - SerpApi caches queries for 1 hour after they are performed. Meaning, a cached search will be returned if you perform another search with the same exact search parameters within an hour of the initial search. I've chosen to disable this parameter to ensure the freshest data possible. Once configured, our node should look like this:

Now, let's go ahead and test the step by executing the step:

Saving the Data

n8n offers a multitude of options for working with and saving data returned by the eBay API. You can write directly to a SQL database, push data to cloud-based data warehouses like Snowflake, or simply log it to a Google Sheet. From this point forward, I would recommend exploring n8n to best fit your specific use case, but in this example, we’ll be using Google Sheets.

For each sold item returned, we want to create a row in our sheet to make it easy to filter and sort information such as the title, condition, and price.

First, we need to extract the organic results from the eBay API response using a Code node. I’ve used JavaScript here, but you can also use Python:

const results = $input.first().json.organic_results

return results

This will output each result individually so we can pass them to the Google Sheets node.

Next, configure the Google Sheets node by selecting the “Append row in sheet” operation. Choose the document and sheet you want to write to, and make sure you’ve already added columns for each data point you plan to include. In this example, I’ve added Title, Link, Condition, and Price.

Finally, map the appropriate attributes from n8n to the corresponding columns in your Google Sheet.

If all went well, all of our results should have been written to a Google Sheet:

Wrap-Up

In Part 2, we completed our journey from an image to actionable data. We used an AI agent to refine the search results and pinpoint the exact product name, then tapped into eBay’s data to find its market price. At this point, our “Visual Search to Price Tracking” workflow is fully functional: you can POST an image and get back the likely item name and its price on eBay within seconds.

Ideas for Scaling Further: Now that the basic workflow is done, you can enhance it even more:

Batch Processing: Have multiple images? You could modify the workflow to handle an array of images in one go, or simply trigger it multiple times. n8n could even watch an email or folder for new images and process them automatically.
Scheduled Price Checks: As mentioned, you could use a Scheduler (Cron) node to periodically run the eBay search for saved item names, building a daily price tracker. This might help you notice trends (e.g., prices going up or down over time).
Alerts for Price Drops or Increases: With a slight extension, you could compare the latest price to a previous one (store the last known price in a workflow variable or database). If the price drops by a certain percentage, have n8n send you an alert – useful for deciding when to sell or buy more.
Integration into Listing Workflow: If you are a seller, you could integrate this into your listing creation process. For example, when you want to list a new item, run it through this workflow to get the suggested title and price, then automatically draft an eBay listing or a listing in your inventory system with those details.

We hope this two-part tutorial showed the power of combining AI, web scraping and automation. By chaining these tools (image recognition, language understanding, and web data retrieval), you can automate typically monotonous manual tasks. Whether you’re a card collector pricing out your collection or an online seller researching products, this workflow can save you a ton of time.

Fetching AI Overviews with Node.js

Nathan Skiles — Fri, 11 Jul 2025 01:14:34 +0000

Fetching AI Overviews with Node.js

AI Overviews are quickly becoming a prominent feature in today’s SEO landscape, and with SerpApi, extracting this data is simple.

While Google hasn’t shared many details about how sources are selected for AI Overviews, early signs suggest that they rely on many of the same SEO principles already in use. This makes tracking AI Overviews a valuable addition to your SEO strategy, offering insight into how users discover and interact with your content.

By targeting the correct queries and optimizing your content accordingly, you may improve your chances of being cited in an AI Overview. And although SerpApi simplifies data collection, there are a few common pitfalls to be aware of. In this post, I’ll walk through what to expect and how to avoid issues during implementation.

Getting Started

For this guide, I'll be using SerpApi's Node.js package (documentation) to retrieve AI Overviews. That said, the same principles apply no matter which programming language you prefer.

I also recommend reviewing the SerpApi Google Search API and AI Overview API documentation to get more familiar with the endpoints and data structure.

Requirements

SerpApi API Key - If you don’t have an account yet, you can sign up for a free plan with 100 searches per month. If you already have an account, your API key is available in your dashboard.
Node.js 7.10.1 or newer
SerpApi Node.js Package - You can install it using npm install serpapi

Search Parameters

AI Overviews are returned for some queries to SerpApi's Google Search API. To get started, let’s define the search parameters:

const searchParams = {
  engine: "google",
  q: query,
  api_key: API_KEY,
  no_cache: true,
};

Here’s what each parameter does:

engine : "google" - Tells SerpApi to use the standard Google Search engine.
q: query - The search query to submit.
api_key: API_KEY - Your SerpApi key for authentication.
no_cache: true - Forces SerpApi to fetch fresh data from Google instead of returning a cached result.

If no_cache is not used, you may receive a cached response that includes an expired page_token. Always use no_cache: true when you intend to follow up with a second request using the page_token.

Perform the Initial Search and Check for AI Overview

const searchResponse = await getJson(searchParams);
const overviewData = searchResponse.ai_overview;

This performs the initial search and attempts to extract the ai_overview object from the response. If the query triggers an AI Overview, the overviewDatathe variable will contain its content; otherwise, it will be undefined.Not all searches return the full AI Overview here. Instead, some may only include a page_token, which is a temporary reference used to fetch the actual AI Overview in a second request.

Check for page_token and Fetch the Full AI Overview

if (overviewData.page_token) {
  const overviewParams = {
    engine: "google_ai_overview",
    api_key: API_KEY,
  };

  overviewParams.page_token = overviewData.page_token;
  const ai_overview_response = await getJson(overviewParams);

  overviewResult = ai_overview_response.ai_overview;
} else {
  overviewResult = overviewData;
}

This checks whether the initial response includes a page_token. If it does:

A second request is made using the google_ai_overview engine and the page_token.
The full AI Overview is returned and stored in overviewResult.

If no token is found, it assumes the AI Overview was included in the original response.

page_token values are short-lived, expiring within ~4 minutes. You must make the follow-up request immediately.

Full Example

In the example below, we iterate over an array of queries, performing a Google Search for each one and checking if an AI Overview is returned.

If an AI Overview is present, we then check whether it includes a page_token. If it does, we make a follow-up request to the google_ai_overview engine using that token to retrieve the full AI Overview.

import { getJson } from "serpapi";
import dotenv from "dotenv";
dotenv.config();

const API_KEY = process.env.SERPAPI_KEY;

const queries = [
  "what makes a good web hosting service",
  "best crm platform",
  "top programming languages to learn in 2025",
];

const fetchAIOverview = async (query) => {
  const searchParams = {
    engine: "google",
    q: query,
    api_key: API_KEY,
    no_cache: true,
  };

  try {
    const searchResponse = await getJson(searchParams);
    const overviewData = searchResponse.ai_overview;
    let overviewResult;

    if (!overviewData) {
      console.log("No AI overview available.");
      return;
    }

    if (overviewData.page_token) {
      const overviewParams = {
        engine: "google_ai_overview",
        api_key: API_KEY,
      };
      console.log("Page token found. Fetching AI overview page...");
      overviewParams.page_token = overviewData.page_token;
      const ai_overview_response = await getJson(overviewParams);
      console.log("AI overview retrieved.");
      overviewResult = ai_overview_response.ai_overview;
    } else {
      console.log("AI overview retrieved.");
      overviewResult = overviewData;
    }

    console.log("AI Overview:\n", overviewResult);
  } catch (error) {
    console.error("Error fetching data from SerpAPI:", error);
  }
};

queries.forEach((query) => {
  console.log(`Fetching AI overview for query: "${query}"`);
  fetchAIOverview(query);
});

Best Practices Summary

Use no_cache: true to avoid stale results and expired tokens.
Make the follow-up request to google_ai_overview immediately if a page_token is present.
Process each query sequentially, not in parallel, to avoid expired page_tokens.

Resources

For a high-level introduction to the Google AI Overviews API, I recommend checking out the following blog post:

How to Scrape Google AI Overviews (AIO)

If you have questions or run into any issues with AI Overviews, don’t hesitate to reach out to SerpApi’s support team at contact@serpapi.com

Unlocking SEO Insights: Leveraging 'People Also Ask' for Smarter Content Strategies

Nathan Skiles — Thu, 27 Feb 2025 22:09:27 +0000

Creating content that resonates with your target audience requires more than just keyword research and competitor analysis. Gaining insight into your audience’s questions, concerns, and information-seeking behaviors can provide a deeper understanding of their needs. This knowledge helps in crafting content that directly addresses their interests and challenges.

The "People Also Ask" (PAA) boxes have become a prominent feature in Google's search engine results pages (SERPs), which are the pages displayed by search engines in response to a user's query. These dynamic elements offer valuable insights into user intent and reveal emerging content opportunities that can enhance any content strategy.

Understanding People Also Ask

The People Also Ask feature represents Google's effort to provide users with additional, relevant questions related to their search queries. These expandable boxes appear within search results and contain a series of questions that users frequently ask about related topics. When a user clicks on a question, the box expands to reveal a brief answer, typically extracted from a high-ranking webpage.

Expandable "People also ask" box

What makes PAA boxes particularly valuable is their dynamic nature. For example, if a user searches for "content strategy" and clicks on a PAA question like "What are the steps to creating a content strategy?", new related questions will appear, such as "How do you measure the success of a content strategy?" or "What tools help with content strategy development?" This continuous expansion of queries provides deeper insights into user search intent. This behavior reflects real user interests and search patterns, providing authentic insights into an audience's thought processes and information needs.

While PAA data is often associated with straightforward Q&A content, it also uncovers:

The various angles and perspectives from which users research a topic.
Common misconceptions or concerns that need addressing.
Related topics that users frequently explore together.
The level of expertise in their target audience's questions.

Analyzing PAA data enables content strategists to craft detailed plans that target key search terms, cover topics comprehensively, and guide user journeys. This approach aligns with Google's quality guidelines, which prioritize comprehensive topic coverage and demonstrable subject expertise. In doing so, it paves the way for improved search visibility and engagement.

Transforming PAA Insights into Content

Converting PAA data into actionable content strategies requires a systematic approach. When analyzing PAA questions, start by identifying patterns in user inquiries. For instance, if you're creating content about digital marketing, you'll often notice a progression—from basic definitional queries to more complex implementation questions. This progression can help structure your content, guiding readers from foundational concepts to advanced applications.

Consider a real-world example: a software company launching a new project management tool. By analyzing PAA data related to "project management software," they might discover that users frequently ask questions about integration capabilities, pricing models, and team collaboration features. This intelligence helps prioritize which aspects of the product to highlight in their content marketing efforts. By leveraging PAA data, content teams can take several concrete steps:

Build detailed FAQ sections that proactively address common audience queries.
Structure blog posts and articles to naturally answer related questions.
Create content that covers topics exhaustively.
Spot content gaps to uncover overlooked topics and opportunities for expansion.

These strategies help ensure your content directly addresses user needs and fills critical gaps in your existing offerings.

The Role of PAA in SEO Strategy

Incorporating PAA insights into an SEO strategy can improve search visibility and audience engagement. By targeting PAA questions, content creators can:

Increase their chances of ranking for featured snippets.
Improve content relevance and user engagement.
Enhance topic authority and site credibility.
Drive organic traffic by addressing real search queries.

Optimizing content for PAA involves structuring answers concisely, using schema markup, and ensuring that content provides direct, valuable responses to user queries. Additionally, updating content based on newly emerging PAA questions can help maintain relevance and rankings over time.

Implementing PAA Insights with SerpApi

SerpApi is a powerful tool designed to scrape and extract structured data from live search engine results pages, including the "People Also Ask" feature. Using SerpApi, businesses and SEO professionals can automate the collection of PAA data, ensuring continuous insights into evolving user queries.

With real-time search results and an easy-to-use API, SerpApi eliminates the need for manual research or time-intensive in-house solutions. This allows content teams to focus on strategy and execution rather than data collection.

Google Search API

SerpApi’s Google Search API enables users to extract various search result elements, including organic listings, ads, featured snippets, and the People Also Ask section. By integrating this API, businesses can track PAA questions dynamically, monitor changes, and identify emerging trends in user queries. This allows content creators to refine their SEO and content strategies with up-to-date information.

Google Search API Documentation

The remainder of this post will focus specifically on the Google Related Questions API, I highly recommend checking out the following blog post to get similar with SerpApi and the Google Search API:

How to Scrape Google Search Results (SERPs) - 2025 Guide

Additionally, it's worth performing test queries in our playground environment to get a better idea of the results returned as well as the structure of data returned from both the Google Search and Related Questions APIs:

SerpApi Playground - SerpApi

The Google Related Questions API specifically focuses on retrieving PAA data. This API provides structured data on related questions that users frequently ask, along with their respective answers. By leveraging this API, content strategists can:

Identify trending PAA questions for their niche.
Monitor how PAA questions evolve over time.
Optimize content to align with high-impact queries.

Using SerpApi’s capabilities, businesses can efficiently gather, analyze, and implement PAA insights into their SEO and content marketing efforts.

Google Related Questions API - SerpApi

Google Related Questions API Documentation

Fetching PAA Questions Using SerpApi

In this section, we will walk through using Node.js to fetch PAA questions using SerpApi's Google Search API and Google Related Questions API. I will use JavaScript, but the examples can be easily translated into other programming languages. You can find a complete list of SerpApi integrations here:

SerpApi: Integrations

Step 1: Install SerpApi’s JavaScript Package

While you can use any of your favorite packages to make API calls, we will use the SerpApi Node.js package (documentation) to keep things clean and consistent.

npm install serpapi

Step 2: Fetch Data Using Google Search API

PAA questions appear organically in Google Search results, so to start with, we can query the Google Search API to return a typical SERP.

Below, we perform a search using the google engine, and our query is "content strategy." We then store the related questions returned in an array that we can add to later.

import { getJson } from "serpapi";

const response = await getJson({
  engine: "google",
  api_key: "YOUR_API_KEY",
  q: "content strategy"
});

// Store PAA questions
const relatedQuestionsArray = response["related_questions"]);

While related questions are returned for most queries, I recommend adding some more robust error handling for edge cases where no related questions are returned.

Example Response:

{
  ...
  "related_questions": [
    {
      "question": "What are the 7 steps in creating a content strategy?",
      "snippet": ""
      "title": "7-step content development process: step-by-step guide",
      "date": "Jun 17, 2024",
      "link": "https://contentsnare.com/content-development-process/",
      "list": [
        "Step 1: Do your research. ... ",
        "Step 2: Analyze the information. ... ",
        "Step 3: Plan your strategy. ... ",
        "Step 4: Write. ... ",
        "Step 5: Editing, SEO and publishing. ... ",
        "Step 6: Take to social media. ... ",
        "Step 7: Analyze and begin again."
      ],
      "displayed_link": "https://contentsnare.com › content-development-process",
      "source_logo": "https://serpapi.com/searches/67bf85dac71a675e242edf6b/images/b63f19fbb9fff39987251b02abdc8c2d341978702805ce28c8d754c35cf0eb70.png",
      "next_page_token": "eyJvbnMiOiIxMDA0MSIsImZjIjoiRXFFQkNtSkJRUzFMVkdoa1...",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google_related_questions&google_domain=google.com&next_page_token=eyJvbnMiOiIxMDA0MSIsImZ..."
    }
    ...
  ]
  ...
}

Typically, 3-4 PAA questions are returned on any given Google SERP. However, as mentioned previously, expanding these questions returns additional related questions.

Each question returned in Step 2 will include a next_page_token, which we can use to retrieve additional questions related to the question using the Google Related Questions API.

const relatedResponse = await getJson({
  engine: "google_related_questions",
  api_key: "YOUR_API_KEY",
  next_page_token: "NEXT_PAGE_TOKEN"
});

// Save additional question data
const additionalQuestions = relatedResponse["related_questions"]);

The response to this query will be similar to the one seen in Step 2, returning next_page_tokens for each additional question.

By automating the extraction of PAA insights, businesses can refine their SEO approach, ensuring content aligns with what users are actively searching for.

Example Node.js CLI App

To get you started, I've created a sample CLI app that allows you to enter a query and specify the depth at which you would like to retrieve PAA questions.

The tool takes your search query, sends it to the Google Search API, and extracts the "People Also Ask" questions from the search results. For each question, it recursively fetches additional related questions up to the specified depth.

The output displays the relationship between questions in a tree structure:

1 → First level question
1.2 → Second question that stems from the first question
2.1.3 → Third question that stems from the first child of the second question

GitHub - NateSkiles/paa-example

Measuring Success

After implementing PAA insights into your content and SEO strategy, measuring success is crucial. Key performance indicators (KPIs) to track include:

Organic traffic growth: Increased visits from search engines.
SERP rankings: Improved positions for targeted queries.
Click-through rates (CTR): Higher engagement on PAA-optimized content.
User engagement metrics: Time on page, bounce rate, and session duration.
Conversions: Lead generation and customer inquiries driven by content.

Regularly updating content based on fresh PAA insights ensures that your strategy remains effective and aligned with user search behavior.

Conclusion

The "People Also Ask" feature is a powerful tool for understanding user intent and enhancing content strategies. By leveraging SerpApi’s capabilities, marketers and SEO professionals can efficiently extract and analyze PAA data to create highly relevant, user-focused content.

By systematically integrating PAA insights, businesses can improve their search visibility, attract more organic traffic, and establish authority in their niche. Leveraging these insights allows content creators to address user queries effectively, optimize for featured snippets, and provide comprehensive information that enhances engagement. Regularly refining content based on PAA data ensures continued relevance and strengthens a brand’s presence in search results. Whether you're developing blog posts, FAQs, or in-depth guides, PAA-driven content strategies can set your brand apart in an increasingly competitive digital landscape.

No-code Solutions for Turning Search Results Into Markdown for LLMs

Nathan Skiles — Thu, 27 Feb 2025 21:56:35 +0000

Intro
In my last post, I walked through a coding solution in which we used Node.js to scrape webpages returned in Google Search results and parse them to Markdown for use in LLMs or other use cases. Today, we will do something similar, but we will utilize no-code solutions to return results from SerpApi's Google Search API and parse to markdown using the Reader API by Jina.ai.

Turning Search Results Into Markdown for LLMs
Intro This article will walk through converting search results into Markdown format, suitable for use in large language models (LLMs) and other applications. Markdown is a lightweight markup language that provides a simple, readable way to format text with plain-text syntax. Checkout Markdown Guide for more information: Markdown GuideA free

SerpApi
Nathan Skiles

Reader by Jina
Jina AI’s Reader API (https://jina.ai/reader/) offers an alternative for extracting structured content from web pages, maintaining document structure, and handling diverse content types.

💡
Jina AI provides 1 million test credits to get started, but keep in mind that tokens are used based on the amount of content parsed, not the number of requests made.
Scraping Search Results
Several options integrate well with SerpApi for those who prefer no-code solutions. This section will walk through configuring Make and Zapier to query search results through SerpApi.

Make
Starting with Make, we will create a Google Doc from the data returned by SerpApi and the Reader API. This is a relatively simple example, but Make provides many integrations with AI providers such as OpenAI, Anthropic, and more. Instead of sending the data to a Google Doc, you can replace the last module and send it wherever you need it!

SerpApi has an official Make app, making it much more straightforward than Zapier to configure. While I will only be walking through how to configure the SerpApi Google Search module, for more information on our Make app, check out this blog post:

Announcing SerpApi’s Make App
We are pleased to announce the release of our Custom App on Make.com. This makes it easier than ever to integrate SerpApi with other systems without having to write any code.

SerpApi
Alex Barron

Create a new scenario and search for the SerpApi app to get started. Select your preferred search engine module; we will use Google for this example.

After selecting your search engine, we can start configuring our call to SerpApi. You'll add your SerpApi API key by creating a connection. Click "Create a connection."

Enter your API key and name your connection:

From here, we can configure our query:

You can fetch additional pages by increasing the “Pagination Limit” setting but note that each page will cost 1 SerpApi search credit.

Click “Run once” and verify the data coming back from SerpApi:

Now that we have results we can work with, let’s add an “Iterator” module to allow us to use the data returned in the organic_results array. The Iterator module (documentation) converts arrays into a series of bundles. Each element in the array will output as a separate bundle, allowing us to query the Reader API for each link returned in our Google results.

To query the Reader API for each link returned, let’s add an HTTP “Make a request” module that allows us to send HTTP requests.

Set the URL parameter to https://r.jina.ai/, then click the box again to pull up a list of available data we can pass to this parameter. From our Iterator module, select the “link” attribute. This passes the link from our organic Google search results to the Reader URL. Here’s what that should look like:

Now, we need to set our header for authorization by passing our Bearer token provided by Jina AI:

Before running again, let’s add our final step to create a Google Doc to store the results. Again, this final step can be changed depending on your use case and where you wish to send the data.

Add the Google Docs “Create a Document” module and add a connection to your Google account. Now you can set a name for the document as well as pass data from the Reader API as the content of the doc:

Now lets run our scenario once more to ensure that everything is working correctly.

💡
While testing, we can set a limiter filter after the Iterator module to prevent the flow from continuing more than once.

Click the wrench icon under the link between the Iterator and HTTP modules. Create a condition only to continue if the Iterator’s “position” attribute is numerically equal to “1”:

Make sure to remove the limiter before publishing if you wish to query for all the results. Alternatively, you set the num parameter in the SerpApi Search Google module to limit results returned by SerpApi.

Zapier
While this article is focused on no-code solutions, SerpApi currently does not have a Zapier integration. Meaning we must use a little code to make our API requests to SerpApi. I have provided a code below that you can drop into Zapier Code steps without issue, though some adjustments may be needed to fit your use case better. I recommend starting with the code I've provided and then adjusting once you get results from SerpApi and the Reader API.

In this section, we will walk through:

Use the Schedule Trigger to initiate Zap
Retrieve search results URLs from SerpApi
Process webpages from URLs through Reader API to Markdown
Send formatted content to your desired apps
As this Zap requires API calls to both SerpApi and the Reader API, you will likely need a Trial or Professional/Team plan. Zapier limits code runtime to 1 second on the free plan, and both API requests will likely take longer (documentation).

To get started, add a trigger to your Zap. For this post, we will use the Schedule by Zapier trigger to our Zap every day at 12:00 a.m. You can select a trigger that best fits your needs.

💡
Make sure to test each step before continuing to ensure you've created test data for future steps. Zapier will typically force you to test a step after changes before publishing.

Next, we add a Code by Zapier action to make our request to SerpApi. This example uses JavaScript, but you can choose Python if you prefer to use the code below as a guideline.

I've included the JavaScript code for this action below:

const { SERPAPI_KEY, QUERY, LOCATION } = inputData;

async function fetchGoogleSearchResults(query, location) {
try {
const response = await fetch(
https://serpapi.com/search.json?engine=google&q=${encodeURIComponent( query )}&location=${encodeURIComponent( location )}&gl=us&hl=en&api_key=${SERPAPI_KEY}
);

if (!response.ok) {
  throw new Error(`HTTP error! status: ${response.status}`);
}

const data = await response.json();

if (!data) {
  throw new Error("No data returned from SerpAPI");
}

return data;

} catch (error) {
// Instead of just logging, throw the error
throw new Error(SerpAPI request failed: ${error.message});
}
}

try {
const searchResults = await fetchGoogleSearchResults(QUERY, LOCATION);

output = {
organicResults: searchResults?.organic_results || "No Organic Result Found"
};

return output;
} catch (error) {
console.error("Error:", error.message);
// Return a default response or throw error based on your needs
return { organic_results: [] };
}
After adding the above code to the action, add your input data:

SERPAPI_KEY - your SerpApi key goes here. You can find your API key in your SerpApi dashboard.
QUERY - query parameter for the search you wish to perform.
LOCATION - location parameter for the search you wish to perform.
If you wish to add additional search parameters, you can set them as Input Data or update the URL in the code directly. Setting them as Input Data allows you to update them more easily in the future.

Your first code step should now look something like this:

Our request from SerpApi will return an array of links for our organic results. To send each link to the Reader API, we must iterate through the links using the Looping by Zapier action.

This action takes an array of values and runs the actions within the loop for each value. For our "Values to Loop," we can pass the "Organic Results Link" values from our second step to a key called "links".

Let's also set the maximum number of loop iterations to 1 for testing to ensure we aren't spending Reader API credits needlessly.

Inside the Looping by Zapier action, we can add another Code by Zapier action to make our request to the Reader API. Paste the JavaScript below in the code section of this action:

const { link, READERAPI_KEY } = inputData;

async function fetchReaderData(link, apiKey) {
try {
// Encode URL for path parameter
const encodedUrl = encodeURIComponent(link);

const response = await fetch(`https://r.jina.ai/${encodedUrl}`, {
  method: 'GET',
  headers: {
    'Authorization': `Bearer ${apiKey}`,
    'Accept': 'application/json'
  }
});

if (!response.ok) {
  throw new Error(`HTTP error! status: ${response.status}`);
}

const data = await response.json();

if (!data) {
  throw new Error('No data returned from Reader API');
}

return data;

} catch (error) {
console.error('Reader API request failed:', error.message);
throw error;
}
}

try {
const readerResults = await fetchReaderData(link, READERAPI_KEY);

output = {
readerResults
};

return output;
} catch (error) {
return {
error: error.message,
content: null,
title: null,
metadata: null
};
}
Again, set your Input Data for this action:

link - the value of the link from our Looping action.
READERAPI_KEY - your Jina AI Reader API key, found on the Reader API website.
Once complete, your second Code by Zapier action should look like this:

Finally, we can send our data from the Reader API to an app. For this example, we will use Google Docs, but feel free to explore Zapier's integrations and find the action that best fits your use case. I recommend trying with Google Docs first to better understand the output data.

Add the "Create Document From Text in Google Docs" action to your Loop. This will create a new Google Doc for each URL the Loop processes. Feel free to adjust as needed.

If you have not done so previously, you may need to authorize Zapier to access your Google account. Once complete, your Setup tab should look something like this:

Click "Continue" or navigate to the "Configure" tab. Here, we can set a name for the document and the Google Drive folder in which you would like to save the document and pass the content returned from the Reader API to the document's content.

And you're all set! Make sure to test the Zap thoroughly to ensure there are no issues, and check your Google Drive to ensure the document was created.

Conclusion
No-code tools such as Make and Zapier provide powerful tools for users who might not be comfortable writing or maintaining scripts necessary to scrape valuable data found in search results. While SerpApi isn't necessarily a no-code solution, it abstracts away many complex concepts required to scrape search engines. When paired with no-code tools, even non-technical users can leverage this data.

No-code Solutions for Turning Search Results Into Markdown for LLMs

Nathan Skiles — Wed, 08 Jan 2025 22:59:36 +0000

Intro

In my last post, I walked through a coding solution in which we used Node.js to scrape webpages returned in Google Search results and parse them to Markdown for use in LLMs or other use cases. Today, we will do something similar, but we will utilize no-code solutions to return results from SerpApi’s Google Search API and parse to markdown using the Reader API by Jina.ai.

Turning Search Results Into Markdown for LLMs ↗

Reader by Jina

Jina AI’s Reader API (https://jina.ai/reader/) offers an alternative for extracting structured content from web pages, maintaining document structure, and handling diverse content types.

💡 Jina AI provides 1 million test credits to get started, but keep in mind that tokens are used based on the amount of content parsed, not the number of requests made.

Scraping Search Results

Several options integrate well with SerpApi for those who prefer no-code solutions. This section will walk through configuring Make and Zapier to query search results through SerpApi.

Make

Starting with Make, we will create a Google Doc from the data returned by SerpApi and the Reader API. This is a relatively simple example, but Make provides many integrations with AI providers such as OpenAI, Anthropic, and more. Instead of sending the data to a Google Doc, you can replace the last module and send it wherever you need it!

Announcing SerpApi’s Make App ↗

Create a new scenario and search for the SerpApi app to get started. Select your preferred search engine module; we will use Google for this example.

After selecting your search engine, we can start configuring our call to SerpApi. You’ll add your SerpApi API key by creating a connection. Click “Create a connection.”

Enter your API key and name your connection:

From here, we can configure our query:

You can fetch additional pages by increasing the “Pagination Limit” setting but note that each page will cost 1 SerpApi search credit.

Click “Run once” and verify the data coming back from SerpApi:

Now that we have results we can work with, let’s add an “Iterator” module to allow us to use the data returned in the organic_results array. The Iterator module (documentation) converts arrays into a series of bundles. Each element in the array will output as a separate bundle, allowing us to query the Reader API for each link returned in our Google results.

To query the Reader API for each link returned, let’s add an HTTP “Make a request” module that allows us to send HTTP requests.

Set the URL parameter to https://r.jina.ai/, then click the box again to pull up a list of available data we can pass to this parameter. From our Iterator module, select the “link” attribute. This passes the link from our organic Google search results to the Reader URL. Here’s what that should look like:

Now, we need to set our header for authorization by passing our Bearer token provided by Jina AI:

Before running again, let’s add our final step to create a Google Doc to store the results. Again, this final step can be changed depending on your use case and where you wish to send the data.

Now lets run our scenario once more to ensure that everything is working correctly.

While testing, we can set a limiter filter after the Iterator module to prevent the flow from continuing more than once.

Click the wrench icon under the link between the Iterator and HTTP modules. Create a condition only to continue if the Iterator’s “position” attribute is numerically equal to “1”:

Make sure to remove the limiter before publishing if you wish to query for all the results. Alternatively, you set the num parameter in the SerpApi Search Google module to limit results returned by SerpApi.

Zapier

While this article is focused on no-code solutions, SerpApi currently does not have a Zapier integration. Meaning we must use a little code to make our API requests to SerpApi. I have provided a code below that you can drop into Zapier Code steps without issue, though some adjustments may be needed to fit your use case better. I recommend starting with the code I’ve provided and then adjusting once you get results from SerpApi and the Reader API.

In this section, we will walk through:

Use the Schedule Trigger to initiate Zap
Retrieve search results URLs from SerpApi
Process webpages from URLs through Reader API to Markdown
Send formatted content to your desired apps

As this Zap requires API calls to both SerpApi and the Reader API, you will likely need a Trial or Professional/Team plan. Zapier limits code runtime to 1 second on the free plan, and both API requests will likely take longer (documentation).

To get started, add a trigger to your Zap. For this post, we will use the Schedule by Zapier trigger to our Zap every day at 12:00 a.m. You can select a trigger that best fits your needs.

💡 Make sure to test each step before continuing to ensure you’ve created test data for future steps. Zapier will typically force you to test a step after changes before publishing.

Next, we add a Code by Zapier action to make our request to SerpApi. This example uses JavaScript, but you can choose Python if you prefer to use the code below as a guideline.

I’ve included the JavaScript code for this action below:

const { SERPAPI_KEY, QUERY, LOCATION } = inputData;

async function fetchGoogleSearchResults(query, location) {
  try {
    const response = await fetch(
      `https://serpapi.com/search.json?engine=google&q=${encodeURIComponent(
        query
      )}&location=${encodeURIComponent(
        location
      )}&gl=us&hl=en&api_key=${SERPAPI_KEY}`
    );
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    const data = await response.json();
    if (!data) {
      throw new Error("No data returned from SerpAPI");
    }
    return data;
  } catch (error) {
    // Instead of just logging, throw the error
    throw new Error(`SerpAPI request failed: ${error.message}`);
  }
}

try {
  const searchResults = await fetchGoogleSearchResults(QUERY, LOCATION);
  output = {
    organicResults: searchResults?.organic_results || "No Organic Result Found"
  };
  return output;
} catch (error) {
  console.error("Error:", error.message);
  // Return a default response or throw error based on your needs
  return { organic_results: [] };
}

After adding the above code to the action, add your input data:

SERPAPI_KEY - your SerpApi key goes here. You can find your API key in your SerpApi dashboard.
QUERY - query parameter for the search you wish to perform.
LOCATION - location parameter for the search you wish to perform.

If you wish to add additional search parameters, you can set them as Input Data or update the URL in the code directly. Setting them as Input Data allows you to update them more easily in the future.

Your first code step should now look something like this:

Our request from SerpApi will return an array of links for our organic results. To send each link to the Reader API, we must iterate through the links using the Looping by Zapier action.

This action takes an array of values and runs the actions within the loop for each value. For our “Values to Loop,” we can pass the “Organic Results Link” values from our second step to a key called “links”.

Let’s also set the maximum number of loop iterations to 1 for testing to ensure we aren't spending Reader API credits needlessly.

Inside the Looping by Zapier action, we can add another Code by Zapier action to make our request to the Reader API. Paste the JavaScript below in the code section of this action:

const { link, READERAPI_KEY } = inputData;

async function fetchReaderData(link, apiKey) {
  try {
    // Encode URL for path parameter
    const encodedUrl = encodeURIComponent(link);
    const response = await fetch(`https://r.jina.ai/${encodedUrl}`, {
      method: 'GET',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Accept': 'application/json'
      }
    });
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    const data = await response.json();
    if (!data) {
      throw new Error('No data returned from Reader API');
    }
    return data;
  } catch (error) {
    console.error('Reader API request failed:', error.message);
    throw error;
  }
}

try {
  const readerResults = await fetchReaderData(link, READERAPI_KEY);
  output = {
    readerResults
  };
  return output;
} catch (error) {
  return {
    error: error.message,
    content: null,
    title: null,
    metadata: null
  };
}

Again, set your Input Data for this action:

link - the value of the link from our Looping action.
READERAPI_KEY - your Jina AI Reader API key, found on the Reader API website.

Once complete, your second Code by Zapier action should look like this:

Finally, we can send our data from the Reader API to an app. For this example, we will use Google Docs, but feel free to explore Zapier’s integrations and find the action that best fits your use case. I recommend trying with Google Docs first to better understand the output data.

Add the “Create Document From Text in Google Docs” action to your Loop. This will create a new Google Doc for each URL the Loop processes. Feel free to adjust as needed.

If you have not done so previously, you may need to authorize Zapier to access your Google account. Once complete, your Setup tab should look something like this:

Click “Continue” or navigate to the “Configure” tab. Here, we can set a name for the document and the Google Drive folder in which you would like to save the document and pass the content returned from the Reader API to the document’s content.

And you’re all set! Make sure to test the Zap thoroughly to ensure there are no issues, and check your Google Drive to ensure the document was created.

Conclusion

No-code tools such as Make and Zapier provide powerful tools for users who might not be comfortable writing or maintaining scripts necessary to scrape valuable data found in search results. While SerpApi isn’t necessarily a no-code solution, it abstracts away many complex concepts required to scrape search engines. When paired with no-code tools, even non-technical users can leverage this data.