DEV Community: Nathan Skiles

Searching eBay with AI and Automation using n8n

Nathan Skiles — Mon, 04 Aug 2025 23:33:55 +0000

Welcome to Part 2 of our tutorial series. Previously (Part 1), we created an automated workflow that takes an image and returns potential item names using Google’s visual search.

In case you missed it, you can find Part 1 here:

Uploading Images and Searching with Google Lens via SerpApi

In this tutorial, we’ll build an automation that identifies an item using visual search. By leveraging Google Lens (via SerpApi) for image recognition, you can quickly determine the exact name of a card or collectible.

serpapi.com

Now, we’ll close the loop by integrating AI and price lookup. The goal for this second part is to automatically figure out which of those candidate titles is the correct one for our item, and then find how much it’s selling for on eBay.

We will use AI to pick the best title, and then use SerpApi’s eBay Search API to fetch live pricing data. We’ll also touch on ways to log or store this info for future use.

What You’ll Need

n8n: the same n8n setup from Part 1. The workflow will be extended, so have that ready.
SerpApi API key: continue using your SerpApi account for the eBay search step. (No new setup if you have it from Part 1.)
OpenAI API Key: an OpenAI account (or another supported LLM in n8n). You’ll input this in n8n’s credentials so the AI Agent node can access the model.
(optional) Google Sheet or Database: If you want to store the results (item names and prices) for record-keeping or price tracking, you can prepare a Google Sheet or database. This is optional, but we’ll discuss how you could send data to Google Sheets as an example.

Workflow Continuation

Now we will extend the workflow from Part 1. After extracting the list of possible titles, we’ll add steps for AI processing and eBay lookup. Referring to the workflow diagram (from Part 1) for context, our new nodes will come after the “Extract Visual Match Titles” step.

Your workflow should currently look like this:

Using the AI Agent to Refine Results

n8n offers a variety of AI nodes, with capabilities ranging from text summarization to classification. For our use case, we’ll use the AI Agent node, which lets you attach multiple chat models and tools to the agent.

docs.n8n.io

We’ll use the AI Agent to determine the most likely product name based on the title list we generated in the previous blog post.

Add the AI Agent node to your workflow and set the Source for Prompt option to Define below. This allows us to pass our title list to the chatbot and provide it with context.

For the prompt (user message), I’ve used the following:

Here are the top visual matches from a Google Lens search for a product image: {{ $json.titlesList }}

Return the most likely product title from these matches that best describes the specific item in the image.

Do not include bundle listings, condition, or unrelated items.

Next, add a System Message (not shown by default) to give the chatbot additional instructions:

You are an AI assistant that specializes in identifying and generating accurate, search-optimized product titles from visual search data.

Given a list of titles extracted from a reverse image search, return the most likely full product title for the specific item in the image.

Your output should be a clean, concise string that includes the key identifiers (brand, model, year, type, edition, etc.) — formatted like a product listing title someone might search for on a marketplace (e.g., eBay or Amazon).

Only return the title — no explanations or extra words.

Do not include specific model numbers unless they are relevant to the product. For example:

Sony PlayStation 5 Digital Edition Console (Model CFI-1218B)

Can just be:

Sony PlayStation 5 Digital Edition Console

Your AI Agent parameters should now look like this:

We’re not quite done configuring the AI Agent. We still need to add a Chat Model, along with any tools that might help the AI Agent process the data.

To add a Chat Model, click the plus (+) button shown at the bottom of the previous screenshot. I’ll be using an OpenAI chat model, but feel free to experiment with others to see which best suits your use case.

Configuration will vary depending on the model you choose. In most cases, you’ll need to provide an API key. Refer to n8n’s documentation for setup instructions specific to your model.

Once configured, execute the step to ensure the model returns a clear and relevant product title for the input provided:

Searching eBay via SerpApi

Next, we'll use eBay to get a list of recent sales for our product to determine a fair price.

From the SerpApi node, add the Search eBay action to the workflow. From the parameters menu, we'll only need to set the following:

Search Query: {{ $json.output }} - This will pass the output from our AI Agent step as the query parameter.
Results filter (additional field): Sold - It is important that this parameter is capitalized. This parameter ensures that we only get results back for recently sold items.
Disable Caching (optional): true - SerpApi caches queries for 1 hour after they are performed. Meaning, a cached search will be returned if you perform another search with the same exact search parameters within an hour of the initial search. I've chosen to disable this parameter to ensure the freshest data possible. Once configured, our node should look like this:

Now, let's go ahead and test the step by executing the step:

Saving the Data

n8n offers a multitude of options for working with and saving data returned by the eBay API. You can write directly to a SQL database, push data to cloud-based data warehouses like Snowflake, or simply log it to a Google Sheet. From this point forward, I would recommend exploring n8n to best fit your specific use case, but in this example, we’ll be using Google Sheets.

For each sold item returned, we want to create a row in our sheet to make it easy to filter and sort information such as the title, condition, and price.

First, we need to extract the organic results from the eBay API response using a Code node. I’ve used JavaScript here, but you can also use Python:

const results = $input.first().json.organic_results

return results

This will output each result individually so we can pass them to the Google Sheets node.

Next, configure the Google Sheets node by selecting the “Append row in sheet” operation. Choose the document and sheet you want to write to, and make sure you’ve already added columns for each data point you plan to include. In this example, I’ve added Title, Link, Condition, and Price.

Finally, map the appropriate attributes from n8n to the corresponding columns in your Google Sheet.

If all went well, all of our results should have been written to a Google Sheet:

Wrap-Up

In Part 2, we completed our journey from an image to actionable data. We used an AI agent to refine the search results and pinpoint the exact product name, then tapped into eBay’s data to find its market price. At this point, our “Visual Search to Price Tracking” workflow is fully functional: you can POST an image and get back the likely item name and its price on eBay within seconds.

Ideas for Scaling Further: Now that the basic workflow is done, you can enhance it even more:

Batch Processing: Have multiple images? You could modify the workflow to handle an array of images in one go, or simply trigger it multiple times. n8n could even watch an email or folder for new images and process them automatically.
Scheduled Price Checks: As mentioned, you could use a Scheduler (Cron) node to periodically run the eBay search for saved item names, building a daily price tracker. This might help you notice trends (e.g., prices going up or down over time).
Alerts for Price Drops or Increases: With a slight extension, you could compare the latest price to a previous one (store the last known price in a workflow variable or database). If the price drops by a certain percentage, have n8n send you an alert – useful for deciding when to sell or buy more.
Integration into Listing Workflow: If you are a seller, you could integrate this into your listing creation process. For example, when you want to list a new item, run it through this workflow to get the suggested title and price, then automatically draft an eBay listing or a listing in your inventory system with those details.

We hope this two-part tutorial showed the power of combining AI, web scraping and automation. By chaining these tools (image recognition, language understanding, and web data retrieval), you can automate typically monotonous manual tasks. Whether you’re a card collector pricing out your collection or an online seller researching products, this workflow can save you a ton of time.

Fetching AI Overviews with Node.js

Nathan Skiles — Fri, 11 Jul 2025 01:14:34 +0000

Fetching AI Overviews with Node.js

AI Overviews are quickly becoming a prominent feature in today’s SEO landscape, and with SerpApi, extracting this data is simple.

While Google hasn’t shared many details about how sources are selected for AI Overviews, early signs suggest that they rely on many of the same SEO principles already in use. This makes tracking AI Overviews a valuable addition to your SEO strategy, offering insight into how users discover and interact with your content.

By targeting the correct queries and optimizing your content accordingly, you may improve your chances of being cited in an AI Overview. And although SerpApi simplifies data collection, there are a few common pitfalls to be aware of. In this post, I’ll walk through what to expect and how to avoid issues during implementation.

Getting Started

For this guide, I'll be using SerpApi's Node.js package (documentation) to retrieve AI Overviews. That said, the same principles apply no matter which programming language you prefer.

I also recommend reviewing the SerpApi Google Search API and AI Overview API documentation to get more familiar with the endpoints and data structure.

Requirements

SerpApi API Key - If you don’t have an account yet, you can sign up for a free plan with 100 searches per month. If you already have an account, your API key is available in your dashboard.
Node.js 7.10.1 or newer
SerpApi Node.js Package - You can install it using npm install serpapi

Search Parameters

AI Overviews are returned for some queries to SerpApi's Google Search API. To get started, let’s define the search parameters:

const searchParams = {
  engine: "google",
  q: query,
  api_key: API_KEY,
  no_cache: true,
};

Here’s what each parameter does:

engine : "google" - Tells SerpApi to use the standard Google Search engine.
q: query - The search query to submit.
api_key: API_KEY - Your SerpApi key for authentication.
no_cache: true - Forces SerpApi to fetch fresh data from Google instead of returning a cached result.

If no_cache is not used, you may receive a cached response that includes an expired page_token. Always use no_cache: true when you intend to follow up with a second request using the page_token.

Perform the Initial Search and Check for AI Overview

const searchResponse = await getJson(searchParams);
const overviewData = searchResponse.ai_overview;

This performs the initial search and attempts to extract the ai_overview object from the response. If the query triggers an AI Overview, the overviewDatathe variable will contain its content; otherwise, it will be undefined.Not all searches return the full AI Overview here. Instead, some may only include a page_token, which is a temporary reference used to fetch the actual AI Overview in a second request.

Check for page_token and Fetch the Full AI Overview

if (overviewData.page_token) {
  const overviewParams = {
    engine: "google_ai_overview",
    api_key: API_KEY,
  };

  overviewParams.page_token = overviewData.page_token;
  const ai_overview_response = await getJson(overviewParams);

  overviewResult = ai_overview_response.ai_overview;
} else {
  overviewResult = overviewData;
}

This checks whether the initial response includes a page_token. If it does:

A second request is made using the google_ai_overview engine and the page_token.
The full AI Overview is returned and stored in overviewResult.

If no token is found, it assumes the AI Overview was included in the original response.

page_token values are short-lived, expiring within ~4 minutes. You must make the follow-up request immediately.

Full Example

In the example below, we iterate over an array of queries, performing a Google Search for each one and checking if an AI Overview is returned.

If an AI Overview is present, we then check whether it includes a page_token. If it does, we make a follow-up request to the google_ai_overview engine using that token to retrieve the full AI Overview.

import { getJson } from "serpapi";
import dotenv from "dotenv";
dotenv.config();

const API_KEY = process.env.SERPAPI_KEY;

const queries = [
  "what makes a good web hosting service",
  "best crm platform",
  "top programming languages to learn in 2025",
];

const fetchAIOverview = async (query) => {
  const searchParams = {
    engine: "google",
    q: query,
    api_key: API_KEY,
    no_cache: true,
  };

  try {
    const searchResponse = await getJson(searchParams);
    const overviewData = searchResponse.ai_overview;
    let overviewResult;

    if (!overviewData) {
      console.log("No AI overview available.");
      return;
    }

    if (overviewData.page_token) {
      const overviewParams = {
        engine: "google_ai_overview",
        api_key: API_KEY,
      };
      console.log("Page token found. Fetching AI overview page...");
      overviewParams.page_token = overviewData.page_token;
      const ai_overview_response = await getJson(overviewParams);
      console.log("AI overview retrieved.");
      overviewResult = ai_overview_response.ai_overview;
    } else {
      console.log("AI overview retrieved.");
      overviewResult = overviewData;
    }

    console.log("AI Overview:\n", overviewResult);
  } catch (error) {
    console.error("Error fetching data from SerpAPI:", error);
  }
};

queries.forEach((query) => {
  console.log(`Fetching AI overview for query: "${query}"`);
  fetchAIOverview(query);
});

Best Practices Summary

Use no_cache: true to avoid stale results and expired tokens.
Make the follow-up request to google_ai_overview immediately if a page_token is present.
Process each query sequentially, not in parallel, to avoid expired page_tokens.

Resources

For a high-level introduction to the Google AI Overviews API, I recommend checking out the following blog post:

How to Scrape Google AI Overviews (AIO)

If you have questions or run into any issues with AI Overviews, don’t hesitate to reach out to SerpApi’s support team at contact@serpapi.com

Unlocking SEO Insights: Leveraging 'People Also Ask' for Smarter Content Strategies

Nathan Skiles — Thu, 27 Feb 2025 22:09:27 +0000

Creating content that resonates with your target audience requires more than just keyword research and competitor analysis. Gaining insight into your audience’s questions, concerns, and information-seeking behaviors can provide a deeper understanding of their needs. This knowledge helps in crafting content that directly addresses their interests and challenges.

The "People Also Ask" (PAA) boxes have become a prominent feature in Google's search engine results pages (SERPs), which are the pages displayed by search engines in response to a user's query. These dynamic elements offer valuable insights into user intent and reveal emerging content opportunities that can enhance any content strategy.

Understanding People Also Ask

The People Also Ask feature represents Google's effort to provide users with additional, relevant questions related to their search queries. These expandable boxes appear within search results and contain a series of questions that users frequently ask about related topics. When a user clicks on a question, the box expands to reveal a brief answer, typically extracted from a high-ranking webpage.

Expandable "People also ask" box

What makes PAA boxes particularly valuable is their dynamic nature. For example, if a user searches for "content strategy" and clicks on a PAA question like "What are the steps to creating a content strategy?", new related questions will appear, such as "How do you measure the success of a content strategy?" or "What tools help with content strategy development?" This continuous expansion of queries provides deeper insights into user search intent. This behavior reflects real user interests and search patterns, providing authentic insights into an audience's thought processes and information needs.

While PAA data is often associated with straightforward Q&A content, it also uncovers:

The various angles and perspectives from which users research a topic.
Common misconceptions or concerns that need addressing.
Related topics that users frequently explore together.
The level of expertise in their target audience's questions.

Analyzing PAA data enables content strategists to craft detailed plans that target key search terms, cover topics comprehensively, and guide user journeys. This approach aligns with Google's quality guidelines, which prioritize comprehensive topic coverage and demonstrable subject expertise. In doing so, it paves the way for improved search visibility and engagement.

Transforming PAA Insights into Content

Converting PAA data into actionable content strategies requires a systematic approach. When analyzing PAA questions, start by identifying patterns in user inquiries. For instance, if you're creating content about digital marketing, you'll often notice a progression—from basic definitional queries to more complex implementation questions. This progression can help structure your content, guiding readers from foundational concepts to advanced applications.

Consider a real-world example: a software company launching a new project management tool. By analyzing PAA data related to "project management software," they might discover that users frequently ask questions about integration capabilities, pricing models, and team collaboration features. This intelligence helps prioritize which aspects of the product to highlight in their content marketing efforts. By leveraging PAA data, content teams can take several concrete steps:

Build detailed FAQ sections that proactively address common audience queries.
Structure blog posts and articles to naturally answer related questions.
Create content that covers topics exhaustively.
Spot content gaps to uncover overlooked topics and opportunities for expansion.

These strategies help ensure your content directly addresses user needs and fills critical gaps in your existing offerings.

The Role of PAA in SEO Strategy

Incorporating PAA insights into an SEO strategy can improve search visibility and audience engagement. By targeting PAA questions, content creators can:

Increase their chances of ranking for featured snippets.
Improve content relevance and user engagement.
Enhance topic authority and site credibility.
Drive organic traffic by addressing real search queries.

Optimizing content for PAA involves structuring answers concisely, using schema markup, and ensuring that content provides direct, valuable responses to user queries. Additionally, updating content based on newly emerging PAA questions can help maintain relevance and rankings over time.

Implementing PAA Insights with SerpApi

SerpApi is a powerful tool designed to scrape and extract structured data from live search engine results pages, including the "People Also Ask" feature. Using SerpApi, businesses and SEO professionals can automate the collection of PAA data, ensuring continuous insights into evolving user queries.

With real-time search results and an easy-to-use API, SerpApi eliminates the need for manual research or time-intensive in-house solutions. This allows content teams to focus on strategy and execution rather than data collection.

Google Search API

SerpApi’s Google Search API enables users to extract various search result elements, including organic listings, ads, featured snippets, and the People Also Ask section. By integrating this API, businesses can track PAA questions dynamically, monitor changes, and identify emerging trends in user queries. This allows content creators to refine their SEO and content strategies with up-to-date information.

Google Search API Documentation

The remainder of this post will focus specifically on the Google Related Questions API, I highly recommend checking out the following blog post to get similar with SerpApi and the Google Search API:

How to Scrape Google Search Results (SERPs) - 2025 Guide

Additionally, it's worth performing test queries in our playground environment to get a better idea of the results returned as well as the structure of data returned from both the Google Search and Related Questions APIs:

SerpApi Playground - SerpApi

The Google Related Questions API specifically focuses on retrieving PAA data. This API provides structured data on related questions that users frequently ask, along with their respective answers. By leveraging this API, content strategists can:

Identify trending PAA questions for their niche.
Monitor how PAA questions evolve over time.
Optimize content to align with high-impact queries.

Using SerpApi’s capabilities, businesses can efficiently gather, analyze, and implement PAA insights into their SEO and content marketing efforts.

Google Related Questions API - SerpApi

Google Related Questions API Documentation

Fetching PAA Questions Using SerpApi

In this section, we will walk through using Node.js to fetch PAA questions using SerpApi's Google Search API and Google Related Questions API. I will use JavaScript, but the examples can be easily translated into other programming languages. You can find a complete list of SerpApi integrations here:

SerpApi: Integrations

Step 1: Install SerpApi’s JavaScript Package

While you can use any of your favorite packages to make API calls, we will use the SerpApi Node.js package (documentation) to keep things clean and consistent.

npm install serpapi

Step 2: Fetch Data Using Google Search API

PAA questions appear organically in Google Search results, so to start with, we can query the Google Search API to return a typical SERP.

Below, we perform a search using the google engine, and our query is "content strategy." We then store the related questions returned in an array that we can add to later.

import { getJson } from "serpapi";

const response = await getJson({
  engine: "google",
  api_key: "YOUR_API_KEY",
  q: "content strategy"
});

// Store PAA questions
const relatedQuestionsArray = response["related_questions"]);

While related questions are returned for most queries, I recommend adding some more robust error handling for edge cases where no related questions are returned.

Example Response:

{
  ...
  "related_questions": [
    {
      "question": "What are the 7 steps in creating a content strategy?",
      "snippet": ""
      "title": "7-step content development process: step-by-step guide",
      "date": "Jun 17, 2024",
      "link": "https://contentsnare.com/content-development-process/",
      "list": [
        "Step 1: Do your research. ... ",
        "Step 2: Analyze the information. ... ",
        "Step 3: Plan your strategy. ... ",
        "Step 4: Write. ... ",
        "Step 5: Editing, SEO and publishing. ... ",
        "Step 6: Take to social media. ... ",
        "Step 7: Analyze and begin again."
      ],
      "displayed_link": "https://contentsnare.com › content-development-process",
      "source_logo": "https://serpapi.com/searches/67bf85dac71a675e242edf6b/images/b63f19fbb9fff39987251b02abdc8c2d341978702805ce28c8d754c35cf0eb70.png",
      "next_page_token": "eyJvbnMiOiIxMDA0MSIsImZjIjoiRXFFQkNtSkJRUzFMVkdoa1...",
      "serpapi_link": "https://serpapi.com/search.json?device=desktop&engine=google_related_questions&google_domain=google.com&next_page_token=eyJvbnMiOiIxMDA0MSIsImZ..."
    }
    ...
  ]
  ...
}

Typically, 3-4 PAA questions are returned on any given Google SERP. However, as mentioned previously, expanding these questions returns additional related questions.

Each question returned in Step 2 will include a next_page_token, which we can use to retrieve additional questions related to the question using the Google Related Questions API.

const relatedResponse = await getJson({
  engine: "google_related_questions",
  api_key: "YOUR_API_KEY",
  next_page_token: "NEXT_PAGE_TOKEN"
});

// Save additional question data
const additionalQuestions = relatedResponse["related_questions"]);

The response to this query will be similar to the one seen in Step 2, returning next_page_tokens for each additional question.

By automating the extraction of PAA insights, businesses can refine their SEO approach, ensuring content aligns with what users are actively searching for.

Example Node.js CLI App

To get you started, I've created a sample CLI app that allows you to enter a query and specify the depth at which you would like to retrieve PAA questions.

The tool takes your search query, sends it to the Google Search API, and extracts the "People Also Ask" questions from the search results. For each question, it recursively fetches additional related questions up to the specified depth.

The output displays the relationship between questions in a tree structure:

1 → First level question
1.2 → Second question that stems from the first question
2.1.3 → Third question that stems from the first child of the second question

GitHub - NateSkiles/paa-example

Measuring Success

After implementing PAA insights into your content and SEO strategy, measuring success is crucial. Key performance indicators (KPIs) to track include:

Organic traffic growth: Increased visits from search engines.
SERP rankings: Improved positions for targeted queries.
Click-through rates (CTR): Higher engagement on PAA-optimized content.
User engagement metrics: Time on page, bounce rate, and session duration.
Conversions: Lead generation and customer inquiries driven by content.

Regularly updating content based on fresh PAA insights ensures that your strategy remains effective and aligned with user search behavior.

Conclusion

The "People Also Ask" feature is a powerful tool for understanding user intent and enhancing content strategies. By leveraging SerpApi’s capabilities, marketers and SEO professionals can efficiently extract and analyze PAA data to create highly relevant, user-focused content.

By systematically integrating PAA insights, businesses can improve their search visibility, attract more organic traffic, and establish authority in their niche. Leveraging these insights allows content creators to address user queries effectively, optimize for featured snippets, and provide comprehensive information that enhances engagement. Regularly refining content based on PAA data ensures continued relevance and strengthens a brand’s presence in search results. Whether you're developing blog posts, FAQs, or in-depth guides, PAA-driven content strategies can set your brand apart in an increasingly competitive digital landscape.

No-code Solutions for Turning Search Results Into Markdown for LLMs

Nathan Skiles — Thu, 27 Feb 2025 21:56:35 +0000

Intro
In my last post, I walked through a coding solution in which we used Node.js to scrape webpages returned in Google Search results and parse them to Markdown for use in LLMs or other use cases. Today, we will do something similar, but we will utilize no-code solutions to return results from SerpApi's Google Search API and parse to markdown using the Reader API by Jina.ai.

Turning Search Results Into Markdown for LLMs
Intro This article will walk through converting search results into Markdown format, suitable for use in large language models (LLMs) and other applications. Markdown is a lightweight markup language that provides a simple, readable way to format text with plain-text syntax. Checkout Markdown Guide for more information: Markdown GuideA free

SerpApi
Nathan Skiles

Reader by Jina
Jina AI’s Reader API (https://jina.ai/reader/) offers an alternative for extracting structured content from web pages, maintaining document structure, and handling diverse content types.

💡
Jina AI provides 1 million test credits to get started, but keep in mind that tokens are used based on the amount of content parsed, not the number of requests made.
Scraping Search Results
Several options integrate well with SerpApi for those who prefer no-code solutions. This section will walk through configuring Make and Zapier to query search results through SerpApi.

Make
Starting with Make, we will create a Google Doc from the data returned by SerpApi and the Reader API. This is a relatively simple example, but Make provides many integrations with AI providers such as OpenAI, Anthropic, and more. Instead of sending the data to a Google Doc, you can replace the last module and send it wherever you need it!

SerpApi has an official Make app, making it much more straightforward than Zapier to configure. While I will only be walking through how to configure the SerpApi Google Search module, for more information on our Make app, check out this blog post:

Announcing SerpApi’s Make App
We are pleased to announce the release of our Custom App on Make.com. This makes it easier than ever to integrate SerpApi with other systems without having to write any code.

SerpApi
Alex Barron

Create a new scenario and search for the SerpApi app to get started. Select your preferred search engine module; we will use Google for this example.

After selecting your search engine, we can start configuring our call to SerpApi. You'll add your SerpApi API key by creating a connection. Click "Create a connection."

Enter your API key and name your connection:

From here, we can configure our query:

You can fetch additional pages by increasing the “Pagination Limit” setting but note that each page will cost 1 SerpApi search credit.

Click “Run once” and verify the data coming back from SerpApi:

Now that we have results we can work with, let’s add an “Iterator” module to allow us to use the data returned in the organic_results array. The Iterator module (documentation) converts arrays into a series of bundles. Each element in the array will output as a separate bundle, allowing us to query the Reader API for each link returned in our Google results.

To query the Reader API for each link returned, let’s add an HTTP “Make a request” module that allows us to send HTTP requests.

Set the URL parameter to https://r.jina.ai/, then click the box again to pull up a list of available data we can pass to this parameter. From our Iterator module, select the “link” attribute. This passes the link from our organic Google search results to the Reader URL. Here’s what that should look like:

Now, we need to set our header for authorization by passing our Bearer token provided by Jina AI:

Before running again, let’s add our final step to create a Google Doc to store the results. Again, this final step can be changed depending on your use case and where you wish to send the data.

Add the Google Docs “Create a Document” module and add a connection to your Google account. Now you can set a name for the document as well as pass data from the Reader API as the content of the doc:

Now lets run our scenario once more to ensure that everything is working correctly.

💡
While testing, we can set a limiter filter after the Iterator module to prevent the flow from continuing more than once.

Click the wrench icon under the link between the Iterator and HTTP modules. Create a condition only to continue if the Iterator’s “position” attribute is numerically equal to “1”:

Make sure to remove the limiter before publishing if you wish to query for all the results. Alternatively, you set the num parameter in the SerpApi Search Google module to limit results returned by SerpApi.

Zapier
While this article is focused on no-code solutions, SerpApi currently does not have a Zapier integration. Meaning we must use a little code to make our API requests to SerpApi. I have provided a code below that you can drop into Zapier Code steps without issue, though some adjustments may be needed to fit your use case better. I recommend starting with the code I've provided and then adjusting once you get results from SerpApi and the Reader API.

In this section, we will walk through:

Use the Schedule Trigger to initiate Zap
Retrieve search results URLs from SerpApi
Process webpages from URLs through Reader API to Markdown
Send formatted content to your desired apps
As this Zap requires API calls to both SerpApi and the Reader API, you will likely need a Trial or Professional/Team plan. Zapier limits code runtime to 1 second on the free plan, and both API requests will likely take longer (documentation).

To get started, add a trigger to your Zap. For this post, we will use the Schedule by Zapier trigger to our Zap every day at 12:00 a.m. You can select a trigger that best fits your needs.

💡
Make sure to test each step before continuing to ensure you've created test data for future steps. Zapier will typically force you to test a step after changes before publishing.

Next, we add a Code by Zapier action to make our request to SerpApi. This example uses JavaScript, but you can choose Python if you prefer to use the code below as a guideline.

I've included the JavaScript code for this action below:

const { SERPAPI_KEY, QUERY, LOCATION } = inputData;

async function fetchGoogleSearchResults(query, location) {
try {
const response = await fetch(
https://serpapi.com/search.json?engine=google&q=${encodeURIComponent( query )}&location=${encodeURIComponent( location )}&gl=us&hl=en&api_key=${SERPAPI_KEY}
);

if (!response.ok) {
  throw new Error(`HTTP error! status: ${response.status}`);
}

const data = await response.json();

if (!data) {
  throw new Error("No data returned from SerpAPI");
}

return data;

} catch (error) {
// Instead of just logging, throw the error
throw new Error(SerpAPI request failed: ${error.message});
}
}

try {
const searchResults = await fetchGoogleSearchResults(QUERY, LOCATION);

output = {
organicResults: searchResults?.organic_results || "No Organic Result Found"
};

return output;
} catch (error) {
console.error("Error:", error.message);
// Return a default response or throw error based on your needs
return { organic_results: [] };
}
After adding the above code to the action, add your input data:

SERPAPI_KEY - your SerpApi key goes here. You can find your API key in your SerpApi dashboard.
QUERY - query parameter for the search you wish to perform.
LOCATION - location parameter for the search you wish to perform.
If you wish to add additional search parameters, you can set them as Input Data or update the URL in the code directly. Setting them as Input Data allows you to update them more easily in the future.

Your first code step should now look something like this:

Our request from SerpApi will return an array of links for our organic results. To send each link to the Reader API, we must iterate through the links using the Looping by Zapier action.

This action takes an array of values and runs the actions within the loop for each value. For our "Values to Loop," we can pass the "Organic Results Link" values from our second step to a key called "links".

Let's also set the maximum number of loop iterations to 1 for testing to ensure we aren't spending Reader API credits needlessly.

Inside the Looping by Zapier action, we can add another Code by Zapier action to make our request to the Reader API. Paste the JavaScript below in the code section of this action:

const { link, READERAPI_KEY } = inputData;

async function fetchReaderData(link, apiKey) {
try {
// Encode URL for path parameter
const encodedUrl = encodeURIComponent(link);

const response = await fetch(`https://r.jina.ai/${encodedUrl}`, {
  method: 'GET',
  headers: {
    'Authorization': `Bearer ${apiKey}`,
    'Accept': 'application/json'
  }
});

if (!response.ok) {
  throw new Error(`HTTP error! status: ${response.status}`);
}

const data = await response.json();

if (!data) {
  throw new Error('No data returned from Reader API');
}

return data;

} catch (error) {
console.error('Reader API request failed:', error.message);
throw error;
}
}

try {
const readerResults = await fetchReaderData(link, READERAPI_KEY);

output = {
readerResults
};

return output;
} catch (error) {
return {
error: error.message,
content: null,
title: null,
metadata: null
};
}
Again, set your Input Data for this action:

link - the value of the link from our Looping action.
READERAPI_KEY - your Jina AI Reader API key, found on the Reader API website.
Once complete, your second Code by Zapier action should look like this:

Finally, we can send our data from the Reader API to an app. For this example, we will use Google Docs, but feel free to explore Zapier's integrations and find the action that best fits your use case. I recommend trying with Google Docs first to better understand the output data.

Add the "Create Document From Text in Google Docs" action to your Loop. This will create a new Google Doc for each URL the Loop processes. Feel free to adjust as needed.

If you have not done so previously, you may need to authorize Zapier to access your Google account. Once complete, your Setup tab should look something like this:

Click "Continue" or navigate to the "Configure" tab. Here, we can set a name for the document and the Google Drive folder in which you would like to save the document and pass the content returned from the Reader API to the document's content.

And you're all set! Make sure to test the Zap thoroughly to ensure there are no issues, and check your Google Drive to ensure the document was created.

Conclusion
No-code tools such as Make and Zapier provide powerful tools for users who might not be comfortable writing or maintaining scripts necessary to scrape valuable data found in search results. While SerpApi isn't necessarily a no-code solution, it abstracts away many complex concepts required to scrape search engines. When paired with no-code tools, even non-technical users can leverage this data.

No-code Solutions for Turning Search Results Into Markdown for LLMs

Nathan Skiles — Wed, 08 Jan 2025 22:59:36 +0000

Intro

In my last post, I walked through a coding solution in which we used Node.js to scrape webpages returned in Google Search results and parse them to Markdown for use in LLMs or other use cases. Today, we will do something similar, but we will utilize no-code solutions to return results from SerpApi’s Google Search API and parse to markdown using the Reader API by Jina.ai.

Turning Search Results Into Markdown for LLMs ↗

Reader by Jina

Jina AI’s Reader API (https://jina.ai/reader/) offers an alternative for extracting structured content from web pages, maintaining document structure, and handling diverse content types.

💡 Jina AI provides 1 million test credits to get started, but keep in mind that tokens are used based on the amount of content parsed, not the number of requests made.

Scraping Search Results

Several options integrate well with SerpApi for those who prefer no-code solutions. This section will walk through configuring Make and Zapier to query search results through SerpApi.

Make

Starting with Make, we will create a Google Doc from the data returned by SerpApi and the Reader API. This is a relatively simple example, but Make provides many integrations with AI providers such as OpenAI, Anthropic, and more. Instead of sending the data to a Google Doc, you can replace the last module and send it wherever you need it!

Announcing SerpApi’s Make App ↗

Create a new scenario and search for the SerpApi app to get started. Select your preferred search engine module; we will use Google for this example.

After selecting your search engine, we can start configuring our call to SerpApi. You’ll add your SerpApi API key by creating a connection. Click “Create a connection.”

Enter your API key and name your connection:

From here, we can configure our query:

You can fetch additional pages by increasing the “Pagination Limit” setting but note that each page will cost 1 SerpApi search credit.

Click “Run once” and verify the data coming back from SerpApi:

Now that we have results we can work with, let’s add an “Iterator” module to allow us to use the data returned in the organic_results array. The Iterator module (documentation) converts arrays into a series of bundles. Each element in the array will output as a separate bundle, allowing us to query the Reader API for each link returned in our Google results.

To query the Reader API for each link returned, let’s add an HTTP “Make a request” module that allows us to send HTTP requests.

Set the URL parameter to https://r.jina.ai/, then click the box again to pull up a list of available data we can pass to this parameter. From our Iterator module, select the “link” attribute. This passes the link from our organic Google search results to the Reader URL. Here’s what that should look like:

Now, we need to set our header for authorization by passing our Bearer token provided by Jina AI:

Before running again, let’s add our final step to create a Google Doc to store the results. Again, this final step can be changed depending on your use case and where you wish to send the data.

Now lets run our scenario once more to ensure that everything is working correctly.

While testing, we can set a limiter filter after the Iterator module to prevent the flow from continuing more than once.

Click the wrench icon under the link between the Iterator and HTTP modules. Create a condition only to continue if the Iterator’s “position” attribute is numerically equal to “1”:

Make sure to remove the limiter before publishing if you wish to query for all the results. Alternatively, you set the num parameter in the SerpApi Search Google module to limit results returned by SerpApi.

Zapier

While this article is focused on no-code solutions, SerpApi currently does not have a Zapier integration. Meaning we must use a little code to make our API requests to SerpApi. I have provided a code below that you can drop into Zapier Code steps without issue, though some adjustments may be needed to fit your use case better. I recommend starting with the code I’ve provided and then adjusting once you get results from SerpApi and the Reader API.

In this section, we will walk through:

Use the Schedule Trigger to initiate Zap
Retrieve search results URLs from SerpApi
Process webpages from URLs through Reader API to Markdown
Send formatted content to your desired apps

As this Zap requires API calls to both SerpApi and the Reader API, you will likely need a Trial or Professional/Team plan. Zapier limits code runtime to 1 second on the free plan, and both API requests will likely take longer (documentation).

To get started, add a trigger to your Zap. For this post, we will use the Schedule by Zapier trigger to our Zap every day at 12:00 a.m. You can select a trigger that best fits your needs.

💡 Make sure to test each step before continuing to ensure you’ve created test data for future steps. Zapier will typically force you to test a step after changes before publishing.

Next, we add a Code by Zapier action to make our request to SerpApi. This example uses JavaScript, but you can choose Python if you prefer to use the code below as a guideline.

I’ve included the JavaScript code for this action below:

const { SERPAPI_KEY, QUERY, LOCATION } = inputData;

async function fetchGoogleSearchResults(query, location) {
  try {
    const response = await fetch(
      `https://serpapi.com/search.json?engine=google&q=${encodeURIComponent(
        query
      )}&location=${encodeURIComponent(
        location
      )}&gl=us&hl=en&api_key=${SERPAPI_KEY}`
    );
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    const data = await response.json();
    if (!data) {
      throw new Error("No data returned from SerpAPI");
    }
    return data;
  } catch (error) {
    // Instead of just logging, throw the error
    throw new Error(`SerpAPI request failed: ${error.message}`);
  }
}

try {
  const searchResults = await fetchGoogleSearchResults(QUERY, LOCATION);
  output = {
    organicResults: searchResults?.organic_results || "No Organic Result Found"
  };
  return output;
} catch (error) {
  console.error("Error:", error.message);
  // Return a default response or throw error based on your needs
  return { organic_results: [] };
}

After adding the above code to the action, add your input data:

SERPAPI_KEY - your SerpApi key goes here. You can find your API key in your SerpApi dashboard.
QUERY - query parameter for the search you wish to perform.
LOCATION - location parameter for the search you wish to perform.

If you wish to add additional search parameters, you can set them as Input Data or update the URL in the code directly. Setting them as Input Data allows you to update them more easily in the future.

Your first code step should now look something like this:

Our request from SerpApi will return an array of links for our organic results. To send each link to the Reader API, we must iterate through the links using the Looping by Zapier action.

This action takes an array of values and runs the actions within the loop for each value. For our “Values to Loop,” we can pass the “Organic Results Link” values from our second step to a key called “links”.

Let’s also set the maximum number of loop iterations to 1 for testing to ensure we aren't spending Reader API credits needlessly.

Inside the Looping by Zapier action, we can add another Code by Zapier action to make our request to the Reader API. Paste the JavaScript below in the code section of this action:

const { link, READERAPI_KEY } = inputData;

async function fetchReaderData(link, apiKey) {
  try {
    // Encode URL for path parameter
    const encodedUrl = encodeURIComponent(link);
    const response = await fetch(`https://r.jina.ai/${encodedUrl}`, {
      method: 'GET',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Accept': 'application/json'
      }
    });
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    const data = await response.json();
    if (!data) {
      throw new Error('No data returned from Reader API');
    }
    return data;
  } catch (error) {
    console.error('Reader API request failed:', error.message);
    throw error;
  }
}

try {
  const readerResults = await fetchReaderData(link, READERAPI_KEY);
  output = {
    readerResults
  };
  return output;
} catch (error) {
  return {
    error: error.message,
    content: null,
    title: null,
    metadata: null
  };
}

Again, set your Input Data for this action:

link - the value of the link from our Looping action.
READERAPI_KEY - your Jina AI Reader API key, found on the Reader API website.

Once complete, your second Code by Zapier action should look like this:

Finally, we can send our data from the Reader API to an app. For this example, we will use Google Docs, but feel free to explore Zapier’s integrations and find the action that best fits your use case. I recommend trying with Google Docs first to better understand the output data.

Add the “Create Document From Text in Google Docs” action to your Loop. This will create a new Google Doc for each URL the Loop processes. Feel free to adjust as needed.

If you have not done so previously, you may need to authorize Zapier to access your Google account. Once complete, your Setup tab should look something like this:

Click “Continue” or navigate to the “Configure” tab. Here, we can set a name for the document and the Google Drive folder in which you would like to save the document and pass the content returned from the Reader API to the document’s content.

And you’re all set! Make sure to test the Zap thoroughly to ensure there are no issues, and check your Google Drive to ensure the document was created.

Conclusion

No-code tools such as Make and Zapier provide powerful tools for users who might not be comfortable writing or maintaining scripts necessary to scrape valuable data found in search results. While SerpApi isn’t necessarily a no-code solution, it abstracts away many complex concepts required to scrape search engines. When paired with no-code tools, even non-technical users can leverage this data.