DEV Community: SerpApi

Self-host SerpBear with Coolify

Josef Strzibny — Thu, 02 Apr 2026 08:47:23 +0000

Tracking organic positions on search engines is one of the main concerns of SEO. SerpBear is a SERP position tracking tool that will help you track how well you are doing in Google search. This post will take you through all the steps to start tracking your keywords with your own SerpBear instance on Coolify.

What is SerpBear?

SerpBear is an Open Source search engine position tracker. It allows you to track your or your competitors' keyword positions in Google and to also get notified of these changes. Some of the features include:

Unlimited keywords: You can add as many domains and keywords to track as you want. SerpBear will use a 3rd-party API such as SerpApi for actual tracking behind the scenes.
Flexible scraping strategy: Choose your scraping strategy to control how many Google pages are checked for each domain. It's built to handle the Google's 2025 removal of the num=100parameter.
Email notification: Get notified of all your keyword position changes via email with at the frequency of your choosing.
Keyword research: You have the option to do keyword research with the integration of Google Ads.
Google Search Console: You can see the actual visit count and impressions for each keyword. Discover new keywords thanks to direct integration with Google Search Console.
Exports: Export your domain keyword data in CSV files.

If you want to track your keywords, but don't want to use SerpBear, have a look how to do your custom SERP tracking in JavaScript.

What is Coolify?

Coolify is an Open Source Platform as a Service (PaaS) that will help you self-host SerpBear or any other self-hostable software out there. You can also deploy applications, databases, and one-click services.

I am assuming you already have Coolify installed on your server. If you don't, you can learn how to deploy Coolify on any VPS or take advantage of Coolify Cloud version where you don't have to host Coolify yourself.

💡 We are going to deploy SerpBear 3.0 on Coolify 4.0. Coolify 4.0 is still beta software for the time being.

Signing up for SerpApi

SerpApi is a service providing access to various SERP results with a simple API. SerpBear is free and open source software, but it uses SerpApi to do the heavy lifting of querying and parsing Google results. SerpBear cannot do this on its own due to the nature of managing proxies and solving CAPTCHAs, but you can still start with free searches to set everything up and see it working before committing to a paid plan.

So first of all, sign up for SerpApi and then note your private API key from serpapi.com/manage-api-key page:

We are going to use this API key from SerpBear, but you can also use it to integrating one the many SerpApi SDKs to build anything needing SERP API.

Configuring DNS

Head over to your domain registrar console and set the DNS A records for domains or subdomains you want to use for SerpBear:

coolify.example.com -> 178.104.25.112
serpbear.example.com -> 178.104.25.112

Choose "A" type records to directly connect a domain name or subdomain to the IP address (replace the values, the above is just an example). If you'll run SerpBear on the same host as Coolify, the IP address remains the same for both.

Adding SerpBear

If your Coolify instance is fully set up, it's time to add SerpBear. To run a new Coolify project head over to Projects (from the left menu) and click on + Add next to the Projects headline. Give it a name and description.

From Projects, click on + Add Resource next to your project name or select your project first and click on + New next to Resources. From here, we can select Docker Compose Empty under Docker Based on the right:

SerpBear provides Docker Compose configuration which Coolify supports. Since SerpBear uses SQLite to save SERP data, it's not necessary to spin up process-based databases and the Compose file is relatively simple:

services:
  app:
    image: towfiqi/serpbear:latest
    # Build from source instead of pulling the image:
    # build: .
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"
    environment:
      - USER_NAME=${USER:-admin}
      - PASSWORD=${PASSWORD}
      - SECRET=${SECRET}
      - APIKEY=${APIKEY}
      - SESSION_DURATION=${SESSION_DURATION:-24}
      - NEXT_PUBLIC_APP_URL=${NEXT_PUBLIC_APP_URL:-http://localhost:3000}
      # Optional: Google Search Console integration
      - SEARCH_CONSOLE_CLIENT_EMAIL=${SEARCH_CONSOLE_CLIENT_EMAIL:-}
      - SEARCH_CONSOLE_PRIVATE_KEY=${SEARCH_CONSOLE_PRIVATE_KEY:-}
    volumes:
      - serpbear_data:/app/data
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:3000 || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s

volumes:
  serpbear_data:

The SerpBear service definition includes the official Docker image, container restart policy, ports, environment variables, volumes, and health check. The official guide suggests using the latest image (towfiqi/serpbear:latest) which will let you update the application with Redeploy at any later point, but you can choose a stable release to lock-in a particular version.

The required environment variables reference variables like {USER:-admin} and {SECRET} which let us manage the environment directly from the Coolify admin interface. Notice that SerpBear also supports Google Search Console, so if you can obtain these credentials as well. Finally notice the serpbear_data volume which is the future location of your SerpBear database.

Paste the configuration above. Click Save.

Adding the new resource should take you to the resource configuration. Under General we can change the resource name, under Persistent Storages we can check our persistent volume, and finally under Environment Variables we can go fill up the variables from the Compose file:

Here's the list of environment variables we should fill in:

USER_NAME: The username you want to use to login to the app.
PASSWORD: The password you want to use to log in to the app.
SECRET: A secret key that will be used for encrypting 3rd party API keys & passwords.
APIKEY: API key that will be used to access the app's API. This is not SerpApi account key!
SESSION_DURATION: The duration(in hours) of the user's logged-in session.
NEXT_PUBLIC_APP_URL: The URL where your app is hosted and can be accessed like https://serpbear.example.com

Note that every environment entry has its own Update button and they won't be saved all at once. You can add the Google Search Console credentials if you have them, but they aren't necessary to start tracking your keywords with SerpApi.

Restart the service.

Tracking keywords with SerpBear

If you carefully followed all the steps, you should be now able to access the SerpBear instance on the (sub)domain of your choosing. Log in using the chosen credentials from the previous step. You should see a red prompt at the top to configure your Scrapper/Proxy. Click on it to open settings, choose SerpApi.com as the Scraping Method and insert your SerpApi API token under Scraper API Key Or Token:

Click Update Settings to close the right sidebar and you'll be ready to add your first keywords to track. Start by adding a domain name. After entering your domain name you should end up on the Tracking tab for the domain.

Click Add Keyword and start adding keywords and phrases you care about. They should appear in a nice table with their last position, best position, history graph, volume, and URL. Here's an example of tracking "serp api" for serpapi.com in the Czech Republic:

And that's it! You can now add as many domains and keywords as you want to. Remember that SEO today is not just about tracking organic results in Google, so have a look at AI overviews as well.

Next steps

As next steps you can consider adding your Google Search Console data to SerpBear, handle external backups in Coolify, or exploring more of SerpApi. The Free tier comes with 250 free searches per month.

How to start self-hosting with Coolify 4 on a VPS

Josef Strzibny — Wed, 01 Apr 2026 11:57:51 +0000

Coolify is an open source Platform as a Service (PaaS) that can help you self-host a lot of different software from content management tools like Ghost to SEO tracking software like SerpBear. The only thing you'll need is a virtual private server (VPS) from a hosting provider of your choice and a little bit of set up which is exactly what we'll go through in this post.

💡

Coolify v4 is still beta software as of the time of writing of this post. If you need a little bit more stability, wait for the final release.

What is Coolify?

Coolify is a self-hostable alternative to Heroku, Vercel, or Netlify. It's an open source PaaS that allows developers to deploy their applications as well as manage 3rd-party services and databases.

Some of the biggest advantages include:

Self-hosted control: You own your data on your own servers which saves money and avoids vendor lock-in.
Hosting versatility: Hosts web applications, static websites, databases (PostgreSQL, MySQL, MongoDB, etc.), and services
Automated DevOps: Automatically installs dependencies, sets up databases, and handles deployments from GitHub, GitLab, or Bitbucket.
Web console: Offers a web-based dashboard for managing multiple servers, viewing logs, and monitoring resource usage like CPU and RAM.
Built-in tools: Features automatic S3-compatible backups, auto provisioning of SSL certificates, webhooks, or preview deployments.

You can run Coolify in various ways. You can use a single server for everything you host or do a separate deployment for the services you'll run.

Applications, databases and services

Coolify supports deploying custom applications and 3rd-party services. It generally distinguish between 3 different types of resources:

Applications come with a git source or are built using Docker from Dockerfile. You can deploy anything from PHP to Java.
Databases are preconfigured Docker images for running databases, like MySQL or PostgreSQL.
Services are deployments based on Docker Compose files that are stored directly on the server. You deploy well known software like Ghost.

You can mix and match any of these on a single Coolify instance.

Setting up the virtual server

There are two variants to self-hosting. One with physical servers that are usually dedicated to you and one with virtual servers where you share the underlying physical servers with others.

We'll set up a simple virtual server on Hetzner which is a provider praised for its affordable costs. You are free to use any other provider you like or already have. Some other popular providers include Digital Ocean, Vultr, or OVHcloud.

💡

Coolify currently offers $20 credit for deploying to Hetzner by following the https://coolify.io/hetzner link.

To create a virtual private server (VPS), you'll need an SSH key pair which you'll authenticate with in the future. You can optionally set a password for the private key as well. To generate a new one, run ssh-keygen:

ssh-keygen

The command will interactively ask you about your key details. If you name it coolify you should end up with two files inside the ~/.ssh directory, one for private key and one for public key (coolify.pub).

💡

Protect your SSH private key! You always provide others with your public key, not the private key. The private key should stay with you on your computer.

Once you have your key pair, it's fairly straightforward to spin up a new server with the provider of your choice.

On Hetzner, create a new project and on the project overview click Add Server.

Select a location close to you, Ubuntu 24.04 or your favorite operating system (Debian, SUSE, and Fedora based systems are supported), the size you want, and optionally enable full virtual machine backups. Under SSH Keys upload your public SSH key which should disable password-based access on most providers and let you use your private key instead.

As for the size of the box, Coolify recommends at least 2 vCPUs, 2 GB of RAM, and 30 GB of storage space. If you are going to self-host projects on the same instance, you generally need to increase the box size accordingly. For only running one or two extra applications, consider increasing RAM.

This is all that's needed for things to work, but we can also provide a custom cloud-init script under Cloud config. For example, we can set up automatic system updates and install fail2ban for extra protection of our SSH port:

# Do a system update
apt update;
DEBIAN_FRONTEND=noninteractive apt upgrade -y

# Install essential packages
apt install -y curl unattended-upgrades fail2ban

# Set up unattented updates
echo -e "APT::Periodic::Update-Package-Lists \"1\";\nAPT::Periodic::Unattended-Upgrade \"1\";\n" > /etc/apt/apt.conf.d/20auto-upgrades
/etc/init.d/unattended-upgrades restart

# Install fail2ban
systemctl start fail2ban.service
systemctl enable fail2ban.service

You can optionally install Docker at this step from the system package manager but do not install Docker from Snap as that's not supported by Coolify.

Finally, give the server a name (you can call it coolify, but do not use a domain name) and click Create & Buy now. Provisioning the server will take some minutes and once done you should be able to find the public server IPv4 address next to its name on the server page:

You should now be able to log in from your terminal as:

ssh-add ~/.ssh/path-to-your-key
ssh root@[IP_ADDRESS]

There is quite a bit more to running and securing servers. You should ideally choose a different user than root and/or disable public SSH access altogether in the cloud's firewall. If you want to do that, have a look at WireGuard or Tailscale.

Configuring DNS

Now that you have a public IP address, you can create DNS records to get nice URLs for accessing Coolify, SerpBear, and whatever else you decide to self host.

Head over to your domain registrar console and set the DNS A records for domains or subdomains you want to use for Coolify and the applications you want to host:

# For accesing Coolify web console (admin area)
coolify.example.com -> 178.104.25.112

# Example application
serpbear.example.com -> 178.104.25.112

Choose "A" type records to directly connect a domain name or subdomain to the IP address (replace the values, the above is just an example). You can use Porkbun or GoDaddy to register your first domain name if you don't have one yet.

Since we'll run everything on a single host, the IP address remains the same for all instances.

Installing Coolify

Now that we have our Linux box up and running, we can SSH into it and run the official installer script:

curl -fsSL https://cdn.coollabs.io/coolify/install.sh | sudo bash

You can change some installation details with environment variables, so go and review this list before running the script.

For example, you might want to choose the admin username, email, and password:

env ROOT_USERNAME=admin \
ROOT_USER_EMAIL=admin@example.com \
ROOT_USER_PASSWORD=SecurePassword123 \
bash -c 'curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash'

Once run, you'll see a Congratulations! screen with the list of the IP addresses, version as well as a log file:

____ _ _ _ _ _
  / ___|___ _ ____ _ _ ____ _| |_ _ _| | ___| |_(_)___ _ _____ | |
 | | / _ \| '_ \ / _` | ' __/ _` |__ | | | | |/ _` | __| |/ _ \| '_ \/__ | |
 | |__| (_) | | | | (_| | | | (_| | |_| |_| | | (_| | |_| | (_) | | | \__ \_|
  \ ____\___ /|_| |_|\ __, |_| \__ ,_|\ __|\__ ,_|_|\ __,_|\__ |_|\ ___/|_| |_|___ (_)
                   |___/


Your instance is ready to use!

You can access Coolify through your Public IPV4: http://178.104.25.112:8000
You can access Coolify through your Public IPv6: http://[2a01:4f8:1c19:f81a::1]:8000

If your Public IP is not accessible, you can use the following Private IPs:

http://10.0.0.1:8000
http://10.0.1.1:8000
http://2a01:4f8:1c19:f81a::1:8000
http://fdb0:c330:3639::1:8000

WARNING: It is highly recommended to backup your Environment variables file (/data/coolify/source/.env) to a safe location, outside of this server (e.g. into a Password Manager).


============================================================

[2026-03-19 11:02:44] Installation Complete
============================================================
[2026-03-19 11:02:44] Coolify installation completed successfully
[2026-03-19 11:02:44] Version: 4.0.0-beta.468
[2026-03-19 11:02:44] Log file: /data/coolify/source/installation-20260319-110120.log

After installation, you should find your self-hosted Coolify instance at http://[IP_ADDRESS]:8000:

You should see a Welcome to Coolify screen. Click Let's go!

Create your root account and on the next screen choose This Machine as your server type (we'll run everything on a single instance):

After the initial wizard, let's go to Settings (from the left menu) and input the instance URL with our (sub)domain we prepared for Coolify under General (make sure to include full URL including the leading https://).

Optionally you can also explore the Backup and Transactional Email tabs for setting up Coolify backups and transactional emails (for things like forgotten password).

Then head over to Servers (from the left menu) where you should see our localhost instance (running Coolify). Here you'll find settings for the Coolify components. Open it and go to the Proxy tab:

Checking that the proxy is running is important as it will route the traffic to the applications you'll self host. Even if Coolify is running and you are logged in inside the console, the coolify-proxy might not be.

In case the proxy is not running, open the logs and see what's wrong. In my case, the proxy wasn't running and I found the following inside the logs:

Creating required Docker Compose file.
Pulling docker image.
 Image traefik:v3.6 Pulling 
 Image traefik:v3.6 Pulled 
Ensuring network coolify exists...
Ensuring network havzqyfu212ybs9n5lg48gzy exists...
Starting coolify-proxy.
ParseAddr("fdb0:c330:3639::1/64"): unexpected character, want colon (at "/64")

There seemed to be an issue with parsing an IPv6 address. To resolve it I decided to turn off IPv6 support on the system. I opened an SSH connection to the server again to edit Docker's daemon.json file:

vi /etc/docker/daemon.json

I added an entry with "ipv6": false:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    {"base":"10.0.0.0/8","size":24}
  ],
  "ipv6": false
}

Then I restarted Docker:

systemctl restart docker

After that I could restart the proxy and see coolify-proxy container running on the system.

And that was it! If you can log in and see that the proxy is running, you are ready to start adding individual applications and services. You can still also use a hosted version of Coolify if you never set up a server before and all of this looks daunting.

Deploying resources

Resources in Coolify needs a project, so open Projects from the left side menu, and click + Add next to Projects. Then click + New next to the Resources headline. You should arrive on a New Resource page which lets you choose what you want to run. You can deploy directly from a git repository, Dockerfile or custom Docker Compose.

One-click services

One-click services are pre-configured Docker Compose templates provided directly by Coolify, skipping the complexity of manual setup and configuration. They are the easiest to start with.

Scroll a bit down and you should see them:

Coolify also maintains a directory for these services on the web. Some of services you can add are:

Internal tools: Appsmith, Budibase, NocoDB.
Chat & messaging: Matrix, Mattermost, Rocket.Chat.
Development & CI/CD: VS Code Server, Gitea, Jenkins.
Monitoring & analytics: Bugsink, CloudBeaver.
CMS: WordPress, Ghost.

They are all pretested, optimized, and coming with sensible defaults.

Custom services

Custom services in Coolify are those that you deploy with your own Docker Compose file. They require a little bit more work, but allow you to run almost anything. Have a look at how to deploy SerpBear, a SEO tool for keyword tracking as an example of a custom service.

Next steps

If you are new to self-hosting make sure to learn a little bit more about SSH, DNS, Linux, Docker, and related topics. You should also have a look at the Coolify documentation for even more information and tips.

Grok AI API tutorial - from Chat to Web Search

Hilman Ramadhan — Mon, 30 Mar 2026 02:03:37 +0000

The xAI Grok API provides access to powerful frontier models like Grok 4 series, supporting chat completions (text + vision), image generation, tool calling (function calling + built-in tools like web search), and more advanced features.

Quick Intro

Sign up at https://x.ai/api
Generate an API key from the console
Install: pip install xai-sdk
Set env var: export XAI_API_KEY="your_key_here"
Models list: https://docs.x.ai/developers/models

I'll share some samples in Python.

Basic Chat API Call

Let's first prepare our project before making the API call

Install the xai-sdk

pip install xai-sdk

Set env var: export XAI_API_KEY="your_key_here" or use .env file

Now, create a new file and this basic setup:

import os
from xai_sdk import Client
from xai_sdk.chat import user, system

from dotenv import load_dotenv
load_dotenv()
XAI_API_KEY =  os.environ.get("XAI_API_KEY")

client = Client(api_key=XAI_API_KEY)

Ensure you can print out your XAI_API_KEY correctly at this stage.

Next, let's call the chat function

...
model = "grok-4-1-fast-non-reasoning"
chat = client.chat.create(model=model)
chat.append(system("You are Grok, a highly intelligent, helpful AI assistant."))
chat.append(user("How can I be a good developer?"))

response = chat.sample()
print(response.content)

Feel free to switch the model based on your needs or preferences.

Here is an example output:

Image Generation API

Let's see how to generate an image with Grok API. We'll need to use the "grok-imagine-image" model for this.

...
response = client.image.sample(
    model="grok-imagine-image",  
    prompt="detective cat searching on website"
)

print(f"Generated image: {response.url}")

The output is a URL like this:

Video Generation API

Generating a video is as easy as generating an image with Grok API. We'll need to use the "grok-imagine-video" model for this.

response = client.video.generate(
    prompt="A glowing crystal-powered rocket launching from the red dunes of Mars, ancient alien ruins lighting up in the background as it soars into a sky full of unfamiliar constellations",
    model="grok-imagine-video",
    duration=10,
    aspect_ratio="16:9",
    resolution="720p",
)
print(response.url)

You can set the duration, aspect ratio, and resolution.

The xAI Grok API features powerful tool calling capabilities, allowing Grok to go far beyond simple text generation. It can take real actions such as performing web searches, running code, retrieving information from your own data sources, or invoking any custom functions you've defined.

Let's start by calling a custom function, as it'll help us call any internal or external API or function.

Let's say we want to call a function to look for an item's price. First, we need to define the function, such as adding the name, description, and parameters.

...
import json
from xai_sdk.chat import user, tool, tool_result
...

# Define tools
tools = [
    tool(
        name="get_item_price",
        description="Get the price of an item from the store",
        parameters={
            "type": "object",
            "properties": {
                "item_name": {"type": "string", "description": "Name of the item to get the price for"},
            },
            "required": ["item_name"]
        },
    ),
]

Upon calling the client method, we now need to include the tool we declared above.

chat = client.chat.create(
    model="grok-4.20-reasoning",
    tools=tools,
)
chat.append(user("What is the price of a laptop?"))
response = chat.sample()

print("========= response ===========")
print(response)
print("==========================")

Important: At this stage, Grok doesn't care if we have the actual function to check the price or not. The AI simply wants to know "what tools are available" for them to use.

Try to run the code to see the output from the chat call.

As you can see, Grok can detect the tool we need to call. You can see it from outputs > message > tool_calls . It consists of the name of the function and the arguments that are extracted from the user's prompt, so it'll be dynamic.

Function call simulation

Next, let's create a fake function to call. In real life, it could be a call to a database or APIs.

def get_item_price(item_name):
    prices = {
        "laptop": 999.99,
        "smartphone": 499.99,
        "headphones": 199.99,
    }
    return {"item_name": item_name, "price": prices.get(item_name, "Item not found")}

Following up from the latest code, we can check if the response has a "tool_calls" object or not. If so, we'll call the actual function we just declared above.

# Handle tool calls
if response.tool_calls:
    chat.append(response)
    for tc in response.tool_calls:
        args = json.loads(tc.function.arguments)
        result = get_item_price(args["item_name"])
        chat.append(tool_result(json.dumps(result)))


    response = chat.sample()

print(response.content)

We need to loop through the tool_calls object
We need to extract the argument to pass to the function
Call the actual function alongside the argument value
Add the information back to our chat method

Now, calling the chat.sample() method, will include all the information we received from calling the "fake function" before.

Let's try with a different prompt

chat.append(user("I need to buy two laptops and a smartphone. Can you tell me how much that will cost?"))

Here is the result

Web Search API

Grok can access real-time information through this feature, so you can get up-to-date content. Unlike the function calling above, we don't need to declare a custom function, as it's an internal tool. Here is a simple example:

import os
from xai_sdk import Client
from xai_sdk.chat import user
from xai_sdk.tools import web_search

from dotenv import load_dotenv
load_dotenv()
XAI_API_KEY =  os.environ.get("XAI_API_KEY")

client = Client(api_key=XAI_API_KEY)

chat = client.chat.create(
    model="grok-4.20-reasoning",  # reasoning model
    tools=[web_search()],
    include=["verbose_streaming"],
)

chat.append(user("Grok VS OpenAI API"))

is_thinking = True

for response, chunk in chat.stream():
    for tool_call in chunk.tool_calls:
        print(f"\nCalling tool: {tool_call.function.name} with arguments: {tool_call.function.arguments}")
    if response.usage.reasoning_tokens and is_thinking:
        print(f"\rThinking... ({response.usage.reasoning_tokens} tokens)", end="", flush=True)
    if chunk.content and is_thinking:
        print("\n\nFinal Response:")
        is_thinking = False
    if chunk.content and not is_thinking:
        print(chunk.content, end="", flush=True)

print("\n\nCitations:")
print(response.citations)

Use tools=[web_search()]
To show what's happening in the process, we use include=["verbose_streaming"],
is_thinking variable is to check if the process is still running (a boolean variable)

As you can see, it'll perform several searches on the internal with different queries. It'll then visit a specific URL after that to get more context.

Allowed domains

You can search only in specific domains using allowed_domains.

 tools=[
        web_search(allowed_domains=["grokipedia.com"]),
    ],

Exclude Domains

Vice versa, you can exclude specific domains

chat = client.chat.create(
    model="grok-4.20-reasoning",
    tools=[
        web_search(excluded_domains=["grokipedia.com"]),
    ],
)

Better Web Search API

While you can specifically choose the domain, the keyword Grok uses to find answers on the internet is random. For example, when I'm asking for "Top 3 pizza restaurants from Google Maps in Boston. Share some reviews and ratings for each place."

This is what I saw from the thinking process:

It needs to perform multiple queries before returning the answer.

Another sample, when asking simply for 3 images:

It runs on multiple pages, and unfortunately, the links are not valid. Grok may hallucinate at this point.

Web Search API alternative

In some cases, AI-generated keywords are fine, but if you're building an app where you want efficiency and full control of the process, the native "Web Search Tool" can be replaced with a simple API call to a specific API your app needs. For example, to find answers on the internet, SerpApi provides 100+ APIs for you.

Need a generic Google answer? We have:

- Google Search API

- Google AI Overview

- Google AI Mode

and more!

See how SerpApi is the Web Search API for your AI apps, LLM, and agents.

Using Grok API with SerpApi

To get some idea of how SerpApi works, feel free to test the results on our playground. You can play with different parameters and directly see the JSON sample we return.

Sample case

Let's say we want to find images via Google Image API like this:

Step 1: Preparation

You can register for free at serpapi.com to get your API key.

Step 2: Parsing keyword

Let's say we need 3 images from Google. Since users can type anything, we need to parse the keyword, as SerpApi simply performs a search using a particular keyword.

USER_QUERY = "Show me 3 cute cat images from the internet"

# Step 1: Ask Grok to extract a search keyword from the user's natural language
keyword_chat = client.chat.create(model="grok-3-fast")
keyword_chat.append(system("Extract the most relevant search keyword or phrase from the user's message. Reply with only the keyword, nothing else."))
keyword_chat.append(user(USER_QUERY))

keyword_response = keyword_chat.sample()
search_keyword = keyword_response.content.strip()
print(f"Extracted keyword: {search_keyword}")

Step 3: Search via SerpApi

We now have the keyword. Let's run a search on SerpApi

# Step 2: Search via SerpAPI using simple requests (Google Images)
serpapi_params = {
    "api_key": SERPAPI_API_KEY,
    "engine": "google_images",
    "q": search_keyword,
    "hl": "en",
    "gl": "us",
}

serpapi_url = "https://serpapi.com/search"
serpapi_response = requests.get(serpapi_url, params=serpapi_params)
results = serpapi_response.json()

At this stage, you already have the answers you're looking for.

Step 4: (Optional) Filter results

Sometimes, we don't need all the information. It's good to filter it programmatically first, so we don't use too many tokens.

For example, I'm only interested in the top 5 answers:

image_results = results.get("images_results", [])
[:5]
formatted_results = "\n".join(
    f"- {img.get('title', 'No title')}: {img.get('original', img.get('thumbnail', 'No URL'))}"
    for img in image_results
)
print(f"\nSerpAPI results:\n{formatted_results}")

We can also format the answer as a bonus.

Step 5: (Optional) Reply in a natural language

Depending on your application, you may want to answer the user back in natural language. We just need to pass the answers above back to the AI:

# Step 3: Feed results back to Grok for a final response
final_chat = client.chat.create(model="grok-3-fast")
final_chat.append(system("You are a helpful assistant. Use the provided search results to answer the user's question."))
final_chat.append(user(f"User question: {USER_QUERY}\n\nSearch results from SerpAPI:\n{formatted_results}\n\nPlease answer the user's question based on these results."))

final_response = final_chat.sample()
print(f"\nFinal Response:\n{final_response.content}")

Final result:

Sidenote

It's also possible to call the API with the OpenAI SDK. Sample:

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("XAI_API_KEY"),
    base_url="https://api.x.ai/v1",
)

Introducing SerpApi Google Travel Explore API

Matteo Vedovati — Thu, 11 Dec 2025 08:46:11 +0000

Google Travel Explore is Google's premier search tool for destination discovery. It helps travelers decide where to go and when to plan their next trip, based on real-time flight deals and hotel pricing. Often found via the homepage of Google Flights, it allows user to browse destinations from a departure city or airport, visually navigating the explore map and uncover affordable trips across entire regions, and interests - whether for specific dates or flexible dates.

SerpApi's new Google Travel Explore API lets you scrape these explore-style travel data via a simple endpoint (engine=google_travel_explore), returning flight prices, trip dates, travel durations, and more-all in structured JSON. No need to reverse-engineer Google's complex travel experience or handle anti-bot measures yourself.

What Is Google Travel Explore?

Google Travel Explore is a powerful, price-based travel discovery interface built directly into the Google Flights search engine. When you use Google Flights for a traditional flight search, you typically need a specific destination and fixed dates (or specific dates) to find airfare. However, this rigid approach can often limit your travel planning options.

In contrast, Google Flights Explore offers far more dynamic functionality. It lets users browse popular destinations based on flexible dates (or flexible date ranges), interests, regions, and budget constraints. Whether you are looking for round-trip, one-way, or multi-city flight options, the explore map visualizes the cheapest flights from your departure city or home airport. You can zoom in on specific regions to find the best flight, filter by cabin class (such as business class), and set preferences for nonstop flights versus layovers.

It is the ultimate search tool to find flight deals before you book flights on an online travel agency or the airline's website. In fact, Explore is the engine behind many popular travel hacks, allowing users to answer an early travel experience question: "Where can I go, and when, for the best flight value?"

How the Explore Search Works

The functionality of the Explore tool changes based on how specific the user's plans are. In all cases, a departure city or airport is required, but the arrival input dictate the mode:

Open-Ended Discovery: Users leave the arrival blank or select a broad area (like "Europe"). This allows them to zoom in and out of the map to see prices change dynamically. This is the cheapest option for travelers who are flexible, as the tool returns a list of destinations where every result has its own "best dates" pre-calculated based on the chosen duration (e.g., "one week") and time window.
Specific Destination Analysis: If a user already knows where they want to go, they can input a specific arrival city. In this case, there is no exploration; the tool simply returns the best available flight options for that route. It provides a clear snapshot of the cost, the airline, the total trip duration, and the number of stops of layovers involved.

Why Scrape Google Travel Explore?

Google Flights Explore display pricing and destination data that's especially valuable for early-stage trip planning. Whether you are building an online travel agency or a niche flight deal site, scraping this data allows you to unlock capabilities that major aggregators often miss and offer users better flight options:

Flexible date discovery: Quickly identify the best time to travel without brute-forcing many date combinations.
Destination discovery: Suggest popular destinations to travel even when the user doesn't have a destination in mind, both globally or within broader regions.
Flight + hotel price context: Understand not just airfare but full Google Flights price picture by including rough lodging costs to estimate total trip value.
Automatically suggested travel windows: Use Explore's proposed date ranges to highlight the best deals and highest-value periods.
Smart budget matching: Filter results to show only destinations that meet user cost constraints, helping them find cheap flights effortlessly.

Building Next-Gen Travel Tools

By leveraging this API, developers can build sophisticated applications that go beyond what a standard airline's website offers. Here is how you can use this data to enhance your users' search:

Advanced Filtering: Create tools that filter by cabin class (Economy vs. Business class), number of stops, or prioritize nonstop flights for convenience.
Cost Analysis: Help users calculate the true cost of travel by factoring in carry-on baggage fees or specific airlines (filtering for airline alliances) to maximize credit card points or savings.
Comparison Features: While standard Google Flights is great for simple lookups, your app could compare multi-city possibilities by aggregating Explore data for different regions.
Actionable Insights: Provide travel tips on when to book flights by programmatically comparing prices across flexible dates. This helps build a custom price tracking engine that identifies the cheapest flight possible for a specific window.

Why use SerpApi for Google Travel Explore Scraping?

Building and maintaining a scraper for Google Travel Explore is challenging: the UI is dynamic, pricing layers update frequently, and anti-bot countermeasures can block automated extraction. SerpApi solves this complexity so you can build on the data:

Structured results: Receive consistently formatted JSON ready for immediate use.
Fully managed scraping and anti-bot protection: need to maintain headless browsers, manage proxy pools, or deal with captchas and rate limits. SerpApi handles them all for you.
Up-to-Date Support: The SerpApi team continuously monitors for UI and structural changes in Google Travel Explore and update extraction logic as needed. Your data pipeline keeps working even as Google evolves the product.
Built for scale: Whether running a few queries or thousands, SerpApi provides robust infrastructure to handle concurrency, traffic spikes, and large-volume workloads without additional engineering effort.

Querying the Google Travel Explore API

Getting started is easy. Sign up at SerpApi and grab your API key. Then you can query the Google Travel Explore API by sending a GET request to SerpApi with engine=google_travel_explore. At minimum, you'll want to specify a departure_id, which represents the city or airport you're flying from. The API can then retrieve suggested destinations, price windows, and travel details. You can also try it interactively in the SerpApi Playground by selecting the "Google Travel Explore" engine.

Key parameters:

departure_id: Indicates where the trip begins. You can pass either an airport code (e.g., JFK) or an Google Knowledge Graph ID (e.g., /m/02_286 → New York City – this ID can be easily found its Wikidata page under Freebase ID).
arrival_area_id: Useful when you want to browse broader areas instead of a single city.
arrival_id: Use when you already know where you want to go, ideal for date selection and routes lookups.

For more informations on all the parameters refer o the API documentation.

Python example

from serpapi import GoogleSearch

params = {
    "engine": "google_travel_explore",
    "departure_id": "/m/02_286",  # New York City
    "api_key": "YOUR_SERPAPI_KEY"
}

search = GoogleSearch(params)
result = search.get_dict()

print("Top destination recommendations:")
for d in result.get("destinations")[:3]:
    name = d.get("name")
    start = d.get("start_date")
    end = d.get("end_date")
    price = d.get("flight_price")
    print(f"{name}: {start} → {end} - ${price}")

This will print the top three recommended destinations from the specified departure city, including their date ranges and flight prices.

JavaScript (Node.js) example

const SerpApi = require('google-search-results-nodejs');
const search = new SerpApi.GoogleSearch("YOUR_SERPAPI_KEY");

search.json(
  {
    engine: "google_travel_explore",
    departure_id: "EWR",     // Airport code
    outbound_date: "2026-01-20",   // fixed start date
    return_date: "2026-01-25",     // fixed return date
  },
  (data) => {
    console.log(`Top destination recommendations for ${params.outbound_date} → ${params.return_date}:`);
    const top3 = data.destinations.slice(0, 3);
    top3.forEach(dest => {
      console.log(`${dest.name}: $${dest.flight_price}`);
    });
  }
);

This will print the top three results for the chosen travel dates, showing each destination and its corresponding flight price.

Try It Out!

The new Google Travel Explore API is available now to all SerpApi users. To get started, create a free SerpApi account (the free tier includes 250 queries/month) and find your API key. Then visit the Playground or use the endpoints above to begin scraping Google Travel Explore answers. You can find API documentation here.

Whether you are building a dedicated flight search engine, a blog sharing travel tips, or a tool to help users book flights cheaper, the Google Travel Explore API makes it simple.

Like what you see? Head to SerpApi’s Blog for a lot more: https://serpapi.com/blog/

Introducing SerpApi's MCP Server

Emirhan Akdeniz — Fri, 05 Dec 2025 17:26:27 +0000

As the AI Agents ecosystem and usage grow, we at SerpApi are looking to support that effort and simplify how AI developers can get fast and reliable access to the latest Google and other Search web data sources. Today, we are releasing an open-source Model Context Protocol (MCP) server that exposes our web search API to AI agents and developers through a familiar and common interface.

This server lets any MCP-compatible AI model client call SerpApi’s search tools without writing any custom code or dealing with complicated SDKs. The agent can ask the server what tools it offers and then invoke them as needed to achieve the desired outcome. SerpApi’s MCP server includes a unified search tool so that LLM (Large Language Models) agents can retrieve real-time search results through simple function calls. The search tool is flexible and supports all schemas and parameters that are supported in our REST API.

The server supports multiple engines (Google, Bing, Yahoo, DuckDuckGo, Yandex, Baidu, YouTube, eBay, Walmart, and more) and specialised queries such as weather or stock lookups, returning clean JSON results optimised for AI application consumption. You may register for SerpApi to claim free credits.
What is the Model Context Protocol (MCP)?

What is the Model Context Protocol (MCP)?

The Model Context Protocol (MCP) is an open protocol created by Antrophic to let AI models operate in richer, more context-aware ways inside real-world development environments and applications. Instead of relying solely on static training data, MCP lets models access live information—such as local files in a file system, project metadata, or connected external systems—all through a unified, standardised protocol. It works over simple transports like stdio, sse, or even USB-C (when implemented over serial-style connections), making it easy to embed into many platforms. To support broad interoperability, MCP also clarifies how transport methods such as server-sent events or process pipes can carry messages between tools and clients. MCP is defined through a clear protocol specification built on JSON-RPC, and many implementations explicitly align with JSON-RPC 2.0, ensuring that tools and clients can communicate reliably without vendor lock-in. As adoption grows, MCP aims to function as a truly standardized protocol that any AI system can use—whether it’s ChatGPT from OpenAI, Claude from Anthropic, Gemini, or any other model—so long as they connect through a compatible endpoint. These foundations also help address security risks by enabling structured access control, controlled sandboxing, and predictable interactions between models and tools.

With MCP, developers can build powerful MCP tools that expose datasets, APIs, services, or an entire workspace to AI assistants and chatbots. This enables models to read docs, run development tools, inspect code, or even debug problems interactively using natural language while still following strict structured messaging. Because everything follows the same protocol, tools become reusable across editors, terminals, and other apps. This growing MCP integration ecosystem means that not only OpenAI models but also third-party AI systems can participate in shared workflows. MCP supports emerging agentic AI patterns, where models autonomously sequence many small tool calls to complete complex use cases, while still respecting boundaries around sensitive data and maintaining scalability as environments grow. With these enhancements, MCP turns AI from a passive text generator into an active participant that can interact with your environment in a safe, structured way—unlocking smarter automation, deeper integrations, and more capable AI-assisted workflows.

Key Features

Multi-Engine Search: A single search tool covers dozens of engines that are providers of external data sources—Google (full and light), Bing, Yahoo, DuckDuckGo, Yandex, Baidu, YouTube, eBay, Walmart, and others.
Structured Results: The server autogenerates structured JSON block templates (external tools such as answer boxes, knowledge graph data, calculations, and more) along with organic, news, image, and shopping results. AI Assistant Agents get exactly what they need without manual parsing.
Raw and Clean Output: Developers can request raw JSON or a clean, normalised version with fields like title, snippet, link, etc.
Robust Reliability: Error handling and retry logic are included by default. Authentication failures, rate limits, and network timeouts return helpful messages that agents can use to respond gracefully.
Open Source: We at SerpApi believe that open source is an integral part of the development of AI Tools. Anyone with some Python knowledge should be able to contribute to Artificial Intelligence tooling and general AI Systems. You may head to our GitHub Repository to access an open-sourced version of the MCP Server.

Getting Started

You can start by using our remotely hosted MCP Server at https://mcp.serpapi.com/SERPAPI_API_KEY/mcp. The MCP server supports requests from any origin.

You can also self-host the MCP server locally to have full control of the deployment. You can start with it by cloning the repository and installing the dependencies:

git clone https://github.com/serpapi/serpapi-mcp.git
cd serpapi-mcp/
uv sync

Set your SerpApi API key in a .env file:

SERPAPI_API_KEY=your_key_here

Then start the server:

uv run src/server.py

Configure your MCP-client to connect to the server. Once connected, the AI-powered agent will detect SerpApi’s search tool and any additional resources the server provides in automation.

You can also use the provided Docker configuration to deploy the service as a container.
Interacting with the server

You can use the MCP inspector to interact and explore the MCP service. You can start by installing the package:

npm install -g  @modelcontextprotocol/inspector

Then start the inspector:

npx @modelcontextprotocol/inspector

The service will become available on localhost:6274. You can try listing the available tools, see their description and parameters.

You can also test the MCP server inside Microsoft’s VS Code IDE. You can add the MCP server config to the .vscode/mcp.json file and test it inside the VS Code Copilot.

{
  "servers": {
    "serpapi-mcp": {
      "type": "http",
      "url": "https://mcp.serpapi.com/<API_KEY>/mcp"
    }
  }
}

Accelerating AI Integration

By adopting the MCP standard, SerpApi makes web search a native capability for AI agents. Instead of bespoke integration code with complicated connectors, agents discover and use SerpApi’s tools automatically. This reduces hallucinations and accelerates the development of agentic applications.

SerpApi releases this server to give AI developers fast, reliable access to live search data. We’re excited to support the growing ecosystem of AI agents and provide the infrastructure that helps them understand the real world with open standards and with minimal permissions needed. The MCP server is open-source, and we invite developers to explore it, contribute, and integrate it into their own AI workflows.

Try out the SerpApi MCP server and see how it fits into your agent stack.

How to scrape 100 search results on Google (num parameter)

Hilman Ramadhan — Thu, 25 Sep 2025 01:48:54 +0000

Finally, there is a solution to scrape 100 search results from Google after they dropped the num parameter: introducing the Google Fast Light API

Google recently discontinued support for showing up to 100 results per page in search, a parameter that many SEO professionals, data companies, and researchers have relied on for years. This change disrupts established workflows, making large-scale data collection, competitor analysis, and keyword research less efficient.

API to scrape 100 search results from Google

Introducing Google Light Fast API

With this API, you can scrape 100 search results from Google with one single request. This API only contains the organic_results , which makes it faster compared to our regular Google Search API.

You can access this API with a single GET request:

https://serpapi.com/search.json?engine=google_light_fast&q=coffee&api_key=YOUR_API_KEY&num=100

You can replace coffee with any keyword you want to search for.
To get your API key, you can register for free at SerpApi.

Scraping other rich results from Google

If you need the complete Google results on the first page, you can still use our Google Search API in order to scrape AI Overview answer, knowledge graph, and so on. So, if you need both 100 organic results and complete rich results , you only need to perform two searches:

1 to get the 100 organic results via the new Google Fast Light API
2 to get the rich results on the Google Search API.

No need to paginate the results or run 10 searches anymore.

Conclusion

The removal of the num=100 (&num=100) parameter from Google has a lot of impacts on many SEO companies that need to serve ranking data for their clients. Many search API providers fail to scrape 100 search results, which results in many errors and misinformation on their platform.

This issue was also mentioned a couple of times on X (Twitter) by many search engine professionals,

Barry Schwartz - founder of the Search Engine Roundtable

Brodie Clark - Independent SEO consultant

SEO companies looking for a way to track their clients’ SEO rankings programmatically, as well as AI companies seeking more real-time data from Google, should try this new API.

How to scrape Tripadvisor (2025 Tutorial)

Hilman Ramadhan — Fri, 19 Sep 2025 02:17:42 +0000

Scraping Tripadvisor listings allows you to gather detailed information about various attractions, hotels, or restaurants, including descriptions, amenities, pricing, and user-generated ratings. This data can be instrumental in understanding market trends, identifying competitors, and optimizing your own listings or offerings in the travel industry.

Luckily, you can do this easily using our brand new Tripadvisor API. We'll see how to scrape it in cURL, Python, and JavaScript. We can scrape restaurants, things to do, hotels, destinations, vacation rentals, and forum information.

Available data on the Tripadvisor Search API

Here is the list of data you can retrieve from this Tripadvisor Search API:

title
description
rating
reviews
location
thumbnail
highlighted overview

This is perfect if you need to collect place data from Tripadvisor.

How to scrape the Tripadvisor website?

Now, let's see how to use this simple API from SerpApi to collect the data!

Get your API Key

First, ensure you register at serpapi.com to get your API Key. You can get 250 free searches per month. You can use this API Key to access all of our APIs, including the Tripadvisor Search API.

Available parameters

On top of running the basic search, you can see all of our available parameters here.

cURL Implementation

Here is the basic implementation in cURL:

curl --get https://serpapi.com/search \
 -d api_key="YOUR_API_KEY" \
 -d engine="tripadvisor" \
 -d q="Rome"

The q parameter is responsible for the search query.

Sample response from cURL request

Scrape Tripadvisor search results in Python

Next, let's see how to scrape the Tripadvisor search results in Python.

Preparation for accessing the SerpApi API in Python

Create a new main.py file
Install requests with:

pip install requests

Here is what the basic setup looks like:

import requests
SERPAPI_API_KEY = "YOUR_REAL_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, #replace with your real API Key
    # soon
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

With these few lines of code, we can access all of the search engines available at SerpApi, including the Tripadvisor Search API.

import requests
SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "tripadvisor",
    "q": "indonesia"
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

To make it easier to see the response, let's add indentation to the output.

import json

# ...
# ...
# all previous code

print(json.dumps(response, indent=2))

Running this Python file should show you the listing for that keyword from Tripadvisor website.

Print specific information

Let's say we only need the title, description, rating, and reviews. This is how we can print specific columns from the locations results:

# ...
search = requests.get("https://serpapi.com/search", params=params)
response = search.json()

for result in response.get("locations", []):
    title = result.get("title")
    rating = result.get("rating")
    reviews = result.get("reviews")
    description = result.get("description")

    print(f"Title: {title}")
    print(f"Rating: {rating}")
    print(f"Reviews: {reviews}")
    print(f"description: {description}")
    print("-" * 10)

Export data to a CSV file

Let's see how to export this Tripadvisor product data into a CSV file in Python

# ...

import csv

with open("tripadvisor_results.csv", "w", newline="", encoding="utf-8") as csvfile:
    fieldnames = ["title", "rating", "reviews", "description"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    for result in response.get("locations", []):
        writer.writerow({
            "title": result.get("title"),
            "rating": result.get("rating"),
            "reviews": result.get("reviews"),
            "description": result.get("description")
        })
print("Data exported successfully.")

JavaScript implementation

Finally, let's see how to scrape the Tripadvisor search results in JavaScript.

Install the serpapi package:

npm install serpapi

Run a basic query:

const { getJson } = require("serpapi");
getJson({
  engine: "tripadvisor",
  api_key: API_KEY, // Put your API Key
  q: "indonesia"
}, (json) => {
  console.log(json["locations"]);
});

Other programming languages

While you can use our APIs using a simple GET request with any programming language, you can also see our ready-to-use libraries here: SerpApi Integrations.

How to customize the search?

We provide many filters to customize your search.

tripadvisor_domain: Parameter defines the Tripadvisor domain to use. It defaults to tripadvisor.com. Head to Tripadvisor domains for a full list of supported domains.
lat and lon: Specify the lat and lon GPS coordinates when performing the search

Example using a different Tripadvisor domain:

curl --get https://serpapi.com/search \
 -d api_key="YOUR_API_KEY" \
 -d engine="tripadvisor" \
 -d q="cafe" \
 -d tripadvisor_domain="wwww.tripadvisor.ca"

Advanced Parameters:

ssrc: This parameter specifies the search filter you want to use for the Tripadvisor search.

Available options:

a - All Results

r - Restaurants

A - Things to Do

h - Hotels

g - Destinations

v - Vacation Rentals

f - Forums

How to paginate the results?

You can scrape beyond the first page using the offset and limit parameter.

limit defines how many results you want to receive per page. By default, it's 30. Maximum value is 100.
Offset skips the given number of results. For example, 0 for the first page, 30 for 2nd page, 60 for 3rd page, and so on. Or if you're using 100 as the limit, then the 2nd page will be 100, 3rd page will be 200, and so on.

Frequently Asked Questions (FAQs)

Is it legal to scrape the Tripadvisor website?

Scraping publicly available data from websites like Tripadvisor is generally permitted under U.S. law.

How much does it cost?

Register at serpapi.com to start for free. If you want to scale, we offer tiered plans based on your usage.

Why do you need to scrape Tripadvisor listing?

Scraping Tripadvisor listings can provide valuable data for market analysis, competitive insights, and understanding customer preferences in the travel and hospitality industry. This information can help businesses optimize their offerings and improve customer engagement.

Closing

That's it! Thank you very much for reading this blog post. You can play around for free on our playground here.

Scrape YouTube videos in Python

Hilman Ramadhan — Fri, 08 Aug 2025 08:10:37 +0000

Scraping YouTube videos enables developers and businesses to extract detailed YouTube video metadata at scale, including titles, descriptions, view counts, thumbnails, channel names, related videos, and comments/replies. It streamlines what would otherwise require complex scraping and anti‑blocking measures.

Get your API Key

First, ensure to register at SerpApi to get your API Key. You can get 250 free searches per month. Use this API Key to access all of our APIs, including the YouTube Video API.

Available parameters

In addition to running the basic search, you can view all YouTube Video API parameters here.

How to scrape YouTube video data with Python

Create a new main.py file
Install requests with:

pip install requests

Here is what the basic setup looks like:

import requests
SERPAPI_API_KEY = "YOUR_REAL_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, #replace with the actual API Key
    # soon
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

With these few lines of code, we can access all of the search engines available at SerpApi, including the YouTube Video API.

import requests
SERPAPI_API_KEY = "YOUR_SERPAPI_API_KEY"

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "youtube_video",
    "v": "j3YXfsMPKjQ" # YouTube video ID
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(response)

To make it easier to see the response, let's add indentation.

import json

# ...
# ...
# all previous code

print(json.dumps(response, indent=2))

Here is the result:

Here is how to scrape the comments on a video.

From the request we performed previously, you should be able to see this in the response.

We can scrape the "Top comments" or "Newest first" comments using the token that is available for each. We can put this token in the next_page_token parameter.

Here is an example:

params = {
    "api_key": SERPAPI_API_KEY, 
    "engine": "youtube_video",
    "v": "j3YXfsMPKjQ", # YouTube video ID
    "next_page_token": "Eg0SC2ozWVhmc01QS2pRGAYyOCIRIgtqM1lYZnNNUEtqUTABeAIwAUIhZW5nYWdlbWVudC1wYW5lbC1jb21tZW50cy1zZWN0aW9u"
}

search = requests.get("https://serpapi.com/search", params=params)
response = search.json()
print(json.dumps(response, indent=2))

Here is the result:

If you're interested in scraping the reply on the comment, you can repeat the same action, but this time using the replies_next_page_token.

Bonus

You can also scrape YouTube search results using our YouTube Search API. Here is how to scrape YouTube search results using Python.

Why Use It?

Real‑time data: Fetch up‑to‑date video details.
Rich metadata access: Retrieve comments and nested replies, related videos, and viewer stats in one request.
Automated proxy and captcha handling: SerpApi manages the complexities of scraping so you can focus on analysis.

Key Features

Feature: Comprehensive video metadata
- Benefit: Includes title, description, duration, chapters, views, publishing date, channel, thumbnails
Feature: Comments & replies
- Benefit: Access nested comments for engagement analysis
Feature: Related videos
- Benefit: Discover context and competition around a video

Build a SERP rank tracker app with this API

Hilman Ramadhan — Mon, 23 Sep 2024 07:48:06 +0000

If you're interested in building a SERP ranking tracker app, you'll love our API. SerpApi provides a simple API to access live data from various search engines, including Google, Bing, DuckDuckGo, Yahoo, and others. It enables you to build an app like a SERP ranking tracker.

The idea

To get the ranking position of a website, we need to access the organic results and check where the domain first appears. The organic results data is available through our API. Here are the three APIs we're going to use:

Google Search API: Scrape the results from Google search.
Bing Search API: Scrape the results from the Bing search.
DuckDuckGo Search API: Scrape the results from the DuckDuckGo search.

API Design

We'll create a single endpoint where people can receive the ranking results from the above search engine using this parameter:

Domain name (String): The website domain we want to track.
Keywords (Array): List of keywords that we want to search for.
Engines (Object): List of search engines we want to search on. We can also adjust the parameter details based on the API. Refer to the relevant documentation to check the available parameters for each APIs.

POST: /api/rankings

Data:
 - domain (string)
 - keywords (array[string])
 - engines ((array[name, params]))

Example:
{
  "domain": "archive.org",
  "keywords": ["internet archive"],
  "engines": [
    {
      "name": "google",
      "params": {
        "domain": "google.com",
        "gl": "es"
      }
    }
  ]
}

The source code is available on GitHub; feel free to take a look at the detailed implementation here:
https://github.com/hilmanski/rank-tracker-api/

Let's write the code!

I'll use Nodejs for this API; feel free to use other languages/frameworks.

Install Express

Let's use Express to help us clean up the code structure.

npm i express --save

Export your API Key

You can either export your API key in a terminal like the sample below or save it on an .env file.

export SERPAPI_API_KEY=YOUR_ACTUAL_API_KEY

Basic route

Prepare the POST endpoint with relevant parameters

const express = require('express')
const app = express()
const port = 3000

app.use(express.json());

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

  // detail implementation later

  res.json({ keywords, engines });
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Validate the input type

Make sure API users use the correct types.

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

   // Validate keywords
  if (!Array.isArray(keywords) || !keywords.length) {
    return res.status(400).json({ error: 'Keywords and engines must be arrays.' });
  }

  // Validate engines
  for (const engine of engines) {
    if (typeof engine !== 'object' || !engine.name) {
      return res.status(400).json({ error: 'Each engine must be an object with a "name" property.' });
    }
    if (engine.params && typeof engine.params !== 'object') {
      return res.status(400).json({ error: 'Engine "params" must be an object.' });
    }
  }

  // coming soon

})

Run parallel search

Since we're enabling multiple keywords and multiple search engines, we need to run the function in parallel to save us some time.

// Parallel search
  const results = await Promise.all(engines.map(async engine => {
    const rankings = await Promise.all(keywords.map(async keyword => {
      return await getRanking(keyword, engine, cleanDomain);
    }));

    // map keywords - rankings in one array
    const rankingResults = keywords.map((keyword, index) => {
      return [keyword, rankings[index]];
    });

    console.log(rankingResults);

    return { domain, engine, rankingResults };
  }))

*getRanking method coming soon.

GetRanking method implementation

Here is the function that is responsible for running the search for each search engine.

const suportEngines = ['google', 'bing', 'duckduckgo'];

async function getRanking(keyword, engine, domain) {
  const engineName = engine.name.toLowerCase();

  if(!suportEngines.includes(engineName)) {
      console.error(`Error: Engine ${engineName} is not supported.`);
      return;
  }

  return new Promise(async (resolve, reject) => {
      switch(engineName) {
          case 'google':
            resolve(await searchGoogle(keyword, engine.params, domain))
          break;
          case 'bing':
            resolve(await searchBing(keyword, engine.params, domain))
          break;
          case 'duckduckgo':
            resolve(await searchDuckDuckGo(keyword, engine.params, domain))
          break;
          default:
          break;
      }
  })
}

We'll use the native fetch method in NodeJS to request the actual SerpApi endpoint for each search engine.

API to access ranking position in Google

function searchGoogle(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=google&num=100&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

API to access ranking position in Bing

function searchBing(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=bing&count=50&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

API to access ranking position in DuckDuckGo

function searchDuckDuckGo(keyword, params, domain) {
  let endpoint = `https://serpapi.com/search?q=${keyword}&engine=duckduckgo&api_key=${SERPAPI_API_KEY}`
  if(params) {
      endpoint += `&${new URLSearchParams(params).toString()}`
  }

  return fetch(endpoint)
    .then(response => response.json())
    .then(data => {
      const organic_results = data.organic_results;
      let ranking = organic_results.findIndex(result => result.link.includes(domain))
      return ranking + 1;
    })
    .catch(error => {
      console.error(error);
    });
}

Final Endpoint

Here's what our endpoint looks like

app.post('/api/rankings', async(req, res) => {
  const { keywords, engines, domain } = req.body;

  // Validate keywords
  if (!Array.isArray(keywords) || !keywords.length) {
    return res.status(400).json({ error: 'Keywords and engines must be arrays.' });
  }

  // Validate engines
  for (const engine of engines) {
    if (typeof engine !== 'object' || !engine.name) {
      return res.status(400).json({ error: 'Each engine must be an object with a "name" property.' });
    }
    if (engine.params && typeof engine.params !== 'object') {
      return res.status(400).json({ error: 'Engine "params" must be an object.' });
    }
  }

  // CLean up domain
  // Since people can include https:// or http:// or a subdomain, strip all of it?
  const cleanDomain = domain.replace(/^https?:\/\//, '').replace(/\/$/, '');

  // Parallel search
  const results = await Promise.all(engines.map(async engine => {
    const rankings = await Promise.all(keywords.map(async keyword => {
      return await getRanking(keyword, engine, cleanDomain);
    }));

    // map keywords - rankings in one array
    const rankingResults = keywords.map((keyword, index) => {
      return [keyword, rankings[index]];
    });

    console.log(rankingResults);

    return { domain, engine, rankingResults }
  }))

  res.json(results);
})

Test the API

Let's try out this API via cURL.

No parameter example

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
  -d '{
    "keywords": ["internet archive"],
    "domain": "archive.org",
    "engines": [
      {
       "name": "google"
     }
    ]
  }'

Using multiple keywords and search engine parameter sample

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
-d '{
    "keywords": ["internet archive", "digital library archived internet"],
    "domain": "archive.org",
    "engines": [
      {
        "name": "google",
        "params": {
            "google_domain": "google.co.id",
            "gl": "id"
        }
      }
    ]
  }

Using multiple search engines

curl -X POST http://localhost:3000/api/rankings \
  -H 'Content-Type: application/json' \
  -d '{
    "keywords": ["internet archive", "digital archive", "internet library"],
    "domain": "archive.org",
    "engines": [
    {
      "name": "google",
      "params": {
        "domain": "google.com",
        "gl": "es"
      }
    },
    {
      "name": "Bing",
      "params": {
        "cc": "gb"
      }
    },
{
      "name": "duckduckgo",
      "params": {
        "kl": "uk-en"
      }
    }
  ]
  }'

That's it! I hope you like this post. Please let us know if you have any questions. Feel free to also contribute to this project on GitHub.

Spiking Neural Networks

Emirhan Akdeniz — Wed, 12 Jun 2024 09:40:00 +0000

In this blog post, we will discuss the differences between spiking neural networks, and non-spiking neural networks, potential use cases of these algorithms, and open source a simple example script to compare a simple SNN model to an ANN model.

At SerpApi Blog, we discuss different topics around web scraping:

What is a spiking neural network?

A spiking neural network (SNN) is a type of artificial neural network that more closely mimics the behavior of biological neurons. Unlike traditional artificial neural networks (ANNs), which use continuous activation functions, SNNs use discrete events called spikes. These spikes represent the times at which neurons fire and enable the network to process information in a way that is more similar to synapses of the brain. This enabling mechanism is what makes SNNs superior to ANNs in inference times for temporal (in-time steps) data.

What is the difference between spiking and non-spiking neural networks?

The primary difference between spiking and non-spiking neural networks lies in how they handle information processing:

Spiking neural networks: Use spikes to represent information, with firing neurons only when their membrane potential reaches a certain threshold. These neuronal modelings are often associated with neuromorphic computing and use learning rules such as spike-timing-dependent plasticity (STDP) instead of backpropagation.
Non-spiking neural networks (e.g. ANNs): Use continuous activation functions like ReLU or sigmoid to process information and typically use backpropagation for learning.

How Does Spiking Neural Network Work?

SNNs work by simulating the behavior of biological neurons. When a neuron’s membrane potential exceeds a certain threshold, it generates a spike that propagates to other spiking neurons. These spikes can modify synaptic weights through learning rules such as spike-timing-dependent plasticity (STDP), enabling the network to learn from temporal patterns in the data.

What are the most effective ways to use neural networks for pattern recognition?

For pattern recognition, deep learning models like convolutional neural network models (CNNs) are highly effective. However, SNNs are gaining attention for their ability to recognize spatiotemporal (belonging to both space and time or to space-time) patterns with high precision and low power consumption.

What could it mean for the future of scraping?

In my humble opinion, SNNs could hold a place in finding patterns within changing and evolving HTML structures. Instead of classifying items and parsing them, SNNs may be useful for identifying where specific parts of the HTML are within the overall body. This could reduce human interaction and pave the way for the future of fully automated parsers with higher precision and lower inference times.

SNN vs ANN Comparison

The following is a simple demonstration script for comparing SNN and ANN models under the same conditions. I have to give a disclaimer that this is for demonstration purposes, and not proving purposes, or definite benchmarking. As I repeat time and again, I am not an expert in machine learning, I am just an enthusiast.

Let's import the libraries. We will be using PyTorch for the framework, sklearn for simple dataset-splitting tasks, and snntorch for creating SNN models in PyTorch:


import numpy as np

import torch

import torch.nn as nn

import torch.optim as optim

import snntorch as snn

from sklearn.model_selection import train_test_split

import time

Let's create a function that simulates motion data (a kind of temporal data):


def generate_motion_data(num_samples, event_length, num_events, noise_level):
    X = []
    y = []
    for _ in range(num_samples):
        motion_indices = np.random.randint(0, event_length, size=num_events)
        event_data = np.zeros(event_length)
        event_data[motion_indices] = 1
        noise = np.random.normal(0, noise_level, size=event_length)
        event_data += noise

        # Introduce variability in the patterns
        if np.random.rand() < 0.5:
            event_data = np.roll(event_data, np.random.randint(1, event_length))

        X.append(event_data)
        y.append(1 if np.sum(event_data) > 0 else 0)
    return np.array(X), np.array(y)

The output of this encoding will be a binary where 1 representing motion and 0 representing no motion. We also introduce a Gaussian noise with standard deviation to make the data more consistent with real-world data. Alongside noise, we introduce some random variability patterns to make the task harder. The model should be able to take into consideration all of these factors and predict the motion output within the series.

Let's create our data:


# Parameters

num_samples = 1000

event_length = 100

num_events = 100

noise_level = 0.1

# Generate data

X, y = generate_motion_data(num_samples, event_length, num_events, noise_level)

# Convert to PyTorch tensors

X = torch.tensor(X, dtype=torch.float32)

y = torch.tensor(y, dtype=torch.float32)

# Split into training and validation sets

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

Let's define an SNN model and train it:


# Define SNN model
class SpikingNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(SpikingNN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.lif1 = snn.Leaky(beta=0.9)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        mem1, spk1 = self.lif1(x)
        x = self.fc2(spk1)
        return x

# Model, loss function, and optimizer for SNN
input_dim = event_length
hidden_dim = 64
output_dim = 1  # Binary classification

snn_model = SpikingNN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(snn_model.parameters(), lr=0.001)

# Training loop for SNN
num_epochs = 100
snn_training_start = time.time()

for epoch in range(num_epochs):
    snn_model.train()
    optimizer.zero_grad()
    outputs = snn_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    snn_model.eval()
    with torch.no_grad():
        val_outputs = snn_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'SNN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

snn_training_time = time.time() - snn_training_start
print(f"SNN Training Time: {snn_training_time:.4f} seconds")

Let's create an ANN model and train it for comparison:


# Define ANN model

class ANN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(ANN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Model, loss function, and optimizer for ANN
ann_model = ANN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(ann_model.parameters(), lr=0.001)

# Training loop for ANN
ann_training_start = time.time()

for epoch in range(num_epochs):
    ann_model.train()
    optimizer.zero_grad()
    outputs = ann_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    ann_model.eval()
    with torch.no_grad():
        val_outputs = ann_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'ANN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

ann_training_time = time.time() - ann_training_start
print(f"ANN Training Time: {ann_training_time:.4f} seconds")

Let's define a function to run predictions, calculate the inference time, and compare the two models:


# Function to predict and measure inference time

def predict_and_measure_time(model, new_data):
    start_time = time.time()
    model.eval()
    with torch.no_grad():
        new_data_tensor = torch.tensor(new_data, dtype=torch.float32)
        outputs = model(new_data_tensor)
    inference_time = time.time() - start_time
    return outputs, inference_time

# Generate new test data
X_test, y_test = generate_motion_data(5, event_length, num_events, noise_level)

# Predictions with SNN
snn_outputs, snn_inference_time = predict_and_measure_time(snn_model, X_test)
snn_predictions = torch.round(torch.sigmoid(snn_outputs)).squeeze().numpy()
print("SNN Predictions:", snn_predictions)
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")

# Predictions with ANN
ann_outputs, ann_inference_time = predict_and_measure_time(ann_model, X_test)
ann_predictions = torch.round(torch.sigmoid(ann_outputs)).squeeze().numpy()
print("ANN Predictions:", ann_predictions)
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Comparison Summary
print(f"Comparison Summary:")
print(f"SNN Training Time: {snn_training_time:.4f} seconds")
print(f"ANN Training Time: {ann_training_time:.4f} seconds")
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Final validation accuracies (from the last epoch)
snn_model.eval()
with torch.no_grad():
    snn_val_outputs = snn_model(X_val)
    snn_val_accuracy = ((torch.sigmoid(snn_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

ann_model.eval()
with torch.no_grad():
    ann_val_outputs = ann_model(X_val)
    ann_val_accuracy = ((torch.sigmoid(ann_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

print(f"Final SNN Validation Accuracy: {snn_val_accuracy:.4f}")
print(f"Final ANN Validation Accuracy: {ann_val_accuracy:.4f}")

Following numbers are not definitive as the task is simple. But it should give a clear idea as to how SNN can beat ANN:

Comparison Summary:

SNN Training Time: 0.6785 seconds

ANN Training Time: 0.3952 seconds

SNN Inference Time: 0.0007 seconds

ANN Inference Time: 0.0017 seconds

Final SNN Validation Accuracy: 1.0000

Final ANN Validation Accuracy: 1.0000

It took more time to train SNN due to framework and the architecture not in favor of training the model. However in the future we may see different approaches to optimize training time of SNN models. For inference time, SNN was faster than ANN model. This is where the energy efficiency is taking place. Because SNN is much easier to execute under same accuracy, it is consuming less CPU power.

Are spiking neural networks the future?

Spiking neural networks (SNNs) are increasingly being considered as a promising frontier in the future of artificial intelligence, particularly due to their closer resemblance to the neural computation seen in biological brains. Leveraging principles from neuroscience, SNNs process information through spikes, or action potentials, which offers a unique form of temporal coding and rate coding.

In contrast to traditional deep neural networks, which rely on gradient descent and back propagation, SNNs utilize spike-based learning algorithms and synaptic plasticity, making them more efficient in certain types of neural computation. The initialization of these spiking networks involves setting up multi-layer architectures capable of handling the temporal dynamics and correlations within spike trains.

One of the significant advantages of SNNs is their potential for lower energy consumption, especially when implemented on neuromorphic hardware. These processors mimic the brain’s architecture and function, enabling real-time processing with minimal latency. This is particularly beneficial in applications like robotics, computer vision, and large-scale network models, where real-time and efficient computations are crucial.

SNNs also offer improved interpretability compared to traditional deep-learning models. Each single neuron in an SNN can be examined for its specific role in the network, which aids in understanding how neural computations propagate through the system. Feedforward and recurrent neural networks can both be implemented within the SNN framework, providing versatility in handling different types of data and tasks.

Despite these advantages, SNNs face challenges in terms of learning algorithms and network models. The nonlinear nature of spike-based communication and the need for precise temporal synchronization complicate the development of effective supervised learning techniques. Additionally, the number of spikes and their timing (latency) play a crucial role in the plausibility and performance of SNNs.

Recent advances in state-of-the-art neuromorphic processors and spiking neuron models show promise for overcoming these hurdles. As research in neuroscience and artificial intelligence continues to converge, SNNs may become more viable for practical applications, enhancing the capabilities of both AI and computational neuroscience.

In summary, spiking neural networks hold significant potential for the future of AI, particularly in areas requiring efficient, real-time processing with low energy consumption. Their biologically inspired approach offers a plausible and powerful alternative to traditional deep learning, potentially revolutionizing fields such as robotics, computer vision, and beyond.

Conclusion

I’m grateful to the reader for their attention. I hope this blog post will help people understand the potential of spiking neural networks and their use cases. The reader may find the full comparison script below:

Full script:


!pip install torch snntorch scikit-learn

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import snntorch as snn
from sklearn.model_selection import train_test_split  
import time

# Generate synthetic event-based motion data
def generate_motion_data(num_samples, event_length, num_events, noise_level):
    X = []
    y = []
    for _ in range(num_samples):
        motion_indices = np.random.randint(0, event_length, size=num_events)
        event_data = np.zeros(event_length)
        event_data[motion_indices] = 1
        noise = np.random.normal(0, noise_level, size=event_length)
        event_data += noise

        # Introduce variability in the patterns
        if np.random.rand() < 0.5:
            event_data = np.roll(event_data, np.random.randint(1, event_length))

        X.append(event_data)
        y.append(1 if np.sum(event_data) > 0 else 0)
    return np.array(X), np.array(y)

# Parameters
num_samples = 1000
event_length = 100
num_events = 100
noise_level = 0.1

# Generate data
X, y = generate_motion_data(num_samples, event_length, num_events, noise_level)

# Convert to PyTorch tensors
X = torch.tensor(X, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.float32)

# Split into training and validation sets
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Define SNN model
class SpikingNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(SpikingNN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.lif1 = snn.Leaky(beta=0.9)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        mem1, spk1 = self.lif1(x)
        x = self.fc2(spk1)
        return x

# Model, loss function, and optimizer for SNN
input_dim = event_length
hidden_dim = 64
output_dim = 1  # Binary classification

snn_model = SpikingNN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(snn_model.parameters(), lr=0.001)

# Training loop for SNN
num_epochs = 100
snn_training_start = time.time()

for epoch in range(num_epochs):
    snn_model.train()
    optimizer.zero_grad()
    outputs = snn_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    snn_model.eval()
    with torch.no_grad():
        val_outputs = snn_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'SNN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

snn_training_time = time.time() - snn_training_start
print(f"SNN Training Time: {snn_training_time:.4f} seconds")

# Define ANN model
class ANN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(ANN, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Model, loss function, and optimizer for ANN
ann_model = ANN(input_dim, hidden_dim, output_dim)
criterion = nn.BCEWithLogitsLoss()  # Binary cross-entropy loss
optimizer = optim.Adam(ann_model.parameters(), lr=0.001)

# Training loop for ANN
ann_training_start = time.time()

for epoch in range(num_epochs):
    ann_model.train()
    optimizer.zero_grad()
    outputs = ann_model(X_train)
    loss = criterion(outputs.squeeze(), y_train)
    loss.backward()
    optimizer.step()

    # Calculate training loss
    train_loss = loss.item()

    # Validation
    ann_model.eval()
    with torch.no_grad():
        val_outputs = ann_model(X_val)
        val_loss = criterion(val_outputs.squeeze(), y_val)
        val_loss = val_loss.item()

    print(f'ANN Epoch {epoch+1}/{num_epochs}, Loss: {train_loss:.4f}, Validation Loss: {val_loss:.4f}')

ann_training_time = time.time() - ann_training_start
print(f"ANN Training Time: {ann_training_time:.4f} seconds")

# Function to predict and measure inference time
def predict_and_measure_time(model, new_data):
    start_time = time.time()
    model.eval()
    with torch.no_grad():
        new_data_tensor = torch.tensor(new_data, dtype=torch.float32)
        outputs = model(new_data_tensor)
    inference_time = time.time() - start_time
    return outputs, inference_time

# Generate new test data
X_test, y_test = generate_motion_data(5, event_length, num_events, noise_level)

# Predictions with SNN
snn_outputs, snn_inference_time = predict_and_measure_time(snn_model, X_test)
snn_predictions = torch.round(torch.sigmoid(snn_outputs)).squeeze().numpy()
print("SNN Predictions:", snn_predictions)
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")

# Predictions with ANN
ann_outputs, ann_inference_time = predict_and_measure_time(ann_model, X_test)
ann_predictions = torch.round(torch.sigmoid(ann_outputs)).squeeze().numpy()
print("ANN Predictions:", ann_predictions)
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Comparison Summary
print(f"Comparison Summary:")
print(f"SNN Training Time: {snn_training_time:.4f} seconds")
print(f"ANN Training Time: {ann_training_time:.4f} seconds")
print(f"SNN Inference Time: {snn_inference_time:.4f} seconds")
print(f"ANN Inference Time: {ann_inference_time:.4f} seconds")

# Final validation accuracies (from the last epoch)
snn_model.eval()
with torch.no_grad():
    snn_val_outputs = snn_model(X_val)
    snn_val_accuracy = ((torch.sigmoid(snn_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

ann_model.eval()
with torch.no_grad():
    ann_val_outputs = ann_model(X_val)
    ann_val_accuracy = ((torch.sigmoid(ann_val_outputs) > 0.5).squeeze().float() == y_val).float().mean().item()

print(f"Final SNN Validation Accuracy: {snn_val_accuracy:.4f}")
print(f"Final ANN Validation Accuracy: {ann_val_accuracy:.4f}")

Create a super fast AI assistant with Groq (Without a database)

Hilman Ramadhan — Thu, 16 May 2024 02:47:59 +0000

Last week, I tried to build a voice AI assistant using OpenAI AI assistant. It takes a while to generate a response, which is not suitable for a voice assistant. So, I'm looking for an alternative to make my assistant faster. That's how I found out about Groq. This post will cover how I build an AI assistant using Groq.

Pros and Cons summary

Pro:

Easy to implement with only one API (Groq API).

Respond is fast.

Cons:

The longer we chat, the higher the chance that we might lose some context along the way.

What is Groq?

Groq is a service that provides a super fast engine to run AI applications. It's not an AI model! We can run different AI models like Llama, Mixtral, Gemma and more!

Ref: Why Groq?

How I build a fast AI assistant

Many AI models exist, but only OpenAI offers an easy way to implement a chat-like experience using the Assistants API. By default, these models won't know or understand the context of our previous chat. So, we have to re-explain everything if we want the AI to understand the context of each message.

There are some alternatives out there, such as using LangChain chat history. But I prefer to find a simple way (*with the caveat, of course). Luckily, I found some ideas on the internet (Thank you, Internet!).

The idea below can be implemented for any AI model/engine, not just Groq. You can try this with OpenAI itself, Mixtral, Claude, and so on.

Here is the flow:

The user sends the initial message
The AI responds to the message
We ask AI to summarize the conversation
We send the response and summary back to the user
The user will send the summary back later alongside the new message
AI now will reply based on the fresh message and with help of the conversation summary to provide some context.

The caveat of this method

By summarizing a conversation, we may lose some information along the way. That's why it's a good idea in certain cases to store the message history on a database (Vector database).

One way I can reduce this shortage is by attaching the recent reply from AI. I've also read an article that suggests keeping the latest 2-3 conversations and providing them as additional context later.

Code implementation

I'll use NodeJs for this tutorial. Feel free to use any language you want. The final code is available at GitHub:

GitHub -assistants-api-with-groq-ai

Install dependencies

npm i express groq-sdk dotenv --save

Express for creating a route for the endpoint
Groq-sdk is the official package for using Groq in Javascript
dotenv to store our API key safely.

Add API Key

Create a new .env file. Add your Groq API key in this file like this:

GROQ_API_KEY=YOUR_GROQ_API_KEY

Make sure to sign up to Groq and get your API key here.

Basic Setup

Let's create a new index.jsfile, and we'll write everything in this file. We prepare one endpoint called chat where we'll send these parameters:

- message: user's message

- latestReply: The latest reply from AI

- messageSummary: The conversation summary so far

In this endpoint, we'll do two things:

- Respond to new user message (with latestReply and messageSummary as context)

- Create a new conversation summary by providing the fresh reply from AI.

const express = require('express');

// Express Setup
const app = express();
app.use(express.json());
const port = 3000

require("dotenv").config();
const { GROQ_API_KEY } = process.env;

// GROQ Setup
const Groq = require("groq-sdk");
const groq = new Groq({
    apiKey: GROQ_API_KEY
});

async function chatWithGroq() { } // soon
async function summarizeConversation() { } // soon

app.post('/chat', async (req, res) => {
    const { message, latestReply, messageSummary } = req.body;

    // request chat completion
    const reply = await chatWithGroq(message, latestReply, messageSummary)

    // request chat summary
    const summary = await summarizeConversation(message, reply, messageSummary)

    // Always return chat history/summary
    res.send({
        reply,
        summary
    })
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Chat with Groq method

Here is the chatWithGroq method implementation:

async function chatWithGroq(userMessage, latestReply, messageHistory) {
    let messages = [{
        role: "user",
        content: userMessage
    }]

    if(messageHistory != '') {
        messages.unshift({
            role: "system",
            content: `Our conversation's summary so far: """${messageHistory}""". 
                     And this is the latest reply from you """${latestReply}"""`
        })
    }

    console.log('original message', messages)

    const chatCompletion = await groq.chat.completions.create({
        messages,
        model: "llama3-8b-8192"
    });

    const respond = chatCompletion.choices[0]?.message?.content || ""
    return respond
}

We only provide a conversation summary when we have one (look at the if statement). So, it won't be included in our first message.

Conversation summary method

Here is the summarizeConversation method implementation:

async function summarizeConversation(message, reply, messageSummary) {
    let content = `Summarize this conversation 
                    user: """${message}""",
                    you(AI): """${reply}"""
                  `

    // For N+1 message
    if(messageSummary != '') {
        content = `Summarize this conversation: """${messageSummary}"""
                    and last conversation: 
                    user: """${message}""",
                    you(AI): """${reply}"""
                `
    }

    const chatCompletion = await groq.chat.completions.create({
        messages: [
            {
                role: "user",
                content: content
            }
        ],
        model: "llama3-8b-8192"
    });

    const summary = chatCompletion.choices[0]?.message?.content || ""
    console.log('summary: ', summary)
    return summary
}

In this method, we ask the AI to create a summary based on the latest summary and recent reply.

Demo Time!

You can use any API client, like Postman, Thunder (VS Code), etc.

Don't forget to run your program with node index.js

Create a POST request for the /chat endpoint and provide message endpoint and provide the first message parameter.

We can display the reply from the response on our user interface. This is the actual reply to our message.

We'll save the summary for the next request.

Now, this is how the JSON looks like for the N+1 message:

The next messages should include the latestReply and messageSummary as parameters.

message: *Don't forget to add a new message. This is you talking to the AI. Notice that I use here on my question, to validate that the AI knows what's the previous context here.
latestReply: Send the latest reply from AI (from previous response)
messageSummary: Send the conversation summary so far (from previous response)

Here is the result to this request:

As you can see, the AI knows that when I said here I was talking about Indonesia. You can try to send a follow-up message (create a new request) by asking something like "Can you tell me more about number 4?" as an example. But don't forget that we always need to update the latestReply and summaryConversation on each request.

To return the response and summarize the conversation, I only need to wait around 2s. This is much faster than using OpenAI AI assistants.

Reference:

- Build a smart AI voice assistant

- Basic tutorial: Assistants API by OpenAI

Build an AI Voice assistant like Siri (use OpenAI AI Assistant)

Hilman Ramadhan — Tue, 30 Apr 2024 00:23:48 +0000

Hi! Today, we'll learn how to build an AI Voice assistant like Siri that can understand what we say and speak back to us.

Demo AI Voice assistant

Here's what we're going to build:

This post will focus on implementing it as a web application. Therefore, we will use HTML for the interface and Javascript for the voice features. You might want to adjust this if you build for another platform (mobile, desktop, etc.).

Regardless of the platform, knowing the components of how to build this will help us along the way.

Basic Voice AI assistant Structure

Just like other applications, we will have this basic structure:

Here is what it looks like on our app:

You can replace each of these elements with the more advanced option. I'm trying to stick with what we already have in the browser. These are the alternatives:

Voice input: AssemblyAI or OpenAI Whisper
Voice output: Elevenlabs or OpenAI Whisper

Code Tutorial on how to build a Voice AI assistant

We'll split our codebase into two parts, one for the front end and one for the back end.

First, we just want to make sure that we can get a text from the user's voice and read a text out loud; there is no AI or conversation involved yet.

Here is the final code result of this post:
GitHub - hilmanski/simple-ai-voice-assistant-openai-demo

Step 1: Basic frontend Views

Since we're building a web application, we'll use HTML.

- We need two buttons to start and stop the recording

- A div to display the text

<button id="record">Record</button>
<button id="stop">Stop</button>

<div id="output">Output</div>

(Optional) If you want to copy the style I implemented:

<style>
body {
    margin: 50px auto;
    width: 500px;
}

#output {
    margin-top: 20px;
    border: 1px solid #000;
    padding: 10px;
    height: 200px;
    overflow-y: scroll;
}

#output p:nth-child(even) {
    background-color: #f8f6b1;
}
</style>

Step 2: Listen to speech

Let's trigger the actions from Javascript:

<script>
    // Set up
    const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition;
    const SpeechGrammarList = window.SpeechGrammarList || window.webkitSpeechGrammarList;
    const SpeechRecognitionEvent = window.SpeechRecognitionEvent || window.webkitSpeechRecognitionEvent;

    const recognition = new SpeechRecognition();
    const speechRecognitionList = new SpeechGrammarList();

    recognition.grammars = speechRecognitionList;
    recognition.continuous = true;
    recognition.lang = 'en-US';
    recognition.interimResults = false;
    recognition.maxAlternatives = 1;


    // Start recording
    document.getElementById('record').onclick = function() {
        recognition.start();
    }

    // Stop recording
    document.getElementById('stop').onclick = function() {
        recognition.stop();
        console.log('Stopped recording.');
    }

    // Output
    recognition.onresult = async function(event) {
        // Get the latest transcript 
        const lastItem = event.results[event.results.length - 1]
        const transcript = lastItem[0].transcript;
        document.getElementById('output').textContent = transcript;
        recognition.stop(); 
        // await sendMessage(transcript); // we'll implement this later
    }

    recognition.onspeechend = function() {
        recognition.stop();
    }
</script>

Step 3: Backend structure

We're using NodeJS/Express for the backend. Make sure to install the necessary packages in your new directory (separate from the frontend code):

npm init -y #initialize NPM package
npm init express cors dotenv openai --save
touch index.js #create a new empty file

const express = require('express');
const cors = require('cors')

// Setup Express and allow CORS
const app = express();
app.use(express.json());
app.use(cors()) // allow CORS for all origins

// Main route
app.post('/message', async (req, res) => {
    const { message } = req.body;
    res.json({ message: 'Received: ' + message });
});

// Start the server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
});

Run node js in your terminal to run the server. Your application is running on localhost:3000.

We only have one route, which is /message , which receives a message from the client and echoes it back. This is to ensure that the input and output parts are running smoothly.

Step 4: Send a message from frontend

Let's add a fetch method to send the message to that endpoint.

Update your recognition.onresult event to call the sendMessage function like this:

recognition.onresult = async function(event) {
    // Get the latest transcript 
    const lastItem = event.results[event.results.length - 1]
    const transcript = lastItem[0].transcript;
    document.getElementById('output').textContent = transcript;

    await sendMessage(transcript); // New addition
}

Now, let's declare the sendMessage method:

async function sendMessage(message) {
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message })
    });

    const data = await response.json();
    console.log(data);

    speak(data.message);
}

function speak(message) {
    if (synthesis) {
        const utterance = new SpeechSynthesisUtterance(message);
        synthesis.speak(utterance);
    }
}

I also added a new method called speak to use the Web Speech API, specifically SpeechSynthesis method to speak. The SendMessage will hit the endpoint we prepared previously.

Step 5: Try the App

Now, run your HTML file, you can use something like VS Code Live server.

press record
say anything

AI Voice assistant web application screenshot

Now, you should see your message echo back to you (it's coming from the server). Press the record button again to send a different message.

Now, we have an app that can listen and speak to us. Let's dive into the AI part!

Smart AI assistant

We'll use the OpenAI Assistants API as the brain.

Step 1: Install the OpenAI package

We've previously installed the OpenAI package for NodeJS. Make sure you've installed it as well.

Step 2: Grab your API Key

Get your API Key from the OpenAI dashboard. Create a new .env file and paste it there.

OPENAI_API_KEY=YOUR_API_KEY_FROM_OPENAI

Step 3: Import OpenAI and dotenv

require("dotenv").config();
const OpenAI = require('openai');
const { OPENAI_API_KEY } = process.env;

// Set up OpenAI Client
const openai = new OpenAI({
    apiKey: OPENAI_API_KEY,
});

Step 4: Implement assistants API

If you're not familiar with the assistants API, I suggest to read this introduction blog post first:

[

Assistant API by OpenAI (Basic Tutorial)

Learn the basics of Assistant API by OpenAI. We’ll create a super simple coding example to understand the barebone of Assistant API: https://serpapi.com/blog/assistant-api-openai-beginner-tutorial/

We'll use the same logic and code for this tutorial (with some updates soon). You can also get the code sample for the API assistants here:

GitHub - hilmanski/assistant-API-openai-nodejs-sample: A simple example for Assistant API by OpenAI using NodeJS

Note: I won't explain and show the whole code here.

Step 5: Add the assistants API

We need to tell the AI assistants that they will act as general helpers who can help us with anything.

*Make sure to update the assistant_id key on your code.

Step 6: Assign a thread ID

Since the API needs a thread unique ID for each conversation, I'll add a new fetch request on our frontend to automatically ask for a thread id on the first visit. The ID will also updated on browser refresh.

Reminder: we have this route to create a thread ID from the OpenAI assistant tutorial

// Open a new thread
app.get('/thread', (req, res) => {
    createThread().then(thread => {
        res.json({ threadId: thread.id });
    });
})

Let's add this block to our HTML file (frontend part)

let threadId = null;

// onload
window.onload = function() {
    fetch('http://localhost:3000/thread')
        .then(response => response.json())
        .then(data => {
            console.log(data);
            assistant_id = data.threadId;
        });
}

We'll adjust our sendMessage method to attach the threadId when sending a message.

 async function sendMessage(message) {
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message, threadId }) // <- update here
    });

    // continue

Step 7: Return the last message only

Let's update our checkingStatus method to only return the latest message from the AI

async function checkingStatus(res, threadId, runId) {
    const runObject = await openai.beta.threads.runs.retrieve(
        threadId,
        runId
    );

    const status = runObject.status;
    console.log(runObject)
    console.log('Current status: ' + status);

    if(status == 'completed') {
        clearInterval(pollingInterval);

        const messagesList = await openai.beta.threads.messages.list(threadId);
        const lastMessage = messagesList.body.data[0].content[0].text.value

        res.json({ message: lastMessage });
    }
}

Our voice assistant is ready!

Step 8: Show all messages

If you want to add all previous transcriptions to the user interface, here is the code to collect all the previous messages in the div.

First, every time we speak:

recognition.onresult = async function(event) {
    // Get the latest transcript 
    const lastItem = event.results[event.results.length - 1]
    const transcript = lastItem[0].transcript;

    // Update: Append new text to div
    const newText = "<p>" + transcript + "</p>";
    document.getElementById('output').insertAdjacentHTML("afterbegin", newText);

    recognition.stop();
    await sendMessage(transcript);
}

Second, every time we got a response from the AI:

async function sendMessage(message) {
    console.log('Sending message: ', threadId);
    const response = await fetch('http://localhost:3000/message', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ message, threadId })
    });

    const data = await response.json();

    // update: add new text here
    const newText = "<p>" + data.message + "</p>";
    document.getElementById('output').insertAdjacentHTML("afterbegin", newText);

    speak(data.message);
}

Make sure to re-run your NodeJS app. Now try to have a chat with your AI!

Things we can improve

There are several things we can improve for this voice assistant

AI Instruction

Since we're building a voice assistant, it shouldn't explain things too long for a simple question. We can adjust the instructions like this:

You're a general AI assistant that can help with anything. You're a voice assistant, so don't speak too much, make it clear and concise.

Voice and listening

We use native browser API to listen and speak. Better alternatives exist, such as Elevenlabs, AssemblyAI, and more.

Knowledge Limitation

We're using one of the OpenAI models, where the knowledge is cut off at a particular year. We can expand its knowledge by providing PDF files or connecting the assistant API to the internet.