DEV Community: Terry Tan

SerpApi's Google Maps Scraper

Terry Tan — Thu, 14 Nov 2024 06:09:34 +0000

In SerpApi, we make web scraping on search engines easy and fast. Google Maps Scraper API is one of our customers' favourite to scrape local business, it provides comprehensive google maps data.

It only takes one API call to retrieve all the data available in Google Maps. Above is an example from our playground (Try it out). Upon sign up, you will receive 100 free searches every month.

What type of data can I extract from the Google Maps API?

Title - Name of the business entity
GPS Coordinates - An accurate GPS coordinates of the business entity
Rating - Rating of the business entity
Reviews - Total number of reviews of the business entity
Price - For some business (e.g. restaurant), price indicate the estimated cost of a meal
Type - The business entity type
Address - Human readable full address of the business entity
Operating Hours - The opening and closing hours of the business from Monday to Sunday
Phone - Phone number of the business entity
Website - Website URL of the business entity
Description - A short summary about the business entity
Service Options - Indicate of the business entity offers Dine In, Takeout, Delivery, and many others
Order Online - Order URL of the business entity
Menu - Menu URL of the business entity
Thumbnail - Image of the business entity
User Review - Featured review at the front page
Amenities - Amenities of the business entity
Photos - Get all the user posted photos with pagination. Refer to Google Maps Photos API
Reviews - Get all the user reviews with pagination. Refer to Google Maps Reviews API
Extensions - Highlights for the business entity (e.g. Fast service, Great coffee, Popular for Breakfast, Wheelchair accessible entrance and many more)
People Also Search For - Popular search of a similar business entity

We also provide a few more Google Maps Scraper APIs, which are Google Autocomplete API, Google Contributor Reviews API and Google Directions API.

How to scrape Google Maps?

Here is a step-by-step tutorial on using SerpApi’s Google Maps API to retrieve Google Maps data.

Step 1: Go to SerpApi's Google Maps Playground
The playground is the most intuitive tool to get started. You can feel no different from searching in Google Maps.

The link should direct you to Google Maps. Please verify that it is correctly set to Google Maps.

Step 2: Enter the search query and GPS Coordinates to scrape

Search query can be anything you will usually search in Google Maps. GPS Coordinates is optional, however, we recommend to always search with a GPS Coordinates for higher accurate results. You can find GPS Coordinates from a place using www.gps-coordinates.net or any tool you can find online.

Step 3: Start the search
The search would take averagely 2.5s (or less than 1 second if you are on Ludicrous Speed plan). The results are presented in JSON format with a preview at the left side. Feel free to test more variations of the search query to see if the data matches your needs.

At the top right corner - Export to Code, you can copy the JSON URL or follow the code to integrate with your workflow. You do have to register a free SerpApi account to use the API. The HTML can be useful when you would like to inspect the data.

Example in Python

If you would like to straight dive into the code, this should be helpful to get you started on Google Maps data extraction. We also provide many others programming languages (e.g. Javascript, Java, etc), checkout our integration guide.

First, you need to install the SerpApi client library (Github).

pip install google-search-results

Set up the SerpApi credentials and search.

from serpapi import GoogleSearch
import os, json

params = {
    'api_key': 'YOUR_API_KEY',         # your serpapi api
    'engine': 'google_maps',           # SerpApi search engine  
    'q': 'Coffee',                     # query
    'location': "New York, NY, USA"    # location
}

To retrieve the Google Maps local results for a given search query, you can use the following code:

results = GoogleSearch(params).get_dict()['local_results']

You can store Google Maps local results JSON data in databases or export them to a CSV file.

import csv

header = ['position', 'title', 'place_id', 'address', 'rating', 'reviews', 'price', 'phone']

with open('google_maps_results.csv', 'w', encoding='UTF8', newline='') as f:
    writer = csv.writer(f)

    writer.writerow(header)

    for item in results:
        print(item)
        writer.writerow([item.get('position'), item.get('title'), item.get('place_id'), item.get('address'), item.get('rating'), item.get('review'), item.get('price'), item.get('phone')])

FAQs

Is it legal to extract data from Google Maps?
Scraping public available data is generally safe provided the use cases is legal. Refer to our blog post (Scraping public pages is legal in the US) for more information.

Why use SerpApi?
There are some common challenges when scraping huge amounts of data from Google Maps: bypassing CAPTCHAs, IP blocking, rate limits, session timeouts, dynamic web structure, pagination and infinite scrolling, data variability complexity, data parsing and labelling, and legal and ethical considerations. Fortunately you don't need to tackle these problems yourself. SerpApi takes care of them and provides high-quality Google Map Search Results APIs for your usage by using high quality proxies. SerpApi saves you a lot of time so you can concentrate on your business as scraping Google Maps data and other Google search results should be easy.

Am I able to get all of the businesses in a particular city/country?
Getting all the businesses in a specific city or country is possible through our API. But, you will still need a way to list all of the specific businesses. If you already have that list, you can just submit each business name as a search on our API. If not, you can search for that specific business in each city or local subdivision of the country using our API until you build up a list that you’re happy with.

Is there a no-code solution?
Yes, you can use our API in Google Sheets using our SerpApi extension in Google Sheets. We have a comprehensive tutorial for you.

We strive to provide the best SERP scraper out there. Other than the Google Maps data scraper, you can find abundance of Google APIs, Bing APIs, and etc, feel free to checkout SerpApi for more details. Also if you have any questions, please feel free to contact me.

I wish Google worked this way

Terry Tan — Thu, 20 Jun 2024 06:26:36 +0000

LLMs are extremely effective at summarizing content. While hallucination remains a critical problem, the current stage of OpenAI’s GPT-4 is already useful for many tasks. One improvement I wish Google would implement is the ability to ask follow-up questions. This could be useful when quickly scanning through restaurant reviews. For example, we could ask a follow-up question like, “What’s good and bad about restaurant A?” and it would then extract the important pieces of information based on our question. Another example could be, “I just want to grab something quick; which restaurants should I choose?”. There are many use cases that can benefit from this improvement, and I believe it can significantly enhance everyone’s lives.

The current stage of AI implementation by Google is only able to work with factual information most of the time. For example, asking the different between EV and ICE.

When asked about the best sushi restaurant to dine in, the results are filtered based on user comments mentioning ‘best sushi’, which is fine but it appears across all the restaurants shown. Therefore, it is important to go through each of the user reviews to accurately gauge the quality for aspects you probably care about, such as service quality, cleanliness, freshness, etc.

A DIY Solution

At SerpApi, we help users scrape Google and other search engines. We have good amount of coverage on Google data, including, Google Search, Google Maps, Google Maps Reviews, Google Trends, Google Flights, Google Hotels, and many others. This give me the idea that I could build a simple DIY solution using the data we can access.

This DIY solution is made possible with OpenAI’s GPT-4 and SerpApi’s Google Maps and Google Maps Reviews APIs. We first retrieve the places and reviews using SerpApi’s API and then pass each place’s data to OpenAI’s GPT-4 for summarization.

The inputs

Building a follow-up question UI could take some times to tweak and make it work well. This UI could serve as a proof of concept, mainly aiming to have LLM perform the tasks for us. We have to provide the keyword we want to search and the location in GPS coordinates. You could search "Sushi restaurant", "Car Servicing", and etc. The preference is optional but it could improve the final result.

The results

Looking for a quick grab at a sushi restaurant in New York

Honest and on time car servicing workshop in New York

This is great because I can now read through all the good and bad reviews at a glance for each place. An obvious improvement I should make is to attach a reference link to each line.

This is the prompt given to the LLM

You’re an expert reviewer. You are great at looking at the reviews for a given place and summarizing the good and bad aspects. It is also great to show a highlights section to the user. Take user preference (if present) into consideration when deciding if you recommend or not. When a JSON is provided by the user, look for the good and bad points, then return the result in JSON format with new keys ‘good’ and ‘bad’.

Response format:
{ good: [string], bad: [string], highlights: [string], recommended: true/false, reason: string }

Github Repository: https://github.com/tanys123/thegoto

Conclusion

LLM is a great piece of technology that can help improve everyone’s life when it is well implemented. Google most probably developing a ChatGPT version of Google, hopefully it is going to be good. For now, I would rely on Perplexity.

Every Google &udm=? in the world

Terry Tan — Tue, 18 Jun 2024 05:52:23 +0000

Appends udm=14 to the Google search URL to instruct Google to only return the Web results. There is even an extension and website built specifically for this use case. This is suitable for user who would like to have a clean search results without the clutter from Knowledge Graph, Local Results, Related Questions and etc. Now, there are others udm=x than just 14. I have been doing some searching and compile all of them here.

In fact, you most probably have been using some of the common udm. For example, udm=2 is Images.

Or udm=12 which shows all the relevant news results.

An example for not so common udm is udm=15 which shows Attractions results.

And udm=18 which shows Forums results.

udm could be a useful capability in Google search, thus I have compile a intuitive list of country and their udm.

Please note that the list is very long, it is impossible to test all of them with all the countries. Only use this as an exploration guide.

The List

#2: Images
#6: Learn
#7: Videos
#12: News
#14: Web
#15: Attractions
#18: Forums
#28: Shopping
#36: Books
#37: Products
#44: Visual matches
#48: Exact matches

This set of udm should work with all the countries. The countries below are limited to the udm mentioned above.

Canada (ca), United States (us), Mexico (mx), Belize (bz), Costa Rica (cr), Cuba (cu), Dominican Republic (do), El Salvador (sv), Guatemala (gt), Haiti (ht), Honduras (hn), Jamaica (jm), Nicaragua (ni), Panama (pa), Puerto Rico (pr), Trinidad and Tobago (tt), Argentina (ar), Bolivia (bo), Brazil (br), Chile (cl), Colombia (co), Ecuador (ec), Guyana (gy), Paraguay (py), Peru (pe), Suriname (sr), Uruguay (uy), Venezuela (ve), Andorra (ad), Monaco (mc), San Marino (sm), Switzerland (ch), United Kingdom (uk, gb), Vatican City (va), Albania (al), Belarus (by), Bosnia and Herzegovina (ba), Moldova (md), North Macedonia (mk), Russia (ru), Serbia and Montenegro (rs), Ukraine (ua), Bahrain (bh), Egypt (eg), Iran (ir), Iraq (iq), Israel (il), Jordan (jo), Kuwait (kw), Lebanon (lb), Oman (om), Palestinian Territory (ps), Qatar (qa), Saudi Arabia (sa), Syria (sy), United Arab Emirates (ae), Yemen (ye), Algeria (dz), Angola (ao), Benin (bj), Botswana (bw), Burkina Faso (bf), Burundi (bi), Cameroon (cm), Cape Verde (cv), Central African Republic (cf), Chad (td), Comoros (km), Congo (cg), Democratic Republic of the Congo (cd), Djibouti (dj), Egypt (eg), Equatorial Guinea (gq), Eritrea (er), Ethiopia (et), Gabon (ga), Gambia (gm), Ghana (gh), Guinea (gn), Guinea-Bissau (gw), Ivory Coast (ci), Kenya (ke), Lesotho (ls), Liberia (lr), Libya (ly), Madagascar (mg), Malawi (mw), Mali (ml), Mauritania (mr), Mauritius (mu), Morocco (ma), Mozambique (mz), Namibia (na), Niger (ne), Nigeria (ng), Rwanda (rw), Senegal (sn), Seychelles (sc), Sierra Leone (sl), Somalia (so), South Africa (za), Sudan (sd), Swaziland (sz), Tanzania (tz), Togo (tg), Tunisia (tn), Uganda (ug), Western Sahara (eh), Zambia (zm), Zimbabwe (zw), Kazakhstan (kz), Kyrgyzstan (kg), Tajikistan (tj), Turkmenistan (tm), Uzbekistan (uz), Afghanistan (af), Bangladesh (bd), Bhutan (bt), India (in), Maldives (mv), Nepal (np), Pakistan (pk), Sri Lanka (lk), China (cn), Japan (jp), Mongolia (mn), North Korea (kp), South Korea (kr), Taiwan (tw), Brunei (bn), Cambodia (kh), Indonesia (id), Laos (la), Malaysia (my), Myanmar (mm), Philippines (ph), Singapore (sg), Thailand (th), Timor-Leste (tl), Vietnam (vn), Australia (au), Fiji (fj), Kiribati (ki), Marshall Islands (mh), Micronesia (fm), Nauru (nr), New Zealand (nz), Palau (pw), Papua New Guinea (pg), Samoa (ws), Solomon Islands (sb), Tonga (to), Tuvalu (tv), Vanuatu (vu), American Samoa (as), Christmas Island (cx), Cocos (Keeling) Islands (cc), Cook Islands (ck), Faroe Islands (fo), Gibraltar (gi), Greenland (gl), Guam (gu), Hong Kong (hk), Macao (mo), Netherlands Antilles (an), Niue (nu), Norfolk Island (nf), Northern Mariana Islands (mp), Pitcairn (pn), Saint Helena (sh), Svalbard and Jan Mayen (sj), Tokelau (tk), Turks and Caicos Islands (tc), United States Minor Outlying Islands (um), Virgin Islands, U.S. (vi), Antarctica (aq), Bouvet Island (bv), Heard Island and McDonald Islands (hm)

#9: Product sites

The udm mentioned prior plus the udm above are available for the countries below.

Anguilla (ai), Bermuda (bm), British Indian Ocean Territory (io), Cayman Islands (ky), Falkland Islands (fk), Montserrat (ms), South Georgia and the South Sandwich Islands (gs), Turks and Caicos Islands (tc), Virgin Islands, British (vg)

#1: Places
#3: Products
#5: Lodging
#8: Jobs
#10: Job sites
#11: Places sites
#13: Airline options
#31: Flight sites
#32: Trains
#33: Buses
#34: Transport sites

All the mentioned udm plus the udm above are available for the countries below, they are usually country from Europe.

Aruba (aw), Austria (at), Belgium (be), Bulgaria (bg), Croatia (hr), Cyprus (cy), Czech Republic (cz), Denmark (dk), Estonia (ee), Finland (fi), France (fr), French Guiana (gf), French Polynesia (pf), French Southern Territories (tf), Germany (de), Greece (gr), Guadeloupe (gp), Hungary (hu), Iceland (is), Ireland (ie), Italy (it), Latvia (lv), Liechtenstein (li), Lithuania (lt), Luxembourg (lu), Malta (mt), Martinique (mq), Mayotte (yt), Netherlands (nl), New Caledonia (nc), Norway (no), Poland (pl), Portugal (pt), Reunion (re), Romania (ro), Saint Pierre and Miquelon (pm), Slovakia (sk), Slovenia (si), Spain (es), Sweden (se), Turkey (tr), Wallis and Futuna (wf)

Conclusion

There you have it. If you find this list useful, please share it to your friends, family and colleagues.

If you find any mistakes, feel free to contact me (terry@serpapi.com).

Join us on X | YouTube

Add a Feature Request💫 or a Bug🐞

My Experience building with OpenAI's Assistants API

Terry Tan — Fri, 24 Nov 2023 06:40:36 +0000

TLDR: Like most API products in the market, Assistants API allow us to build the core of our product

A lot of amazing stuffs are announced on OpenAI DevDay like GPT-4 Turbo with 128k context, GPT with Vision and lower pricing. The one I am fond with is the new Assistants API.

I have already experimenting with chatbot apps for awhile now before the launch of Assistants API. If you are going to build a chatbot with your own knowledge base, you are most likely to stumble upon the concept of RAG (Retrieval-augmented generation). It is fairly easy to get started and build something that actually work. Langchain has a great tutorial on that.

However, there are actually a lot of steps and decisions to be taken care of behind the scene.

Trunacating the document
- Because of the limited context length, we have to split a document into multiple chunks.
Calculate document embeddings and store it in a Vector DB
Decide the number of documents to retrieve
- Retrieving too much document will lead to unrelavant result
- Retrieving too little document might miss out important information
Avoiding duplicated documents
Managing chat history
- We have to truncate the chat in case it grow too large
- We have to decide how to truncate so that the context is retain
Document citation

All of this is taken care by Assistants API which reduced the steps to just uploading the document. The downside are the number of documents is limited to 20 as of now and the overall cost might be higher compare to doing all yourself.

If you are still interested, I will share the experience building with Assistants API for Wander which focus on Function Call. I was trying to replicate the Wanderlust demo shown in the DevDay

OpenAI DevDay, Opening Keynote - YouTube

Join us for the opening keynote from OpenAI DevDay — OpenAI’s first developer conference.We’re gathering developers from around the world for an in-person da...

youtube.com

Wander Demo:

Wander Demo - YouTube

Demo for github.com/serpapi/wander

youtube.com

Check out the Wander Github Repo if you are interested. I enhanced it with real-time data using SerpApi Google Maps API.

First time seeing the demo, I thought probably that's how the future of UIUX is gonna be like. Even though the demo of showing markers on map is relative small use case but I believe people will find way to implement them effectively on their applications. Here is an great article that explain the concept I want to talk about.

I believe you can build all this with Chat Completions API alone, but Assistants API just make everything easier. For example, the flow of chat is managed. It tells you when it is time to handle Function call and the output from the Function call should submit to the API to get the next step.

Currently the execution especially those include Function call is taking too long (50s+) and streaming is not supported yet. These are the two most important improvements that can really make it usuable in production.

I haven't use Assistants API for scale and production level, so the information I can share is limited. If you have deployed it to production, do let me know!

Real-time data for OpenAI Assistants API

Terry Tan — Thu, 16 Nov 2023 03:48:30 +0000

The Assistants API by OpenAI makes creating chatbot applications easier than ever. Before this, we had to manage chat history for the users and implement Retrieval Augmented Generation (RAG) on our own, but all of that is already taken care of now by Assistants API.

The impressive demo at OpenAI DevDay made me think about the changes to the future UIUX. The demo is accomplished by using Assistants API.

After a few failed attempts to find the repository for Wanderlust, I tried to recreate my own Wander to satisfy my curiosity. Annotating the map is cool and I thought it would be even more useful if we could get real-time Google Maps data to show local businesses like cafes, restaurants, convenience stores etc.

Check out the Wander Github Repo if you are interested. The entire solution is built on OpenAI Assistants API and SerpApi Google Maps API. You have to sign up on the respective website for the API key. SerpApi provides free searches upon signing up.

Conclusions

OpenAI Assistants API is a powerful feature for building a very well-rounded chat application. Providing it with real-time data makes it even more capable.

Scrape Google Trends Trending Now in Javascript

Terry Tan — Mon, 30 Oct 2023 08:24:45 +0000

We have already covered scraping Google Trends Trending Now in Python and if you are looking to extract data from the Google Trends Explore, we already have a blog post that demo it in detail. In this tutorial, we will be using SerpApi's Google Trends Trending Now API to scrape Google Trends Trending Now in Javascript.

Google Trends' Trending Now

Google Trends Explore allow you to get data for specific query or compare trends between queries. As for Google Trends’ Trending Now, it shows the top queries for the day from pool of Google Trends data.

There are two parts of Trending Now: the first one is Daily Search Trends, and the second is Real-time Search Trends.

Daily Search Trends focus on a single query and its performance throughout a specific day. Unique to Daily Search Trends, it offers data for the past few weeks or even months. It is suitable if you would like to query historical data of trending searches.

Realtime Search Trends provide a dynamic and up-to-the-minute view of what is currently trending. It usually combines with multiple queries. This section displays topics and search queries as they unfold in real time. Unlike Daily Search Trends, Real-time Search Trends are not confined to a 24-hour window. They continuously update as new trends emerge and fade, making them ideal for tracking breaking news, sports events, and social media trends as they happen.

In summary, the main difference between Daily Search Trends and Real-time Search Trends lies in their time frames and purposes. Daily Search Trends provide a retrospective view of the previous 24 hours or even weeks, making them suitable for analyzing what was trending on a specific day. On the other hand, Real-time Search Trends offer a dynamic, live view of what is currently trending, making them ideal for staying up to date with rapidly evolving events and topics.

Google Trends’ Trending Now can be valuable tool for various use cases including content creation, news, journalism, product research, marketing and SEO, trend analysis, location-based insights and many others.

Why use an API?

No need to create a parser from scratch and maintain it.
Bypass blocks from Google: solve CAPTCHA or solve IP blocks.
No need to pay for proxies, and CAPTCHA solvers.
Don't need to use browser automation.

SerpApi take care of everything mentioned above with fast response times under ~2.47 seconds (~1.33 seconds with Ludicrous speed) per request. The result is a well structured data in JSON with only single API call.

Response times and success rates are shown under SerpApi Status page.

Full code

If you don't need an explanation, feel free to copy and paste the code into your favorite IDE and start running. Otherwise, you can follow the step-by-step-guide code explanation to know what each of the line is doing. You can also check out our playground, which is very handy to play around with the API and get a sense of how things work.

import { getJson } from "serpapi"

const SERPAPI_KEY = "..." // Get your API_KEY from https://serpapi.com/manage-api-key

const params = {
  api_key: SERPAPI_KEY,
  engine: "google_trends_trending_now",
  frequency: "daily", // daily or realtime
  geo: "US",
  hl: "en"
}

const response = await getJson(params)

Prerequisite

Install library:

npm install serpapi

serpapi is an official SerpApi API package. Follow this guide to get your Node.js and npm install if you haven’t done so.

Code Explanation

First, we need to import the library

import { getJson } from "serpapi"

Next, we set the required parameters. A list of parameters and their detailed explanation can be found on our documentation.

const params = {
  api_key: SERPAPI_KEY, // https://serpapi.com/manage-api-key
  engine: "google_trends_trending_now", // Make sure the right engine is set
  frequency: "daily", // daily or realtime
  geo: "US", // Location to search for
  hl: "en" // Language of the results
}

Once parameters is defined, everything else will be taken care by the library, for example, making the actual API request. Calling getJson to make the API request and JSON data will be returned and assigned it to response.

const response = await getJson(params)

Output:

console.log(response)

"daily_searches": [
  {
    "date": "20231025",
    "searches": [
      {
        "query": "Maine shooting",
        "google_trends_link": "https://trends.google.com/trends/explore?q=Maine+shooting&date=now+7-d&geo=US",
        "serpapi_google_trends_link": "https://serpapi.com/search.json?data_type=TIMESERIES&date=now+7-d&engine=google_trends&geo=US&q=Maine+shooting&tz=420",
        "traffic": 5000000,
        "related_queries": [
          {
            "query": "lewiston maine",
            "google_trends_link": "https://trends.google.com/trends/explore?q=lewiston+maine&date=now+7-d&geo=US",
            "serpapi_google_trends_link": "https://serpapi.com/search.json?data_type=TIMESERIES&date=now+7-d&engine=google_trends&geo=US&q=lewiston+maine&tz=420"
          },
          ...
        ],
        "articles": [
          {
            "title": "Manhunt for Maine shooting suspect enters second day",
            "link": "https://www.bbc.com/news/world-us-canada-67232966",
            "snippet": "Hundreds of law enforcement personnel are part of the search as towns remain under shelter in place orders.",
            "source": "BBC News",
            "date": "1h ago",
            "thumbnail": "https://t1.gstatic.com/images?q=tbn:ANd9GcR-vgroBO4SB7JAB8HE4J8fOBmF01lY6K-HUBMB0Vbc2lfIGqc_p3qDuE6wuZPTXjIEEBycQD0O"
          },
          ...
        ]
      },
      {
        "query": "Celtics",
        "google_trends_link": "https://trends.google.com/trends/explore?q=Celtics&date=now+7-d&geo=US",
        "serpapi_google_trends_link": "https://serpapi.com/search.json?data_type=TIMESERIES&date=now+7-d&engine=google_trends&geo=US&q=Celtics&tz=420",
        "traffic": 200000,
        "related_queries": [
          {
            "query": "Knicks",
            "google_trends_link": "https://trends.google.com/trends/explore?q=Knicks&date=now+7-d&geo=US",
            "serpapi_google_trends_link": "https://serpapi.com/search.json?data_type=TIMESERIES&date=now+7-d&engine=google_trends&geo=US&q=Knicks&tz=420"
          },
          ...
        ],
        "articles": [
          {
            "title": "Kristaps Porzingis scores 30, seals win at MSG in Celtics debut ...",
            "link": "https://www.espn.com/nba/story/_/id/38746584/kristaps-porzingis-scores-30-seals-win-vs-knicks-celtics-debut",
            "snippet": "Kristaps Porzingis set a franchise record with 30 points in his Celtics debut and took over down the stretch to lead his new team past his old one Wednesday ...",
            "source": "ESPN",
            "date": "22h ago",
            "thumbnail": "https://t0.gstatic.com/images?q=tbn:ANd9GcTdwRxz8dK0cXCRMX35ovYaRffEbtVoTvpFbfmsYuCC7K76Y5DSnOKKJKi4xE24vqwFFXbs7A6h"
          },
          ...
        ]
      },
      {
        "query": "Spurs",
        "google_trends_link": "https://trends.google.com/trends/explore?q=Spurs&date=now+7-d&geo=US",
        "serpapi_google_trends_link": "https://serpapi.com/search.json?data_type=TIMESERIES&date=now+7-d&engine=google_trends&geo=US&q=Spurs&tz=420",
        "traffic": 200000,
        "related_queries": [
          {
            "query": "Wembanyama",
            "google_trends_link": "https://trends.google.com/trends/explore?q=Wembanyama&date=now+7-d&geo=US",
            "serpapi_google_trends_link": "https://serpapi.com/search.json?data_type=TIMESERIES&date=now+7-d&engine=google_trends&geo=US&q=Wembanyama&tz=420"
          },
          ...
        ],
        "articles": [
          {
            "title": "Victor Wembanyama finishes with 15 points in debut as Spurs lose ...",
            "link": "https://www.espn.com/nba/story/_/id/38746746/victor-wembanyama-15-points-debut-spurs-lose",
            "snippet": "SAN ANTONIO -- As Victor Wembanyama walked on to the Frost Bank Center court during pregame warmups for his NBA regular-season debut, he spread his arms ...",
            "source": "ESPN",
            "date": "15h ago",
            "thumbnail": "https://t0.gstatic.com/images?q=tbn:ANd9GcSlm55dFiq8eyKV7otfGAXOGknwusoMZamiiIPvtiyiGcLfQ--O9_6DErU1V8oAwS1jguLgXnlD"
          },
          ...
        ]
      },
      ...
    ]
  }
],
"serpapi_pagination": {
  "current_date": "20231025",
  "next_date": "20231024",
  "next": "https://serpapi.com/search.json?date=20231024&engine=google_trends_trending_now&frequency=daily&geo=US&hl=en"
}

Next Date Result

Similar code but to get the next date result, we have to get and set the date with next_date return by the first API call. You can get the next date result as long as the API is returning next_date, which mean you reach the end of result when there is no next_date return from the API.

const nextDate = response["serpapi_pagination"]["next_date"]
const nextDateResponse = await getJson({
  ...params,
  date: nextDate
})

Other Parameters

Here is a list of combination of parameters that you can use to get different results. api_key is omitted for simplicity.

Daily Search Trends in France

const params = {
  engine: "google_trends_trending_now",
  frequency: "daily",
  geo: "FR",
  hl: "en"
}

Daily Search Trends in France at 1st of October 2023

const params = {
  engine: "google_trends_trending_now",
  frequency: "daily",
  date: "20231001",
  geo: "FR",
  hl: "en"
}

Realtime Search Trends in United States

const params = {
  engine: "google_trends_trending_now",
  frequency: "realtime",
  geo: "US",
  hl: "en"
}

Realtime Search Trends in the United States for the Sports category

const params = {
  engine: "google_trends_trending_now",
  frequency: "realtime",
  geo: "US",
  hl: "en",
  cat: "s"
}

Next page results of the Realtime Search Trends in the United States for the Sports category

const params = {
  engine: "google_trends_trending_now",
  frequency: "realtime",
  geo: "US",
  hl: "en",
  cat: "s",
  next_page_token: "WyJVU19sbmtfNnNEUm1BQUFBQUE3Mk1fZW4iLCJVU19sbmtfSHFrZ2xnQUFBQUEtdk1fZW4iLCJVU19sbmtfbnhqSWx3QUFBQUJYRE1fZW4iLCJVU19sbmtfOC1iWW1BQUFBQUFyX01fZW4iLCJVU19sbmtfMUl2TGx3QUFBQUFmbk1fZW4iLCJVU19sbmtfWnozLW1BQUFBQUNaSk1fZW4iLCJVU19sbmtfb1E0Qm1RQUFBQUNnRk1fZW4iLCJVU19sbmtfeUVmc21BQUFBQUFrWE1fZW4iLCJVU19sbmtfWF9uZ2x3QUFBQUNfN01fZW4iLCJVU19sbmtfR3BWcG1BQUFBQUJ6ak1fZW4iLCJVU19sbmtfLXpmdW1BQUFBQUFWTE1fZW4iLCJVU19sbmtfZkVqYmx3QUFBQUNuWE1fZW4iLCJVU19sbmtfMzVtRmx3QUFBQUJhak1fZW4iLCJVU19sbmtfWWdSS21BQUFBQUFvSE1fZW4iLCJVU19sbmtfTGlMMW1BQUFBQURiT01fZW4iLCJVU19sbmtfODFBSm1BQUFBQUQ2U01fZW4iLCJVU19sbmtfZDY1NW1BQUFBQUFPdE1fZW4iLCJVU19sbmtfNHhMcGx3QUFBQUFLQk1fZW4iLCJVU19sbmtfV3VVdGx3QUFBQUIzOE1fZW4iLCJVU19sbmtfTUJIem1BQUFBQUREQ01fZW4iXQ=="
}

Documentation

Conclusions

Google Trends' Trending Now is a great data source for understanding the events happening in real time or in the past. SerpApi provides a simple yet powerful solution to scrape those data without any headache on blocking and maintenance as compare to your typical web scraping experience. Alternatively, you can checkout pytrends github repository which is a popular Python open source google trends scraper.

How many percent of your work is not building the core business logic?

Terry Tan — Thu, 10 Jun 2021 06:09:07 +0000

We have built a chatbot + commerce solution on Messenger for businesses. Businesses can use our solution for Facebook Live Selling and Facebook Ads, no website is needed. Customer make comment and the bot will lead them to Messenger to complete the purchase, it works seamlessly.

We have actually deployed to few clients. The thing is each client will have different CRM system, Payment Gateway, Logistic system. Before we launch, our team need to spend few weeks building the integration. It is necessary but actually not the core of our software.

I am not complaining about it. But I think there must be a lot of software out there always need to write this integration layer, which it is identical to the other system. So wouldn't it be great if there is an integration layer we can plug into our software and it will work just with few lines of code?

For example, the integration layer already integrated with Payment Gateway A, B and C. My software just need to integrate with the integration layer and all 3 payment gateways will be available on my system.

This is very high level view, let me know if you have any questions. More important, feel free to share your opinion 😊.