DEV Community

Cover image for Web scraping Google Jobs Listing with Nodejs
SerpApi profile image Mikhail Zub
Mikhail Zub for SerpApi

Posted on

Web scraping Google Jobs Listing with Nodejs

#webscraping #node #serpapi

Intro

In this blog post I'll show you how to use Google Jobs Listing Results API from SerpApi.

The main advantages of our API is that you don't need to use browser automation to scrape results, create the parser from scratch and maintain it.

There's also a chance that the request might be blocked at some point from Google, we handle it on our backend so there's no need to figure out how to do it yourself or figure out which CAPTCHA, proxy provider to use.

What will be scraped

what

Full code

If you don't need an explanation, have a look at the full code example in the online IDE

const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(process.env.API_KEY);

const jobId =
  "eyJqb2JfdGl0bGUiOiJCYXJpc3RhIiwiY29tcGFueV9uYW1lIjoiQmx1ZSBCb3R0bGUgQ29mZmVlIiwiY29tcGFueV9taWQiOiIvbS8wM3AxMnFkIiwiYWRkcmVzc19jaXR5IjoiU2FuIEZyYW5jaXNjbyIsImFkZHJlc3Nfc3RhdGUiOiJDYWxpZm9ybmlhIiwiaHRpZG9jaWQiOiJBNnFRZGw1VjZvU1ZxbnI0QUFBQUFBPT0iLCJ1dWxlIjoidytDQUlRSUNJa1UyRnVJRVp5WVc1amFYTmpieUJDWVhrZ1FYSmxZU3hWYm1sMFpXUWdVM1JoZEdWeiIsImZjIjoiRW80QkNtY3dZVFZSY0RkWE9VTkJSek42WVdocVVHRkZXV2hqTkdGamJGSm5lRUpmWVhOc2NVRmlhRXhKWHpaek1Hb3hWamx2UWxKSlpGVnBVRUpyV1ZkeU1sQXpXRmhVVDB3MVRHbFNjamh5T0VzelRVcGhhM2x6WjFsQ1NuZFBWMjVJTlVKQlJWTjJOREZCRWhaME5rUXdXSE5pYjB4TmRWVXRaMVEwTkc5SFNVTkJHZ3RQTFVKSGRqaFNVWEp4TkEiLCJmY2VudiI6IjEiLCJmY3YiOiIyIiwiZmNfaWQiOiJmY183In0";

const params = {
  engine: "google_jobs_listing", // search engine
  q: jobId, // job id from https://serpapi.com/playground?engine=google_jobs&q=Barista
};

const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};

const getResults = async () => {
  const json = await getJson();
  const { apply_options = "No apply options for this job", salaries = "No salaries for this job", ratings = "No ratings for this job" } = json;
  return { applyOptions: apply_options, salaries, ratings };
};

getResults().then(console.log);
Enter fullscreen mode Exit fullscreen mode

Preparation

First, we need to create a Node.js* project and add npm package google-search-results-nodejs to scrape and parse what you need using SerpApi.

To do this, in the directory with our project, open the command line and enter:

$ npm init -y
Enter fullscreen mode Exit fullscreen mode

And then:

$ npm i google-search-results-nodejs
Enter fullscreen mode Exit fullscreen mode

*If you don't have Node.js installed, you can download it from nodejs.org and follow the installation documentation.

Code explanation

First, we need to declare SerpApi from google-search-results-nodejs library and define new search instance with your API key from SerpApi:

const SerpApi = require("google-search-results-nodejs");
const search = new SerpApi.GoogleSearch(API_KEY);
Enter fullscreen mode Exit fullscreen mode

Next, we write the job ID and the necessary parameters for making a request. You can get the job Id parameter using Web scraping Google Jobs organic results with Nodejs blog post:

const jobId =
  "eyJqb2JfdGl0bGUiOiJCYXJpc3RhIiwiY29tcGFueV9uYW1lIjoiQmx1ZSBCb3R0bGUgQ29mZmVlIiwiY29tcGFueV9taWQiOiIvbS8wM3AxMnFkIiwiYWRkcmVzc19jaXR5IjoiU2FuIEZyYW5jaXNjbyIsImFkZHJlc3Nfc3RhdGUiOiJDYWxpZm9ybmlhIiwiaHRpZG9jaWQiOiJBNnFRZGw1VjZvU1ZxbnI0QUFBQUFBPT0iLCJ1dWxlIjoidytDQUlRSUNJa1UyRnVJRVp5WVc1amFYTmpieUJDWVhrZ1FYSmxZU3hWYm1sMFpXUWdVM1JoZEdWeiIsImZjIjoiRW80QkNtY3dZVFZSY0RkWE9VTkJSek42WVdocVVHRkZXV2hqTkdGamJGSm5lRUpmWVhOc2NVRmlhRXhKWHpaek1Hb3hWamx2UWxKSlpGVnBVRUpyV1ZkeU1sQXpXRmhVVDB3MVRHbFNjamh5T0VzelRVcGhhM2x6WjFsQ1NuZFBWMjVJTlVKQlJWTjJOREZCRWhaME5rUXdXSE5pYjB4TmRWVXRaMVEwTkc5SFNVTkJHZ3RQTFVKSGRqaFNVWEp4TkEiLCJmY2VudiI6IjEiLCJmY3YiOiIyIiwiZmNfaWQiOiJmY183In0";

const params = {
  engine: "google_jobs_listing", // search engine
  q: jobId, // job id from https://serpapi.com/playground?engine=google_jobs&q=Barista
};
Enter fullscreen mode Exit fullscreen mode

Next, we wrap the search method from the SerpApi library in a promise to further work with the search results:

const getJson = () => {
  return new Promise((resolve) => {
    search.json(params, resolve);
  });
};
Enter fullscreen mode Exit fullscreen mode

And finally, we declare the function getResult that gets data from the page and return it:

const getResults = async () => {
  ...
};
Enter fullscreen mode Exit fullscreen mode

In this function, we get json with results from the page (getJson function). Then we destructure received json, set default values to each destructured property and return object with results:

const json = await getJson();
const { 
    apply_options = "No apply options for this job",
    salaries = "No salaries for this job",
    ratings = "No ratings for this job"
    } = json;
return { applyOptions: apply_options, salaries, ratings };
Enter fullscreen mode Exit fullscreen mode

After, we run the getResults function and print all the received information in the console:

getResults().then(console.log);
Enter fullscreen mode Exit fullscreen mode

Output

{
  "applyOptions": "No apply options for this job",
  "salaries": [
    {
      "job_title": "Barista",
      "link": "https://www.ziprecruiter.com/Salaries/Barista-Salary-in-San-Francisco,CA?utm_campaign=google_jobs_salary&utm_source=google_jobs_salary&utm_medium=organic",
      "source": "ZipRecruiter",
      "salary_from": 21000,
      "salary_to": 40000,
      "salary_currency": "$",
      "salary_periodicity": "year",
      "thumbnail": "https://serpapi.com/searches/6363e399a3f4ef79bb0ff373/images/f1d87c26c4968270a8e938f7cf10b5cdc1a6c6457fd1184e.png",
      "based_on": "Based on local employers"
    },
    {
      "job_title": "Coffee Barista",
      "link": "https://www.salary.com/research/salary/alternate/coffee-barista-salary/san-francisco-ca?utm_campaign=google_jobs_salary&utm_source=google_jobs_salary&utm_medium=organic",
      "source": "Salary.com",
      "salary_from": 27000,
      "salary_to": 36000,
      "salary_currency": "$",
      "salary_periodicity": "year",
      "thumbnail": "https://serpapi.com/searches/6363e399a3f4ef79bb0ff373/images/f1d87c26c496827075cfe00b6aeb887d3533b03d6f8540fc.png",
      "based_on": "Based on local employers"
    },
    {
      "job_title": "Barista",
      "link": "https://www.payscale.com/research/US/Job=Barista/Hourly_Rate/ae670208/San-Francisco-CA?utm_campaign=google_jobs_salary&utm_source=google_jobs_salary&utm_medium=organic",
      "source": "Payscale",
      "salary_from": 14,
      "salary_to": 20,
      "salary_currency": "$",
      "salary_periodicity": "hour",
      "thumbnail": "https://serpapi.com/searches/6363e399a3f4ef79bb0ff373/images/f1d87c26c4968270a20489154adfa8fce64f52ba237d9ed1.png",
      "based_on": "Based on local employers"
    }
  ],
  "ratings": [
    {
      "link": "https://www.glassdoor.com/Reviews/Blue-Bottle-Coffee-Reviews-E803867.htm?utm_campaign=google_jobs_reviews&utm_source=google_jobs_reviews&utm_medium=organic",
      "source": "Glassdoor",
      "rating": 3.6,
      "reviews": 303
    },
    {
      "link": "https://www.indeed.com/cmp/Blue-Bottle-Coffee-Company/reviews?utm_campaign=google_jobs_reviews&utm_source=google_jobs_reviews&utm_medium=organic",
      "source": "Indeed",
      "rating": 3.3,
      "reviews": 70
    },
    {
      "link": "https://www.comparably.com/companies/blue-bottle-coffee?utm_campaign=google_jobs_reviews&utm_source=google_jobs_reviews&utm_medium=organic",
      "source": "Comparably",
      "rating": 2.5,
      "reviews": 40
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

If you want to see some projects made with SerpApi, write me a message.

Join us on Twitter | YouTube

Add a Feature Request💫 or a Bug🐞

Top comments (0)

Read next

imsushant12 profile image

Monitoring and Logging in Node.js Applications: Best Practices and Tools

Sushant Gaurav -

kamlesh_gupta_539c974fb0d profile image

Overview of Nodemailer: Easy Email Sending in Node.js

Kamlesh Gupta -

runtime_terrors profile image

Exploring Deno 2: Features, Benefits, and Getting Started

Bhavsagar -

vjygour profile image

Understanding the Difference Between package.json and package-lock.json

Vijay Kumar -