DEV Community: Caleb David

Building a Google SERP extension in Google Sheets (Apps Script + RapidAPI) 📈

Caleb David — Sat, 13 Aug 2022 11:13:04 +0000

Introduction

So you want to monitor some keywords for your next SEO research and analysis and want the data readily available in a Google Sheets file all, without leaving the four walls (corners?) of Google Sheets? 🤔 Say no more -- This article will help you achieve just that. So, in a nutshell, we'll be building a webapp inside Google Sheets that interacts with the same spreadsheet file. We'll use Google Apps Script and RapidAPI platforms to achieve this.

Google Apps Script is a scripting platform developed by Google for light-weight application development, in the Google Workspace platform (wikipedia) and RapidAPI is a marketplace for APIs.

Goal

Create a custom menu in Google Sheets.
When that menu is clicked, launch a modal window that accepts Google search parameters and other search options
Send request to Google Search API hosted on RapidAPI to retrieve results.
Write the results to a sheet on the same spreadsheet.
Jump 3 times and sip a glass coffee then sleep? 🤷‍♂️

Folder sctructure

We'll require two files:

index.gs
index.html

Create the custom menu

Now let's get our hands dirty with some code. Create a new spreadsheet file in your Google account and locate Extensions from the menu bar. Then select Apps Script from the submenu. That will launch a text editor where we will be working from.

Next is to create our custom menu titled SEO Tools with a submenu called Google SERP. Below is the code for that.

// index.gs
function onOpen(){
  SpreadsheetApp.getUi()
    .createMenu('SEO Tools')
    .addItem('Google SERP', 'openDialog')
    .addToUi()
}

function openDialog() {
  var html = HtmlService.createHtmlOutputFromFile('index');
  SpreadsheetApp.getUi()
    .showModalDialog(html, 'Google SERP');
}

The onOpen function fires immediately after the spreadsheet file loads hence, injecting our custom menu. When the submenu item is clicked, it triggers the openDialog function which in turn, launches our modal window. The modal window contains our html file.

Send data from HTML to Apps Script

We need a way to send data from our html file to Apps Script. In the index.html file, just before the body closing tag, paste the following script

<!-- index.html -->
...
<script>
  document.querySelector(`#submit-btn`).addEventListener(`click`, () => {
    let query = document.querySelector(`#query`)?.value || `q=tesla+stocks&num=100`;
    let rapid_api_key =
        document.querySelector(`#rapid_api_key`)?.value || null;
    let proxy_location =
        document.querySelector(`#proxy_location`)?.value || `United States`;
    let device = document.querySelector(`#device`)?.value || `desktop`;
    let domain = document.querySelector(`#domain`)?.value || `google.com`;

    let params = {
      query,
      rapid_api_key,
      proxy_location,
      device,
      domain
    }

    google.script.run.handleSubmission(params)
  })
</script>
...

Form parameters

The form accepts the following parameters:

query
rapid_api_key
proxy_location
device
domain

query is any valid Google search parameters. Example:

q=seo+use+cases&num=100&ie=UTF-8
q=serp&num=20&ie=UTF-8&start=100
q=google+sheets&num=100&ie=UTF-8&start=100
q=javascript&num=50&ie=UTF-8

You can find a list of some of the possible parameters here: The Ultimate Guide to the Google Search Parameters. Big shouts-out to the folks at Moz 🙌

Searches are geographically bounded, hence, you can search from up to 127 countries/regions by selecting the desired proxy location on the form. You also need to subscribe to Google Search API to obtain your free API key for this. Since results may vary between desktop and mobile devices, you can also choose the type of device to be used for the search. Lastly, all Google domains are supported and you can simply select the domain from the list of all the available options.

After the form has been filled and the button clicked, it will run the function handleSubmission in Apps Script via google.script.run. That will get all the search parameters we provided in the form and our script will have access to them at this point, to process it further.

// index.gs
async function handleSubmission(params){
  try {
    let data = await search(params)
    let spreadsheet = SpreadsheetApp.getActiveSpreadsheet()

    // organic result sheet
    writeOrganicResults(data, spreadsheet)
  } catch (error) {
    // Handle error as desired
    Logger.log(error)
  }
}

The handleSubmission function does two things:

Make an API call to Google Search API on RapidAPI
Write the returned data to the designated sheet. You can find a sample of the Google SERP data here.

Make the API request

The search function accepts the parameters that we have already obtained and then it makes the request. A little 'gotcha!' with Apps Script is that it does not provide the URL object 😯 and we need to pass those parameters to our endpoint's URL https://google-search65.p.rapidapi.com/search. So the next hurdle to cross is cracking the cocunut fruit with our bare hands! 😆 Ready for that? Actually, what we need to do is to somehow append the parameters (of type object) to the url string as query params. There are many ways to achieve this but I find the following solution very elegant, and efficient.

// index.gs
// Object to querystring - https://gist.github.com/tanaikech/70503e0ea6998083fcb05c6d2a857107
String.prototype.addQuery = function(obj) {
  return this + Object.keys(obj).reduce(function(p, e, i) {
    return p + (i == 0 ? "?" : "&") +
      (Array.isArray(obj[e]) ? obj[e].reduce(function(str, f, j) {
        return str + e + "=" + encodeURIComponent(f) + (j != obj[e].length - 1 ? "&" : "")
      },"") : e + "=" + encodeURIComponent(obj[e]));
  },"");
}

What we did above is that, we added a custom method to the Strings object, and can now call that method on every instance of that object. addQuery will now accept our params object (remember the data we got from the form?), and inject the needed query strings to our endpoint url. The code block below shows how to achieve that.

// index.gs
function search(params) {
  return new Promise((resolve, reject) => {
    try {
      // ensure that API key is provided
      if (!params.rapid_api_key) {
        reject(`Please subscribe to https://rapidapi.com/microworlds/api/google-search65 to get a free 'X-RapidAPI-Key' key`)
      }

      const X_RapidAPI_Key = params.rapid_api_key
      delete params.rapid_api_key

      let url = `https://google-search65.p.rapidapi.com/search`
      url = url.addQuery(params)

      let response = UrlFetchApp.fetch(url, {
        method: 'GET',
        headers: {
          'X-RapidAPI-Key': X_RapidAPI_Key,  // guard this API key
          'X-RapidAPI-Host': `google-search65.p.rapidapi.com`
        }
      })

      let data = response.getContentText()
      resolve(JSON.parse(data))
    } catch (error) {
      reject(error)
    }
  })
}

You must have noticed that we are deleting the rapid_api_key param from that object. That is because we do not want to include that API key in the URL, but as a header, hence, storing it in the X_RapidAPI_Key constant.

Write the data to a sheet

Now that we have the SERP data returned as JSON from our API call, we can write that into our spreadsheet. For the sake of this article, we are only interested in the organic search results, but for what it's worth, you may write all the returned data if you so wish.

// index.gs
function writeOrganicResults(data, spreadsheet){
  Logger.log(`Writing data to sheet...📝`)
  let organic_results = data?.data?.organic_results

  if (organic_results.length < 1){
    return
  }

  let organicResultsSheet = spreadsheet.getSheetByName(`organic_results`)

  if (!organicResultsSheet) {
    spreadsheet.insertSheet(`organic_results`)
  }

  // Append search info at top of the file
  writeSearchInfo(data, organicResultsSheet)

  // Append headers row
  organicResultsSheet.appendRow(Object.keys(organic_results[0]))

  // append the rest of the data
  organic_results.forEach((item) => {
    const keys = Object.keys(item)

    let rowData = keys.map((key) => {
      return item[key].toString()
    })

    organicResultsSheet.appendRow(rowData)
  })
  Logger.log(`Finished writing to sheet! ✅`)
}

Once again, you might have noticed another foreign function -- writeSearchInfo. 😁 That will write all the search parameters to the sheet so we can easily know what results we are looking at.

// index.gs
function writeSearchInfo(data, organicResultsSheet){
  let search_query = data?.data?.search_query
  let headerContent = Object.keys(search_query)

  organicResultsSheet.appendRow(headerContent)

  let bodyContent = headerContent.map((item) => {
    return search_query[item]
  })

  organicResultsSheet.appendRow(bodyContent)
}

At this point, it is safe to call ourselves Apps Script heroes of the century, even if Google doesn't recognize that title! 😎 You can test this by reloading the spreadsheet file. After refreshing, our new custom menu item SEO Tools will appear on the menu bar, click on it to launch the app. Google may ask you for permissions to integrate the script with your spreadsheet -- proceed and grant that permission, and viola!!! Says the Spanish? Oh, I think it's the French!

Sample Sheet

Here's an example of how the sheet would look like after running several queries ✅
https://docs.google.com/spreadsheets/d/1UuwWujNyf2g0aMWwX26TUpiKjWvWeVQltL6oqxlMEAc/edit#gid=1142278110

Limitations

No graceful error handling.
Only a few data points written to sheet - organic results.
We are not spreading nested objects on the sheet.

You may proceed to tweak and customize your version as desired.

Source code

The source code is available on Github - google-serp-app-script

Conclusion

That's it! You can see how easy it is to get your Google SERP data from the comfort of your Google Sheets file in a matter of seconds. You can also integrate this API with your other applications, and must not necessarily consume the data on Google Sheets directly as we have done here.

Happy SERPing!!! 🎉

Can I Read? Analyze if a webpage is safe for reading (Web scraping using Puppeteer)

Caleb David — Thu, 03 Dec 2020 16:51:14 +0000

What are we building?

As a frontend engineer, you probably have come across the site caniuse.com - it tells you what web features are supported by different browsers. It also helps you make guided decisions upfront regarding your site's accessibility using different browsers.

The tool we will be building today is going to assess any given webpage and then give us its verdict whether that page is worth our time to read or not based on the the occurences of some certain keywords. It is very handy for avid article readers.

Costly assumptions

We are going to make a few assumptions here:

If a webpage contains more "positive words" than "negative words", we presume the page to be safe for reading.
If a webpage contains more "negative words" than "positive words", then we presume that the page is not worth our time - it may ruin our day 😆.

Keyword files

The "positive" and "negative" keywords list can be obtained on Github by jeffreybreen, that's an incredible compilation, a big shout-out to the author. Links to the two files on Github below:

Citation: Minqing Hu and Bing Liu. "Mining and Summarizing Customer Reviews."
Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2004), Aug 22-25, 2004, Seattle, Washington, USA,

Limitations

The approach we are using for analyzing the page will not always work out 100% correctly because we are not using natural language processing to analyze the whole context in which the keywords are used in the page (as in sentiment analysis systems), we are solely relying on word count frequency. I have plans to add support for NLP in subsequent series of this post, so keep an eye here if that interests you.

Thought process

As usual, it always pays to highlight our thought process because it will help steer our direction big time in this wild ride.

Convert the keywords text files to JSON files. The resulting json files will hold all the keywords in an array.
Get the address of the webpage we want to analyze and make a request to it.
Extract the text content of that page.
Compare each keyword in the JSON files with the extracted text and record each keyword's frequency in that extracted text (for both keyword files).
Sum the total number of positive keywords appearances and do same for the negative keywords
The category with the higher number determines if the page is worth reading or not. That is, if we end up with more positive keywords in the page, we mark that webpage as safe for reading and the opposite for the negative keywords.

Alright, let's wish ourselves success and hope our terminal don't get mad at us because we are just about to bounce on it. 😄.

Initialize a new project directory

mkdir can-i-read && cd can-i-read && npm init -y

We will download two packages: puppeteer and keyword-count

npm i --save puppeteer keyword-count

Note: Don't panic if it takes ages to finish the installation. Puppeteer usally takes longer time to install

Folder structure

For reference and clarity purposes, this is how our project directory will look like at the end. We come to that and explain later. Don't mind the dev.md file you are seeing there, it is the file that contains what you are reading now, so technically not part of the project files 😆.

Prepare the keyword files

We want to have both the keywords files stored in the assets directory in the root of our project directory. I have cleaned up these two files (positive and negative text files from the Github repo) to contain only the keywords because the original files from that repo contain acknowledgement notes which will require us to take extra step to clean them up.

Convert the text files to json files

Since we want to count the number of occurences of each keyword in the document body, I found an npm package that does this job extremely well - keyword-count. The package demands that we provide paths to three files:

{
    target: "/path/to/the/text/file/we/want/to/analyze", 
    keywordsList: "/path/to/json/file/containing/the/keywords", 
    outputPath: "/path/to/json/file/we/want/to/write/the/result/of/the/analysis"
}

That is why in the assets directory structure above, you are seeing one million and one files there 😄. Let me explain what each file in that directory is doing.

input.txt - This is the file containing the text of the document body we want to analyze. Visit any webpage, pop open devtools and type document.body.textContent in the console. The text it returns is what we want to analyze, and that is what goes to this file. We haven't created it yet but we'll come to that.
negative.txt - This is the same file we got from the keywords text file but we have removed the acknowledgement note for easy parsing. We want to convert this file to a json file so that the keyword-count package can be able to use it. This leads us to the third file in that directory.
negative.json - This is the json file that the keyword-count package expects to see the list of keywords to work with.
negativeResult.json - The keyword-count package spits the result of its computation in this json file. That's why you are are seeing it here.

The same explanation goes for the remaining files (positive.txt, positive.json and positiveReslt.json).

Wait, do I have to manually create all these files myself? Nope! We are going to create them programatically from our code. You should only have two files there - positive.txt and negative.txt (I will include them in the project repo).

Now that we have a clear understanding of the assets directory, let's remind ourselves of what we want to achieve - Convert the text files to json files ( positive.txt ==> positive.json, same for negative ).

At the root project directory, open your terminal window again and type the following:

mkdir utils && cd utils && touch textToJson.js

We created the utils directory and navigated into it and then created a textToJson.js file with that one-liner bash command. Open the textToJson.js file in your text editor of choice and paste in this block of code

const fs = require('fs')
const path = require('path')
const readline = require('readline')

async function processLine(filename) {    
    const fileStream = fs.createReadStream(path.join(__dirname, `./../assets/${filename}`))

    // This will contain the list of words in the text file
    let linesArray = []

    const rl = readline.createInterface({
        input: fileStream,
        crlfDelay: Infinity
    })

    // Iterate over each line and populate the `linesArray`
    for await (const line of rl) {
        linesArray.push(line)
    }

    // Initialize an empty object and populate it with the array of words
    let json = {}
    json["keywords"] = linesArray

    // Write the populated object to a json file on our filesystem
    try {
        await writeJSON(filename, json)
    } catch(err) {
        console.log(err)
    }
}

First we are requiring three modules to work with and then creating the processLine() function. This function takes in a filename (think of the filename variable as the name of the .txt file we want to convert to json - positive.txt or negative.txt) and creates a readable stream off of that file on our filesystem. We then pass this readable stream to the readline to extract every word line by line in the file passed. For every word found in that file, we push the word to the linesArray variale. After the for...of iteration is done, by this time, we have an array containing all the keywords extracted from the text file. We want to save this keywords as a json file on our disk - in the assets directory, we then called the writeJSON() function, passing it the filename and the json data we want to write to the file.

function writeJSON(filename, json){

    // Rename the filename from .txt to .json extension
    filename = filename.split('.')[0] + '.json'

    const filePath = path.join(__dirname, `./../assets/${filename}`)

    const data = JSON.stringify(json)


    fs.writeFile(filePath, data, (err) => {
        if (err) {
            return Promise.reject(`Error generating ${filename}`)

            // Exit program at this point, because this file will be used by the scraper API
            process.exit(1)
        }

        console.log(`${filename} generated`)

        return Promise.resolve()
    })
}

Remember, the filename argument passed to the writeJSON() function is just a string that will probably look like this: positive.txt or negative.txt. We then replaced the .txt extension to .json and append it to the full filepath. We also turned the json data to a string and call the standard writeFile method of the fs module.

Now whenever the processLine() gets called, it is going to generate an equivalent json file containing the keywords that the keyword-count package will use. E.g,

await processLine('positive.txt')
// It will generate a positive.json file in the assets directory

await processLine('negative.txt')
// It will generate a negative.json file in the assets directory

Shameless plug: if you are still here with me then kudos! You may also want to checkout my site webscrapingzone.com where I teach web scraping indepth, by building real-life projects and potential ways you can monetize it. Get a 50% discount on the course 😄

At this point, we are done with the functionality of generating the json files for the keyword-count package to work with. Also, we are exporting the whole of this module in the textToJSON function. Next is to make request to the webpage we want to analyze.

Old-school Linux hackers, (ohh, I forgot to mention the Powershell and Steve's fans too 😆 too), pop open your terminal and hit this one-liner again, creating the lib directory at the project's root, navigating inside and creating the scraper.js file.

mkdir lib && cd lib && touch scraper.js

Open the scraper.js and paste this huge block of code

'use strict';

const puppeteer = require('puppeteer');
const fs = require('fs');
const path = require('path');

// Browser and page instance
async function instance(){
    const browser = await puppeteer.launch({
        headless: false
    })

    const page = await browser.newPage()
    return {page, browser}
}

// Extract all body text from the page
async function getBodyText(){
    const {page, browser} = await instance()

    // Get the url we want to visit from the user
    let baseURL = process.argv[2] ? process.argv[2] : "https://webscrapingzone.com"

    try {
        await page.goto(baseURL, {waitUntil: 'networkidle0'})
        await page.waitForSelector('body')

        let bodyText = await page.evaluate(() => {
            let scriptTags = document.querySelectorAll('script')
            let styleTags = document.querySelectorAll('style')

            // Remove a tag from the page
            function removeTagsFromPage(tags) {
                tags.forEach((tag) => {
                    tag.innerText = ""
                })
            }

            removeTagsFromPage(scriptTags)
            removeTagsFromPage(styleTags)

            // Get the page's text content
            let text = document.body.textContent

            // Remove line-break from text
            text = text.replace(/\n/g, "").trim()

            return text
        })

        await browser.close()

        return bodyText

    } catch (err) {

        // ***Handle errors properly in production environment***
        console.log(err)
    }
}

Quite a bit happening there, but c'mon, it's not far beyond reach. Sip your drink one more time because we still have some journey across. That's our punishment for building the next billion-dollar product 😄.

Back from the digression, now let's explain what's happening in scraper.js. We created a instance() function, setup the browser and page objects from the puppeteer package and then returned them. The page object represents a blank browser page that we will use to make web requests.

We also created an async function called getBodyText(), as the name implies, we want to get all the text from the body tag of the page we want to analyze. Inside this function, we then made a request to the website we want to analyze, in our case here - webscrapingzone.com and then wait for the body tag to render before traversing the DOM.

The bodyText variable will contain all the text in the body tag of that page. But wait a second, including all the script and style tags? Allowing these two tags in our text we want to analyze will drop the accuracy rate of our program because they are codes and not actual text written by the author of that page. So we want to get rid of these two tags completely from the page.

The removeTagsFromPage() is a helper function that removes any tag from the page. Actually, we are not removing the tag, but setting the innerText property of the tag to an empty string. Example, open your devtools and paste the following block of code into your console and see what happens to the page. You just hacked yourself and saw a bright light staring at you 😆.

let bodyTags = document.querySelectorAll('body')

// Remove all tags from the page
function removeTagsFromPage(tags) {
    bodyTags.forEach((tag) => {
        tag.innerText = ""
    })
}

removeTagsFromPage(body Tags)

That's the kind of behaviour we want, but this time not removing everything on the page, instead just the script and style tags. We want to remove them so that our algorithm doesn't analyze some random piece of gibberish. We then extracted the text left in the page by running document.body.textContent. The text may contain some formatting and so many line breaks, we removed the line-breaks and then trimmed the text. Now this is the text we want to run our analysis on. Remember the input.txt file? This text will then go into this file, but not so just yet.

Give me a stretch at this time because we will continue in the next post of this series. In our next post, we will run the actual analysis and determine if a page is worth our time or not. In this section we have achieved two things:

Creating functionality for generating keyword files
Creating functionality for extracting the text we want to run our analysis

Stay tunned for the next post, you can follow me on Twitter to get updates when it's published @microworlds

Motivation

If you have found this article helpful in any way and generosity is the state of your mind right now, you can put a smile on my face like this one here😄 with a cold bottle of anything right here below:

cold bottle of anything 😆

Gracias 🙏

Image downloader with puppeteer and the fetch API

Caleb David — Tue, 24 Nov 2020 11:26:07 +0000

In this tutorial, we are going to build a webpage image downloader. Assuming you visit a webpage and saw that the images in that page are cool and you want to have your own copy without downloading them one by one, this simple tool we will build is going to be a life saver for you. This little project is also a good way to practice and hone your webscraping skills.

We will create a new directory called image-downloader and navigate into it. Pop open your terminal window and type in the following commands.

mkdir image-downloader && cd image-downloader

I will assume that you have node js and npm installed on your machine. We will then initialize this directory with the standard package.json file by running npm init -y and then install two dependencies namely puppeteer and node-fetch. Run the following commands to get them installed.

npm install --save puppeteer node-fetch --verbose

You probably just saw a new npm flag --verbose. When installing puppeteer, what happens behind the scenes is that npm also installs the chromium browser because it is a dependency of puppeteer. This file is usually large and we are using the --verbose flag to see the progress of the installation, nothing fancy, but let's just use it because we can.

One more thing to do before getting our hands dirty with code is to create a directory where we want all our images to be downloaded. Let's name that directory images. We will also create index.js file where all the app's logic will go.

mkdir images && touch index.js

Actually, it's great to clearly outline our thought process before writing a single line of code.

Get all image tags from the page and extract the href property from each of these image tags
Make request to those href links and store them into the images directory (Saving images to disk)

Step one 1: Getting all image tags and `href` property

'use strict';

const puppeteer = require('puppeteer');
const fetch = require('node-fetch');
const fs = require('fs')

// Extract all imageLinks from the page
async function extractImageLinks(){
    const browser = await puppeteer.launch({
        headless: false
    })

    const page = await browser.newPage()

    // Get the page url from the user
    let baseURL = process.argv[2] ? process.argv[2] : "https://stocksnap.io"

    try {
        await page.goto(baseURL, {waitUntil: 'networkidle0'})
        await page.waitForSelector('body')

        let imageBank = await page.evaluate(() => {
            let imgTags = Array.from(document.querySelectorAll('img'))

            let imageArray = []

            imgTags.map((image) => {
                let src = image.src

                let srcArray = src.split('/')
                let pos = srcArray.length - 1
                let filename = srcArray[pos]

                imageArray.push({
                    src,
                    filename
                })
            })

            return imageArray
        })

        await browser.close()
        return imageBank

    } catch (err) {
        console.log(err)
    }
}

Now let me explain what is happening here. First, we created an async function called extractImageLinks. In that function, we created an instance of a browser page using puppeteer and stored it in the page constant. Think of this page as the new page you get after launching your chrome browser. We can now heedlessly control this page from our code. We then get the url of the page we want to download the image from the user and stored it in a variable named baseURL. We then navigate to that URL using the page.goto() function. The {waitUntil: 'networkidle0'} object passed as the second argument to this function is to ensure that we wait for the for the network request to complete before we proceed with parsing the page. page.waitForSelector('body') is telling puppeteer to wait for the html body tag to render before we start extracting anything from the page.

The page.evaluate() function allows us to run JavaScript code in that page instance as if we were in our Google Chrome Devtools. To get all image tags from the page, we call the document.querySelectorAll("img") function. However, this function returns an NodeList and not an array. So to convert this to an array, we wrapped the first function with the Array.from() method. Now we have an array to work with.

We then store all the image tags in the imgTags variable and initialized imageArray variable as a placeholder for all the href values. Since imgTags has been converted into an array, we then map through every tag in that array and extract the src property from each image tag.

Now time for some little hack, we want to download the image from the webpage maintianing the original filename as it appears in the webpage. For instance, we have this image src https://cdn.stocksnap.io/img-thumbs/960w/green-leaf_BVKZ4QW8LS.jpg. We wnat to get the green-leaf_BVKZ4QW8LS.jpg from that URL. One way to do this is to split the string using the "/" delimeter. We then end up with something like this:

let src = `https://cdn.stocksnap.io/img-thumbs/960w/green-leaf_BVKZ4QW8LS.jpg`.split("/")

// Output
["https:", "", "cdn.stocksnap.io", "img-thumbs", "960w", "green-leaf_BVKZ4QW8LS.jpg"]

Now the last index of the array after running the split array method on the image source contains the image's name and the extension as well, awesome!!!

Note: to get the last item from any array, we subtract `1` from the `length`m of that array like so:

let arr = [40,61,12] 
let lastItemIndex = arr.length - 1 // This is the index of the last item

console.log(lastItemIndex)
// Output
2

console.log(arr[lastItemIndex])
// Output
12

So we store the index of the last item in the pos variable and then store the name of the file in the filename variable as well. Now we have the source of the file and the file name of the current image in the loop, we then push these values as an object in the imageArray variable. After the mapping is done, we return the imageArray because by now it has been populated. We also return the imageBank variable which now contains the image links (sources) and the filenames.

Saving images to disk

function saveImageToDisk(url, filename){
    fetch(url)
    .then(res => {
        const dest = fs.createWriteStream(filename);
        res.body.pipe(dest)
    })
    .catch((err) => {
        console.log(err)
    })
}


// Run the script on auto-pilot
(async function(){
    let imageLinks = await extractImageLinks()
    console.log(imageLinks)

    imageLinks.map((image) => {
        let filename = `./images/${image.filename}`
        saveImageToDisk(image.src, filename)
    })
})()

Now let's decipher this little piece. In the anonymous IIFE, we are running the extractImageLinks to get the array containing the src and filename. Since the function is returns an array, we run the map function on that array and then pass the required parameters (url and filename) to saveImageToDisk. We then use the fetch API to make a GET request to that url and as the response is coming down the wire, we are concurrently piping it into the filename destination, in this case, a writable stream on our filesystem. This is very efficient because we are not waiting for the image to be fully loaded in memory before saving to disk but instead saving every chunk we get from the response directly.

Lets's run the code, cross our fingers and checkout our images directory

node index.js  https://stocksnap.io

We should see some cool images there in. Wooo! You can add this to your portfolio. There are so many improvements that can be done to this little software, such as allowing the user to specify the directory they want to download the image, handling Data URI images, proper error handling, code refactoring, creating a standalone CLI utility for it. Hint: use the commander npm package for that, etc. You can go ahead and extend this app and I'll be glad to see what improvements you will make it.

Full code

'use strict';

const puppeteer = require('puppeteer');
const fetch = require('node-fetch');
const fs = require('fs')

// Browser and page instance
async function instance(){
    const browser = await puppeteer.launch({
        headless: false
    })

    const page = await browser.newPage()
    return {page, browser}
}

// Extract all imageLinks from the page
async function extractImageLinks(){
    const {page, browser} = await instance()

    // Get the page url from the user
    let baseURL = process.argv[2] ? process.argv[2] : "https://stocksnap.io"

    try {
        await page.goto(baseURL, {waitUntil: 'networkidle0'})
        await page.waitForSelector('body')

        let imageLinks = await page.evaluate(() => {
            let imgTags = Array.from(document.querySelectorAll('img'))

            let imageArray = []

            imgTags.map((image) => {
                let src = image.src

                let srcArray = src.split('/')
                let pos = srcArray.length - 1
                let filename = srcArray[pos]

                imageArray.push({
                    src,
                    filename
                })
            })

            return imageArray
        })

        await browser.close()
        return imageLinks

    } catch (err) {
        console.log(err)
    }
}

(async function(){
    console.log("Downloading images...")

    let imageLinks = await extractImageLinks()

    imageLinks.map((image) => {
        let filename = `./images/${image.filename}`
        saveImageToDisk(image.src, filename)
    })

    console.log("Download complete, check the images folder")
})()

function saveImageToDisk(url, filename){
    fetch(url)
    .then(res => {
        const dest = fs.createWriteStream(filename);
        res.body.pipe(dest)
    })
    .catch((err) => {
        console.log(err)
    })
}

Cheers! 🎉

Introduction to webscraping

Caleb David — Tue, 24 Nov 2020 10:42:44 +0000

What is web scraping?

Web scraping or screen scraping or web data extraction, etc is the act of extracting data from web pages in an automated way. The extracted data is usually in an unstructured format. After the extraction, the data usually needs to be cleaned up and be presented in a format that will be useful depending on the purpose why it was extracted.

There are different techniques and languages employed in web scraping and in my opinion, puppeteer has redefined how web scraping is done with its incredible simple API - no language war intended 😄. The goal is to extract the data and turn it into something useful, nobody cares how you get the data, well, probably your engineering manager does 😎.

Who needs web scraping?

Web scraping can be applied to countless niches ranging from e-commerce sites, real estate, finance, legal firms, entartainment, news, fashion, social media, etc. The e-commerce niche seems to be the primary target of web scrapers. Many e-commerce stores monitor prices of products on competitors' websites just to set optimum prices of products on their stores.

For example, if Amazon and eBay are selling a similar product, both stores can monitor each other's products prices to ensure they are not offering that product at a charity price and at the same time, they are not going way overboard with the price. In short, web scraping helps e-commerce stores maintain competitive edge in that niche.

Marketers can also generate quality leads through web scraping public databases, some people may consider this act as less than legal - and that leads us to the next point. Regardless of the case, extracting contact information of potential customers can be done in a split second with web scraping.

Is web scraping legal and free?

Whatever niche webscraping is used, we are talking about "free data" which translates to free value. Well, not so fast, "free" here can come with some legal implications. Knowing the restrictions attached to web scraping is a most-know for every aspiring data miner because it's going to save you alot of headaches in the future.

Most websites have a file called robots.txt placed at the root directory of their site. Example amazon.com/robots.txt. This file contains rules for scraping the site, it specifies which endpoints are allowed to be hit and which ones are not. It is very important you adhere to these rules to avoid being blocked from that site or even being sued. If you are new to robots.txt concepts, here's an article by Patrick Sexton, he did an excellent job breaking the nitty-gritty concepts The robots.txt file.

A rule of thumb applies - never overload any website that you are scraping. When you spam them with thousands of concurrent requests per second, you are making them burn more resources (bandwidth is expensive 💳). Performance of the service will be adversely affected which is something you don't want to do. oOher users may experience slow response or even server downtime. Always ensure the servers are blinking green when scraping. Let's be our brothers' keepers 👍

What and how can I monetize my web scraping skills?

It is only natural to think of the benefits of learning web scraping. After you have dedicated time and effort into learning web scraping, and then what next??? Do you just scrape for fun? Would it be cool if I tell you that you can learn this skill and generate some kind of revenue from it? Here are some few ways you can monetize your new found skills.

Getting hired - some companies hire developers solely for web scraping purposes. You don't have to be a genius in backend development before you get hired as a web scraper by a company. There are "web-scraping-specific" jobs that you can apply for. Example Ziprecruiter Webscraping Jobs
Freelancing - There are tonnes of gigs available on freelancing sites that only have to do with web scraping. You can set your price, do the task and watch the money flowing Truelancer Webscraping Jobs
Build a startup - If you like the taste of freedom from "bosses", you can decide to venture into the startup world by creating a web service that people can use and pay you for your services. There are companies that offer products using webscraping. Example Truelancer Webscraping Jobs

If you have come this far in this journey, thank you very much for your time and patience. Tab yourself three times on the back because you are a hero. 🍷

Shameless plug 😊

If you enjoyed this article and are feeling super pumped, I run 🔗 webscrapingzone.com where I teach advanced webscraping techniques by building real-world projects and how you can monetize your webscraping skills instantly without even being hired. It's still in beta stage but you can join the waiting list and get 💥 50% 💥 off when the course is released.

You can follow me on twitter - @microworlds

Thank you for your time 👍

DEV Community: Caleb David

Building a Google SERP extension in Google Sheets (Apps Script + RapidAPI) 📈

Introduction

Goal

Folder sctructure

Create the custom menu

Send data from HTML to Apps Script

Form parameters

Make the API request

Write the data to a sheet

Sample Sheet

Limitations

Source code

Conclusion

Can I Read? Analyze if a webpage is safe for reading (Web scraping using Puppeteer)

What are we building?

Costly assumptions

Keyword files

Limitations

Thought process

Initialize a new project directory

Folder structure

Prepare the keyword files

Convert the text files to json files

Motivation

Image downloader with puppeteer and the fetch API

Step one 1: Getting all image tags and href property

Note: to get the last item from any array, we subtract 1 from the lengthm of that array like so:

Saving images to disk

Full code

Introduction to webscraping

What is web scraping?

Who needs web scraping?

Is web scraping legal and free?

What and how can I monetize my web scraping skills?

Shameless plug 😊

Step one 1: Getting all image tags and `href` property

Note: to get the last item from any array, we subtract `1` from the `length`m of that array like so: