DEV Community: Waqasabi

Optimizing and Deploying Puppeteer Web Scraper

Waqasabi — Sat, 08 Feb 2020 12:31:24 +0000

In this post, we'll look into how we can optimize and improve our puppeteer Web Scraping API. We'll also look into several puppeteer plugins to improve our API and have it produce more consistent results. Even though this post refers to a Web Scraping API that we built with puppeteer, some of these tips can be applied to general web scraping and can be implemented with other web scrapers built with different tools and languages i.e. Python.

This is the 3rd part of the 3-Part Series Web Scraping with Puppeteer:

1st Part: Basics of Puppeteer and Creating a Simple Web Scrapper.
2nd Part: Creating Search Engine API using Google Search with Node/Express and Puppeteer.
3rd Part: Optimising our API, Increasing Performance, Troubleshooting Basics and Deploying our Puppeteer API to the Web.

Table Of Contents - Part 3

Headless Mode Off (Troubleshooting)
Getting to know your Webpage
Blocking Images and CSS
Using getElementsByTagName Wherever Possible
Useful Puppeteer Plugins (Adblock & Stealth)
Deploying Your Puppeteer API
Conclusion

Headless Mode Off (Troubleshooting)

The simplest way to troubleshoot puppeteer is to turn headless mode off. Doing so shows the full version of the Chromium browser and you can see exactly what puppeteer is trying to do. To do this, we can set the headless option to false before launching a browser:

const browser = await puppeteer.launch({headless: false}); // default is true

Now if we execute our API, we can see exactly what puppeteer is trying to do! Don't forget to turn it off after you're done troubleshooting as this increases execution-time.

For advanced troubleshooting, you can refer to the troubleshooting docs.

Improving Performance

To get started with improving our API's performance, we need to first measure the execution time. This will help us measure the difference after we apply all the optimizations. Since our puppeteer code lies in the file searchGoogle.js we'll modify it a bit and execute that file separately.

We can use performance.now() to measure the performance by doing:

const averageTime = async () => {
  //snapshot in time
  const t0 = performance.now();

  //wait for our code to finish
  await searchGoogle('cats');

  //snapshot in time 
  const t1 = performance.now();

  //console logs the difference in the time snapshots
  console.log("Call to searchGoogle took " + (t1 - t0) + " milliseconds.");
}

To use performance.now() We need to install the library perf_hooks :

npm install perf_hooks

Now we can import it with:

const {performance} = require('perf_hooks');

We can create an averageTime function that runs searchGoogle 20 times and calculates the average execution time. This will take a long-time to execute, however, it will give us a good average (you can increase it for an even better average). Due to the total-time required, I don't recommend calculating the average however I wanted to mention this for anyone curious about how to measure execution time. Please keep in mind that this performance is dependent on your network connection and computer. Adding this to our searchGoogle.js file:

const puppeteer = require('puppeteer');
const {performance} = require('perf_hooks');

//minimised code
const searchGoogle = async (searchQuery) => {...};

//calculates average time by executing searchGoogle 20 times asynchronously
const averageTime = async () => {
    const averageList = [];

    for (let i = 0; i < 20; i++) {
        const t0 = performance.now();

        //wait for our function to execute
        await searchGoogle('cats');

        const t1 = performance.now();

        //push the difference in performance time instance
        averageList.push(t1 - t0);
    }

    //adds all the values in averageList and divides by length
    const average = averageList.reduce((a, b) => a + b) / averageList.length;

    console.log('Average Time: ' + average + 'ms');
};

//executing the average time function so we can run the file in node runtime.
averageTime();

module.exports = searchGoogle;

To execute the file we can run the command:

node searchGoogle.js

Now we can go ahead and start optimizing our API.

Getting to know your Webpage

This is one of the most important steps to optimizing your API's performance. Sometimes playing around with a webpage/website reveals different and faster ways to get the necessary information.

In our example, we were manually typing the search query in the google search bar and waiting for the results to load. We did this to see how typing behaves with puppeteer, however, we can instead use a faster way of displaying google search results for our search query and that is to use URL Params with Google Search, and we can do this by simply entering our search query after the q= in the URL https://www.google.com/search?:

https://www.google.com/search?q=cats

This will display all the results for the search query 'cats'. To add this, we need to first remove the code that navigates to www.google.com and enters the search query into the search bar:

//finds input element with name attribute 'q' and types searchQuery
await page.type('input[name="q"]', searchQuery);

//finds an input with name 'btnK', after so it executes .click() DOM Method
await page.$eval('input[name=btnK]', button => button.click());

Removing this and adding the Google Search with URL Params to our searchGoogle.js file::

const puppeteer = require('puppeteer');
const {performance} = require('perf_hooks');

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();

    //use google search URL params to directly access the search results for our search query
    await page.goto('https://google.com/search?q='+searchQuery);

    //wait for one of the div classes to load
    await page.waitForSelector('div[id=search]');

    //minimised - Find all div elements with ...  the information we need
    const searchResults = await page.$$eval('div[class=bkWMgd]', results => {...});

    await browser.close();

    return searchResults;
};

//minimised - Calculates average time by executing searchGoogle 20 times asynchronously
const averageTime = async () => {...};

module.exports = searchGoogle;

Sometimes the website you are trying to scrape provides better ways that you can use to optimize your web scrapper. In our case Google Search can be used through URL Params and we don't need to manually enter queries into the Google Search bar and press Enter (Saving us some time). This is why it's very important to get to know the webpage you're trying to scrape.

Blocking Images and CSS

A significant amount of webpages on the web make use of images and they are known to reduce page load-time due to their size. Since we don't really care about the images or the CSS of the webpage, we can just prevent the page from making requests to images or stylesheet files. This way we can focus on the HTML (The part we care about). The difference in the load time will depend on the webpage you're trying to scrape. This example was taken from official docs.

To proceed to block images we need to add a Request Interceptor.

This provides the capability to modify network requests that are made by a page.

This means that we can prevent the webpage from making any requests to certain resources. In our case, we can use it to prevent the webpage from making requests to images and stylesheets. Setting this up is very simple, we need to turn the Request Interceptor on and abort requests made to images:

//turns request interceptor on
await page.setRequestInterception(true);

//if the page makes a  request to a resource type of image then abort that request
page.on('request', request => {
  if (request.resourceType() === 'image')
    request.abort();
  else
    request.continue();
});

Similarly, we can also do the same thing if the resource type is a stylesheet:

//turns request interceptor on
await page.setRequestInterception(true);

//if the page makes a  request to a resource type of image or stylesheet then abort that request
page.on('request', request => {
  if (request.resourceType() === 'image' || req.resourceType() === 'stylesheet')
    request.abort();
  else
    request.continue();
});

Adding this to our searchGoogle.js:

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();

    //turns request interceptor on
    await page.setRequestInterception(true);

    //if the page makes a  request to a resource type of image or stylesheet then abort that            request
    page.on('request', request => {
        if (request.resourceType() === 'image' || request.resourceType() === 'stylesheet')
            request.abort();
        else
            request.continue();
    });

    //use google search URL params to directly access the search results for our search query
    await page.goto('https://google.com/search?q='+searchQuery);

    //wait for one of the div classes to load
    await page.waitForSelector('div[id=search]');

    //minimised - Find all div elements with ...  the information we need
    const searchResults = await page.$$eval('div[class=bkWMgd]', results => {...});

    await browser.close();

    return searchResults;
};

This way of blocking supports other types of resources:

document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, other.

Using getElementsByTagName Wherever Possible

This might not necessarily lower the execution-time but it might help, document.getElementsByTagName method is described as:

The getElementsByTagName method of Document interface returns an HTMLCollection of elements with the given tag name.

This means that if we want all the <a> tags on the page then we do:

nodes = document.getElementsByTagName('a');

the alternative to doing this would be using document.querySelectorAll and this is more widely used:

nodes = document.querySelectorAll('a');

Based on tests it seems that document.getElementsByTagName() executes a little bit faster than document.querySelectorAll() when the aim is to select all tags on a page, this might not come as a surprise however I thought I should mention this since it's not very commonly used. In our case, this is not really applicable since we don't necessarily need to fetch a certain HTML tag.

Useful Puppeteer Plugins (Adblock & Stealth)

With the help of puppeteer-extra we can make use of plugins and teach puppeteer new tricks through plugins. We'll only be going through puppeteer-extra-plugin-adblocker and puppeteer-extra-plugin-stealth. If you want to check out all available plugins, you can do so here.

We need to first install puppeteer-extra, puppeteer-extra-plugin-adblocker & puppeteer-extra-plugin-stealth:

npm install puppeteer-extra puppeteer-extra-plugin-adblocker puppeteer-extra-plugin-stealth

Please keep in mind that these plugins might not necessarily help the execution time.

Stealth Plugin

We will be using Stealth Plugin to create consistent environments and make the results more similar to what we see when we browse the webpage, this is because webpages are able to detect if the user browsing the webpage is Headless and they might choose to serve different content or not serve any content at all. For this reason, this plugin can allow us to create a consistent environment when scraping. According to the docs:

Applies various evasion techniques to make detection of headless puppeteer harder.

It's very easy to use the plugins, to make use of plugins we need to first replace our puppeteer client with puppeteer-extra client and we do the following to add Stealth Plugin:

const puppeteer = require('puppeteer-extra')

// Add stealth plugin and use defaults (all tricks to hide puppeteer usage)
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

However before we execute it, we need to make sure that we provide {headless: false} config to our puppeteer client during launch, otherwise puppeteer-extra will throw an error:

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch({headless: true});

  const page = await browser.newPage();

  ...

  await browser.close();

  return searchResults;
};

However, this plugin isn't necessarily designed to reduce page-load time so we'll likely not see any difference in the execution time.

Adblock Plugin

We will be using the Adblock-Plugin to block any ads or trackers on our page since ads/trackers can play a role in our page load-time. According to the docs:

Ads and trackers are on most pages and often cost a lot of bandwidth and time to load pages. Blocking ads and trackers allows pages to load much faster because fewer requests are made and less JavaScript needs to run.

This automatically blocks all the ads when using puppeteer. However, at the moment there is conflict between this plugin and our method of blocking requests to images and stylesheets, this is because Adblock-plugin and our method of blocking image/stylesheet make use of Request Interception and puppeteer doesn't expect multiple entities to be interested in using Request Interception, therefore for your solution you have to either block images/stylesheets / other resources or use this Adblock plugin. For use-case, I would recommend testing both and seeing which one yields better results.

Adding this to our searchGoogle.js:

const puppeteer = require('puppeteer-extra')
const {performance} = require('perf_hooks');

// Add stealth plugin and use defaults (all tricks to hide puppeteer usage)
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

// Add adblocker plugin, which will transparently block ads in all pages you
// create using puppeteer.
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker')
puppeteer.use(AdblockerPlugin({blockTrackers: true}))

//minimised searchGoogle with Image / Stylesheet blocking removed
const searchGoogle = async (searchQuery) => {...};

//minimised averageTime
const averageTime = async () => {...};

module.exports = searchGoogle;

This will block all ads and trackers that might be present on our page. There are other options available with the Adblock Plugin:

interface PluginOptions {
  /** Whether or not to block trackers (in addition to ads). Default: false */
  blockTrackers: boolean
  /** Persist adblocker engine cache to disk for speedup. Default: true */
  useCache: boolean
  /** Optional custom directory for adblocker cache files. Default: undefined */
  cacheDir?: string
}

Deploying Your Puppeteer API

Now that we know about different ways of lowering execution time and creating more consistent results, we can look into how we can deploy our puppeteer API to the cloud. For this post, we'll be deploying to Heroku but the process is very similar for other platforms. If you're interested in deploying to other cloud platforms such as AWS, Google App Engine etc, please refer to this troubleshooting guide.

Before we deploy to Heroku, we need to edit our server.js express file so that Heroku can use ports and IP it needs for the express server. We need to add dynamic port and IP, this will allow Heroku to use the port and IP it needs:

const ip = process.env.IP || '0.0.0.0';
const port = process.env.PORT || 8080;

app.listen(port, ip);

Adding this to our server.js file:

const express = require('express');
const app = express();

const ip = process.env.IP || '0.0.0.0';
const port = process.env.PORT || 8080;

//Import puppeteer function
const searchGoogle = require('./searchGoogle');

//Catches requests made to localhost:3000/search
app.get('/search', (request, response) => {

    //Holds value of the query param 'searchquery'.
    const searchQuery = request.query.searchquery;

    //Do something when the searchQuery is not null.
    if (searchQuery != null) {

        searchGoogle(searchQuery)
            .then(results => {
                //Returns a 200 Status OK with Results JSON back to the client.
                response.status(200);
                response.json(results);
            });
    } else {
        response.end();
    }
});

//Catches requests made to localhost:3000/
app.get('/', (req, res) => res.send('Hello World!'));

//Initialises the express server on the port 30000
app.listen(port, ip);

Once we have that setup, we can start uploading our server to Heroku. You need to make sure you have a Heroku account before proceeding

#skip git init if you already have a git repository initialized
git init

git add .
git commit -m "Added files"
heroku login

After logging in through the browser/terminal, we can create a new Heroku app.

heroku create

Please make sure that you do not already have 5 apps on your Heroku account as free accounts are only limited to 5 apps. After Heroku creates the app, all you need to do is push the code onto Heroku:

git push Heroku master

If this command gives you an error:

fatal: 'heroku' does not appear to be a git repository
fatal: 'heroku' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Then you need to go to your Heroku dashboard and copy the name of the app you just created and do:

heroku git:remote -a your-app-name

We are almost done. We need to now take care of puppeteer dependencies. The list of Dependencies can be found here. No matter where you deploy it, you need to make sure that these dependencies are installed on the machine hosting puppeteer. Luckily for us, Heroku has build packs. Buildpacks are a collection of dependencies that instruct Heroku on what is needed to be installed for the project.

Running Puppeteer on Heroku requires some additional dependencies that aren't included on the Linux box that Heroku spins up for you.

The URL of the buildpack:https://github.com/jontewks/puppeteer-heroku-buildpack

To add the buildpack to our project we can just do:

heroku buildpacks:add https://github.com/jontewks/puppeteer-heroku-buildpack.git

Before we push the changes, we need to add one last configuration to our searchGoogle.js We need to use '--no-sandbox' mode when launching Puppeteer. This can be done by passing it as an argument to your .launch():

const puppeteer = require('puppeteer-extra');
const {performance} = require('perf_hooks');

// Add stealth plugin and use defaults (all tricks to hide puppeteer usage)
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

// Add adblocker plugin, which will transparently block ads in all pages you
// create using puppeteer.
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');
puppeteer.use(AdblockerPlugin({blockTrackers: true}));

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch({headless: true, args: ['--no-sandbox']});
  ...
  ...
  await browser.close();

  return searchResults;
};

We can now commit the changes and push to heroku master:

git add .
git commit -m 'Disabled sandbox mode'
git push heroku master

After a while, our Puppeteer API should be deployed, we can simply click the URL from the terminal or go to our dashboard and open our app through there and we can simply just make requests to the URL provided by Heroku:

https://yourappname.herokuapp.com/search?searchquery=cats

And we can change the search query by changing the URL parameter search query. Our Search Engine API is ready!

Please make sure that you are not using Adblock Plugin and blocking Images/ Resources with Request Interception together and to only use one of them, otherwise the Heroku server will run into errors.

The code for this project can be found on Github.

Conclusion

This is the end of this post and the end of the 3-Part Series Web Scraping with Puppeteer! I hope you enjoyed this series and found it to be useful!

If you're interested in other use-cases, check out the Net-Income Calculator, which uses Node/Express Puppeteer API to scrap information about state taxes and average rent in cities from websites. You can check out it's Github Repo.

If you enjoyed reading this and would like to provide feedback, you can do so anonymously here. Any feedback regarding anything is appreciated!

Building a Search Engine API with Node/Express and Puppeteer using Google Search

Waqasabi — Wed, 22 Jan 2020 17:27:28 +0000

In this post, we're going to build a Search Engine API with Node/Express & Puppeteer. It will use web scraping to get top results from Google

If you haven't read the first post, I highly recommend reading it! It goes over the basics of web scraping with puppeteer.

Note: Unfortunately, the concepts discussed in Part 2 and 3 are still valid, however, the examples used to demonstrate these concepts, no longer work. This is the nature of web scrapping. If a website decides to change its class name for a certain HTML element, then the web scrapper needs to be adjusted to those class names. In this example, we used class names which Google used at the time of writing this post, however, those class names have changed since then and so the example no longer works.

This is why sometimes it's better to find a dynamic way to target an element so that if the class name or element id was to change, the web scrapper would still continue to operate.

This is part a 3 Part Series:

1st Part: Basics of Puppeteer and Creating a Simple Web Scrapper.
2nd Part: Creating Search Engine API using Google Search with Node/Express and Puppeteer.
3rd Part: Optimising our API, Increasing Performance, Troubleshooting basics and Deploying our Puppeteer API to the Web.

Table Of Contents - Part 2

API Requirements
Setting up a Node/Express Server
Creating the Search Engine API with Puppeteer

API Requirements

Before we get started, It's important to know what we're trying to build. We're going to build an API, that will take in a search request and return a JSON with the top results from Google's Search Results.

The information we care about from the results:

Website Title
Website Description
Website URL

The search request will be a GET request and we're going to make use of URL Query Params to specify the search query. The user will send a request to /search with search query searchquery=cats:



localhost:3000/search?searchquery=cat

Our API is expected to return the top Results about cats from Google in JSON:



[
    {
      title: 'Cats Are Cool',
      description: 'This website is all about cats and cats are cool',
      url: 'catsarecool.com'
    },
    ...
    {
      title: 'Cats funny videos',
      description: 'Videos all about cats and they are funny!',
      url: 'catsfunnyvideos.com'
    }
]

Now that we know our requirements, we can go ahead start building our API

Setting up a Node/Express Server

If you want to skip setting up the Node/Express server, you can skip right ahead to the part where we start writing the code for puppeteer to crawl Google. Although I recommend reading this part.

To get started, we're going create a new project directory and initilise npm:



mkdir search-engine-api
cd search-engine-api
npm init -y

For this API, we're going to use Express.js to create a simple API and so we need to install express, puppeteer and nodemon. We're going to use nodemon for development. Nodemon will detect any changes in our server file and automatically restart our server. This will save us time in the long run.



npm i express puppeteer nodemon

We can now create our server file:



touch server.js

After doing so, we need to configure our package.json and add scripts for npm start to start our server. For development purposes, we can create a script with nodemon. We will use npm run dev for running the nodemon script:



{
  "name": "search-engine-api",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
    "start": "node server.js",
    "dev": "nodemon server.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1",
    "nodemon": "^2.0.2",
    "puppeteer": "^2.0.0"
  }
}

Now if we run npm run dev and try to make changes in our server.js file, nodemon will automatically restart the server. We can now start writing code for our server.

Before we get into building our API, we need to setup a simple Express server. We're going to use the Hello World example provided by Express Docs:



const express = require('express');
const app = express();
const port = 3000;

//Catches requests made to localhost:3000/
app.get('/', (req, res) => res.send('Hello World!'));


//Initialises the express server on the port 30000
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

This creates an express server on the port 3000 of our local machine. If someone sends a GET Request to localhost:3000/ our server responds with Hello World. We can see it working by opening the URLlocalhost:3000/ in a browser.

We're going to create a new route for our search. This is where we will pass information in the URL with Query Params, for example, If we want search results for the query "dogs", we can send a request to:



localhost:3000/search?searchquery=dogs

To implement this, we need to create a new GET request function with express and since we expect this to be a GET request, we can make use of app.get(route, callbackFunc)



const express = require('express');
const puppeteer = require('puppeteer');
const app = express();
const port = 3000;

//Catches requests made to localhost:3000/search
app.get('/search', (request, response) => {
    //Do something when someone makes request to localhost:3000/search
    //request parameter - information about the request coming in
   //response parameter - response object that we can use to send a response
});

//Catches requests made to localhost:3000/
app.get('/', (req, res) => res.send('Hello World!'));


//Initialises the express server on the port 30000
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

Now that we have a function that catches requests made to localhost:3000/search, we can start looking into how we can make use of any query params that are in the URL. Any requests made to this route will execute the callback function in this handler.

Express allows us to access the query params through the request parameter. In our case, since we named our query field searchquery, we can access it through that:



//Catches requests made to localhost:3000/search
app.get('/search', (request, response) => {

  //Holds value of the query param 'searchquery' 
    const searchQuery = request.query.searchquery;
});

However, if this query does not exist then we have nothing to search for, so we can handle that case by only doing something when the search query is provided. If the search query does not exist then we can quickly end the response without any data with response.end()



//Catches requests made to localhost:3000/search
app.get('/search', (request, response) => {

  //Holds value of the query param 'searchquery'.
    const searchQuery = request.query.searchquery;

  //Do something when the searchQuery is not null.
  if(searchQuery != null){

  }else{
    response.end();
  }
});

Now that we have our Node/Express server setup, we can start writing code for our scraper.

Creating the Search Engine API with Puppeteer

When it comes to web-scraping Google, one way to search something directly on Google Search is to pass the search query as a URL query parameter:



https://www.google.com/search?q=cat

This will show us results for the keyword 'cat' on Google. This would be the ideal approach, however, for the purpose of this post, we're going to do things the difficult way by opening google.com(Homepage) having puppeteer type in the search box and press Enter to get the results.

We'll do it this way because not all websites make use of query parameters and sometimes the only way to get to the next step of the website (in our case the results page) is to do things manually in the first step.

At this point our server.js looks like this:



const express = require('express');
const puppeteer = require('puppeteer');
const app = express();
const port = 3000;

//Catches requests made to localhost:3000/search
app.get('/search', (request, response) => {

    //Holds value of the query param 'searchquery'.
    const searchQuery = request.query.searchquery;

    //Do something when the searchQuery is not null.
    if(searchQuery != null){

    }else{
      response.end();
    }
});

//Catches requests made to localhost:3000/
app.get('/', (req, res) => res.send('Hello World!'));


//Initialises the express server on the port 30000
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

We're going to create a new function called searchGoogle. This will take in the searchQuery as an input parameter and return an array of JSON with the top results.

Before we go ahead to write searchGoogle with puppeteer, we're going to write the footprint of the function so we know how the code should behave:



const express = require('express');
const puppeteer = require('puppeteer');
const app = express();
const port = 3000;

//Catches requests made to localhost:3000/search
app.get('/search', (request, response) => {

    //Holds value of the query param 'searchquery'.
    const searchQuery = request.query.searchquery;

    //Do something when the searchQuery is not null.
    if (searchQuery != null) {

        searchGoogle(searchQuery)
            .then(results => {
                //Returns a 200 Status OK with Results JSON back to the client.
                response.status(200);
                response.json(results);
            });
    } else {
        response.end();
    }
});

//Catches requests made to localhost:3000/
app.get('/', (req, res) => res.send('Hello World!'));


//Initialises the express server on the port 30000
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

Since puppeteer works asynchronously, we need to wait for the results to be returned from searchGoogle. For this reason, we need to add a .then so that we make sure to wait until searchGoogle processes and fetches the results before we make use of them and the results can be accessed with a callback function which will have the results as the first parameter. After which we can respond back to the client by using response.json().

response.json() returns a JSON back to the client. There are different methods that you can use with the response. You can read more about them on the official Express docs.

We can now start writing code and building the puppeteer function searchGoogle. To do this, we're going to create a new file in the same directory. This is because having a separate file will allow us to test our puppeteer file without having to make a manual request to our server, which can be a time-consuming process. We'll name it searchGoogle.js:



touch searchGoogle.js

Now we need to initialize the function in the file:



const puppeteer = require('puppeteer');

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.goto('https://google.com');

    await browser.close();
};

export default searchGoogle;

Right now, we are just launching a Headless Instance of Chrome and browsing to Google. We need to now find the Search Bar, where we can write the query. For this, we need to inspect the source code of Google's Homepage.

After using the Mouse tool for selecting elements, we can see the HTML for this search bar:

We can see that it has name="q" We can use it to Identify and target the input through puppeteer. To type in our search query, puppeteer provides a function for the page page.type(selector, textToType). With this we can target any forms and input our values directly:



const puppeteer = require('puppeteer');

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.goto('https://google.com');

    //Finds input element with name attribue 'q' and types searchQuery
    await page.type('input[name="q"]', searchQuery);

  await browser.close();
};

export default searchGoogle;

Just to make sure, everything is working we can take a screenshot after it is done typing:



const puppeteer = require('puppeteer');

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.goto('https://google.com');

    //Finds input element with name attribue 'q' and types searchQuery
    await page.type('input[name="q"]', searchQuery);

    await page.screenshot({path: 'example.png'});

    await browser.close();
};

//Exports the function so we can access it in our server
module.exports = searchGoogle;

searchGoogle('cats');

As you can see, at the end of the file we make a call to the searchGoogle function. This is so we can start testing it. We can now go to our command line and execute:



node searchGoogle.js

After few seconds, the file should finish executing and you should be able to view the screenshot:

Now, all we need to do is either have puppeteer press 'Enter' on the keyboard or click 'Google Search' button below the Search Bar.

Both approaches are suitable solutions, however, for precision, we're going to have puppeteer press 'Google Search'. However, if you were to press Enter this is how you would do it:



 await page.keyboard.press('Enter');

We're going to inspect the page once again and find information regarding the 'Google Search' Button. Doing so reveals this:

We can see that it has a name "btnK". We can use this to target the element and click it:



//Finds the first input with name 'btnK', after it is found, it executes .click() DOM Event Method
await page.$eval('input[name=btnK]', button => button.click());

Adding it to our file:



const puppeteer = require('puppeteer');

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.goto('https://google.com');

    //Finds input element with name attribue 'q' and types searchQuery
    await page.type('input[name="q"]', searchQuery);

    //Finds an input with name 'btnK', after so it executes .click() DOM Method
    await page.$eval('input[name=btnK]', button => button.click());

    await page.screenshot({path: 'example.png'});

    await browser.close();
};

searchGoogle('cats');

//Exports the function so we can access it in our server
module.exports = searchGoogle;

Executing the file and seeing the screenshot yields this result:

We need to make sure to wait for Google to load up all the results before we do anything. There are different ways we can do this. If we want to wait for a certain time we can use:



await page.waitFor(durationInMilliseconds)

Alternatively, if we already know the element we are looking for, then we can use waitForSelector to wait for puppeteer to load the first element with the matching selector before proceeding:



await page.waitForSelector('selector');

This will wait for the selector to load before proceeding. To use this, we need to first identify the selector for our results, so that puppeteer can wait for the results selector to load before proceeding. You should keep in mind that this will only wait for the first selector it finds.

After going through the HTML source code for the search results, I found that all the search results are stored in a div with an id search:

So we can use waitForSelector(selector) and target the div with id=search:



const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.goto('https://google.com');

    //Finds input element with name attribue 'q' and types searchQuery
    await page.type('input[name="q"]', searchQuery);

    //Finds an input with name 'btnK', after so it executes .click() DOM Method
    await page.$eval('input[name=btnK]', button => button.click());

    //Wait until the first div element with id search laods
    await page.waitForSelector('div[id=search]');

    await page.screenshot({path: 'example.png'});

    await browser.close();
};

searchGoogle('cats');

//Exports the function so we can access it in our server
module.exports = searchGoogle;

Now that our results have loaded, we can start parsing them. If you want to skip the part where we try to find the divs with relevant information then you can skip right ahead to the implementation.

If we take a closer look at the source code to make meaning out of the HTML, we can see that the information we're looking for is stored in divs with class=bkWMgd however not all divs with this class contain relevant information, some of these div contain video recommendations, news stories etc. The ones we're interested in, are the ones with an h2 title with Web Results text.

If we take a closer look at that div, we can see that it's nested very deeply. For this reason, we're going to make use of special selectors to target deep children. The main information is stored in the div with class 'g':

We can target the specific divs we care about. We're going to use '>' CSS Selectors known as Child-Combinators to target the nested information.

We can target nested elements like so:



<div class='1'>
    <div class='2'>
        <div class='3'>
            <p>Information</p>
        </div>
    </div>
</div>

For an HTML file with structure like this, we can access the paragraph by doing:



'div[class=1] > div[class=2] > div[class=3] > p'

We can select the div with results:



//Finds the first div with class 'bkWMgd' and returns it
const parent = await page.$eval('div[class=bkWMgd]', result => result);

Since the parent variable represents a DOM node returned from page.$eval(), we can run HTML DOM methods on this object. Since all the information is available in the div with class g we can set the parent to its immediate child.



//Sets the parent to the div with all the information 
parent = parent.querySelector('div[class=g]');

With this we can now target the information we care about, this information can be seen in this image:

Title



//Targets h3 Website Title i.e. 'Cats  (2019 film)  - Wikipedia'
const title = parent.querySelector('div[class=rc] > div[class=r] > a >  h3').innerText;

URL



//Targets the <a> href link i.e. 'https://en.wikipedia.org/wiki/Cats_(2019_film)'
const url = parent.querySelector('div[class=rc] > div[class=r] > a').href;

Description



const desc = parent.querySelector('div[class=rc] > div[class=s] > div > span[class=st]').innerText;

Now that we know how to target our information we can add this to our file. We only looked at parsing information from one search result, but there are multiple search results so we need to use page.$$eval to target ALL divs with h2 Web results and target divs with class g, we can see here that some divs have multiple search results:

When there are multiple divs with class g they are nested in another div with class srg. Let's start adding all of this to our code so we can start putting all the pieces together. Please read this code carefully, it might seem confusing but it's based on the screenshot above.



//Find all div elements with class 'bkWMgd'
const searchResults = await page.$$eval('div[class=bkWMgd]', results => {
        //Array to hold all our results
        let data = [];

        //Iterate over all the results
        results.forEach(parent => {

            //Check if parent has h2 with text 'Web Results'
            const ele = parent.querySelector('h2');

            //If element with 'Web Results' Title is not found  then continue to next element
            if (ele === null) {
                return;
            }

            //Check if parent contains 1 div with class 'g' or contains many but nested in div with class 'srg'
            let gCount = parent.querySelectorAll('div[class=g]');

            //If there is no div with class 'g' that means there must be a group of 'g's in class 'srg'
            if (gCount.length === 0) {
                //Targets all the divs with class 'g' stored in div with class 'srg'
                gCount = parent.querySelectorAll('div[class=srg] > div[class=g]');
            }

            //Iterate over all the divs with class 'g'
            gCount.forEach(result => {
                //Target the title
                const title = result.querySelector('div[class=rc] > div[class=r] > a >  h3').innerText;

                //Target the url
                const url = result.querySelector('div[class=rc] > div[class=r] > a').href;

                //Target the description
                const desciption = result.querySelector('div[class=rc] > div[class=s] > div > span[class=st]').innerText;

                //Add to the return Array
                data.push({title, desciption, url});
            });
        });

        //Return the search results
        return data;
    });

The code above will parse the page and get us our results in an Array. We can now return that array from our main function searchGoogle:



const puppeteer = require('puppeteer');

const searchGoogle = async (searchQuery) => {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.goto('https://google.com');

    //Finds input element with name attribue 'q' and types searchQuery
    await page.type('input[name="q"]', searchQuery);

    //Finds an input with name 'btnK', after so it executes .click() DOM Method
    await page.$eval('input[name=btnK]', button => button.click());

    //Wait for one of the div classes to load
    await page.waitForSelector('div[id=search]');

    const searchResults = await page.$$eval('div[class=bkWMgd]', results => {
        //Array to hold all our results
        let data = [];
        ...
        ...
                //Return the search results
        return data;
    });

    await browser.close();

    return searchResults;
};

module.exports = searchGoogle;

We can now remove the last line, where we manually call the function. We are now finished with this Search Engine API! Now, all we need to do is to import this function in our main server.js file:



const express = require('express');
const app = express();
const port = 3000;

//Import puppeteer function
const searchGoogle = require('./searchGoogle');

//Catches requests made to localhost:3000/search
app.get('/search', (request, response) => {

    //Holds value of the query param 'searchquery'.
    const searchQuery = request.query.searchquery;

    //Do something when the searchQuery is not null.
    if (searchQuery != null) {

        searchGoogle(searchQuery)
            .then(results => {
                //Returns a 200 Status OK with Results JSON back to the client.
                response.status(200);
                response.json(results);
            });
    } else {
        response.end();
    }
});

//Catches requests made to localhost:3000/
app.get('/', (req, res) => res.send('Hello World!'));


//Initialises the express server on the port 30000
app.listen(port, () => console.log(`Example app listening on port ${port}!`));

Now if we start our server with npm start and go to our browser and browse to:



http://localhost:3000/search?searchquery=cats

We get a JSON! I'm using a JSON Viewer Chrome Extension to be able to view JSON in my browser

The code for this project can be found on Github

However, we are not done yet. At the moment, our API is ready but it's a bit slow. It's also currently running on our local machine, so we need to deploy it somewhere. This will all be covered in Part 3!

Part 3 will cover:

Optimizing and Improving Performance
Troubleshooting Basics
Deploying the API

This is the end of this post! I hope you enjoyed reading this and found this to be useful. Stay tuned for Part 3!

If you enjoyed reading this and would like to provide feedback, you can do so anonymously here. Any feedback regarding anything is appreciated!

Turn Any Webpage into your Personal API with Puppeteer

Waqasabi — Sat, 18 Jan 2020 23:44:21 +0000

In this 3-part-series we're going to learn how to convert any webpage into our personal API. We'll do this with an example of creating a Search API that uses Google Search to obtain its results. All of this will be done with Web Scraping with puppeteer

This is a 3 Part Series:

In the 1st Part: We'll go over the basics and create a Simple Web Scraper.
In the 2nd Part: We'll create a Search Engine API using Google Search.
In the 3rd Part: We'll go over different ways we can optimize our API, increase performance, troubleshooting basics and how we can deploy our puppeteer API to the web.

Table Of Contents - Part 1

Basics of Puppeteer
Creating a Simple Web Scraper

Basics of Puppeteer

We need to first understand what puppeteer is and what you can do with it.

What exactly is Puppeteer?

The definition according to official docs:

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

In simple words, puppeteer is a library that allows you to access the Chromium browser programmatically (a.k.a headless).

A headless browser is a great tool for automated testing and server environments where you don't need a visible UI shell. For example, you may want to run some tests against a real web page, create a PDF of it, or just inspect how the browser renders an URL.

The Chrome browser is very powerful and has a lot of features. Anything that is possible in Chrome is possible with puppeteer and this includes everything possible in DevTools. You can learn more about what you can do with DevTools here.

Here are some use-cases for puppeteer:

Generate screenshots and PDFs of pages.

Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).

Automate form submission, UI testing, keyboard input, etc.

Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.

Capture a timeline trace of your site to help diagnose performance issues.

Test Chrome Extensions.

However, in this post, we're going to use puppeteer exclusively for web-scraping.

Creating a Simple Puppeteer Scraper

To get started, we need to first initialize a directory and initialize npm (or yarn) using the command line:

mkdir hello-puppeteer
cd hello-puppeteer
npm init -y

We can then install the puppeteer library:

npm i puppeteer

To make sure that the library works as intended on all devices, the Chromium browser comes with the library. This ensures that the library is guaranteed to work and can avoid having to get the user to configure path and/or download /install Chrome.

The library downloads a recent version of Chromium (~170MB Mac, ~282MB Linux, ~280MB Win) that is guaranteed to work with the API.

For those of you interested, the puppeteer team is currently also working on an experimental version to bring Puppeteer to Firefox.

Since puppeteer is a Node.js library, we need to create a node.js file and run it with node. For the purpose of this post, we'll name it server.js:

touch server.js

To start our API, we need to configure the package.json file so that we can have node run our server file. We can do this by adding an npm start script in scripts:

{
  "name": "hello-puppeteer",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
     "start": "node server.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "puppeteer": "^2.0.0"
  }
}

We are now ready to write code to create a simple scraper in our server.js file:

const puppeteer = require('puppeteer');

(async () => {
    //Creates a Headless Browser Instance in the Background
    const browser = await puppeteer.launch();

    //Creates a Page Instance, similar to creating a new Tab
    const page = await browser.newPage();

    //Navigate the page to url
    await page.goto('https://example.com');

    //Closes the Browser Instance
    await browser.close();
})();

This creates an anonymous function that gets executed when we run npm start. It creates a Browser Instance, with a new page and navigates to https://example.com. Afterward, It closes the Browser Instance, and node finishes executing the file.

To make sure this is working as intended we can take a screenshot of the page after puppeteer is finished navigating to the page:

page.screenshot({path: 'example.png'});

After adding this to our file:

const puppeteer = require('puppeteer');

(async () => {
    //Creates a Headless Browser Instance in the Background
    const browser = await puppeteer.launch();

    //Creates a Page Instance, similar to creating a new Tab
    const page = await browser.newPage();

    //Navigate the page to url
    await page.goto('https://example.com');

    //Takes a screenshot of the page after navigating there and saves it as 'example.png'
    await page.screenshot({path: 'example.png'});

    //Closes the Browser Instance
    await browser.close();
})();

We can replace https://example.com with a working url. For this example, we'll use https://google.com. We can now run npm start and after a while, we can see example.png show up in our file directory, and opening it shows the homepage of Google.

We're almost done with our simple web scraper. We can now choose to get any information we want from Google's homepage. For now, we'll just get the image source for Google's Homepage Logo

This has no inherent purpose. However, the point is that we can access this information programmatically.

To do this, we need to go to our browser and navigate to the URL and find the element we're looking for by Inspect Elementing the page. You can Right-Click on the page and choose Inspect or you can open Dev-Tools directly and navigate to the HTML (Elements Tab).

After using the Mouse tool to highlight the Logo Element, this is what it points to (This might be different for you):

The important thing to look for is anything that can identify the HTML element. In our case, the img element has an id hplogo. So we can use this information to get access to the image source.

There are many different ways to get the specific element(s) from the DOM/page.

To target a single element we can use $eval method where we substitute the name of the id, class or any identifying attribute of the element we're looking for as the selector parameter.

page.$eval(selector, callbackFunc);

This method runs document.querySelector within the element and passes it as the first argument to callbackFunc. If there's no element matching selector, the method throws an error.

To target multiple elements we can use:

page.$$eval(selector, callbackFunc);

This method runs document.querySelectorAll within the element and passes it as the first argument to callbackFunc. If there's no element matching selector, the method throws an error.

If the element is found, it is passed as the first argument to the callback function and therefore we can use it to get the information we need.

const googleLogo = await page.$eval('#hplogo', (img) => img.src);

The targeting of an element is similar to how it is targeted in CSS or Javascript selectors

In our case since we only need to get a single image so we can use $eval and access the src directly:

const puppeteer = require('puppeteer');

(async () => {
    //Creates a Headless Browser Instance in the Background
    const browser = await puppeteer.launch();

    //Creates a Page Instance, similar to creating a new Tab
    const page = await browser.newPage();

    //Navigate the page to url
    await page.goto('https://google.com');

    //Takes screenshot of the page after navigating there
    await page.screenshot({path: 'example.png'});

    //Finds the first element with the id 'hplogo' and returns the source attribute of that element
    const googleLogo = await page.$eval('#hplogo', img => img.src);
    console.log(googleLogo);

    //Closes the Browser Instance
    await browser.close();
})();

After running npm start , The server console.logs:

https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png

If we open this URL in a browser tab, we can see that it's the Image we were looking for! Our simple web-scraper is now complete!

The code for this simple web scraper can be found on Github

In the next part, we'll create a Search Engine API using Google Search. The user will be able to request our API with a search query. Our API will then scrape Google and return the top 10 search results.

This is the end of Part 1. I hope you enjoyed reading this, and stay tuned for Part 2! Any feedback is appreciated!

Server-Side Rendered App with Next.js, React and Redux

Waqasabi — Sun, 12 Jan 2020 17:50:31 +0000

There are many known benefits to pre-rendering your web application or rendering your web app on the server-side, some of which include better SEO, faster-load times, a better user experience for users with poor connections and many more.

This post will guide you towards getting quickly started with Next and using it to develop a React-Redux web application.

Note: This post is outdated and was written for Next.js versions 9.2 and lower. Please refer to next-redux-wrapper to see how to create an updated version Next.js-redux web app.

Prerequisites for this post:

Understanding of Basic React Concepts
Understanding of Basic Redux Concepts

For the purpose of this post, after going through all the concepts, we will be creating a simple counter app with our server-side rendered app.

Getting Started with Next.JS

Next.js is a React-Framework which makes it really easy to develop react server-side rendered apps. It also provides additional features but in this post, we will only go over rendering applications server-side with Next.js.

I highly recommend going over the docs. This part goes over the basic principles of next and there is a lot of overlap with the documentation. I recommend going through the documentation and then proceeding to the next part of this article. Nevertheless, if the documentation are not sufficient, then you can keep reading!

If you already have Next.js installed and know the basics, you can skip to the following

To get started, we first create a project directory:

mkdir hello-next

We then initialize the project with npm:

cd hello-next
npm init -y

We then need to install next, react & react-dom, these are necessary dependencies for next:

npm install --save react react-dom next

We then need to create a 'pages' directory within our project directory. Any React files in this directory by default are mapped to the url routes based on the file-name for our server-side rendered app:

mkdir pages

A file named index.jsx will be mapped to the root url i.e. localhost:3000/.
Similarily a file named login.jsx will be mapped to localhost:3000/login
This feature is enabled by default and for our use-case, this is sufficient.

To get started with next, we need to edit our package.json in our project directory and replace the scripts with the following:

"scripts": {
  "dev": "next",
  "build": "next build",
  "start": "next start"
}

After doing so, everything is ready. You can now run this command in the project directory:

npm run dev

After a few seconds, the development server should be up and running and visiting localhost:3000 will yield "404 | Page Not Found". This is because our pages directory does not have an "index.jsx" file. We can proceed to create it:

touch pages/index.jsx

We can then create a simple Hello-World index page:

import React from 'react';

class App extends React.Component {
    render() {
        return (
            <h1> Hello World! </h1>
        );
    }
}

export default App;

Here we create a React Component that renders "Hello World" and visiting the root path will show the result.

Next only recognizes the default imports in the React files in the pages directory, and will only render the default Component when browsed to the URL path.

Creating a Simple Counter App (Without Redux)

To create a simple counter app, we can do the following:

import React from 'react';

class App extends React.Component {

    constructor(props) {
        super(props);

        //Initialise state
        this.state = {
            counter: 0
        };
    }

    //Updates the counter in state by +1
    incrementCounter = () => {
        this.setState(prevState => {
            this.setState({
                counter: prevState.counter + 1.
            });
        });
    };

    //Updates the counter in state by  -1
    decrementCounter = () => {
        this.setState(prevState => {
            this.setState({
                counter: prevState.counter - 1.
            });
        });
    };

    render() {
        return (
            <div>
                <button onClick={this.incrementCounter}>Increment</button>
                <button onClick={this.decrementCounter}>Decrement</button>
                <h1>{this.state.counter}</h1>
            </div>
        );
    }
}

export default App;

Doing so will show this result:

Clicking on the appropriate buttons increment and decrements.

As you can see, next makes use of React and therefore working with next is simple just working with React, the only difference is that next automatically (Behind-the-scenes) renders the application serverside.

Understanding Server-Side Rendering with Redux

Similar to how working with Next is basically just working with React. Redux web apps behave in the same way. Everything works similar to how it would work if the app was rendered on the client-side. The only challenge working with Redux is the initial Redux setup with server-side rendering and this is exactly what the following part covers.

The official documentation, Redux provides a good explanation of how server-side rendering is expected to work with Redux. The explanation states that:

When using Redux with server rendering, we must also send the state of our app along in our response, so the client can use it as the initial state. This is important because, if we preload any data before generating the HTML, we want the client to also have access to this data. Otherwise, the markup generated on the client won't match the server markup, and the client would have to load the data again.

To send the data down to the client, we need to:

create a fresh, new Redux store instance on every request;

optionally dispatch some actions;

pull the state out of store;

and then pass the state along to the client.

On the client-side, a new Redux store will be created and initialized with the state provided by the server. Redux's only job on the server-side is to provide the initial state of our app.

This might seem confusing but the important part is:

Initialize and create a new redux store for new user request
(Optional) populate the store with information, for example, you could make use of the user-cookies in the request to identify the user and populate the store with the user information.
Send the redux state to the client
The client then uses the received state to initialize the client-side redux store.

The next part will cover how we can achieve this.

Setting-up Redux

To get started with Redux, we will create a basic Redux app where we keep track of the counter in our state.

We need to first install redux and react-redux:

npm install --save redux react-redux

This is how our project structure will look like:

hello-next
    +- .next
    +- node_modules
    +- pages
    +- redux
        +- actions
        +- reducers

To do so we can do the following:

mkdir redux redux/actions redux/reducers

We will now create a counterReducer, which will keep track of our counter state. We can place this in the reducers folder:

touch redux/reducers/counterReducer.js

This is how the counterReducer file will look like:

const counterReducer = (state = {value: 0}, action) => {
    return {...state};
};

export default counterReducer;

This will create an initial state with the counter value being set to 0

Right now our counterReducer does nothing. We can proceed to creating actions:

touch redux/actions/counterActions.js

We will just specify two actions - Increment and Decrement:

//Action Types
export const INCREMENT_COUNTER = "INCREMENT_COUNTER";
export const DECREMENT_COUNTER = "DECREMENT_COUNTER";


//Action Creator
export const incrementCounter = () => ({
   type: INCREMENT_COUNTER
});

export const decrementCounter = () => ({
    type: DECREMENT_COUNTER
});

We can now modify our reducer to include these actions:

import {DECREMENT_COUNTER, INCREMENT_COUNTER} from '../actions/counterActions';

const counterReducer = (state = {value: 0}, action) => {
    switch (action.type) {
        case INCREMENT_COUNTER:
            return {...state, value: state.value + 1};
        case DECREMENT_COUNTER:
            return {...state, value: state.value - 1};
        default:
            return {...state};
    }
};

export default counterReducer;

This will either increment or decrement our counter when INCREMENT_COUNTER or DECREMENT_COUNTER actions are dispatched.

We can now proceed to creating the root reducer, which will be responsible for combining all our reducers. In our case we only have 1 reducer "counterReducer", however for common practice we will proceeding to combining reducers.

Create the rootReducer file:

touch redux/reducers/rootReducer.js

This is how our rootReducer file will look like:

import counterReducer from './counterReducer';
import {combineReducers} from 'redux';

const rootReducer = combineReducers({
    counter: counterReducer
});

export default rootReducer;

This combines all our reducers into one rootReducer which we can use to initalise our redux store.

We can now proceed to creating our redux store:

touch redux/store.js

import {createStore} from 'redux';
import rootReducer from './reducers/rootReducer';

const store = createStore(rootReducer);

export default store;

Now that we have our redux logic setup, we can link our application with redux, using react-redux. However to do this, we need to create a special file named "_app.jsx" located in our pages directory:

touch pages/_app.jsx

next uses the App component to initialize pages. We created the "_app.jsx" file to override the default App Component. To begin with, our new App Component needs to extend the default App component, so that next can still use it to initalise pages.

We can import the default App Component from "next/app" and create our own App component:

import App from 'next/app';

class MyApp extends App {


}

export default MyApp;

However, at this moment we are doing nothing. Similar to how Redux is connected to Client-side react apps, we can connect our server-side rendered application here.

We use "Provider" provided by react-redux and connect our store:

import App from 'next/app';
import {Provider} from 'react-redux';
import React from 'react';

class MyApp extends App {

    render() {
        return (
            <Provider store={}>

            </Provider>
        );
    }

}

export default MyApp;

But what do we put as the argument for store inside the Provider Component? To finish the setup we must use a static function getInitialProps. This function according to the next docs is responsible for:

Next.js comes with getInitialProps, which is an async function that can be added to any page as a static method.

getInitialPropsallows the page to wait for data before rendering starts.

Using getInitialProps will make the page opt-in to on-demand server-side rendering.

Every page that has getInitialProps will be server-side rendered. If you do not include this method then the file will be rendered to static HTML at next build time. Including this function will allow this page to render on the server, and everything inside that function will be executed before sending the page to the client. This is helpful in cases where our page needs data that needs to be fetched. Returning anything from this function will allow that information to be sent to the client. The client can access the information returned from this function using the props of the React component.

This is also where we can choose to optionally populate our redux state before sending it to the client, adding this function to our "_app.jsx" looks like:

import App from 'next/app';
import {Provider} from 'react-redux';
import React from 'react';
import {INCREMENT_COUNTER} from '../redux/actions/counterActions';

class MyApp extends App {

    static async getInitialProps({Component, ctx}) {
        const pageProps = Component.getInitialProps ? await Component.getInitialProps(ctx) : {};

        //Anything returned here can be access by the client
        return {pageProps: pageProps};
    }

    render() {
        //Information that was returned  from 'getInitialProps' are stored in the props i.e. pageProps
        const {Component, pageProps} = this.props;

        return (
            <Provider store={}>
                <Component {...pageProps}/>
            </Provider>
        );
    }

}

export default MyApp;

ctx is a getInitialProps parameter referring to Context. You can read more about it here

Using getInitialProps in _app.jsx has a different interface. When using it on normal pages, getInitialProps only has 1 parameter ctx. However in our case, since we are overriding the default App Component, we have access to the default App Component. We need to make sure if the default App component makes use of getInitialProps then we need to send whatever that function returned to the client.

Moving on, to pass the store to the client, we need to wrap the original component with React-Redux's Provider. To make all of this work, we need to install one last library: next-redux-wrapper

npm install --save next-redux-wrapper

Next-redux-wrapper will enable us to create a store at every new request and it will pass it to MyApp (Our App Implementation) as props.

We need to make use of Next-redux-wrapper's withRedux wrapper and wrap our App component with it.

The withRedux function accepts makeStore as first argument. The makeStore function will receive initial state and should return a new instance of Redux store each time when called, no memoization needed here, it is automatically done inside the wrapper.

After connecting with next-redux-wrapper:

import App from 'next/app';
import {Provider} from 'react-redux';
import React from 'react';
import withRedux from "next-redux-wrapper";
import store from '../redux/store';

class MyApp extends App {

    static async getInitialProps({Component, ctx}) {
        const pageProps = Component.getInitialProps ? await Component.getInitialProps(ctx) : {};

        //Anything returned here can be accessed by the client
        return {pageProps: pageProps};
    }

    render() {
        //pageProps that were returned  from 'getInitialProps' are stored in the props i.e. pageprops
        const {Component, pageProps, store} = this.props;

        return (
            <Provider store={store}>
                <Component {...pageProps}/>
            </Provider>
        );
    }
}

//makeStore function that returns a new store for every request
const makeStore = () => store;

//withRedux wrapper that passes the store to the App Component
export default withRedux(makeStore)(MyApp);

After the following the changes, our app is ready! We can now use Redux as we normally would. Changing our index.jsx to incorporate redux.

import React from 'react';
import {connect} from 'react-redux';
import {decrementCounter, incrementCounter} from '../redux/actions/counterActions';

class App extends React.Component {

        static getInitialProps({store}) {}

    constructor(props) {
        super(props);
    }

    render() {
        return (
            <div>
                <button onClick={this.props.incrementCounter}>Increment</button>
                <button onClick={this.props.decrementCounter}>Decrement</button>
                <h1>{this.props.counter}</h1>
            </div>
        );
    }
}

const mapStateToProps = state => ({
    counter: state.counter.value
});

const mapDispatchToProps = {
    incrementCounter: incrementCounter,
    decrementCounter: decrementCounter,
};

export default connect(mapStateToProps, mapDispatchToProps)(App);

We use React-Redux's connect to connect the Redux state to our page, and we use mapStateToProps and mapDispatchToProps to connect our state and actionCreators to our page.

After running the page, our React-Redux app works as expected! Clicking on the buttons Increments and/or Decrements!

Congratulations, you now know the basics of how to create a Server-side Rendered React-Redux Application using next.js

One thing to note is that at the moment Redux only works as a Single Page Application, what this means is that Client-side routing is the only way for the Redux store to be transferred between pages.

This means that if the user navigates to a different URL (i.e. Server-side routing) then the server will treat it as a new client and serve an empty redux state. To learn how to persist the redux-state so that the counter values stays the same for each refresh refer to next-redux-wrapper guide. However please ensure that you update your Next.js version and the next-redux-wrapper version first and follow the updated guide.

The code for this project can be found on Github

This is the end of this post! This was my first post and I hope you enjoyed reading it! Any feedback is appreciated!

If you would like to read more yourself refer to next-redux-wrapper repository