DEV Community: Rohit Dalal

Getting Started With Puppeteer

Rohit Dalal — Thu, 13 Aug 2020 14:31:15 +0000

In this post, I will try to walk you through the basics of Puppeteer, a browser automation library for Node.js. Puppeteer is created and maintained by Google Chrome and it's the de-facto standard when it comes for browser automation in JavaScript.

Let's get started with this post 🚀.

What is Puppeteer?

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

This is the definition of Puppeteer' official Website. Simply stated, it is a headless browser API which gives you the ability to run Chrome or Chromium browser automatically based on the code you wrote to automate it. Now, you will ask "What is a headless browser?". So, the headless browser is a browser without GUI. You can also run Puppeteer in non-headless( GUI ) mode (as given in the above definition). More on that further.

It can do various things for you and some of them are listed below:

Web Scrapping
Take a screenshot of the page
Generate PDF's of the page/s
Automate certain repetitive tasks
... and many more.

Let's see how to install this awesome package now!

Installation

There are two ways to install this library in your machine.

The standard way (Library + Browser):

If you install this way, it will download a new version of Chromium Browser in your project directory of size ~180MB. This download will definitely take time and depends on your internet speed. After installing, you don't need to do any custom settings in order to run the code. Puppeteer will register the locally installed browser in your pwd as default to run any code involving Puppeteer.

npm install --save puppeteer

Well, what if you don't wanna download this ~180MB browser? There's the next step for this.

The short way (Only Library):

This is the short and less in size solution to avoid the browser download. This will only install the core package (~3MB), not the browser. If you do this way, you must have a working version of Chrome or Chrome Canary browser installed in your machine which you use for daily purposes which you can use for Puppeteer by passing additional info while writing code specifying the path of the Chrome installation. (We will see this later in the post. Don't worry!)

npm install --save puppeteer-core

Note

Please note that puppeteer-core works only for development purpose. If you want to deploy such application to the web, you must use the complete package because the local path you specify while developing will be invalid in production.

If you want to read more on puppeteer vs puppeteer-core, here is the link

Now that we have completed the installation, let's write some code using this library.

Setup

In this post, we will see two working examples using Puppeteer.

Scrapping Google Search Results
Take a screenshot of any Webpage

To get up and running for this demo, create a new Node.js project by typing

npm run init

After initialization, you can install the package by either of the above-mentioned ways. If you are using the short way, there is only one place where you have to make changes to the code. That will be clear as we see in action.

Grab some coffee and let us see the examples in action.

Scrapping Google Search Results

Now, here we will be scrapping search results for any query of your choice from Google. We will store the scrapped results in an array of objects. The actual application may require DB access after scrapping. I leave that up to you.

Firstly, we import puppeteer from puppeteer-core and then we create a browser object with puppeteer.launch() passing it launchOptions , which is an object containing optional parameters. I have used async/await while writing this code. If you want to use .then() , you can use that as well, it is basically a way to handle the returned Promise.

Description of the used launchOptions properties:

headless : Whether to open Puppeteer in headless mode or not? The default value is true.
defaultViewport : An object with width and height properties, which depicts its purpose itself.
executablePath : Path of Chrome/ Chrome Canary/ Chromium installed in your machine. Here is an easy guide on how to find that path. You should use this property only if you are using puppeteer-core. Double "\" denotes character escaping.

You can find a detailed list of arguments here.

After this, we create a new page using browser.newPage(), which opens a new tab in the launched browser and navigates to https://www.google.com/search?q=coffee to scrape search results from. Upon successful page load, we grab the page content using page.content(). If you try to print the scraped content at this point, you will see the entire page source in the console, but we are interested in only the search title and the associated link with the result. For that, we shall use a separate package named cheerio. Cheerio is a package which can parse and do all the things with the page-source at back-end/ server which jQuery does on the front-end.

Cheerios is a Fast, flexible, and lean implementation of core jQuery designed specifically for the server.

We parse the content using cheerio and store it in a variable $(used to show similarity with jQuery ). A div with class 'r' is a container for both, the search title and the actual link of one result. We then loop over all the "divs" elements with class='.r' to get the title, which is an "h3" heading with class="LC20lbDKV0Md".Now, grab the link from the children anchor tag of the parent div using the "href " property with .attr('href') and then push the {title, link} to the links array and here we finish the process by closing the tab and the browser.

Here is the full working code for the same:

//scrapeGoogle.js
const puppeteer = require('puppeteer-core')
const cheerio = require('cheerio')

const run = async () => {
  let launchOptions = {
    headless: false, //to see the execution as it happens
    executablePath:
      'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
  }

  let browser = await puppeteer.launch(launchOptions)
  let page = await browser.newPage()

  try {
    await page.goto('https://www.google.com/search?q=coffee', {
      waitUntil: 'domcontentloaded',
    })
  } catch (err) {
    if (err instanceof puppeteer.errors.TimeoutError) {
      throw new Error(err)
      await browser.close()
    }
  }

  let content = await page.content()
  //cheerio
  let $ = cheerio.load(content)

  var links = []
  $('.r').each(function (i, el) {
    var title = $(this).find('.LC20lb').text()
    var link = $(this).children('a').attr('href')
    if (title.length > 0 && link.length > 0) {
      links.push({ title, link })
    }
  })
  console.log(links)

  await page.close()

  await browser.close()
}

run()

In this way, we have successfully scrapped Google search results using Puppeteer. You can improve this further by adding more and more features and scrapping more data. We completed the first example here.

Taking a screenshot of any Webpage

Now, this section will be very similar as above, except content scrapping. We take the screenshot with page.screenshot() which returns a Promise and on its successful resolution, our image will be saved in the folder path you specify.

//screenshot.js
const ss = async () => {
  let launchOptions = {
    headless: false,
    executablePath:
      'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
    defaultViewport: {
      width: 1536,
      height: 763,
    },
  }

  let browser = await puppeteer.launch(launchOptions)
  let page = await browser.newPage()

  try {
    await page.goto('https://www.google.com/search?q=chelsea', {
      waitUntil: 'domcontentloaded',
    })
  } catch (err) {
    if (err instanceof puppeteer.errors.TimeoutError) {
      throw new Error(err)
      await browser.close()
    }
  }

  //main line
  await page.screenshot({ path: 'screenshot.png' })

  await page.close()
  await browser.close()
}

ss()

As said, everything is the same here except just one line were to take the screenshot and save it with name 'screenshot.png'. {path:"your_path"} is necessary, without which it will not save the screenshot.

Conclusion

Hooray, that's it for this post guys. If you have any queries regarding this post, feel free to contact me personally. If you liked this post, share it with your developer friends and social media.

Thank you. See you next time ;)

How to connect to MongoDB Atlas using Node.js

Rohit Dalal — Thu, 23 Jul 2020 18:48:35 +0000

In this post, I will tell you how you can connect to MongoDB Atlas, a cloud-based DB service, from your local Node.js / Express.js environment.

You can use Atlas for any project which needs to be deployed, whether for development purpose or production environment.

Let's get started!

Step 1: Register/Login and create your plan

Create your account at Atlas and register yourself to their service. It's free of cost. If you already have an account, go ahead. If you logging in for the first time, you will need to choose a plan for your account. There are 3 types to choose from, for testing and learning purpose, "FREE PLAN" will suit your needs perfectly. After doing all basic registration and setup, it's time to create a Cluster.

Step 2: Create your Cluster

After Registration/Login and choosing your plan, let's get our Cluster setup done.

If you have no created clusters, you will see the below page on your dashboard.

Upon clicking "Build a cluster", you will be given with the following options:

Here, on this page, you need to select Cloud Provider, and your Region. Upon selecting those, you can leave all other settings to their default and can go ahead. If you want to change the name of your Cluster, you have to do it in this step only. Name cannot be changed once the cluster is created.

Completing all the above steps is what you need to create a Cluster. Now, MongoDB will configure and create your cluster and this process will take about 5-10 minutes (pretty long tbh). Come back to the same page once all is finished.

We will now look at how to create and configure database collections and integrate it with our express server

Step 3: Add User and whitelist IP address

Till this step, we have successfully created our Cluster. Now, its time to add Database User and IP address of your current machine.

Add database user Go to Database access under SECURITY and click on "Add New Database User". Fill out the username and password, leaving all settings to default and make sure you remember your credentials. We will need it later on to create a connection string (URI) of our DB.

Whitelist your IP address Now, the next step is to Whitelist your IP address so it can recognize your machine for regular access. To whitelist your IP address, go to SECURITY > Network Access and click on "Add IP Address". Upon clicking that, you will be given with the following page. Click on Allow access from anywhere which sets it to global access. Don't worry, its not any danger.

Now, we've all set up done to use it to our Node.js server. It's time for some code.

Step 4: Connect to your database

In this step, we will connect our server to the database. To do this, go to the main page of your cluster and click on "Connect" which will give you a modal like this:

Select "Connect your application". On clicking, you have to select "Node.js" in the next page and save the connection string provided by Atlas somewhere for further use.

Now, its time to create our database connection using mongoose. Install mongoose in your project by npm install mongoose. Copy-paste the following code to db.js.

//db.js

const mongoose = require('mongoose')

const url = `mongodb+srv://sample_user:<password>@my-sample-cluster-b3ugy.mongodb.net/<dbname>?retryWrites=true&w=majority`;

const connectionParams={
    useNewUrlParser: true,
    useCreateIndex: true,
    useUnifiedTopology: true 
}
mongoose.connect(url,connectionParams)
    .then( () => {
        console.log('Connected to database ')
    })
    .catch( (err) => {
        console.error(`Error connecting to the database. \n${err}`);
    })

Replace "password" with your user password and also the "dbname" with the name you'd like to have for your database. Run the following code by typing node db.js and you will see Connected to the database in your console. You can take this to the next step by configuring your DB model by creating a "Collection" with various fields and connecting it with the express server. I leave it up to you to use according to your need. This was the basic setup needed to get Atlas up and running.

Conclusion

Hooray, this way we have successfully connected to Atlas using Node.js. If you have any doubts, feel free contact me and I will try to resolve it on a personal basis.

Thank you.

Live Stream your Webcam to HTML Page

Rohit Dalal — Thu, 09 Jul 2020 18:14:34 +0000

Ever wondered how you can stream your own webcam recording live on HTML page?

Well, in this quick and easy post, we will have a look at how you can live to stream your webcam straight to your HTML page.

Let's get started.

Setup

HTML required for this short demo is a one-liner. Create a file index.html and paste the following:

<video id="myVidPlayer" controls muted autoplay></video>

This embeds a Simple HTML5 video player to your page. controls muted autoplay are optional parameters to video tag, which clearly describe their purposes.

Now, it's time to add some life to the video player. This code will Stream your Webcam recording to the created video player

<script type="text/javascript">
    //Selector for your <video> element
    const video = document.querySelector('#myVidPlayer');

    //Core
    window.navigator.mediaDevices.getUserMedia({ video: true })
        .then(stream => {
            video.srcObject = stream;
            video.onloadedmetadata = (e) => {
                video.play();
            };
        })
        .catch( () => {
            alert('You have give browser the permission to run Webcam and mic ;( ');
        });

</script>

Here, getUserMedia() is a function which needs an object as a parameter with either of audio/video as a compulsory property set to true. Else, you will get an error. This function then returns a Promise.

Upon running this code, the browser will ask for our permission for the usage of Microphone and Camera. If you "allow" it to use, then our Promise resolves with "stream" as a parameter which we assign it to video.srcObject. This sets the stage up for playing the video. Upon successful loading of metadata of our video player (video.onloadmetadata), we start the actual streaming with video.play() and you will see your "beautiful" face right there on your HTML page, else it is rejected with an alert box popup.

Bonus: Snapshot of the current video frame

Now, we will add a button which will get the snapshot of the current video frame on click and will display it to you in a canvas element.

<button onclick="snapshot()" >Snapshot</button>

<div class="mycanvas">
    <h6>Captured snapshot</h6>
    <canvas></canvas>
</div>

Let's rewrite our complete JavaScript

<script type="text/javascript">
    var canvas = document.querySelector("canvas");
    var context = canvas.getContext("2d");
    const video = document.querySelector('#myVidPlayer');

    //w-width,h-height
    var w, h;
    canvas.style.display = "none";

    //new
    function snapshot(){
        context.fillRect(0, 0, w, h);
        context.drawImage(video, 0, 0, w, h);
        canvas.style.display = "block";
    }

    window.navigator.mediaDevices.getUserMedia({ video: true, audio: true })
        .then(stream => {
            video.srcObject = stream;
            video.onloadedmetadata = (e) => {
                video.play();

                //new
                w = video.videoWidth;
                h = video.videoHeight

                canvas.width = w;
                canvas.height = h;
            };
        })
        .catch(error => {
            alert('You have to enable the mike and the camera');
        });
</script>

On clicking the button, canvas context is filled with a 2D rectangle with w=width and h=height. (0,0) is passed to tell that we want to fill the context from the left top corner of the canvas. Then, we attach our video frame snapshot to the canvas and make it visible with display: "block" which was hidden by default. More info on Canvas API is here.

Here is the Code pen link Codepen.

Conclusion

This was it for this short and sweet tutorial. See you next time ;)

Sticky navbar from scratch using react

Rohit Dalal — Sun, 21 Jun 2020 09:29:19 +0000

In many modern websites, you must have seen this effect of Navigation bar/ Header sticking to the top of the page even if you scroll down the page for your better navigation so you don’t have to go up always to choose between links for moving between the pages. Haven’t you?
If you have and are curious about how does that work, then this post is your go-to guide to create that very simple yet beautiful effect/feature for you next project which adds better UX to the page.
What are you waiting for? Let’s get started 🚀

Initial Setup

First, we are going to create a react-app from start. For that enter this command in your favourite terminal



create-react-app sticky-navbar

We are going to delete some of the files created out of the box from create-react-app. After deleting some files, project directory will look like this:

We will start by creating components, two components will suffice for this little project. Create a directory called Components in /src and add two files Navbar.js and Content.js
Start by editing App.js. Add the following code to start with a basic layout



//App.js
import React from 'react';
import Navbar from './components/Navbar';
import Content from './components/Content';
function App() {
  return (
    <div className="App">
      <Navbar />
      <Content />
    </div>
  );
}
export default App;

Here we are rendering two functional components Navbar and Content, their name clearly explains their purpose.
This below code fills up Navbar.js. Navbar component is basically a header section which consists of your website Logo and one navigation menu with 4/5 links



//Navbar.js
import React from 'react';
import './navbar.scss';
import Logo from './../img/logo.svg';
const Navbar=() => {
  return (
    <header className={navbarClasses.join(" ")}>

        <div className="logo">
            {/* your logo */}
        </div>
        <nav className="navigation">
            {/* your navigation */}
        </nav>

    </header>
  )
};
export default Navbar;

Now, it’s time to give some styling to the page. You can have your own set of styling or if you just want to use mine, copy below given styles to navbar.scss



//navbar.scss
.navbar{
  width: 100%;
  min-height: 6vh;
  box-sizing: border-box;
  display: flex;
  justify-content: space-between;
  align-items: center;
  background-color: #eee;
  transition: all .7s ease-in;
}
//IMPORTANT
.scrolled{
  position: fixed;
  top: 0;
  left: 0;
  background-color: lightblue;
}
.logo img{
  width: 50px;
}
.navigation{
    //your navigation styles
}

Now, we would fill up Content.js with some meat to add some content on the page along with the Navbar.
This code fills up dummy boxes to take ample amount of height on-page. You can use any real-life content. This is just for demo



//Content.js
import React from 'react';
import './content.scss';
const Content=() => {
  const data=(
    <div className="box">
        <h2>My div content </h2>
    </div>
  );
return (
    <main className="content">
        {data}
        {data}
        {data}
        {data}
        {data}
    </main>
  )
}
export default Content;

Style for Content.js



//content.scss
.content{
    width: 80%;
    margin:2rem 10% 0 10%;
    max-width: 100%;
    box-sizing: border-box;
    min-height: 100vh;
}
.box{
    width:80%;
    margin:2rem 10% 0 10%;
    height:100vh;
    background-color:plum;
}

After all this setup, our page will look like this:

Main Logic

Now that we have our meat set up for the Sticky navbar, we will add the main Logic to get that sticky effect. For this, we are going to do a little bit of state-management using React hooks. If you are not aware of react-hooks, don’t worry about it. You can use normal state management with classful component too.
So, the idea is as you scroll down, you check for the scrollY (i.e. total height you have scrolled from the top in pixels ) in componentDidMount() or useEffect() using window.scrollY and if that scrolled height is greater than some value, say 200px in this case, then we change the class of our Navbar to navbar scrolled. Styles for .scrolled is already in your navbar.scss if you have copied my styles. I have used array approach to switch between classes from navbar to navbar scrolled.

Confused?



//Navbar.js
import React,{useEffect} from 'react';
import './navbar.scss';
const Navbar=() => {
  const [scrolled,setScrolled]=React.useState(false);
const handleScroll=() => {
    const offset=window.scrollY;
    if(offset > 200 ){
      setScrolled(true);
    }
    else{
      setScrolled(false);
    }
  }

  useEffect(() => {
    window.addEventListener('scroll',handleScroll)
  })
let navbarClasses=['navbar'];
  if(scrolled){
    navbarClasses.push('scrolled');
  }

   return (
    /* rest remains same*/
   )
};
export default Navbar;

In this way, we have successfully created our simple sticky Navbar from scratch.
If you want to look at the complete code, check my GitHub repo.

Conclusion

This was all for this simple demo application. I hope this post was helpful to you and I’ve solved your problem by explaining it in your words. If you have any queries regarding this post, feel free to ask me personally. I will be happy to answer those on an individual level.
Thank you. Have a nice day!