Hello folks, today I will be sharing information on Web Scraping. Web scraping is simply the process of extracting content and data from a website. This post is only for Educational purposeโ
Prerequisite
๐จโ๐ป Nodejs
๐จโ๐ป Developer Tool knowledge (DevTools)
๐จโ๐ป Document Object Model knowledge
Let's Start
๐ฅฆ Make a new directory in my case nodescraping and initiate a node js app
npm init -y
๐ฏ Result: Creates your package.json file 
๐ฅฆ Install dependencies
npm i express axios cheerio
๐ฏ Result:
๐ฅฆ Install Dev dependencies (for development purposes), nodemon restarts our node app automatically when files change..
npm i nodemon -save-dev
๐ฏ Result:
๐ฅฆ Edit your start script
"start": "node app.js ",
"dev:": "nodemon app.js"
๐ฏ Result:
๐ฅฆ create a file app.js and import the packages
const axios = require('axios');
const cheerio = require('cheerio');
const express = require('express');
const port = process.env.PORT || 4000;
const app = express();
๐ฅฆ I will be using the axios package to fetch the website. I will be using a site called Dev.to๐. Be at liberty to use any website of your choice. We will be scraping and exporting our result into a plain text file CSV.
๐ฅฆ Right-click, to inspect the website to select elements (class, is) and their respective attributes (a, li).
๐ฏ This gives us the ability to inspect the classes we want to select.
๐ฅฆ I want to target the following. Blog title, link, author, and read the time.
Side note:
Always use . before the class name you want to target.
axios.get('https://dev.to/')
.then(res => {
const $ = cheerio.load(res.data)
$('.crayons-story').each((index, element) => {
const blogTitle = $(element).find('.crayons-story__title').text()
}).catch(err => console.error(err))
In the logic above, i am target the child element of the class crayons-story.
The .text() method is converting the result to text.
๐ฅฆ I repeated the whole process to select the Blog link, author, and read the time.
๐ฅฆ Final logic is:
const axios = require('axios');
const cheerio = require('cheerio');
const express = require('express');
require('dotenv').config();
const fs = require('fs');
const writeStream = fs.createWriteStream('devBlog.csv');
const port = process.env.PORT || 4000;
const app = express();
//write headers
writeStream.write(`author, BlogTitle, bloglink, readtime \n`);
axios.get('https://dev.to/')
.then(res => {
const $ = cheerio.load(res.data)
$('.crayons-story').each((index, element) => {
const author = $(element).find('.profile-preview-card__trigger').text().replace(/\s\s+/g, '')
const blogTitle = $(element).find('.crayons-story__title').text().replace(/\s\s+/g, '')
const blogLink = $(element).find('a').attr('href');
const readTime = $(element).find('.crayons-story__tertiary').text()
const dev = 'https://dev.to'
const joinedBlogLink = `${dev}` + `${blogLink}`;
writeStream.write(`Author: ${author}, \n Blog title is : ${blogTitle} ,\n Blog link: ${joinedBlogLink}, \n Blog read time : ${readTime} \n`);
});
}).catch(err => console.error(err))
//Listen to server
app.listen(port, () => {
console.log(`Server Established and running on Port โก${port}`)
})
View source code here: here
Notes and explanation
- fs module was used to write the final result into the devBlog.csv file
- \n equals to a new line
- .replace(/\s\s+/g, '') removes white spaces between the author's field.
- axios fetch markup data from the URL
- cheerio grabs the html data from the URL. Cheerio is a tool for parsing HTML and XML in Node.js.
- the cheerio.load method loads the website mark up and stores the value in the declared variable, in my case
$ -
.eachmethod loops through the selected elements.
๐ฅฆ Run server npm run dev
๐ฏ Result:
author, BlogTitle, bloglink, readtime
Author: Gracie Gregory (she/her),
The blog title is : What was your win this week? ,
Blog link: https://dev.to/devteam/what-was-your-win-this-week-5h25,
Blog read time : for Oct 8
1 min read
Author: Jeremy Friesen,
Blog title is : Trick or Treat, I've Joined the DEV Team ,
Blog link: https://dev.to/jeremyf/trick-or-treat-i-ve-joined-the-dev-team-4283,
Blog read time : Oct 8
5 min read
Author: Michael,
Blog title is : How To See Which Branch Your Teammate Is On In Android Studio ,
Blog link: https://dev.to/gitlive/how-to-see-which-branch-your-teammate-is-on-in-android-studio-2n3i,
Blog read time : for Oct 8
1 min read
Author: Iain Freestone,
Blog title is : ๐10 Trending projects on GitHub for web developers - 8th October 2021 ,
Blog link: https://dev.to/iainfreestone/10-trending-projects-on-github-for-web-developers-8th-october-2021-102e,
Blog read time : Oct 8
3 min read
Author: AM,
Blog title is : Django Cloud Task Queue ,
Blog link: https://dev.to/txiocoder/django-cloud-task-queue-27g2,
Blog read time : Oct 8
1 min read
Author: Ankit Anand โจ,
Blog title is : AWS X-Ray vs Jaeger - key features, differences and alternatives ,
Blog link: https://dev.to/signoz/aws-x-ray-vs-jaeger-key-features-differences-and-alternatives-322,
Blog read time : for Oct 8
6 min read
Author: Raquel Romรกn-Rodriguez,
Blog title is : Algo Logging: the Longest Substring of Unique Characters in JavaScript ,
Blog link: https://dev.to/raquii/algo-logging-the-longest-substring-of-unique-characters-in-javascript-4i3,
Blog read time : Oct 8
3 min read
Author: Shaher Shamroukh,
Blog title is : Working With Folders & Files In Ruby ,
Blog link: https://dev.to/shahershamroukh/working-with-folders-files-in-ruby-2l97,
Blog read time : Oct 8
3 min read
Author: Roberto Ruiz,
Blog title is : Untangling Your Logic Using State Machines ,
Blog link: https://dev.to/robruizr/untangling-your-logic-using-state-machines-2epj,
Blog read time : Oct 8
5 min read
Author: Cubite,
Blog title is : How To Manage Open edXยฎ Environment Variables Using Doppler and Automating The Deployment ,
Blog link: https://dev.to/corpcubite/how-to-manage-open-edx-environment-variables-using-doppler-and-automating-the-deployment-4c5e,
Blog read time : Oct 8
5 min read
Author: OpenReplay Tech Blog,
Blog title is : Building an Astro Website with WordPress as a Headless CMS ,
Blog link: https://dev.to/asayerio_techblog/building-an-astro-website-with-wordpress-as-a-headless-cms-47mo,
Blog read time : Oct 8
9 min read
Author: Anamika,
Blog title is : How to setup Appwrite on Ubuntu ,
Blog link: https://dev.to/noviicee/how-to-setup-appwrite-on-ubuntu-3j67,
Blog read time : Oct 8
4 min read
Author: Bryan Robinson,
Blog title is : Building server-rendered search for static sites with 11ty Serverless, Netlify, and Algolia ,
Blog link: https://dev.to/algolia/building-server-rendered-search-for-static-sites-with-11ty-serverless-netlify-and-algolia-13e2,
Blog read time : for Oct 8
8 min read
Author: bhupendra,
Blog title is : Understanding Redux without React ,
Blog link: https://dev.to/bhupendra1011/understanding-redux-without-react-223n,
Blog read time : Oct 8
4 min read
Author: Rizel Scarlett,
Blog title is : Add Fuzzy Search to Your Web App with this Open Source Tool ,
Blog link: https://dev.to/github/add-fuzzy-search-to-your-web-app-with-this-open-source-tool-22d7,
Blog read time : for Oct 8
6 min read
Author: Marcelo Sousa,
Blog title is : Ship / Show / Ask With Reviewpad ,
Blog link: https://dev.to/reviewpad/ship-show-ask-with-reviewpad-47jh,
Blog read time : for Oct 8
5 min read
Author: Shantanu Jana,
Blog title is : Random Gradient Generator using JavaScript & CSS ,
Blog link: https://dev.to/shantanu_jana/random-gradient-generator-using-javascript-css-529c,
Blog read time : Oct 8
6 min read
Author: Miles Watson,
Blog title is : URL Shortener with Rust, Svelte, & AWS (6/): Deploying to AWS ,
Blog link: https://dev.to/mileswatson/url-shortener-with-rust-svelte-aws-6-deploying-to-aws-2gi0,
Blog read time : Oct 8
4 min read
Author: Jon Deavers,
Blog title is : Publishing my first NPM package ,
Blog link: https://dev.to/lucsedirae/publishing-my-first-npm-package-200g,
Blog read time : Oct 8
3 min read
Author: Anjan Shomooder,
Blog title is : CSS positions: Everything you need to know ,
Blog link: https://dev.to/thatanjan/css-positions-everything-you-need-to-know-2ng4,
Blog read time : Oct 8
4 min read
Author: Alvaro Montoro,
Blog title is : Divtober Day 8: Growing ,
Blog link: https://dev.to/alvaromontoro/divtober-day-8-growing-1182,
Blog read time : Oct 8
1 min read
Author: Jambang J,
Blog title is : Deploying an discordjs bot to Qovery ,
Blog link: https://dev.to/jambang067/deploying-an-discordjs-bot-to-qovery-51e,
Blog read time : Oct 8
7 min read
Author: Sadee,
Blog title is : How to create responsive navbar {twitter clone} with HTML CSS ,
Blog link: https://dev.to/codewithsadee/how-to-create-responsive-navbar-twitter-clone-with-html-css-6fa,
Blog read time : Oct 8
1 min read
Author: Jeremy Grifski,
Blog title is : Support The Sample Programs Repo This Hacktoberfest ,
Blog link: https://dev.to/renegadecoder94/support-the-sample-programs-repo-this-hacktoberfest-42ad,
Blog read time : Oct 8
5 min read
Author: Sebastian Rindom,
Blog title is : Making your store more powerful with Contentful ,
Blog link: https://dev.to/medusajs/making-your-store-more-powerful-with-contentful-3efk,
Blog read time : for Oct 8
7 min read
Author: Shalvah,
Blog title is : A practical tracing journey with OpenTelemetry on Node.js ,
Blog link: https://dev.to/shalvah/a-practical-tracing-journey-with-opentelemetry-on-node-js-5706,
Blog read time : Oct 8
16 min read
Author: Kingsley Ubah,
Blog title is : How to build an Accordion Menu using HTML, CSS and JavaScript ,
Blog link: https://dev.to/ubahthebuilder/how-to-build-an-accordion-menu-using-html-css-and-javascript-3omb,
Blog read time : Oct 7
6 min read
Author: mike1237,
Blog title is : Create Proxmox cloud-init templates for use with Packer ,
Blog link: https://dev.to/mike1237/create-proxmox-cloud-init-templates-for-use-with-packer-193a,
Blog read time : Oct 8
3 min read
Author: Prosper Yong,
Blog title is : Get Paid Writing ,
Blog link: https://dev.to/yongdev/get-paid-writing-2i2j,
Blog read time : Oct 8
1 min read
Author: Debbie O'Brien,
Blog title is : Understanding TypeScript ,
Blog link: https://dev.to/debs_obrien/understanding-typescript-378g,
Blog read time : Oct 8
5 min read
Author: Matias D,
Blog title is : Show me your portfolio ,
Blog link: https://dev.to/matiasdandrea/show-me-your-portfolio-1l9h,
Blog read time : Oct 8
1 min read
Author: Marcos Henrique,
Blog title is : You should use Buildpacks instead Dockerfile and I'll tell you why ,
Blog link: https://dev.to/wakeupmh/you-should-use-buildpack-instead-dockerfile-and-i-ll-tell-you-why-2n6,
Blog read time : Oct 8
2 min read
Author: Gaurav Gupta,
Blog title is : Smart Notes - A Build-in Public Product. BuildLog[1] ,
Blog link: https://dev.to/gauravgupta/smart-notes-a-build-in-public-product-buildlog-1-kj6,
Blog read time : Oct 8
4 min read
Author: Andrea Giammarchi,
Blog title is : About bitwise operations ,
Blog link: https://dev.to/webreflection/about-bitwise-operations-29mm,
Blog read time : Oct 8
10 min read
Author: AbcSxyZ,
Blog title is : Business models of Free and Open Source software ,
Blog link: https://dev.to/abcsxyz/business-models-of-free-and-open-source-software-2cg8,
Blog read time : Oct 8
4 min read
Author: Saharsh Laud,
Blog title is : Face Detection in just 15 lines of Code! (ft. Python and OpenCV) ,
Blog link: https://dev.to/saharshlaud/face-detection-in-just-15-lines-of-code-ft-python-and-opencv-37ci,
Blog read time : Oct 8
4 min read
Author: Kaustubh Joshi,
Blog title is : Hello, I'm HTTP and these are my request methods๐๐ป ,
Blog link: https://dev.to/elpidaguy/hello-i-m-http-and-these-are-my-request-methods-co,
Blog read time : Oct 8
3 min read
Author: SilvenLEAF,
Blog title is : Easiest way to create a ChatBOT from Level 0 ,
Blog link: https://dev.to/silvenleaf/easiest-way-to-create-a-chatbot-from-level-0-31pf,
Blog read time : Oct 8
6 min read
Author: whykay ๐ฉ๐ปโ๐ป๐๐ณ๏ธโ๐ (she/her),
Blog title is : ๐ New EuroPython Fellows ,
Blog link: https://dev.to/europython/new-europython-fellows-2ob2,
Blog read time : for Oct 8
1 min read
Author: Zaw Zaw Win,
Blog title is : How to pass props object from child component to parent ,
Blog link: https://dev.to/hareom284/how-to-pass-props-object-from-child-component-to-parent-2a8d,
Blog read time : Oct 8
2 min read
Author: Zack DeRose,
Blog title is : The "DeRxJSViewModel Pattern": The E=mc^2 of State Management [Part 1] ,
Blog link: https://dev.to/zackderose/the-derxjsviewmodel-pattern-the-e-mc-2-of-state-management-part-1-3dka,
Blog read time : Oct 8
23 min read
Author: john methew,
Blog title is : Serverless Cloud Application Development with AWS Lambda ,
Blog link: https://dev.to/johnmethew18/serverless-cloud-application-development-with-aws-lambda-3o7l,
Blog read time : Oct 8
1 min read
Author: Antonio-Bennett,
Blog title is : Hacktoberfest Week 1 ,
Blog link: https://dev.to/antoniobennett/hacktoberfest-week-1-4ebc,
Blog read time : Oct 8
2 min read
Author: ZigRazor,
Blog title is : Hacktoberfest Beginners and Advanced Repos to Contribute to ,
Blog link: https://dev.to/zigrazor/hacktoberfest-beginners-and-advanced-repos-to-contribute-to-p1,
Blog read time : Oct 8
1 min read
Author: Rahul kumar,
Blog title is : Added option to share the blog on any social media | @dsabyte.com ,
Blog link: https://dev.to/ats1999/added-option-to-share-the-blog-on-any-social-media-dsabyte-com-57oo,
Blog read time : Oct 8
2 min read
Author: Kavindu Santhusa,
Blog title is : Top 10 trending github repos of the week๐. ,
Blog link: https://dev.to/ksengine/top-10-trending-github-repos-of-the-week-k7,
Blog read time : Oct 8
1 min read
Author: Andre Willomitzer,
Blog title is : OpenAQ - My first open source PR :) ,
Blog link: https://dev.to/andrewillomitzer/openaq-my-first-open-source-pr-3k32,
Blog read time : Oct 8
2 min read
Author: Kinanee Samson,
Blog title is : Observables Or Promises ,
Blog link: https://dev.to/kalashin1/observables-or-promises-29a8,
Blog read time : Oct 8
9 min read
Author: Amador Criado,
Blog title is : How to enable versioning in Amazon S3 ,
Blog link: https://dev.to/aws-builders/how-to-enable-versioning-in-amazon-s3-17m8,
Blog read time : for Oct 8
2 min read
Author: Bartosz Zagrodzki,
Blog title is : React Context - jak efektywnie go uลผywaฤ? ,
Blog link: https://dev.to/bartek532/react-context-jak-efektywnie-go-uzywac-41l,
Blog read time : Oct 8
8 min read
Conclusion:
This is a quick guide on how to scrape websites, there are other packages that can be used to perform the same function such as puppeteer, fetch, request and so on.
Reference
Web scraping by Thomas W.Smith
Web scraping by Traversy Media
Cheerio Docs
Thanks for reading
Top comments (11)
Thanks for putting this together Abayomi.
I am glad you found it useful
For dynamic content this not works :)
try to use selenium webdriver, this will work perfectly
Thanks for your contribution
Hi, please how did this come about: require('dotenv').config()?
It is giving me error of module not found
You will install it. npm install dotenv
Thanks Abayomi.
You are welcome
How can I get data in the next page if that page has pagination? Thanks for your article!
Thanks for the feedback will read on it