DEV Community: Richard Jedlička

Web scraping with Node.js and Typescript - the crawler part (2/3)

Richard Jedlička — Sun, 06 Mar 2022 19:14:42 +0000

Hello, I'm glad to see you here again 🙂. Since the previous article, you've known how to scrape a single webpage. Today, I will teach you how to find all the pages you want to scrape and scrape them.

How crawling works?
Let's crawl
- Finding URLs
- Generators
- Putting it together
- Fixing the unexpected
- Polishing
Final code
Conclusion

❗❗ Important thing to remind. You should be careful when scraping any website. Web scraping isn’t illegal by itself but you should care about how you do it and what you do with the data. There is also an ethical side of it. Do not harm the website and check if you have the rights to use the data the way you are going to. Read more here: https://blog.apify.com/is-web-scraping-legal/. If you are not sure, ask your lawyer.

Disclaimer: I am not taking any responsibility for your web scraping activities. Do it at your own risk.

How web crawling works?

The crawling process use similar principles as you saw when scraping. You look for the HTML link elements in the page and instead of storing their data, you follow them. You repeat this process until you find the right pages. There are two approaches how to do it. It depends on how the website is structured and what do you know about it.

First approach is you can build your crawler that it starts at some page (e.g. homepage) and follows every link (probably only internal so you stay on the same website) it finds. And when it detects the wanted page, it will scrape it. This requires you to know how to detect the right page and it can be time-consuming because it can crawl the entire website and visit irrelevant pages.

Search engines do this because they want to visit every page to scrape and index its metadata.
Second approach is you will crawl only a needed minimum of pages. This requires you to know the website's structure. Most of the pages have some kind of tree-like hierarchy, so you can start somewhere in the middle and only "go down".

E.g. online store, where homepage links to categories, categories link to sub-categories and products' details. So if you want to scrape only mobile phones' data, you will start at mobile phones category page and follow only the sub-category and product links.

Let's crawl

🎓 Our goal now is to find all the URLs of European capital city pages on Wikipedia.

Wikipedia is full of user generated content, and doesn't have any strict rules for its structure. It is like a network of pages where each page references other related pages. It seem we need the first approach here. Fortunately, there are some pages which can help us, like the List of European countries by area with this nice table.

Each country is linked to its wiki page, where we can find a link to a capital city's page. This gives us the hierarchy, and the second approach can be used here 🤞.

Finding URLs

Let's start coding! We need to scrape the country links.

💻 See the complete project in the GitHub repository

Take a look at the HTML code.

Links are located in the second <td> of rows of the <table> with class wikitable.

Add new method crawlCountries to our scraper class.

async crawlCountries(url: string) {
    const response = await axios.get(url);
    const html = response.data;

    const $ = cheerio.load(html);

    const links = $('.wikitable td:nth-child(2) > a').get();

    for (let link of links) {
        const href = $(link).attr('href')!;
        const countryPageUrl = new URL(href, url).toString();
        console.log(countryPageUrl);
    }
}

And modify the main() function.

async function main() {
    const scraper = new CapitalCityScraper();
    await scraper.crawlCountries("https://en.wikipedia.org/wiki/List_of_European_countries_by_area");
}

💻 See the commit 02efe310

I should explain a bit more how Cheerio works. As I mentioned in the previous article, cheerio.load() creates a querying function bound to a document based on the provided HTML markup. This function, when called with a CSS selector, returns a Cheerio object representing a set of matched elements. This object provides convenient methods to make operations on this set. Some of them work only with a single element like .attr(...) which returns the specified attribute of the first matched element. Some of them work with the whole set like .text() which returns a combined text content of all matched elements.

To get the matched elements in a raw form, the .get() method is used. When called without an argument, it returns an array of matched elements. Each element is a DOM-like object, you can work with it directly or if you pass it the Cheerio querying function ($) you will get the Cheerio object representing this single element (as you can see in the for loop).

Run the code and you will see the list of country page URLs 🚩.

$ npx ts-node index.ts
https://en.wikipedia.org/wiki/Russia
https://en.wikipedia.org/wiki/Ukraine
https://en.wikipedia.org/wiki/France
https://en.wikipedia.org/wiki/Spain
https://en.wikipedia.org/wiki/Sweden
...

What now? Instead of printing the URL we would like to visit it and find the link to the capital city page. Let's take one of the URLs and do the inspection 🕵️.

It looks similar to the data we scraped in the previous article. Create another method for that.

async crawlCountry(url: string) {
    const response = await axios.get(url);
    const html = response.data;

    const $ = cheerio.load(html);

    const capitalCityRow = $('.infobox th:contains(Capital)').parent();

    if (capitalCityRow.length === 0) {
        // city state is capitaly city itself
        return url;
    }

    const capitalCityLink = capitalCityRow.find('td > a').attr('href');

    if (!capitalCityLink) {
        // in some cases the capital city cannot be scraped
        // (possible, but too complicated for this article)
        return undefined;
    }

    const capitalCityUrl = new URL(capitalCityLink, url).toString();

    return capitalCityUrl;
}

City states are the capital cities by themselves, so we return the same URL. And, in some cases, the capital city cannot be scraped (explained below).

🙁️️ Due to the lack of strict structure on Wikipedia, scraping of the capital city link is too complicated in some cases, so I decided to sacrifice the completeness for the code simplicity. If you are interested in more information and possible solutions, ask me in the comments.

Replace the console.log in the crawlCountries method with

const capitalCityPageUrl = await this.crawlCountry(countryPageUrl);

if (capitalCityPageUrl) {
    console.log(capitalCityPageUrl);
}

💻 See the commit f3f4da52

Now run the code and see what's happening.

$ npx ts-node index.ts
https://en.wikipedia.org/wiki/Moscow
https://en.wikipedia.org/wiki/Kyiv
https://en.wikipedia.org/wiki/Paris
https://en.wikipedia.org/wiki/Madrid
https://en.wikipedia.org/wiki/Stockholm
...

The URLs of capital cities are printed line by line and there is a delay between each. It is because we are making many HTTP requests, one for each country, and it takes time for the server (Wikipedia) to respond.

❗❗ Think about a server's performance. Try to be gentle and don't hit the server with too many requests in a short time. It can lead to server exhaustion, so the website is not usable when you are scraping and the server may have some protection again it and will block you. If it's not necessary, scrape rather slowly by adding additional delay between requests.

Generators

Our crawler goal is almost finished. We are able to find all the URLs of capital city pages. Now, we need to get the list of URLs out of the crawlCountries method, so we can later loop through them. But how? 🤔

We can store the URLs in an array and return it whole.

async crawlCountries(url: string) {
    const response = await axios.get(url);
    const html = response.data;

    const $ = cheerio.load(html);

    const links = $('.wikitable td:nth-child(2) > a').get();

    const urls: string[] = [];

    for (let link of links) {
        const href = $(link).attr('href')!;
        const countryPageUrl = new URL(href, url).toString();
        const capitalCityPageUrl = await this.crawlCountry(countryPageUrl);

        if (capitalCityPageUrl) {
            urls.push(capitalCityPageUrl);
        }
    }

    return urls;
}

async function main() {
    for (let url of await scraper.crawlCountries("https://en.wikipedia.org/wiki/List_of_European_countries_by_area")) {
        console.log(url);
    }
}

💻 See the commit a3ae3829

This looks fine and it works, but there is a catch. The problem is we have to build the whole array before we can return it from the function. What does it mean? It means we have to wait until the "crawler" finds all the URLs before we can start using them. In our case, it takes not such a long time, but imagine we are crawling hundreds or thousands of pages. It can take very long time, in the order of minutes or hours.

Try to run the code and you will see. The output is the same as before but the timing has changed. Nothing is happening for some time and then the whole list is printed.

Modern versions of JavaScript and TypeScript offer one very useful construct, generators. It allows us to "return" values from a function multiple times and even before it has finished. To make it work, you must mark the function as generator function by prefixing its name with * and instead of return keyword you use a yield. The function then actually returns an iterator, an object on which you call .next() to get the next value. You can iterate (loop trough) the iterator same way as an array (e.g. in a for loop). The advantage here is that we have access to the "returned" value immediately after yielding. You can imagine it as the function pauses in the place of the yield statement and waits until someone requires the next value from the iterator, and then continue. Read more about generators here.

So let's try the generators instead.

async *crawlCountries(url: string) {
    const response = await axios.get(url);
    const html = response.data;

    const $ = cheerio.load(html);

    const links = $('.wikitable td:nth-child(2) > a').get();

    for (let link of links) {
        const href = $(link).attr('href')!;
        const countryPageUrl = new URL(href, url).toString();
        const capitalCityPageUrl = await this.crawlCountry(countryPageUrl);

        if (capitalCityPageUrl) {
            yield capitalCityPageUrl;
        }
    }
}

async function main() {
    const scraper = new CapitalCityScraper();
    for await (let url of scraper.crawlCountries("https://en.wikipedia.org/wiki/List_of_European_countries_by_area")) {
        console.log(url);
    }
}

💻 See the commit 5ad62c43

Notice the * before crawlCountries method's name. The yield keyword. This creates a generator.

In the main function, there is a for await contruct. Why? Because the crawlCountries is async so it returns async generator (iterator). Which means the .next() call is asynchronous (returs a promise) and for await takes care of it by awaiting on each iteration.

If you now run the code, the output will be the same as when we printed the URL directly in crawlCountries method (printing URLs one by one), but notice we now print them outside of the crawlCountries function, isn't it cool?

Putting it together

Do you know what's next? Yes, put everything together: the crawler methods with the scraper ones we built in the previous article. And the change is very simple, just one line. Are you excited as much as me 🤩?

Replace the console.log(url) in the main function by scrapeCity method call.

async function main() {
    const scraper = new CapitalCityScraper();
    for await (let url of scraper.crawlCountries("https://en.wikipedia.org/wiki/List_of_European_countries_by_area")) {
        console.log(await scraper.scrapeCity(url));
    }
}

💻 See the commit e77e7e27

Go go, run the code! 🙏

$ npx ts-node index.ts
{
  name: 'Moscow',
  country: 'Russia',
  area: NaN,
  population: NaN,
  flagImagePath: 'flags\\Flag_of_Moscow%2C_Russia.svg'
}
{
  name: 'Kyiv',
  country: 'Ukraine',
  area: 839,
  population: 2962180,
  flagImagePath: 'flags\\Flag_of_Kyiv_Kurovskyi.svg'
}
{
  name: 'Paris',
  country: 'France',
  area: NaN,
  population: NaN,
  flagImagePath: 'flags\\Flag_of_Paris_with_coat_of_arms.svg'
}
...

Wow, it works!! Yes, yes ... or ... oh, wait. 😮 What's that NaN?? It doesn't look like a number, yes it's not a number. Have you looked in flags folder? There is a weird file with a name undefined. What happened? Again, the Wikipedia's lack of strict structure. The problem is some city pages are formatted slightly differently, and our scraper doesn't handle it well.

Don't worry. This is quite common when you try to scrape a website. Even though you do some analysis before, you will probably encounter unexpected problems when you scrape the bigger portion of the website. Some page will have missing values, some will have different formatting, some pages cannot be loaded, etc. This will likely to happen and you need to make your scraper robust enough to handle all the cases. Be prepared that sometimes you won't be able to scrape everything or with too much effort for a little outcome.

Fixing the unexpected

I did the hard work for you and analyzed the pages. Before we begin, we need to update our model because not every value is possible to scrape.

interface City {
    name: string;
    country?: string;
    area?: number;
    population?: number;
    flagImagePath?: string;
}

Then update the scrapeCity method itself.

After cheerio parses the HTML, remove the elements with reference links from infobox (the box with the data).

They make the scraping harder and we don't need them.

const $ = cheerio.load(html);

$('.infobox .reference').remove();

City states don't have the country specified, and the row doesn't always have the class mergedtoprow.

const country = $('.infobox th:contains(Country) + td').first().text().trim() || undefined;

Area values don't have always the same label and are not in the same place. Sometimes the value is even next to the "Area" label (see Paris). So I decided to find and use the first value in the correct format (a number with km squared unit).

const areaRows = $('.mergedtoprow th:contains(Area)').parent().nextUntil('.mergedtoprow').addBack();
const areaValues = areaRows.find('td').filter((i, el) => !!$(el).text().trim().match(/^[0-9,.]+\s+km2.*$/));
const areaText = areaValues.first().text().trim().replace(/ km2.*$/, '');
const area = parseFloat(areaText.replace(/,/g, '')) || undefined;

Population scraping is similar to the area. Only the matching regular expression is different (a number optionally followed with a text in parentheses, nothing else).

const populationRows = $('.mergedtoprow th:contains(Population)').parent().nextUntil('.mergedtoprow').addBack();
const populationValues = populationRows.find('td').filter((i, el) => !!$(el).text().trim().match(/^[0-9,.]+(\s+\(.*\))?$/));
const populationText = populationValues.first().text().trim();
const population = parseFloat(populationText.replace(/,/g, '')) || undefined;

Not every city has a flag and also the row doesn't always have the class mergedtoprow.

const flagPageLink = $('.infobox a.image + div:contains(Flag)').prev().attr('href')!;
const flagPageUrl = flagPageLink && new URL(flagPageLink, url).toString();
const flagImagePath = flagPageUrl && await this.scrapeImage(flagPageUrl);

💻 See the commit 0402d629

Run the code now. Despite the data are not 100% complete, you can be quite satisfied with the result.

Full completeness of the scraped data from sites with a loose structure, such as Wikipedia, is very hard to achieve. Sometimes almost not even possible.

Polishing

We can say we are finished but I would prefer to polish the scraper class a little bit more.

I would rather encapsulate the scraper, so it exposes only one method scrape which will return a generator of scraped City items.

Make all the methods in the scraper class protected and add new public method scrape along with the class property holding the starting URL.

readonly startUrl = "https://en.wikipedia.org/wiki/List_of_European_countries_by_area";

async *scrape() {
    for await (let url of this.crawlCountries(this.startUrl)) {
        yield await this.scrapeCity(url);
    }
}

Actually, we moved the code from the main function to the scrape method. And in the main we only get the scraped city object without caring about the internals.

async function main() {
    const scraper = new CapitalCityScraper();

    for await (let city of scraper.scrape()) {
        console.log(city);
    }
}

💻 See the commit 32f506c9

Hey, we are finished. Let's celebrate! 🥳🎉🍾🥂🍸

Final code

💻 See the complete project in the GitHub repository

import fs from 'fs';
import path from 'path';
import axios from 'axios';
import cheerio from 'cheerio';

interface City {
    name: string;
    country?: string;
    area?: number;
    population?: number;
    flagImagePath?: string;
}

export class CapitalCityScraper {
    readonly startUrl = "https://en.wikipedia.org/wiki/List_of_European_countries_by_area";

    async *scrape() {
        for await (let url of this.crawlCountries(this.startUrl)) {
            yield await this.scrapeCity(url);
        }
    }

    async *crawlCountries(url: string) {
        const response = await axios.get(url);
        const html = response.data;

        const $ = cheerio.load(html);

        const links = $('.wikitable td:nth-child(2) > a').get();

        for (let link of links) {
            const href = $(link).attr('href')!;
            const countryPageUrl = new URL(href, url).toString();
            const capitalCityPageUrl = await this.crawlCountry(countryPageUrl);

            if (capitalCityPageUrl) {
                yield capitalCityPageUrl;
            }
        }
    }

    async crawlCountry(url: string) {
        const response = await axios.get(url);
        const html = response.data;

        const $ = cheerio.load(html);

        const capitalCityRow = $('.infobox th:contains(Capital)').parent();

        if (capitalCityRow.length === 0) {
            // city state is capitaly city itself
            return url;
        }

        const capitalCityLink = capitalCityRow.find('td > a').attr('href');

        if (!capitalCityLink) {
            // in some cases the capital city cannot be scraped
            // (possible, but too complicated for this article)
            return undefined;
        }

        const capitalCityUrl = new URL(capitalCityLink, url).toString();

        return capitalCityUrl;
    }

    async scrapeCity(url: string) {
        const response = await axios.get(url);
        const html = response.data;

        const $ = cheerio.load(html);

        $('.infobox .reference').remove();

        const cityName = $('#firstHeading').text().trim();

        const country = $('.infobox th:contains(Country) + td').first().text().trim() || undefined;

        const areaRows = $('.mergedtoprow th:contains(Area)').parent().nextUntil('.mergedtoprow').addBack();
        const areaValues = areaRows.find('td').filter((i, el) => !!$(el).text().trim().match(/^[0-9,.]+\s+km2.*$/));
        const areaText = areaValues.first().text().trim().replace(/ km2.*$/, '');
        const area = parseFloat(areaText.replace(/,/g, '')) || undefined;

        const populationRows = $('.mergedtoprow th:contains(Population)').parent().nextUntil('.mergedtoprow').addBack();
        const populationValues = populationRows.find('td').filter((i, el) => !!$(el).text().trim().match(/^[0-9,.]+(\s+\(.*\))?$/));
        const populationText = populationValues.first().text().trim();
        const population = parseFloat(populationText.replace(/,/g, '')) || undefined;

        const flagPageLink = $('.infobox a.image + div:contains(Flag)').prev().attr('href')!;
        const flagPageUrl = flagPageLink && new URL(flagPageLink, url).toString();
        const flagImagePath = flagPageUrl && await this.scrapeImage(flagPageUrl);

        const city: City = {
            name: cityName,
            country,
            area,
            population,
            flagImagePath
        };

        return city;
    }

    protected async scrapeImage(url: string) {
        const response = await axios.get(url);
        const html = response.data;

        const doc = cheerio.load(html);

        const imageLink = doc('#file a').attr('href')!;
        const imageUrl = new URL(imageLink, url).toString();

        const imagePath = await this.downloadFile(imageUrl, 'flags');

        return imagePath;
    }

    protected async downloadFile(url: string, dir: string) {
        const response = await axios.get(url, {
            responseType: 'arraybuffer'
        });

        fs.mkdirSync(dir, { recursive: true });

        const filePath = path.join(dir, path.basename(url));
        fs.writeFileSync(filePath, response.data);

        return filePath;
    }
}

async function main() {
    const scraper = new CapitalCityScraper();

    for await (let city of scraper.scrape()) {
        console.log(city);
    }
}

main();

Conclusion

Now, you have all you need to build a web scraper, at least a simple one. You saw that sometimes you have to compromise and decide between completeness or code simplicity. But still you can mostly retrieve valuable information even from not well-structured websites.

Hope you like web scraping. 😉

There are some advanced topics, how you can improve the scraper, like execution management, auto throttling, etc. In the next article, I will show you one: the use of a proxy.

👉 Stay tuned! See you soon.

Web scraping with Node.js and Typescript - the scraper part (1/3)

Richard Jedlička — Mon, 21 Feb 2022 14:25:13 +0000

Internet is full of information these days. Almost every website display them to the user in a human readable form. But what if you want to process these data programmatically, do some analysis, present them in a different form or store them in a database to make queries on them later? E.g. collect all the product names with a description, image and a price from your favorite online store. Well, you can open the page by page and copy&paste the data you need, but you won't 🤦‍♂️. What you definitely can and should is to check if the page has an API which will provide you the data easily. If not, I'm sorry bro there is no way to ... just kidding! 😝

... the web scraping comes into play. Yay!

👉 In this article series (3 parts) I will guide you through the whole process of building a web scraper in Node.js and Typescript.

In the first article you will learn how to scrape data from a single webpage. In the second article you I will teach you how to crawl the website to find and scrape all the wanted pages. And in the last article I will show you how to use a proxy with the scraper (coming soon) which can have some advantages in certain situations.

If you are a beginner or a more skilled programmer who is new to web scraping, transitioning from different programming language or just curious how others do it, you will benefit from this.

Prerequisites
What is web scraping exactly?
Let's scrape something!
- What are we going to scrape?
- First simple scraping
- More difficult selector
- Scraping elements "in context"
- Parsing the scraped data
- Scraping images
Data model
Final code
Conclusion

Prerequisites

I assume you are familiar with Javascript and Typescript, know HTML and CSS selectors and have Node.js installed.

If not, check out these resources:

Javascript: https://javascript.info
Typescript: https://www.typescriptlang.org/docs
HTML/CSS: https://learn.shayhowe.com

What is web scraping exactly?

As you should know the websites are built using HTML and CSS. HTML describes the structure of the information in the page using tags. What web scraper does is extracting required information from the specified HTML tags. CSS selectors are very good way how to tell the scraper which tags to look at.

So the input for the scraper is the URL of a page (e.g. product detail). The scraper then loads the HTML source code, parse it, filters the tags specified by CSS selectors and extracts text from them. Then outputs the extracted data in a structured way (e.g. JSON). Easy, right?

Wait! You may ask ... Where do I get the page URLs? Do I have to copy&paste them to the crawler manually?

Of course not! Web scrapers are usually more robust and also contain the "crawler" part to automate the whole process.

The crawler will go through (crawls) the website and search for the pages which have the data to be scraped.

Actually, it is a special type of scraper which usually starts at homepage and looks for the hyperlinks according to specific rules and follows them and repeats the process until it finds the desired pages.

The term "web scraper" is often used interchangeably with "web crawler".

❗❗ Important thing to know is you should be careful when scraping any website. Web scraping isn’t illegal by itself but you should care about how you do it and what you do with the data. There is also an ethical side of it. Do not harm the website and check if you have the rights to use the data the way you are going to. Read more here: https://blog.apify.com/is-web-scraping-legal/. If you are not sure, ask your lawyer.

Disclaimer: I am not taking any responsibility for your web scraping activities. Do it at your own risk.

Let's scrape something!

🎓 For an example, consider we want to have a list of all European capital cities with a basic data like its name, name of the country, current population, area and an image of a city flag. The Wikipedia can be used as a good source of information.

First, init the project



npm init
npm install --save-dev typescript ts-node
npx tsc --init

and install the packages we will need.



npm install axios cheerio @types/cheerio

Axios is an HTTP client which we will use for fetching website data. It is more robust and feature-rich alternative to Fetch API.

Cheerio is a tool to parse HTML and gives you the ability to make queries on HTML tags and extract data from them. It is similar to jQuery but more suitable for server side.

💻 See the complete project in the GitHub repository

What are we going to scrape?

As we are prepared, we will start with a "scraper" part, so go and look at the capital city page we are going to scrape, e.g. https://en.wikipedia.org/wiki/Prague

There it is, the data we need. Ah, ok, but how do we know the location of the data in the page's HTML 🤔? Easily, we use dev tools. I'm using Chrome browser (other modern browsers usually have dev tools too) so right click the article's title element and select Inspect.

As you can see, the name of the city resides in <h1> tag with the ID firstHeading. I'm sure you are getting the idea.

First simple scraping

Stop talking and create some code!

Create a file index.ts and put in this code



import axios from 'axios';
import cheerio from 'cheerio';

export class CapitalCityScraper {
    async scrapeCity(url: string) {
        const response = await axios.get(url);
        const html = response.data;

        const $ = cheerio.load(html);

        const cityName = $('#firstHeading').text().trim();
        console.log(cityName);
    }
}

async function main() {
    const scraper = new CapitalCityScraper();
    await scraper.scrapeCity("https://en.wikipedia.org/wiki/Prague");
}

main();

💻 See the commit f13ccee0

Are you excited? Run the code



$ npx ts-node index.ts
Prague

🎉 Congratulations, your first scraping! Isn't it beautiful 🤩?

I think the code is quite self-explanatory, still I will go through some interesting moments



const response = await axios.get(url);
const html = response.data;

Axios makes an HTTP GET request to the specified URL and returns a promise which will hold the response with an HTML source code (in our case).

If you are not familiar with async/await check this https://javascript.info/async-await. Basically, it is very comfortable way to work with JS promises. The code "waits" until the promise is resolved and returns its data.



const $ = cheerio.load(html);

Cheerio parses the HTML and returns a querying function bound to a document based on that HTML markup. The querying function ($) accepts CSS selector and finds corresponding element(s) in the document.



const cityName = $('#firstHeading').text().trim();

Here we find the element with ID firstHeading and get its text content. It is also a good practice to trim the leading and trailing whitespace.

Ok, this was easy, right? Let's move on to something more difficult.

More difficult selector

The country's name is in <a> tag, but the tag has no ID or a class. We have to loot at its parent elements. The interesting one is the table row <tr> with a class mergedtoprow. But, there is a catch. If you look around, there are lots of table rows with the same class. Hmm 🤔, how do we select the correct row? Maybe we can use row's index? I wouldn't count on that as the other pages may have different number of info rows. I think there is no easy way with regular CSS selectors. What we can expect is that the row's label will always be "Country". Cheerio supports the same selectors as jQuery and it has a special selector :contains() (see jQuery doc) which checks if the element contains specific text. So the idea is we find the <td> element which is after the <th> element (row's label) containing text "Country".

Add this to the end of the scrapeCity method.



const country = $('.mergedtoprow th:contains(Country) + td').text().trim();
console.log(country);

💻 See the commit 07db4fbe

Run the code again



$ npx ts-node index.ts
Prague
Czech Republic

Nice!

Scraping elements "in context"

When you look at area and population, the rows we are interested in have the same label Capital city, therefore we can't use the same selector as for the country name directly. We need to find the relevant row according to the previous label Area or Population. You might be getting the impression that the row is nested inside a box, but, if you look closely, there are no boxes actually.

There are "top level" rows with mergedtoprow class which may have a "sub rows" with a class mergedrow. The "sub rows" are placed between two "top level" rows and relate to the first one. This is all we need to know.



const areaRows = $('.mergedtoprow th:contains(Area)').parent().nextUntil('.mergedtoprow');
const area = areaRows.find('th:contains(Capital city) + td').text().trim();
console.log(area);

First line find the "Area" label, the parent() method select the wrapping row and with nextUntil() we select all the next elements (rows) before next "top level" row. With this we get a context (areaRows) where we find the value with the same principle as for the country's name.

The same for population



const populationRows = $('.mergedtoprow th:contains(Population)').parent().nextUntil('.mergedtoprow');
const population = populationRows.find('th:contains(Capital city) + td').text().trim();
console.log(population);

💻 See the commit 95c3bb09

And after running



$ npx ts-node index.ts
Prague
Czech Republic
496 km2 (192 sq mi)
1,335,084

All right. We have got the information, but in a formatted shape 🤔. We want numbers!

Parsing the scraped data

It happens quite often when scraping that the information you scrape is formatted as human readable and not structured very well. So you have to make another step to parse (extract) the right data from the strings you scrape. In our case we want to have are as number of squared kilometers and the population as a count of persons.

Regular expressions for the win!

Modify the code slightly



const areaText = areaRows.find('th:contains(Capital city) + td').text().trim().replace(/ km2.*$/, '');
const area = parseFloat(areaText.replace(/,/g, ''));

const populationText = populationRows.find('th:contains(Capital city) + td').text().trim();
const population = parseFloat(populationText.replace(/,/g, ''));

Notice this will work for English localization, different languages can have numbers in different format.

In area text, we first drop everything from the unit to the end. And before converting to a number with parseFloat the commas must be removed.

💻 See the commit e88b6bae

Looks better now!



$ npx ts-node index.ts
Prague
Czech Republic
496
1335084

Scraping images

When scraping images, you can just scrape the image's URL or download the file itself. URL is fine if you want to display the image on another website or just want to store the link to it. But if you want to make some modifications to the image or you can't rely on the image's availability on the source website, you need to download it. I will show you the second case.

Still, we need to obtain the image's URL first. Let's analyze the HTML for the city flag.

The image of the flag is wrapped in <a> tag with image class which is in front of the <div> with a text "Flag". However, the <img> tag doesn’t keep the original SVG file, only the small PNG thumbnail. The anchor tag looks like it keeps the image’s URL.

Actually, it links to another webpage.

There it is. The <a> tag there has the URL we are looking for.

Get the flag image page URL.



const flagPageLink = $('.mergedtoprow a.image + div:contains(Flag)').prev().attr('href')!;
const flagPageUrl = new URL(flagPageLink, url).toString();
const flagImagePath = await this.scrapeImage(flagPageUrl);
console.log(flagImagePath);

I made the selector more universal by using a template string with a city's name. The flagPageLink variable keeps the relative path. The URL object will help us to obtain the full URL, the second argument is a base URL, the one of the city's wiki page in our case.

To make the scraper more organized, I moved the code for the image scraping into the separated method scrapeImage. The method can be used to scrape the image from any Wikipedia's image detail page.



protected async scrapeImage(url: string) {
    const response = await axios.get(url);
    const html = response.data;

    const $ = cheerio.load(html);

    const imageLink = $('#file a').attr('href')!;
    const imageUrl = new URL(imageLink, url).toString();

    const imagePath = await this.downloadFile(imageUrl, 'flags');

    return imagePath;
}

Everything should be already familiar to you. And again the code related to downloading of the image is separated to another method downloadFile.



protected async downloadFile(url: string, dir: string) {
    const response = await axios.get(url, {
        responseType: 'arraybuffer'
    });

    fs.mkdirSync(dir, {recursive: true});

    const filePath = path.join(dir, path.basename(url));
    fs.writeFileSync(filePath, response.data);

    return filePath;
}

This method is universal for downloading any file to a specified directory. The option responseType: 'arraybuffer' is crucial here. Axios will then consider the URL as a source of binary data and don't try to parse the response as a text.

💻 See the commit 2e0dec92

Now, if you run the code, you will see this



$ npx ts-node index.ts
Prague
Czech Republic
496
1335084
flags/Flag_of_Prague.svg

And if you look into the folder flags you will find the file Flag_of_Prague.svg here 🥳.

Data model

Great, we can scrape all the data we need. But all of them are just printed to the console in the moment when they are obtained. This is not good to work with. We want to return them in some form from our scrapeCity method. Plain object is sufficient.

For type safety, we will use an interface. Put it above the scraper class.



interface City {
    name: string;
    country: string;
    area: number;
    population: number;
    flagImagePath: string;
}

Remove all the console.log commands and put the this code at the end of scrapeCity function.



const city: City = {
    name: cityName,
    country,
    area,
    population,
    flagImagePath
};

return city;

Now this is much better, out scraped data has a specific shape and we can manipulate with them later. For now, we will just modify our main function to get the city object and print it to the console (in whole).



async function main() {
    const scraper = new CapitalCityScraper();
    const city = await scraper.scrapeCity("https://en.wikipedia.org/wiki/Prague");
    console.log(city);
}

💻 See the commit 96426d2d

Run the script.



$ npx ts-node index.ts
{
  name: 'Prague',
  country: 'Czech Republic',
  area: 496,
  population: 1335084,
  flagImagePath: 'flags/Flag_of_Prague.svg'
}

I feel quite satisfied now 😎. What about you?

Final code

💻 See the complete project in the GitHub repository



import fs from 'fs';
import path from 'path';
import axios from 'axios';
import cheerio from 'cheerio';

interface City {
    name: string;
    country: string;
    area: number;
    population: number;
    flagImagePath: string;
}

export class CapitalCityScraper {
    async scrapeCity(url: string) {
        const response = await axios.get(url);
        const html = response.data;

        const $ = cheerio.load(html);

        const cityName = $('#firstHeading').text().trim();

        const country = $('.mergedtoprow th:contains(Country) + td').text().trim();

        const areaRows = $('.mergedtoprow th:contains(Area)').parent().nextUntil('.mergedtoprow');
        const areaText = areaRows.find('th:contains(Capital city) + td').text().trim().replace(/ km2.*$/, '');
        const area = parseFloat(areaText.replace(/,/g, ''));

        const populationRows = $('.mergedtoprow th:contains(Population)').parent().nextUntil('.mergedtoprow');
        const populationText = populationRows.find('th:contains(Capital city) + td').text().trim();
        const population = parseFloat(populationText.replace(/,/g, ''));

        const flagPageLink = $('.mergedtoprow a.image + div:contains(Flag)').prev().attr('href')!;
        const flagPageUrl = new URL(flagPageLink, url).toString();
        const flagImagePath = await this.scrapeImage(flagPageUrl);

        const city: City = {
            name: cityName,
            country,
            area,
            population,
            flagImagePath
        };

        return city;
    }

    protected async scrapeImage(url: string) {
        const response = await axios.get(url);
        const html = response.data;

        const doc = cheerio.load(html);

        const imageLink = doc('#file a').attr('href')!;
        const imageUrl = new URL(imageLink, url).toString();

        const imagePath = await this.downloadFile(imageUrl, 'flags');

        return imagePath;
    }

    protected async downloadFile(url: string, dir: string) {
        const response = await axios.get(url, {
            responseType: 'arraybuffer'
        });

        fs.mkdirSync(dir, { recursive: true });

        const filePath = path.join(dir, path.basename(url));
        fs.writeFileSync(filePath, response.data);

        return filePath;
    }
}

async function main() {
    const scraper = new CapitalCityScraper();
    const city = await scraper.scrapeCity("https://en.wikipedia.org/wiki/Prague");
    console.log(city);
}

main();

Conclusion

Now you know how to scrape a web page in Javascript/Typescript. I hope you agree it is quite easy and fun.

Of course it depends on the website you want to scrape, the less the data you want are structured, the harder is it to get them. There are always many ways how to achieve the goal, sometimes it is straightforward, sometimes tricky. But if you managed it, the result can be quite satisfying when you are giving an order to something unorganised 😁.

Currently, we can handle a single capital city page only. In the next article I will teach you how to crawl the Wikipedia website to scrape all of them.

DEV Community: Richard Jedlička

Web scraping with Node.js and Typescript - the crawler part (2/3)

How web crawling works?

Let's crawl

Finding URLs

Generators

Putting it together

Fixing the unexpected

Polishing

Final code

Conclusion

Web scraping with Node.js and Typescript - the scraper part (1/3)

Prerequisites

What is web scraping exactly?

Let's scrape something!

What are we going to scrape?

First simple scraping

More difficult selector

Scraping elements "in context"

Parsing the scraped data

Scraping images

Data model

Final code

Conclusion

Web scraping with Node.js and Typescript - the crawler part (2/3)

Richard Jedlička ・ Mar 6 '22