DEV Community: Sebastian Scheibe

Part 2: Parse website HTML with NodeJS and cheerio

Sebastian Scheibe — Mon, 04 May 2020 15:37:12 +0000

Hello back again! If you missed out on how to create the selector for HTML data, please check out part one of the series! Otherwise, let us proceed with the automatic parsing of the data!

We begin with creating a new project. Open up your terminal and enter the following to create a NodeJS project.

mkdir web-data-retrieve && cd web-data-retrieve && npm init

After that, let us add got (HTTP request client) and cheerio (jQuery equivalent for NodeJS). The former will be used to request the content and the later to parse it.

npm install --save cheerio got

Now, let us create an index file which will contain our code.

touch index.js

Open up the index.js file and add the following code inside:

const cheerio = require('cheerio');
const got = require('got');

const url = 'https://www.marketwatch.com/investing/stock/aapl/financials';

(async () => {
    try {
        const response = await got(url);
        const $ = cheerio.load(response.body)
        const selected = $('.financials tr.partialSum:nth-child(1) td.valueCell')
        const cellData = selected.toArray().map(cell => cell.firstChild.data)
        console.log(cellData)
    } catch (error) {
        console.log(error.response.body);
    }
})();

Now let me explain what is going on here, top to bottom. The first two lines are the libraries we will be using:

const cheerio = require('cheerio');
const got = require('got');

The next line is the URL which will be getting the content from, it is identical with the one from the browser.

const url = 'https://www.marketwatch.com/investing/stock/aapl/financials';

The next line is a bit more complex, it is a wrapper, which let us use asynchronous functions. To have that explained, it is better to follow another tutorial :). The more interesting part happens inside of it!

The first line calls the library got with the URL which was defined in the beginning of our file. It's response is saved in the response variable.

try {
        const response = await got(url); # <- This line
        const $ = cheerio.load(response.body)
        const selected = $('.financials tr.partialSum:nth-child(1) td.valueCell')
        const cellData = selected.toArray().map(cell => cell.firstChild.data)
        console.log(cellData)
} catch (error) {
        console.log(error.response.body);
}

After that, the response will be loaded with cheerio. It gives us a query operator, similar to what was used with jQuery in the browser.

try {
        const response = await got(url);
        const $ = cheerio.load(response.body) # <- This line
        const selected = $('.financials tr.partialSum:nth-child(1) td.valueCell')
        const cellData = selected.toArray().map(cell => cell.firstChild.data)
        console.log(cellData)
} catch (error) {
        console.log(error.response.body);
}

What follows is the use of our selector which we created before in the browser. It will return us the selected elements.

try {
        const response = await got(url);
        const $ = cheerio.load(response.body)
        const selected = $('.financials tr.partialSum:nth-child(1) td.valueCell') # <- This line
        const cellData = selected.toArray().map(cell => cell.firstChild.data)
        console.log(cellData)
} catch (error) {
        console.log(error.response.body);
}

The last magical line in the try converts the selected data to standard JS array which allows us to map over it. In the map we take the first child and look at its data.

try {
        const response = await got(url);
        const $ = cheerio.load(response.body)
        const selected = $('.financials tr.partialSum:nth-child(1) td.valueCell')
        const cellData = selected.toArray().map(cell => cell.firstChild.data) # <- This line
        console.log(cellData)
} catch (error) {
        console.log(error.response.body);
}

The application will print out the sales for the Apple stocks:

# node index.js
[ '231.28B', '214.23B', '228.57B', '265.81B', '259.97B' ]

Where to go from here? Well, you can modify the application in ways to retrieve more data of the Apple stock or even fetch different stocks. The possibilities are endless, whatever suits your purpose.

References:

https://github.com/sindresorhus/got

https://cheerio.js.org

https://www.w3schools.com/jquery/jquery_ref_selectors.asp

https://api.jquery.com/category/selectors/

Part 1: Locating the HTML for the website data you are interested

Sebastian Scheibe — Mon, 04 May 2020 15:37:05 +0000

You maybe might be in the same position, you want to gather some data of different objects (like financial data from different entities) from a website, and you copy/paste thousands times from website to your local worksheet. The work of collection makes you exhausting. Believe me, I have been there as you did.

To collect and format data more efficiently and nicely, we will create a little program to solve the problem for us. Afterwards, it will only take a few seconds to retrieve the online data.

The article is split into two parts:

The first part introduces how to locate the data points we want in the HTML of the website.
The second part shows you how to apply that knowledge in an automated way with a small NodeJS script.

This part is going to start with an example of collecting financial information of Apple Inc. from a stock market website, martketwatch.com.
The URL path is: https://www.marketwatch.com/investing/stock/aapl/financials.

The approach I will show is based on the assumption that there is no API available which could be used directly. In the case of martketwatch.com, they render the data on the server, which means the rendered page needs to be parsed. So let us do that!

Now, to help us research how to get to the precious data, let us make use of jQuery. It has the same interface as cheerio, which will be used later.

To use jQuery on the current website, execute the following in the developer console. It adds a script tag to the website which will load the jQuery library for our use.

var script = document.createElement('script');script.src = "https://code.jquery.com/jquery-latest.min.js";document.getElementsByTagName('head')[0].appendChild(script);

Then the $ should be available. In case it is not available, maybe due to CSR policy of the website, the last option is to copy the content of https://code.jquery.com/jquery-latest.min.js and paste and execute it in the developer console.

Give it a test and select all input tags on the website:

$('input')

You should now see that there are some input tags found by the jQuery selector. This is great, the groundwork is laid for our adventure!

Now, let us see how we can select the data we want, in this case, the financial data of the stocks of brand with a bitten fruit.

The data we are interested here are the sales from the year 2015 to 2019, the question is, how can we extract them? Well, you might have guessed it, with the jQuery selector. Let us have a look at the HTML of it. With the element selector of your developers tools, you can select the element in the view.

After selecting a field, the element will be located in the HTML.

Looking at the surrounding HTML, there is a div tag with the class financials. Inside of that tag is the actual table tag, which has a tbody tag. That one contains a tr tag which contains the td tags with our precious data we want to collect! I hope you could follow me here so far :)

Let us try to come up with a selector that will work for jQuery. Clearly the financials class looks like a perfect anchor, due to being unique in the HTML. Next let us try to select only the class valueCell.

$('.financials td.valueCell')

Now this is a bit to broad and we have too many valueCell class tags selected, we need to narrow it down a bit. After looking at what kind of other tags were selected, it seems that we need to constrain it down to the first row of valueCell class tags. Luckily, there is a tr tag which has the class partialSum.

$('.financials tr.partialSum:nth-child(1) td.valueCell')

This leaves us with exact the cells we want to look up! Next step is too use it in our automation!

References:

https://www.w3schools.com/jquery/jquery_ref_selectors.asp

https://api.jquery.com/category/selectors/

#discuss Best course for functional programming?

Sebastian Scheibe — Wed, 15 Apr 2020 19:18:02 +0000

Hey everyone!

What course would you recommend for learning functional programming which also incorporates real world examples?

I read already materials on the subject but they mostly evolve around the basics. I would like to see a course/video/post where a small project is built from ground up in a functional approach.

Any ideas?

How to migrate from Sails.js to Express.js (or how to finally reach land)

Sebastian Scheibe — Thu, 10 May 2018 10:23:34 +0000

At the company I am working for we had a pretty large project written in Sails.js. Now, after such a long time of development the project grew and grew and there was a need now for structural changes. Also it just became necessary to use a compiler for syntax checking.

So, we introduced TypeScript. This step was easy and could be quickly achieved with just a new Grunt task and some modifications at the folder structure.

There is just one thing in Sails.js which made TypeScript less powerful_._ It makes all controllers, services and models available to the GLOBAL variable. This limits the possibilities of TypeScript since Sails.js always expects

module.exports = {
 ...
}

to be set.

With that size of a code base, it just becomes necessary to rely on features like type checking and code completion from TypeScript.

For these features we needed to implement classes. Now, having classes and the Node.js standard export module.exports is not such an ideal combination.

A class looked like this:

So, after some try and error, it seemed like the keyword export would work for Sails.js and we could use our required features from TypeScript.

This concept worked for some time, but in the end we faced sometimes issues with functions not being defined, depending on how the instance was accessed, via GLOBAL or via an import of the file.

This brought me then to the idea of the removal of Sails.js and the implementation of Express.js in combination with a full class driven approach for our code base.

It had also another huge benefit.

We could finally group files into sub folders. This was not possible, since Sails.JS just reads the first layer of the folders it works with (services, controllers, models).

The migration guide

So, how did we migrate in the end?

After some research of what modules are needed, it was more a task of try and error to see if the application boots and the unit tests are still running. :)

Custom app.js and server.ts(js) file

So, first step is to create a custom entry file for the application.

First, we created a new app.js and a server.ts file. The server.ts was created somewhere in the source directory and being a TypeScript file it has the benefits of being checked by the compiler.

The app.js file in the root folder would just call the compiled version of the server.ts file to start the application.

The server.ts file would look like your average Express.js file except that you would add some extra in there, to make it work like Sails.js in the beginning.

Constructing the server file was the major part of the migration, in the end.

There are a couple of things that need to be done:

Global Sails.js object

Sails.js makes an object globally available which contains features like logging, config object, i18n.

To get the code up and running, it was the easiest to just simulate this behaviour:

Setup all the middleware

CSRF
CORS
Locals (translations)
wantsJSON (which has the same behaviour as Sails.js)
Skipper (file uploads)
Default response methods (res.ok() / res.serverError()/ …)

Routing and policies

In Sails.js the routing and the policies are both set up with files, not in the code itself. This makes the migration quite time intensive, if you need to rewrite every route and its policies into code for the Express.js router setup.

If the application is small, this would be not a huge problem. Our application though contains 700 REST routes and the equivalent amount of policies.

In the end, I ended up writing two parsers. One for the route setup, which would parse the routes.js and one for the policies, which would parse the policies.js file.

This had also the huge benefit of other developers could just proceed with their day to day development and extend these files all while I was modifying the core parts of the application. Business as usual could proceed.

Template engine

Sails.js uses by default the EJS template engine.

This gave me a bit trouble, since the default setup of EJS did not work from the beginning with the setup of our EJS templates. There was an issue with how we use sub templates.

After some experimentation I found that it works properly with the package express-ejs-layouts.

This was the setup in the server file:

Also the render methods needed to be changed.

Sails.js implements a behaviour which, based on the controller, detects the correct template file.

The migration would go from:

to:

What about the database layer?

Sails.js uses its own written database connector, Waterline.

This made it a bit more complex since Waterline is made to run inside of Sails.js. Now that there is no Sails.js anymore, how would you trigger the startup? The documentation of the Github page does not give a lot of information on how to work with Waterline in your own project.

After some debugging of the documentation I came up with a class which replicates the behaviour of Sails.js.

The startup method would be called during the Express.js startup.

I came up with this:

Also I made it possible to access the model via the import and have the functions of Waterline available (find()/remove()/save()/…).

A model can now look like this:

Socket.IO

Since we rely heavily on the socket implementation, we were in need to reimplement in nearly the same way.

To init the socket.io we first init the express server. The instance we get from the express server is then used to initiate an instance of socket.io.

Here we use the Redis adapter to keep multiple instances of our application synchronised.

The cookie parser is being used to, as it says, parse the cookies on the first connection from a browser.

After that the Socket.io instance gets started and as the final stage, there is some middleware applied to the Socket.io instance.

In the monitor you can listen on events which are coming in.

As you can see, this methods differs from the controller mapping approach of the Sails.js Socket.io implementation. It should not be too difficult to adapt to the Socket.io event listening approach.

Last words

I am very happy with how everything turned out and how it works.

The next step for the future would be the migrate away from Waterline and towards Mongoose.

I hope you had the patience to read until this point and it might be helpful for you.