DEV Community: George Gkasdrogkas

The definitive guide on compiling and linking Boost C++ libraries for Visual Studio projects

George Gkasdrogkas — Mon, 19 Apr 2021 21:06:18 +0000

Recently, a requirement came up to migrate a legacy C++ codebase, containing quite a few of custom implemented functionalities, into a more robust architecture. After spending some time at searching, we decided to include the Boost libraries, due to the large set of utilities they offer.

Most of the C++ Boost libraries are header-only; they consist entirely of header files, and require no separately-compiled library binaries. However, there are some libraries that need to be built separately. The "Getting Started" guide on Boost website is quite informative, but do not provide clear guidance on how to build for multiple architectures and targets (debug/release). We're using Visual Studio as our IDE and finding the best way to build those binaries was quite tiresome.

For that reason, we dedicated this guide on how to build and link the C++ Boost libraries in Visual Studio projects.

Downloading the Boost libraries

To get the latest version of Boost (or any other version for that matter), go to the official download page. We are targeting Windows platforms, so we need to choose the respective version. The "Getting Started" guide recommends downloading the 7-Zip archive.

We no longer recommend .zip files for Boost because they are twice as large as the equivalent .7z files. We don't recommend using Windows' built-in decompression as it can be painfully slow for large archives. - "Getting Started on Windows", www.boost.org

The current release, as of this day, is version 1.76.0, but the methodology we are going to follow works (and will continue to work in the future) for any release.

After we successfully downloaded the archive, we need to extract the containing folder into the desired location.

The documentation assumes that the path for any library version is inside C:\Program Files\boost, so we're going to respect that. Create a directory called boost inside C:\Program Files\ and extract the archive there.

Building the binaries

The Boost libraries includes a really nice build system, which we are definitely going to use it. The build system is triggered from the command line. First we have to open the cmd window and navigate into the root folder of the Boost library.

Then we have to initialize the build system by running the bootstrap.bat file.

After bootstraping is done, it's time to build the actual binaries. Remember what we said earlier: We have to build

artifacts for x64 architecture
artifacts for x86 architecture
artifacts for static linking (i.e. .lib files)
artifacts for dynamic linking (i.e. .dll files)
release artifacts
debug artifacts

In general, we need to trigger 8 builds. Fortunately for us, the build command allow us to specify multiple targets in one build. In the end, we'll only trigger two builds, one for each architecture type.

Building for x86 architecture

Run the following command in you terminal:

b2 --build-dir=build\x86 address-model=32 threading=multi --stagedir=.\bin\x86 --toolset=msvc -j 16 link=static,shared runtime-link=static,shared --variant=debug,release

This command will build and place the binaries into C:\Program Files\boost\boost_1_76_0\bin\x86\lib\. The output will contain all the required files we might need for most of our projects.

Building Debug for x64 architecture

Execute the following command on you terminal:

b2 --build-dir=build\x64 address-model=64 threading=multi --stagedir=.\bin\x64 --toolset=msvc -j 8 link=static,shared runtime-link=static,shared --variant=debug,release

This command will build and place the binaries into C:\Program Files\boost\boost_1_76_0\bin\x64\lib\. As before, the output will contain all the required files we might need for most of our projects.

Explaining the build command arguments

Let's take a quick look on the arguments used in the previous two commands.

--build-dir: Specify the directory to place all the intermediate files while building.
address-model: Specify the targeting address model.
threading: Compile Boost to be thread aware. (see this question on stackoverflow for more info.)
--stage-dir: the directory where the binaries will be placed.
--toolset: The compiler and linker that will be used to build the artifacts. We chose msvc; this should be the default on Windows, but in case we have more compilers installed on the system, it's better to explicitly define it.
-j: how many threads to use for building. This can drastically improve the build times on multi-threading environments, i.e. most modern machines.
link: declare a library target.
runtime-link: linking statically/dynamically to the C++ standard library and compiler runtime support libraries.
--variant: Build debug or release version of the binaries.

Including the Boost libraries on Visual Studio

To be able to work with the Boost libraries in Visual Studio, we have to define the root path, that is C:\Program Files\boost\boost_1_76_0, into each individual project properties. In this guide, we use Visual Studio 2019, but this process can be applied to older versions as well. To include the libraries, do the following:

On a Visual Studio solution → Select the project you want to add the library into → Right Click → "Properties"

The properties window should now be open. An example is attached bellow:

Before we proceed take a moment to ensure that both "Configuration" and "Platform" values are selected as shown in the following screenshot. This will apply the changes we're going to make into all the platforms.

We're going to modify the "Additional Include Directories" option of "C/C++" → "General" property.

Select the drop-down button and then click < Edit.. >.

Now we have to add the library path and hit OK.

The option value should be updated as shown in the following screenshot.

Note that %(AdditionalIncludeDirectories) macro was automatically included in the value. This is expected and includes some default values specified by the system.

We are ready to work with most of the Boost libraries. We can include them in our source code without any issues. However, if we use libraries that require linking (like boost/chrono, boost/thread, etc) we have to link the binaries we built earlier to the linker.

Linking the Boost libraries on Visual Studio

This is the final step to our journey. The only think we have to do, is to tell the linker to include our binaries. Like before, open the properties window:

On a Visual Studio solution → Select the project you want to add the library → Right Click → "Properties"

Before we proceed take a moment to ensure that the "Configuration" and "Platform" values are selected as shown in the following screenshot. This will apply the changes we're going to make into all the platforms.

We're going to modify the "Additional Library Directories" option of "Linker" → "General" property.

(Note: In case your project builds a static library, the "Linker" menu will be shown as "Librarian".)

Select the drop-down button and then click < Edit.. >.

Add the following value: C:\Program Files\boost\boost_1_76_0\bin\$(PlatformTarget)\lib, and hit OK.

The property value should be updated as shown in the following screenshot.

Take a quick note on $(PlatformTarget) macro. Recall when we built our targets and created two directories, one for each architecture. This macro defines the correct folder for use depending on the selected build platform.

Note that %(AdditionalLibraryDirectories) macro was automatically included in the value. This is expected and includes some default values specified by the system.

Testing the boost libraries

We should now be able to build and run our projects in both x86 and x64 architectures, targeting both debug and release versions. In case you want a quick, test program, we provide the following example from boost/chrono library.

Bellow is a running example screenshot using Debug and x64:

Conclusion

In this guide we described how we can set up Boost C++ libraries in Windows and link them into Visual Studio projects. We tried to be as detailed as possible in our explanation. We wanted to provide a unified reference for future readers who might encounter the same problems we did.

As a final note, we inform the readers that we tested all the steps provided in this article, on multiple projects, without any issues. However, if you have a different set-up, that might or might not cause problems. If this is the case, feel free to comment your question or point any ambiguity you have encountered.

I have plans to write more articles regarding project organization and software development in the future. Feel free to post suggestions and articles ideas you might want to read about.

There is no such thing as the Spread Operator in JavaScript!

George Gkasdrogkas — Fri, 10 Apr 2020 15:09:30 +0000

Have you heard about the Spread Syntax? Introduced in ES2015, we love it due to its simple semantics and ubiquitous use cases. What about the Spread Operator? Yes, it's the three dots (...) used by the Spread Syntax!

…and by saying such things, we start digging into the horrifying world of a bug's life…

A quick review of the Spread Syntax

A simple use case where we can leverage the use of the Spread Syntax is when we want to concatenate multiple arrays. Check the following snippet:

const clientErrors = ['err1', 'err2', 'err3'];
const serverErrors = ['err4', 'err5'];

function numberOfErrors(clientErrors, serverErrors) {
  // Assuming that both inputs are arrays to prevent TypeErrors.
  return [...clientErrors, ...serverErrors].length;
}
numberOfErrors(clientErrors, serverErrors); // => 5

The function numberOfErrors concatenates two arrays and returns the length of the new array. But what if the parameters are falsy values, like null or undefined?

const clientErrors = ['err1', 'err2', 'err3'];
const serverErrors = null;
function numberOfErrors(clientErrors, serverErrors) {
  return [...clientErrors, ...serverErrors].length;
}
numberOfErrors(clientErrors, serverErrors);
// => TypeError

We know that if we try to spread a null or undefined variable, this will make the interpreter nag. In real world scenarios, we want to guard ourselves from such edge cases. With a minor tweak, we end up writing something like this:

const clientErrors = ['err1', 'err2', 'err3'];
const serverErrors = null
function numberOfErrors(clientErrors, serverErrors) {
  return [...(clientErrors || []), ...(serverErrors || [])].length;
}
numberOfErrors(clientErrors, serverErrors) // => 3

Because serverErrors is falsy, the logical OR operator will return an empty array, which then will be spread gracefully. The final result from calling numberOfErrors is equal to the length of the clientErrors array, which is 3.

Spread Operator Precedence

Now that we covered a basic example, let's see something more interesting. For each of the following questions, mark the correct answer. The solutions will be presented immediately after. (Hint: You can run the code snippets and see the results yourself!)

Question A

const a1 = null;
const b1 = [1, 2];
const c1 = [...a1 || b1];

What is the value of c1?

c1 has no value. The expression ...a1 will throw TypeError, because a1 is null.
c1 is [1, 2]. The expression a1 || b1 will be evaluated first, which then will return [1, 2], which will be spread.

Question B

const a2 = [1, 2];
const b2 = null;
const c2 = [...a2 || b2];

c2 is [1, 2]. The expression a2 || b2 will be evaluated first, which will be spread.
c2 is [1, 2]. The expression …a2 will be evaluated first, which will be spread.

Question C

const a3 = null;
const b3 = null;
const c3 = [...a || b];

c3 has no value. The expression ...a3 will throw TypeError, because a3 is null.
c3 has no value. The expression a3 || b3 will evaluate first which will return null and then the Spread Syntax will throw TypeError.

Answers

A. 2
B. 1 
C. 2

If it happens that you have not answered correctly to at least one of the above questions, then you might have fallen into the trap of the operator precedence. Does the dots punctuator … have higher precedence over the logical OR ||, or is the other way around? What is the precedence of the Spread Operator? The correct answers is: It does not matter, because there is no such thing as Spread Operator in JavaScript!

The Spread Operator does not exist!

When we try to evaluate expressions like […array || []] it's logical to examine the precedence of our operators. There exist a common misconception in the Web regarding the Spread Syntax, which is presented as an operator.

A great answer was posted in Stack Overflow by Andrew Li, which is worth mentioning and summarizes the nature of the Spread Syntax.

One of the most memorable arguments can be retrieved directly by the ECMAScript 2015 specification itself:

The complete list of operators is listed in Clauses §12.5 through §12.15 in the ECMAScript 2015 Language Specification, the specification in which … is introduced, which doesn't mention .... - Andrew Li answer

Another worth-mentioning point is that "An operator is a builtin function [..] that **evaluates to exactly one value.". If we try to run a statement like const a = …b in our Web Console, where b is an array, then we'll SyntaxError.

The way that the Spread Syntax works, is by evaluating its arguments first, and then spreading the result. Thus, […a || b] behaves exactly the same way as […(a || b)]. Putting a set of parentheses around a || b expression helps to remove the ambiguity.

As a practical reference, Spread Syntax's arguments are evaluated first and then spread.

An Example-Based Introduction to Finite-State Machines

George Gkasdrogkas — Sat, 08 Feb 2020 16:01:33 +0000

If you are enrolled in a CS program or happen to be fascinated by the aspects of theoretical Computer Science, you would most likely be introduced at some point to finite-state machines. It also might be the case that you were overwhelmed by their principles. But considering how ubiquitous this simple computational model is, it's time to unravel its mysteries once and for all!

An everyday example

It's Saturday morning. You'll wake up early today to enjoy the benefits of a morning walk, along the seaside. The clock rings at 8 am. Your eyes open, but your body feels quite relaxed under the sheets. For the next few moments, the motion of the second hand in the alarm clock is ticking rhythmically in your ears.

You finally decide to leave the bed and get ready. It's time to go, you're inserting the key in the front door to lock it, but the key does not match. You're using the wrong key! It doesn't take more than a few seconds to find the correct one and with a gentle motion, you turn the key to the right.

The beach is not far from home, but quite enough to make your head quickly to the driver's seat and start the engine. The alarm beeps reminding you to fasten the seat belt. You comply successfully with the demands of your vehicle.

Heading to your favorite spot, where the sea breeze leaves a gentle, liquid sense in the air, way off the crowded area you live in, the yellow color of the traffic lights suggests slowing your speed. After two seconds, the traffic lights turn red, commanding you to stop. On your left, two people are buying from the vending machine near the coffee shop, where a couple is playing a board game. The lights turn green and you continue the ride to the desired location.

When you finally reach it, all the previous images start crossing your mind:

the alarm clock
the key hole
the car commanding you to fasten the seat belt
the traffic lights
the vending machine
the board game

You suddenly make the connection! They all represent some kind of finite-state machine.

Finite-State Machines (FSMs)

Think of a computer with extremely limited memory capacity. A device with such limited resources can seem inefficient at first glance but don't rush to a conclusion yet. What makes a machine like this special in a world where every computer tends to be big, complicated and fancy, is the simplicity of its design.

You see, modern computers are extremely complex to allow us to create a feasible mathematical theory of them directly. Sometimes scientists need only the basics to work on a solution. For that reason, we created an idealized computer, a computational model. The simplest model is called finite-state machine (FSM).

We can visualize one using state diagrams. Look at the image below:

For those who know the basics of graph theory, this is clearly a directed graph. We can see the nodes, represented by circles, and edges, represented by arrows. But unlike graphs, in state diagrams, we call them states and transitions respectively.

States represent the status of a system. For instance, automatic sliding doors in supermarkets have two states: either close or open. Traffic lights contain three states: red, yellow, green. More sophisticated examples can contain quite a lot of states, but never an infinite number. Having no states doesn't make sense either.

When a finite-state machine is running, one or many states are active in each execution step. Those active states represent the current value of the system.

In the first execution step, a default state, called the starting state, is being activated. In state diagrams, those states are indicated by an arrow pointing to them. In the above diagram, state s is the starting state.

Finite-state machines may also contain one or multiple accepting states. Accepting states are represented by a double circle, like states p and r. A finite-state machine is not required to have accepting states; it might be designed to run indefinitely. The purpose of accepting states is quite straight forward: when the processing ends, depending on if we are on any accepting states or not, the output of the finite-state machine is either accept or reject.

States are connected with each other using transitions. Each transition contains a set of symbols. The set of all symbols presented in the FSM is called the alphabet. When the source state is active and the transition is executed, it changes the active state from the source state to the destination state only if the current input matches the symbol of the transition.

It's like using a vending machine:

The active state when you first approach the machine is idle, which happens to also be the starting state. By inserting coins, you move to the next state choose product.

To recap, finite-state machines:

have a set of states and transitions
have at least one state
cannot have an infinite number of states
have one starting state, which is the first state that is activated upon their execution
may or may not have accepting states
have for each transition a set of symbols,
have an alphabet which is the union of all the sets of transition symbols

There are two types of finite-state machines: deterministic and non-deterministic. Based on those types, the execution is handled differently. Remember that finite-state machines are a computational model: they calculate an output based on the provided input. Below we discuss these two types and provide a complete running example for each one.

Deterministic Finite Automatons (DFAs)

The first type of finite-state machine is the Deterministic Finite Automaton (DFA). The word deterministic has quite a value here. It means that you can transition from one state to the next while not being in multiple states concurrently. Think the example with the traffic signals: they can't be in green and red state at the same time.

Let's take a look at the bellow automaton:

The above DFA contains 4 states. Starting state is s and accepting state is r. The alphabet contains the symbols a, b, and c.

Observe that for each state, there is only one transition arrow for each symbol. In some cases, a transition arrow can contain multiple symbols, like the one from q to r, but this does not break the requirement as there is no other exiting transition from q with the same symbols (b and c).

In deterministic finite automatons, the transition from one state to another can happen only if the input matches the symbol(s) of the transition. For instance, we can move from s to q only if the current symbol of the input is b.

A running example

But enough with the theory! Let us see what the above DFA outputs for the input baabb.

When the execution begins, the starting state is being activated. To run the automaton, for each symbol in the input, from left-to-right, we're going to see if there is an exiting transition from the active state that match this symbol.

The first symbol of the input baabb is b. The current active state is s. Is there any exiting transition from s that contains the symbol b? Yes, the one that connects s to q.

The current active state is q. The next symbol of the input baabb is a. Is there any exiting transition from q that match the symbol a? Yes, the one that connects q with p.

The current active state is p. The next symbol of the input baabb is a. Is there any exiting transition from p that match the symbol a? Yes, the one that connects p with itself.

The current active state is p. The next symbol of the input baabb is b. Is there any exiting transition from p that match the symbol b? Yes, the one that connects p with q.

The current active state is q. The next symbol of the input baabb is b. Is there any exiting transition from q that match the symbol b? Yes, the one that connects q with r.

At this point we have traverse all the symbols of the input. The final active state is r. Is r an accepting state? Yes it is! That means the output is accept. In case that after traversing the input the final state was not an accepting state, the output would be reject.

Formal definition

It would be a shame not to mention the formal definition of a DFA, now that we have covered all the required details to do so.

A deterministic automaton is a 5-tuple ⟨Q, Σ, δ, q₀, F⟩, where

Q is a finite set of all states
Σ is a finite set of all symbols, the alphabet
δ: Q × Σ → Q is the transition function
q₀ ∈ Q is the starting state
F ⊆ Q is the set of accepting states

What may bother you in the above definition, is the transition function. Remember that we can move from one state to another only if the current input matches the symbol of the transition. For instance, consider that a finite automaton has a transition labeled 1 from a state p to a state q, then we can indicate the same thing with a transition function δ(p, 1) = q. This notion is just a kind of mathematical shorthand.

Non-deterministic Finite Automatons (NFAs)

The second type of finite-state machine is the Non-deterministic Finite Automaton (NFA). Compared to deterministic finite automatons, there are some major differences.

Consider the following automaton:

The above NFA contains 5 states. Starting state is s and accepting states are v and d. The alphabet contains the symbols a, b and c.

But wait a moment! What is this strange label ε? It is a Greek character called epsilon and is not part of the alphabet. In non-deterministic automatons, transitions can be labeled either with a symbol from the alphabet or ε.

In non-deterministic automatons, there are cases where more than one transitions are applicable. In those cases, the automaton splits into multiple copies of itself and continues to run in non-determinism manner like before. We can see this behavior as a kind of parallel computation, where the machine is forking into several children, each processing independently.

Label ε is the first difference between deterministic and non-deterministic automatons. The second difference is the use of multiple transition arrows that contain the same symbols for each state in the NFA. For instance, state s has two exiting transitions labeled with the symbol a. This is the opposite of a DFA which has only one symbol per exiting transition for each state.

A running example

A running example will clear any misconceptions.

When the execution begins, the starting state is being activated. We're going to check if there are any ε transition from s.

Is there any ε transition from s? Yes, the one that connects s to w. Activate state w.

For the new state, we'll do the same procedure until no more ε transitions exist.

Is there any ε transition from w? No, there isn't.

The initial active states are s and w.

For each symbol of the input, from left-to-right, we'll check if there are exiting transitions from the active states that match that symbol. Then we'll follow the previous process and activate all the states that connect with epsilon transitions.

The first symbol of the input aabc is a. The current active states are s and w.

Any exiting transition from s that contains the symbol a? Yes, the ones that connect s to w and a.
Any exiting transition from w that contains the symbol a? No, there isn't.

For the new activated states w and a, we're going to follow all the epsilon transitions and repeat until we run out of them:

Any ε transition from w? No, there isn't.
Any ε transition from a? Yes, the one that connects a to v.
Any ε transition from v? Yes, the one that connects v to itself. Because this transition forms a self loop, we won't test v again.

The new active states are w, a and v.

The next symbol of the input aabc is a. The current active states are w, a and v.

Any exiting transition from w that contains the symbol a? No.
Any exiting transition from a that contains the symbol a? Yes, the ones that connect a to d and v.
Any exiting transition from v that contains the symbol a? No.

For the new activated states d and v, we're going to follow all the epsilon transitions and repeat until we run out of them:

Any ε transition from d? No.
Any ε transition from v? Yes, the one that connects v to itself. Because this transition forms a self loop, we won't test v again.

The new active states are d and v.

The next symbol of the input aabc is b. The current active states are d and v.

Any exiting transition from d that contains the symbol b? Yes, the one that connect d to s.
Any exiting transition from v that contains the symbol b? Yes, the one that connect v to a.

For the new activated states s and a, we're going to follow all the epsilon transitions and repeat until we run out of them:

Any ε transition from s? Yes, the one that connects to w.
Any ε transition from w? No.
Any ε transition from v? Yes, the ones that connects v to itself and a.
Any ε transition from a? Yes, the one that connects to v. Because we already tested v we won't test it again.

The new active states are s, w, a and v.

The last symbol of the input aabc is c. The current active states are s, w, a and v.

Any exiting transition from s that contains the symbol c? No.
Any exiting transition from w that contains the symbol c? No.
Any exiting transition from a that contains the symbol c? No.
Any exiting transition from v that contains the symbol c? Yes, the one that connects to a.

For the new activated state a, we're going to follow all the epsilon transitions and repeat until we run out of them:

Any ε transition from a? Yes, the one that connects to v .
Any ε transition from v? Yes, the one that connects v to itself.

The final active states are a and v. Is any of them an accepting state? Yes, state v is an accepting state. That means the output is accept. In case that after traversing the input the final state was not an accepting state, the output would be reject.

Formal definition

The formal definition of a non-deterministic automaton is similar to the definition of a deterministic one. It consists of states, transitions, a transition function, an alphabet, and some accepting states. The only difference is the kind of transition function.

A non-deterministic automaton is a 5-tuple ⟨Q, Σ, δ, q₀, F⟩, where

Q is a finite set of all states
Σ is a finite set of all symbols, the alphabet
δ: Q × Σε → P(Q) is the transition function
q₀ ∈ Q is the starting state
F ⊆ Q is the set of accepting states

In deterministic automatons, the transition function takes as input a state, a symbol and returns the next state, as output. In non-deterministic automatons, the transition function takes as input a state, a symbol or ε and returns the set of all possible next states, as output.

Conclusion

This article was intended to provide you with all the required details and examples to get a better understanding of Finite-state Machines. Thank you for taking the time to read it. 🙏

Anonymous Web Scraping with Node.js, Tor, Puppeteer and cheerio

George Gkasdrogkas — Sat, 25 Jan 2020 20:47:27 +0000

Web Scraping is the technique of extracting data from websites. The term is used typically for automated data extraction. Today, I am going to show you how to crawl websites anonymously. The reason why you want to hide your identity is due to the fact that many web servers apply rules to websites which ban IPs after a certain amount of continuous requests. We are going to use Puppeteer for accessing web pages, cheerio for HTML parsing, and Tor to run each request from a different IP address.

While the legal aspects of Web Scraping vary, with many grey zones,
remember to always respect the Terms of Service of each web page you
scrape. Ben Bernard has wrote a nice article about those legal issues.

Setting Up Tor

First things first, we have to install our Tor client by using the following command.

sudo apt-get install tor

Configure Tor

Next, we are going to configure our Tor client. The default Tor configuration uses a SOCKS port to provide us with one circuit to a single exit node (i.e. one IP address). This is handy for everyday use, like browsing, but for our specific scenario we need multiple IP addresses, so that we can switch between them while scraping.

To do this, we'll simply open additional ports to listen for SOCKS connections. This is done by adding multiple SocksPort options to main configuration file under /etc/tor.

Open /etc/tor/torrc file with your preferred editor and add the next lines in the end of the file.

There a couple of things to notice here:

The value of each SocksPort is a number, the port that Tor will listen for connections from SOCKS-speaking applications, like browsers.
Because SocksPort value is a port to be open, the port must not already be used by another process.
The initial port starts with value 9050. This is the default SOCKS of the Tor client.
We bypass value 9051. This port is used by Tor to allow external applications who are connected to this port to control Tor process.
As a simple convention, to open more ports, we increment each value after 9051 by one.

Restart the tor client to apply the new changes.

sudo /etc/init.d/tor restart

Create a new Node project

Create a new directory for your project, I’ll call it superWebScraping.

mkdir superWebScraping

Navigate to superWebScraping and initialize an empty Node project.

cd superWebScraping && npm init -y

Install the required dependencies.

npm i --save puppeteer cheerio

Browse with Puppeteer

Puppeteer is a headless browser that uses DevTools Protocol to communicate with Chrome or Chromium. The reason why we don’t use a request library, like tor-request, is due to the fact that request libraries cannot process SPA websites that load their content dynamically.

Create an index.js file and add the below script. The statements are documented inline.

Run the script with

node index.js

You should see the Chromium browser navigating to https://api.ipify.org like the following screenshot

There is a reason why I chose the specific web page in my example. https://api.ipify.org is able to send us our public IP address. This is the IP you are browsing the web, without using Tor.

Change the above code by adding the following key in puppeteer.launch statement:

We provide the --proxy-server argument to our browser. The value of this argument tells the browser to use a socks5 proxy in our local machine on top of port 9050. The value of the port is one of the values we provided earlier in torrc file.

Now run again the script.

node index.js

This time you should see a different IP address. This is the IP that comes with Tor circuit.

Mine is 144.217.7.33, but you might have a different one. Note that if you run the script again on the same port 9050 you'll get the same IP address as before.

This is the reason why we opened many ports in our Tor configuration. Try to use another port, like 9051. The IP will not be the same.

Scrape content with Cheerio

Now that we have a nice way to get our pages, it's time to scrape them. We are going to use the cheerio library. Cheerio is an HTML parser designed to use the same API as jQuery. Our task is to scrape the last 5 post titles of Hacker News.

Let's navigate to Hacker News.

We want to scrape the first 5 titles ("HAKMEM (1972), "Larry Roberts has died", etc). Inspecting the title of an article using my browser's DevTools I can see that each article is being wrapped with an HTML link element which has storylink class.

The procedure we are going to follow can be denoted by the bellow list:

Launch a browser instance in headless mode with Tor proxy
Create a new page
Navigate to https://news.ycombinator.com/
Get page HTML content
Load the HTML content in Cheerio
Create an array to save the article titles.
Access all the elements that have storylink class
Get only the first 5 such elements, using Cherrio's slice() method.
Iterate through those 5 elements using Cherrio's each() method.
Append each article title in the array.

And here is the output of the above code.

Continuous scraping with different IP

The last thing to consider is to take advantage of all the SOCKS ports we defined in torrc file. This is quite easy. We’ll define an array with each item to be a different port number. Then we’ll rename main() to scrape() and we will define a new main() function that will call scrape() each time with a different port.

Here is the complete code.

Conclusion

This article was intended to provide you with a gentle introduction to Web Scraping. Thank you for taking time to read it. 🙏