DEV Community: Nya

RxJS Lessons: Understanding Multicasting Operators

Nya — Tue, 02 Feb 2021 13:17:07 +0000

Introduction

When it comes to RxJS, multicasting is probably one of its most daunting features. Multicasting requires a prior RxJS knowledge base and is, in itself, a feature which some programmers struggle to grasp. To this we add that there exist quite a few multicasting operators, and that knowing which one to pick can prove to be quite difficult if we don't understand how they each work, and which particular problem they solve.

This is an introductory guide to the RxJS multicasting operators. By the end of this guide, you'll (hopefully) understand these operators a little better, and know when and why you need to use each one. We will start by taking a look at the hot/cold Observable concept, and we'll then make our way through all of the multicasting operators, discussing their particularities and looking at examples, until we understand them all.

I've created a StackBlitz project for each and every one of the examples that we'll see in this guide, so that you can play around with the code snippets, since, in my humble opinion, this considerably helps to understand how the operators work. You can find the link to the corresponding StackBlitz project in the caption below each example.
Before we get started, here is a list of all the operators that we'll cover today:

Hot & cold Observables

Note: If you're already familiar with these concepts, feel free to skip this section and move on to the next.

Most Observables are cold by default. Each time we subscribe to a cold Observable, its producer is recreated. So what does this mean exactly? First, we must understand what a producer is: in a nutshell, it's the source of our Observable's values. It can be a DOM event, a callback, an HTTP request, an iterator and so on. In short, anything that can produce values and pass them to an observer.

Now that we know what a producer is, it'll be easier to understand the meaning of the previous statement, which basically comes to say that our Observable's producer is being created over and over again with each subscription. Let's take a look at an example:

StackBlitz

As you can see, because our Observable is cold, and its producer is being recreated with each subscription, the side effect is being executed twice, one time for each subscription. If the Observable were hot, the side effect would be performed only once, regardless of how many times we subscribe.

Some may think, after seeing the previous code, that the consequences derived from this behavior aren't particularly important, which is why I always like to explain this concept, and its considerable importance, by using an HTTP request as an example.

Let's imagine that we have an Ajax Observable, that gets some data for us. Because the Ajax Observable is cold, each time we subscribe to it, a new HTTP request is made. Yes, you read correctly, a new request is made for each subscription. 20 subscriptions = 20 HTTP requests. Let's take a look at some code:

StackBlitz

Upon seeing this, I believe that the importance of properly handling hot/cold Observables becomes exceedingly clear. Having the producer recreated with each subscription is something we most certainly don't want happening in our code. So, how do we fix this serious issue? By making our cold Observables hot. How do we do that? With multicasting operators! So, without further ado, let's get started on those operators.

Note: There is a wonderful article by Ben Lesh that treats the topic of hot/cold Observables in-depth. You can find it here.

multicast

multicast shares the source Observable by using a Subject. Let's take a look at an example using multicast:

StackBlitz

Have you tried the StackBlitz? Noticed anything strange? If we run the previous code, we won't receive any values at all! Why isn't our source Observable emitting anything?

Because multicast returns a special kind of Observable: a ConnectableObservable. This special type of Observable has a connect() method, which, when called, is responsible for subscribing to the source Observable with the Subject that we provided.

This means that if we don't call connect(), the source will never be subscribed to, and will never begin emitting values. So, let's change our previous code, adding a call to connect():

StackBlitz

Et voilà! Our code is now working as it should. Since multicast is sharing the source Observable, the side effect will only be executed once, even if we were to subscribe 1000 times.

Unsubscribing

As with all Observables, it's important to unsubscribe from our multicast Observables to avoid memory leaks. We need to bear in mind that, when dealing with multicasting operators that return a ConnectableObservable, we need to unsubscribe from the multicast subscription.

Let's take our previous code snippet, remove the take(2) from the source that was taking care of ending our Observable for us, and unsubscribe:

StackBlitz

Memory leak successfully avoided!

What about late subscribers?

In a perfect sandbox environment, all the subscriptions to the multicast Observable happen at the same time. However, what are the odds of finding this kind of behavior in real life? I can assure you that they are not very good. Let's take a look at a more realistic example, where we have different subscriptions occurring at different times:

StackBlitz

Well, it seems that our late observer is missing out on the values that were emitted before it subscribed to the multicast Observable. This can lead to unexpected behavior and major headaches trying to discover the cause.

So, how can we solve this problem? It's actually quite simple, all we have to do is use a ReplaySuject instead of a regular Subject. Since ReplaySubjects replay old values to new subscribers, our issue is effectively solved:

StackBlitz

Et, voilà, our late observers now have access to the previously emitted values.

publish()

We can all agree that multicast is an amazing operator, but having to type multicast(() => new Subject()) each time we want to multicast our streams can get a little verbose…
publish operator to the rescue! publish basically uses multicast plus a Subject under the hood, so that we don't have to go the trouble of typing it. Pretty cool, right? Let's take a look at an example:

StackBlitz

Remember, we still have to call connect() if we want our source Observable to be subscribed to!

The publish variants

Remember the issue we had with late subscribers and multicast? How do we deal with them in this case? Since publish is equivalent to using multicast(() => new Subject()), we can't just change the Subject for a ReplaySubject manually. Luckily for us, publish has several variants, one for every kind of Subject that there is. Let's take a look at them:

publishReplay()

publishReplay is equivalent to multicast(() => new ReplaySubject()). Since we've already seen an example with multicast + ReplaySubject, we know that it allows late subscribers to receive the emissions prior to their subscription:

StackBlitz

publishLast()

publishLast is equivalent to multicast(() => new AsyncSubject()). It will wait until the source Observable has completed to emit the last value. Here's an example:

StackBlitz

publishBehavior

publishBehavior is equivalent to multicast(() => new BehaviorSubject()). Since it uses a BehaviorSubject, publishBehavior allows us to specify an initial value:

StackBlitz

refCount()

We now know about several amazing operators to share our streams. However, having to call connect() gets old quick. It's verbose and, should we forget to call it, we'd probably end up wasting time figuring out why our Observables aren't emitting. So, isn't there a better alternative?

Of course there is! Allow me to introduce the refCount operator. refCount is in charge of counting the number of subscriptions to the source internally, which takes care of two crucial things for us:

If the number of subscriptions is larger than 0, a.k.a. there's at least one subscriber, refCount subscribes (once only) to the source, calling connect().
If the number of subscriptions is smaller than 1, a.k.a. there aren't any subscribers, refCount unsubscribes from the source.

Let's refactor our previous code to include refCount:

StackBlitz

As you can see, refCount takes care of calling connect() and of unsubscribing from the source Observable for us.

share()

Last, but not least, we have the share operator, which is equivalent to using multicast(() => new Subject()) + refCount. It's the easiest and most commonly used multicasting operator, since it takes care of everything under the hood. Here's our previous example, refactored to use share:

StackBlitz

Just for fun, let's take a look at a slightly more realistic example, featuring a shared Ajax Observable instead of a boring old interval:

StackBlitz

shareReplay()

Once again, we must remember our late subscribers. In this case, share only has one variant, shareReplay. As you can imagine, shareReplay is equivalent to multicast(() => new ReplaySubject()) + refCount. Here's an example:

StackBlitz

Wrapping up

💡 publish is equivalent to multicast(() => new Subject()).
💡 publishBehavior is equivalent to multicast(() => new BehaviorSubject()).
💡 publishLast is equivalent to multicast(() => new AsyncSubject()).
💡 publishReplay is equivalent to multicast(() => new ReplaySubject()).
💡 With refCount, we no longer have to manually call connect() nor do we have to take care of unsubscribing.
💡 share is equivalent to multicast(() => new Subject()), refCount().
💡 shareReplay is equivalent to multicast(() => new ReplaySubject()), refCount().

Conclusion

That's all folks! I hope this post helped you understand multicasting operators a little better, and realize that multicasting is not as hard as it initially seems.

As I mentioned earlier, I've created a StackBlitz project for each and every example in this guide, so feel free to play around with the code, I promise that it really helps to better understand how the operators work.

If you enjoyed this guide, feel free to leave a nice comment! If you have any questions, you can drop them in the comments and I'll try my best to answer.

RxJS Best Practices

Nya — Sat, 28 Mar 2020 17:14:34 +0000

RxJS Best Practices

RxJS is the most popular framework for reactive functional programming in
JavaScript. This means that a lot of people are using RxJS daily in their
projects. Most developers are aware of the common clean code practices, but…
What about RxJS best practices? Are you aware of the dos and don’ts when it
comes to functional reactive programming? Are you applying them in your code?

This tutorial will focus on several best practices that I use daily when writing
code, with practical examples. We will be covering the following points:

Avoid logic inside the subscribe function
Using Subjects to force completion
Avoid duplicated logic
Avoid nesting — Use chaining instead
Sharing to avoid stream duplication
Don’t expose Subjects
Use marble diagrams for testing

Without further ado, let’s get started:

Avoid logic inside the subscribe function

This statement may seem pretty obvious to some of you, but it’s a common pitfall for RxJS beginners. Until you learn how to think reactively, you may be tempted to do something like this:

Our pokemon$ Observable emits Pokemon objects, and, in a very non-reactive way, we are subscribing to it in order to access these objects and perform some actions, like returning early if the Pokemon type is Water, making a call to a getStats() function, logging the stats that this function returns and finally saving the data to the Pokedex. All our logic is inside the subscribe function.

However, doesn’t this code look exactly like something we would see in the traditional imperative programming paradigm? Since RxJS is a functional reactive programming library, we have to say goodbye to our traditional way of thinking, and start thinking reactively (streams! Pure functions!).

So, how do we make our code reactive? By using the pipeable operators that RxJS provides us with:

Et voilá, our code has gone from imperative to reactive with a few simple changes. It even looks cleaner, doesn’t it?

Note: I am fully aware of the fact that a part of the logic (the
saveToPokedex() function) still remains in the subscribe. I find that by keeping the last part of the logic inside the subscribe makes it easier for me to read the code. You are all free to choose whether or not to keep the subscribe completely empty :)

The operators we have used are pretty straightforward: filter and map work exactly the same as the Array operators that they share name with, and tap is used to perrform side effects.

Wikipedia: An operation, function or expression is said to have a side effect if it modifies some state variable value outside its local environment.

Using Subjects to force completion

Memory leaks are a real danger when it comes to using Observables. Why? Because, once we subscribe to an Observable, it will keep emitting values indefinitely until one of the following two conditions are met:

We manually unsubscribe from the Observable.
It completes.

Seems simple enough, right? Let’s take a look at how to unsubscribe from an Observable:

As you can see in the above example, we have to store the subscription of our pokemon$ Observable in a variable, and then manually call unsubscribe on that stored subscription. Doesn’t seem too difficult so far…

But what happens if we have more Observables that we need to subscribe to?

As you can see, as we add more Observables to our code, we need to keep track of more and more subscriptions, and our code starts looking a bit crowded. Isn’t there a better way for us to tell our Observables to stop emitting values? Luckily for us, there is, and it’s very, very simple:

We can use a Subject, together with the takeUntil() operator, to force our Observables to complete. How? Here’s an example:

Let’s understand what’s going on above. We’ve created a stop$ Subject, and we’ve piped our three Observables with the takeUntil operator. This operator is used for an Observable to keep emitting values, until a notifier Observable emits. Which means that our three Observables will stop emitting values when the stop$ Subject emits.

So how do we make our stop$ Observable emit? By calling the next()
function on it, which is exactly what we are doing inside our
stopObservables() function. Therefore, whenever we call our
stopObservables() function, our stop$ Observable will emit and all our Observables will automatically complete. Sounds cool, doesn’t it?

No more having to store any subscriptions and call unsubscribe, no more messing around with arrays? All hail the takeUntil operator!

Avoid duplicated logic

We all know that duplicated code is a bad sign, and something that should be
avoided. (If you didn’t know that, I would recommend that you go and read
this,
and then come back.) You may be wondering which scenarios could lead to having duplicate RxJS logic. Let’s take a look at the following example:

As you can see, we have a number$ Observable, which emits every second. We subscribe twice to this Observable: Once to keep score with scan() and once to call the getPokemonByID() function every ten seconds. Seems quite straightforward, but…

Notice how we have duplicated the takeUntil() logic in both Observables? This should be avoided, as long as our code allows it. How? By attaching this logic to the source observable, like this:

Less code && no duplication === Cleaner code. Awesome!

Avoid nesting — Use chaining instead

Nested subscriptions should be avoided at all cost. They make our code complex, dirty, difficult to test and can cause some pretty nasty bugs. What is a nested subscription, you may ask? It’s when we subscribe to an Observable in the subscribe block of another Observable. Let’s take a look at the following code:

Doesn’t look very neat, does it? The above code is confusing, complex, and, should we need to call more functions that return Observables, we’re going to have to keep adding more and more subscriptions. This is starting to sound suspiciously like a ‘subscription hell’. So, what can we do to avoid nested subscriptions?

The answer is to use higher order mapping operators. Some of these operators are: switchMap, mergeMap etc.

To fix our example, we are going to make use of the switchMap operator. Why? Because switchMap unsubscribes from the previous Observable, and switches (easy to remember, right?) to the inner Observable, which, in our case, is the perfect solution. However, please note that depending on which behavior you need, you may need to use a different higher order mapping operator.

Just look at how lovely our code looks now.

Sharing to avoid stream duplication

Ever had your angular code make duplicate HTTP requests and wondered why? Read on and you will discover the reason behind this widespread bug:

Most Observables are cold. This means that their producer is created and activated when we subscribe to them. This might sound a bit confusing, but is simple enough to understand. With cold Observables, every time we subscribe to them, a new producer is created. So if we subscribe to a cold Observable five times, five producers will be created.

So, what is a producer exactly? It’s basically the source of our Observable’s values (for example, a DOM event, an HTTP request, an array etc.) What does this imply for us reactive programmers? Well, if we, for example, subscribe twice to an observable that makes an HTTP request, two HTTP requests will be made.

Sounds like trouble.

The following example (borrowing Angular’s HttpClient) would trigger two
different HTTP requests, because pokemon$ is a cold Observable, and we are subscribing to it twice:

As you can imagine, this behavior can lead to nasty bugs, so, how can we avoid it? Isn’t there a way to subscribe multiple times to an Observable without triggering duplicated logic as its source its created over and over again? Of course there is, allow me to introduce: The share() operator.

This operator is used to allow multiple subscriptions to an Observable, without recreating its source. In other words, it turns an Observable from cold, to hot. Let’s see how it’s used:

Yes, that is really all we need to do, and our problem is ‘magically solved’. By adding the share() operator, our previously cold pokemon$ Observable now behaves as if it were hot, and only one HTTP request will be made, even though we subscribe to it twice.

A word of caution: Since hot Observables don’t replicate the source, if we subscribe late to a stream, we won’t be able to access previously emitted values. The shareReplay() operator can be a solution for this.

Don’t expose Subjects

It’s a common practice to use services to contain Observables that we reuse in our application. It’s also common to have Subjects inside such services. A common mistake many developers make is exposing these Subjects directly to the ‘outside world’, by doing something like this:

Don’t do this. By exposing a Subject, we’re allowing anyone to push data into it, not to mention that we are completely breaking the encapsulation of our DataService class. Instead of exposing our Subject, we should expose our Subject’s data.

Isn’t it the same thing, you may be wondering? The answer is no. If we expose a Subject, we are making all of its methods available, including the next() function, which is used to make the Subject emit a new value. On the other hand, if we just expose its data, we won’t make our Subject’s methods available, just the values that it emits.

So, how can we expose our Subject’s data but not it’s methods? By using the asObservable() operator, which transforms a Subject into an Observable. Since Observables do not have the next() function, our Subject’s data will be safe from tampering:

We have four different things going on in the above code:

Both our pokemonLevel and stop$ Subjects are now private, and therefore not accessible from outside our DataService class.
We now have a pokemonLevel$ Observable, that has been created by calling the asObservable() operator on our pokemonLevel Subject. This way, we can access the pokemonLevel data from outside the class, while keeping the Subject safe from manipulation.
You may have noticed that, for the stop$ Subject, we did not create an Observable. This is because we don’t need to access stop$’s data from outside the class.
We now have two public methods, named increaseLevel() and stop(). The latter is simple enough to understand. It allows us to make the private stop$ Subject emit from outside the class, thus completing all Observables that have piped takeUntil(stop$).
increaseLevel() acts as a filter and only allows us to pass certain values to the pokemonLevel() Subject.

This way, no arbitrary data will be able to find its way to into our Subjects,which are nicely protected inside the class.

Note: Please bear in mind that Observables have complete() and error() methods, which can still be used to mess with the Subject.

Remember everyone, encapsulation is the key.

Use marble diagrams for testing

As we all (should) know, writing tests is as important as writing code itself. However, if the thought of writing RxJS tests sounds a bit daunting to you… Fear not, from RxJS 6+, the RxJS marble testing utils will make life very, very easy for us. Are you familiar with marble diagrams? If not, here’s an example:

Even if you’re a newbie to RxJS, you should more or less understand these
diagrams. They’re all over the place, are quite intuitive, and make it quite easy to understand how some of the more complicated RxJS operators work. The RxJS testing utils allows us to use these marble diagrams to write simple, intuitive and visual tests. All you have to do is import TestScheduler from the rxjs/testing module, and start writing tests!

Let’s take a look at how it’s done, by testing our number$ Observable:

Since deep diving into marble testing isn’t the goal of this tutorial, I will only briefly cover the key concepts that appear in the above code, so we have a basic understanding of what’s going on:

TestScheduler: Is used to virtualize time. It receives a callback, which can be called with helper objects (in our case, the cold() and expectObservable() helpers.)
Run(): Automatically calls flush() when the callback returns.
- : Each - represents 10ms of time.
Cold(): Creates a cold Observable whose subscription starts when the test begins. In our case, we are creating a cold Observable that will emit a value very 10ms, and complete.
|: Represents the completion of an Observable.
Therefore, our expectedMarbleDiagram, expects for a to be emitted at 20ms.
The expectedValues variable contains the expected values of every item that is emitted by our Observable. In our case, a is the only value that will be emitted, and equals 10.
ExpectObservable(): Schedules an assertion that will be executed when the testScheduler flushes. In our case, our assertion expects the number$ Observable to be like the expectedMarbleDiagram, with the values contained in the expectedValues variable.

You can find more information about helpers etc. in the official RxJS
docs.

Advantages of using RxJS marble testing utils:

You avoid a lot of boilerplate code. (Jasmine-marbles users will be able to appreciate this.)
It’s very easy and intuitive to use.
It’s fun! Even if you aren’t a big fan of writing tests, I can guarantee you’ll enjoy marble testing.

Since I enjoy making all my code examples Pokemon-themed, I’ll throw in another spec, this time featuring a pokemon$ Observable test:

Conclusion

That’s all folks! Today we discussed a few of the RxJS best practices that I always take careful care to apply in my code. I hope that you found them useful, if you didn’t already know about them.

Do you know any more RxJS best practices? If you do, let me know in the comments below. This way we can all contribute to writing better and cleaner reactive code!

If you enjoyed this post, don’t forget to share with your friends/colleagues, and maybe give me a lil’ clap :) If you have any questions, don’t hesitate to ask, either in the comments, or by reaching out to me via Twitter. See you all in the next tutorial!

Contingency tables in R

Nya — Mon, 23 Mar 2020 19:07:14 +0000

Contingency tables in R

Introduction

Contingency tables are very useful when we need to condense a large amount of data into a smaller format, to make it easier to gain an overall view of our original data.

To create contingency tables in R we will be making use of the table() function. Since table() is so versatile, we can build contingency tables from something as simple as character strings, to something as complex as data frame objects.

Something that we have to keep in mind is the fact that as we add more data, or as the complexity of our data increases, our contingency table will grow more complex too.

As well as the creation of contingency tables, we will cover the calculation of marginal distributions, creating a frequency or percentage contingency table, and selecting fragments of our table. We will also learn three of the most popular techniques that are used to test for independence between variables, and how to calculate the measures of association between these same variables.

Here is the index for this tutorial:

Getting started - Loading data
Creating a table object
Contingency tables

Selecting parts of a contingency table
Frequencies in contingency tables
Calculating marginal distributions

Testing for Independence

The Chi-Square test
Fisher's Exact test
The G-Test

Measures of Association

The Phi Coefficient
Cramer's V and Tschuprow's T

Conclusion

Getting started - Loading data

Before we begin with contingency tables, I would like to briefly discuss creating table objects in R. This way, we will be able to truly appreciate the advantages that contingency tables have (when our goal is overall legibility) over traditional tables.

In order for us to be on the same page, I would propose using the following built-in R dataset: mtcars, although you can use a different one if you wish to.

Tip: To load a built-in R dataset, you can use data(), as follows: data(mtcars). To check if the data has been loaded correctly, simply type the name of the dataset, and you should see a preview on your console.

Once we have loaded our data, we can start building tables.

Creating a table object

Let us begin by taking a look at the number of gears that each car model has. To do this, we will use the $ syntax:

It's a bit difficult to read, isn't it? If we use table(), the information will be much more readable:

It looks a lot better this way, doesn't it? We can now tell, at a glance, that there are 15 cars with 3 gears, 12 cars with 4 gears and 5 cars with 5 gears.

Now let's take a look at the number of cylinders for each car. As previously, we will first show all the data at once, and then use table() to summarize it:

Simple enough, right? Let's take it one step further. What if we want to see the number of gears that each car has, in relation to the number of cylinders? Here is where contingency tables come in.

Contingency tables

Following the previous examples, we will create a contingency table showing the number of gears of each car, related to their number of cylinders. How do we do this? By once again, using table(), this time with two arguments: The first argument will correspond to the rows, and the second to the columns:

As you can see, each row corresponds to each of the gear values, which are 3,4 and 5. Each column corresponds to each of the cyl values, which are 4, 6 and 8. We can now easily know that there are 12 cars with 3 gears and 8 cylinders, or that there are no cars that have 4 gears and 8 cylinders (this can lead us to ask ourselves if perhaps this combination is mechanically incompatible?)

Selecting parts of a contingency table

Have you ever tried to select a table property by using the $ notation? Then you'll be familiar with the $ operator is invalid for atomic vectors error. Why does this happen? Well, objects of class table are essentially vectors (arrays) which means that we can't use the $ syntax. This syntax is only valid for recursive objects, and vectors are atomic objects, hence the error.

How do we interact with tables then? Here are several useful techniques to select and interact with tables:

str() - Examines and outputs the structure of a table object. The only drawback to this useful little function is that the output it provides us with is hard to interpret when you see it for the first time, which is why we are going to analyze what is going on with str() output.

Let's break down that nasty output, to be able to understand what's going on:

In the first line, str() is telling us that we have an object of type table: 'table', which is a matrix of integers, with three rows and three columns: int [1:3, 1:3], and finally shows us all the values contained in our matrix, grouped by columns 1 8 2 2 4...
The strange-looking attr(, "dimnames")* code fragment in the second line is there because R stores the names of the columns and rows in an attribute called dimnames (dimension names). It also tells us that this dimnames attribute is a list of two components: = List of 2.
The ..$ : in the beginning of the fourth line indicates the beginning of a component, which is a character vector containing three values: chr [1:3], and ends with a list of the three values contained in the vector: "3" "4" "5", which you may have noticed, are the names of the three rows in our table.
The fifth row is exactly the same as the fourth, save for the values contained in the vector, which are "4" "6" "8" and correspond with the names of the three columns of our table.

length() - Returns the length of the table object.

table[x,] - Returns the xᵗʰ row:

table[1:x,] - Returns the first x rows of the table:

table[1:x,1:y] - Returns the first x rows of the first y columns:

table["3",] - Returns the row named 3:

table[,"8"] - Returns the column named 8:

Frequencies in contingency tables

Sometimes, we would prefer to display frequencies. By using prop.table() together with table(), we can create a contingency table of frequencies:

Would you like to display percentages instead of frequencies? It's as simple as multiplying by 100, like this:

Calculating marginal distributions

Here are several useful functions for calculating marginal distributions in contingency tables:

addmargins() - Will calculate the total values of both columns and rows:

rowSums() - Will calculate the total values of each row:

colSums() - Will calculate the total values of each column:

Three-way contingency tables

So far we've worked with a two-way contingency table, but, what if we need to work with three variables? Is it possible? The answer is yes, it is. Let's create a three-way table based on the cyl and carb and gear variables:

As you can see, what is really going on is that we have three separate two-way contingency tables, one for each of the gear values. This third gear variable is known as a layer variable, or as a control variable.

Hint: Instead of using table(), we could use ftable(), for a more concise view of our data.

As you can see, at the leftmost of our three-way contingency table, we have our cyl rows, followed by the carb rows, and on top, we have our gear columns. Is it possible to create contingency tables with more variables? The answer is yes, but the complexity will increase as we add more and more variables. Here is an example of a four-way contingency table:

As you can see, it grows more complex as we increase the number of variables, and loses legibility in the process.

Testing for independence

We may want to test the independence of the row and column variables of our contingency table. To achieve this, there are several tests we can perform on our table:

The Chi-Square test

One of the simplest ways to test for independence is to use the Chi-Square test function: chisq.test()

The important value of this test is the p-value, which, as you can see, equals 0.001214. Since p-value is below the significance level (which is 5%, or 0.05), it looks like the variables are not independent. However…

You may have noticed the warning. This is because the value of at least one of the expected values in the contingency table is below 5.

Important: please bear in mind that expected values are not the values that appear in each cell, they are the result of calculating (row total*column total)/overall total). You can have low cell values in your table, yet have perfectly fine expected values, if you have a large enough sample.

Therefore, we shouldn't trust the result of this test, and should look for a different way to test for independence: in this case, we should use the Fisher test instead of the Chi-Square test. Before diving into the Fisher test, I'd like to introduce:

The Monte Carlo simulation

What happens if the expected values of our contingency table are below 5, but we can't use a different test and have to stick with the Chi-Square test? In these cases, the Monte Carlo simulation is a great alternative. It allows us to perform the Chi-Square test with simulated values, that attempt to make up for the low expected values of our table.To achieve this, all we need to do is set the simulate.p.value to true when calling chisq.test():

As you can see, the warning no longer appears, and the test has been performed with simulated p-value based on 2000 replicates, which is the default.

The Fisher Exact test

The general rule of thumb is, when at least one expected value is lower than five, use the Fisher test rather than the Chi-Square test. There are some who think that it may perhaps be time to retire this rule of 'no expected values lower than five' and use some other method to calculate whether or not a sample size is too small for the Chi-Square test.

Tip: You can find a great post talking about small numbers in Chi-square tests here.

Regardless of the debate, our mtcars sample is definitely too small for the Chi-Square test. Therefore, we will use the Fisher test. To perform this test in R, we can use the fisher.test() function:

Once again, the p-value is below the significance level (0.05). Therefore, we can conclude that the variables are not independent.

The G-test

There is yet another method we can use to test for independence, and that is the G-test. Once again, just like the Chi-Square test, it is only reliable when used with large samples. To be able to use the G-Test function in R, we must first import the DescTools package, and then use the GTest() function:

Tip: Remember, for small samples, use the Fisher test. For large samples, use the Chi-Square test or the G-Test

Measures of association

We now know how to test for independence between contingency table variables. However, how can we know how strong this independence is? By calculating association measures.

Phi Coefficient

The simplest way to calculate the strength of association in a 2x2 contingency table is by using the Phi Coefficient. This measure is related to the Chi-Square test for a two-way contingency table in the following way:

Where:

ϕ is the Phi Coefficient.
x² is the chi-squared statistic from the Chi-Square test.
n is the total number of observations.

The range of the Phi Coefficient is [-1, 1], and its interpretation is the following:

0 means there is no association between variables. They are independent.
1 means a perfect positive association. This is when most of the data falls along the diagonal cells.
-1 means a perfect negative relationship. This is when most of the data falls away from the diagonal cells.

Since we need a 2x2 table to perform this test, we will calculate the Phi Coefficient for the following contingency table:

To calculate the Phi Coefficient in R, we can use the phi() function in the psych package:

Since the result is negative, we know that we have a negative relationship. Since the result is close to 0, we know that the strength of the relationship is very weak, which translates to there being next to no relationship.

Tip: Remember that the Phi Coefficient can only be used for 2x2 contingency tables. For larger tables, we should use Cramer's V or Tschuprow's T.

Cramer's V and Tschuprow's T

Both Cramer's V and Tschuprow's T are closely related to the Chi-Square test. Because the latter is dependent on the size of the sample, it's a poor reflection of the strength of association. Here is where Cramer's V and Tschuprow's T come in:

Where:

φ is the Phi Coefficient.
x² is the chi-squared statistic from the Chi-Square test.
n is the total number of observations.
k is the number of columns.
r is the number of rows.

And here is Tschuprow's T:

Where:

x² is the chi-squared statistic from the Chi-Square test.
n is the total number of observations.
k is the number of rows.
c is the number of columns.

Both Cramer's V and Tschuprow's T, produce a value in the [0,1] range. The interpretation of this value is the following:

0 means there is no association between variables. They are independent.
1>x>0 (any number larger than 0, smaller than 1) means that there exists a relationship of association. The only problem with both Cramer's V and Tschuprow's T is that we have no way of knowing whether the obtained value represents a weak, intermediate or strong association.
1 means perfect dependency in the case of Tschuprow's T, but not in the case of Cramer's V, where it only means that there exists a high dependency.

To calculate Cramer's V in R, we need to use the CramerV() function from the DescTools package (since we already imported DescTools previously, for the G-Test, we don't need to do so again):

Since the value is larger than 0, we know that the variables are associated. However, we can't know whether or not that association is weak or strong…
To calculate Tschuprow's T in R, we can use the TschuprowT() function, once again from the DescTools package:

Bias correction

From A bias-correction for Cramér's V and Tschuprow's T: It's important to know that both Cramer's V and Tschuprow's T are considerably biased, and tend to overestimate the strength of association.

How do we avoid this? By applying a bias correction. Here is an example of Cramer's V with a bias correction applied in R:

And an example of Tschuprow's T with an applied bias correction in R:

Conclusion

And that's all for today, folks. We covered everything from the concept of a contingency table to independence tests and calculations of association measures. There are some widely used tests, like the Cochran-Mantel-Haenszel test for conditional independence or the Breslow-Day test for homogeneous association that we didn't mention, perhaps we could talk about them in another tutorial.

As always, I hope you enjoyed this tutorial and/or found it useful. Don't forget to let me know your thoughts in the comments below!

P.S. I am a full stack web developer with an interest in R. I initially wrote most of this material as part of my own learning process, and decided to publish it in case it helped others who are also learning R. Comments, tips and advice are welcome!

The Art of Refactoring: 5 tips to Write Better Code

Nya — Thu, 10 Oct 2019 20:16:16 +0000

Bad code works. We all know this. Developers have been writing code for years without giving a single thought to whether they were doing it right or not. It's understandable, isn't it? After all, we already have to deal with the stress of keeping up with the industry and the demands of our job…

The answer is no. Writing bad code comes at a price. Have you ever faced the issue of not understanding your own code after a couple of weeks, and having to spend hours, or even days figuring out what was going on?

The solution to this (extremely) common problem is to make your code as clear and informative as possible. I will go as far as to say that your code should be understood even by a non-technical person. It's time to drop our excuses, and improve the quality of your code.

Writing clean code isn't that complicated. This tutorial will show you 5 simple techniques to improve your code, with practical examples:

Get rid of switch statements
Make your conditionals descriptive
Use guard clauses to avoid nested if statements
Avoid code duplication
Functions should only do one thing

Get rid of switch statements

We normally use switch statements to avoid large if else if statements. However, switch statements are very verbose, hard to maintain and even harder to debug. They clutter up our code, and, in my humble opinion, have an odd, uncomfortable syntax. When adding more cases, we have to manually add each case and break statement, which is quite error-prone.

Let's take a look at an example of a switch statement:

Imagine that we need to add more cases to the switch statement. The amount of code that we would have to write is considerable. We would probably end up copy-pasting code, and we all know how that ends.

So, how do we avoid switch statements? By using an object literal. Object literals are simple, easy to write, read and maintain. We are all used to handling objects in JavaScript, and the syntax is a lot fresher than that of the switch statement. Here is an example:

As you can see, we can add a default value by using the || operator. If the type isn't found in the pokemon object, the getPokemon function will return 'Mew' as a default value.

Note: As you will probably have noticed, we declare the pokemon object outside the function, instead of inside. We do this to prevent it from being created each time we execute the function.

We can also use a map to achieve the same result. A map is a collection of key value pairs, just like an object. The difference is that map allows keys of any type, while objects only allow strings as keys. Also, map has an interesting series of properties and methods. You can read more about the map structure here.

Here's how to use map:

As you can see, our code looks a lot cleaner and straightforward when replacing switch statements with an object literal or map.

Make your conditionals descriptive

Conditional statements are an absolute necessity when writing code. However, they can quickly get out of hand, and end up being impossible to understand. This leads to either having to write comments explaining what the statement does, or having to spend valuable time tracing back our own code to understand what's going on. This is bad.

Take a look at the following statement:

If we only look at the code inside the if statement in the previous function, it's difficult to understand what's going on. Our code isn't clear, and unclear code leads only to technical debt, bugs, and significant headaches.

How can we improve our conditional? By extracting it into a function. Here's how:

By extracting the conditional into a function with a descriptive name: isGameLost(), our checkGameStatus function is now understandable at a glance. Why? Because our code is informative, it tells us what is going on, which is what we should always strive for.

Use guard clauses to avoid nested if statements

Nested if statements are one of the worst things we can encounter in code. I've seen nested ifs 10 levels deep… Believe me when I tell you that it was an absolute nightmare to be able to fully grasp what was going on in that code. Here's an example of a nested if statement (only three levels deep though, I'm not a monster):

You probably have to take a couple of minutes, and read up and down to follow the flow of the function. Nested if statements are hard to both read and understand. So, how do we get rid of the nasty nested if statement? By reversing the logic and using what we call a guard clause.

In computer programming, a guard is a boolean expression that must evaluate to true if the program execution is to continue in the branch in question. - Wikipedia

By reversing the logic of our function, and placing the conditions that cause an early exit in the beginning of the function, they will act as guards, and only allow our function to continue executing if all the conditions are met. This way, we can avoid else statements. Here's how to refactor our previous function to use guard clauses:

As you can see, the code is a lot cleaner and easier to understand. We can see what the function does simply by reading down, following the natural flow of the function, unlike before, where we had to read up and down.

Avoid code duplication

Duplicating code always ends badly. It leads to situations such as: "I fixed this bug here, but forgot to do it there" or "I need to make a change/add a new feature, and have to do it in five different places".
Just as the DRY (don't repeat yourself) principle states:

Every piece of knowledge or logic must have a single, unambiguous representation within a system.

Therefore, having less code is good: It saves us both time and effort, is easier to maintain, and reduces the chances of bugs appearing.

So, how do we get rid of duplicated code? The answer is not always simple, but extracting logic to functions/variables usually works just fine. Let's take a look at the following code, which I ran across when refactoring an application:

You'll probably have noticed that the for loop is exactly the same in both functions, except for one little detail: the type of news that we want, which is either JavaScript or Rust news. To avoid this duplication, we can extract the for loop into a function, which we then call from the getJavascriptNews, getRustNews and getGolangNews functions. Here's how:

After extracting the for loop into the getNewsContent function, our getJavascriptNews, getRustNews and getGolangNews functions have turned into simple, clear one liners.

Further refactoring

However, have you realized that, once again, both functions are exactly the same except for the type string that we pass into the getNewsContent function? This is something that usually happens when we refactor our code. More often than not, one change leads to another change, and so on, until our refactored code ends up being half the size of the original. Let your code tell you what it needs:

Further refactoringWhere did our getJavascriptNews, getRustNews and getGolangNews functions go? We substituted them for a getNews function, which receives the type of news as an argument. This way, no matter how many more types of news we add, we always use the same function. This is called abstraction, and allows us to reuse functions, thus being incredibly useful. Abstraction is one of the techniques I use most frequently in my code.

Bonus: Make the for loop more readable with ES6 features

This is the last refactoring, I swear.
For loops aren't precisely readable. With the introduction of ES6 Array functions, we can avoid using them 95% of the time. In our case, we can use Array.filter combined with Array.map to substitute the original loop:

With Array.filter we return only the elements whose type equals the type passed as an argument.
With Array.map, we return only the content property of the item object, instead of the whole item.

Congratulations, after three simple refactorings, our initial three functions have been reduced to two, which are much easier to understand and maintain. Also, by using abstraction, we made the getNews function reusable.

Functions should only do one thing

Functions should only do one thing, and one thing only. Functions that do more than one thing are the root of all evil, and one of the worst things we can encounter in code (together with nested ifs). They are messy, and make our code hard to understand. Here's an example of a complex function from a real application:

Note: Since the handlers for the event listeners were not needed for this example, I chose to remove them.

As you can see, it's confusing, and hard to understand what's going on in there. If any bugs come up, it will be quite difficult to find and fix them. How can we improve our startProgram function? By extracting common logic into functions. Here's how:

Let's go through the changes made to the startProgram function:

First, we got rid of the if else statement by using a guard clause. Then, we extracted the logic needed to start the database into an initDatabase function and the logic to add event listeners to a setListeners function.

The logic for printing the employee list is slightly more complex, so we created three functions: printEmployeeList, formatEmployeeList, and getEmployeeList.

The getEmployeeList is responsible for making a GET request to employeeList.json, and returning the response in json format.

It is then called by the printEmployeeList function, which takes the list of employees, and passes it to the formatEmployeeList function, which formats and returns it. Then, the list is printed.

As you can see, every function is responsible for doing only one thing.

We could still make a few more changes to the function, and honestly, the application is begging for the separation of the view from the controller, but on the whole, our startProgram function is now more informative, and there is absolutely no difficulty in understanding what it does. We would have no problem at all if we had to come back to this code after a couple of months.

Conclusion

Programmers are the only ones responsible for writing good, quality code. We should all make it a habit to write good code from the very first line. Writing clean code isn't complicated, and doing so will help both you and your colleagues.

By applying the 5 simple techniques shown in this tutorial, your code quality should improve considerably, and so will your productivity.

If you have any questions, don't hesitate to ask. Thank you for reading.

Understanding the JavaScript Spread Operator - From Beginner to Expert Part 2

Nya — Wed, 25 Sep 2019 17:18:11 +0000

Introduction

The spread operator, …, was first introduced in ES6. It quickly became one of the most popular features. So much so that despite the fact it only worked on arrays, a proposal was made to extend its functionalities to objects. This feature was finally introduced in ES9.

The goal of this tutorial, which is divided into two parts, is to show you why the spread operator should be used, how it works, and to deep dive into its uses, from the most basic to the most advanced. If you haven't read the first part of this tutorial, I encourage you to do so! Here is the link:

Understanding the JavaScript Spread Operator - From Beginner to Expert

Here is a short summary of the contents of this tutorial:

Part 1

Why the spread operator should be used
Cloning arrays/objects
Converting array-like structures to array
The spread operator as an argument
Adding elements to arrays/objects
Merging arrays/objects

Part 2

Destructuring nested elements
Adding conditional properties
Short circuiting
The rest parameter (…)
Default destructuring values
Default properties

Cloning arrays/objects with nested elements

In the first part of this article, we learnt about reference data types, accidental variable mutation, and how we could solve this problem by cloning arrays/objects immutably, with the spread operator.

However, there's a slight problem with this approach, when it comes to nested reference data types: The spread operator only performs a shallow clone. What does this mean? If we attempt to clone an object that contains an array, for example, the array inside the cloned object will contain a reference to the memory address where the original array is stored… This means that, while our object is immutable, the array inside it isn't. Here's an example to illustrate this:

As you can see, our squirtleClone has been cloned immutably. When we change the name property of the original pokemon object to 'Charmander', our squirtleClone isn't affected, its name property isn't mutated.

However, when we add a new ability to the abilities property of the original pokemon object… Our squirtleClone's abilities are affected by the change. Because the abilities property is a reference data type, it isn't cloned immutably. Welcome to the reality of JavaScript :)

One of the solutions to this problem would be to use the spread operator to clone the nested properties, as shown in the following example:

For obvious reasons, this isn't an ideal approach to solving our problem. We would need to use the spread operator for every single reference type property, which is why this approach is only valid for small objects. So, which is the optimal solution? Deep cloning.

Since there is plenty to say about deep cloning, I won't be going into too much detail. I'd just like to say that the correct of deep cloning is either using an external library (for example, Lodash), or writing ourselves a function that does it.

Adding conditional properties

Sometimes we need to add properties to an object, but we don't know whether or not those properties exist. This doesn't pose much of a problem, we can always check if the property exists with an if statement:

There is, however, a much simpler way of achieving the same result, by using short circuiting conditionals with the && operator. A brief explanation:

Short circuiting

When we evaluate an expression with &&, if the first operand is false, JavaScript will short-circuit and ignore the second operand.

Let's take a look at the following code:

If starterPokemon.length > 0 is false (the array is empty), the statement will short circuit, and our choosePokemon function will never be executed. This is why the previous code is equivalent to using the traditional if statement.

Going back to our original problem, we can take advantage of the logical AND operator to add conditional properties to an object. Here's how:

What's going on here? Allow me to explain:

As we already know, by using the && operator, the second part of the statement will only be executed if the first operand is true. Therefore, only if the abilities variable is true (if the variable exists), will the second half of the statement be executed. What does this second half do? It creates an object containing the abilities variable, which is then destructured with the spread operator placed in front of the statement, thus adding the existent abilities variable into our fullPokemon object immutably.

Before we can introduce our final advanced spreading use, adding default properties to objects, we must first dive into two new concepts: default destructuring values, and the rest parameter. Once we are familiar with these techniques, we will be able to combine them to add default properties to objects.

Default destructuring values

If we try to destructure an array element or object property that doesn't exist, we will get an undefined variable. How can we avoid undefined values? By using defaults. How does this work?

We can assign default values to the variables we destructure, inside the actual destructuring statement. Here's an example:

As you can see, by adding the default value 'Water' to the type variable in the destructuring statement, we avoid an undefined variable in the case of the pokemon object not having the type property.

The rest parameter (…)

You may be surprised to hear that the spread operator is overloaded. This means that it has more than one function. It's second function is to act as the rest parameter.

From the MDN docs: A function's last parameter can be prefixed with ... which will cause all remaining (user supplied) arguments to be placed within a "standard" javascript array. Only the last parameter can be a "rest parameter".

Simply put, the rest operator takes all remaining elements (this is the reason it's named rest, as in the rest of the elements :p ) and places them into an array. Here's an example:

As you can see, we can pass as many abilities as we want to the printPokemon function. Every single value we introduce after the type parameter (the rest of the parameters) will be collected into an array, which we then turn into a string with the join function, and print out.

Note: Remember that the rest parameter must be the last parameter, or an error will occur.

The rest parameter can also be used when destructuring, which is the part that interests us. It allows us to obtain the remaining properties in an object, and store them in an array. Here's an example of the rest parameter used in a destructuring assignment:

As shown above, we can use the rest operator to destructure the remaining properties in the pokemon object. Like in the previous example, our pokemon object can have as many properties as we want defined after the id property, they will all be collected by the rest parameter.

Now that we know how the rest parameter works, and how to apply it in destructuring assignments, let's return to dealing with default properties.

Adding default properties

Sometimes, we have a large amount of similar objects, that aren't quite exactly the same. Some of them lack properties that the other objects do have. However, we need all of our objects to have the same properties, simply for the sake of order and coherence. How can we achieve this?

By setting default properties. These are properties with a default value that will be added to our object, if it doesn't already have that property. By using the rest parameter combined with default values and the spread operator, we can add default properties to an object. It may sound a bit daunting, but it's actually quite simple. Here's an example of how to do it:

What's going on in the previous code fragment? Let's break it down:

As you can see, when we destructure the abilities property, we are adding a default value ([]). As we already know, the default value will only be assigned to the abilities variable if it doesn't exist in the pokemon object. In the same line, we are gathering the remaining properties (name and type) of the pokemon object into a variable named rest, by making use of the awesome rest parameter.

On line 7, we are spreading the rest variable (which, as you can see, is an object containing the name and type properties) inside an object literal, to generate a new object. We are also adding the abilities variable, which in this case, is an empty array, since that is what we specified as its default value on the previous line.

In the case of our original pokemon object already having an abilities property, the previous code would not have modified it, and it would maintain its original value.

So, this is how we add default properties to an object. Let's place the previous code into a function, and apply it to a large collection of objects:

As you can see, all the pokemon in the array now have an abilities property. In the case of charmander and bulbasur, they have an empty array, since that is the default value we assigned. However, the squirtle object maintains its original array of abilities.

There are, of course, other ways of adding default properties to an object, mainly by using if statements. However, I wanted to show an interesting new way of doing it, by using a combination of default values, the rest parameter, and the spread operator. You can then choose which approach suits you best :)

Conclusion

This is the second and final part of the Understanding the JavaScript Spread Operator - From Beginner to Expert tutorial. Here's a link to the first part.

In this second part of the tutorial we have learnt some more advanced uses of the spread operator, which include destructuring nested elements, adding conditional properties and adding default properties. We have also learnt three interesting JS concepts: short-circuiting, default destructuring values and the rest parameter.

I sincerely hope you've found this piece useful, thank you for reading :) If you can think of any more uses of the spread operator or would like to comment something, don't hesitate to reach out, here's a link to my Twitter page.

Understanding the JavaScript Spread Operator - From Beginner to Expert

Nya — Mon, 16 Sep 2019 19:54:00 +0000

Introduction

The spread operator '…' was first introduced in ES6. It quickly became one of the most popular features. So much so, that despite the fact that it only worked on Arrays, a proposal was made to extend its functionalities to Objects. This feature was finally introduced in ES9.

The goal of this tutorial, which is divided into two parts, is to show you why the spread operator should be used, how it works, and deep dive into its uses, from the most basic to the most advanced.

Here is a short summary of the contents of this tutorial:

What we are going to learn

Part 1

Why the spread operator should be used
Cloning Arrays/Objects
Converting Array-like structures to Array
The spread operator as an argument
Adding elements to Arrays/Objects
Merging Arrays/Objects

Part 2

Destructuring nested elements
Adding conditional properties
Short circuiting
The rest parameter (…)
Default destructuring values
Default properties

Why you should use the spread operator

After reading the previous list, you may be thinking something along these lines: "But JavaScript has functions to do all those things… Why would I use the spread operator?" Allow me to introduce you to immutability:

From Oxford Lexico: Immutability - unchanging over time or unable to be changed.

In software development, we use the term immutable to refer to values whose state cannot change over time. In fact, most of the values that we normally use (Primitive values, such as Strings , Integers etc.) are immutable.

However, JavaScript has a peculiar behavior when it comes to Arrays and Objects; they are, in fact, mutable. This can become a big problem. Here's an example, illustrating why:

As you can see in the previous code fragment, we have a Squirtle. Our Squirtle has a hp of 100, since we just visited the Pokemon Center.

Since we want another Squirtle, we declare the variable anotherSquirtle, assigning our original Squirtle as its value. After an arduous battle, anotherSquirtle is defeated. We therefore access anotherSquirtle's hp, and change it to 0. The next step is to check on our original Squirtle. We console.log and…

Wait, what? Our original Squirtle's hp is down to 0! How can this be? What happened to our poor Squirtle? JavaScript mutation happened. Let me explain what's going on.

When we created the anotherSquirtle variable, and assigned our original Squirtle as its value, what we really did was assign a reference to the memory location of the original Squirtle Object. This is because JavaScript Arrays and Objects are reference data types. Unlike Primitive data types, they point to the memory address where the actual Object/Array is stored.

To make it easier to understand, you can imagine reference data types as pointers to a global variable. By changing a reference data type's value, what we are really doing is changing the value of the global variable.

This means that, when we changed the anotherSquirtle's hp value to 0, we were really changing the hp value of the Squirtle Object stored in memory to 0. This is why mySquirtle's hp value is 0, because mySquirtle holds a reference to the Object stored in memory, which we changed via the anotherSquirtle variable. Thank you JavaScript.

How do we solve this problem?

To avoid the accidental mutation of variables, what we have to do is create a new instance of our Array/Object whenever we want to copy an Array/Object. How do we achieve this?

With the spread operator!! :)

How does the spread operator work?

From the MDN docs: Spread syntax allows an iterable such as an array expression or string to be expanded in places where zero or more arguments (for function calls) or elements (for array literals) are expected, or an object expression to be expanded in places where zero or more key-value pairs (for object literals) are expected
.
To put it simply, the spread operator '…' spreads the items that are contained in an iterable (an iterable is anything that can be looped over, like Strings, Arrays, Sets…) inside a receiver (A receiver is something that receives the spread values). Here are several simple examples with Arrays that will allow you to understand it better:

As you can see, when we use the spread operator on an Array, we obtain each individual item contained in the Array. In all the previous cases, the receiver was a function, the console.log function. Easy enough, right?

Cloning Arrays and Objects

Now that we now how the spread operator works, we can make use of it to copy Arrays and Objects immutably. How? By spreading the contents, and then using either the Array or Object literals ([] and {} respectively) to generate a new instance of the Array/Object. Let's take the previous Squirtle example, and fix it, by cloning the *mySquirtle * variable immutably:

By destructuring the contents of the mySquirtle variable with the spread operator, and using the Object literal, we are creating a new instance of the Squirtle Object. This way, we prevent accidental variable mutation.

To copy an Array, we use exactly the same syntax:

Note: Bear in mind the fact that the spread operator only performs shallow copies. This means that if you have a reference data type stored inside your Array/Object, when you make a copy with the spread operator, the nested Array/Object will contain a reference to the original, and will thus be mutable.

Converting Array-like Objects to Arrays

Array-like Objects are very similar to Arrays. They usually have numbered elements and a length property. However, they have one crucial difference: Array-like Objects do not have any of the Array functions.

Among the Array-like Objects are the HTML node lists returned by most DOM methods, the arguments variable generated automatically in every JS function and a few others.

With the same syntax as when cloning arrays, we can use the spread operator to transform Array-like structures to Array, as an alternative to using Array.from. Here's an example, converting a NodeList to an Array:

With this technique, we can transform any Array-like structure to Array, and thus have access to all the Array functions.

The spread operator as an argument

Some functions accept a variable number of parameters. A great example of these types of functions are the ones in the Math collection. For our example, let's pick the Math.max() function, which accepts n numeric parameters, and returns the largest one. Imagine we have an Array of numbers, which we want to pass to the Math.max() function. How do we do it?

We could do something like this (don't hate me for the following code):

But, of course, doing this would be suicide. What if we had 20 values? Or 1000? Are we really going to access each value by index? The answer is no. As we already know, the spread operator takes an Array and extracts each individual value. This is just what we're looking for! Therefore, we can do this:

Spread operator to the rescue!

Adding new elements

Adding items to an Array

To add new elements to an array, we first spread the Array's contents, and use the Array literal [] to create a new instance of the Array, containing the original array's contents, plus the values we want to add :

As you can see, we can add as many new items as we want.

Adding properties to an Object

By using the same syntax as with Arrays, we can easily add new properties when cloning an Object. To switch it up a little, here's a different syntax to add properties to an Object (it can also be used with Arrays):

As you can see, we can declare and initialize new variables directly inside the Object literal, instead of doing so outside.

Merging Arrays/Objects

Arrays

We can merge two arrays, by spreading them and using the Array literal, like in the previous examples. However, instead of simply adding a new element, we're going to add another (spread) array:

It also works if we have an Array of Objects:

Objects

We can merge two (or more) Objects into a single Object, by using the same syntax as before (you may have noticed by now, that the spread operator is used in a very similar same way, both for Arrays and Objects):

Conclusion

In this first part of the tutorial we have learnt why we should use the spread operator (immutability!), how it works, and several basic uses of said operator. In the second part of the tutorial we will deepen our knowledge this operator with several advanced techniques and use cases. Here's the link to the second part.

Thank you very much for reading :) If you have any doubts or comments don't hesitate to reach out to me, here's a link to my Twitter page.

Using variables in Docker-Compose

Nya — Thu, 05 Sep 2019 15:45:15 +0000

Introduction

Sooner or later, we all have to deal with environment variables in our Compose files. They can become a pain, especially if we don't know how to use them properly. This piece is a recollection of everything I've learned about environment variables, and aims to make using these variables easy, and above all, secure.

First of all, how do we use environment variables?

Docker Compose allows us to pass environment variables in via command line, or to define them in our shell. However, it's recommended to keep these values inside the actual Compose file, and out of the command line.

Why, you may ask?

Because this way, we don't have to remember all the environment variables we use every time we deploy our container. By storing them in the Compose file, we are able to maintain consistency during our builds.

There are several ways to do this:

Using the environment option

Using the Compose environment option allows us to declare environment variables and their values inside our Compose file, like this:

This is the easiest, quickest way of storing environment variables inside Compose files. However, it has a (in my opinion) huge drawback. It is related to security. Can you guess what it is?

That's right.

Storing the values of your environment variables in the Compose file, which, 9 and a half times out of ten will go to straight to source control, is a huge security risk. Luckily, we have an alternative: using an external file to store our environment variables.

Using a .env file

The main advantage of using an external file for your environment variables is the fact that you can keep said file out of your source control. After all, no one enjoys their passwords/API keys/other super secret information to be displayed all over the internet, for anyone to see :p

.env files are plain text files. You don't need to name them, the extension is the actual name of the file. They must be created at the root of your project, which is also where your docker-compose.yml file should be.

We declare and assign variables in our .env file. You can name the variables however you want, since we will only be accessing their values. Here's my .env file:

You can also create and populate your .env file from the command line, by using the linux cat command:

Tip: Remember not to leave any spaces between the "=" sign and the value assigned to your variable, as they will be added to the string.

So, now that we have our variables stored in our .env file, let's use them in our Compose file. Now's the time to use string interpolation (that's a fancy name for using this notation: ${string} ) to assign the values of our .env variables to the environment variables in the Compose file, like so:

As you can see, we maintain the environment option, and simply assign our external values to the Compose environment variables.

To check that everything is working properly, run the following command:

docker-compose up

Tip: You can check which values are assigned to the environment variables by running the following command (in a different terminal):

docker-compose config

Environment variable priority

Something very important we must keep in mind is the priority used by Compose to choose which environment value it uses. What does this mean?

If we declare the same environment variable in several files, for example, both in the Compose file and in the external .env file, with different values, Compose will use the value of the variable declared in the Compose file. Why?

Because depending on where the variable is declared, it is given a higher or lower priority by Compose. Here's the order, ranking from highest priority, to lowest:

1. Compose file
2. Shell environment variables
3. Environment file
4. Dockerfile
5. Variable is not defined

If for some reason, Compose is picking up and assigning a value you weren't expecting, this is probably the cause. Make sure to have all your variables declared exactly where you want them to be.

Using the env-file option

In the previous section, we talked about .env files, and we said that these files aren't named. However, if for whatever reason we do want our .env file to have a name, a name like secret-stuff.env, Compose has a nifty little option named env-file.

This option allows us to tell Compose which .env file it has to look for, as opposed to its default behavior, which is to look for an unnamed .env file. This is how we use the env-file option:

As you can see, we added the env_file option, which points to a file named secret-stuff.env file. All that is left is to rename our previous .env file to secret-stuff.env .

You may have noticed that the environment option is no longer present in our Compose file. This is because using the env-file option raises a problem (that gave me quite a headache). Allow me to explain:

To be able to assign values declared in an external named .env file to Compose variables requires said file to be defined in the primary Compose service. This is related to the order that Compose follows when performing operations. However, we have no primary service, since we are only using a db service. Therefore, if we try to deploy our Compose file, it will complain about our variables not being defined, and substitute them with blank strings.

Comment: You don't have to take my word for it, go ahead and try it! Run docker-compose up and see what happens :)

So, how do we solve this problem? Here's what I figured out:

If we remove the environment option from the Compose file, upon deployment, Compose will search for the specified secret-stuff.env file, something it doesn't do when the environment option is present. Problem solved!

Bear in mind though, since we don't have the environment option any longer, we must declare the environment variables directly in the secret-stuff.env file, like this:

Once again, to check everything works properly, run:

docker-compose up

Conclusion

That's all folks! By now you've (hopefully!) learnt the different ways of securely dealing with environment variables in Compose files. If you have any doubts or would like to discuss anything, don't hesitate to reach out to me. Here's a link to my Twitter page.

Thank you for reading :)

PokeAPI REST in NodeJS with Express, Typescript, MongoDB and Docker — Part 3

Nya — Thu, 11 Jul 2019 11:26:03 +0000

Foreword

This is part 3 of a series of posts which will show you how to create a RESTful API in NodeJS. For further reading please check out the following links:

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 1

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 2

If you prefer to check out the full code, you can find the full PokeApi project here.

Introduction

In the previous post we deployed an instance of MongoDB with docker-compose, and connected our application to it. We also created our Mongoose Schema and data Model.

In this post we are going to implement the rest of the routes that are necessary to create a basic CRUD, as well as their respective database query functions. These functions will make use of the Model we created previously to query our MongoDB database.

The coding begins

Preview

As always, we will begin with a preview of how our directory will look by the end of this post:

You may note that, directory-wise, nothing has changed since the previous post. Content-wise, however, there are quite a few changes.

Just as a reminder, to run our project we are currently using the following command:

npm run start

To start our dockerized MongoDB instance, use the following command:

docker-compose up

This said, let us begin.

PokeService: Querying the database

It is now time to create our database query functions. For this, as previously mentioned, we are going to make use of our Pokemon Model. Since our goal is to implement the four basic CRUD operations, the first function we are going to implement is one for reading the contents of the db. Open up the pokemon.service.ts file, and type in the following:

//src/services/pokemon.service.ts

import { Request, Response } from 'express';
import { MongooseDocument } from 'mongoose';
import { Pokemon } from '../models/pokemon.model';
import { WELCOME_MESSAGE } from '../constants/pokeApi.constants';

export class PokeService {
  public welcomeMessage(req: Request, res: Response) {
    res.status(200).send(WELCOME_MESSAGE);
  }

//Getting data from the db
  public getAllPokemon(req: Request, res: Response) {
    Pokemon.find({}, (error: Error, pokemon: MongooseDocument) => {
      if (error) {
        res.send(error);
      }
      res.json(pokemon);
    });
  }
 }

As you can see, we’ve created a new function, named “getAllPokemon”. It uses the Pokemon Model to interact with MongoDB and find all the Pokemon in the db.

Since Mongoose’s helper functions are extensively documented in the Mongoose docs, I don’t think it’s necessary to break them down here. I will, however, comment on the guard clause inside the callback:

Note: Guard clauses are, put simply, a check that exits the function, either with a return statement or an exception, in case of an error or fulfilled condition. They allow us to avoid unnecessary complexity (if else if structures) in our code.

This is our guard clause:

if (error) {
   res.send(error);
  }
  res.json(pokemon);

By reversing the logic, and checking first for errors, we can avoid an “else” statement. If any errors are encountered, we exit the function by sending the error. If we find no errors, then the pokemon result is sent. We will make use of this technique throughout the rest of this post.

Implementing GET routing

We now have our getAllPokemon function in our PokeService. To be able to interact with this function, we must create another GET route. Let’s open our controller, and add a new route:

//src/main.controller.ts

import { PokeService } from "./services/pokemon.service";
import { Application } from "express";

export class Controller {
  private pokeService: PokeService;

  constructor(private app: Application) {
    this.pokeService = new PokeService();
    this.routes();
  }

  public routes() {
    this.app.route("/").get(this.pokeService.welcomeMessage);

    //Our new route
    this.app.route("/pokemons").get(this.pokeService.getAllPokemon);
  }
}

As you can see, the endpoint to access this new route is “/pokemons”. (Excuse the glaring grammatical error, it’s to avoid confusion further on.)

From here on, I recommend using Postman to test our routes. You can find more information about Postman here and install it here.

If all goes well, you should be obtaining output like the following from Postman:

Since we haven’t introduced any data into our db, we are receiving an empty array. We have now completed our first db query successfully!

Adding a new Pokemon

Let’s implement a function to add a new pokemon to our db. Go back to the PokemonService and type:

//src/services/pokemon.service.ts

import { Request, Response } from 'express';
import { MongooseDocument } from 'mongoose';
import { Pokemon } from '../models/pokemon.model';
import { WELCOME_MESSAGE } from '../constants/pokeApi.constants';

export class PokeService {
  public welcomeMessage(req: Request, res: Response) {
    res.status(200).send(WELCOME_MESSAGE);
  }

  public getAllPokemon(req: Request, res: Response) {
    Pokemon.find({}, (error: Error, pokemon: MongooseDocument) => {
      if (error) {
        res.send(error);
      }
      res.json(pokemon);
    });
  }

  //Adding a new pokemon

  public addNewPokemon(req: Request, res: Response) {
    const newPokemon = new Pokemon(req.body);
    newPokemon.save((error: Error, pokemon: MongooseDocument) => {
      if (error) {
        res.send(error);
      }
      res.json(pokemon);
    });
  }
}

To explain briefly, we create a Mongoose Document (newPokemon) from the request body, and we save it into the db.

Note: Just as a reminder, Mongoose Documents are instances of Mongoose Models

Let’s create the route to interact with our function. In our controller:

//src/main.controller.ts

import { PokeService } from "./services/pokemon.service";
import { Application } from "express";

export class Controller {
  private pokeService: PokeService;

  constructor(private app: Application) {
    this.pokeService = new PokeService();
    this.routes();
  }

  public routes() {
    this.app.route("/").get(this.pokeService.welcomeMessage);
    this.app.route("/pokemons").get(this.pokeService.getAllPokemon);

    //Our new route
    this.app.route("/pokemon").post(this.pokeService.addNewPokemon);
  }
}

Note that our new route is accessed through a POST request. Head over to Postman, and let’s add a new Pokemon to our db:

If everything goes well, you should be receiving the Pokemon you just added as output. To double-check, we can make use of our GET route:

As you can see, we now have a Squirtle in our db. Don’t worry about the “_id” and “__v” fields. They are generated automatically by Mongooose, and we will deal with them later.

Deleting a Pokemon

To implement a function to delete a Pokemon, open up the PokeService, and type the following:

//src/services/pokemon.service.ts

import { Request, Response } from 'express';
import { MongooseDocument } from 'mongoose';
import { Pokemon } from '../models/pokemon.model';
import { WELCOME_MESSAGE } from '../constants/pokeApi.constants';

export class PokeService {
  public welcomeMessage(req: Request, res: Response) {
    res.status(200).send(WELCOME_MESSAGE);
  }

  public getAllPokemon(req: Request, res: Response) {
    Pokemon.find({}, (error: Error, pokemon: MongooseDocument) => {
      if (error) {
        res.send(error);
      }
      res.json(pokemon);
    });
  }

  public addNewPokemon(req: Request, res: Response) {
    const newPokemon = new Pokemon(req.body);
    newPokemon.save((error: Error, pokemon: MongooseDocument) => {
      if (error) {
        res.send(error);
      }
      res.json(pokemon);
    });
  }

  public deletePokemon(req: Request, res: Response) {
    const pokemonID = req.params.id;
    Pokemon.findByIdAndDelete(pokemonID, (error: Error, deleted: any) => {
      if (error) {
        res.send(error);
      }
      const message = deleted ? 'Deleted successfully' : 'Pokemon not found :(';
      res.send(message);
    });
  }
}

We obtain the ID of the Pokemon to delete from the request parameters, this is to say, the parameters in the query string in the URL to which we make the GET request. It would look something like this:

localhost:9001/pokemon/123pokemonId

Mongoose has a very useful findByIdAndDelete helper function, which allows us to delete a document (in our case, a Pokemon) by said document’s “_id” field. This function is shorthand for findOneAndDelete({_id: pokemonId}).

Note: If you have the need of deleting a document by any field other than _id, you may use the findOneAndDelete function aforementioned. More information about it here.

I’d now like to draw your attention to the following line:

const message = deleted ? "Deleted successfully" : "Pokemon not found :(";

Here we have a ternary expression, which assigns a different value to the “message” variable, depending on the value of the second parameter (“deleted”) passed to the callback.

This is because Mongoose’s findByIdAndDelete function finds a matching document, deletes it, and then passes the found document (if any) to the callback. Therefore, if Mongoose finds a document, it will be deleted, in which case we return the “Deleted successfully” message. If not, we return the “Pokemon not found” message.

Once we have our function ready, let’s create our route. In our controller:

//src/main.controller.ts

import { PokeService } from "./services/pokemon.service";
import { Application } from "express";

export class Controller {
  private pokeService: PokeService;

  constructor(private app: Application) {
    this.pokeService = new PokeService();
    this.routes();
  }

  public routes() {
    this.app.route("/").get(this.pokeService.welcomeMessage);
    this.app.route("/pokemons").get(this.pokeService.getAllPokemon);
    this.app.route("/pokemon").post(this.pokeService.addNewPokemon);

    //Our new route
    this.app.route("/pokemon/:id").delete(this.pokeService.deletePokemon);
  }
}

In the route we have just created, we are indicating that we will receive a request parameter in the URL, a parameter we have named “id”. This is the parameter we previously used in the Pokemon Service to obtain the id.

Note that this route is accessed through a DELETE request.

Once again, we open Postman, and test our new route by deleting the Squirtle (or whichever Pokemon you chose) we added to our db earlier:

As you can see, we receive the “Deleted successfully” message. If no Pokemon with the id we specified were to be found, we would receive the “Pokemon not found” message instead.

We can double check that our squirtle has been deleted correctly by obtaining all Pokemon from the db:

Empty array = no Pokemon = Squirtle has been deleted successfully.

Updating a Pokemon

In our Pokemon Service:

//src/services/pokemon.service.ts

import { Request, Response } from 'express';
import { MongooseDocument } from 'mongoose';
import { Pokemon } from '../models/pokemon.model';
import { WELCOME_MESSAGE } from '../constants/pokeApi.constants';

export class PokeService {
  public welcomeMessage(req: Request, res: Response) {
    res.status(200).send(WELCOME_MESSAGE);
  }

  public getAllPokemon(req: Request, res: Response) {
    Pokemon.find({}, (error: Error, pokemon: MongooseDocument) => {
      if (error) {
        res.send(error);
      }
      res.json(pokemon);
    });
  }

  public addNewPokemon(req: Request, res: Response) {
    const newPokemon = new Pokemon(req.body);
    newPokemon.save((error: Error, pokemon: MongooseDocument) => {
      if (error) {
        res.send(error);
      }
      res.json(pokemon);
    });
  }

  public deletePokemon(req: Request, res: Response) {
    const pokemonID = req.params.id;
    Pokemon.findByIdAndDelete(pokemonID, (error: Error, deleted: any) => {
      if (error) {
        res.send(error);
      }
      const message = deleted ? 'Deleted successfully' : 'Pokemon not found :(';
      res.send(message);
    });
  }

  //Updating a pokemon

  public updatePokemon(req: Request, res: Response) {
    const pokemonId = req.params.id;
    Pokemon.findByIdAndUpdate(
      pokemonId,
      req.body,
      (error: Error, pokemon: any) => {
        if (error) {
          res.send(error);
        }
        const message = pokemon
          ? 'Updated successfully'
          : 'Pokemon not found :(';
        res.send(message);
      }
    );
  }
}

Note that we have used precisely the same technique as in the deletePokemon function. Obtaining the ID as a request parameter, using Mongoose’s findByIdAndUpdate helper function, and returning a message according to the value of the second callback parameter.

In our controller, let’s create the final route:

//src/main.controller.ts

import { PokeService } from "./services/pokemon.service";
import { Application } from "express";

export class Controller {
  private pokeService: PokeService;

  constructor(private app: Application) {
    this.pokeService = new PokeService();
    this.routes();
  }

  public routes() {
    this.app.route("/").get(this.pokeService.welcomeMessage);
    this.app.route("/pokemons").get(this.pokeService.getAllPokemon);
    this.app.route("/pokemon").post(this.pokeService.addNewPokemon);

    //Chaining our route

    this.app
      .route("/pokemon/:id")
      .delete(this.pokeService.deletePokemon)
      .put(this.pokeService.updatePokemon);
  }
}

Considering that both the delete and put routes have exactly the same endpoint, we can chain them as shown above. This way, we don’t have to declare the same route twice, one for each verb.

Let’s head over to Postman, and test our final route. Don’t forget to add a new Pokemon, or you won’t have any data to update! I chose to add another Squirtle, which I will now update:

Let’s obtain all our Pokemon to check on our Squirtle:

Congratulations! Your Squirtle has evolved into a Wartortle, and you have successfully implemented all the basic CRUD functions and their respective routes.

Conclusion

In this post we’ve learnt how to query MongoDB by means of Mongoose Models and Documents. We have also implemented routes to access our CRUD functions.

If you’d like to see the full code for this post, you can do so here (branch “part3” of the pokeAPI project).

Thank you so much for reading, I hope you both enjoyed and found this post useful. Feel free to share with your friends and/or colleagues, and if you have any comments, don’t hesitate to reach out to me! Here’s a link to my twitter page.

PokeAPI REST in NodeJS with Express, Typescript, MongoDB and Docker — Part 2

Nya — Thu, 11 Jul 2019 11:25:06 +0000

Foreword

This is part 2 of a series of posts which will show you how to create a RESTful API in NodeJS. For further reading please check out the following links:

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 1

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 3

If you prefer to check out the full code, you can find the full PokeApi project here.

Introduction

In the previous post we set up our server, and implemented our first GET route, which returned a lovely welcome message. Since our goal is to perform basic CRUD operations on our Pokemon data, we need to have a database to store our information.

In this post we are going to create and deploy a docker container for our MongoDB database. We are also going to define our Pokemon data model using Mongoose.

Let’s code

Preview

Once again, before we start, a little preview of how our directory tree will look by the end of this post:

Just as a reminder, to run our project we are currently using the following command:

npm run start

This said, let us begin.

Creating our docker-compose file

The first thing we are going to do is create a docker-compose.yml file, on the same level of our “src” directory, which is to say, outside the “src” directory. Once this is done, copy and paste the following code into the newly created file:

Let’s explain briefly what all of these configuration options mean:

version: Specifies the docker-compose version we are going to use.

services: We can specify a list of services which will be deployed with our container. In our case, we want a database, which is why we use the following attribute:

db: We indicate that we are going to deploy a database.

container_name: This is optional, it allows us to specify a custom container name. If we omit this option, a default container name will be generated.

image: Specifies the image that the container will be built from. In our case, the latest MongoDB image.

restart: Always restart the container if it stops. If it is manually stopped, it is restarted only when Docker daemon restarts or the container itself is manually restarted.

volumes: This is a very interesting option. It allows us to have data persistence. What does this mean? All of our data is going to be stored in a docker container. However, docker containers can be stopped, restarted etc. In such cases, what happens to our data? Does it disappear? The answer is, it won’t disappear if we use the volumes option. We can specify a directory in our local machine where our data will be stored. In our case, this directory is named “pokeData”.

environment: We can specify environment variables. In our case, we are creating a database named “Pokemon” when the container starts.

ports: Specifies the ports that will be exposed (Host port: Container port). In our case, we are mapping our local port 27017 to the container port 27017 (27017 is the default port for MongoDB).

Note: For more information about docker-compose files, you can check out this link.

Now that we have our docker-compose file ready, let’s start the container. Fire up your terminal, and type this command:

docker-compose up

If you’ve done everything correctly, you should be seeing output similar to this on your terminal:

You should also see that a new directory named pokeData has appeared in your directory tree. This is the directory we specified earlier in our docker-compose file, by using the “volumes” attribute. Our pokeData directory will store all our database data (once we insert some), and keep it safe and sound.

Isn’t docker awesome and easy to use? A simple, intuitive, extensively documented configuration file and one command are all we need to have our database instance up and running. Beautiful.

Troubleshooting: If you’re getting the following output after executing the docker-compose up command:

ERROR: Couldn’t connect to Docker daemon at http+docker://localhost — is it running?

It means that your docker daemon isn’t running. Execute this command to start the docker daemon:

sudo systemctl start docker

And try the docker-compose up command again. The error should be gone.

Connecting our app to our dockerized MongoDB database

We have our database container deployed and running, so we now need to connect our application to it. Open the app.ts file, and add the following code:

//src/app.ts

import express, { Application } from 'express';
import { Controller } from './main.controller';
import bodyParser from 'body-parser';
import cors from 'cors';
import mongoose from 'mongoose';

class App {
  public app: Application;
  public pokeController: Controller;

  constructor() {
    this.app = express();
    this.setConfig();
    this.setMongoConfig();

    this.pokeController = new Controller(this.app);
  }

  private setConfig() {
    this.app.use(bodyParser.json({ limit: '50mb' }));
    this.app.use(bodyParser.urlencoded({ limit: '50mb', extended: true }));
    this.app.use(cors());
  }

  //Connecting to our MongoDB database
  private setMongoConfig() {
    mongoose.Promise = global.Promise;
    mongoose.connect("mongodb://localhost:27017/Pokemon", {
      useNewUrlParser: true
    });
  }
}

export default new App().app;

You may have noticed that once again, we are hard coding a variable: the mongoose connection string. To avoid this, let’s open our constants file, and store it there:

//src/constants/pokeAPI.constants.ts

export const PORT = 9001;
export const WELCOME_MESSAGE = "Welcome to pokeAPI REST by Nya ^^";
export const MONGO_URL = "mongodb://localhost:27017/Pokemon";

Back in our app.ts, we can now change the hard coded String for our newly defined constant:

//src/app.ts

import express, { Application } from 'express';
import { Controller } from './main.controller';

//importing our MONGO_URL constant
import { MONGO_URL } from './constants/pokeApi.constants';
import bodyParser from 'body-parser';
import cors from 'cors';
import mongoose from 'mongoose';

class App {
  public app: Application;
  public pokeController: Controller;

  constructor() {
    this.app = express();
    this.setConfig();
    this.setMongoConfig();

    this.pokeController = new Controller(this.app);
  }

  private setConfig() {
    this.app.use(bodyParser.json({ limit: '50mb' }));
    this.app.use(bodyParser.urlencoded({ limit: '50mb', extended: true }));
    this.app.use(cors());
  }

  private setMongoConfig() {
    mongoose.Promise = global.Promise;

//using our constant instead of the hard coded String
    mongoose.connect(MONGO_URL, {
      useNewUrlParser: true
    });
  }
}

export default new App().app;

If we’ve done everything correctly, we should now be seeing the following output in our terminal where we ran our “docker-compose up” command (if, for any reason, you stopped docker-compose previously, run the command again):

As you can see, our docker container has accepted the connection we made from our application. So far, so good.

Creating our data model

Now that we are connected to our database, we need a way to interact with it. To achieve this, we are going to use Mongoose, which provides us with several data modelling tools, such as Schemas and Models. Mongoose makes interacting with MongoDB exceedingly easy and simple.

From the mongoose docs: Models are fancy constructors compiled from Schema definitions. An instance of a model is called a document. Models are responsible for creating and reading documents from the underlying MongoDB database.

To store our data models, we are going to create a models directory in src, which will contain a file named “pokemon.model.ts”. Inside this file, we are going to import Mongoose and create our data model:

//src/models/pokemon.model.ts

import mongoose from "mongoose";

const PokemonSchema = new mongoose.Schema({
  name: String,
  gender: String,
  type: String,
  height: Number,
  weight: Number,
  photo: String
});

Once we’ve created our pokemon Schema, we need to create a Mongoose model. To do this, we will part from our newly created Schema. Therefore, in the same file:

//src/models/pokemon.model.ts

import mongoose from "mongoose";

const PokemonSchema = new mongoose.Schema({
  name: String,
  gender: String,
  type: String,
  height: Number,
  weight: Number,
  photo: String
});

//Creating our model
export const Pokemon = mongoose.model("Pokemon", PokemonSchema);

Note: I am fully aware of the fact that we are defining both our schema and model in the same file; a file named “pokemon.model”. Despite being a control freak, I refuse to create two separate files for 10 lines of code. If you are even more obsessed than I, you may, of course, create a separate file for your schema. Don’t count me in though :p

With our Pokemon model just created, it is now time to import it in the PokeService:

//src/services/pokemon.service.ts

import { WELCOME_MESSAGE } from "../constants/pokeAPI.constants";
import { Request, Response } from "express";
//importing our model
import { Pokemon } from "../models/pokemon.model";

export class PokeService {
  public welcomeMessage(req: Request, res: Response) {
    return res.status(200).send(WELCOME_MESSAGE);
  }
}

The Pokemon model will later be used to query our MongoDB database, once we create the CRUD routes and their respective db query functions. This, however, we will leave for the following post.

Conclusion

In this post we’ve learnt how to deploy an instance of MongoDB with docker-compose, and how to connect our application to it. We’ve also used Mongoose to create both a Schema and a Model for our database.

If you’d like to see the full code for this post, you can do so here (branch “part2” of the pokeAPI project).

In the following post we will be implementing the rest of the routes that are necessary to create a basic CRUD, as well as their respective database query functions.

Here is the link to the next post:

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 3

PokeAPI REST in NodeJS with Express, Typescript, MongoDB and Docker — Part 1

Nya — Thu, 11 Jul 2019 11:24:00 +0000

Foreword

This is part 1 of a series of posts which will show you how to create a RESTful API in NodeJS. For further reading please check out the following links:

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 2

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 3

If you prefer to check out the full code, you can find the full PokeApi project here.

Introduction

In these series of posts we will be learning how to create our own RESTful API in NodeJS, using an excellent web framework named Express. However, before we start, a little theory:

REST = Representational State Transfer. An awesome style of software architecture designed by Roy Fielding for his doctoral dissertation. REST allows us to design loosely coupled applications by using the HTTP protocol.

HTTP provides us with the following verbs, or methods: GET, POST, PUT and DELETE, which correspond to Read, Create, Update and Delete (CRUD operations) respectively. There are a few other verbs, but they aren’t used as frequently. We will be using these verbs to make requests, which will perform various operations on our data.

Since (in my humble opinion) TypeScript is the best thing since sliced bread, this is the language that we will be using. For those unfamiliar with it, TypeScript is a typed superset of JavaScript, that we compile to plain ol’ JavaScript, and, among many other things, allows us to add types to JavaScript (TS FTW).

Since we need a database to store our data, we will be using a dockerized instance of MongoDB, together with Mongoose, an ODM which makes interacting with MongoDB that much easier.

Now that we know what we are going to be working on, and how, let us get down and start coding.

Setting up our project

Preview

Before we start, I’d like to show you a preview of how our directory tree will look by the end of this post:

In case anyone is curious about how I generated the directory tree image, I used the linux “tree” command, and snapped a screenshot of my terminal. Pretty simple.

Installing NodeJS

Since we will be using NodeJS for our project, the first thing to do is make sure it is installed on our machine.

Tip: Open up your terminal and type this command to check if Node is installed: node -v

If not, you can install it here.

Installing Docker and Docker-compose

Docker is a fantastic tool that allows us to create, deploy and run applications (or pretty much anything we want) by using containers. We can deploy a MongoDB database (or any other database, you name it) in a few minutes, with a couple of simple commands.

Tip: Run this command to see if you have docker on your machine: docker -v

If not, you can install it here.

As I’ve just mentioned, Docker is awesome. However, and this is purely personal taste, I prefer to deploy my containers using Docker Compose. This is a tool offered by Docker, which allows us to create a .yml configuration file, where we can specify all the details of our container, and deploy said container with a just one simple command.

Tip: Run this command to check if you have docker-compose: docker-compose -v

If not, you can install it here.

Let the coding Begin

Getting started

With all our pre-requisites out of the way, we can now get down and start coding for real. Let us begin:

The first step is to create the file where our project is going to live. I’m going to name our project file “pokeApi”. Open up your terminal and type this:

mkdir pokeApi
cd pokeApi

Once inside our project file, we want to create our package.json file. Again, type the following command in your terminal:

npm init

After running this command, we will be asked a series of questions, and upon answering them, our package.json file will be created.

Tip: If you don’t feel like answering npm’s questions, use this command instead: npm init -y
You can always go back to the package.json file and edit it later.

Installing dependencies

To be able to use express, mongoose, TypeScript etc. we must install a few dependencies. To do so, run the following command:

npm i body-parser cors express mongoose

We also need to install several dependencies needed only for development. Type:

npm i -D @types/body-parser @types/cors @types/express @types/mongoose @types/node nodemon ts-node typescript

Tip: By adding -D or --save-dev to npm i command, the dependencies installed will be listed under “devDependencies” in the package.json file

Adding npm scripts

To be able to run our projects, we must create the following script in our package.json file:

"scripts": {
"start": "nodemon"
},

Configuring nodemon

Nodemon is a neat tool for developing nodeJS applications. It automatically restarts the application whenever it detects changes in the code (basically, whenever you save).

Create a file named nodemon.json, and type in the following:

{
"watch": ["src"],
"ext": "ts",
"exec": "ts-node ./src/server.ts"
}

This tells nodemon which files to watch and execute.

Configuring TypeScript

To generate our tsconfig.json file, run the following command:

tsc --init

Note that this file contains many, many configuration options. You may, of course, configure TypeScript according to your preferences. If not, here’s the configuration I use:

"compilerOptions": {
"target": "es6",
"module": "commonjs",
"outDir": "./dist",
"resolveJsonModule": true,
"strict": true,
"esModuleInterop": true,
"experimentalDecorators": true,
"emitDecoratorMetadata": true
},                         
"exclude": ["node_modules"], 
"include": ["src/**/*.ts"]

Creating a .gitignore file

In this file we can list all the files/directories that we want git to ignore, this meaning that when we add and commit the changes made to our project, these files while remain “invisible” to git.

Tip: It is very, very, very important to add node_modules to our .gitignore. We most definitely don’t want this huge file pushed to our repository!!

To create our .gitignore file, type this (while in the root of the directory, of course):

touch .gitignore

Then, add the following lines inside the file:

//.gitignore

node_modules
package-lock.json
dist

Enough with the config, where’s the real coding?

It begins now, I swear. Let’s go:

Setting up our server

The first thing we are going to do is create our basic directory structure. We are going to create a directory named src, which will contain all of our project files (aside from config):

mkdir src
cd src 
mkdir constants
touch server.ts
touch app.ts

Let us open the app.ts file we just created, which will contain our basic express configuration:

//src/app.ts

import express, { Application } from 'express';
import bodyParser from 'body-parser';
import cors from 'cors';

class App {
  public app: Application;

  constructor() {
    this.app = express();
    this.setConfig();
  }

  private setConfig() {
    //Allows us to receive requests with data in json format
    this.app.use(bodyParser.json({ limit: '50mb' }));

    //Allows us to receive requests with data in x-www-form-urlencoded format
    this.app.use(bodyParser.urlencoded({ limit: '50mb', extended:true}));

    //Enables cors   
    this.app.use(cors());
  }
}

export default new App().app;

We will go over Express’ config quickly:

Body parser allows us to receive requests with data in different formats, such as json, or x-www-form-urlencoded.
CORS (Cross-Origin Resource Sharing) uses additional HTTP headers let our browser know that is has to allow a web application running at one domain to access resources from a server at a different origin.

Once this is done, we’re going to create a file to store our app’s constants. Why? Because this way we only have to declare each constant once. Whenever we need to make use of it, we just have to import it.

Furthermore, if the value of our constant changes (yes, even though it’s a constant, sometimes we need to change its value), it will change everywhere in our project, since it’s only declared in one place. All of this said, let’s create our constants file:

cd constants
touch pokeApi.constants.ts

The first constant we are going to declare is our PORT, which will store the number of the port we are going to open for our server:

//src/constants/pokeApi.constants.ts

export const PORT = 9001;

Now, head over to our server.ts file, where we will set up our server:

//src/server.ts

import app from "./app";
import { PORT } from "./constants/pokeApi.constants.ts";

app.listen(PORT, () => console.log(`Listening on port ${PORT}`));

Tip: If anyone is unfamiliar with the syntax I’m using in the console.log, it’s a technique named template literals, where you type everything inside grave quotes (also known as backticks), and use interpolation (${}) to embed variables. More about this technique here.

Note that we are importing both the app we created previously, and our PORT constant.

And with just these three lil’ files, we’ve created our very own server! Fire up your terminal and execute the npm start script we created previously. You can do this by typing:

npm run start

Tip: Since we are using nodemon to watch our project files, we only need to execute the previous command once. Every time we save our changes, nodemon will automatically restart our app for us.

After executing the command, you should be seeing the “Listening on port 9001” message on your terminal. Yeah! We now have our server up and running.

You can also head over to your favorite browser to check it out. Type this:

localhost:9001

You should be seeing a message similar to this: “Cannot GET /”. I know, not very exciting… But if you’re seeing this message, it works! If not, go back and re-check your code to make sure nothing is missing.

Creating our first GET route

Since we now have our server up and running, we are going to create the first GET route and display a nice welcome message. After all, “Cannot GET /” isn’t very welcoming…

To do this, create a file named “main.controller.ts”, and type in the following:

//src/main.controller.ts

import { Application } from 'express';

export class Controller {
  private pokeService: PokeService;

  constructor(private app: Application) {
    this.routes();
  }

  public routes() {
    this.app.route('/').get();
  }
}

As you may have noted, our Controller is going to act as a router; it is where we will define all of our routes for this project. Each route will execute a different action, which will be defined in a service file.

Why are we going to separate our actions in a different file? Say you defined all of the functions that interact with the database in your controller. For this project, we are going to use MongoDB as our database. Now imagine you want to change the database, and use MySQL instead. You would have to go back to your controller, and change everything, to adapt it to a MySQL database. If, however, you’ve declared all of your database functions in a different file, you wouldn’t need to change the controller at all. You could just swap the file with MongoDB query functions for one with MySQL query functions. By using a service, we keep our code loosely coupled.

Therefore, we will now create a file named “pokeApi.service.ts”, in a directory named “services”, and type in the following:

//src/services/pokemon.service.ts

import { Request, Response } from "express";

export class PokeService {
  public welcomeMessage(req: Request, res: Response) {
    return res.status(200).send("Welcome to pokeAPI REST by Nya ^^");
  }
}

A very simple function, which returns our cute welcome message. Now, head over to our controller, and import the service we have just created:

//src/main.controller.ts

import { Application } from 'express';
import { PokeService } from './services/pokemon.service';

export class Controller {
  private pokeService: PokeService;

  constructor(private app: Application) {
    this.pokeService = new PokeService();
    this.routes();
  }

  public routes() {
    this.app.route('/').get(this.pokeService.welcomeMessage);
  }
}

As you can see, our main GET route will call the welcomeMessage function we have just created in our pokemon service.

So far, so good. It’s time to import our Controller into our app.ts:

//src/app.ts

import express, { Application } from 'express';

//importing our controller
import { Controller } from './main.controller';
import bodyParser from 'body-parser';
import cors from 'cors';

class App {
  public app: Application;

  //declaring our controller
  public pokeController: Controller;

  constructor() {
    this.app = express();
    this.setConfig();

    //Creating and assigning a new instance of our controller
    this.pokeController = new Controller(this.app);
  }

  private setConfig() {
    this.app.use(bodyParser.json({ limit: '50mb' }));
    this.app.use(bodyParser.urlencoded({ limit: '50mb', extended: true }));
    this.app.use(cors());
  }
}

export default new App().app;

And we’re done! Head over to your browser, and if you’ve done everything correctly, you should be seeing your welcome message displayed, like this:

A little bit of refactoring

Remember we created a file which would store all of our constants? You may have realized that in our welcomeMessage function (in our pokemon service), we were returning a String containing the message, which we “hard coded” into our service. Not very neat, right? What if we want to change the message? I’d have to modify the service. Not good.

Therefore, we are going to take the message, and declare it in our constants file:

//src/constants/pokeApi.constants.ts

export const PORT = 9001;
export const WELCOME_MESSAGE = "Welcome to pokeAPI REST by Nya ^^";

One we’ve done this, we are going to import the constants file in our service, like so:

//src/services/pokemon.service.ts

import { Request, Response } from "express";
import { WELCOME_MESSAGE } from "../constants/pokeApi.constants";

export class PokeService {
  public welcomeMessage(req: Request, res: Response) {
    return res.status(200).send(WELCOME_MESSAGE);
  }
}

If you go back to your browser, you should still be seeing the welcome message.

Conclusion

In this post we’ve covered everything from setting up our project’s configuration, to defining our first route and successfully making our first GET request.

If you want to check out the full code for this post, you can find it here (branch “part1” of the pokeAPI project).

In the following post we will be connecting our application to a dockerized instance of MongoDB, deployed with docker-compose. We will also be using Mongoose to create a data Model and Schema.

Last, but not least, here is the link to the following post:

PokeAPI REST in NodeJS with Express, TypeScript, MongoDB and Docker — Part 2

Array.map() + async/await

Nya — Thu, 02 May 2019 15:06:25 +0000

MDN Web Docs: The map() method creates a new array with the results of calling a provided function on every element in the calling array.

Today I found myself in a situation where I had to use an asynchronous function inside of an Array.map. Needless to say, Array.map is a synchronous function. Therefore, I had a slight problem in my hands.

I came across this problem while refactoring my code to get rid of a nasty “for..of”. Basically, I had an Array of objects upon which I had to make several calculations. One of these calculations involved making a call to an API which, of course, is asynchronous. Before my refactoring, I had no problem with using async/await inside my for..of loop. However, after making the switch to Array.map, I realized it wouldn’t be as simple.

Here’s what I did:

Asynchronous Array.map()

As you can see, my getDistance function simulates a call to an API, and returns a Promise.

Note: The return value of async functions is always a Promise

This means that our Array.map function will return an Array of unresolved promises. Therefore, outside of the Array.map, we have to wait for all the promises to be resolved before we can make use of the resulting Array. This is why we use Promise.all().

Note: If you aren’t comfortable with async/await syntax, you can always resort to good ol’ “.then”

You can check out and play with my solution in the following CodePen:

Pretty simple, right? It took me a while to figure it out…
Which is why I decided to share my solution, in case someone else encounters this issue. I hope it helps! Thank you for reading, feel free to share or leave a comment :)

P.S. You can check out my Github and Twitter pages if you like...