DEV Community: Teddy MORIN

Stop Using Insecure and Inefficient Dockerfiles

Teddy MORIN — Sun, 24 Sep 2023 14:39:10 +0000

Often, I see terrible Dockerfiles used at length. It has a negative impact on security, productivity, and overall cost.

Today, I want to show you how to improve from a basic to a very efficient Dockerfile, step by step.

Getting Started

Its a very short Dockerfile, which has the advantage of being simple while working perfectly fine. It does the job of:

Using the latest version of the official NodeJS image as a base
Define a working directory (/app)
Copying source files
Installing dependencies
Exposing port (8000)
Running the app (yarn start)

But is it efficient? Obviously not. If it was, there would be no point to the article youre about to read.

Parent Image

Here, there are three important points.

Using a fixed version

Maybe it seems like a good idea to use the latest version of an image as a base (to stay up-to-date), but its not. In practice, its often the source of issues.

If you use the last version of an image (latest), you dont have control over when it gets updated. Its possible (and will happen) that a new version of an image is published while being incompatible with your app.

Between two builds, your application will suddenly go from working to failing, without any apparent reason.

Thats why we prefer using a fixed version instead of latest. But that also means you have the responsibility to update your base images from time to time.

Long-Term-Support

Usually, a software is provided with specific LTS versions. That means creators provide better support for those versions, and should be preferred.

NodeJS provides a list of LTS versions on its official website.

Alpine

Docker images are built on top of a given distribution, such as Ubuntu and Debian. Among those distributions is Alpine.

Its known for being very lightweight compared to others and is a huge help in keeping an efficient image.

Lower Privileges

The default user being ROOT, he has unlimited access. For security, its a better idea to provide a user with limited privileges.

Fortunately, with the node image comes a user called node.

Working Directory

The working directory is the one used by default. Its a good idea to define a specific one for your app.

The most common practice is to use /usr/src/app.

Caching

Docker works with a caching system, which is often ignored. You can think of each step in a Dockerfile as a layer, which is cached.

When one layer changes, all subsequent layers are invalidated. When the image is rebuilt, instead of retrieving a layer from the cache, the necessary command is simply restarted.

One layer changes frequently: the source code. So its best to copy the source code as late as possible.

The most common mistake is to copy the list of dependencies at the same time as the source code, then install the dependencies.

In this case, every time the source code changes and the image is rebuilt, the dependencies will be reinstalled. Thats why we prefer to copy the files defining the dependencies first, then install them, and finally copy the source code.

Configure the Working Directory

Previously, we defined a new Working Directory and set up a user with limited privilege. That means we need to create the necessary directory first and give necessary access to the node user.

Update Package List

With Alpine, we arent using APT, but APK to manage packages. No matter which tool you use to manage packages, you need to update its list from remote repositories.

That way, you ensure no out-of-date packages are installed. Its mandatory to avoid packages with security and performance issues.

Adding Packages

I recommend using theno-cache flag to avoid generating cache you wont use, but will still make your image heavier.

Multiple RUN

Inside Best practices for writing Dockerfiles, we learn to:

always combine RUN apt-get update with apt-get install in the same RUN statement.

Using apt-get update alone in a RUN statement causes caching issues and subsequent apt-get install instructions to fail.

For more information, I highly recommend you give best practices a read.

Entrypoint

I like to keep a basic Dockerfile that doesnt start my application directly. Instead, I use Tini, which keeps the container alive and improve how processes are managed.

Multi-stage builds | Docker Docs

Learn about multi-stage builds and how you can use them to improve your builds and get smaller images

docs.docker.com

Then, you can either use multi-stage builds to augment your basic stage, or start and configure your application from outside the Dockerfile.

Locally you can use the Docker CLI. Also, when you deploy your app, any container management system allows you to configure startup scripts.

That way, you can have a single Dockerfile that is used in different configurations, instead of multiple configurations with barely any difference.

It becomes the source of truth about which environment your app run on.

Final Result

Now, you might be wondering: whats the difference?

Apart from better security and faster re-build, one of my real-world project is 1GB lighter with the improved image:

Do you want to learn more backend skills, you can effectively use in a professional environment?

[-15%] Bundle - Scalable Backend: from Zero to hero

Learn how to create clean and hyper-scalable applications from A to Z. Includes both courses "Enterprise Grade Back-End Development with NodeJS" and "Microservices: the ultimate guide (using NestJS)".

scalablebackend.com

Cover photo by Thais Morais on Unsplash

How To Fail at Micro-Services

Teddy MORIN — Fri, 18 Aug 2023 17:18:25 +0000

Doing micro-services the wrong way can be a death sentence for your project. If your architecture resonates with some elements of todays list, it might be a good time to stop what youre doing and reflect on it.

First, we need to define the basic properties of micro-services. Amazing resources like Building Microservices and Microservice Architecture tell us they are supposed to be:

Small in size
Messaging enabled
Bounded by contexts
Autonomously developed
Independently deployable
Decentralized
Built and released with automated processes
Loosely coupled
Focused on one thing

If you built your back-end with micro-services in mind, but dont respect some of those properties, its an obvious sign you made a mistake.

But sometimes, wrong architecture choices sneak up on you. You end up with something close to micro-services but dont respect some of their rules.

Distributed monolith

The most common is to build a distributed monolith instead of micro-services. This is a complete subject in itself, I highly recommend you read more about it.

Understand the Difference Between Monolith, MicroServices, and Distributed Monolith

Verify your knowledge of modern software architecture

blog.scalablebackend.com

Shared database

Lets get this straight: its impossible to create independent and loosely coupled services with a shared database.

You will quickly realize how shared ownership of a database is a nightmare and has a negative impact on development and maintenance. It doesnt integrate too well with ORMs, and Im not even talking about deployment.

Yes, in a monolith you can have a single database, which makes things easy. With micro-services, you dont have a choice but to make it more complex by separating databases for each service.

Its one of the trade-offs with micro-services, its an added complexity that allows you to take advantage of micro-services.

Lack of automation

Micro-services definitely require more effort to develop, deploy, and maintain. Thats why one of the cornerstones of your system should be automation.

I guess you could try to create a few micro-services without automation, but on a real project thats simply unrealistic. Mainly, testing and deployment quickly become impossible without a completely automated CI/CD system.

Shared business logic

Sharing business logic goes against the properties expected from micro-services, as it makes your services highly coupled.

And, in practice, you will also realize it has fewer benefits than expected. Using a shared package for business logic forces you to update the package itself, before updating your services.

It makes development longer and causes context-switching for developers, without bringing any real value.

Starting with micro-services

In recent times, micro-services became trendy. While they solve several specific challenges, theyre not a silver bullet.

In practice, they should only be used when you already have a monolith, but have challenges micro-services solve.

Lets say you started your back-end application, day one, as micro-services. There is a higher probability you did so because you wanted to use micro-services than to solve a real challenge.

Lack of tests

Micro-services should be tested as much, if not more, compared to a monolith.

They should be tested in isolation, but thats not all. If your services communicate (usually through events), it means they agree on a contract (implicitly or explicitly).

This contract should definitely be a part of the scope of your tests.

Not independently deployable

If, for some reason, you are unable to deploy a new version of a service in autonomy, its an obvious sign your services are highly coupled.

You shouldnt need the other services at any point in the development and deployment process of a service. In Hands-on Microservices with Kotlin, we learn:

Having the ability to deliver constantly is one of the advantages of the microservices architecture; any constraints should be removed, as much as we remove bugs from our applications.

We should take care of deployments from the beginning of the design of our microservices and architecture; finding a constraint on this area at late stages could have a big impact on the overall application.

Synchronous communication between services

Using synchronous (request-response) communication between services breaks the independent & loosely coupled properties.

If you use synchronous communication between services, you are actually building a distributed monolith. After reading about distributed monolith, you should know its the exact opposite of what we want.

It doesnt bring the added value of micro-services but also loses the simplicity that comes with a monolith.

Moreover, with micro-services, we need to guarantee messages get delivered. With request-response communication, when a process extends to more than one service, its simply impossible.

Need for other services to work

Maybe the most important , but still the most often ignored.

One of the downsides of synchronous communication between services is the need for multiple services to work at the same time.

In theory, one service should be able to work perfectly fine even if all other services are down. Obviously, the system itself wont work as expected, but services in isolation should.

This is generally made possible by:

Limiting relationships between services
Using deduplication
Communicating through events.

How To Handle Authentication with Micro-Services

Teddy MORIN — Wed, 16 Aug 2023 16:58:06 +0000

With micro-services, and more generally distributed systems, come a lot of new challenges. One of them is how to handle authentication.

If you are unsure about the exact properties of micro-services and the challenges that come along, feel free to read about it more in-depth:

Understand the Difference Between Monolith, MicroServices, and Distributed Monolith

Verify your knowledge of modern software architecture

blog.scalablebackend.com

Authentication & Authorization

Actually, authentication is not the only step we have to manage, but also authorization.

If youre not sure about the difference, Auth0 reminds us that:

In simple terms, authentication is the process of verifying who a user is, while authorization is the process of verifying what they have access to.

In practice, you can see authentication as the process where you send your email and password. You usually retrieve something that identifies you as the user, which you send along with other requests.

That way, on every other request, the server knows who you are and what resources you have access to. He can then decides if you can access certain endpoints, this is authorization.

Now, I would like to differentiate authentication systems into two groups with different challenges. They are stateful and stateless authentication systems.

Stateful authentication

In a stateful authentication system, you need to retrieve the users information every time hes making a request.

For example, after authentication, you can retrieve a unique id that ties you to a user in database. That way, the server can find who you are, without sending your e-mail and password every time.

But the server still needs to verify the users information that is tied to the unique id youre sending. Only then can he decides if you can access a given resource.

That means, even if youre sending a unique ID that identifies you, the server has to retrieve the user information before handling a request.

This authorization step becomes problematic when it comes to micro-services. If youre doing micro-services well, you want to avoid high coupling, which implies two things (among others):

Separate database
No synchronous communication (request/response) between services.

That means, with the current state of things, we cannot authorize a user on a service outside of the one dedicated to authentication.

Now, we have to think about a way to make authorization possible but keep micro-services independent and lowly coupled.

There is one main solution: managing authentication through an API gateway.

Here, we use an API gateway whose role is to proxy requests to the right service. But, before proxying requests, it checks if the user sent an ID he wants to authenticate with.

If he did, the API gateway retrieves the data related to the user and adds it to the request before it gets sent to the right service.

For example, it encodes the user information inside a string, that is passed with a header.

This way, any service can decode the user information, without relying on the user service.

In my opinion, stateful authentication is simply not a great authentication system when it comes to micro-services.

It makes development and maintenance harder, has a bad impact on performance, and opens the door to security exploits.

While the previous solution is possible, I would prefer using a stateless authentication system.

Stateless authentication

In a stateless authentication system, the users information is stored by the client. That means, once you are authenticated, there is no need to retrieve the users information.

For example, a stateless authentication system can be set up using JSON Web Token.

https://blog.scalablebackend.com/vulnerabilities-in-authentication-with-jwt

Here, the process becomes much more straightforward. We generate a JWT by authenticating to the user service and sending it with requests to other services.

Other services dont need to reach the user service as the user information is self-contained.

It makes everything simple by design and helps a lot in keeping our services completely independent.

While this is my opinion, its definitely my favorite way to handle authentication with micro-services.

Are you looking for a more in-depth , and practical course on micro-services? Good news, I teach micro-services in a complete guide :

Enterprise Grade Back-End Development with NodeJS

Learn how to write reliable backend applications using Node.JS & NestJS!

scalablebackend.com

Cover photo by Max Harlynking on Unsplash

Vulnerabilities in Authentication with JWT

Teddy MORIN — Mon, 14 Aug 2023 16:49:41 +0000

After working with JWT more in-depth for the past few months, I realized most of the learning materials are of poor quality.

Today, I want to make it clear how JWT should be used in your authentication flow, what are its security vulnerabilities, and how to avoid them.

What is a JWT

From its introduction page, we learn the following:

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object

JSON Web Tokens - jwt.io

JSON Web Tokens are an open, industry standard RFC 7519 method for representing claims securely between two parties.

jwt.io

Content

In practice, a JWT is a string that looks like xxxxx.yyyyy.zzzzz , where the sections are respectively the header (xxxxx ), payload (yyyyy ), and signature (zzzzz ).

Header

The header is a JSON object, which typically defines the algorithm used, and the type of the token JWT. Its encoded in base64 to be used as a string.

Payload

The payload is also a JSON object, where you define the user information. its also encoded in base64.

Signature

The signature is generated based on the header and payload, using an algorithm (such as HMAC SHA256) and your secret.

Result

In the end, you generate a JWT, which anyone can read the information of. But you are the only one able to verify a JWT has not been tampered with. Nobody can change a JWT payload and sign it with your own secret.

Processes

Authentication

In the authentication process, a user typically sends his credentials to an API, which tries to find the corresponding account in a database.

If an account is found, a JWT is generated with the user information, such as id, name, or even his roles (admin? user?).

Authorization

Then, a user is able to request a private endpoint, where he needs to be authenticated. This is known as the authorization process:

Because a JWT contains user information (its stateless/self-contained), the API doesnt need to request a database. This is amazing in terms of performance, and even better on distributed architecture.

This is the normal use case of a JWT, if youre making requests to your database during authorization, you defeat the purpose of JWT.

Limitations

On one side, the self-contained aspect of JWT makes it amazing. On the other, because youre not requesting to your database, you cannot invalidate a JWT.

This is an issue for both functionality & security. If a user gets his token stolen, or if you have a role system and someone gets a role removed, he can still use a previous token when he shouldnt be allowed to (while the token is not expired).

This problem is solved by limiting the lifetime of tokens to a short duration, such as 5 minutes. But you dont want to ask a user credentials every five minutes.

Thats why you need to implement refresh tokens. Its often treated as beyond the scope of basic learning materials, but its mandatory.

What Are Refresh Tokens and How to Use Them Securely

Learn about refresh tokens and how they help developers balance security and usability in their applications.

auth0.com

Refresh tokens

Refresh tokens are completely different from the regular JWT you use for authorization (which are called identity tokens). They are long-lived tokens (~7 days) and can be used a single time to generate a new identity token.

If you respect those properties, you can implement identity tokens in different ways. You can have a table that stores refresh tokens and the corresponding user, which is updated every time its used.

For refresh tokens, I usually generate a JWT where the payload contains two properties, a sub, and userId. The sub contains a UUID, which is stored in a database, and map to its corresponding user.

When a user tries to log in based on a refresh token, I find the corresponding sub in my database and verify the userId it maps to is the right one (it avoids a potential situation where a user connects with a previous user refresh token UUID).

In the end, you should have two endpoints for login, one with user credentials, and one with a refresh token. Now, there are multiple ways to generate and store those tokens, which leads us to the next section: vulnerabilities.

Security vulnerabilities

XSS attack

There is one implementation issue Ive seen too many times, how to store a JWT. In most online examples, you can see a JWT being returned inside a request response (body), and stored in localStorage or simply in memory.

This is the source of a huge security vulnerability, XSS attack.

owasp.org

An XSS attack is possible when a malicious third party manage to inject code inside your application.

From here, anything allowed by your runtime is possible. A third party could secretly read the content of localStorage and send it to their own server, stealing JWTs for example.

Storing your JWT in-memory isnt enough. A third party can easily intercept a request response, and read users JWTs from there. There is a single solution: storing them inside secured cookies.

A secure cookie is configured with, at least, the Secure and HttpOnly attributes. Its also a good practice to use SameSite to avoid CSRF.

There is also some arguments in favor of storing refresh tokens inside localStorage instead of cookies. The impact of a refresh token being stolen is reduced by its one-time only validity.

CSRF

Cookies are much better than localStorage for our use case, but theyre not perfect. With cookies, you dont control when they are sent, your browser sends them with every request.

Its the source of CSRF, short for Cross Site Request Forgery.

owasp.org

From a general perspective, CSRF can happen when a third party trick a user into making a malicious request. The CSRF page from OWASP gives an amazing scenario with a bank transfer endpoint.

Using cookies with SameSite mitigates CSRF, but only a CSRF token can completely get rid of it.

Signing

There are two potential vulnerabilities when you sign a JWT token, bad algorithm and secret key.

JWT can be signed with different algorithms. The list can differ depending on which library you are using. Library authors are responsible to implement those.

The default algorithm is usually HS256, but using a bad implementation or wrong configuration might end up with you using the none algorithm.

Critical vulnerabilities in JSON Web Token libraries

Which libraries are vulnerable to attacks and how to prevent them.

auth0.com

What this algorithm does is nothing! It generates an empty signature, which allows any third party to modify the JWT payload, and your server will still believe the modified JWT is valid.

My advice is to use well-known/maintained libraries and not try to use the none algorithm. You can also verify the content of tokens you generate using jwt.io.

Brute Forcing HS256 is Possible: The Importance of Using Strong Keys in Signing JWTs

Cracking a JWT signed with weak keys is possible via brute force attacks. Learn how Auth0 protects against such attacks and alternative J...

auth0.com

Previously, we talked about refresh tokens, but there is a vulnerability introduced by using different types of tokens.

If you configure your refresh tokens to be signed with the same secret as the identity tokens, a malicious user could send an identity token where you expect a refresh token and vice-versa.

Depending on your implementation, you might grant access to a malicious user where you shouldnt. There is a single solution: use a different secret for your identity & refresh tokens.

Validation

During the validation phase, bad implementation could introduce security vulnerabilities.

jsonwebtoken - npm

JSON Web Token implementation (symmetric and asymmetric). Latest version: 9.0.1, last published: a month ago. Start using jsonwebtoken in your project by running `npm i jsonwebtoken`. There are 24736 other projects in the npm registry using jsonwebtoken.

npmjs.com

There is an example with the NodeJS jsonwebtoken library. The right implementation is to use the verify method to ensure the token is valid and decode it.

On the other hand, it provides a decod method that doesnt check if the token is valid. Some developers not used to working with JWT might use the second method instead, virtually accepting any JWT token.

Ensure you are using well-known libraries and learn them properly before implementing anything sensitive.

Authors note

I advocate for the use of cookies over localStorage to mitigate XSS attacks, but thats not a reason to ignore potential XSS attacks altogether. I definitely recommend following good practices regarding XSS attacks.

For example, using JS eval should be avoided. You can also verify your dependencies using tools like snyk.io, and only use trusted CDN.

Auth0: Secure access for everyone. But not just anyone.

Rapidly integrate authentication and authorization for web, mobile, and legacy applications so you can focus on your core business.

auth0.com

Building your own authentication system is an arduous task, I would recommend anyone to either use an authentication provider such as Auth0, or have a dedicated team working full time on authentication.

Do you want to learn how to create a backend application, add a secure authentication system , and much more?

Enterprise Grade Back-End Development with NodeJS

Learn how to write reliable backend applications using Node.JS & NestJS!

scalablebackend.com

Cover photo by Edwin Hooper on Unsplash

Understand the Difference Between Monolith, MicroServices, and Distributed Monolith

Teddy MORIN — Wed, 09 Aug 2023 18:26:46 +0000

Often, inexperienced teams want to create micro-services but end up with a distributed monolith. What is it and why should you avoid it

First, we need to understand the difference between a simple monolith and micro-services.

Monolith

Impressive only by name, monolithic is in fact the most common architecture style. It's when everything is contained in a single piece, or in other words, self-contained.

Architecture

In practice, a monolithic backend looks like a single and large codebase that provides all the APIs you need.

Now, the questions are, why is monolithic the traditional architecture, and why would you want to create micro-services instead?

A monolithic architecture comes with much more simplicity at the beginning of a project and is sufficient for most projects.

When your whole codebase is in the same place, development, testing, monitoring, and deployment are much more straightforward, faster, and cheaper.

Challenges

But when your application grows, with more functionality and users, new challenges appear. They could be gathered under two main categories, codebase maintainability and scalability.

In a monolith, it's a common practice to organize code by creating abstractions, such as modules.

But those abstractions and their boundaries usually break down with time, and similar code starts to become spread all over.

With a large codebase, it becomes difficult to know where a change needs to be made, making it harder to fix bugs and implement new features.

Scaling

Then, come the issues with scalability. When a system needs more resources, there are two ways to scale it: horizontally, and vertically.

Vertical scaling refers to adding more resources to an existing machine, such as GPU or RAM. Horizontal scaling refers to spreading out your application on multiple machines and adding more machines to your pool of resources.

Vertical scaling can be enough but will be limited at some point, as there is a limit to how much resource a single machine can have.

Horizontal scaling is in theory infinite. On the other hand, it requires your application to be able to work when spread out on multiple machines.

Even a monolithic application could work if it's spread out on multiple machines, as long as it's stateless. But monoliths have a weakness that micro-services don't have when it comes to horizontal scaling.

As a monolith contains the whole system in a single codebase, everything must be scaled together. If you only need to scale a subset of endpoints, you have no choice but to deploy the whole system.

Microservices

Unfortunately, microservices are described in a lot of different ways.

Architecture

A succinct definition I particularly like comes from "Building Microservices" by Sam Newman:

Microservices are small, autonomous services that work together.

That's enough for a first approach but lacks details to understand microservices at a deeper level. From "Microservice Architecture", we learn microservices should share the following traits:

Small in size
Messaging enabled
Bounded by contexts
Autonomously developed
Independently deployable
Decentralized
Built and released with automated processes

Now, what does that mean, in practice? In the real world, your app is split into multiple services with its own set of related functionalities (while ensuring low coupling between them).

There are, at the same time, multiple instances of the same service depending on the load to handle. They communicate asynchronously to ensure messages get delivered and avoid coupling.

Challenges

Opposite to monoliths, why is development, testing, monitoring, and deployment slower and more expensive with microservices?

New challenges appear with multiple applications. For example, managing data in separate but related databases comes with more challenges than a single, unified database.

Then, you need some sort of communication between services, usually through an event bus. Making your micro-services work together, testing them correctly, as well as deployment will be a serious headache.

In the end, you should expect development to take more time with micro-services. The increased complexity also means you need a bigger team to manage it, setting up micro-services with a single and small team is a bad idea.

Scaling

There is one primary benefit with micro-services. They can be easily deployed as you need them, and scale more efficiently.

With micro-services, you can just scale the service that needs scaling, making it possible to run it on smaller, less powerful hardware. It makes it faster and more cost-effective.

Micro-services provide other benefits compared to monoliths, such as resilience, ease of deployment, and replaceability. As a monolith contains your whole app, if one of its components fails, everything else becomes unavailable.

Also, you might have experienced how hard it can be to refactor a huge codebase, whereas the size of a micro-service is usually limited. The cost to replace a service is then much more manageable.

On large projects, with numerous teams, micro-services might even have a positive impact on productivity

Distributed Monolith

A distributed monolith is the result of splitting a monolith into multiple services, that are heavily dependent on each other, without adopting the patterns needed for distributed systems.

In practice, it's the result of splitting a monolith into separate services, but keeping them tightly coupled.

That means, they still rely heavily on each other. In this context, you lose the simplicity that comes with a monolithic architecture, but don't enjoy the benefits of independent microservices.

Don't do it!

Being lowly coupled doesn't mean you have absolutely no relationship either. For example, you might send and listen to events.

Also, you should write some integration tests, or to be more precise contract tests (if you're doing microservices well).

That means, at some point, you need information on the contracts from other services. There is some relationship going on, but it doesn't make your services tightly coupled.

Do you want to learn more about micro-services?
Good news, I teach micro-services in a practical course.

Cover photo by Crawford Jolly on Unsplash

How To Write Efficient Unit Tests with Prisma ORM

Teddy MORIN — Fri, 09 Dec 2022 09:55:55 +0000

During my life as a full-stack developer, I tried out lots of ORM, ODM, and query builders. Some were fantastic, while others still give me nightmares (we see you Sequelize).

Current state

Among the fantastic tools, there is Prisma. It works perfectly, has a great DX, documentation, and much more. I encountered a single issue when working with it, unit testing.

While they have a documentation page on unit testing, with a great introduction, their method of mocking is unsatisfactory at best.

Unit testing with Prisma

Learn how to setup and run unit tests with Prisma Client

prisma.io

At best, that allows you to mock, one by one, the responses for the requests you believe will be called. I think this is inefficient for a few reasons:

It requires too much setup, on every single test.
There is a very low level of trust in the mocks you define, as you are not sure Prisma will return the same data.
By defining these mocks yourself, you get further away from testing your app behavior, and closer to testing implementation details.
With new versions of Prisma, you are at risk of seeing your tests being obsolete.

kentcdodds.com

Lets look at what application we want to test, which tests to write, and then define how to write unit tests.

Application structure

My backend applications are usually split into a few layers. If we take an example with a basic ExpressJS app, it has at least 3 layers: module, controller, and service.

Each set of functionality is separated into a dedicated module , which handles routing, validation, and passes the request to the controller.

The controller is where the business logic is found, and often makes use of services.

A service provides high-level functions that make requests to the database. This is where I use ORM such as Prisma.

What should be tested

I usually have two primary types of tests when working on a backend application (and many more depending on the context!). They are unit and end-to-end tests.

I try to avoid writing e2e tests when possible, as they have limitations like being slow, expensive, and giving late feedback. I usually write unit tests for low-level functions such as middlewares.

On the other hand, I dont test my services or controllers in complete isolation. In those cases, I feel like its not giving me enough value for the time it takes to write tests. Instead, I want to test my endpoint behavior as a whole.

To do it, I write my tests using a test version of my backend app. Outside dependencies, like Prisma, are mocked (in an efficient way) which allows me to simulate queries in isolation using tools like SuperTest.

In this context, I write unit tests for most use-cases. It includes validation (such as parameters), authorization (ensure youre connected with the right access), but I also verify I receive the right response, using the mocked version of Prisma. Depending on my app, there is even more use-case.

I also end up writing e2e tests, to ensure my successful response and database-dependent errors, are the expected ones, in a real environment.

Those tests may partially overlap with my unit tests, but this time using a real database. It gives me the quick and early feedback of unit tests while having the high confidence of e2e tests.

How to write those tests

In the following examples, I expect you to understand the basics of testing. That includes Jest, which is used as part of the examples.

Testing with Node.js: Understand and Choose the Right Tools

What are the tools you need for testing with Node.js and which ones to choose for your purpose?

blog.scalablebackend.com

Middleware

Middlewares dont need a special environment to be tested in, they can be considered just like any other function.

Only thing is, they have a defined format. For Express, they must return a function that takes a request , response , and next function. Lets have a look at the following snippet:

import { Request, Response, NextFunction } from 'express';

type Validator = (req: Request, res: Response) => boolean;

export function validator(validate: Validator) {
  return (req: Request, _res: Response, next: NextFunction) => {
    const valid = validate(req);
    if (!valid) next(new Error());
    else next();
  };
}

Validator takes a function that determines if a request is considered valid, by returning a boolean based on the received request. It returns a middleware, which will throw an error if the received request is determined as invalid.

With Jest, this middleware can be tested quite easily by calling it with different arguments, and verifying next has been called with the right element.

import validator from '../validator.middleware';

describe('validator', () => {
  const req = { params: { mock: 'true' } };
  const res = {};

  it('Should call next with an error if request is invalid', () => {
    const mockNext = jest.fn();
    validator((req) => req.params.mock === 'false')(req, res, mockNext);
    expect(mockNext).toHaveBeenCalledWith(new Error());
  });

  it('Should call next otherwise', () => {
    const mockNext = jest.fn();
    validator((req) => req.params.mock === 'true')(req, res, mockNext);
    expect(mockNext).toHaveBeenCalled();
  });
});

Endpoints (Unit)

I already explained I use SuperTest for my endpoints. I also talked about using a test version of my backend app. To be more precise, I have a few helper functions, dedicated to bootstraping my backend during tests and managing mocked data.

The following snippet is a good example of a unit test. We demonstrate how we can test an endpoint dedicated to creating an article.

We use helpers to bootstrap our app, generate the tokens needed for our use case, and send the request with SuperTest.

import { Express } from 'express';
import supertest from 'supertest';

import { ArticleFixture, ServerMock, UserMock } from '../../../../testing';

describe('POST /article', () => {
  let app: Express;
  const tokens = UserMock.generateTokens();

  beforeAll(async () => {
    app = await ServerMock.createApp();
  });

  test("Should return error if user isn't authenticated", () => {
    return supertest(app).post('/article').send(ArticleFixture.articles.newArticle).expect(401);
  });

  test("Should return error if user doesn't have ADMIN role", () => {
    return supertest(app)
      .post('/article')
      .set('Authorization', `Bearer ${tokens.user}`)
      .send(ArticleFixture.articles.newArticle)
      .expect(401);
  });

  it('Should create article', () => {
    return supertest(app)
      .post('/article')
      .set('Authorization', `Bearer ${tokens.admin}`)
      .send(ArticleFixture.articles.newArticle)
      .expect(201);
  });
});

Mocking with Prisma

But we still didnt solve the issue that comes with Prisma. In the above snippet, it looks like nothing is mocked.

There is a single solution if we want to write tests with no dependence to a database or heavy mocking: using an in-memory implementation of Prisma.

Introducing prismock. Disclaimer: I am indeed its creator.

prismock - npm

A mock for PrismaClient, dedicated to unit testing.. Latest version: 1.17.0, last published: 2 days ago. Start using prismock in your project by running `npm i prismock`. There are no other projects in the npm registry using prismock.

npmjs.com

As there was no satisfying solution to efficiently write unit tests with Prisma, I decided to write my own solution.

It actually reads your schema.prisma and generates models based on it. It perfectly simulates Prismas API and store everything in-memory for fast, isolated, and retry-able unit tests.

Remember how I use a helper to build a test version of my backend app?

In production I build my app, using a genuine PrismaClient , which is then bootstrapped. During my test, I replace PrismaClient using dependency injection.

In the above snippet, its done as part of ServerMock.createApp(), which makes it virtually invisible when I write my tests.

Endpoints (E2E)

In a context where our article endpoint and authorization process are already tested, we could argue that its not mandatory to test authentication on every single endpoint during e2e tests.

For example, we could end up with the following test:

import supertest from 'supertest';

import { ArticleFixture } from '../../fixtures';
import { ArticleMock } from '../../mocks';
import E2EUtils from '../EndToEndUtils';

describe('POST /article', () => {
  let tokens: { admin: string };

  beforeAll(async () => {
    tokens = await E2EUtils.generateTokens();
  });

  it('Should return created article', async () => {
    return supertest('http://localhost')
      .post('/article')
      .set('Authorization', `Bearer ${tokens.admin}`)
      .send(ArticleFixture.articles.newArticle)
      .expect(201)
      .then((response) => {
        expect(response.body).toEqual({
          title: ArticleMock.articles.newArticle.title,
          content: ArticleMock.articles.newArticle.content,
          slug: ArticleMock.articles.newArticle.slug,
        });
      });
  });
});

This test must be written in a different environment, where we have access to a seeded database, and our endpoint has been built and runs in a near-production environment.

Conclusion

In this context, we should cover the entirety of our codebase with unit tests, giving us fast and early feedback, with a strong trust in our test results.

Using an in-memory implementation of Prisma instead of manual mocks increases our confidence even more. Together with our testing strategy (defined in what should be tested ), we also end up with amazing productivity.

Finally, we write E2E tests exclusively for use-cases that make requests to our database. It does overlap with some unit tests, is slower, and more expensive with late feedback, but it eliminates the remaining grey areas.

We are then able to answer:

Are you confident in shipping your app to production?

Enterprise Grade Back-End Development with NodeJS

Learn how to write reliable backend applications using Node.JS & NestJS!

scalablebackend.com

Photo by Roman Mager on Unsplash

React Testing Library Configuration for Productive Unit Testing

Teddy MORIN — Mon, 11 Jul 2022 17:52:01 +0000

Too often, I join a new React project where unit tests are lacking, both in amount and quality. There are a few causes, but the one I want to discuss today is the poor test environment.

Indeed, testing requires skills, thoroughness, and is definitely time-consuming (even if thats worth it!). If testing is more painful than necessary, it becomes a signal to avoid writing tests altogether.

Environment

With React, the tools I recommend are Jest and React Testing Library. Nothing fancy here; they are the de-facto standard in the community.

React Testing: Understand and Choose the Right Tools

Save time and effort with React and React Native by choosing the appropriate testing tools.

morintd.hashnode.dev

Example

To demonstrate how to write great tests, in a good environment, we need a component to test, right? Lets use a common functionality: a counter.

Ill start with the following component, inside its own Counter.tsx file.

import { useState } from 'react';

const Counter = () => {
  const [counter, setCounter] = useState(0);
  const decrement = () => setCounter((count) => count - 1);
  const increment = () => setCounter((count) => count + 1);

  return (
    <div>
      <h1>Counter: {counter}</h1>
      <button onClick={decrement}>-</button>
      <button onClick={increment}>+</button>
    </div>
  );
};

export default Counter;

I would like to specify my application entrypoint is still App.tsx. It will be modified later in this article to demonstrate our configuration!

import Counter from './Counter';

function App() {
  return <Counter />;
}

export default App;

Tests

A complete repository can be found on GitHub.

React testing library

After following the introduction and adding jest-dom, I set up the first version of my test, as shown below:

import { fireEvent, render, screen } from '@testing-library/react';

import Counter from '../Counter';

describe('Counter', () => {
  it('Should display a default value', () => {
    render(<Counter />);
    expect(screen.queryByText('Counter: 0')).toBeInTheDocument();
  });

  it('Should increment', () => {
    render(<Counter />);
    fireEvent.click(screen.getByText('+'));
    expect(screen.queryByText('Counter: 1')).toBeInTheDocument();
  });

  it('Should decrement', () => {
    render(<Counter />);
    fireEvent.click(screen.getByText('-'));
    expect(screen.queryByText('Counter: -1')).toBeInTheDocument();
  });
});

We can find three basic tests. They help us verify weve displayed a default value, which increments or decrements when the corresponding button is clicked.

With the current configuration, Im able to run my test successfully:

But issues arise when working with a bigger codebase, more functionalities, and dependencies. In this article, I would like to demonstrate how we handle Redux and GraphQL which are fairly common.

Adding Redux

Im following RTK Quick Start, which conveniently shows an example with a counter app. I end up with the following slice:

import { createSlice } from '@reduxjs/toolkit';

export interface CounterState {
  value: number;
}

const initialState: CounterState = {
  value: 0,
};

export const counterSlice = createSlice({
  name: 'counter',
  initialState,
  reducers: {
    increment: (state) => {
      state.value += 1;
    },
    decrement: (state) => {
      state.value -= 1;
    },
  },
});

export const { increment, decrement } = counterSlice.actions;
export default counterSlice.reducer;

The following store:

import { combineReducers, configureStore } from '@reduxjs/toolkit';

import counterReducer from './counter.slice';

export const rootReducer = combineReducers({
  counter: counterReducer,
});

export const store = configureStore({
  reducer: rootReducer,
});

export type RootState = ReturnType<typeof store.getState>;
export type AppDispatch = typeof store.dispatch;

And an updated Counter.tsx component:

import { useDispatch, useSelector } from 'react-redux';
import { decrement, increment } from './store/counter.slice';

import { RootState } from './store/store';

const Counter = () => {
  const counter = useSelector((state: RootState) => state.counter.value);
  const dispatch = useDispatch();

  return (
    <div>
      <h1>Counter: {counter}</h1>
      <button onClick={() => dispatch(decrement())}>-</button>
      <button onClick={() => dispatch(increment())}>+</button>
    </div>
  );
};

export default Counter;

The behavior of my counter didnt change, but tests are failing with the following error message:

Could not find react-redux context value; please ensure the component is wrapped in a .

Its quite straightforward. We are trying to test a component in isolation, but it needs a react-redux provider to work. I added it to my App.tsx, but from my test perspective, its nowhere to be seen.

Now, for every test, we need to declare a new store and render our component with the Provider from react-redux. Technically, it would work with the following code:

import { fireEvent, render, screen } from '@testing-library/react';
import { Provider } from 'react-redux';
import { configureStore } from '@reduxjs/toolkit';

import Counter from '../Counter';
import { rootReducer } from '../store/store';

describe('Counter', () => {
  it('Should display a default value', () => {
    const store = configureStore({ reducer: rootReducer });
    render(
      <Provider store={store}>
        <Counter />
      </Provider>
    );

    expect(screen.queryByText('Counter: 0')).toBeInTheDocument();
  });

  it('Should increment', () => {
    const store = configureStore({ reducer: rootReducer });
    render(
      <Provider store={store}>
        <Counter />
      </Provider>
    );

    fireEvent.click(screen.getByText('+'));
    expect(screen.queryByText('Counter: 1')).toBeInTheDocument();
  });

  it('Should decrement', () => {
    const store = configureStore({ reducer: rootReducer });
    render(
      <Provider store={store}>
        <Counter />
      </Provider>
    );

    fireEvent.click(screen.getByText('-'));
    expect(screen.queryByText('Counter: -1')).toBeInTheDocument();
  });
});

But, is that the right solution? Thats a lot of unneeded boilerplate code. If you ever have more dependencies, your tests will grow exponentially. It makes them hard to write and maintain.

Instead, React Testing Library explains how to set up a Custom Render.

Setup | Testing Library

React Testing Library does not require any configuration to be used. However,

testing-library.com

Setup improvement

After following the Custom Render section, I end up creating a tests/ directory with a testing.tsx file:

import React, { FC, ReactElement } from 'react';
import { render, RenderOptions } from '@testing-library/react';
import { Provider } from 'react-redux';

import { rootReducer } from '../src/store/store';
import { configureStore } from '@reduxjs/toolkit';

const AllTheProviders: FC<{ children: React.ReactNode }> = ({ children }) => {
  const store = configureStore({ reducer: rootReducer });
  return <Provider store={store}>{children}</Provider>;
};

const customRender = (ui: ReactElement, options?: Omit<RenderOptions, 'wrapper'>) => {
  return render(ui, { wrapper: AllTheProviders, ...options });
}

export * from '@testing-library/react';
export { customRender as render };

I add an index.ts file that re-exports everything from my tests/ directory Then, instead of importing my Redux boilerplate code and utilities from @testing-library/react, I import utilities from this directory:

import { fireEvent, render, screen } from '../../tests';
import Counter from '../Counter';

describe('Counter', () => {
  it('Should display a default value', () => {
    render(<Counter />);
    expect(screen.queryByText('Counter: 0')).toBeInTheDocument();
  });

  it('Should increment', () => {
    render(<Counter />);
    fireEvent.click(screen.getByText('+'));
    expect(screen.queryByText('Counter: 1')).toBeInTheDocument();
  });

  it('Should decrement', () => {
    render(<Counter />);
    fireEvent.click(screen.getByText('-'));
    expect(screen.queryByText('Counter: -1')).toBeInTheDocument();
  });
});

Thats much better! But, what if we need to trigger some change to our Redux store during our test? Also, when our app grows, adding dozens of providers inside our testing.tsx will make it harder to read and maintain.

First, I want to make my render more customizable. If you look at the customRender method, you can see it takes some options related to React Testing Library.

We can use those options to customize our providers. Ill allow a new property, providers, which is an object with the data related to our providers. For now, it takes the following:

export type ProvidersRenderOptions = {
  store?: Store;
};

export type CustomRenderOptions = {
  providers?: ProvidersRenderOptions;
};

The implementation looks like this:

import { ReactElement } from 'react';
import { render, RenderOptions } from '@testing-library/react';
import { Provider } from 'react-redux';

import { rootReducer } from '../src/store/store';
import { configureStore, Store } from '@reduxjs/toolkit';

export type ProvidersRenderOptions = {
  store?: Store;
};

export type CustomRenderOptions = {
  providers?: ProvidersRenderOptions;
};

const AllTheProviders = (options: ProvidersRenderOptions = {}) => ({ children }: { children: React.ReactNode }) => {
  const store = options.store ?? configureStore({ reducer: rootReducer });
  return <Provider store={store}>{children}</Provider>;
};

const customRender = (ui: ReactElement, options: CustomRenderOptions & Omit<RenderOptions, 'wrapper'> = {}) => {
  const { providers, ...others } = options;
  render(ui, { wrapper: AllTheProviders(providers), ...others });
};

export * from '@testing-library/react';
export { customRender as render };

Looks good, Ill even throw in a helper function to build a store:

export const generateStore = (preloadedState: PreloadedState<typeof rootReducer>) => {
  return configureStore({
    preloadedState,
    reducer: rootReducer,
  });
};

This way, Im able to write a more advanced test:

  it('Should display a default value', () => {
    const store = generateStore({ counter: { value: 5 } });
    render(<Counter />, { providers: { store } });
    expect(screen.queryByText('Counter: 5')).toBeInTheDocument();
  });

That looks quite good! But we still have one issue: the testing.tsx file will grow to become unmaintainable with the current state of things.

Instead, I would like to move the declaration for providers to different files, and build the function allTheProviders on the fly. Also, there is a great pattern to build it, function composition!

Let me explain. Instead of having one huge function, I create one for each provider I want to add. These new functions take options, a React node, and return a React node (with potentially a new provider).

This function, for Redux, would look like this:

export const withRedux = (children: ReactElement, options: ProvidersRenderOptions) => {
  return <Provider store={options.store ?? configureStore({ reducer: rootReducer })}>{children}</Provider>;
};

If they have this structure, I can chain them, or in other words, compose them, to build a component with multiple providers. If I split my list of providers, the function dedicated to composing them, and the function AllTheProviders, it looks like the following:

type ComposableProvider = (children: ReactElement, options: ProvidersRenderOptions) => ReactElement;

const providers: ComposableProvider[] = [withRedux];

const composeProviders = (children: ReactElement, options: ProvidersRenderOptions) => {
  return providers.reduce((component, provider) => {
    return provider(component, options);
  }, children);
};

const AllTheProviders = (options: ProvidersRenderOptions = {}) => {
  return ({ children }: { children: ReactElement }) => {
    return composeProviders(children, options);
  };
};

And thats it! This way, we can declare new (composable) provider functions, in a different file, and add them to our list.

Adding GraphQL

Lets improve our demonstration by adding GraphQL. Well add functionalities to load and save the current counter.

My schema and resolvers look like the following:

type Query {
  counter: Int
}

type Mutation {
  setCounter(counter: Int): Int
}


let savedCounter = 0;

export default {
  Query: {
    counter() {
      return savedCounter;
    },
  },
  Mutation: {
    setCounter(_: any, { counter }: { counter: number }) {
      savedCounter = counter;
      return savedCounter;
    },
  },
};

After setting up Apollo on the server and frontend, I added two queries to my counter:

export const GET_COUNTER = gql`
  query counter {
    counter
  }
`;

export const SET_COUNTER = gql`
  mutation setCounter($counter: Int) {
    setCounter(counter: $counter)
  }
`;

Then, I updated my Redux slice, and added two buttons in order to save and load the current counter:

export const counterSlice = createSlice({
  name: 'counter',
  initialState,
  reducers: {
    increment: (state) => {
      state.value += 1;
    },
    decrement: (state) => {
      state.value -= 1;
    },
    load: (state, action: PayloadAction<number>) => {
      state.value = action.payload;
    },
  },
});

export const { increment, decrement, load } = counterSlice.actions;


const Counter = () => {
  const counter = useSelector((state: RootState) => state.counter.value);
  const dispatch = useDispatch();

  const [getCount, query] = useLazyQuery(GET_COUNTER);
  const [mutate, mutation] = useMutation(SET_COUNTER, { variables: { counter } });

  useEffect(() => {
    if (query.data) dispatch(load(query.data.counter));
  }, [query.data]);

  return (
    <div>
      <h1>Counter: {counter}</h1>
      <button onClick={() => dispatch(decrement())}>-</button>
      <button onClick={() => dispatch(increment())}>+</button>
      <button disabled={mutation.loading} onClick={() => mutate()}>
        save
      </button>
      <button disabled={mutation.loading} onClick={() => getCount()}>
        load
      </button>
    </div>
  );
};

But now, just like for Redux, our tests throw an error:

Invariant Violation: Could not find client in the context or passed in as an option. Wrap the root component in an , or pass an ApolloClient instance in via options.

Lets follow the testing section from Apollo, and integrate it into our custom render. We can start by adding an option for GraphQL mocks and create a composable test provider for apollo:

export type ProvidersRenderOptions = {
  store?: Store;
  graphql?: MockedResponse[];
};


import { ReactElement } from 'react';
import { MockedProvider } from '@apollo/client/testing';

import { ProvidersRenderOptions } from '../testing';

export function withApollo(children: ReactElement, options: ProvidersRenderOptions) {
  return (
    <MockedProvider mocks={options.graphql ?? []} addTypename={false}>
      {children}
    </MockedProvider>
  );
}

Then, we can add this composable provider to our providers. Keep in mind the order of providers will have an impact on how our providers are added.

This is related to how function composition works. The most left in the list will be the inner providers, while the most right will be the outer providers.

const providers: ComposableProvider[] = [withRedux, withApollo];

Then, Im able to write the following test:

  it('Should load counter', async () => {
    const query = { request: { query: GET_COUNTER }, result: { data: { counter: 5 } } };
    render(<Counter />, { providers: { graphql: [query] } });

    fireEvent.click(screen.getByText('load'));

    await waitFor(() => {
      expect(screen.queryByText('Counter: 5')).toBeInTheDocument();
    });
  });

  it('Should save counter', async () => {
    const query = {
      request: { query: SET_COUNTER, variables: { counter: 1 } },
      result: jest.fn().mockReturnValue({ data: { setCounter: 1 } }),
    };

    render(<Counter />, { providers: { graphql: [query] } });

    fireEvent.click(screen.getByText('+'));
    fireEvent.click(screen.getByText('save'));

    await waitFor(() => {
      expect(query.result).toHaveBeenCalled();
    });
  });

And thats it! You are now able to write proper tests in a healthy environment. Moreover, you wont have any issues when your app gets bigger, as long as you continue to create composable test providers.

Dont forget a complete repository is available on GitHub.

Cover photo by sharonmccutcheon on Unsplash

Testing with Node.js: Understand and Choose the Right Tools

Teddy MORIN — Mon, 27 Jun 2022 05:17:17 +0000

If youre here today, there is a 99% chance you want to get started with testing in a higher-level environment, but lack the low-level knowledge. You might be working on Frontend or Backend with Node.js, but dont know where to start. Congratulations, youre in the right place.

Test types

If you are not familiar with testing, you might get a bit lost. There are a lot of test types, but some are more common.

The following article from Atlassian provides a good starting point.

As we will take our first step in the Node.js test environment, our scope will revolve around unit and integration tests.

Different tools

Now, you might be wondering about where to start. Youve seen a Node.js project that uses Jest, heard about Mocha or Chai but you are not sure what those are about.

Great, thats exactly what we will be discussing today. To have a clearer picture, we can separate the different kinds of tools we need.

Test runners

You have this vague idea of writing tests but how do you organize and run them? This is the role of test runners. Some of their functionalities are:

Finding test files and running tests
Reporting a test success or failure
Configuring which tests to run or re-run when updated
Setting up an environment

And a lot more.

Here is a graph comparing the most popular Node.js test runners by download:

Assertions

Now that you have an environment where you can write your tests, we come to the most important question: How do I write my first test?

From our previously cited article, Atlassian says the following about unit tests:

They consist in testing individual methods and functions of the classes, components, or modules used by your software.

From a practical perspective you can view it as the following:

Get the result from your individual unit (function for example) in a situation (with defined parameters).
Ensure the given result match the expected one.

Thats what assertions are about: comparing the received result with the expected one to report the test as a success or failure.

Like tests runners, there are several libraries helping you with assertions:

Spies

With unit and integration testing comes a difficulty:

You might want to unit test a function that relies on an external dependency.
For unit as well as integration tests, your function can have a native dependency you cannot use in a JS environment.
Your function is making an API call but your server is not set up in your testing environment.

Thats where spies are used. They help you create fakes (or mock objects) and replace dependencies with an object you defined.

In the same way, several spy libraries can help you:

Making a choice

Great, you now have a general idea of what tools you need and a list of the most popular ones. Now comes the hardest part: which tools to choose?

Test runner

To be honest, I never really considered Jasmine, Ava, and QUnit.

I first started testing with Backend projects. At the time, Mocha was the more widely used, thats the only reason why I started using it (with Chai).

Mocha worked great for a few years and then two changes happened:

I started to specialize in React
Jest started to overthrow Mocha

At this point, I had to do a serious comparison.

Mocha is a test runner that lets you choose an external assertion and spy library. It doesnt restrict you but gives you more flexibility.

Jest is a whole testing framework, shipped with React and maintained by Facebook. From the previous sections and graphs, you might have noticed that more than a test runner, Jest includes its own assertion and mock libraries.

Its interesting to note that Jest was built on top of Jasmine.

Jest · 🃏 Delightful JavaScript Testing

jestjs.io

Both are very similar. Of course, their APIs change, and you need to get used to it when going from one to the other.

Im a big fan of having a single library/framework/utility that fulfills a given role. After trying out Jest and experimenting with React, I definitely adopted it.

Afterward

Once you understand how to write tests, your best move would be to write great tests.

Test-Driven Development is a software development process that will definitely help you in writing great tests. Another useful tool is Code Coverage.

I would like to complete the article from Atlassian on Code Coverage by saying the following:

If your Code Coverage is below 100% you can be sure you didnt test everything. If its 100% you dont know.

Code Coverage can tell you if your code has been executed during your tests. Sometimes, youll end up executing code without really testing it. To be sure that you really tested everything, you must resort to Mutation Testing.

Stryker Mutator

Stryker Mutator: Test your tests with mutation testing.

stryker-mutator.io

Mutation Testing is great but really expensive, and cannot be used as much as unit/integration testing.

Once you get into testing, you often see the following pyramid:

What this represents is how many tests you should have. As unit tests are cheap to write and run, you should have plenty of them.

Still cheap but less than unit tests are: integration tests. Its expected for an application to have a lot of integration tests but less than unit tests.

Finally, youll have e2e tests but in fewer numbers, as those are expensive to set up, write, and run.

Good luck on your testing journey.

Cover photo by flowforfrank on Unsplash

What 4 Years Of Programming Taught Me About Writing Typed JavaScript

Teddy MORIN — Fri, 04 Dec 2020 17:41:19 +0000

Types of types

Mostly, statically typed languages are criticized for restricting developers. On the other hand, theyre loved for bringing early information about errors, documenting components (such as modules, methods, etc.), and now other more advanced functionalities such as auto-completion.

A preliminary study from 2009 on untyped languages gives us some reference on exactly those pros and cons. Today, another type of language is also widely used: dynamically typed languages.

A dynamically typed language is different from its counterpart by bringing types but at runtime. This way, you can have far more freedom than strongly typed languages while keeping their advantages.

From our list, we have a single dynamically typed language: TypeScript. And thats not completely exact, TS could also be called a soft-typed language, that is between a dynamically and statically typed language. As this is not todays subject, curious readers can have a look at the following article:

TypeScript: Static or Dynamic? The war is over. | by Vlad Balin | ITNEXT

Growing popularity of TypeScript revived an old static vs dynamic languages battles. When you’re engaged in it, you want yourself to be…

itnext.io

Why am I saying there is only one? Of course, JavaScript is considered untyped (or weakly typed), while PropTypes is a package allowing type-checking at runtime.

prop-types

prop-types - npm

Runtime type checking for React props and similar objects.. Latest version: 15.8.1, last published: 2 years ago. Start using prop-types in your project by running `npm i prop-types`. There are 54655 other projects in the npm registry using prop-types.

npmjs.com

Flow is neither. In practice, it looks a lot like TypeScript, and both are often compared. Inside your IDE and CLI, they are similar but their engines differ: TypeScript is a language while Flow is called a static type checker.

In Flow, you write annotations to set types. At compile time, those annotations must be removed, which creates JavaScript files without any superset.

It has been an argument in favor of Flow: performance. Both solutions have almost the same functionalities, but Flow removes any overhead that TypeScript has, once compiled.

My experience

I started my career in the JavaScript & front-end world in 2016 with Angular2 (and TypeScript).

Before this front-end project, I mainly worked on C#, Java, and a bit of Vanilla JavaScript. I hated it. To me, vanilla JavaScript had no structure, type, or object-oriented concepts, it was HELL.

With more experience, practice with Angular2 (shipped with TypeScript) and then React, I almost started to enjoy it. Thats when I seriously considered a way to type JavaScript: I used TypeScript for a short time (which I didnt master at the time) but I was back to untyped JavaScript with React.

React with untyped JavaScript was working not-so-bad but I felt like a piece was missing:

After a few weeks on a React project, code could easily get messy and difficult to understand, even more for newcomers.
We needed a lot of time to read a piece of code.
The number of errors after builds was too high.

Among the practices needed to avoid these problems, my experience told me that typed JavaScript was the top priority. After some research: here I was, looking at TypeScript, Flow, and PropTypes.

PropsTypes allowed me to type-check my props but that was not enough. What about types outside of my React components? Can I validate that my app is type-safe as part of my CI pipeline? Can I validate it as a commit hook?

Well, You can find some ways to validate types during your tests and use them outside of components but PropTypes was not designed with that in mind. It was intended to give you information in real time (at runtime).

As I was not convinced by PropTypes, I was left with two choices: TypeScript and Flow. Functionality-wise they looked the same, with some good points on Flow side:

No overhead meant better performance.
Backed by Facebook, as well as React.

Those were enough to make a difference and thats why I started using Flow with React.

Flow

Around 2 years. Thats the time Ive been using Flow and only Flow. It worked with React / React Native like a charm, boosted our team productivity, was a great tool as part of our CI pipeline, and generally helped us deliver.

And then, we hit a wall. Once our projects started getting bigger, flow server struggled to start, and was getting slower and slower. At some point we were simply unable to run it, our CI running time and its cost skyrocketed.

Unfortunately, at the same time, I started getting into other projects. Among them: an embedded project with Arduino using johnny-five. Thats when I realized the second weakness of Flow: its community support.

Thats how typing systems in JavaScript work: you write your module with one solution and write an independent type definition for the other. TypeScript had a lot of support, I think that even today, the number of libraries without support that I used is less than 10.

Flow was different: during the time I used it in the React ecosystem, there were a few libraries without support but I could still work without it. Outside of this ecosystem, support was a mess, I was unable to find even one library working with Flow. I found solutions to transform TypeScript definitions to Flow but in the end, it didnt work.

At some point, I wondered if I would really have to use TypeScript, even for React / RN projects. How inconsistent is that: using a technology backed by a company such as Facebook, and replacing its typing system with one from Microsoft? Moreover, support for TypeScript in React must be a mess, right?

TypeScript

I (re)discovered a whole new world.

After, getting back to TypeScript on side projects with Johnny-Five or React, I couldnt believe how I loved it.

Not only did my performance and library support problems disappear, but I got to love its syntax. Since 2016 and a lot of updates, I could not find anything to reproach TypeScript for.

React / React Native support was perfect, with the whole ecosystem such as eslint with prettier, jest, etc. Also, an extract from State of JavaScript (2022) gives us an idea of the difference between TypeScript and other flavors:

My next move was simple: As I previously wrote templates/boilerplates for my team using Flow, they were quickly replaced with the equivalent in TypeScript. Since then, weve only been using TypeScript.

Flow was great for a long time, its performance and support problems got the best of him while TypeScript later evolved to be amazing: my choice was simple.

Conclusion

If you were to ask me what to choose between untyped JavaScript, PropTypes, Flow, and TypeScript for your project: I would tell you the following:

Untyped JavaScript can be a good choice if you work on a project for less than a week, throw it after, and dont wish to learn any type solution.
PropTypes is a great tool if, for any reason, you cannot use Flow or TypeScript but is not enough for a big project.
Flow was great, since then I did not try it again, but would not bet on it.
TypeScript is a great solution, in my own opinion the best out there and required for any JavaScript project with high stakes.

Regarding using a typing library I experienced some objections, that I would like to answer here.

Loss of performance: todays solutions are great and the infinitely small loss of performance you could sustain would be balanced by the structure that typed code brings.
Lack of knowledge in TypeScript / Flow: if your project has high stakes and you dont want to use typed JavaScript because your developers dont know it, the project is bound to fail. Change your team or train them, thats the only way.
Loss of time: Ill quote Clean Code: A Handbook of Agile Software Craftsmanship from Robert C. Martin: Indeed, the ratio of time spent reading vs. writing is well over 10:1. If one thing, typed JavaScript will help you read code, thus increasing your productivity rather than decreasing it.

TL;DR

TypeScript.

Photo by Kevin Canlas on Unsplash