DEV Community: Antoine Coulon

🚨Avoid this mistake when running containerized applications in production

Antoine Coulon — Mon, 06 Nov 2023 16:30:00 +0000

Hello everyone!

Let's talk about things we must manage when running containerized applications and how this relates to proper management of termination signals.

If you prefer the GitHub with examples format, I created a repository that follows the same track with practical examples.

Before specifically talking about containers, let's put them aside and see how we run applications on a daily basis. We all use various operating systems that can run huge amount of tasks. These tasks are executed within processes, one of the fundamental units of an operating system.

Consequently each application is assigned to a process that will live until the application properly terminates.

With Node.js, we can check what is the process id currently running our application:

$ node -e "console.log(process.pid)"
> 39829

Overall, one of the core responsabilities of an operating system is to manage the life cycle of these processes, whether they should be created, killed, paused, restarted, etc.

Processes can also interact with other processes, for instance it's pretty common for a parent process to manage a set of child processes to run a multi-process application.

Termination signals

Termination signals are the primitive used by the OS to tell a specific process to terminate.

On Unix, you can send these termination signals through the kill command.

$ kill 39829 // SIGTERM by default
$ kill -9 39829 // SIGKILL

There are many existing termination signals but we'll talk about the three most widely used:

SIGTERM: this is the most common termination signal. This termination is called "soft" because the signal simply orders the process to stop but the process in question can decide to ignore it. This is the signal used by default on Unix OS, when using the kill command. This is the equivalent of the OFF button on your remote control.
SIGKILL: this is the most brutal termination signal because it does not allow the target process to react to or ignore the signal. This signal is used to terminate the process immediately and by definition does not allow a graceful shutdown of the application (explained a little further down). It is the equivalent of suddenly unplugging your power cable.
SIGINT: this is the interrupt signal which is for example sent when the user sends CTRL+C command in the terminal currently executing the process.

Graceful shutdown: one of the main responsabilities of a production-grade application

Now that we know what termination signals are, we could wonder what is the responsibility of the application regarding these signals?

Well, the application itself must properly handle these signals, otherwise there is no room for graceful shutdown.

A graceful shutdown for an application refers to a clean and controlled program termination where there is no data loss nor leak of resources and where the program has the ability to perform other types of critical operations before exiting, such as logging. This is performed by the "Eject" action of a computer drive that allows a clean disconnect. I'm assuming you already know the consequences when doing that 💣.

How to perform a graceful shutdown?

Most of the time, process supervisors will first emit signals that are "soft" such as SIGTERM or SIGINT which don't terminate right away processes.

During a specific period of time (depending on the supervisor), the application process will have time to clean up everything it has to do and then exit i.e. performing the graceful shutdown itself. In other words, graceful shutdown begins with reacting to a termination signal and then trigger application level code to manage the teardown.

// Node.js

// Graceful shutdown on SIGTERM
process.on("SIGTERM", () => {
 // close server, close database connection, write logs...
 releaseAllResources();
 // then manually exit
 process.exit(143);
});

As we can see to handle these signals, we can attach process handlers with our own custom logic. Note that it's important for the handler to exit the process manually after the clean up, otherwise the process will hang and still live.

// Node.js

process.on("SIGTERM", () => {
 // If we don't manually exit, the process will hang forever 
 // until a SIGKILL is fired. Once attaching these types of handler, 
// it's then your responsibility to release the last ressource: the process itself.
});

Adding SIGTERM process handlers allows us to take control over the logic that will be performed, but we're also responsible to properly exit after that.

We could also be tempted to not exit after and keep the process running, but this is really disadvised and the handler should only be used for the graceful shutdown itself.

But can't attaching these handlers be harmful if we don't exit after?

Of course it can be! This is why most of the supervisors usually don't only rely on SIGTERM and after firing SIGTERM they wait until a certain period of time and send SIGKILL if the process was still not terminated (this is essentially the difference between "Quit" and "Force Quit" when using Task Manager). Note that depending on the OS, some can just let the process hang indefinitely.

Consequently even applications on your own host machine should properly handle these signals, not only the ones in production, even though the consequences will most likely be less serious and cost less money.

Production environment

Let's come back on the main, topic, running containerized applications in production-grade environments.

For this article, I consider the fact that you know what a container is. If you don't, please check Docker's documentation before going deeper into this subject.

Life cycle of a container

For most production systems, applications will be deployed and run into containers, allowing applications to run in a sandboxed and lightweight environment. There are many different types of containers, but we won't go into these details in this series as most of the good practices and things to avoid can be applied to all types of containers.

Containers are usually managed by supervisors or orchestrators that determine whether a container should be started, restarted, stopped, scaled up/down etc. One of the most famous container orchestrator is Kubernetes (K8S) originally developed by Google.

The goal of K8S is to orchestrate containers at scale, managing clusters of containers including lot of different services and applications of a company. K8S in a nutshell manages containers deployments, containers life cycles by determining whether they need to be scaled up or down based on the load, but also deals with deployments and when to take arbitrary decision like stop/restart them when new versions of containers are available.

For Kubernetes to be working efficiently, one fundamental criteria for containers has to be respected: an application running in a container should properly react to signals coming from the orchestrator.

What is the danger of not reacting properly to these signals in the context of containers?

As it was described in the previous section for OS processes, the supervisor, which is Kubernetes in the context of containers, will first try to send SIGTERM and after the default value of 30 seconds (can be customized) is elapsed, a SIGKILL signal will be sent to be sure that the container is shut down.

Here a list of container services with their respective behavior with SIGTERM and SIGKILL

Service	Termination signal behavior
AWS Elastic Container Service	SIGTERM, wait 30s, then SIGKILL
Kubernetes	SIGTERM, wait 30s, then SIGKILL
Azure App Service	SIGTERM, wait 30s, then SIGKILL
Docker	SIGTERM, wait 10s, then SIGKILL

We could say that it's fine and we don't really care about handling termination signals as the SIGKILL will be sent anyway at some point and will terminate both our application process and the container initially hosting the app process.

Putting the graceful shutdown process aside that we already explained in the previous section, there is also another problem that can become critical in most production systems, which is latency.

Imagine that a container needs to be rollbacked because a bug was introduced, it means that during 30s we won't be able to do anything else but waiting. Also, all operations such as scale down or stop will keep using resources until the SIGKILL was fired. At scale on hundreds/thousands of containers, 30s can quickly become a lot of overhead and introduce performance critical penalties.

Remember to attach termination signal handlers to your applications and handle graceful shutdowns properly. This will help both the application and all the production systems build around it to be more reliable, resilient and effective.

Ensure that termination signals can be propagated

Usually, when your application properly manages terminal signal, it's already a good point. However, there is a prerequisite to that: the termination signal must be propagated correctly starting for the supervisor at the very top to the very bottom where your application process resides.

What could go wrong?

Let's take a very simple example with Docker containers, where a Dockerfile specifies an ENTRYPOINT such that it spawns a shell process as the PID1, known as the shell form of the ENTRYPOINT command.

Please don't use the following Dockerfile for shipping Node.js applications in production. This is a simplified example that will pull a deprecated Node.js version (v12) that only suits for educative purposes.

You can test all the examples that will follow on your own using the repository I created

Dockerfile

FROM ubuntu:22.04

RUN apt-get update && \
    apt-get install -y nodejs

ENTRYPOINT node index.js

Building your own Node.js image is discouraged, here we build it from Ubuntu for educative purposes. Please rely on official images for production.

This will result in two processes in the container, shell being the parent one and the Node.js application being a child process of the shell (subshell). The consequence is that shell does not propagate termination signals correctly, meaning that the Node.js process will never receive these signals.

# Command run within the container

$ ps aux
USER       PID  COMMAND
root         1  /bin/sh -c node index.js
root         7  node index.js

Now when we do docker stop <container-id> we can see that the Node.js process does not receive the SIGTERM signal and keeps living until Docker sends the SIGKILL signal after 10s.

One way to circumvent that is to avoid shell to be PID1 and turn the application process itself to PID1 using the exec form of the ENTRYPOINT.

Dockerfile

FROM ubuntu:22.04

RUN apt-get update && \
    apt-get install -y nodejs

ENTRYPOINT ["node", "index.js"]

If we run again ps aux within the container we can see that the Node.js process is now PID1:

$ ps aux
USER       PID  COMMAND
root         1  node index.js

Because our application follows the good practices we talked about in the first part, it correctly handles signals and is able to perform graceful shutdown.

Don't let your application process be PID1

The good part of having our application process being the only process in the container and given that our application has setup the expected process handlers is that we are able to properly manage termination signals.

Nonetheless, we have a new problem, which is that our application now is the PID1 also known as init process.

What is PID1 alias "init" process?

PID1 or "init" process has very precise responsibilities regarding the operating system (or more precisely in the container in that context).

Before explaining the issue around PID1, let's quickly return to the basis of an OS with the organization of processes.

The process register is represented as a graph, where PID1 represents the root node, commonly called "init".

Unlike other processes, the "init" process (PID1) is assigned very specific responsibilities by the Kernel, which are in particular:

initialize all services (processes) required by the operating system.

manage and reap so-called “zombie” or “orphan” processes. A "zombie" process is a process that has finished executing but for which resources continue to be monopolized.

ensure that child process' exit code is forwarded properly outside of the container.

All of these responsibilities cannot and should not be shouldered by your application nor the runtime your application is being executed on (JVM, Node.js, etc).

Who should be PID1 then?

There are a lot of solutions out there but Tini is the most famous and battle-tested one.

Tini has one goal, which is to provide an “init” process that works as expected. It is an independent executable, but it is important to mention that it is embedded by default in Docker since v1.13, usable with docker run --init or with docker-compose (v2.2) using init: true from the config file for a service.

Still in the context of a Node.js application, here is an example of a minimalist but closer to production-ready Dockerfile using Tini:

FROM node:18-alpine

RUN apk add --no-cache tini

# Copy app files

ENTRYPOINT ["/sbin/tini", "--", "node", "index.js"]

Hopefully you are now aware of some of the traps that you should avoid when running containerized apps in production and more generally understand the way processes communicate and how supervisors manage their life cycle.

🖼️ Unleash graph visualization power to take full control over your project with skott

Antoine Coulon — Mon, 02 Oct 2023 09:30:38 +0000

Hello everyone, exciting announcement about skott, the web application automatically embedded has now reached v2, after several months of development and after many issues were opened by the community asking for improvements!

If you never heard of skott, here is the first blog post introducing the tool. The API is not really up-to-date and much more feature were added since then but the introduction still remains valid!

skott is all-in-one devtool to automatically analyze, search and visualize dependencies from your JavaScript and TypeScript projects.

Installation and quick run

npm install skott -g
skott --displayMode=webapp

Here is a pretty quick overview of what you can achieve with the new version of the web application:

Graph Visualization is hard, and being able to extract valuable information from graphs with hundreds or thousands of nodes and edges is not an easy thing.

In that web application v2, the focus was on improving the experience dealing with the information that was generated by skott.

Graph configuration

Probably the most asked section! One of the biggest drawbacks of the v1 of the web application was its inability to render graphs accordingly to their sizes, shapes, and user preferences. Graph Visualization is complex because depending on the size and on the shape of the links between all the nodes, some layouts and parameters might fit better. Because it's hard to automatically determine the best configuration out there for anyone's project, I made the choice to offer some customization over graph rendering, let's see that:

Cluster layout

Cluster layout builds graph based on connections between nodes. Especially useful when having a graph with a lot of interconnected nodes, including circular dependencies.

Three node spacing algorithms are currently available: "Barnes Hut", "Repulsion", "Force Atlas 2". These are all provided by vis-js, the library skott uses to draw graphs.

Hierarchical layout

Hierarchical arranges nodes in a hierarchical fashion. It helps visualizing graph structures that naturally use some sort of hierarchical architecture that is no circular dependencies between parent and children nodes.

Note that depending on graph size, some layouts, algorithms or options might perform and fit better. For instance "Improve Spacing" option improves node spacing and layout rendering but takes up to 100x more time to render the graph.

In most cases, what you want is to reduce the dataset upfront to avoid rendering huge graphs that will both be a costly operation but also will be harder to visualize correctly.

File Explorer

File Explorer section provides a VSCode like explorer to browse files and directories. The whole purpose is to have an easy way to navigate through the graph by recreating the file tree as it would appear in an IDE. It also provides ways of filtering folders (either via the glob pattern or command actions available on each folder) and focusing on files.

Dependencies

Section that allows to enhance the rendered graph by highlighting circular, third-party, or builtin dependencies.

As of today this section just ported the existing features that were initially introduced in web application v1.

This section is pretty minimalist for now but it has a lot of potential, such as displaying unused dependencies (unused npm dependencies are already collected by skott itself).

Summary

Summary sections aims to provide global stats about the project itself, number of files traversed, number of circular dependencies, exhaustive lists of third-party dependencies (npm) and builtin (Node.js) dependencies used throughout the project, etc.

More to come!

A lot of information collected by skott is not yet displayed in the web application, but it's only a matter of time.

Cycles explorer

I think this will be a super useful section to fight against circular dependencies. Currently, skott only highlights nodes involved in circular dependencies. However, highlighting is not enough and can be misleading when you want to have a clear and granular vision about each independent cycle.

This section shape is not settled yet so don't hesitate to suggest features if you have ideas that could be useful for you in mind!

Interactive playground

skott provides a Graph API that can help recreating sub-graphs where only direct or deep nodes and parent/children are involved.

This would provide capabilities to focus on specific modules and recreate the sub graph that these modules are involved in, reducing the overall noise that can be introduced by drawing the whole graph.

Thanks for reading! If you ever come to use skott, share your feedback, open issues/discussions, I'm trying to make the project very interactive and reactive to contributions. Also if you like it, don't forget to drop ⭐️ :)

💉 Test-Driven Development and Dependency Injection are the way

Antoine Coulon — Wed, 26 Jul 2023 10:04:40 +0000

Welcome back into these series everyone, past few days have been incredible as skott just reached 130 stars on GitHub, 100k of total downloads and lately around +12k weekly downloads since I started open-sourcing it. It's very far for being mainstream but it's a good start, isn't it? Anyway, let me put my personal satisfaction aside, and let's talk about what you came for!

For the newcomers of the series, let me do a quick skott introduction to put things in context.

skott is a all-in-one devtool to automatically analyze, search and visualize dependencies from JavaScript, TypeScript (JSX/TSX) and Node.js (ES6, CommonJS). Basically what skott does is deeply traversing a file directory, parsing all supported files (only JS(X)/TS(X) for now), resolving all module imports and build the whole graph from these and then expose a graph API from it.

Having that graph generated, you can then use it in many ways, such as visualizing it using a web application. Here is an overview of the next version of the skott web application I'm currently revamping:

If you're familiar with static analysis tools (if not feel free to check my previous blog post on that subject), you might already be aware of some of the challenges we can face. In the context of skott, here is a list of some of the challenges:

extracting workspace information such as manifest files (package.json) or lockfiles (package-lock.json, yarn.lock, pnpm-lock.yml), configuration files (tsconfig.json, .eslintrc.json)
traversing the file system, using ignore patterns, discarding ".gitignore-d" files
parsing files using multiple parsers for each supported language (JS/TS) and managing to cover all specificities that can be enabled or not within that language (e.g: JSX/TSX for JavaScript/TypeScript)
walking the previously emitted AST to collect all the module declarations from each file AST, for both CommonJS and ESM
resolving all the module declarations to their real file system location, in order for us to incrementally resolve the entire graph
building the graph by using all the resolved modules

That's quite a lot to do even though it's a simplified version.

The question now is, how do you manage that complexity in a way that you're fast, confident and safe enough to add/remove/update features, fix bugs and process huge refactorings? Basically, how do you enable true software Agility?

Testing

The first thing that could come in mind are tests. Overall writing good tests brings safety and helps to gain confidence. Nevertheless, there is a double-edged side with tests, because depending on how you manage to write them and essentially what is being tested i.e: the system under test (SUT) and how it is tested, it could lead to wrong confidence leading to unexpected bugs, sneakily introduce wrong design solutions such as coupling code to implementation details leading to painful and unsafe refactorings. As software engineers, this is something we should avoid at all cost.

In my previous blog post Don't target 100% coverage, I quickly demonstrate how having tests written in a certain way can produce unexpected behaviors and false sense of security.

Before continuing the post, I highly recommend to read ☝️ this blog post if you're not familiar with the different ways of bringing tests to the table I'm referring to, namely Test-First, Test-Last, Test-Driven Development, Mutation Testing, Code Coverage.

Test-First, Test-Last and Test-Driven Development

Let's see the differences between these three.

Test-First

The process of writing the test before writing any line of code of the targeted feature. This aims to put a precise plan on what should be implemented but is indeed missing the most important part, the how you manage to achieve the implementation, as Test-First is not providing an incremental feedback loop from which you can benefit a lot.

Doing Test-First can be useful and can generally help increasing the covering of specifications with tests but kind of omits the fact that new constraints, hints and ways of doing things will be discovered along the way. Unfortunately, Test-First does not allow tests to drive the implementation, and favors the writing of big chunks of code at once, because you start from 0 lines of code to X lines of code where X is the number of lines of code that were needed for the test to pass. Amongst these chunks, some might be useless or take too much time to be introduced given the overall complexity that you have to deal with at once.

Test-Last

Probably the most popular way of testing, consists of writing all the tests after all the lines of the feature code have been written. This aims to bring safety around the already produced code but does it really help with that? Not really. One of the drawbacks of Test-Last is that it's often used as a way of converting a desire to gain confidence -by adding tests and increase the code coverage percentage- into a false sneaky feeling of safety.

Remember the Goodhart's law: When a measure becomes a target, it ceases to be a good measure.

Consequently, a lot of unnecessary and noisy tests will be added along the way as they will try to cover parts of the code that don't essentially need to, either because they could have been covered by higher-level tests oriented by system state expectations or just that they test useless or repeated things already covered by other tests. How could you know if some test is useless or covers an behavior already covered by another test? You add a test, it passes, but is it due to the code you just added or was it the case before? 🧐

Because Test-Last introduces tests in the very last step of the feature life cycle, it comes with a lot of pain points, such as revealing design smells or that implementation details were introduced or that the code is not easily testable. This is a waste of time, because you could have figured it out before. This will also most likely favor the introduction of mocks (⚠️ I'm not referring to the test doubles in general, but especially a Mock which is one type of test double) that should be considered as a smell in most cases as it introduces a structural coupling between tests and the code (asserting that X will called Y with abc parameters, this highly reduces flexibility and ability to refactor).

In that case, tests become very fragile as they depend on implementation details and at the next function change they break, making you feel the anger towards the tests themselves.

"Tests prevent me from refactoring as they break all the time when changing the code, I'm losing flexibility and it's not a convenient way of working" 😡

Does it exist a better way of achieving efficiency and safety while keeping the code highly flexible, easily refactorable and abstracted from implementation details that we don't want to depend on feature-wise?

Test-Driven Development (TDD)

Disclaimer: My desire is not to advocate TDD at all cost, Test-Driven Development is not a silver bullet. The whole purpose is just to offer an overview about an highly misunderstood and under estimated discipline. It's then your choice to try it, blame it, blame me. But believe me, when you start being decent with TDD, you never look back.

Test-Driven Development is a more evolved version of Test-First, it's essentially a software development discipline that drives you to find the quickest and best path of writing each line of code targeting a domain specification through fine-grained decomposed steps (so called baby steps), dealing with complexity incrementally.

It appears that Test-Driven Development relies on automated tests which its best companion to-date.

By using tests, TDD will drive the writing of code required to make a failing test ❌ pass to turn to GREEN ✅ in very short feedback loop cycles.

If there is no prerequisite, that is no failing test, why would I add some lines of code? Nothing in my system justifies that need for adding lines of code yet. Maybe that the behavior is already implemented? The first step is to be sure that we have a failing test which is our first checkpoint to reach.

Once we have that, we can start writing the code required to make the test pass. We are now sure that we did something useful, that is adding lines of code justified by the specification that turned from red (failing) to green (succeeding).

Just after that, we have room for the third step of the TDD process which is the refactoring part during which you're free to refactor and produce the best code while keeping the test to green. That's why Test-Driven Development fundamentally helps designing software as it allows to infinitely and safely refactor (at any point in time) the code produced while ensuring that the expected behavior is still intact, which is one of the main benefits.

Test-Driven Development requires good Software Engineering skills to be effective

TDD helps designing software because it creates a perpetual confrontation between the design choices to be made at a time depending on the current system requirements. Nevertheless, TDD requires a good set of design and refactoring skills upfront in order for the discipline to be effective and increase productivity, otherwise you might end up being blocked at some point in the process or taking the wrong decisions (or even worse, no decisions at all).

Test-Driven Development is all about feedback loop, so is the Software Engineering in general!

Let's put the TDD aside for a little bit and let's talk about some of the most common feedback loops we're all working with on a daily basis:

Integrated Development Environment (IDE): all about feedback loop! How fast you want to know that the code is correctly written in the host language? The red underlines, warnings, auto-format, popups and whatever alerts can go through the whole IDE are part of the feedback loop, they help you know whether you are heading to the good path or not.
Compilers: all about feedback loop! Same here, the compiler provides you a way to know if the written code is semantically correct to some extent. Statically typed languages such as TypeScript are a great way of improving the feedback loops by adding type-safety + great developer experience with real-time integrated Language Server Protocol (LSP).
CI/CD Pipelines: all about the feedback loop, where you continuously want to know whether the product could be successfully deployed in production, passing all the verification steps. If that's not the case, you want to know it quickly to fix the problem right away.

All of these examples of feedback loops tend to leverage the fail fast principle whose goal is to quickly identify failures, rather than letting them persist or be discovered later in a development process or worse letting the customers discover the bugs. Feedback loop is the foundation of continuous improvement.

Test-Driven Development is all about having the shortest feedback loop towards the written code, asserting whether the system is producing the expected outcome, when adding, removing or updating code. It appears that thanks to automated tests, the loop is quick enough in most cases as long as you respect the Fast nature from the F.I.R.S.T principles!

One downside of TDD is that it can become counter-productive if not correctly applied. IMHO it is better not to practice TDD than to practice it the wrong way. Also, the learning curve is steep as it requires strong technical skills and mental shifts and a different way of approaching software development.

Putting that into small practical examples

Ok so now that we talked about Test-First, Test-Last and Test-Driven Development, let's come back with our initial question:

How do you manage that complexity in a way that you're fast, confident and safe enough to add/remove/update features, fix bugs, continuously improve the code using refactoring?

For now, the only way I have been putting in practice that works for all these points, is Test-Driven Development. Of course you might be able to do the same without it, but at what cost, with what level of confidence and how fast? Without introducing any regression? I have been there too, now I could never do it again without it.

Let's take an highly simplified version of a skott feature.

Given two JavaScript modules with one being a dependency of the other one, both should be analyzed and a graph should contain two vertices with one edge representing that relationship:

index.js

import { add } from "./feature.js";

// Rest of the code consuming the function, we don't care about that

feature.js

export function add(a, b) {
  return a+b;
}

We want to:

read both files,
find the module declarations and see that index.js imports feature.js module
then build a graph of two vertices, index.js and feature.js, with a directed edge from index.js to feature.js representing the import dependency, we say that index.js is "adjacent to" feature.js.

Consequently the expected outcome of our system is a graph shaped that way:

{
  "index.js": {
    "adjacentTo": ["feature.js"]
  },
  "feature.js": {
    "adjacentTo": []
  }
}

This is our expected final shape, this is the expected outcome that we want skott to produce.

Test-First

Practicing Test-First here would suggest us to start by writing the test first, with the whole expectation that we have from our system.

describe("Graph construction", () => {
  describe("When having two JavaScript modules", () => {
    describe("When the first module imports the second", () => {
      test("Should produce a graph with two nodes and one edge from the index module to the imported module", () => {
        createFile("feature.js").withContent(
          `
            export function add(a, b) {
                return a + b;
            }
          `
        );
        createFile("index.js").withContent(
          `
            import { add } from './feature.js';
          `
        );

        const graph = new GraphResolver();

        expect(graph.resolve()).toEqual({
          "index.js": {
            adjacentTo: ["feature.js"],
          },
          "feature.js": {
            adjacentTo: [],
          },
        });
      });
    });
  });
});

The test itself includes the three main components, Arrange/Act/Assert. Running that test will make it fail for sure, as we don't have any code written yet. But now, how do you go from that test failing from passing?

Little reminder of all the steps that we must go through to make the test pass:

file traversal
file parsing
module extraction
module resolution
graph construction

This represents such a long way and so many steps and components involved, hence we might spend a little a bit of time doing that. Unfortunately the test won't be useful during that whole time, it will only be there to assert the final result once we already produced all the required code.

Test-Last

Nothing happens there, Test-Last does not want us to write any test yet! Even though sometimes I hear it whispering something in my head:

Test-Driven Development

Ah finally, something that will help us achieving the desired behavior. As already said, Test-Driven Development wants us to take an incremental approach instead and let the code flow and grow in complexity in various steps. By design, it wants us to adopt a baby step approach where we just add the minimum of code to make the test pass.

In the context of TDD, here is the first test that crosses my mind:

describe("Graph construction", () => {
  describe("When not having any modules", () => {
    test("Should produce an empty graph", () => {
      const graph = new GraphResolver();

      expect(graph.resolve()).toEqual({});
    });
  });
});

First things first, we are focused on the shape of the contract we want to expose, it constraints the thinking around that, one problem at a time.

One benefit of TDD is that the test become the first client of the code itself and you can let the design emerge progressively.

Here is the quickest way of making the test pass:

class GraphResolver {
  resolve() {
    return {};
  }
}

Really easily, we just have to create a raw class with a method returning an hardcoded empty object.

Then comes the wonder of what should be the next test? Don't forget that the end goal is to be able to traverse files, and build module dependencies between each of them.

TDD efficiency is enabled by the choice of the tests, there is no magic. The developer is responsible for thinking and finding the right path for the test order and each missed step or step that is too big will be a missed opportunity to entirely benefit from TDD. But don't worry if you end up in that situation and think of an intermediate step that can be valuable, you can still downshift gear.

So what test comes next? After validating the case where we don't have any module, what about introducing the case where we finally have one module to analyze?

describe("When having one JavaScript module", () => {
    test("Should produce a graph with the module", () => {
      /**
       * ARRANGE
       * 
       * How can we create a file system context including that file specifically
       * for this test?
       */
      const graph = new GraphResolver();

      expect(graph.resolve()).toEqual({
        "index.js": {},
      });
    });
  });

Even if this appears to be simple, it grows in complexity as we introduce the notion of file in the Arrange part of the test. Our tool will be based on the file system, meaning that it will need to read from it at some point. Does it mean that the test itself should read from the file system? Not at all.

But why not using the real filesystem right away, using the Node.js API for instance?

Introducing the use of the real file-system is something we want to avoid with unit tests, we want them to be fast, isolated and repeatable across executions. This is not the case as the filesystem layer would not check any of these criteria. We want to have a fully manageable and specialized version of the Node.js filesystem module. Do we need to fake everything within it? Not at all, we just need to cover the subset of it we need for now.

As you can see, that small test brings a whole new level of thinking around the new feature. You could be thinking that it's something we could avoid if we were not writing any test and that it complicates our task for not much. But indeed, this forces us to think about building a testable code on which we have full control over and that already brings us to designing solutions.

What we want first is to create a minimalistic and in-memory file system that can fake a real one.

Let's go back at our second test:

test("Should produce a graph with the module", () => {
  const fileSystem = new FakeFileSystem();
  fileSystem.createFile("index.js").empty();
});

Now that we have that in-memory context created, we want it to be able to be used within that Graph Resolver context, in other words we want it to be injected in the Graph Resolver context.

This naturally leads us to Dependency Injection (DI).

Dependency Injection

Dependency injection is a pattern for decoupling the usage of dependencies from their actual creation process. In other words, it is a process of injecting dependencies of a service from the outside world. The service itself doesn't know how to create its dependencies.

The dependency there (the FileSystem module) is created outside of the GraphResolver context and can be injected right away:

class FakeFileSystem {
 // ...
}

const fileSystem = new FakeFileSystem();
fileSystem.createFile("index.js").empty();

class GraphResolver {
  constructor(private readonly fileSystem: FakeFileSystem) {}
}

// Dependency Injection
const graphResolver = new GraphResolver(fileSystem);

Now that we have the dependency injection in place, we can fake the required behavior from the file system module

class FakeFileSystem {
  fs = {};

  createFile(name) {
    return {
      empty: () => {
        this.fs[name] = "";
      },
    };
  }

  readFiles() {
    return Object.keys(this.fs);
  }
}

This is a simple fake filesystem that emulates the operations we need for now, that is createFile and readFiles. Nothing much! We don't need to cover everything, only what's needed in the context of the test.

To make the test pass, we can now consume that newly available implementation in the GraphResolver service.

class GraphResolver {
  graph = {};

  constructor(private readonly fileSystem: FakeFileSystem) {}

  resolve() {
    const [filename this.fileSystem.readFiles()) {
      this.graph[filename] = {};
    }

    return this.graph;
  }
}

And now that passes, just matching our expectations. Note how much we're only focused on the current desired behavior, we just produced the only minimum of code we needed. For instance the readFiles method only yields the filename as the test only requires us to have a record including that name, even though we know for sure that later one we'll also need the file content.

Also, we're fully in-memory and isolated from the real filesystem, the dependency only produce the outcome we need, we don't need to fake the whole filesystem API of Node.js or whatever runtime we are using!

Note that in this case I'm voluntary skipping some of the steps in between as I know for sure that we want to read all files from the directory at some point. Instead of just introducing readFile method (as we only have one file), I go straight into what would have been the next step (readFiles). As a rule of thumb, it's generally a good practice to follow a sequence of Transformations that you should go through to produce the minimal code in order for the test to pass in short loops. If you're interesting in knowing more about that, you can read the Transformation Priority Premise from Robert C. Martin

After the green step, TDD comes in with a third phase which is the refactoring.

Refactoring is about changing the system design without altering that is changing/adding/removing parts of the system behavior.

Luckily, we can use that phase to improve a tiny thing that you might have noticed in the Dependency Injection system we currently have.

Dependency Injection can be used in a way that decouples from concrete implementations of the dependencies. In the case above, GraphResolver is directly coupled to a FakeFileSystem instance letting no room for flexibility and offers no way of injecting anything else that would match the same contract. More importantly, it leaks implementation details and things that should not be part of the contract. In our case, FakeFileSystem exposes a createFile method that exists just for the purpose of the test but has no value in the context of the GraphResolver, so GraphResolver should not know about that method but only what it cares about, that is the readFiles method.

Consequently, we can improve that by introducing a generic interface in a way that GraphResolver now depends on abstractions and not on concrete implementation. This brings us to the Dependency Inversion Principle (the D from SOLID principles). That way, skott could have an runtime-agnostic way of traversing file systems.

interface FileSystem {
   readFiles(): string[];
}

class FakeFileSystem implements FileSystem {
  // unchanged
}

class GraphResolver {
  constructor(private readonly fileSystem: FileSystem) {}
                                           // ^ this changes
}

All tests are still passing because we just played a bit more with interfaces and with the static compiler to narrow the correct types into the GraphResolver.

No more refactoring? Let's get closer to the objective.

Now we want to start introducing the concept of dependencies between the JavaScript modules. As already said, dependencies between JS modules can be modeled using a Directed Graph, in which files would be represented as vertices and relationships between files would be represented as directed edges between vertices.

Using the Graph terminology, a vertex A that depends on another vertex B is said to be "adjacent to B".

Once again, we introduce one concept at a time. This time, we don't really need to add a new test, we can just update the last one:

expect(graph.resolve()).toEqual({
-        "index.js": {},
+        "index.js": {
+          adjacentTo: [],
+        },
      });

This fails, now we make it pass:

resolve() {
    for (const filename of this.fileSystem.readFiles()) {
-      this.graph[filename] = {};
+      this.graph[filename] = {
+        adjacentTo: [],
+      };
    }

    return this.graph;
  }

// What is the next step? Adding two modules would not allow us to go forward. We need to find a test that will drive us to the cases where we introduce the concept of import that will create an edge between two modules.

Let's add a test that will force us to start dealing with dependencies between modules:

describe("When having two modules with one dependency", () => {
    test("Should produce a graph with both modules and a  dependency between the two", () => {
      const fileSystem = new FileSystem();
      const fileWithModuleImport = `import "./feature.js";`

      fileSystem.createFile("index.js").content(fileWithModuleImport);
      fileSystem.createFile("feature.js").empty();

      const graph = new GraphResolver(fileSystem);

      expect(graph.resolve()).toEqual({
        "index.js": {
          adjacentTo: ["feature.js"],
        },
        "feature.js": {
          adjacentTo: [],
        },
      });
    });
  });

Still with the idea of writing the easiest code possible to make the test pass, we can add a little if statement in the resolve method of the GraphResolver. From the Transformation Priority Premise, this is ordered as the middle of the transformation we should process (unconditional->if) splitting the execution path, splitting the execution path to match expectations:

  resolve() {
    for (const [fileName, fileContent] of this.fileSystem.readFiles()) {
+      if (fileContent.includes("import")) {
+        const moduleName = fileContent.split("./")[1].split("'")[0];
+
+        this.graph[fileName] = {
+         adjacentTo: [moduleName],
+        };
+
+       continue;
+     }

      this.graph[fileName] = {
        adjacentTo: [],
      };
    }

    return this.graph;
  }

Test is passing ✅

Note how ugly that "split" is, actually it might be the ugliest statement I have even written but we don't care, it fits our needs: it turns the test to green.

Remember after this step, you're free the refactor as much as you want, so no stress, we'll be able to do better just after.

Let's try to slightly improve the code during the refactoring step:

resolve() {
    for (const [fileName, fileContent] of this.fileSystem.readFiles()) {
      if (fileContent.includes("import")) {
+        const moduleImportParser = /import '(.*)';/g;
+        const [moduleImport] = +moduleImportParser.exec(fileContent);
         // path comes from Node.js "path" module
+        const moduleName = path.basename(moduleImport);

        this.graph[fileName] = {
          adjacentTo: [moduleName],
        };

        continue;
      }

      this.graph[fileName] = {
        adjacentTo: [],
      };
    }

    return this.graph;
  }

The way parsing is done in the two previous steps is an implementation detail there and is part of our own business logic, it should remain hidden allowing heavy refactoring without breaking the API surface. As long as the resolve method returns the expected graph we are fine because it's what matters the most.

While progressively adding cases where you want to reliably parse all types of module imports (and there is a bunch, including fun edge cases), you will realize that using a RegExp does not scale well and becomes to complex to manage. But it's fine, nothing prevents you there from introducing an ECMAScript-compliant parser, such as meriyah, acorn, swc, that does the job for you! The best part is that this would still remain hidden inside the internals of your use case, allowing infinite refactoring and improvements.

I'm not going to into the implementation of the parser itself (you can check skott source code to see a complete example), but hopefully you start being able to imagine what are the next steps of this use case. I also wish that you were able to see the advantages that Test-Driven Development brings in that little context.

Wrap up

By dealing with few use cases, we were able to see how Test-Driven Development incrementally drives us to the development of a feature.

Not only this helps us facing constraints very early in the process thanks to the feedback loop, it also naturally forces us to introduce a way to make the code easily testable, fast, isolated, repeatable, via Dependency Injection. This also reduces the level of complexity we have to deal with at each new step and test added, while going way faster and safer.

Moreover, we can also abuse from heavy refactoring phases to make the code cleaner (even though clean code and great design are prerequisites to process efficient refactoring) while being sure that the system behavior still matches our expectations.

Personal take:

One thing I know for sure is that I would not feel confident with skott features if I hadn't brought 130+ unit tests with Test-Driven Development. Features can be added without any fear, refactoring can be done with ease and confidence.

Nevertheless this does not mean that skott can not produce bugs, it simply means that for the use cases where skott is expected to work, it will most likely work as expected. In other words, a bug will most of the time be due to a missing system behavior description or edge case. And it's totally fine, because to fix that, you only have to add that test case reproducing the missing behavior, and let the TDD flow!

Link to skott's GitHub repository: https://github.com/antoine-coulon/skott

See you at the next episode 👋🏻

Don't target 100% coverage

Antoine Coulon — Thu, 19 Jan 2023 08:39:04 +0000

Don't target 100% coverage... but achieve it anyway!

I recently noticed that a lot of people were advising to reach 100% of code coverage and were presenting it as a primary indicator of code quality. While I agree on the fact that it's an interesting metric, everyone must be aware that having 100% of coverage does not indicate the quality of tests and the way to achieve a good coverage is more important that being able to produce 100% of coverage as an end, it must not be a target in the first place. Let's see why.

Code Coverage

First things first let's briefly explain what is code coverage for those who are not really familiar with the concept.

Code coverage is a technique aiming to determine what parts of your program is being covered by a test. Basically, the code coverage tool instantiates counters and they are incremented once a line of your program is traversed by one of your tests.

Here is a quote from istanbul, one of the most used code coverage tool:

Istanbul instruments your ES5 and ES2015+ JavaScript code with line counters, so that you can track how well your unit-tests exercise your codebase.

Code coverage tools often collect various metrics such as:

% of statements traversed
% of branches traversed
% of functions traversed
% of lines traversed

Overall, code coverage tools just collect the amount of production code traversed by a test, whether the assertion is relevant or not, which is where the devil resides.

The trap 👹

Let's see a trap example with a simple project using c8 as our code coverage tool (istanbul unfortunately does not natively work with ESM).

You can grab the source code here.

index.js

export function add(a, b) {
  return a + b;
}

Now here is one simple test that we would expect to be truthy:

index.spec.js

import assert from "assert";
import { add } from "./index.js";

it("should return 1", function () {
  assert.equal(add(1, 0), 1);
});

When running the test, we can see that it turns to green and the coverage is automatically generated for us:

As we can see, everything is 100% covered by tests, all the statements, all the functions, all the branches, etc. So everything is fine, so we could just ship the feature right? 🧐

But what if someday someone just changes a little detail, pretty confident that it will work as the unit test still succeeds after the change.

Thanks to Michaël Azerhad who provided me that example while back!

export function add(a, b) {
+  return a - b;
-  return a + b;
}

The test suite still passes and coverage is still showing us 100% of coverage.

As simple as the example may seem, it demonstrates that coverage only checks that a given chunk of code is at least traversed once, but it will never tell you if a test is valuable nor useful in some kind of way. Consequently there is a danger which is that it can make you feel (wrongly) safe about your code, just based on the fact that they are traversed by a test at some point in time.

Don't let "100% coverage" be your primary concern 🖐

Don't get me wrong, I'm not saying that coverage is not useful in itself.

However aiming towards reaching 100% after having written the code will most likely produce false positive tests in a "Test-Last" fashion, just to make statistics look better. I would even say that Code Coverage Driven Development (CCDD) sometimes inherits from the disadvantages of the "Test-Last" approach, that is most of the time either writing irrelevant or even worse useless tests. They both make you feel safe and confident as the console only shows green lights, but it won't avoid you to get in trouble.

Mutation testing, an under-estimated tool 🔍

Because I'm lazy, I'll once again simply quote the definition from the Stryker documentation

Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

Basically, it does something fundamentally different from code coverage which is adding variants to branches of your program. By mutating your code, the mutation testing tool can detect whether your test suite was correctly covering or not the behavior produced by the mutation.

Let's try it on our small example using Stryker.

$ npm run mutation-test

Running the test produces the following output:

As you can see, one mutant survived which is the one changing the Arithmetic Operator which is exactly what we were looking for. Consequently, we can see that it does a step forward by allowing tests to be tested via the mutation of the production code itself. The more your tests kill mutants, the more they tend to be relevant. You could have 100% of coverage by having only 20 to 30% of the mutants killed, which is a big smell. Be careful with code coverage!

You might have guessed it, but mutation testing tools can also provide code coverage reports.

Automatically reaching 100% coverage and 100% of mutants killed using Test-Driven Development 🤹‍♀️

With a simple example we just showed that coverage as such is not enough to ensure tests quality. We also saw that mutation testing helps finding incomplete/missing test suites. What if I told you that a good coverage and a high rate of mutants killed can be a positive side effect of a discipline that does not care about any coverage in the first place?

Test-Driven Development

This article is not dedicated to Test-Driven Development (TDD) is, but I'll try to make a short introduction.

Test-Driven Development is a software development discipline that aims to drive the writing of code required to make a failing test ❌ (prerequisite) pass i.e. turn to GREEN ✅ in very short feedback loop cycles. Test-Driven Development fundamentally helps designing software as it allows to infinitely and safely refactor (at any point in time) the code produced, which is one of the main benefits.

When doing TDD correctly, because each line of code produced must be justified by a failing test in the first place, 100% coverage will be naturally achieved. All the mutants will also be killed if TDD was done correctly, as changing a line of code or one statement will without a doubt break one test, otherwise it means that some code was shamefully produced without having a failing test first!

Not only you'll achieve 100% of code coverage and 100% of killed mutants, but you'll be able to produce a code with a higher level of quality, depending on your refactor skills (because TDD can drive you writing a well designed code but the skillset required to do so has nothing to do with TDD).

Of course Test-Driven Development is not a silver bullet, but it is to date the best way of mixing both code quality with code coverage and the best part is that you'll most likely do it unconsciously.

Wrap up

Don't get me wrong, code coverage is useful in many ways, as it can show how well a codebase is covered by tests at a very high level, for instance having a very low percentage of coverage means that there is not enough tests, so it's good metric to start with.

Nevertheless, having the opposite which is a high percentage of code coverage does not mean that tests are relevant. The best way of achieving both a high percentage of coverage and tests reliability is by not targeting it in the first place, otherwise it might lead people to write more or less (often less) relevant tests just make the percentage a little bit better, the worst part being that it makes everyone feel safer.

By also using mutation testing tools and by mastering disciplines such as Test-Driven Development (or at least Test-First), you can end up achieving 100% coverage and 100% mutants killed by not caring at all (0%)% of all these metrics. That, is the target.

Project sample can be found there: https://github.com/antoine-coulon/dont-target-100-coverage

Exploring the latest features of Skott: road to V1

Antoine Coulon — Mon, 16 Jan 2023 09:30:00 +0000

Hello everyone, I hope you're doing well.

Following the series of My journey of building Skott, an open-source Node.js library, I wanted to share with you a quick update about the latest features added. Implementing these features will be the opportunity for me to share with you what it took to build them.

For those who are not familiar with the tool, I suggest you to quickly check my first article introducing Skott 😎

Summary

Web application (v0.11.0) 👩‍💻
Incremental comparison (v0.12.0) 🕰
Detecting unused third-party dependencies (v0.13.0) 🔍

Web application (introduced in v0.11.0)

For its first versions, Skott's visualization was only rendered in our beloved command line interface which is cool but not really ideal to represent real-world graph structures.

Since v0.11.0, Skott now embeds the new "webapp" display mode, which is the new default display mode.

$ skott --displayMode=webapp

If you want third-party dependencies (npm) and built-in dependencies (Node.js) to be rendered in the app, don't forget to provide the flags to the CLI, otherwise they won't be collected nor displayed:

$ skott --displayMode=webapp --trackThirdPartyDependencies --trackBuiltinDependencies

Once the graph generated, the web application is automatically opened in your default browser on a free port:

The application renders a 2D network in which files are represented by the nodes of the network and the links between these files are represented by directed edges.

Within the left sidebar, few things are displayed:

Some statistics are exposed (number of files, circular dependencies, etc)
Some visualization options can be toggled on/off to highlight or make appear additional nodes/edges representing third-party or built-in dependencies.

To make the search of specific files easier when processing large graphs, a global search was recently introduced.

By using the CMD+K/CTRL+K command, files can be searched and then focused on:

Thanks to @bam-charlesbo for suggesting new features about the webapp.

Incremental graph processing (introduced in v0.12.0)

Computing graph is expensive as it includes a static analysis on each file of the project mainly involving parsing and AST walking. Depending on the language, some parsers are faster than others, for instance JavaScript parsers are naturally faster than TypeScript ones as TypeScript embeds a lot more of information encoded in the language at the type-level.

Introducing an incremental comparison

Even if Skott can analyze thousands of files in a matter of few seconds, performance always matter for a better DX. So here is a first step in order to make it nicer!

In the past, I reproduced a very minimalist implementation of the Affected/Incremental pattern that most of the monorepo tools embed natively allowing to gain a lot of performance on heavy project graphs.

If you're interested in knowing more about how it works under the hood, you can directly refer to this article.

Because most of the time project graphs do not entirely change (think of few files changes per commit), using incremental comparison would allow us to heavily benefit from caching.

From v0.12.0, a first version of an incremental comparison can done by providing the --incremental argument from the CLI. A folder ".skott" will be generated at the same location the command was run.

$ skott --incremental

After re-running the same command, you should be able to see the difference between the time took for the analysis without the cache. Note that difference may seem slim if the project doesn't contain much files.

Note that because resolving paths from a cache involves many edge cases, incremental mode is not turned on by default yet, but will be once the feature covers most of the cases I have in mind. Most of the time using it works well though, so don't hesitate to abuse the feature and open issues if ever you encounter some problems while enabling the incremental feature to make it more stable.

Unused dependencies (introduced in v0.13.0)

The unused dependencies feature will try to cover as much use cases as possible over time. For now what have been introduced in v0.13.0 is the detection of unused npm production dependencies. Note that only production code is analyzed as of now, so if you're using production dependencies in test files, Skott will report them as unused.

$ skott --showUnusedDependencies --trackThirdPartyDependencies --displayMode=raw

--trackThirdPartyDependencies is required for unused npm dependencies to be found.

Thanks to @ild0tt0re for that feature request.

Conclusion

That's it for the latest updates about Skott :) As promised in the latest blog posts of this series, the next one will be about how I leveraged Test-Driven Development and Dependency Injection to confidently develop the whole chain of file exploring, parsing and analysis 😎

Also, here is the repository if you want to know more.

Thanks for reading, see you next time!

📦 Everything you need to know: package managers

Antoine Coulon — Fri, 18 Nov 2022 09:51:21 +0000

Welcome everyone! This article is the first one of the Everything you need to know, a Software Engineering series.

In this series, I will try to give you a solid basic understanding about Software Engineering concepts I consider important.

All modern computer systems include tools that automate the process of installing, uninstalling and updating software.

This responsibility is that of a package manager and several can intervene within the same computer system.

Operating system

The majority of Unix-based operating systems embed a package manager as standard, providing a multitude of different packages very simply.

If you have ever used a Linux distribution such as Ubuntu or Debian, you've probably used a package manager before. If I say apt-get update does that ring a bell?

This command tells APT to update all versions of packages installed. APT (Advanced Packaging Tool) is a package manager embedded very widely as standard on Linux operating systems. To install a package, you can for example enter the command apt-get install <package>.

Programming language

Most programming languages can embed their own with package managers, either natively or provided within their respective ecosystem.

Take for example npm, the default package manager for Node.js. We can also mention pip for Python, NuGet for C#, Composer for PHP, etc. Similar to APT, npm makes it easy to install packages using the npm install <package> command.

For this article, I decided to take npm as an example.
npm is indeed a very good support to highlight the advantages but also the disadvantages that a package manager can have.
The advantages and disadvantages listed in the following part are valid for all package managers.

npm is installed alongside Node.js. To reproduce these examples, [you only need to install Node.js here].

In four parts, we will see what are the main reasons for such an expansion of package managers to all layers of a computer system.

1. Ease of use and maintenance of packages

The main interest of a package manager is obviously to simplify the installation of dependencies external to our application. Before the rise of npm in January 2010, the dependencies of a JavaScript application were mostly installed manually. By "manual installation" I mean:

downloading a zip archive from a remote server
unzipping the archive in the project
manual referencing of the installed version, and this with each update of a dependency.

With a package manager like npm, we therefore benefit from:

Simplified installation of a package npm install <package>
The simplified update of a package npm update <package>
The simplified removal of a package npm uninstall <package>

The packages are installed in a node_modules folder adjacent to the application and which is entirely managed by npm. All packages located in the node_modules folder can be directly imported from the application.

In general, each programming language natively embeds its own module resolution management mechanism.

1.1. Install

In order for a package to be installed, we first need a name which is in most cases used as a unique identifier. Naming conventions can differ from one ecosystem to another.

$ npm install rxjs

With this command, the package manager will search within the registry for a package that has the name rxjs. When the version is not specified, the package manager will usually install the latest available version.

1.2. Use

// ECMAScript Modules (ESM)
import { of } from "rxjs";
// CommonJS
const { of } = require("rxjs");

The module systems integrated into the programming languages make it possible to import a library installed locally and sometimes remotely (like Go or Deno for example). In this case with Node.js, the package must be installed locally in a node_modules folder. With Node.js, the module resolution algorithm allows the dependency to be in a node_modules folder either adjacent to the source code or in a parent folder (which sometimes leads to an unexpected behavior).

2. Managing the consistency of installed packages

Now, let's dive into a little more detail on one very important aspect that a package manager must manage: state consistency between installed packages. So far, installing a package looks like a trivial task, which is just to automate downloading a package of a certain version and making it available in a conventional folder that the application has access to.

However this management of consistency between packages turns out to be relatively difficult and the way of modeling the dependency tree varies according to the ecosystems. Most of the time, we talk about a dependency tree, but we can also talk about a dependency graph, in particular a directed graph.

If you are not familiar with the concept of directed graphs, I invite you to read the series of articles I wrote about it on dev.to with examples in JavaScript.

The implementations of these data structures can be drastically different depending on the ecosystem of a package manager, but also between package managers of the same ecosystem (npm, yarn, pnpm for Node.js for example).

How to ensure that all developers share the same dependencies and therefore the same versions of each underlying library?

Still in the context of npm, let's take for example a very simple list of dependencies, expressed as an object in the package.json file:

package.json

{ 
  "dependencies": {
    "myDependencyA": "<0.1.0"
  }
}

This object describes a dependency of our project on the myDependencyA library downloadable from the npm registry. Semantic Versioning here constrains the version of the library to be installed (here lower than 0.1.0).

Semantic version management (commonly known as SemVer) is the application of a very precise specification to characterize the version of software. For more information on this subject, I invite you to take a look at the official specification https://semver.org/lang/fr/

In our case, by remaining on the classic <major>.<minor>.<patch> scheme, we express the possibility of installing all the versions of myDependencyA from "0.0.1" to "0.0.9". This therefore means that any version of the dependency that respects the range is considered valid. On the other hand, this also means that if a developer A installs the dependency at 2 p.m. and a developer B installs the dependency at 5 p.m., they may both not have the same dependency tree if ever a new version of myDependencyA is released in the meantime.

The npm dependency resolution algorithm will by default favor the installation of the most recent dependency that respects the semantic management described in the package.json. By specifying npm install myDependencyA, the most recent version of myDependencyA will be installed respecting the constraint "<1.0.0" (version strictly lower than "1.0.0").

The major problem with this approach is the lack of stability and reproducibility of the dependency tree from one computer to another, for example between developers or even on the machine used in production. Imagine that version 0.0.9 of myDependencyA has just been released with a bug and your production machine is about to do an npm install on Friday at 5:59 PM…

The very simple example is often referred as version drift. This is why a single description file (in this case package.json) cannot be enough to guarantee an identical and reproducible representation of a dependency tree.

Other reasons include:

using a different version of the package manager whose dependency installation algorithm may change.
publishing a new version of an indirect dependency (the dependencies of the dependencies we list in the package.json here), which would result in the new version therefore being uploaded and updated.
the use of a different registry which for the same version of a dependency exposes two different libraries at a time T.

Lockfiles to the rescue

To ensure the reproducibility of a dependency tree, we therefore need more information that would ideally describe the current state of our dependency tree. This is exactly what lockfiles do. These are files created and updated when the dependencies of a project are modified.

A lockfile is generally written in JSON or YAML format to simplify the readability and understanding of the dependency tree by a human. A lockfile makes it possible to describe the dependency tree in a very precise way and therefore to make it deterministic and reproducible from one environment to another. So it's important to commit this file to Git and make sure everyone is sharing the same lockfile.

package-lock.json

{
  "name": "myProject",
  "version": "1.0.0",
  "dependencies": {
    "myDependencyA": {
      "version": "0.0.5",
      "resolved": "https://registry.npmjs.org/myDependencyA/-/myDependencyA-0.0.5.tgz",
      "integrity": "sha512-DeAdb33F+"
      "dependencies": {
        "B": {
          "version": "0.0.1",
          "resolved": "https://registry.npmjs.org/B/-/B-0.0.1.tgz",
          "integrity": "sha512-DeAdb33F+"
          "dependencies": {
            // dependencies of B
          }
        }
      }
    }
  }
}

For npm, the basic lockfile is called package-lock.json. In the snippet above, we can precisely see several important information:

The version of myDependencyA is fixed at "0.0.5" so even if a new version is released, npm will install "0.0.5" no matter what.
Each indirect dependency describes its set of dependencies with versions that also describe their own versioning constraints.
In addition to the version, the contents of the dependencies can be checked with the comparison of hashes which can vary according to the registers used.

A lockfile therefore tries to accurately describes the dependency tree, which allows it to remain consistent and reproducible over time at each installation.

⚠️ But...

Lockfiles don't solve all inconsistency problems! Package managers implementations of the dependency graph can sometimes lead to inconsistencies. For a long time, npm's implementation introduced Phantom Dependencies and also NPM doppelgangers which are very well explained on the Rush.js documentation website (advanced topics that are out of the scope of this blog post).

3. Provision of distributed and transparent databases via open-source

Distributed registries

A package manager is a client that acts as a gateway to a distributed database (often called a registry). This allows in particular to share an infinite number of open-source libraries around the world. It is also possible to define company-wide private registries in a secured network, within which libraries would be accessible.

Verdaccio allows to setup a private proxy registry for Node.js

The availability of registries has greatly changed the way software is developed by facilitating access to millions of libraries.

Transparent access to resources

The other benefit of open-source package managers is that they most often expose platforms or tools that allow browsing through published packages. Accessing source code and documentation has been trivialized and made very transparent. It is therefore possible for each developer to have an overview or even to fully investigate the code base of a published library.

4. Security and integrity

Using open-source registries with millions of publicly exposed libraries is pretty convenient, but what about security?

It is true that open-source registries represent ideal targets for hackers: all you have to do is take control of a widely used library (downloaded millions of times a week) and inject malicious code into it, and no one will realize!

In this part, we will see the solutions implemented by package managers and registries to deal with these attacks and limit the risks.

Integrity safety for each installed package

Given that a package can be installed from any registry, it is important to implement verification mechanisms at the level of the content of the downloaded package, to ensure that no malicious code has been injected during the download, regardless of its origin.

For this, integrity metadata is associated with each installed package. For example with npm, an integrity property is associated with each package in the lockfile. This property contains a cryptographic hash which is used to accurately represent the resource the user expects to receive. This allows any program to verify that the content of the resource matches what was downloaded. For example for @babel/core, this is how integrity is represented in package-lock.json:

"@babel/core": {
   "version": "7.16.10",
   "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.16.10.tgz",  
   "integrity": "sha512 pbiIdZbCiMx/MM6toR+OfXarYix3uz0oVsnNtfdAGTcCTu3w/JGF8JhirevXLBJUu0WguSZI12qpKnx7EeMyLA=="
}

Let's take a closer look at how integrity can drastically reduce the risk of injecting malicious code by hashing source code.

As a reminder:

We call hash function, a particular function which, from a datum supplied as input, calculates a digital fingerprint used to quickly identify the initial datum, in the same way as a signature to identify a person. Wikipedia

Let's take for example a simple case:

// my-library
function someJavaScriptCode() {
  addUser();
}

Let's imagine that this JavaScript code represents a resource that a user might want to download. Using the SHA1 hash function, we get the hash 7677152af4ef8ca57fcb50bf4f71f42c28c772be.
If ever malicious code is injected, the library's fingerprint will by definition change because the input (source code here) to the hash function will have changed:

// my-library
function someJavaScriptCode() {
  processMaliciousCode(); // this is injected, the user is not  expecting that
  addUser();
}

After injecting the malicious code, still using the same SHA1 hash function, we obtain 28d32d30caddaaaafbde0debfcd8b3300862cc24 as the digital fingerprint.
So we get as results:

Original code = 7677152af4ef8ca57fcb50bf4f71f42c28c772be
Malicious code = 28d32d30caddaaaafbde0debfcd8b3300862cc24

All package managers implement strict specifications on this approach to integrity. For example, npm respects the W3C's "Subresource Integrity or SRI" specification, which describes the mechanisms to be implemented to reduce the risk of malicious code injection.
You can jump directly here to the specification document if you want to dig deeper.

Security constraints at the author level

To strengthen security at the level of open-source packages, more and more constraints are emerging on the side of project authors and maintainers. Recently, GitHub, which owns npm, announced that it is forcing two-factor authentication (2FA) for contributors to the 100 most popular packages. The main idea around these actions is to secure resources upstream by limiting write access to open-source packages and identifying people more precisely.

It's important to also mention that there are tools that can be used to perform automatically scans and audits continuously.

Built-in tools

In order to automate the detection of vulnerabilities, many package managers natively integrate tools allowing to scan the installed libraries. Typically, these package managers communicate with databases that list all known and referenced vulnerabilities. For example, GitHub Advisory Database is an open-source database that references thousands of vulnerabilities across multiple ecosystems (Go, Rust, Maven, NuGet, etc) e.g. npm audit command uses this database.

Third-party tools

NodeSecure

At NodeSecure we are building free open source tools to secure the Node.js & JavaScript ecosystem. Our biggest area of expertise is in package and code analysis.

Here are some example of the available tools:

@nodesecure/cli, a CLI that allow you to deeply analyze the dependency tree of a given package or local Node.js project
@nodesecure/js-x-ray, a SAST scanner (A static analyser for detecting most common malicious patterns)
@nodesecure/vulnera, a Software Component Analysis (SCA) tool
@nodesecure/ci, a tool allowing to run SAST, SCA and many more analysis in CI/CDs or in a local environment

Snyk

Snyk is the most popular all-around solution for securing applications or cloud-based infrastructures. Snyk offers a free-tier with SAST and SCA analysis.

To ensure continuous detection of vulnerabilities, it is recommended to run scans each time packages are installed/modified.

Conclusion

There you go, you now know what issues are addressed and solved by package managers!

Package managers are complex tools that aim to make life easier for us as developers, but can quickly become problematic if misused.

It is therefore important to understand the issues they deal with and the solutions provided in order to be able to put into perspective several package managers of the same ecosystem. In the end, it's a tool like any other and it must mobilize thinking in the same way as when libraries/frameworks/programming languages are used.

Don't also forget to take into account security issues and use automated tools which can drastically reduce the attack surface!

🕶 What it takes to build a Static Analysis tool

Antoine Coulon — Mon, 17 Oct 2022 09:40:00 +0000

Hey dear developer! Welcome to the first chapter of My journey of building Skott, an open-source Node.js library, a little series in which I'll talk about different steps I've been through building the Skott project.

Before diving into the subject of today, here is the blog post introducing the project which is not mandatory to read to make this article valuable though, but it can help providing some background.

Ok without further ado, let's talk about the topic of the day: building a static analysis tool.

What is a Static Analysis?

A Static Analysis represents the process of analyzing the source code without running the application. You probably know many tools leveraging Static Analysis behind the scenes for example:

JavaScript/Node.js: ESLint
Python: Pylint
Rust: Clippy

Linters are a set of tools fully leveraging Static Analyzes to flag programming errors, bugs, stylistic errors and suspicious constructs without having to run the code.

As you can see on the image above, ESLint is able to detect unused variables and flag them as errors (according to the provided configuration).

How does the Static Analysis work behind the scenes?

The answer to that question will be done in three steps:

1. What is a Parser
2. What is an Abstract Syntax Tree
3. How to use an Abstract Syntax Tree

Building static analysis tools is not a trivial task, so we'll only scratch the surface here, but you should have some basics to go further if you want!

1. What is a Parser

A Parser is a program generating an intermediary data structure from an input string. Parsing is a is most typically done in two phases:

Lexical Analysis (also known as Lexer, Tokenizer)

The goal of this tokenization phase is to generate tokens from the input program, which is only a raw string at this point (any programming language file in fact, .js, .rs, .go, .py, etc).

Tokens are a collection of characters allowing to describe pieces of code.

Syntactic Analysis

Following the tokenization, the Syntactic Analysis takes produced tokens and generates an intermediary data structure describing precisely these tokens and their relationships. Spoiler: this intermediary data structure is most of the time an Abstract Syntax Tree (AST).

Here is a schema of a common parsing process:

2. What is an Abstract Syntax Tree

As already said in the previous section, an Abstract Syntax Tree is one of the data structure that can be generated while parsing source code (other structures can be used instead such as Concrete Syntax Trees depending on the use case).

The goal of this intermediate representation is to have a standardized way of representing the code that is easy to work with (but without any loss of information).

The most important keyword here is probably standardized, which is one of the main characteristics an AST should respect to be useful. As the AST is an intermediary data structure, the overall goal is to be able to transform the tree to anything we want for example transform the tree to produce a program in an entirely new language e.g. generating JavaScript from Scala or most commonly JavaScript from TypeScript, etc 😎

To allow that, each ecosystem/language has strict specifications describing what should be the standard shape of its own Abstract Syntax Tree. For the JavaScript language, this is the ESTree spec. This standardized format has the advantage of allowing any type of Parser (written using any programming language) to produce a common AST (following the ESTree spec) that can be then interpreted by any Interpreter (written in any language).

3. How to use an Abstract Syntax Tree

Generally speaking for compilers, an Abstract Syntax Tree is as I said multiple times an intermediary data structure which can be then transformed to produce byte code or any other source target e.g. TypeScript emits JavaScript using its own AST.

Nevertheless in the context of a Static Analysis, an Abstract Syntax Tree will most likely represent our final data structure needed as it will allow us to inspect the source code patterns directly from it (no need further transformations that a compiler would do).

By using the spec describing the shape of the AST, we're able (as Static Analysis tools developers) to rely on it to determine whatever static rule/condition, it's just a matter of finding data structure combinations.

To explore a little bit further the structure of ASTs for various languages, I like a lot AST Explorer

Let's wrap it up by example

To demonstrate a little Static Analysis tool in action, I'll show one use case of Skott, a library that builds a Directed Graph of relationships between files of a Node.js project.

If you are not fully sure of the purpose of Directed Graphs, you can check series I wrote on the topic here

To build that Directed Graph, Skott has to:
1. Use the entrypoint of the project and parse it
2. Statically find imports/exports statements of the entrypoint from the AST
3. Recursively follow imported/exported files and keep doing it until all files have been discovered

Let's do this.

1. Use the entrypoint of the project and parse it

index.js



import { runMain } from "./program.js";
import { makeDependencies } from "./dependencies.js";

// do something with runMain and makeDependencies

Once we read the entrypoint file, we can use Skott to extract import statements (here is above a simplified snippet from the Skott's codebase):

At line 3, we import parseScript from meriyah which is a JavaScript parser (could have been babel, acorn, swc, etc doesn't matter for our use case as long as they correctly implement the spec).

After that, we can parse the file content that will return us the root node of the Abstract Syntax Tree. Once the AST is generated by meriyah, the only thing left to do is to traverse the whole tree recursively and find the import statements.

To simplify the example, we only track ECMAScript modules here, but Skott also tries to deal with CommonJS modules.

How to find an import statement from within the AST?

Okay so now you might wonder how we can use the AST data structure to find import statements 🧐

Let's first take a look at the ESTree spec for es2015 to see how an import statement is represented in the AST:

Well, pretty simple!
Anytime we meet a node with type === "ImportDeclaration" we know that's an import statement.

So here's the function to detect if an AST node is a es2015 import statement:

Great, we're now able to find all import statements from any JavaScript file! Using the AST we could also get back information about where the import is located in the file and make it red highlighted in vscode for instance (but I'll let that vscode thing for 2023).

Thanks to our wonderful Skott performing Static Analysis, we're now able to generate and display the graph from all collected imports:

As we can see, three nodes have been found (the only three files of the sample project) and dependencies have been established between index.js and its two children being imported!

Conclusion

If you made it this far, thanks, I appreciate it :) But it also means that you've probably understood foundations about Parsers and Abstract Syntax Trees and that's even nicer!

Be sure to follow me if you're interested in discovering other topics I've covered throughout the journey of building Skott 💥

For the next episode of this series, I'm planning to talk about Test-Driven Development & Dependency Injection.

Last but not least don't hesitate to bring stars ⭐️, issues, feedbacks directly on GitHub here as it will allow me to drive the project where it will bring the most value for each developer.

Wish everyone a wonderful day ☀️

💥 Introducing Skott, the new Madge!

Antoine Coulon — Mon, 12 Sep 2022 07:49:34 +0000

Little announcement: thanks for anyone showing interest in skott! Just for your own information, this article was not recently updated to include all the latest features that skott has, but it still remains a good introduction to the spirit of the project. Feel free to check the GitHub repository to see the latest CLI/API versions!

Hello everyone, hope you're doing well!

Today I'm very pleased to share a fun project I have been working on for several months now: skott.

skott is a tool that generates a graph of dependencies from your JavaScript/TypeScript/Node.js projects (including TSX/JSX and ES6/CommonJS modules!) and helps you discover circular dependencies, cost imports, file sizes, and many more 💥

Node.js builtin and/or npm third-party dependencies can also be configured to be added in the graph. The main goal is to reproduce the architecture of your application at the file level.

Installation and use



$ npm install skott

skott provides an API to traverse the project graph, deeply search for circular dependencies, find parents or children for a given set of nodes, etc.

Here is a quick overview of the API:



import skott from "skott";

const { 
  getStructure, 
  findCircularDependencies, 
  findParentsOf, 
  findLeaves 
} = await skott({
  /**
   * The entrypoint of the project. Must be either a CommonJS or ES6 module.
   * For now, TypeScript files are not supported as entrypoints.
   */ 
  entrypoint: "dist/index.js",
  /**
   * Define the max depth of for circular dependencies search. This can be useful 
   * for performance purposes. This defaults to POSITIVE_INFINITY.
   */
  circularMaxDepth: 5,
  /**
   * This defines whether the base directory of the entrypoint must be included
   * in all the relatives file paths.
   * For the specified `dist/index.js` above, it would consider the root path
   * to be `./` consequently `dist/` would never appear in any file paths.
   */
  includeBaseDir: false
});

skott can also be used through the CLI.

Web Visualization

Using the webapp display mode skott --displayMode=webapp, skott generates the graph and then automatically opens an interactive web application in which you can visualize the graph more precisely.

As shown above, we can see all the nodes created from files of your project, and edges are simply representing the links between files of your project. Circular dependencies, third-party dependencies and built-in dependencies can be displayed on demand to adapt the amount of information displayed at once.

Console Visualization

Sometimes, you don't want to open a web interface to check what you want. For that, you can use other display mode that will render an UI in the console. Here is a quick preview of the graph generated for the fastify library:

We can also decide to track more dependencies, for example the npm third-party dependencies (by providing the "--trackThirdPartyDependencies" option)

Node.js builtin dependencies can also be tracked by providing the "--trackBuiltinDependencies" option.

File tree display

Different modes of display are available from the CLI, including file-tree which reconstructs a directory of files from the graph which is far more concise:

Static file generation

In addition to various display modes from the CLI, Skott is also able to create static files reflecting your project graph, including .json, .svg, .md. (using mermaid), and .png.

Here is an example creating a static file from skott itself:



$ skott dist/index.js --staticFile=svg

For medium to big size projects, you'll probably want to use the webapp display mode! 🫣

Circular dependencies

skott also helps finding out circular dependencies very efficiently in the project, you can even provide a max depth search to avoid deep searches that could be costly.

If you're not sure to see why circular dependencies can be problematic, I created a section Why you should care about circular dependencies and dead code in the root documentation of skott

Some options can also be provided to the CLI to configure the exit code that will be used when circular dependencies are met (defaults to "1" meaning error exit).

Skott is fast

We can easily compare skott with madge because skott already covers most of the features madge exposes for a Node.js project.

I did some benchmarks about the computing time required to build a set of graphs from popular libraries for both skott and madge and here are the results:

Webpack (+560 files)

webpack is a static module bundler for modern JavaScript applications.

Webpack is probably one of the heaviest open-source Node.js project I know. So let's do a benchmark!

using skott takes 503ms

using madge takes 2.5 seconds

N.B: the difference of files between skott and madge is only because of .json files that are ignored by skott in the graph (as well as other files such as binaries).

Knex.js (+6O files)

knex.js A SQL query builder that is flexible, portable, and fun to use!

using skott takes 60ms
using madge takes 450ms

For building the entire graph of knex.js with even more metadata, skott is 7.5 times faster!

Fastify.js (30 files)

fastify.js is a fast and low overhead web framework, for Node.js

using skott takes 50ms

using madge takes 350ms

In this case, skott is 7 times faster than madge.

Skott is extensible

skott aims to offer features that could be extended to any language, given that specific parsers can be implemented along the way. I started from only JavaScript, incrementally brought TypeScript and TSX/JSX support along the way. Why not bringing other languages around the table?

Stay tuned

Many more features will be implemented (skott does not yet have a major version released) to help you easily discover and demystify what's going on under the hood of your Node.js project! I'm also thinking about developing a Web Application that will help you visualize graphs and every metadata related to each file (number of imports, file size, unused imports, etc).

Sharing journey of building an open-source library

I'll also start a series about the journey of building skott, which notably includes:

[OUT] What it takes to build a static analysis tool (parsers, ast, etc)
How to make all of that fully testable by using dependency injection and Test-Driven Development!

Don't hesitate to bring stars ⭐️, issues, feedbacks directly on GitHub here as it will allow me to drive the project where it will bring the most value for each developer.

Thanks for reading this far, wish everyone a wonderful day ☀️

🛠 Let's implement the Incremental/Affected pattern, the killer feature used by Turborepo, Nx, Rush and many more popular tools

Antoine Coulon — Tue, 29 Mar 2022 11:51:37 +0000

In the introduction of the Master Directed Graphs by example with JavaScript, we saw how useful the directed graphs could be. Before reading this blog post, I highly recommend you to read it before going further in this series.

The goal of this series is to demonstrate the usefulness and power of directed graphs by implementing concrete examples with JavaScript.

Let's start by introducing our today's topic

Our today's objective is to explain in a simplified way how the Affected/Incremental pattern works. The whole idea of detecting affected projects/packages is to process tasks (build, lint, test) only on parts of project that change hence optimizing ressources and saving time when working on heavy projects.

Take for instance a simple JavaScript frontend application (app.jsx) with two libs:

design system library to export your custom UI components (e.g: by wrapping Material UI or Angular Material with custom themes)
date formatting library to configure an universal set of rules to work with dates

Our project would look like this:

project-folder/
│
└───design-system/
|   |   dist/
│   │   component1.jsx

│
└───temporal/
│   |   dist/
│   │   dateFormat.js
|
|   dist/
|   app.jsx
|   package.json

Our app.jsx represents our entry file and uses features from both design-system and temporal libs.

app.jsx

import Component1 from "./design-system/component1.jsx";
import { hoursToMilliseconds } from "./temporal/dateFormat.js";

function myApp() {
  return <Component1> {hoursToMilliseconds(2)} </Component1>;
}

To build the entire application, imagine we have an npm script which bundles all libs and the entry point into a single JavaScript file:

# Bundle the whole app including `design-system` and `temporal` libs
$ npm run build

You might wonder what is the problem with this approach. The answer is that there is no problem yet and this type of bundling is probably fine most of the time for small to medium sized projects.

However if your design-system and temporal end up growing and you even add more libraries (more components, but also including assets such as images, fonts etc) you might encounter performance issues when building the whole app.

Let's demonstrate that very easily:

project-folder/
│
└───design-system/ # 100+ components
│
└───temporal/ # 50+ files
|
└───lib3/ # library using node-sass (nearly 5 MB)
|
└───lib4/ # library using heavy npm modules

Let's say that this project takes X time in seconds to be entirely bundled. Now imagine you only change the color of your design-system/component1.jsx:

const component1 = styled.div`
  color: red;
`;

Nice! Your component now looks great... but you have to rebuild your whole app which also includes the rebuild of the heavy ones lib3 and lib4.
We have to build everything each time we want to ship the main application while not everything has changed and therefore does not necessarily have to be rebuilt.

This type of problem is often encountered in monorepos where many apps and libraries share the same build/test/lint tools.

Introducing the killer feature: the Incremental/Affected pattern

If you're using monorepo tools such as Nx, Turborepo or Rush, you have probably heard of Incremental and/or Affected patterns.

If you are not comfortable what is a monorepo and what are their objectives, I recommend you to take a look at a little reminder about it here.

In this blog post, we will use the Affected word but Incremental means literally the same thing here.

Ok, but what is the point with directed graphs?
The tool in charge of that (e.g: Turborepo, Nx, Rush) can introspect your project, and internally emit a Directed Acyclic Graph responsible for establishing dependencies between pieces of your project. By using the emitted Graph and a persisted cache (also handle by the tool), this enables smart builds and many other tasks that depends on the state of the project (linting, testing, etc).

In the image above, we can see a project using an affected strategy to build only what really needed to be rebuilt. When building the Main App after Library 2 changed, only Library 1 and Library 2 must be rebuilt. The other part of the graph (Library 3, Library 4, Library 5) remains unaffected hence the cached version of these libraries can be used in the final build (no need to rebuild them).

Let's start by implementing a minimalist version of the Affected pattern for a project containing with three distinct libraries.

/**
 * lib1 depends on lib3 (via the use of lib3.MyLib3Component)
 * while library 2 is independent.
 */
const lib1Metadata = {
  id: "lib1",
  adjacentTo: [],
  payload: {
    component: `<lib3.MyLib3Component/ >`,
  },
};

const lib2Metadata = {
  id: "lib2",
  adjacentTo: [],
  payload: { component: `<div>hello from lib2</div>` },
};

const lib3Metadata = {
  id: "lib3",
  adjacentTo: [],
  payload: { component: `<MyLib3Component>hello lib3</MyLib3Component>` },
};

We now have to express our different components in our Graph context that we will use afterwards.
Let's start by adding the core items of a graph: the vertices. A vertex can represent any type of item which in our case is a library of a given project.

import { DiGraph } from "digraph-js";
const projectGraph = new DiGraph();

// Each project is represented with a vertex (node)
projectGraph.addVertices(lib1Metadata, lib2Metadata, lib3Metadata);

Now that we added our vertices, we must represent relationships between them by adding appropriate vertices.
In our example project, the lib1 depends on lib3 (by using the lib3 component). Consequently this import creates an implicit relationship between these nodes hence needs to be represented with an edge.

/**
 * lib1 depends on lib3, we must represent this relationship
 * by adding an edge from lib1 to lib3.
 */
projectGraph.addEdge({ from: lib1Metadata, to: lib3Metadata });

We just finished expressing our relationships between libraries of our project so we are now ready to implement our caching system in order to enable affected builds!

/**
 * Simulating a simple cache, persisting an hashed
 * value of the component.
 * In a real world project, you would use something
 * like the "folder-hash" library to generate hashes
 * for a given folder containing sub directories and files.
 */

const cache = {
  lib1: {},
  lib2: {},
  lib3: {},
};

The first function we need is a function to build a library.
During that build process, its content is hashed and stored in the cache object.

import crypto from "node:crypto";

function buildLibrary(library) {
  // Create a SHA1 using the library content
  const libraryHashedContent = crypto
    .createHash("sha1")
    .update(library.payload.component)
    .digest("hex");

  console.log(`Building library: '${library.id}'`);
  // Webpack, Rollup or any other bundler can be processed here

  // Store the hashed content in cache
  cache[library.id].component = libraryHashedContent;
}

Using that cache, it's pretty straightforward to compare if a library changed. If we are able to detect if a library changed, it means that we are able to detect if it needs to be rebuilt:

function isLibraryAffected(library) {
  const libraryHashedContent = crypto
    .createHash("sha1")
    .update(library.payload.component)
    .digest("hex");

  return libraryHashedContent !== cache[library.id].component;
}

From now on once we change something in a library, we will be able to detect changes and invalidate the cache.

Hashed build + Cache = Affected build

Now that we're able to build a library, generate a hash from it and store it in a cache where build versions can be compared over time, we can put together the core function of the Affected pattern:

function buildAffected(library) {
  /**
   * If the component is still the same (i.e: hash data hasnt changed), we
   * don't want to rebuild it. If its "Affected", we must invalidate it
   */
  if (isLibraryAffected(library)) {
    // Component's hash changed, meaning we must build the library
    buildLibrary(library);

    return { hasLibraryBeenRebuilt: true };
  }

  // The lib has not changed so does not require a new build
  console.log(`Using cached version of '${library.id}'`);

  return { hasLibraryBeenRebuilt: false };
}

Let's try out to run an Affected build on a given library:

// build lib1 using Affected detection
buildAffected(lib1Metadata);
// => prints out:  "Building library: 'lib1'";

// re-run a build without any changes multiple times
buildAffected(lib1Metadata);
buildAffected(lib1Metadata);
buildAffected(lib1Metadata);
// => prints out each time: "Using cached version of 'lib1'";

// change lib1's content
lib1Metadata.payload.component = "<div> Hello lib1 </div>";

// re-run a build
buildAffected(lib1Metadata);
// => prints out:  "Building library: 'lib1'";

Ok cool, we demonstrated how this pattern works for a given library. You may wonder why the Affected pattern is so useful in big projects including hundreds of libraries that are relying on each other. The answer is the Affected pattern proves to be extremely powerful and useful in a project where unnecessary computations are saved. If you're using a project with three libraries, it's probably not worth it to implement the Affected pattern which will cost you more CPU than building from scratch every time.

Enough talk, let's continue our project example with a use case allowing us to feel its rising power.

Implementation of the Affected pattern using digraph-js

digraph-js is a library which helps building directed graphs and allows us to traverse graphs effortlessly

Following the example, we will now implement a more complete version of the pattern which includes multiple libraries detection using the digraph-js library.

As a quick reminder, we still have three libs in our project example:

const lib1Metadata = {
  id: "lib1",
  adjacentTo: [],
  payload: {
    component: `<lib3.MyLib3Component />`,
  },
};

const lib2Metadata = {
  id: "lib2",
  adjacentTo: [],
  payload: { component: `<div>hello lib2</div>` },
};

const lib3Metadata = {
  id: "lib3",
  adjacentTo: [],
  payload: { component: `<MyLib3Component>hello lib3</MyLib3Component>` },
};

lib1 depends on lib3 (lib1 uses "MyLib3Component")
lib2 depends on nothing (no dependency)
lib3 depends on nothing (no dependency)

What directed graphs allow us is that given a root library (can be any node in the graph), we can traverse all dependencies of that library using edges pointing towards other nodes of the graph. Using that feature, we are able to determine on which other libraries the root library depends on.

Let's build a function whose goal is to build all dependencies of a given root node. For now, the root node does not know the status of all its children (either they have been rebuilt or not).

// Given a root node, we traverse the graph looking for dependencies
function* buildAllRootLibraryDependencies(rootLibrary) {
  // Traverse all rootLibrary's dependencies
  for (const rootLibraryDependency of projectGraph.getAdjacentVerticesTo(
    rootLibrary
  )) {
    /**
     * Recursively build affected libraries starting from the deepest dependencies
     * of the root library.
     */
    yield* buildAllLibraryDependencies(rootLibraryDependency);
  }

  /**
   * When we reach a dependency with no other dependencies, we
   * now that we're done going deep into the dependency tree.
   * If the library has been rebuilt, we must inform the
   * parent in order to recursively invalidate nodes in the
   * traversed path.
   */
  const { hasLibraryBeenRebuilt } = buildAffected(rootLibrary);

  yield hasLibraryBeenRebuilt;
}

We can traverse each dependency of the root node and rebuilt it if affected.
Now, we must include the root node in that affected process by using the buildAllRootLibraryDependencies function above.

There are 2 conditions requiring the root library to be rebuilt:

At least one of the dependencies of the library changed (can be determined by keeping track of each children state)
The root library itself changed (same simple hash comparison mentioned in the first part of the blog post)

Let's implement that last function:

/**
 * Build everything with affected strategy including root
 * library itself
 */
function buildEverythingAffectedIncludingRootLibrary(rootLibrary) {
  const rootLibraryDependencies =
    projectGraph.getAdjacentVerticesTo(rootLibrary);
  const allRebuiltLibraries = [];

  for (const dependencyLibrary of rootLibraryDependencies) {
    /**
     * Keep track of all the children builds to know if some
     * were rebuilt. If there is no affected dependency, the
     * array would remain empty
     */
    allRebuiltLibraries.push([
      ...buildAllRootLibraryDependencies(dependencyLibrary),
    ]);
  }

  /**
   * All root library's dependencies were rebuilt if necessary (i.e: affected).
   * However, we now need to determine if the root library has to also be
   * rebuilt. There are 2 conditions requiring the root library to be rebuilt:
   * - The root library itself changed
   * - Atleast one of the dependencies of the library changed
   */
  const HAS_LIBRARY_BEEN_REBUILT = true;
  const atleastOneLibraryChanged = allRebuiltLibraries
    .flat()
    .includes(HAS_LIBRARY_BEEN_REBUILT);

  if (atleastOneLibraryChanged) {
    buildLibrary(rootLibrary);
  } else {
    // Check if library itself changed by running affected detection on the root library
    buildAffected(rootLibrary);
  }
}

That's it! The Affected pattern is now fully demonstrated using directed graphs to traverse all dependencies of a given root node and using a cache comparison to determine if a child node must be invalidated then rebuilt.

You're not convinced? Let's confirm that by testing with a function:

function buildProjectUsingAffectedStrategy() {
  console.log("\n----STEP 1-----");
  // Building for the first time
  buildEverythingAffectedIncludingRootLibrary(lib1Metadata);

  console.log("\n----STEP 2-----");
  /**
   * Building for the second time but no dependencies of lib1 changed (neither
   * lib3 or lib4) so it remains UNAFFECTED (i.e: using cache)
   */
  buildEverythingAffectedIncludingRootLibrary(lib1Metadata);

  console.log("\n----STEP 3-----");
  /**
   * Let's now change the content of lib3's component.
   * Remember, lib1 depends on lib3 via the use of lib3.MyLib3Component so
   * this change should trigger an affected build.
   */
  console.log("Changing lib3's content...");
  // addMutation is a function which update the value of a given vertex in the graph
  projectGraph.addMutation(lib3Metadata, {
    // new lib3 component
    component: `<MyLib3Component>Hello affected lib3!</MyLib3Component>`,
  });

  console.log("\n----STEP 4-----");
  /**
   * Now that lib3 (dependency of lib1) changed, both lib3 and lib1 are considered
   * affected. It means that we must rebuild both, starting with lib3 (lib1 must be built
   * with the latest version of lib3).
   */
  buildEverythingAffectedIncludingRootLibrary(lib1Metadata);
}

Here is the output:

----STEP 1-----
Building library: 'lib3' // building for the first time
Building library: 'lib1' // building for the first time

----STEP 2-----
Using cached version of 'lib3' // building for the second time but nothing changed so we can use a cached version of lib3
Using cached version of 'lib1' // building for the second time but nothing changed neither from lib3 nor from lib1 itself so we can use a cached version of lib1

----STEP 3-----
Changing lib3's content... // updating lib3 content

----STEP 4-----
Building library: 'lib3' // lib3 changed so must be rebuilt
Building library: 'lib1' // lib1 did not change but its direct dependency "lib3" changed so it must be rebuilt

That's finally it! We demonstrated the big picture of how the Affected pattern works under the hood.

GitHub repository

Feel free to check the full example here used in the final part of the blog post.

Coming next

In the next part of this series, we will talk about circular dependency detection (which is for instance implemented by
an ESLint plugin eslint-plugin-import/no-cycle alike) that are made possible with directed graphs !

Thanks for reading!

🔀 Master Directed Graphs by example with JavaScript (introduction)

Antoine Coulon — Tue, 15 Mar 2022 11:51:23 +0000

Introduction to a 3 part series

In mathematics, and more specifically in graph theory, a directed graph is a graph that is made up of a set of vertices (often called nodes) connected by directed edges (often called arcs).

The directed nature of the graph is useful in many cases because it allows us to precisely describe relationships between any vertices of the graph.

You already manipulate Directed Graphs without knowing it

Did you know that you were creating directed graphs whenever an import mechanism is used behind the scenes?

Take for instance the image above with four vertices, each representing a JavaScript file.

Now the question is: what are the relationships between these files? In all programming languages, one file might import one or multiple files. Whenever a file imports another one, an implicit relationship is created.

src/hello.js

  export function sayHello() { }

src/main.js

  import { sayHello } from "hello.js";

As you can see above, main.js imports hello.js to use the sayHello function. The static import creates an implicit relationship between both files.

In the field of graphs, this relationship can be modeled as a directed edge from main.js to hello.js (can be written as main.js ---> hello.js). We say that main.js is adjacent to hello.js but generally speaking, we can allow ourselves to say that main.js depends on hello.js.

Given two vertices A and B, if a directed Edge X starting from A to B we can then say that A is adjacent to B.

We can now update the graph with our edges represented:

Basically this graph says that:

FileA directly depends on FileD and FileB
FileA indirectly depends on FileC (through both FileD and FileB)
FileD directly depends on FileC
FileB directly depends on FileC
FileC depends on nothing

This structure may seem simple but can in fact be used to model very complex schemas. Let's take a look at three examples of interesting use cases for directed graphs.

1. Static dependencies analysis

=> Cycle dependencies detection by ESLint for JavaScript ESLint no-cycle plugin

Circular dependencies can make your program crash or introduce inconsistencies in many various ways in this is not something you should underestimate. Luckily in Node.js, most famous module systems can resolve dependencies cycles and avoid your program to crash (some module systems do better than others, though).

Nevertheless, circular dependencies are often an indicator that there is a more or less deep misconceptions in your project, so I always advise to resolve circular dependencies.

Further exploring of the circular dependencies detection:

See an implementation of a circular dependency detection using the digraph-js library I wrote

2. Incremental/Affected tasks

=> Bundlers/Monorepos tools make extensive use of it (e.g: NX's affected build/test/lint...)

A directed graph can also be used in order to establish affected patterns. The affected pattern consists in analyzing source code and figuring out what can be affected by every code change.

In the image above, we can see a project using an affected strategy to build only what really needed to be rebuilt. When building the Main App after Library 2 changed, only Library 1 and Library 2 must be rebuilt. The other part of the graph (Library 3, Library 4, Library 5) remains unaffected hence the cached version of these libraries can be used in the final build (no need to rebuild them).

In a monorepo or in custom projects setup, this affected pattern can drastically reduce the time taken by trivial tasks such as build/test/lint.

Further exploring of the Affected pattern:

See an implementation of an affected pattern using the digraph-js library I wrote

3. Task orchestration

In the context of task orchestration/scheduling, directed graphs can also be very precious.

Take for instance a tool that everybody knows: Microsoft Excel. Have you ever wondered how changing the formula from one cell could directly affect other cells depending on this formula? I hope you are not disappointed to learn that it is not an infinite loop running under the hood.

In this context, our Directed Graph has a vertex for each cell to be updated and an edge in between whenever one of them needs to be updated earlier than the other.

Due to the fact that we need to consistently schedule the tasks involved, we can't have cycles in our Graph. Dependency Graphs without circular dependencies form Directed Acyclic Graphs (DAGs).

This acyclic constraint allows us to be consistent with the order of the various operations involved by updates.

In another context of Task Orchestration, we can briefly talk about Task Scheduling, which includes sequential vs parallel patterns.

Thanks to DAGs, we can easily determine what tasks can be run in parallel (no common dependencies in the Graph) and what tasks must be run sequentially (one after the other because one depends on the other).

Similar problems of Task Ordering arise in makefiles for program compilation, in YAML files for CI/CD and instruction scheduling for low-level computer program optimization.

Stay tuned

I'm planning to showcase some use cases introducing the use of graphs using the digraph-js library.

Some examples are already available on my GitHub

Thanks for reading :)

🔒 Make your JavaScript project safer by using this workflow

Antoine Coulon — Tue, 01 Feb 2022 18:41:03 +0000

The security issue

Have you ever been thinking about security in your JavaScript projects? No? Well, you should, because with new thousands of package published on npm everyday, vulnerabilities could come from your own code but also from your direct dependencies (node_modules).

Few months ago, coa npm library was used to steal users' personal data by injecting malicious code.
As a reminder coa was:

Downloaded approximately 9 million times per week

Used by about 5 million GitHub projects

And that's just one story among many others...

If you're using npm to download dependencies, you have probably already encountered this message:

After each npm install, npm runs an audit scan against your updated dependencies. Here, we have 79 vulnerabilities, coming from one or many dependencies. Each one represents a potential threat and should be fixed.

Where do these vulnerabilities come from? Basically, npm maintains a vulnerability Database which is updated on a daily basis. Many other databases exist, here is an exhaustive list about most popular open-source databases for the JavaScript ecosystem:

These ressources are great, but we are lazy developers focused on productivity and we want to automate that, so we don't have to manually check all databases at 8 am every day before processing new features.

The security solution

First things first, I want to warn you about the fact that there is no silver bullet for security concerns.

If security were all that mattered, computers would never be turned on, let alone hooked into a network with literally millions of potential intruders. Dan Farmer, pioneer in the development of vulnerability scanners for Unix operating systems and computer networks.

Nevertheless, you can drastically reduce the amount of vulnerabilities by using tools that can be easily integrated with your projects.
However most of the time these tools are not open-source hence not for free use.

NodeSecure Continuous Integration

NodeSecure is an open source organization that aims to create free JavaScript security tools. Our biggest area of expertise is in npm package and code analysis.

To see more, read these NodeSecure series, written by Thomas @fraxken, founder of the GitHub organization.

What is @nodesecure/ci

@nodesecure/ci brings together a set of tools to identify dependencies vulnerabilities and track most common malicious code and patterns using Static Code Analysis and Vulnerabilities Analysis

If your project (custom configuration is available) passes all security checks, the process exit with no error code otherwise, it fails.

Here is a preview:

How to use

- GitHub Action

If you use GitHub Actions, you have a very straightforward way to add the official NodeSecure ci-action action to your workflow:

workflow.yaml

steps:
      - uses: actions/checkout@v2
      - uses: NodeSecure/ci-action@v1

Now your source code and its dependencies will be automatically analyzed, ironically without even adding new dependencies to your projects. That's also a perfect fit if your tech lead doesn't want you to add new dependencies (node_modules already heavier than the universe).

- Node.js Script

Install the @nodesecure/ci package and start using the entry script node_modules/.bin/nsci

As well as for the GitHub Action, you can provide a custom configuration through CLI arguments.

First, reference the binary script in the package.json

{
   "scripts": {
       "nsci": "nsci"
   }
}

Then start it providing different arguments (all can be used at once, by the way):

$ npm run nsci -- --directory=/Users/user1/myproject
$ npm run nsci -- --strategy=npm
$ npm run nsci -- --vulnerability=all
$ npm run nsci -- --warnings=error
$ npm run nsci -- --reporters=console

- Module API

@nodesecure/ci exposes its pipeline runner as an API to allow use in any other combined workflow.

import { runPipeline } from "@nodesecure/ci";

const optionsExample = {
    directory: process.cwd(),
    strategy: "node",
    vulnerabilities: "all",
    warnings: "error",
    reporters: ["console"]
}

await runPipeline(optionsExample);
// => the process can either exit with error code (1) 
// or no error code (0), depending on the pipeline status.

That's it, now you have no more excuses not to practice DevSecOps =)

Any feedback on @nodesecure/ci is welcome, the library is just getting started.

Feel free to reach me on GitHub @antoine-coulon

Thanks for reading.