DEV Community: Kmaschta

As a Team, How Do You Share Your Knowledge?

Kmaschta — Tue, 26 Nov 2019 11:37:49 +0000

Each engineer knowledge is endlessly valuable, and I'm searching for a way to make all those hours spent on problems profitable by disseminating them amongs a growing team of ~15 software engineers.

We already have countless ways to do so:

internal brown-bag lunches (BBL)
a lot of blog posts
regular hackdays with a demo to the team at the end
local meetups, etc

But we needs to formalize this knowledge into a base that is easily searchable, discoverable, and easy to feed.

We've heard of Stack Overflow for Teams, also considering to open a Notion.

What do you recommend?

Releasing Comfygure 1.0

Kmaschta — Tue, 28 May 2019 16:20:36 +0000

Two years ago, we introduced an open-source configuration manager called comfygure. Since then, we've been using this tool in production, and we've tweaked it so it's ready for prime time. Today, we are proud to announce that we released a stable version!

Read on to see what is included in this version, how you can test it, and what's our plan for the future of this project.

The blog post was published and commented on Hacker News : https://news.ycombinator.com/item?id=20029420

What Is Comfygure?

Let's take two minutes to explain what comfygure is and how it works. Not so long ago, I ran a small survey on how software engineers handle the deployment of their environments.

The survey is still online, and you can see the results by following this link: Configuration Management Survey Results

Let me summarize what the participants answered:

Software engineers and ops have full responsibility to trigger production deployment by themselves
Most of the time, 2 to 5 team members can do so
Deployments are automated, at least partially
Each team tends to have 2 to 3 environments per application
Environment variables, JSON files and YAML files are the favorites configuration formats
There is no consensus on how to store these configurations
Most answerers don't use configuration or secret managers but would like to

I also gave them a chance to say what they find frustrating about configuration management:

There were not enough participants to extrapolate. However, I understand every one of the participants because we were in a similar situation: too early to have an utomated and full-featured configuration management, but too far to keep managing configurations by hand.

This is the reason why we decided to build a new solution that fulfilled our needs.

A small tool that could store, version, retrieve, and format our application configurations in order to keep sync between coworkers and environments.

What's New In The Stable Release

Quick Configuration Accessors

This release introduces a new command (comfy set) and improves the comfy get command in order to let you read or update a subset of a configuration:

$ comfy set production version "1.0.0"
$ comfy get production version
1.0.0

$ comfy set production flags.stable true
$ comfy get production flags --json
{
    "stable": true
}

You no longer need to fetch the configuration file, open it with your IDE, update that one flag and update the full version of the file. All of that is managed by comfy set. It's particularly useful to quickly change feature flags or bump the project version.

Keep Track Of Your Configuration History

The previous version already had a comfy log command, showing the latest changes in the configuration. The stable release fixes a few things about it. Also, it is now possible to retrieve a specific version of your configuration.

Combined with tags, these commands will let you deploy a fixed version to your environment and rollback at the speed of light!

$ comfy log production
2019-5-24 10:50:30  production  da70d7d69bb7f158748cf3ea76c08b9c4a12c3c0    latest
2019-5-24 10:50:19  production  ae30ab2567e7316cf5521d26b8c7a03ece0522d3    no tag
2019-5-24 10:50:10  production  964e51df37c0fe2a518998fb6457b461c4013d28    no tag

$ comfy get production version --hash 964e51df37c0fe2a518998fb6457b461c4013d28
0.1.4
$ comfy get productiion version --hash ae30ab2567e7316cf5521d26b8c7a03ece0522d3
1.0.0

You may also note that the tag latest replaces the stable and next tags when your create a new environment. You are still able to create them manually by running comfy tag add production stable <hash>.

Host Your Own Configuration Store And Server

Marmelab hosts the default comfygure server at https://comfy.marmelab.com, for free. But some people don't want to let a third-party company store their configuration, even though it's a zero knowledge platform (the configuration is encrypted client-side), and this is perfectly fine.

This is the reason why we focused on making it easy to host your own comfygure service for that release.

To this end, we published extensive documentation explaining how you can deploy comfygure on ZEIT's now, for example, as well as a Docker image available on Docker Hub: marmelab/comfygure.

We use serverless to deploy our own comfygure origin server. That means you can also deploy your own comfy server on Amazon Web Services, Microsoft Azure, Google Cloud Platform, and many others!

Feel free to deploy your own comfygure server and play with it. The more comfy servers there are out there, the less this service is centralized. Our goal isn't to become the default host, just to make developers life a bit easier.

What's next?

The goal of this release was to implement the major breaking changes, and to publish a stable version. The goal of the 1.1 will be to make this tool easier to use.

Installing comfy locally is easy, since all the configuration is stored in a single file. On a server or on a CI, it might be a bit cumbersome, since it requires no less than six environments variables. How ironic for a tool that wants to make configuration easier to deal with! So the first milestone of the next release is to make server installation fast and easy, and update the related documentation, with some real examples.

Similarly, there is a small permission management system on comfy with read-only and admin tokens. This is not explicit enough and these tokens need to be editable. This will also be an objective of the next release.

Also, I find pretty hard to navigate the configuration history. I'd love to have a comfy diff <env> <tag|hash> <tag|hash> command that would show what changed between two versions! So, it is in the pipes.

Finally and most importantly, I would love to hear what are you thinking of this small tool! Please give it a try and tell me what you think of it.

Here are some links where you can find more details about it:

HTTPS In Development: A Practical Guide

Kmaschta — Wed, 23 Jan 2019 21:50:50 +0000

According to Firefox Telemetry, 76% of web pages are loaded with HTTPS, and this number is growing.

Sooner or later, software engineers have to deal with HTTPS, and the sooner the better. Keep reading to know why and how to serve a JavaScript application with HTTPS on your development environment.

Why Use HTTPS On a Development Environment?

First, should you serve a website in production through HTTPS at all? Unless you really know what your are doing, the default answer is yes. It improves your website at so many levels: security, performance, SEO, and so on.

How to setup HTTPS is often adressed during the first release, and brings a lot of other questions. Should traffic be encrypted from end to end, or is encryption until the reverse proxy enough? How should the certificate be generated? Where should it be stored? What about HSTS?

The development team should be able to answer these questions early. If you fail to do so, you might end up like Stack Overflow wasting a lot of time.

Besides, having a development environment as close as possible from the production reduces risks that bugs reach the production environment, and also tend to decrease the time to debug those bugs. It's also true for end-to-end tests.

In addition, there are features that only work on a page served by HTTPS, such as Service Workers.

But HTTPS is slow! Many people believe that encryption is complicated and in a certain way must be slow to be efficient. But with modern hardware and protocols, this is not true anymore.

How To Generate A Valid Certificate For A Development Environment?

For production systems, it's easy to get a TLS certificate: generate one from Let's Encrypt or buy one from a paid provider.

For the development environment, it seems trickier, but it isn't that hard.

Mkcert: The No Brainer CLI

Filippo Valsorda recently published mkcert, a simple cli to generate locally-trusted development certificates. You just have to run a one-line command:

mkcert -install
mkcert example.com

The fully supported certificate will be available where your ran the command, namely at ./example.com-key.pem.

Manual Installation With OpenSSL

mkcert should fulfill all of your needs, unless you have to share the same certificate with your coworkers, or through other systems than your local env. In that case, you can generate your own certificate thanks to openssl.

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt

The certificate (server.crt) and its key (server.key) will be valid but self-signed. This certificate will be unknown to any Certificate Authority. But all browsers ask well-known certificate authorities to validate certificates in order to accept encrypted connections. For a self-signed certificate, they can't validate it, so they display an annoying warning:

You can accept that inconvenience and manually ignore the warning each time it shows up. But it's very cumbersome, and it may block e2e tests in a CI environment. A better solution is to create your own local certificate authority, add this custom authority to your browser and generate a certificate from it.

That's what mkcert does for you under the hood, but if you want to do it yourself, I wrote a gist that may help you: Kmaschta/205a67e42421e779edd3530a0efe5945.

HTTPS From a Reverse Proxy Or A Third-Party App

Usually, end-users don't directly reach the application server. Instead, user requests are handled by a load balancer or a reverse proxy that distributes requests across backends, stores the cache, protects from unwanted requests, and so on. It's not uncommon to see these proxies take the role of decrypting requests and encrypting responses as well.

On a development environment, we can use a reverse proxy, too!

Encryption via Traefik and Docker Compose

Traefik is a reverse proxy that comes with a lot of advantages for developers. Among others, it's simple to configure and it comes with a GUI. Also, there is an official docker image available on docker hub.

So, let's use it inside the docker-compose.yml of a hypothetical application that only serves static files:

version: '3.4'

services:
    reverse-proxy:
        image: traefik # The official Traefik docker image
        command: --docker --api # Enables the web UI and tells Traefik to listen to docker
        ports:
            - '3000:443'  # Proxy entrypoint
            - '8000:8080' # Dashboard
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
            - ./certs/server.crt:/sslcerts/server.crt
            - ./certs/server.key:/sslcerts/server.key
            - ./traefik.toml:/traefik.toml # Traefik configuration file (see below)
        labels:
            - 'traefik.enable=false'
        depends_on:
            - static-files
    static-files:
        image: halverneus/static-file-server
        volumes:
            - ./static:/web
        labels:
            - 'traefik.enable=true'
            - 'traefik.frontend.rule=Host:localhost'
            - 'traefik.port=8080'
            - 'traefik.protocol=http'
        ports:
            - 8080:8080

In this example, our static file server listens on port 8080 and serves files in HTTP. This configuration tells Traefik to handle HTTPS requests to https://localhost and proxy each of them to http://localhost:8080 in order to serve static files.

We also have to add a traefik.toml to configure the Traefik entry points:

debug = false

logLevel = "ERROR"
defaultEntryPoints = ["https","http"]

[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/sslcerts/server.crt"
      keyFile = "/sslcerts/server.key"

Here, we have two entry points: http and https, listening respectively to ports 80 and 443. The first one redirects to the HTTPS, and the second is configured to encrypt requests thanks to the specified TLS certificates.

Encryption From Docker Compose via Nginx

Obviously, we can do exactly the same with the popular Nginx reverse proxy. As Nginx can also directly serve static files itself, the setup is simpler. Again, the first step is the docker-compose.yml:

version: '3'

services:
    web:
        image: nginx:alpine
        volumes:
            - ./static:/var/www
            - ./default.conf:/etc/nginx/conf.d/default.conf
            - ../../certs/server.crt:/etc/nginx/conf.d/server.crt
            - ../../certs/server.key:/etc/nginx/conf.d/server.key
        ports:
            - "3000:443"

And the nginx configuration at default.conf:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl;

    server_name ~.;

    ssl_certificate /etc/nginx/conf.d/server.crt;
    ssl_certificate_key /etc/nginx/conf.d/server.key;

    location / {
        root /var/www;
    }

    ## If the static server was another docker service,
    ## It is possible to forward requests to its port:
    # location / {
    #     proxy_set_header Host $host;
    #     proxy_set_header X-Real-IP $remote_addr;
    #     proxy_pass http://web:3000/;
    # }
}

Serving HTTPS Directly From The Application

Sometimes security requirements demand end-to-end encryption, or having a reverse proxy just might seem to be overkill on a development environment. Most of the time, it's possible to serve HTTPS directly from your everyday development environment.

Let's take the example of a common stack: a React application with a REST API using express.

Using Create React App or Webpack Dev Server

Your average React app is bootstraped by create-react-app. This awesome tool comes with a lot of built-in features and can handle HTTPS out of the box. To do so, you just have to specify a HTTPS=true environment variable when starting the app:

HTTPS=true npm run start
# or
HTTPS=true yarn start

This command will serve your app from https://localhost:3000 instead of http://localhost:3000 with an auto-generated certificate. But it's a self-signed certificate, so the developer experience is poor.

If you want to use your own HTTPS certificate (signed with an authority that your browser trusts), create-react-app doesn't let you configure it without ejecting the app (npm run eject).

EDIT: A dev.to reader, Zwerge, found a clever workaround to replace the default HTTPS certificate on the fly:

  "scripts": {
    "prestart": "(cat ../../certs/server.crt ../../certs/server.key > ./node_modules/webpack-dev-server/ssl/server.pem) || :",
    "start": "react-scripts start",
  },

Fortunately, if you do eject CRA, or if your project is bundled with webpack, webpack-dev-server is as straightforward as create-react-app when it comes to serve HTTPS! It's possible to configure a custom HTTPS certificate with two lines in the Webpack configuration:

const fs = require('fs');
const path = require('path');

module.exports = {
    mode: 'production',
    // ...
    devServer: {
        https: {
            key: fs.readFileSync(path.resolve(__dirname, '../../certs/server.key')),
            cert: fs.readFileSync(path.resolve(__dirname, '../../certs/server.crt')),
        },
        port: 3000,
    },
};

The next time you'll run webpack-dev-server, it will handle HTTPS requests to https://localhost:3000.

Encrypted HTTP/2 With Express And SPDY

Now that we have our frontend part of the app that is served through HTTPS, we have to do the same with our backend.

For this purpose, let's use express and spdy. No wonder why these two libraries names are about SPEED, it's because they are fast to setup!

const fs = require('fs');
const path = require('path');
const express = require('express');
const spdy = require('spdy');

const CERTS_ROOT = '../../certs/';

const app = express();

app.use(express.static('static'));

const config = {
    cert: fs.readFileSync(path.resolve(CERTS_ROOT, 'server.crt')),
    key: fs.readFileSync(path.resolve(CERTS_ROOT, 'server.key')),
};

spdy.createServer(config, app).listen(3000, (err) => {
    if (err) {
        console.error('An error occured', error);
        return;
    }

    console.log('Server listening on https://localhost:3000.')
});

HTTP/2 isn't required to serve HTTPS, it's possible to serve encrypted content with HTTP first of the name, but while we are at serving HTTPS, we can upgrade the HTTP protocol. If you want to know more about the advantages of HTTP/2, you can read this quick FAQ.

Conclusion

Modern tooling allows to build applications that are safer and faster for end-users and, now, easy to bootstrap. I hope that I convinced you to use these libraries and technologies starting from your project inception, when they are still cheap to install.

All the examples I used in this blog post are gathered on the following repo: marmelab/https-on-dev. Feel free to play with and add your own HTTPS development experience!

How do you manage your web application configurations?

Kmaschta — Mon, 12 Nov 2018 16:40:43 +0000

In order to better understand current practices, I'm doing a survey about configuration management and deployment of web applications.

If you are, like me, willing to learn how the community thrive or struggle with their configurations or just curious about deployment, I propose you to:

Fill the small survey (less than 5min)
See the results live
Debate how and why you are using technics and not another

At the end, I'll write a blog post about the conclusions and what I've learn about that, trying to be neutral.

So, environment variables or YAML?
How do you share configurations between coworkers?

Site Reliability Engineering: Google's Secret Sauce For High Availability And Happy Ops

Kmaschta — Tue, 29 May 2018 14:29:43 +0000

Hope is not a strategy
— Traditional SRE saying

I've read the book Site Reliability Engineering - How Google Runs Production Systems. I learned a lot, and I took away many good practices to apply to our own services. Here is the gist, and what I've learned from it. I'll focus on what web developers can learn from this SRE thing, without entering in the complexity of the Google's infrastructure.

One quick disclaimer before diving in: at Marmelab, we don't run our customers' services in production. Our expertise is eliminating uncertainties through agile iterations, so we usually delegate hosting to a partner company. That said, our job can't be disconnected from the production. We are responsible for the quality of the delivered software, and that includes making software that reports quality of service problems, and an architecture that make it resilient and performant. So even though it's not our job per se, the operation of web services interests me a lot.

What Is Site Reliability Engineering (SRE)?

First and foremost, let's define this emerging profession invented by Ben Treynor Sloss, Senior VP of Engineering at Google. From his own words:

Fundamentally, SRE is what happens when you ask software engineers to design an operations function.

A typical SRE team is formed of 6 to 8 engineers, in order to keep a balanced on-call rotation. Half of it is composed of traditional software engineers (SWE in Google's parlance), and the other half of engineers who are almost qualified to be SWE, but who have skills and interests related to operational fields: Unix internals, networking, and so on.

They occupy a central position in the company's workflow. They are in relation with software engineers, release managers, data center engineers, product owners, accounting service, and upper management.

They are fully responsible for the availability, latency, performance, efficiency, charge management, monitoring, emergency response, and capacity planning of a given application.

In order to face these huge responsibilities and hard scaling challenges, they carefully manage their time. They do it in an unusual way: they must devote less than 50% of their time to operational tasks, toil, and emergency response. Most of their time should be spent writing software and tools that automate their own job, or making sure that their application heals itself.

DevOps vs. SRE

DevOps is a movement initiated in 2008-2009 by Patrick Debois to promote agility in the deployment process, and to reduce the gap between developers and ops.

SRE is rather an engineering field, a way to organize engineers in order to manage reliability as a whole. As explained page 7 of the SRE book:

One could equivalently view SRE as a specific implementation of DevOps with some idiosyncratic extensions.

Balancing Availability and Velocity

I expected this book to teach me Google's secret path to the "Always Available System™". I was wrong, but I learned a far more valuable lesson. Google engineers do not focus on 100% availability, because it's unrealistic and unsustainable.

Instead, according to the business requirements and expectations, each application or production system gets a Service Level Objective (SLO). For a typical Google app, the SLO is set around 99.99% ("four nines" availability), or 99.999% ("five nines" availability) of successful requests rate over all requests.

Here is how the number of nines translates to downtime per year:

99% ("two nines"): 3.65 days
99.9% ("three nines"): 8.77 hours
99.99% ("four nines"): 52.60 minutes
99.999% ("five nines"): 5.26 minutes

At marmelab, most of our projects have a 99.9% uptime (three nines, not that bad!), even though we focus far more on velocity than availability. One important takeway from the SRE book is that, if we ever wanted to reach the next service level grade (four nines), we'd have to increase our efforts on availability ten times.

The difference between 100% and "x nines" availability" is important because it provides a measurable room for maneuver. For example, if the unavailability is measured in terms of unplanned downtime and the SLO is set to 99.9%, over a year it is considered acceptable to have about eight hours of downtime.

This relatively thin margin of acceptable and unplanned downtime is a way to measure the risk induced by the velocity. If the availability decreases, and gets closer to the SLO, the SRE team should slow down the feature delivery, and focus on stabilizing the app. On the contrary, if the availability is far above the SLO, then the team has some margin, and can go ahead and push new features to prod.

Of course, the Service Level Objective can change over time. It should be discussed among all the stakeholders of the project in regard to all the circumstances. It is a valuable metric to have.

Eliminating Toil

Since SRE shouldn't devote more than 50% of their time to operational tasks, they regularly have to automate their work as their application scales. They should focus on reducing toil, defined as:

a manual, repetitive and automatable task that is mandatory to the proper functioning of the app and that grows with it.

Automating toil can be a non-negligible investment, but it comes with several advantages. Of course, it saves engineer's time. But it also leads engineers to focus on a task at the right time (not when the service is broken), in a less error-prone way. Moreover, it reduces context switching, allows everyone to take care of serious issues and focus on what really matters.

As developers, we also have to take care of toil every day - it just takes another form. Manual tests are the most common toil, that's why we write automated tests. The second most common toil is deployment and release management in general. Continuous Delivery is a luxury that I definitely recommend! Other toils are more subtle: updating dependencies that don't have breaking changes, benchmarking performance, auditing security vulnerabilities, etc.

Eliminating toil increases the ability to scale and makes the application nicer to play with. It's up to you to find it and automate it! The best way is to include this routine as a habit on a daily basis, just like any other development practice.

Blameless Culture of Retrospective

SRE has a culture of continuous improvement (what agile methodologies translate as the retrospective) in the form of postmortems. Every time an issue or an accident is big enough or is user-facing, the on-call engineer writes a postmortem, another engineer reviews it and publishes it to the whole team.

Postmortems are extremely useful to keep track of incidents, plan further actions, and avoid repeating mistakes. But SRE teams go beyond that: they write postmortems blamelessly.

Each postmortem focuses on facts and root causes analysis. It assumes that everyone had good intentions, and did the right actions with the pieces of information they had. All finger pointing and shaming are explicitly banned. Moreover, the management doesn't punish mistakes. Instead, they take postmortems as an opportunity to improve the robustness of their systems.

It's a hard thing to do, but removing blame and feels from retrospectives (and in this case postmortems) gives the confidence to escalate the real issues. Moreover, most system failures are due to a human mistake.

It was refreshing to read that from a book by Google engineers. It's a good example of how agile principles can be put in practice, beyond the development process.

Conclusion

This book is full of insights and I wrote about only a very small portion of them. Among other technical tips like "divide and conquer" or "simplicity by design", it also teaches how Google scales human interactions among its employees.

I warmly recommend Site Reliability Engineering to anyone who is interested in production scaling and DevOps, of course, especially the Practices part, which is full of practical stuff.

The whole book is a must for all engineering managers. The fourth part, Management, gives valuable insights on how to deal with interrupts, handle or recover from operational overload, and so on.

But I also recommend reading the Principles part to every product owner and agile guru, to get a better understanding of how ops teams work and collaborate.

The book is modular and easily accessible. You can read just a chapter of it without having to understand how the Google infrastructure works.

The SRE book is free to read online, or you can find it on Amazon!

Site Reliability Engineering is a new and very interesting field, and you might expect to read more about it on this very blog.

Finding And Fixing Node.js Memory Leaks: A Practical Guide

Kmaschta — Mon, 16 Apr 2018 11:54:46 +0000

Fixing memory leaks may not be not the shiniest skill on a CV, but when things go wrong on production, it's better to be prepared!

After reading this article, you'll be able to monitor, understand, and debug the memory consumption of a Node.js application.

When Memory Leaks Become A Problem

Memory leaks often go unnoticed. They become a problem when someone pays extra attention to the production performance metrics.

The first symptom of a memory leak on a production application is that memory, CPU usage, and the load average of the host machine increase over time, without any apparent reason.

Insidiously, the response time becomes higher and higher, until a point when the CPU usage reaches 100%, and the application stops responding. When the memory is full, and there is not enough swap left, the server can even fail to accept SSH connections.

But when the application is restarted, all the issues magically vanish! And nobody understands what happened, so they move on other priorities, but the problem repeats itself periodically.

Memory leaks aren't always that obvious, but when this pattern appears, it's time to look for a correlation between the memory usage and the response time.

Congratulations! You've found a memory leak. Now the fun begins for you.

Needless to say, I assumed that you monitor your server. Otherwise, I highly recommend taking a look at New Relic, Elastic APM, or any monitoring solution. What can't be measured can't be fixed.

Restart Before It's Too Late

Finding and fixing a memory leak in Node.js takes time - usually a day or more. If your backlog can't accomodate some time to investigate the leak in the near future, I advise to look for a temporary solution, and deal with the root cause later. A rational way (in the short term) to postpone the problem is to restart the application before it reaches the critical bloat.

For PM2 users, the max_memory_restart option is available to automatically restart node processes when they reach a certain amount of memory.

Now that we're comfortably seated, with a cup of tea and a few hours ahead, let's dig into the tools that'll help you find these little RAM squatters.

Creating An Effective Test Environment

Before measuring anything, do yourself a favor, and take the time to set up a proper test environment. It can be a Virtual Machine, or an AWS EC2 instance, but it needs to repeat the exact same conditions as in production.

The code should be built, optimized, and configured the exact same way as when it runs on production in order to reproduce the leak identically. Ideally, it's better to use the same deployment artifact, so you can be certain that there is no difference between the production and the new test environment.

A duly configured test environment is not enough: it should run the same load as the production, too. To this end, feel free to grab production logs, and send the same requests to the test environment. During my debugging quest, I discovered siege an HTTP/FTP load tester and benchmarking utility, pretty useful when it comes to measuring memory under heavy load.

Also, resist the urge to enable developer tools or verbose loggers if they are not necessary, otherwise you'll end up debugging these dev tools!

Accessing Node.js Memory Using V8 Inspector & Chrome Dev Tools

I love the Chrome Dev Tools. F12 is the key that I type the most after Ctrl+C and Ctrl+V (because I mostly do Stack Overflow-Driven Development - just kidding).

Did you know that you can use the same Dev Tools to inspect Node.js applications? Node.js and Chrome run the same engine, Chrome V8, which contains the inspector used by the Dev Tools.

For educational purposes, let's say that we have the simplest HTTP server ever, with the only purpose to display all the requests that it has ever received:

const http = require('http');

const requestLogs = [];
const server = http.createServer((req, res) => {
    requestLogs.push({ url: req.url, date: new Date() });
    res.end(JSON.stringify(requestLogs));
});

server.listen(3000);
console.log('Server listening to port 3000. Press Ctrl+C to stop it.');

In order to expose the inspector, let's run Node.js with the --inspect flag.

$ node --inspect index.js 
Debugger listening on ws://127.0.0.1:9229/655aa7fe-a557-457c-9204-fb9abfe26b0f
For help see https://nodejs.org/en/docs/inspector
Server listening to port 3000. Press Ctrl+C to stop it.

Now, run Chrome (or Chromium), and go to the following URI: chrome://inspect. Voila! A full-featured debugger for your Node.js application.

Taking Snapshots Of The V8 Memory

Let's play with the Memory tab a bit. The simplest option available is Take heap snapshot. It does what you expect: it creates a dump of the heap memory for the inspected application, with a lot of details about the memory usage.

Memory snapshots are useful to track memory leaks. A usual technique consists of comparing multiple snapshots at different key points to see if the memory size grows, when it does, and how.

For example, we'll take three snapshots: one after the server start, one after 30 seconds of load, and the last one after another session of load.

To simulate the load, I'll use the siege utility introduced above:

$ timeout 30s siege http://localhost:3000

** SIEGE 4.0.2          
** Preparing 25 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:               2682 hits
Availability:             100.00 %
Elapsed time:              30.00 secs
Data transferred:         192.18 MB
Response time:              0.01 secs
Transaction rate:          89.40 trans/sec
Throughput:             6.41 MB/sec
Concurrency:                0.71
Successful transactions:        2682
Failed transactions:               0
Longest transaction:            0.03
Shortest transaction:           0.00

Here is the result of my simulation (click to see the full size):

A lot to see!

On the first snapshot, there are already 5MB allocated before any request is processed. It's totally expected: each variable or imported module is injected into memory. Analysing the first snapshot allows optimizing the server start for example - but that's not our current task.

What interests me here is to know if the server memory grows over time while it's used. As you can see, the third snapshot has 6.7MB while the second has 6.2MB: in the interval, some memory has been allocated. But which function did?

I can compare the difference of allocated objects by clicking on the latest snapshot (1), change the mode for Comparison (2), and select the Snapshot to compare with (3). This is the state of the current image.

Exactly 2,682 Date objects and 2,682 Objects have been allocated between the two load sessions. Unsurprisingly, 2,682 requests have been made by siege to the server: it's a huge indicator that we have one allocation per request. But all "leaks" aren't that obvious so the inspector shows you where it was allocated: in the requestLogs variable in the system Context (it's the root scope of the app).

Tip: It's normal that V8 allocates memory for new objects. JavaScript is a garbage-collected runtime, so the V8 engine frees up memory at regular intervals. What's not normal is when it doesn't collect the allocated memory after a few seconds.

Watching Memory Allocation In Real Time

Another method to measure the memory allocation is to see it live instead of taking multiple snapshots. To do so, click on Record allocation timeline while the siege simulation is in progress.

For the following example, I started siege after 5 seconds, and during 10 seconds.

For the firsts requests, you can see a visible spike of allocation. It's related to the HTTP module initialization. But if you zoom in to the more common allocation (such as on the image above) you'll notice that, again, it's the dates and objects that take the most memory.

Using The Heap Dump Npm Package

An alternative method to get a heap snapshot is to use the heapdump module. Its usage is pretty simple: once the module is imported, you can either call the writeSnapshot method, or send a SIGUSR2 signal to the Node process.

Just update the app:

const http = require('http');
const heapdump = require('heapdump');

const requestLogs = [];
const server = http.createServer((req, res) => {
    if (req.url === '/heapdump') {
        heapdump.writeSnapshot((err, filename) => {
            console.log('Heap dump written to', filename)
        });
    }
    requestLogs.push({ url: req.url, date: new Date() });
    res.end(JSON.stringify(requestLogs));
});

server.listen(3000);
console.log('Server listening to port 3000. Press Ctrl+C to stop it.');
console.log(`Heapdump enabled. Run "kill -USR2 ${process.pid}" or send a request to "/heapdump" to generate a heapdump.`);

And trigger a dump:

$ node index.js
Server listening to port 3000. Press Ctrl+C to stop it.
Heapdump enabled. Run "kill -USR2 29431" or send a request to "/heapdump" to generate a heapdump.

$ kill -USR2 29431
$ curl http://localhost:3000/heapdump
$ ls
heapdump-31208326.300922.heapsnapshot
heapdump-31216569.978846.heapsnapshot

You'll note that running kill -USR2 doesn't actually kill the process. The kill command, despite its scary name, is just a tool to send signals to processes, by default a SIGTERM. With the argument -USR2, I choose to send a SIGUSR2 signal instead, which is a user-defined signal.

In last resort, you can use the signal method to generate a heapdump on the production instance. But you need to know that creating a heap snapshot requires twice the size of the heap at the time of the snapshot.

Once the snapshot is available, you can read it with the Chrome DevTools. Just open the Memory tab, right-click on the side and select Load.

Fixing the Leak

Now that I have identified what grows the memory heap, I have to find a solution. For my example, the solution is to store the logs not in memory, but on the filesystem. On a real project, it's better to delegate log storage to another service like syslog, or use an appropriate storage like a database, a Redis instance, or whatever.

Here is the modified web server with no more memory leak:

// Not the best implementation. Do not try this at home.
const fs = require('fs');
const http = require('http');

const filename = './requests.json';

const readRequests = () => {
    try {
        return fs.readFileSync(filename);
    } catch (e) {
        return '[]';
    }
};

const writeRequest = (req) => {
    const requests = JSON.parse(readRequests());
    requests.push({ url: req.url, date: new Date() });
    fs.writeFileSync(filename, JSON.stringify(requests));
};

const server = http.createServer((req, res) => {
    writeRequest(req);
    res.end(readRequests());
});

server.listen(3000);
console.log('Server listening to port 3000. Press Ctrl+C to stop it.');

Now, let's run the same test scenario as before, and measure the outcome:

$ timeout 30s siege http://localhost:3000

** SIEGE 4.0.2
** Preparing 25 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:               1931 hits
Availability:             100.00 %
Elapsed time:              30.00 secs
Data transferred:        1065.68 MB
Response time:              0.14 secs
Transaction rate:          64.37 trans/sec
Throughput:            35.52 MB/sec
Concurrency:                9.10
Successful transactions:        1931
Failed transactions:               0
Longest transaction:            0.38
Shortest transaction:           0.01

As you can see, the memory growth is far slower! This is because we no longer store the request logs in memory (inside the requestLogs variable) for each request.

This said, the API takes more time to respond: I had 89.40 transactions per second, now we have 64.37.
Reading and writing to the disk comes with a cost, so do other API calls or database requests.

Note that it's important to measure memory consumption before and after a potential fix, in order to confirm (and prove) that the memory issue is fixed.

Conclusion

Actually, fixing a memory leak once it's been identified is somewhat easy: use well known and tested libraries, don't copy or store heavy objects for too long, and so on.

The hardest part is to find them. Fortunately, and despite few bugs, the current Node.js tools are neat. And now you know how to use them!

To keep this article short and understandable, I didn't mention some other tools like the memwatch module (easy) or Core Dump analysis with llnode or mdb (advanced) but I let you with more detailed readings about them:

Configurable Artifacts: How To Deploy Like a Pro

Kmaschta — Tue, 23 Jan 2018 13:22:54 +0000

When a project team grows, feature deployments become more frequent. Automating these deployments then becomes critical to optimize the development workflow. In my opinion, the best practice is artifact-based deployment, a lifesaver process that I use as much as possible. It's quite popular, and part of the The Twelve-Factor App pattern.

This article illustrates artifact-based deployment in simple terms, through a practical example.

What Is A Deployment Artifact

Artifacts aren't a new thing, but like every good practice, it's better when written down.

A deployment artifact (or a build) is the application code as it runs on production: compiled, built, bundled, minified, optimized, and so on. Most often, it's a single binary, or a bunch of files compressed in an archive.

In such a state, you can store and version an artifact. The ultimate goal of an artifact is to be downloaded as fast as possible on a server, and run immediately, with no service interruption.

Also, an artifact should be configurable in order to be deployable on any environment. For example, if you need to deploy to staging and production servers, you should be able to use the same artifact.

Yes, you read that right: Only the configuration must change, not the artifact itself. It can seem harmful or difficult, but it's the main feature of a deployment artifact. If you have to build twice your artifact for two environments, you are missing the whole point.

The Example Project: a Basic Proxy

Let's take an example project to deploy. I'll write a basic HTTP proxy, adding a random HTTP header to every request, in Node.js.

Tips: The code is available in this gist.

First, install the dependencies:

$ mkdir deployment-example && cd $_ # cd into the directory just created
$ npm init --yes
$ npm install --save express axios reify

Here is the server, proxying all requests to the http://perdu.com backend, after adding a random header:

// server.js
import uuid from 'uuid';
import axios from 'axios';
import express from 'express';

const port = process.env.NODE_PORT || 3000;
const baseURL = process.env.PROXY_HOST || 'http://perdu.com/';

const client = axios.create({ baseURL });
const app = express();

app.get('/', (req, res) => {
    const requestOptions = {
        url: req.path,
        method: req.method.toLowerCase(),
        headers: { 'X-Random': uuid.v4() },
    };

    return client(requestOptions).then(response => res.send(response.data));
});

app.listen(port);
console.log(`Proxy is running on port ${port}. Press Ctrl+C to stop.`);

Tips: Did you notice how I used environment variables via process.env? It's central to my point, keep it in mind for later.

A small makefile to run the server:

start:
    # reify is a small lib to fully support import/export without having to install the babel suite
    node --require reify server.js

The server is runnable on local environment, it's all good, "it works on my machine™":

$ make start
> Proxy is running on port 3000. Press Ctrl+C to stop.

Building an Artifact

Now let's ship this code to a staging environment, in order to test it in somewhat real conditions.

At this point, it's a good idea to freeze a version of the code (with a Git tag or a GitHub release for instance).

The simplest way to prepare the deployment is to build a zip file with the source code and all its dependencies. I'll add the following target to the makefile, creating a zip with the identifier of the latest commit:

build:
    mkdir -p build
    zip 'build/$(shell git log -1 --pretty="%h").zip' Makefile package.json -R . '*.js' -R . '*.json'

And it works:

$ make build
$ ls build/
> 4dd370f.zip

The build step is simplified here, but on real projects it can imply a bundler, a transpiler, a minifier, and so on.
All these lengthy tasks should be done at the build step.

The resulting zip file is what we can call an artifact. It can be deployed on an external server, or be stored in a S3 bucket for later usage.

Tips: Once you find the build process that fits you, automate it! Usually, a Continuous Integration / Continuous Delivery (CI/CD) system like Travis or Jenkins runs the tests, and if they pass, build the artifact in order to store it.

Deploying the Artifact

To deploy the artifact, just copy this file on the server, extract it, and run the code. I automate it again as a makefile target:

TAG ?=
SERVER ?= proxy-staging

deploy:
    scp build/$(TAG).zip $(SERVER):/data/www/deployment-example/
    ssh $(SERVER) " \
        cd /data/www/deployment-example/ \
        unzip $(TAG).zip -d $(TAG)/ && rm $(TAG).zip \      # unzip the code in a folder
        cd current/ && make stop \                          # stop the current server
        cd ../ && rm current/ && ln -s $(TAG)/ current/ \   # move the symbolic link to the new version
        cd current/ && make start \                         # restart the server
    "
    echo 'Deployed $(TAG) version to $(SERVER)'

I use environment variables to specify the tag I want to deploy:

$ TAG=4dd370f make deploy
> Deployed 4dd370f version to proxy-staging

As you can see, the deployment is actually very fast, because I don't need to build in the target environment.

Tips: That means that if the build contains binaries, they must be compiled for the target environment. To simplify, it means you should develop your code on the same system as you run it. It's simpler that it sounds once you use Vagrant or Docker.

Tips: I used the ssh command without any credential arguments. You don't want those credentials in a makefile, so I advise to use your local ~/.ssh/config to save them. This way, you can securely share them with your co-workers, and keep these credentials outside of the repository - while having the deployment instructions in the the Makefile. Here is an exampe SSH configuration for my proxy-staging:

Host proxy-staging
    Hostname staging.domain.me
    User ubuntu
    IdentityFile ~/.ssh/keys/staging.pem

Deploying to Several Environments

The server can now run in the proxy-staging server, with the default configuration. What if I want to deploy the same build to a proxy-production server, but with a different port?

All I need is to be sure that wherever the code runs, the NODE_PORT environment variable is set to the correct value.

There are many ways to achieve this configuration.

The simplest one is to directly write the values of the environment variables in a ~/.ssh/environment file in each server. That way, I don't need to remember how or when to retrieve the configuration: it's loaded automatically each time I log into the machine.

> ssh production-server "echo 'NODE_PORT=8000' >> ~/.ssh/environment"
> ssh production-server "echo 'PROXY_HOST=https://host.to.proxy' >> ~/.ssh/environment"
> ssh production-server
> env
NODE_PORT=8000
PROXY_HOST=https://host.to.proxy

Now I can deploy to proxy-production the same artifact that I've used for proxy-staging, no need to rebuild.

$ TAG=4dd370f SERVER=proxy-production make deploy
> Deployed 4dd370f version to proxy-production

That's it.

Tips: The rollback is as simple as a deployment of the previous artifact.

At this point, it is easy to automate the process: let the CI build an artifact each time a PR is merged, or on every push to master (Travis, Jenkins, and every other CI allow to implement a build phase). Then, store this build somewhere with a specific tag, such as the commit hash or the release tag.

When somebody wants to deploy, they can run a script on the server that downloads the artifact, configures it thanks to the environment variables, and runs it.

Tips: If you don't want to write your environment variables on every production server you have, you can use a configuration manager like comfygure. Disclaimer: We wrote it!

When You Should Use Artifact-Based Deployment

Artifact-based deployment comes with many advantages. It is quick to deploy, instant to rollback, the backups are available and ready to run.

It allows to run the exact same code on every environment. In fact, with artifacts, the deployment becomes a configuration issue. If a bunch of code works on staging, there is no reason that it should fail on production, unless there is a mistake in the configuration - or the environments aren't the same. For that reason, it's a good idea to invest in a staging environment that strictly equals the production environment.

Feature flagging is also easier with such a deployment process: just check the environment variables. And last but not least, the deployment can be automated to such an extent that it can be done by a non-technical person (like a product owner). We do it, and our customers love it.

But automating such a process comes with a substantial cost. It takes time to install and maintain, because when the build process involves more than just zipping a few files, you need to run it with a bundler like Webpack. Also, consider the extra disk space necessary to store the artifacts and the backups.

Tips: Use environment variables, do not check the environment. Try to find instances of if (env !== 'production') in your code, and replace them with a significant environment variable, like if (process.env.LOGGING_LEVEL === 'verbose').

No need of such a deployment process for a proof-of-concept, an early project, or a solo developer. All the advantages come with a team working on a mature project.

Ask yourself how many time you spend on deployment, and take a look at this chart!

Frequently Asked Questions

My project is an SPA / PWA. I must configure my bundle at the build phase!

You don't have to. There is many ways to configure a Single Page Application later in the process. For instance, load a config.json at runtime from a different location than the other static assets. Or, write a window.__CONFIG__ with server side rendering.

Don't be afraid to display the frontend config, it'll be easier to debug. If you have sensible informations in your frontend configuration hidden in your minified and cryptic webpack build, you're already doing it wrong.

What about database migrations?

Migrations don't have to run when the application deploys. They should have their own pipeline, that can be triggered at the appropriate moment. This way, you can handle each deployment on its own, and rollback any of them independently. It clearly implies to do backups a lot, and to have revertable migrations.

In case of big or painful migration, don't hesitate to do it step by step. For example, how to move a table: first create the new table, then copy the data, and finally delete the resulting table in three different migrations. The application can be deployed between the copy and the table deletion. If something goes wrong, the application can be reverted quickly without touching the database.

Conclusion

Once this process is correctly installed, deployments become quicker and safer. It allows teams and product owners to be more confident about deployments.

Artifact-based deployment is a powerful technique that is truly worth considering when a project gets close to its maturity.
It matches perfectly with agile methodologies like SCRUM, even more with Kanban, and it's a prerequisite to continuous delivery.

To go further with deployment techniques, I recommend huge article of Nick Craver: Stack Overflow: How We Do Deployment - 2016 Edition.