DEV Community: Vasyl Boroviak

lil-http-terminator, a tiny JS module to gracefully shutdown your HTTP server

Vasyl Boroviak — Sun, 11 Jul 2021 10:49:42 +0000

Increase your node_modules by 11 KB to get tranquility that your node.js HTTP server is shutting down without data loss risk.

Or how I shrunk a 2.2 MB module to 11 KB.

TL;DR: npm i lil-http-terminator

The problem

I've been coding node.js microservices for almost a decade now. Graceful HTTP server shutdown was always a problem I didn't wanna deal with because it's hard to get right.

The solution

`http-terminator` npm module

Recently I discovered that there is a perfect implementation of graceful shutdown. It's called http-terminator. Here is why I decided to use it (quoting the author):

it does not monkey-patch Node.js API

it immediately destroys all sockets without an attached HTTP request

it allows graceful timeout to sockets with ongoing HTTP requests

it properly handles HTTPS connections

it informs connections using keep-alive that server is shutting down by setting a connection: close header

it does not terminate the Node.js process

Usage:

import { createHttpTerminator } from 'http-terminator';
const httpTerminator = createHttpTerminator({ server });
await httpTerminator.terminate();

Works with any node.js HTTP server out there (Express.js, Nest.js, Polka.js, Koa.js, Meteor.js, Sails.js, Hapi.js, etc).

Wow! Brilliant engineering! Well done author(s)!

But there is a catch.

Being a mere 4 KB codebase it adds 22 dependencies (2.2 MB, 464 files) to your node_modules.

See for yourself:

$ npx howfat -r tree http-terminator
npx: installed 18 in 1.695s

http-terminator@3.0.0 (22 deps, 2.16mb, 464 files)
├── delay@5.0.0 (10.91kb, 5 files)
├─┬ roarr@4.2.5 (19 deps, 2.02mb, 398 files)
│ ├── boolean@3.1.2 (7.9kb, 10 files)
│ ├── detect-node@2.1.0 (2.7kb, 6 files)
│ ├─┬ fast-json-stringify@2.7.7 (9 deps, 1.79mb, 268 files)
│ │ ├─┬ ajv@6.12.6 (5 deps, 1.41mb, 181 files)
│ │ │ ├── fast-deep-equal@3.1.3 (12.66kb, 11 files)
│ │ │ ├── fast-json-stable-stringify@2.1.0 (16.56kb, 18 files)
│ │ │ ├── json-schema-traverse@0.4.1 (19.11kb, 9 files)
│ │ │ ╰─┬ uri-js@4.4.1 (1 dep, 490.54kb, 51 files)
│ │ │   ╰── punycode@2.1.1 (31.67kb, 5 files)
│ │ ├── deepmerge@4.2.2 (29.39kb, 9 files)
│ │ ├── rfdc@1.3.0 (23.48kb, 9 files)
│ │ ╰── string-similarity@4.0.4 (10.73kb, 5 files)
│ ├─┬ fast-printf@1.6.6 (1 dep, 34.32kb, 26 files)
│ │ ╰── boolean@3.1.2 (🔗, 7.9kb, 10 files)
│ ├─┬ globalthis@1.0.2 (2 deps, 114.41kb, 41 files)
│ │ ╰─┬ define-properties@1.1.3 (1 dep, 48.41kb, 21 files)
│ │   ╰── object-keys@1.1.1 (25.92kb, 11 files)
│ ├── is-circular@1.0.2 (5.89kb, 8 files)
│ ├── json-stringify-safe@5.0.1 (12.42kb, 9 files)
│ ╰── semver-compare@1.0.0 (3.96kb, 8 files)
╰── type-fest@0.20.2 (108kb, 42 files)

I got curious. What's that roarr package and if it can be removed from the package? The answer got me by surprise.

Removing the unnecessary dependencies

The three top level dependencies can be easily removed.

`type-fest`

The type-fest can be removed by rewriting the package from TS to JS. Hold saying your "boo" yet.

It's a single function module. You don't need the code completion for just one function. So, rewriting to JS shouldn't be a downside for TypeScript proponents.

`delay`

The delay module can be rewritten as a single-line function. Here it is:

const delay = time => new Promise(r => setTimeout(r, time));

`roarr`

roarr module, the largest of the tree, takes 2 MB of your hard drive. But it is used literally in the single line!!!

if (terminating) {
  log.warn('already terminating HTTP server');

  return terminating;
}

The module will print that warning in case you decide to terminate your HTTP server twice. That's all. There is no more usage of the roarr logger within the whole http-terminator module.

I find it nearly impossible to accidentally call .termiate() twice. It's hard to imagine this ever happens. So I decided to put the log variable to options and assign it to console by default.

We get rid of 20 dependencies and simultaneously allow you, my fellow developers, to customise the termination with the logger of your choice (winston, bunyan, pino, morgan, etc; or even the roarr itself).

Meet `lil-http-terminator`

I forked the http-terminator to lil-http-terminator.

const HttpTerminator = require("lil-http-terminator");
const httpTerminator = HttpTerminator({ server });
await httpTerminator.terminate();

Being as awesome as the origin, the lil- version is:

0 dependencies (original had 3 direct and 18 indirect sub-dependencies);
only 5 files (original was 464 files total);
only 11 KB (original was 2180 KB);
packaged by NPM as 3.9 KB .tar.gz file (original downloads about 522 KB).
takes much less memory (I didn't measure it though);
has 8 devDependencies (original has 17);

Afterword

I'm writing code for money for about 20 years. I'm using node.js and npm for almost a decade. I learnt to develop good and robust node.js services, scripts, serverless functions, apps. I discovered (re-invented) the best practices we better follow. I know how to make code maintainable years after it was written. The hardest bit was always the third party dependencies. I learnt the hard way that each additional sub-dependency can cost a company some thousands of dollars.

I forked and wrote lil-http-terminator in two hours. I foresee saving myself from 8 to 80 hours this way. You can save the same.

lil-csv, a 1k JS module to parse and generate CSV files

Vasyl Boroviak — Sat, 10 Jul 2021 12:43:06 +0000

I was struggling to find a small JavaScript module to parse CSV (Comma Separated Values) files. All the existing modules have one or more shortcomings:

Do not work in browsers;
Large (all I found are at least 10kb min.js.gz).
Cannot parse to deep objects.
Cannot generate CSV from deep objects.

I'm pretty sure CSV parsing can be implemented in less than 1Kb min.js.gz. So I did it.

Please meet the powerful lil-csv.

That's how large it is (v1.3.1):

       1465 B: lil-csv.js.gz
       1313 B: lil-csv.js.br
       1315 B: lil-csv.modern.js.gz
       1201 B: lil-csv.modern.js.br
       1480 B: lil-csv.module.js.gz
       1327 B: lil-csv.module.js.br
       1524 B: lil-csv.umd.js.gz
       1359 B: lil-csv.umd.js.br

There are tradeoffs though.

It doesn't accept streams but only the full CSV file content as a single string. (Also remember: Premature optimisation is the root of all evil.)
...you tell me...

Simple example

CSV to JavaScript objects

Let's say you have a CSV like this:

name,address,course
John Smith,"123 John St, CARLTON",Advanced Calculus
Any Newman,"4a/3a Church Ave, CROYDON",Advanced Calculus

Parse it to objects:

import { parse } from "lil-csv";
const objects = parse(fileContents);
console.log(objects);

/*
[
  {
    "name": "John Smith",
    "address": "123 John St, CARLTON",
    "course": "Advanced Calculus"
  },
  {
    "name": "Any Newman",
    "address": "4a/3a Church Ave, CROYDON",
    "course": "Advanced Calculus"
  }
]
*/

And stringify them back to CSV string:

import { generate } from "lil-csv";
const string = generate(objects);
console.log(string);

/*
name,address,course
John Smith,"123 John St, CARLTON",Advanced Calculus
Any Newman,"4a/3a Church Ave, CROYDON",Advanced Calculus
*/

So, in essence stringifying plus parsing is an idempotent operation:

assert.deepEqual(objects, parse(generate(objects)));

CSV to JavaScript array of arrays

If you just need arrays of strings (not objects) then here is how you do it:

const arrays = parse(fileContents, { header: false });
console.log(arrays);

/*
[
  ["name","address","course"],
  ["John Smith","123 John St, CARLTON","Advanced Calculus"],
  ["Any Newman","4a/3a Church Ave, CROYDON","Advanced Calculus"]
]
*/

Stringyfing back to CSV is simple:

const string = generate(arrays, { header: false });
console.log(string);

/*
name,address,course
John Smith,"123 John St, CARLTON",Advanced Calculus
Any Newman,"4a/3a Church Ave, CROYDON",Advanced Calculus
*/

Complex example

Parsing numbers, dates, booleans

In real world the data is rarely all strings. Often your objects have to have numbers, dates, booleans, etc. Here is how to parse CSV with all kind of data.

Let's parse some strings, dates, numbers and booleans from the following CSV file:

firstName,lastName,dob,price,completed
John,Smith,1999-01-15,123.00,Y
Alice,Dwarf,1991-11-24,123.00,N

Converting custom string to the JS objects, and leaving all other data as strings:

const people = parse(fileContents, {
  header: {
    "*": String,
    dob: v => v ? new Date(v) : null,
    price: v => isNaN(v) ? null : Number(v),
    completed: v => String(v).toUpperCase() === "Y",
  }
});

console.log(people);

/*
[
  {
    "firstName": "John",
    "lastName": "Smith",
    "dob": "1999-01-15T00:00:00.000Z",
    "price": 123.00,
    "completed": true
  },
  {
    "firstName": "Alice",
    "lastName": "Dwarf",
    "dob": "1991-11-24T00:00:00.000Z",
    "price": 123.00,
    "completed": false
  }
]
*/

Generating custom CSV

Here is how you can convert booleans to strings like "Y" and "N", and also convert JS Date to calendar dates like "YYYY-MM-DD", and add custom formatting to numbers like "123.00" instead of the default "123":

const string = generate(people, {
  header: {
    "*": String,
    dob: v => v ? new Date(v).toISOString().substr(0, 10) : "",
    price: v => isNaN(v) ? "" : Number(v).toFixed(2),
    completed: v => v ? "Y" : "N",
  }
});

console.log(string);

/*
firstName,lastName,dob,price,completed
John,Smith,1999-01-15,123.55,Y
Alice,Dwarf,1991-11-24,123.55,N
*/

Renaming column headers

Converting CSV column headers to JS property names

Of course people rarely use JavaScript property names for column headers. You would likely see "Date of birth" in CSV file header instead of "dob". The lil-csv is little but powerful. It can can handle that too.

That's how you can rename headers during CSV file parsing and CSV file generation.

You file:

First name,Last name,Date of birth,Price in dollars,Completed
John,Smith,1999-01-15,123.00,Y
Alice,Dwarf,1991-11-24,123.00,N

Renaming each column to a JS object property:

const people = parse(fileContents, {
  header: {
    "First name": "firstName",
    "Last name": "lastName",
    "Date of birth": {
      newName: "dob",
      parse: v => v ? new Date(v) : null,
    },
    "Price in dollars": {
      newName: "price",
      parse: v => isNaN(v) ? null : Number(v),
    },
    Completed: {
      newName: "completed",
      parse: v => String(v).toUpperCase() === "Y",
    },
  }
});

console.log(people);

/*
[
  {
    "firstName": "John",
    "lastName": "Smith",
    "dob": "1999-01-15T00:00:00.000Z",
    "price": 123.00,
    "completed": true
  },
  {
    "firstName": "Alice",
    "lastName": "Dwarf",
    "dob": "1991-11-24T00:00:00.000Z",
    "price": 123.00,
    "completed": false
  }
]
*/

Renaming JS properties to real world column headers

I hope this code is easy to read:

const string = generate(people, {
  header: {
    firstName: "First name",
    lastName: "Last name",
    dob: {
      newName: "Date of birth",
      stringify: v => v ? new Date(v).toISOString().substr(0, 10) : "",
    },
    price: {
      newName: "Price in dollars",
      stringify: v => isNaN(v) ? "" : Number(v).toFixed(2),
    },
    completed: {
      newName: "Completed",
      stringify: v => v ? "Y" : "N",
    },
  }
});

console.log(string);

/*
First name,Last name,Date of birth,Price in dollars,Completed
John,Smith,1999-01-15,123.00,Y
Alice,Dwarf,1991-11-24,123.00,N
*/

Deep objects support!

Here comes the true powers of lil-csv. You can parse CSV rows directly to deep objects like:

{
   order_id: 51234,
   recipient: {
     firstName: "John",
     lastName: "Smith",
     dob: "1999-01-15T00:00:00.000Z",
     address: {
       street: "123 John St, CARLTON",
       country: "AU",
     }
   }
}

Parsing CSV rows as JS deep objects

Let's parse this CSV to the above object:

ID,First name,Last name,Date of birth,Address,Country
51234,John,Smith,1999-01-15,"123 John St, CARLTON",AU

All you need is to rename headers with dot notation:

const orders = parse(fileContents, {
  header: {
    ID: {
      parse: Number,
      newName: "order_id",
    },
    "First name": "recipient.firstName",
    "Last name": "recipient.lastName",
    "Date of birth": {
      newName: "recipient.dob",
      parse: v => v ? new Date(v) : null,
    },
    Address: "recipient.address.street",
    Country: "recipient.address.country",
  }
});

console.log(orders);

It works similar when generating a CSV file from deep data:

const string = generate(orders, {
  header: {
    order_id: "ID",
    "recipient.firstName": "First name",
    "recipient.lastName": "Last name",
    "recipient.dob": {
      newName: "Date of birth",
      stringify: v => v ? new Date(v).toISOString().substr(0, 10) : "",
    },
    "recipient.address.street": "Address",
    "recipient.address.country": "Country",
  }
});

console.log(string);

/*
ID,First name,Last name,Date of birth,Address,Country
51234,John,Smith,1999-01-15,"123 John St, CARLTON",AU
*/

In the above code the "Date of birth" column gets:

deeply set/accessed within an object.
parsed/stringified in a custom way;
renamed;

Afterword

You get all that power from 1 TCP packet, meaning less than 1460 bytes. Or even fewer bytes if you are using only one of the two functions (treeshaking is supported by lil-csv).

If you need additional features from lil-csv then feel free open an issue here: https://github.com/flash-oss/lil-csv/issues

Not using proxyquire or sinon for unit testing

Vasyl Boroviak — Fri, 11 Sep 2020 15:14:43 +0000

Maintainable, readable, DRY JavaScript unit tests without "magical" sinon, proxyquire, jest, etc.

Disclaimer

This post inspired by @samueldjones article called

Using proxyquire and sinon for unit testing in Node.js

Sam Jones ・ Aug 10 '20

#javascript #testing #node

Thanks Sam!

Intro

Let me start by saying that, as a professional, I found testing to be pretty simple and easy. I created a number of apps that made millions for my employers and all with the help of test coverage! However, having spent the last years working across the full stack and writing a lot of unit and integration tests in the process, I feel urge to share my vast experience. While it does require a slight change of perspective, testing should feel like taking a gentle stroll through the countryside away from the hustle and bustle of business logic.

Practical

Diving straight into our scenario then. We simply want to test our function that fetches all available currency rates from this URL - https://currencylayer.com/, then stores all of them to MongoDB database for historical purposes and caches latest of them to Redis database, but only if your node.js process have access to the databases. A use case for this API could potentially be a serverless function that runs daily (hourly?) to fetch all exchange rates for that day in order to show a historical chart of currency conversion rates and have latest exchange rate cached in an in-memory database (Redis).

"Wow!", you might think, "This is a lot to mock!". Indeed a lot. I intentionally made Sam's original example much more complex to get closer to real world scenarios.

Let's pause here to review the packages that we will be using:

stampit: flexible object factories

That's all! Nothing else. No other dependencies need. Yay!!!

Please note, we are not using proxyquire and sinon (and even chai). Why? Because years of my experience drove we away from the approaches these libraries take. They incur too much JavaScript "magic". The more "magic" your code has, the less maintainable it is, the more problems you'll have upgrading your node_modules. These problems cumulative effect accounted for up to 20% of my work time.

At the end I came to conclusion that "magic" is bad and the more explicit your code - the better.

My general recommendations for maintainable node.js code:

Less magic, more explicit.
Fewer dependencies.
More simplicity.
Less code. And less auto generated code.

Our main code

// ExRateFetcher.js

const CURRENCY_API_URL = "https://api.currencylayer.com";
const ACCESS_KEY = process.env.ACCESS_KEY;

module.exports = require("stampit")({
  name: "ExRateFetcher",

  props: {
    fetch: global.fetch,

    mongoose: require("mongoose"),
    CurrencyPairModel: null,

    redis: require("redis"),
    redisClient: null,
  },

  init() {
    const client = this.redis.createClient(process.env.REDIS_URL);
    client.on('ready', () => {
      this.redisClient = client;
    });

    this.mongoose.connect(process.env.MONGO_URL)
      .then(() => {
        const CurrencyPairSchema = new this.mongoose.Schema({
          _id: String, // currency pair as primary key
          rates: [{ date: String, rate: Number }]
        });
        this.CurrencyPairModel = this.mongoose.model(
          'CurrencyPair',
          CurrencyPairSchema
        );
      });
  },

  methods: {
    async _saveToMongo(rates, date) {
      const date = date.toISOString().substr(0, 10);
      for (const [pair, rate] of rates) {
        await this.CurrencyPairModel.upsert(
          { _id: pair, "rates.date": date },
          { $set: { rate } }
        );
      }
    },

    async _saveToRedis(rates) {
      for (const [pair, rate] of rates) {
        await this.redisClient.set(pair, rate);
      }
    },

    async fetchAndStoreLatest() {
      const responseBody = await this.fetch(`${CURRENCY_API_URL}/live?access_key=${ACCESS_KEY}`);
      const date = new Date(responseBody.timestamp * 1000);
      const rates = Object.entries(responseBody.quotes);

      if (this.CurrencyPairModel) {
        await this._saveToMongo(rates, date);
      }
      if (this.redisClient) {
        await this._saveToRedis(rates);
      }
    }
  }
});

Here is the usage of ExRateFetcher.js:

const ExRateFetcher = require("./ExRateFetcher.js");
ExRateFetcher().fetchAndStoreLatest();

I/O dependencies

Some APIs might be a giant Java Spring server. Some APIs might be too dangerous to call (e.g. some of the AWS APIs). Some APIs can be too expensive (e.g. Authy). Some databases can't be easily rolled out for unit testing purposes (e.g. Kafka). Some I/O can be a third party gRPC, UDP, or WebSocket server. You can't use any of these on a CI or your laptop.

In real world the third party APIs and databases you connect to might not be available in your CI/CD environment. In my experience, about half of the I/O dependencies (APIs, DBs, etc) are typically impossible to have for unit testing purposes. Thus...

INABILITY TO MOCK IS A CODE SMELL in node.js.

Our unit test

const assert = require("assert");
const { describe, it } = require("node:test");

// Let's stub two database dependencies with no-op code.
const ExRateFetcher = require("./ExRateFetcher").props({
  // Attention! Mocking redis!
  redis: { createClient: () => ({ on() {} }) },
  // Attention! Mocking mongoose!
  mongoose: { connect: () => ({ then() {} }) },
});

describe("ExRateFetcher", () => {
  describe("#fetchAndStoreLatest", () => {
    it("should fetch", (done) => {
      const MockedFetcher = ExRateFetcher.props({
        // Attention! Mocking global fetch!
        async fetch(uri) {
          assert(uri.includes("/live?access_key="));
          done();
        }
      });

      MockedFetcher().fetchAndStoreLatest();
    });

    const responseBody = {
      "timestamp": 1432400348,
      "quotes": {
        "USDEUR": 1.278342,
        "USDGBP": 0.908019,
      }
    };

    it("should store in Redis", () => {
      let redisSetCalled = 0;
      const MockedFetcher = ExRateFetcher.props({
        fetch: async () => responseBody,

        // Attention! Mocking redis!
        redis: {
          createClient() {
            return {
              on(event, callback) {
                assert(event === "ready");
                assert(typeof callback === "function");
                setTimeout(callback, 0);
              },
              async set(key, value) { // DB call mocking
                assert(responseBody.quotes[key] === value);
                redisSetCalled += 1;
              }
            };
          }
        },
      });

      const fetcher = MockedFetcher();
      await new Promise(r => setTimeout(r, 1)); // wait connection
      await fetcher.fetchAndStoreLatest();

      assert(redisSetCalled === 2);
    });

    it("should store in MongoDB", () => {
      let mongoUpsertCalled = 0;
      const MockedFetcher = ExRateFetcher.props({
        fetch: async () => responseBody,

        // Attention! Mocking mongoose!
        mongoose: {
          connect() {
            return {
              then(callback) {
                assert(typeof callback === "function");
                setTimeout(callback, 0);
              }
            };
          },
          Schema: function () {},
          model: () => ({
            async upsert(query, command) { // DB call mocking
              assert(command.$set.rate === responseBody.quotes[query._id]);
              assert(query["rates.date"] === "2015-05-23");
              mongoUpsertCalled += 1;
            }
          }),
        },
      });

      const fetcher = MockedFetcher();
      await new Promise(r => setTimeout(r, 1)); // wait connection
      await fetcher.fetchAndStoreLatest();

      assert(mongoUpsertCalled === 2);
    });
  });
});

When I see sinon in a codebase, there is usually a lot of repetitive mocking happening.

Test 1 - mock A, mock B, mock C
Test 2 - mock A, mock B, mock C
Test 3 - mock A, mock B, mock C

Whereas, in the unit test code above we mock only the bare minimum. We do not over mock things. Also, you don't need to go to sinon docs over and over again to remember what is the syntax for, say, event emitter or mongoose model object.

In my experience, the code above is very much stable and quite flexible to mock literally anything.

I often mock setTimeout or other JavaScript/Node.js globals. Whereas, mocking globals in node.js is so very much error prone and unstable if using proxyquire, sinon, jest, etc. Using the approach above you could mock setTimeout only in that specific test and nowhere else. This trick alone saved me days over the years.

See more about stampit module here: https://stampit.js.org/

Serverless of the future

Vasyl Boroviak — Sun, 14 Jul 2019 23:49:05 +0000

Serverless, oh so much in this word.

I made this prediction 5 years ago:

Serverless is a great idea. But all the current implementations won't gonna win.

But let's talk about Docker first.

The overnight success of Docker happened because it was open, versatile, and vendor independent. I could deploy the same Dockerfile in premises, my local computer, or a cloud without code change.

But then AWS Lambda was marketed as a Docker killer. And people bought it.

The idea of serverless is great, but I can't run Google cloud function in AWS or vice versa.

Similar, I can't reliably run any of them on my laptop just like that, without any modifications.

Your serverless functions needs a Dockerfile of its own, say Functionfile.

Is there a Functionfile initiative to run serverless functions on ANY platform? I'd like to join. If not, then we have to create one.

DEV Community: Vasyl Boroviak

lil-http-terminator, a tiny JS module to gracefully shutdown your HTTP server

The problem

The solution

http-terminator npm module

Removing the unnecessary dependencies

type-fest

delay

roarr

Meet lil-http-terminator

Afterword

lil-csv, a 1k JS module to parse and generate CSV files

Simple example

CSV to JavaScript objects

CSV to JavaScript array of arrays

Complex example

Parsing numbers, dates, booleans

Generating custom CSV

Renaming column headers

Converting CSV column headers to JS property names

Renaming JS properties to real world column headers

Deep objects support!

Parsing CSV rows as JS deep objects

Afterword

Not using proxyquire or sinon for unit testing

Disclaimer

Using proxyquire and sinon for unit testing in Node.js

Sam Jones ・ Aug 10 '20

Intro

Practical

Our main code

I/O dependencies

Our unit test

Serverless of the future

`http-terminator` npm module

`type-fest`

`delay`

`roarr`

Meet `lil-http-terminator`