A short tour of ExpressJS implementation (Part 1)

Introduction

ExpressJS is a web framework that most of us know about. I really like how easy and fast it is to set up a new server using Express.

Under the hood, Express is a lightweight wrapper around the Node HTTP module. It abstracts out some complexity of the HTTP module and provides us with a simpler way to build our backend server.

In this article, I want to understand how Express is implemented and see if I can learn something from that.

Start reading the codes

The source code of ExpressJS is available at: https://github.com/expressjs/express

First, let's check the directory structure:

├── index.js
├── lib
│   ├── application.js
│   ├── express.js
│   ├── middleware
│   │   ├── init.js
│   │   └── query.js
│   ├── request.js
│   ├── response.js
│   ├── router
│   │   ├── index.js
│   │   ├── layer.js
│   │   └── route.js
│   ├── utils.js
│   └── view.js

The main codes live in the lib folder, with a few modules such as: application, middleware, request, response, router… It is indeed simple, there are only a few files compared to other web frameworks.

Now, we can jump right into the express.js and read the codes. But I want to approach the reading process in a different way.

Let’s say that we have this tiny application that uses Express.

const express = require('express')
const myapp = express()
const port = 3000

myapp.use(whateverLogger())

myapp.use('/hello', (req, res) => {
  res.send('Hello World!')
})

myapp.post('/world', (req, res) => {
  res.send('ok')
})

myapp.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

We can separate this application into 2 phases:

• Phase 1: Registering the handlers
  • In this phase, the handlers (middleware) have been set as follows: whateverLogger, and the handlers for /hello and /world.
• Phase 2: Processing the requests
  • This phase is where the server receives the requests and processes them

Let’s start reading the source codes following each phase.

Phase 1: Registering handlers

The main method that we need attention to are the two:

• myapp = express()
• myapp.use()

For the express() my first guess is to read the express.js

express.js

...
var proto = require('./application');
var Route = require('./router/route');
var Router = require('./router');
...

exports = module.exports = createApplication;

function createApplication() {
  var app = function(req, res, next) {
    app.handle(req, res, next);
  };
  mixin(app, EventEmitter.prototype, false);
  mixin(app, proto, false);
  ...

  return app
}

(omitted other details for clarity)

The function call returns an app which is also a function. The mixin(app, proto, false) indicates that most of the logic of this app will reside in the ./application

We can skip the mixin logic and assume that its purpose is to add methods and properties to the app

Our next hop is the application.js

application.js

At line 194, we can see the app.use is defined:

...
var Router = require('./router');
...

app.use = function use(fn) {
  ...
  var router = this._router;
  var fns = flatten(slice.call(arguments, offset));
  ...

  fns.forEach(function (fn) {
    if (!fn || !fn.handle || !fn.set) {
      return router.use(path, fn);
    }

    ...
  }, this);

  return this;
};

There are many activities in this method but the first thing that should catch our eyes is the fns.forEach that calls the router.use(path, fn)

app.use receives a list of functions (that are our handlers for a path). Then register each of those functions using the router.

Now let’s jump to the router.js

router.js

A lot of interesting things happen in this file.

First, the constructor codes show a stack is defined.

Whenever I see a stack (or queue), I immediately think of an array that has some sort of order when we insert or access the array.

var proto = module.exports = function(options) {
  ...
  router.stack = [];

  return router;
};

Continue, we find the definition of router.use

The sweet spot is this for loop where it creates a new Layer and pushes it to the stack

proto.use = function use(fn) {
  ...
  var callbacks = flatten(slice.call(arguments, offset));
  ...

  for (var i = 0; i < callbacks.length; i++) {
    var fn = callbacks[I];
    ...
    var layer = new Layer(path, {
      sensitive: this.caseSensitive,
      strict: false,
      end: false
    }, fn);

    layer.route = undefined;

    this.stack.push(layer);
  }

  return this;
};

• Each item on the stack is a Layer
• A Layer can only receive a path and a fn callback

Now at this point, I can imagine how this setup is stored in the stack:

myapp.use('/hello', handler1, handler2)
myapp.use('/world', handler3, handler4)

this will become:

router.stack = [
  Layer('/world', handler4),
  Layer('/world', handler3),
  Layer('/hello', handler2),
  Layer('/hello', handler1) // <-- index: 0
]

I believe we now have a basic understanding of how the registration process works, and what is the data structure behind the scence

Before moving on to Phase 2, here's a diagram to summarize everything.

https://www.codediagram.io/app/shares?token=5dce1a36

On Phase 2: Processing the requests, we’ll focus on how a request is processed through the middleware by looking into the implementation of the next() function

Please stay tune 😄