Quest for a practical NodeJS API Framework

While working on REST APIs with Express JS / Node, I came across some common challenges:

1. I had to configure Express the same way every time using some common stuff - body parser, cors, validators etc.
2. It's a free-for-all in terms of how you organize your code when it comes to Express, best practices in terms of coding and code organization had to be implemented manually every time.
3. There is built in central error handling in Express, but it's not declarative, and you'd want a more elegant solution as well as have more control over errors on specific endpoints.
4. Documenting APIs seemed like a chore since you needed to setup swagger, and maintain a separate swagger JSON file which is kind of removed from your actual code.

Working with Express for the last 3 years, I started thinking about how to use ES6 classes and OOP to implement REST controllers, about common patterns and ways I can modularize the APIs I develop. Each REST endpoint has a URL, an HTTP method, and a function it carries out. Any REST controller would be for handling requests for a given endpoint e.g. "/products", with different functions for handling the different HTTP methods. So I began my quest, and my first iteration of a would-be framework was to have a BaseController that each REST controller would extend. Here is a snippet of such a REST controller.

Let’s say the ProductController manages actions regarding products of an e-commerce site. A productService is injected through the constructor, and a method is implemented for getting a single product based on its ID.

Pay attention to two parts of the code:

1. The API validation is done at the beginning of the method, where it returns out of the method if there are errors. This was done using express-validator.
2. Whatever error is thrown inside the method is caught and sent to the next middleware using the "next" function, to avoid the general "internal server error" response. We will need to remember these two points later in the article.

You may notice that the controller has a property “router”, which is actually an Express router, and the “route_” prefix convention is used to register any endpoint for that router, both of which are implemented code in the BaseController. So now if you wanted to have an API for products, you could do this:

Recently I’ve been working on a project where we have several microservices using Java Spring Boot, and it was interesting to see Spring Boot has a similar convention as I derived here. Not to brag, but I mean it's common sense, right?

Then I thought, what happens if you bring nested APIs into the mix? Let’s say each product has reviews associated with it, and we need a set of endpoints to go with that. How would we organize our code then? We would have a separate controller for reviews with their CRUD functions, but would the URI “/:id/reviews” be repeated for each one? Here’s what I wanted the ReviewController to look like.

This controller allows endpoints to create a review, get all reviews, and get a single review by ID. How would we define the paths for these endpoints? Then came the idea of “subroutes”, where I implemented a mechanism to define nested routes within a controller itself. Just like the “routes_” prefix convention, I added a ‘subroutes’ method in the BaseController which you would extend in your own controller and return an array of controllers to be registered in the Express router internally. Here is an example of how to use the 'subroutes' method.

Let’s say I declare the “/products” path from where my main app is routed like before, and with this particular example what we have done is declare the following APIs:

• GET /products
• POST /products/:id/review
• GET /products/:id/review
• GET /products/:id/review

Okay great, now there was a way to do nested APIs, and we could keep declaring controllers for root level routes or subroutes for nested paths, and the framework handled registering the paths. However, this became kind of a mess, or so I figured after taking a step back. Here is all that was wrong with the ‘framework’ so far (as far as I know):

1. The code was very verbose
2. Different controller classes would be tied to each other
3. You would have to pass around services to controllers that had no business with them specifically
4. Remember how I mentioned the API validation and error handling? As I wrote out a few more APIs, I realized I would have to repeat those two lines in every single controller method. No can do.

This was a mess. It broke the Single Responsibility Principle, and probably a few others that don't want to think about anymore. It’s not the controller’s job to register other controllers is it? Is it the controller’s job to register any endpoints at all? Why should I write the same lines of code to validate APIs and catch errors every single time? I created a monster! Surely this could be done better, where some things are done centrally, and maybe we could just define the controllers and externally handle the routing somehow?

I showed one of my colleagues what I had so far, and discussed ways to take out the subroutes feature and make that mechanism independent of any controller. That seemed doable, so then I ended up with independent controllers that can be configured into any route or subroute. “At this rate, you’ll be left with a single JSON file for defining this whole damn API” - my colleague joked.

It was funny, but as I laughed I immediately thought, why not? If controllers are made up of independent endpoints, and subroutes are just a collection of those endpoints, could we rearrange the code to make all of this fully modular? Since the ‘subroutes’ is just an array of objects, and the individual HTTP routes can also be defined using objects (path, HTTP method, controller, etc.), aren’t we just looking at a big object that has a bunch of objects inside it, that kind of look like the big object itself?

My quest took a recursive turn to a recursive turn to a recursive turn to a…okay you get the point. I figured let’s just have a recursive object to define the routes and their functions. And henceforth, a few weeks of coding later, Expressive was born!

Expressive is meant to be flexible but opinionated, because sometimes it's good to have opinions about best practices. It's still Express JS under the covers, with the middleware flexibility, and now I had a modular way of defining every endpoint. This is what I ended up with:

I made an 'ExpressApp' class that takes the router and recursively registers the express endpoints with it's respective routes and functions. You will also notice that each endpoint has a 'validator' with it, and now since each 'controller' is it's own endpoint, the validation code is refactored into a single line internally to avoid repetition.

That's all in one file, but you can see how you could put the "rootRouter" object in one file, the "productsRouter" in another file, and the "reviewRouter" in another, and define endpoints in a very declarative way. This way you could define your root level entities in your APIs in one router file, and the rest would follow. The code was still verbose, so a little refactoring and I came up with this:

There, that's much better! I introduced two new exports - 'Route' and 'subroute' for their respective definitions. Since I was just using objects with the same properties, I thought why not encapsulate them and make things easier to implement and more readable.

Last but not least, API documentation was a concern. Now I figured since each endpoint itself is an independent object, and Swagger definitions are the same thing, why not add a 'doc' property where I can just put a Swagger JSON object? The next steps were naturally to integrate Swagger and Swagger UI (in development mode, but also configurable), where by default it would create the basic Swagger templates for each endpoint that you declare in your router endpoints using the path and method, and you could declare the full API documentation using the 'doc' property, like so:

If you run the app, the docs are available on the "/docs" route by default.

Great success! Now we have an API framework with built in documentation! And you can see which routes are present right away, even if the docs aren't specifically declared!

The ExpressApp class also takes a second object parameter for defining middleware and other options. For example:

You can generally configure your whole API with this class. I was happy with the framework so far because it solved my aforementioned problems. It had built-in middleware like body-parser, cors, express-validator, etc. that's common for every REST API I want to build. I could specifically define a centralized error handling middleware function. I could define API endpoints in a declarative way that makes sense, while having a maintainable code structure. And I could document my code with ease.

I recently came across this article that compares various NodeJS Rest API frameworks, and it was amusing to see the evolution of the frameworks from Express, and how it seems my own framework's endpoint declaration is similar to LoopBack's. We have been able to use my own framework at my current place of work in a few different projects and since it's built on top of Express which we had already done, integrating it with CI/CD was no challenge. I was happy to see my coworkers had fun while using the framework, and that I wasn't the only one that found the features useful. The journey has been fun and fulfilling, and I know I can use the Expressive framework for building REST APIs pretty confidently, but as usual I'm always learning new things, and trying to find ways to make this framework better. And so, my quest continues!

Links:

• Github: https://github.com/siddiqus/expressive
• NPM: https://www.npmjs.com/package/@siddiqus/expressive
• Inspiration article: https://strongloop.com/strongblog/compare-express-restify-hapi-loopback/

Comments

[Ali Almohsen]

Thanks for sharing, will give this a test drive on my next side project.

[Sabbir Siddiqui]

Thanks! Sounds great. Let me know how it goes :)

[Asaju Enitan]

This is a very great article
Seeing stuffs like this keeps refreshing my love for NodeJs framework, I'll def give it a try.

[Sabbir Siddiqui]

Thank you!

[Rattanak Chea]

Thanks for sharing the processes and challenges. I can pick up very useful patterns from your code.

[Sabbir Siddiqui]

Thanks for reading!

[AllStackDev 👨‍💻]

Nice one... Current project following similar pattern. Been a journey but it worth it. Thank for sharing and making me believe in what I am currently doing.