DEV Community

Bulletproof node.js project architecture 🛡️

Sam on April 18, 2019

Originally posted on softwareontheroad.com Update 04/21/2019: Implementation example in a GitHub repository Introduction Express.js is...

Read full post

Ski • Apr 20 '19

with express if you use async handler always wrap the code with try/catch otherwise in case if something happens express will never respond

route.post('/', async (req, res, next) {
   try {
     ... code
   } catch(e) {
      next(e)
   }
})

Dmytro Pylypenko • Apr 22 '19

Or use wrapper around callback.

const wrapCatch = fn => (...args) => fn(args).catch(args[2])
...
app.get('/ping', wrapCatch(async (req, res) => {
  throw Error('break')

  res.send('pong')
}));

app.use((err, req, res, next) => {
  console.error('NOOO')
})

Victor Zamfir • Apr 23 '19

I'd suggest you go even further, like I did here - github.com/oors/oors/blob/master/p...

You can see here how to use it to generate a CRUD experience using async / await - github.com/oors/oors/blob/master/p...

Dmytro Pylypenko • Apr 23 '19

This code is probably for the specific use case, but looks not good.
If the response is undefined, there is no further handle, just a hang.

...
.then(response => {
  if (typeof response !== 'undefined') {
    res.json(response);
  }
})
...

Victor Zamfir • Apr 23 '19

Not really.

You can render the response however you like (using res.json or res.render etc) or you can just return something anything that will be JSON encoded and rendered as is.

Example:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    res.json(result);
  }),
);

which is the same thing as:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    return result
  }),
);

But in other cases you might want to do something like this:

app.get(
  '/',
  wrapHandler(async (req, res) => {
    const result = await ...
    res.render('template', result)
  }),
);

All of these work as expected.

Dmytro Pylypenko • Apr 23 '19 • Edited

You're right!

And this will make an error by negligence.

...
wrapHandler(async (req, res) => {
  const result = await ...
  res.json(result)

  return result
})
...

Victor Zamfir • Apr 23 '19 • Edited

Whoever writes code like that is negligent :)

It's like saying that express.js is too permissive for letting you write code like this:

...
(req, res) => {
  const result = await ...
  res.json(result);
  res.json(someOtherThing);
  res.render('someView');
}
...

I think everyone knows that it's a bad practice to do other things (including returning something) after sending the response in express.js (there are very special cases though, but they don't include the one you shared).

Dmytro Pylypenko • Apr 23 '19

Yes, my idea to hide real response in a route is not a good idea.
Cause you have mixed flow.
In one case, you can put return in other res.SOMETHING, but not both.
It makes harder to maintain and avoid potential fails in a development team.

Victor Zamfir • Apr 23 '19

Yep, I see your point and you're right in a way.

I only use express.js to build APIs (I never render a template or something). So it's more like a convention (shortcut) as returing JSONs is what I do in 90% of the cases.

Dmytro Pylypenko • Apr 23 '19

Yep, this is why I started from this message: "This code is probably for the specific use case, but looks not good."
For example, if you return await ... with undefined you will go in hang situation.

Abdullah Gira • Aug 27 '19

Or you can use express-async-error.

# terminal
npm -i express-async-error

// index.js
require('express-async-error');

and that's it, any error happens the express-async-error will catch it and passes it to the default express error handler.

Elie • Apr 21 '19

Prefer feature based file structure. Why group all your models and all your services together. They don’t really have anything in common. They just happen to be of the same type. Better to group posts model and posts service together.

sweepyoface • Apr 23 '19

This doesn't make much sense. A model will often apply to multiple request methods and/or routes.

Pin-Sho Feng • Apr 28 '19 • Edited

I've been using this generic structure, which I find works pretty well in practice in several languages and domains:

/common
  /models (shared models are moved here)
  /...
/feature1
  /data
    /models (optional folder, if project is too small, not worth it
  /domain
    /models
  /presentation
/feature2
  /data
  /domain
  /presentation

Sam • Apr 29 '19

That's a good architecture too! :)

Yawar Amin • Jun 25 '19

We might be talking about different 'models' here. E.g. the types that are tightly coupled to a particular controller (request) or a service should probably live in the same file or folder. But types that are not obviously coupled could go in a shared 'models' folder, but then I would argue, why are they not strongly coupled to any controller or service? Another point here is that it's OK for a controller to import the types of a service that it uses, that is just normal layered architecture for a higher level to be aware of the lower level.

Elie • Apr 23 '19

So import it where it's needed. Same way as you do when putting models in its folder etc.

Bence Dányi • Apr 23 '19

i'm not sure about the file structure. i think that when you look at a good architecture, it should be obvious what's the purpose of the application, but if you look at folders like api, services, models, that'll tell you nothing. i usually organize files by their purpose, eg. user, product, order, etc.

SohaibRaza • Oct 21 '20

I agree with you components should be self contained.

Aidan Beale • Apr 19 '19

Great architecture and file structure! I also like your bit about the loaders. Very clean way of doing it.
Do you have a skeleton setup on githib or something? I'd love to play around with this.

Sam • Apr 21 '19

I'm glad you like it :)

Here is the github repository github.com/santiq/bulletproof-nodejs

Aidan Beale • Apr 21 '19

Thanks! :)

Adhityo Agam • Apr 20 '19

yep, waiting for the skeleton too

Isak Keyetn • Apr 23 '19

This architecture is completely different from this one
github.com/i0natan/nodebestpractices
why would you say yours is better?

Thorsten Hirsch • Apr 23 '19

Are you sure? I don't see any conflicts. I would say that the "nodebestpractices" focus on different things than Santiago. Actually I only find these 2 points comparable to the advice given here:

1.1 Structure your solution by components
1.2 Layer your components, keep Express within its boundaries

This is very general advice. Santiago follows this advice, but goes way further and fills it with practical instructions.

Isak Keyetn • Apr 24 '19

But his structure is not by components - but by roles. He has a models folder that holds all components models and services with all components services, instead of having a componentName folder with the respectable model and service.

Sam • Apr 29 '19

Oh yes, the good component base architecture, that's a good way to do it too!! :)

Dellean Santos • Jan 31 '20

better is not the right question, because every architecture has a purpose... what fit yours?

Victor Zamfir • Apr 19 '19

Good points.

Another one would be to adopt and follow a plugin / module - based architecture. This way you can split responsibilities and reuse those modules with other projects as well.

A frameworks that pays good respect to these principles and other similar ones is github.com/oors/oors - a framework I created.

It's modules-based, integrates with express.js, promotes a layered architecture, DI is baked in, has great support for MongoDB and GraphQL, plus much more than that.
Feel free to check out the already existing plugins and ask me any questions about it.

sweepyoface • Apr 23 '19

Just some advice – your project doesn't have any kind of helpful readme or docs. I have no idea how to use it, so I won't. Please consider putting effort into that instead of advertising it on articles :(

Victor Zamfir • Apr 23 '19

It actually does - github.com/oors/oors/tree/master/docs - but it's a bit outdated and incomplete. Nonetheless, it highlights the general idea behind the framework.
But you're completely right and documentation is in the making. It hasn't been a priority so far because I was the one to instruct the people who have been using it so far.

The framework's been out for a while now and it's been used in production on some great products.

That being said, I do plan to improve the framework and write more quality extensions, so if you're into node.js, GraphQL, MongoDB... you might want to stick around :)

Jacob Goh • Apr 20 '19

Thanks for this enlightening post.

It reminds me of NestJS, which is a cool Typescript framework.

Manoj Chandrashekar • Apr 19 '19

This is great! I have worked on many nodejs projects and each time the project structure has been a little better than the previous time. Loaders is a nice touch - I hadn't thought of that. Also shout out to Agenda.js - makes job management super simple and it's very reliable!

Israel Muñoz • Apr 20 '19

Hey Santiago, already followed you on Twitter, and as I said there, I really like the points you made.
It'd be awesome if you have a GitHub repo with a basic project set up with this structure.

Btw, there's an error on your link here:

check my guide on using agenda.js the best task manager for node.js.

It's pointing to:
dev.to/nodejs-scalability-issues

I guess you want it to point to your blog:
softwareontheroad.com/nodejs-scala...

Sam • Apr 21 '19

Thanks, I fixed that broken link :)

Here is the example repository github.com/santiq/bulletproof-nodejs

Have fun!

Kostas Bariotis • Apr 22 '19

Well done for this article! Loved it. I would only advise you to remove the middle services layer as you are using Mongoose and Mongoose is able to provide with fully featured models that can do pretty much everything like validation, custom functions, hooks, dependencies on other models, etc. etc.. The middle Services layer will only create an unneeded abstraction that you gonna hate down the road, I certainly hated mine when I realised how much power Mongoose had and that I was trying to write stuff that were already existing.

Put your model's business logic inside Mongose models, handle dependencies on other models inside those models and then let the controllers handle the glue between these models and other pieces of your architecture like sending emails, etc.

Also, make sure to explore this repository which provides a great starting point for a Node.js project structure and covers most of what you covered already. Read the reasoning behind it here.

xout • May 23 '19 • Edited

You have a typo:

export default class UserService() {

should be

export default class UserService {

Very useful article!
Exactly what I was looking for.

You could also link some of your project with those patterns, I think it would help wrap reader's head around it.

Sam • May 25 '19

Thanks for noticing that!

There is a link to an example project with the patterns implemented, you can check it out here github.com/santiq/bulletproof-nodejs

Lukas Siemon • Apr 22 '19

And then you go microservices + lambda and everything becomes different (and more decentralized). Hah :) Nice writeup!

Now we need an npm packages that configures and manages your project structure for you. Feel free to take a look at my robo-config. I'll get to managing the whole project structure dynamically eventually as well :P

Strahinja Laktovic • Apr 20 '19

Awesome post, mate, very nice and organized. I'm curious about your error handling, though. How do you do it, since you have no try catches in services, nor in route handlers ?

Sam • Apr 21 '19

Thank you!

I didn't write too much on good practices about error handling because I wanted to keep the examples simple and concise.

But here you have the complete repository, with proper error handling and more details.
github.com/santiq/bulletproof-nodejs

Strahinja Laktovic • Apr 21 '19

Cool, i checked out your boilerplate, I'd maybe extract error handling logic from the middleware to separate module, maybe add logging. Also, have you thought about having StatusError extending the regular one ? So that you can have comprehensive errors with statuses even from within services, instead of response 500. I'd also add that is of high importance if one should extend regular error to attach stack trace from the super class.

Blake • Apr 22 '19

All great points!! I was thinking about logging after reading this good guide. Centralized logging would be great. Common context such as user, request-id, timing, etc. can be added to all log output. Lots of log shipping programs like to parse structured logs, and formatting in JSON makes it super easy. Also console.log() is not performant for production

@Strahinja - love of the idea of a StatusError. We do something similar in our application. Controllers can throw new Error() which our Express error handler logs as a server error and responds with an HTTP 500 to the client. Controllers can also use a custom error object and throw new ResponseError(err, 400), which logs a warning and returns an HTTP 400 to the client.

Strahinja Laktovic • Apr 22 '19

@blake That sounds really nice. I made a discussion regarding this. You could maybe respond there and maybe put a bit of code so we can put exchanging of ideas in motion.

Scott Yu • Oct 26 '19

Awesome article! I like the structure of this project and IoC. I came from Ruby on Rails so I've been using a similar MVC (or just MC, depends if serving html) structure borrowed from RoR in my Node projects. I'm going to implement these patterns in my new project right now.

I'm curious on where you would put your helpers/utils files though? Do you have a separate directory for them? Or do you stick them under the services directory that uses them?

Sam • Oct 26 '19

Hi!
I use these concepts as a base for my projects, I stick to the 3 layer pattern and sometimes I create a helper's folders and put there functions not related to any service, but other times those helpers were useful to a couple of microservices so I used a private npm package.

Aurel • Apr 16 '20

Hello and thank you for your post,

I would like to suggest that you add an emphasis on the repositories,
Your post talk about the domain isolation from the controller but I think it would be great if you dive into the directory structure of the domain logic. I am referring to DDD philosophy

Michael Z • Apr 19 '19

This sounds like a lot to setup. Have you tried Adonis.js? It has a similar structure to Laravel / RoR, no custom setup needed.

Jay Moretti • Apr 22 '19

Great write up.
For env variables that are injected in other ways (via CI or Lambda) you could move the dotenv line to run when you start your node app:

"node -r dotenv/config --inspect app"

That will avoid failing to load the .env file when it's only present in one of your environments (local).

Hossein Nedaee • Sep 24 '19 • Edited

I went through your code and learned a lot, I didn't finish it yet.
Articles about structure are very enjoyable for me. I need to learn much more about software structure and design patterns.
Thank you.

Sam • Sep 25 '19

I'm really happy that it helped you !! :)

Sam Hall • Apr 30 '20 • Edited

Love the write up! One thing I'm struggling with is where to put more complex functionality in this structure.

Say for example I had some business need to pull data from an external source, transform it, and the push it to another source. In this scenario I would need something in the api and service layer to kick off the whole thing but from there I have trouble keeping things organized. I need all of the objects from data source A and B and another service to map from A to B. Would you create a module to house all of this logic or try to keep it flat in the structure you have shown here? Thanks!

Pablo Andrés Dorado Suárez • Sep 9 '19

Several years ago, I started out a project. Initially to tackle some issues I saw in Sails, it turned out into —more or less, some issues like env variables to manage secrets are still on the go— this exactly.

If you want to check (also, I'm looking for some contributors and anyone willing to use it) the project, you can find it here

Courtney • Apr 22 '19

Thanks for this! When I built my first app I kind of obsessed over file structure a bit too much and couldn't find any solid resources laying out the whys. I ended up looking at a lot of github projects to figure out mine. I wish I had this back then!

wickedknock • Nov 16 '20

I am new to nodejs , I have learnt express and mongodb crud , now I want to learn more , please can you only tell me the topics I have to learn myself to understand and use your repo .

Things I saw in repo I need to learn first is typescript , something called eslint and etc

Thanks <3

SHARAD SHINDE • Apr 19 '19 • Edited

Great post! This is what I am actually looking for!

Can you please review my project structure?

github.com/shindesharad71/Anstagra...

Sam • Apr 23 '19

Seems very good, keep coding! :)

Be careful with the file uploads to express, it can be a real problem in production

You may want to implement a direct upload solution, here is an example with AWS S3 but I saw that you are using GCP so look for something similar.

SHARAD SHINDE • Apr 23 '19

Thank you for your valuable comment and guidance.

I already implemented the direct upload to GCP, no server involved.

Thanks!

Ilia • Sep 10 '19 • Edited

I wouldn't put stock into this. This is a collection of anti patterns. Quite over-dogmatic. None of these patterns make sense at a small scale, neither they do in a complex codebase where every one of those paradigms get in the way of clean code.

Start small. Organise code into functions. Practice TDD. Follow functional programming. Keep directory structure flat. Don't write custom Express middleware.

The rest will follow.

Sam • Sep 10 '19

It’s seems like you are having a bad day.
I send you a virtual hug buddy :)

Al Karlssen • Jul 15 '19

Node.js is an open-source, JavaScript run-time environment used to execute JavaScript code on the server-side. Node.js itechcraft.com/node-js/ development process has changed the paradigm that JavaScript is used primarily on the client-side. That’s why Node.js has become one of the foundational elements of the “JavaScript everywhere” paradigm.

Mauret 👨🏼‍💻 • Nov 27 '19

Hi, nice work ! I just don't understand your unit test. I understood that you had mock UserModel with create method, but when I try to do the same in my test i have got an error :

Argument of type '{ find: () => IUser[]; }' is not assignable to parameter of type 'UserModel'.
Type '{ find: () => IUser[]; }' is missing the following properties from type 'UserModel': watch, translateAliases, bulkWrite, findById, and 58 more.

Any idea to fix this ? Maybe I have to do this differently

Aroh Sunday • Oct 3 '19

Thanks, Sam, this is a wonderful and detailed explanation. I would appreciate seeing the full example of the pub/sub sample with eventemitter as demonstrated above. How can memory leaks be checked and avoided in a bigger nodejs app with EventEmitter? Once again, thanks

CodingStatus.com • Mar 6 '20

This tutorial suggest me a basic & best way about MVC structure, Using this concept, I have developed MVC folder structure in Express with CRUD Example. So, Thanks for sharing..

Sam Clark • Apr 22 '19

really nice breakdown! Thanks!

Thorsten Hirsch • Apr 19 '19

Awesome project structure. I especially like your addition of the pub/sub layer.

Jonathan Worent • Apr 23 '19

How might the architecture look if the data layer is a 3rd party API?

Sam • Apr 23 '19

Should not be so different.

A wrapper class needs to be written for every API resource, with a compatible interface across all models.


class UserModel(){

   public UpdateOne(obj: { selector: any, update: any }):Promise<any> {
     // ...API Call
   }

}

Some methods, for example, UpdateMany may not be present in some third-party API resources and so a custom logic needs to be written.


class CompanyModel(){
   public UpdateOne(obj: { selector: any, update: any }):Promise<any> {
     // ...API Call
   }
   public UpdateMany(objs: Array<{ selector: any, update: any }>):Promise<Array<any>> {
     return Promise.all(
           updates.map(u => {
              return this.UpdateOne(u.selector, u.update);
          })
     )
   }

}

Andreas Bergqvist • Apr 20 '19

Great post! Thanks for sharing!

Beck Lin • May 7 '21 • Edited

Great article and lots of valuable points ! learn so much from your architecture while Im working on my on nodejs project.
Just want to know your opinion regarding the future of nodejs. It seems that the market of nodejs is not so popular as java or python.

aRtoo • Nov 14 '19

Question. What if I wanted to add react. where should I put it? should there be only one node modules for frontend and backend? and what if I wanted to dockerize each? thank you.

Federico Mengozzi • Jul 1 '19 • Edited

Hello, awesome article. Would it be possible to have a list of all patterns used in the project?

Sam • Jul 9 '19

Sure.

The project follows the 3 tier (or layers) pattern for structuring the code and separate the controllers, services and data access layer.

Also, there are some events involved, following the Pub/Sub pattern and inversion of control

And of course, as other developers pointed, you can go nuts and implement something like the 'Clean Domain Driven Design' architecture but is a little overwhelming for little applications.

The list of patterns used here

3 tiers pattern
Pub/Sub pattern
Inversion of control

Federico Mengozzi • Jul 9 '19

Thanks! This is really helpful

Thomas Leitz • Oct 5 '19

Awesome post! In the app.js listing you have

await loaders.init({ expressApp: app });

yet, loaders/index.js doesn't define in init function or did I miss something?

Meet Zaveri • Apr 22 '19 • Edited

Though I am a frontend dev, this gave me good insights of structuring code architecture and seperating concerns. Great write up!

Roman Mikhailov • Apr 22 '19

Loved it! Simple, clean, logical, and extendable!

Anderson. J • Apr 20 '19

Awesome architecture! It looks really scalable.

Basti Ortiz • Apr 20 '19

This was a really good read. Thanks for this! This is exactly what I needed to improve my "architecture skills". Keep it up!

Mohammad Alatrash • Apr 22 '19

Nice article and architecture, the loaders part is really good. Thank you!

Pedro Ivan Partida Galarza • Apr 20 '23

Why use typedi instead of plain old higher-order functions? Functions are first-class citizens in JS/TS, so why not use HOCs instead of adding an additional dependency that applies a pattern that only makes sense in languages without HOCs?

Mauret 👨🏼‍💻 • Nov 16 '19

Hi, thank's for this architecture, i found it perfect for scale API with Node.js ! Adding Typescript & loaders are great ideas

Ande • Jul 22 '20 • Edited

Why do we need two different api and services folder? The subscribers or pub/sub also falls under services.

TusharShahi • Oct 8 '22

Great article.

Maybe you want to add on to this :
An imperative call to a dependent service is not the best way of doing it.

Can you explain why?

Mohamed Ahmed • Jun 10 '20

Thanks for the great article 🤩, I knew most of them but great to put all these in one place, but what about error handling it would be great if u discuss that too, thanks again 🙏

Nikhil Bhandarkar • Jun 30 '20

Great post, great explanation.
But I can't figure out how to handle errors for different scenarios when I should not return any error message or http status from service layer.