DEV Community: Kalkidan Tesfaye

Your code architecture sucks if …

Kalkidan Tesfaye — Fri, 07 Apr 2023 08:46:14 +0000

These are mistakes I did before I learn to develop the right software architecture mindset and I usually see some of them in 90% of projects. Avoiding these mistakes and knowing the right approach when you structure your project can help you build high-performing, scalable, robust, and maintainable software.

1. Your controller is dependent on the database

I noticed this in a lot of codebases, the database logic and functions getting mixed with the controller business logic. If you change the database, the controller will fail to run. Controllers should be independent of the database. Instead of using the controller to access and manipulate data directly, it is best to add another layer that is responsible for providing functions to communicate with the database. Let’s say you have an API to register users, the method to store the data in mongo db, mysql or postgres might have a different name and implementation and if you directly call the ORM function in the controller, you are still making it dependent on the database or the ORM you use. Instead, you can add data or a model service that defines the create method or other database functions used by the controller. So the model will be responsible for providing functions for the controller. Depending on the language and framework you use you can either define the function on the model together with the data type and structure or you can create a separate data service.

2. You don’t have an automated independent module for development processes.

When building software, there are certain processes that seem to appear in every project. Things like database migration, data seeding, dependency management, clean up and management shouldn’t be mixed with your business logic and it is always best to create a development process module you can use on multiple projects. These can include developing a dashboard to automate creating test data, migrating a table, or doing necessary crud operations instead of manually executing a script on the database itself.

3. You don’t manage and tracklogs

An application can have different messages such as errors, warnings, information, user interaction data, etc. When the application gets bigger, data and error management can be overwhelming and you need to have a separate process for routing and storing log data. Storing information about where the bug source is, who sent the request, successful and failed requests and the time it took will help you to do analytics and improve your application

4. You don’t separately store, serve and deploy static files

Static files such as images, documents, or even your frontend code should be separately stored and deployed. You should never store images or documents uploaded by the user to your code base and also you shouldn’t manually store files together with your codebase. Instead, it is best to use cloud storage such as Amazon S3 or google cloud storage to store and manage files separately. Most backend technologies such as Node.js is not designed to serve static assets, if you use your node js server to serve your static files you will end up overwhelming the CPU. Thus it is suggested to use cloud storage together with a proxy CDN to serve your frontend application. This will make sure to deliver the content efficiently and make it quickly available to users. You can also optionally use Nginx to serve static files. Another important point is if your application is heavily dependent on files and assets you might consider using or creating an asset management dashboard.

5. You don’t have a separate configuration file

Configuration information such as port number, URL, paths, statuses, and API keys should be defined in an independent configuration file such as a .env file and injected into the runtime environment as environment variables and it shouldn’t be stored or mixed with your code. The benefit of keeping configuration settings separate from your application logic is that you can apply configuration settings according to the deployment path development, test, and production. Moreover, the .env file should be git ignored to not expose information about the application.

6. You don’t use a secret manager

Having a separate configuration file is important and helpful to write maintainable and robust applications but it is not enough to keep the app secure. Storing sensitive information such as passwords and API keys needs a bit different approach than storing insensitive data. Most cloud services provide a secret manager that lets you save and manage credentials. These add an additional layer of security and you can limit access to the credentials by specifying who can view and alter the credentials.

7. You mix background tasks with your main code logic

Background tasks are operations that can be done later and don’t need to be executed right away. Such kind of tasks are executed outside of the request-response cycle and don’t need the presence or interaction of the user for example sending a thank you email after the subscription or tracking a user activity that isn’t relevant to a user or that doesn’t need to be available to a user right away. First of all, you need to identify things that can run in the background in your application and you need to remove them from your main logic. Instead, you can create a queue of jobs that will later be executed separately from the user request. Some frameworks have an excellent implementation of it but the idea is you will have a table containing a list of jobs(tasks) containing information needed to perform the job such as user id, status whether it is in progress or it is done, name or type of the job, date, etc. Then you will have a command to retrieve from the job table and execute it in an orderly manner. These can be tedious to implement from scratch but there are plenty of libraries that are very easy to work with depending on the language you use. The right implementation of queue/job can significantly improve the performance and usability of your application by reducing the waiting time and latency.

A new course coming soon on creating an architectural mindset in node js. Join the 6.8k community on Facebook or follow up on linked in to get the advanced node js course I will release soon.

Node JS Best Practices — 2023

Kalkidan Tesfaye — Sun, 29 Jan 2023 16:24:41 +0000

1. Don’t useJSON.parse and JSON.stringify

What? but I have used it for many years and it is very important. Yes I have also used it for many years and it indeed served me well but the problem starts when your input grows, it takes a very long amount of time and it can potentially block the event loop. The time complexity for these functions is O(n) and as the n grows the time it takes also grows. If your application takes and processes JSON objects, from a user, you should be cautious about the size of the objects or strings it takes.

Part of the reason why these functions are blocking is that they are processing the whole input and the solution can be to use a stream to work with the given object or string in a chunk. There are npm modules that offer asynchronous JSON APIs. See for example:

JSONStream, which has stream APIs.
Big-Friendly JSON, which has stream APIs as well as asynchronous versions of the standard JSON APIs.

2. Add a logger

An application can have different messages such as errors, warnings, information, user interaction data, etc. When the application gets bigger data and error management can be overwhelming and you need to have information on where the bug source is, who sent the request, and so on. Logger libraries can be very helpful not only in storing messages and errors but also logs/information of successful requests, failed requests, and how long it took. This will help you to do analytics and improve your application. A library called morgan lets you store logs about HTTP requests that have been sent and in the code below I simply add it as middleware and it writes to the console. You can look more here to add different storage mechanisms.

var express = require('express')
var morgan = require('morgan')

var app = express()

app.use(morgan('combined'))

app.get('/', function (req, res) {
  res.send('hello, world!')
})

For errors and messages logs I recommend looking into Winston. It is an easy-to-use library that supports multiple means of transport such as files, databases, and console. You can also set the level such as notice, error, warning, info, etc.

What I normally do is create a separate logger file and export it.

const { createLogger, format, transports } = require('winston');
const config = require('./config');
const { combine, printf } = format;
const winstonFormat = printf(
  ({ level, message, timestamp, stack }) =>
    `${timestamp} ${level}: ${stack || message}`
);
const { timestamp } = format;
const logger = createLogger({
  level: config.env === 'development' ? 'debug' : 'info',
  format: combine(
    timestamp(),
    winstonFormat,
    config.env === 'development' ? format.colorize() : format.uncolorize()
  ),
  transports: [new transports.Console()],
});
module.exports = logger;

3. Adopt one API for one task

It is sometimes difficult to separate and structure APIs but this approach helps in creating decoupling, maintainability, and separation of responsibility in your API not only that but also it makes your application faster than putting a bunch of operations at one API.

For example, when building a form that accepts text and videos. The API should be separated into two, 1 to store the texts, and 2 to upload the video in real-time, this will result in a better response time.

4. Separate code into npm packages

When you are working on multiple projects and if you constantly need to use similar code then it is time to separate the code into an npm package. This will save you a lot of time you use to maintain the code and also it will make it very easy if you are working in a team.

5. Make heavy calculations asynchronous

While node js is best at handling IO operations, it is not well suited for time-intensive mathematical calculation. If you were to do a sum of n integers for a large number, your application can get stuck and don’t serve a request that is because node js serves every request in a single thread and doesn’t create a thread for every request.

This kind of task is executed in the event loop and all incoming requests and outgoing responses pass through the Event Loop. This means that if the Event Loop spends too long at any point, all current and new clients will not get a turn. Just get into the solution!

Okay, so you can wrap you blocking function with set immediate. what do I mean by that and why? setImmediate runs in the next iteration of the event loop which means node js prioritize what can run immediately such as initialization, assignment, and registering asynchronous codes before calling the setImmediate function.

// other codes
    setImmediate(() => {
        processData(data);
    });
// other codes

The event loop doesn’t immediately execute processData instead it registers the set immediate and lets the other code to run. I have written an article on how you can use set immediate to write a nonblocking code and please don’t tell me you haven’t read it yet.

If you need to do something more complex, the first approach is not a good option. This is because it uses only the Event Loop, and you won’t benefit from multiple cores. So the other solution is to use the Node JS worker_threads module to create a separate thread to handle those tasks.

6. Don’t store large data in a variable.

Variables are stored in RAM (Random Access Memory) making it fast to store and access. If you encounter such situations where you need to store data that requires a lot of space always create a place for it in a database. That is because large data can overwhelm the random access memory and it can have effects on the server hence on your application as well.

7. Avoid

Best practices can sometimes be avoiding what is not best. As much as how funny that might sound you need to take these three points seriously.

Avoid using synchronous functions like readFileSync because it can block the main thread. Instead, use the callback or promise version of it.
Avoid storing large sessions or data on your request or response body since it can lag the response. What if you need to send big data? use stream.
Avoid to require a large set of data such as big JSON files since node js require is synchronous and can block the main thread instead store your data in a database and only fetch what you need. If you have to store it in a file use stream to fetch part of the data asynchronously.

Follow me on linked in and get the full advanced node js course I will release soon.