DEV Community: Chiranjib

Platform Engineering for the uninitiated

Chiranjib — Fri, 29 Aug 2025 07:56:20 +0000

"Oh you do platform engineering, you mean CI/CD and pipelines?"

This was my friend's response when I told him that I'm a Lead Platform Engineer now! SMH. I don't really blame him though, of late I find myself struggling to answer crisply when people ask me what I do for a living. The easy answer of course is that I am a developer. I have a confession though - I've been in tech for a while, have worked on software development and delivery, but when I started work as a Platform Engineer, it made me feel as if I've been living under a rock!

I've come across quite a few people who think that DevOps (and even QA) is a luxury and not a necessity. Convincing them on the value proposition of having these bedrocks is an uphill battle, but that's not what this article is about. I'm flushing my thoughts in here, with just the intention of spreading awareness and maybe help a person or two if they've been contemplating about this line of work.

Put simply, a Platform Engineer creates a Platform, thereby offering Platform as a Service (PaaS). My first taste of this was about a decade ago, when Cloud technologies were still in the process of being adopted. My team was working on creating tooling for our Operations Team, who were not developers but were solely in charge of managing deployments. The software had a few flavours - Windows or Linux VMs, SQL Server or Oracle PL/SQL databases, multiple application server stacks based on services selected (containers were not yet a widespread thing) and any combination of these could go on AWS or Azure. The Operations team were going crazy trying to manage deployments and upgrades, and our job was to make their jobs easier. We started developing bash and powershell scripts, but then quickly pivoted to developing a webapp instead that would help one deploy any combination of the aforementioned flavours - an Internal Operations Portal, if I may call it that.

At this point, I expect you to raise a concern - AWS and Azure provide PaaS themselves, why not use it as is?

Well, you would have a valid point; but here's the thing - I've worked with a few programming languages, followed waterfall and agile methodologies, used subversion and git, even tried to understand Jira and came to the conclusion that the most complex of them all is - people! Teams in charge of an organization's infrastructure and processes are expected to put in the effort to create instructions, knowledge bases, playbooks, et al., but getting people to read and follow them has proven to be a Herculean task - and this is just the beginning of the answer to the why.

Over time, I've learned that YAGNI is not just a cool sounding word, KISS is not just an act of love, DRY is not just the opposite of wet, and SOLID is not just a state of matter. Every developer should ideally live by these, and master design patterns that facilitate these. As cloud technologies keep getting more complex and diverse, cross team functions start becoming hazy, and it gets increasingly difficult to achieve consistency and predictability in a software setup. Throw in security to the mix, and conversations quickly shift away from the application related code. This warranted a separation of concern between software development and software delivery.

The solution to this problem started with setting up different teams for both - and ClickOps was coined. As cloud technologies evolved, people realized that it was getting increasingly difficult to keep systems in sync given the room for human error. Naturally, it evolved to the adoption of scripting based pipelines, and it led to the birth of DevOps. This bridged the gap between development and operations quite a bit, and was based on the idea of you build it, you run it, which inadvertently caused shrinkage of operations teams. It was progress alright, but still left techies yearning for something more. Application developers were getting overloaded with the cognitive load of managing infrastructure, and it was getting in the way of product development. Besides this, the complexity of application development was shifting from monoliths to microservices. Upholding the need for separation of concern, technologies evolved and containerization paved the path for taking things to the next level. Docker became somewhat ubiquitous and with the advent of containers, came the need for container orchestration. Google looked over their shoulders and donated Borg to the world, which came to be better known as Kubernetes or k8s. This is where things started to get really spicy, and the cognitive load on developers quickly started to multiply exponentially.

The DevOps way had already seen adoption of Infrastructure as Code, and tools like Terraform, Chef, Pulumi, etc. enabled us to think of our cloud resources as configurations and code. Cloud Engineering needed to evolve along similar lines, and the elephant in the room was - a need for standardization, enter GitOps!

Some of the brightest minds came together to set up the Cloud Native Compute Foundation and championed the concept of GitOps. This brought about yet another major shift in developer mindsets, and allowed techies to be more declarative with their infrastructure and focus solely on the what; the responsibility of how was abstracted away with the new technologies on the horizon. With Kubernetes widely adopted for container orchestration at scale in the cloud, Helm surfaced as the package manager for deployments to k8s clusters. The packages came to be known as Charts and could be deployed with predictability and consistency. This was a step in the right direction, but still included a little bit of ClickOps. Thanks to CNCF yet again, tools like Flux and Argo CD alleviated that pain aptly, and it became possible to manage Helm deployments in a declarative manner. As one can tell, this is already a lot to deal with for a developer who's supposed to write code for implementing business logic.

Software delivery was getting code heavy very quickly - and that is what brought about the need for Platform Engineering. This was a natural step in the evolution from DevOps, and there are multiple discussions online that DevOps is dead! I wouldn't quite agree though, but my view is that Platform Engineering is the wealthy father that adopted the overburdened DevOps (and it's good friend Site Reliability Engineering). Let's have a closer look at this new father figure at the horizon, and understand the peculiar mix of skills that is required to be successful:

Infrastructure Operations: familiarity with the public cloud platforms and other IaaS services is a must, which will form the foundation
Software Development: there would be requirements to develop services for abstraction, and packages for extending tooling to the application code
Product Management: nobody likes to use a bad product, and one must not forget that we are doing this in the first place to make developers' lives easier; we need to strive for the least possible friction and define the Golden Path for developers and it needs to be well documentated and transparent

With the understanding of roles to play, let's try to understand the responsibilities now, I've tried to depict them below:

At the heart of everything, there needs to be security, and the systems in general need to be Secure by Design. Concepts like Role Based Access Control and Attribute Based Access Control can not be an afterthought, but need to be baked in from the start. Once artifacts start getting generated at scale, and open source packages keep getting integrated in application code, all the guardrails like License scanning, Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), and Software Composition Analysis (SCA) need to be in place to stop malicious code in the integration phase. Additional artifacts like SBOM (Software Bill of Materials), test coverage reports and scan reports need to be published for the artifacts being generated during code integration.

Once all the important tech is in-place to enable software delivery, the necessary augmentation is what comes next. Documentation is a non-negotiable demand of the craft, and tools like PlantUML and Mermaid have enabled maintenance of Documentation as Code (Docs as Code).

Smart instrumentation for Site Reliability Engineering is of utmost importance now, because with so many moving parts in a microservices based architecture it is otherwise near impossible to identify problems. Technologies like ELK and LGTM+ in combination with OpenTelemetry provide respite in that area.

This is of course not an exhaustive list of what's already existing, and with Agentic applications on the horizon, the scope is only ever going to increase - think AIBOM and AI vulnerability scanners. Other possibilities might also include setting up internal MCP servers to enable small language models that power AI-native applications.

I'd like to conclude by saying - it's a lot, and it can get very overwhelming very fast, but it's important that we keep calm and resist the urge to boil the ocean on this journey.

P.S. Before all of this, we must evaluate if the product we're building it for has an actual need for these things at scale. Some food for thought -
This post by David Heinemeier Hansson

Beautify WSL2 terminal with ohmyposh

Chiranjib — Mon, 10 Mar 2025 07:50:25 +0000

Read on if you

Are using the combination: Windows, WSL2, Ubuntu
Don't like the look and feel of the limited terminal
Like me, aren't yet using Windows Terminal for some inexplicable reason

Prerequisites

Have a computer that runs Windows OS (Mine is Windows 10)
Enable WSL2 -> Here's how, if you need it
A little tech knowledge about how these things work, just enough to make small changes and understand what's going on

Step 1: Installation

Open Ubuntu in WSL and download oh-my-posh. Oh My Posh is a prompt theme engine for your shell (I, being a simpleton, am using bash). Their somewhat detailed installation instructions enable you to download and set it up smoothly.

Once the installation has completed, let's take small steps towards enabling it. Let's edit the ~/.bashrc file to ensure that it loads when we start.

export PATH=$PATH:~/.local/bin
if [ -f $(which oh-my-posh) ]; then
  eval "$(oh-my-posh init bash)"
fi

Let's see what we just did:

assuming that oh-my-posh got installed in the location ~/.local/bin, we are ensuring that it's loaded in the PATH. This is necessary to be able to invoke the command itself.
since we are using bash, we invoked the command oh-my-posh init bash at startup, so that things are nice and pretty for us to use from the start.

Step 2: Selecting a theme

Discover all the ohmyposh themes that you can choose from, and download the file from the link provided there or just pick one from the list here: https://github.com/JanDeDobbeleer/oh-my-posh/tree/main/themes

Let's assume we have downloaded the necessary theme .json files in the directory ~/.poshthemes.

Let's now modify the file ~/.bashrc to pick the theme up as shown below:

if [ -f $(which oh-my-posh) ]; then
  eval "$(oh-my-posh init bash --config ~/.poshthemes/jandedobbeleer.omp.json)"
fi

Step 3: Installing a Nerd Font

Oh My Posh requires a Nerd Font to work. So, explore all the nerd fonts, have your pick, and download the font to your windows system. Once, the font is downloaded, navigate to the font location, right click, and click Install.

Once installed, verify that the font is visible in the list of fonts: Win + R, type fonts. I went with Inconsolata.

Step 4: Configure the installed font

As a final step, we need to configure the emulator, that is used by Windows. Launch Command Prompt or Ubuntu App, right click on the toolbar, click Properties. This should allow you to select the installed font. One easy to miss detail is that CMD works with Monospaced fonts only. So, the font that you had chosen, ensure that you've installed the Mono variant.

Working with Node.js Entities and Mongoose Models - III

Chiranjib — Mon, 20 Feb 2023 07:32:14 +0000

Previous: Working with Node.js Entities and Mongoose Models - II

Now that we have established the entities, controllers, models and data-access for elementary CRUD operations, let's go ahead and add a few interactions. We have movies, we have users, now we need to:

enable our users to rate movies
fetch ratings for a movie
fetch all ratings by a user

Step 1 - edit our controller for picking up extensions as:

...
    entityObject.getExtensions().forEach(function (extension) {
        router.use('/', extension);
    });

    return [`/${entityObject.name}`, router];

Step 2 - create router extensions for entities

extension for rating movie

Let's create a file ./entities/movie/extendedRouters/postRateMovie.js as:

const router = require('express').Router({ mergeParams: true });
const MovieRatingModel = require('_data-access/models/MovieRating');

router.post('/rate', async function (req, res, next) {
    try {
        const { movie, user, rating } = req.body;
        const movieRating = await MovieRatingModel
            .findOneAndUpdate({
                movie, user
            }, {
                $set: {
                    rating
                }
            }, {
                upsert: true,
                new: true
            });
        res.json(movieRating);
    } catch (error) {
        next(error);
    }
});

module.exports = router;

The code above simply takes in the request body and persists it to the required collection. You may notice how it uses findOneAndUpdate, upsert: true and new: true. This ensures that mongoose creates the document for us upon first access and helps ensure that there's only ever one combination of a movie and a user (enabling a user to modify their rating if they fancy).

extension for fetching movie ratings

Let's create a file ./entities/movie/extendedRouters/getRatingsByMovieId.js as:

const router = require('express').Router({ mergeParams: true });
const MovieRatingModel = require('_data-access/models/MovieRating');
const { Types: { ObjectId } } = require('mongoose');

router.get('/:movieId/ratings', async function (req, res, next) {
    try {
        const movieRatings = await MovieRatingModel.aggregate([
            {
                $match: { movie: new ObjectId(req.params.movieId) },
            },
            {
                $group: {
                    _id: '$movie',
                    ratings: { $push: '$rating' },
                    averageRating: { $avg: '$rating' },
                },
            },
        ]);
        res.json(movieRatings);
    } catch (error) {
        next(error);
    }
});

module.exports = router;

Now, we're getting a little fancy with our query. Notice now we have used aggregate pipelines. There are multiple aggregation stages in Mongo and it takes some getting used to if you are transitioning from being a relational database user. Let us spend a moment on the pipelines here:

the $match stage of the pipeline simply matches movie id of documents.
the '$group' pipeline collects all the documents that pass the $match stage, and $push them into an Array ratings, and also calculate the $avg of those ratings as the field averageRating.

extension for fetching all ratings by a user

Let's create a file ./entities/user/extendedRouters/getRatingsByUserId.js as:

const router = require('express').Router({ mergeParams: true });
const MovieRatingModel = require('_data-access/models/MovieRating');
const MovieModel = require('_data-access/models/Movie');
const {
    Types: { ObjectId },
} = require('mongoose');

router.get('/:userId/ratings', async function (req, res, next) {
    try {
        const movieRatingsByUser = await MovieRatingModel.aggregate([
            { $match: { user: new ObjectId(req.params.userId) } },
            {
                $group: {
                    _id: '$user',
                    records: {
                        $push: {
                            movie: '$movie',
                            rating: '$rating',
                        },
                    },
                },
            },
        ]);
        await MovieModel.populate(movieRatingsByUser, { path: 'records.movie', select: 'name' });
        res.json(movieRatingsByUser);
    } catch (error) {
        next(error);
    }
});

module.exports = router;

We have used the aggregate pipeline once again, let's understand what's happening once more:

the $match stage matches the user id in question with the available documents
the $group stage again collects all documents that pass the $match stage, and $push them into an array records as an object having attributes movie and rating.

We decide to add a little more spice into our response, and decide that we will populate the Movie name in the response as well, because the person who processes the response would be able to make better sense of it. The line await MovieModel.populate(movieRatingsByUser, { path: 'records.movie', select: 'name' }); iterates all the documents that were received after the aggregate pipeline, and starts populating the records.movie attribute for all of them with the 'Movie name' for relevance.

Step 3 - make the entities return the router extensions

Define the function getExtentions in file ./entities/movie/index.js as:

getExtensions() {
        return [require('./extendedRouters/getRatingsByMovieId'), require('./extendedRouters/postRateMovie')];
    }

Define the function getExtentions in file ./entities/user/index.js as:

getExtensions() {
        return [require('./extendedRouters/getRatingsByUserId')];
    }

And, that's that. We have the entity interactions defined and fully functional.

Next: Typing...

Working with Node.js Entities and Mongoose Models - II

Chiranjib — Sun, 19 Feb 2023 14:14:52 +0000

Previous: Working with Node.js Entities and Mongoose Models - I

The finish line is very close now, let's do it!

Step 1 - understand that the magic happens 'async'

Node.js is single-threaded and JavaScript is asynchronous by design. If you don't know or understand how it works, then I'd strongly advise that you should familiarize with the concepts of asynchronous processing first. Here's a good explanation of the concepts.

Step 2 - modify our controller to support asynchronous processing

In the file ./controllers/index.js, we need to make the router functions async, and remember to await before returning the response:

const BaseEntity = require('_entities').getBaseEntity();
const logger = require('_utils/logger');

/**
 * 
 * @param {BaseEntity} entityObject 
 * @returns {[string, Router]}
 */
function getController(entityObject) {
    const router = require('express').Router({ mergeParams: true });
    router.post('/', async (req, res, next) => {
        try {
            const response = await entityObject.create(req.body);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.get('/', async (req, res, next) => {
        try {
            const response = await entityObject.read();
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.get('/:entityObjectId', async (req, res, next) => {
        try {
            const response = await entityObject.read(req.params.entityObjectId);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.put('/:entityObjectId', async (req, res, next) => {
        try {
            const response = await entityObject.update(req.params.entityObjectId, req.body);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.delete('/:entityObjectId', async (req, res, next) => {
        try {
            const response = await entityObject.delete(req.params.entityObjectId);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    return [`/${entityObject.name}`, router];
}

module.exports = {
    getController
};

Step 3 - Create entities for Movie and User

./entities/movie/index.js

const BaseEntity = require('_entities/BaseEntity');

module.exports = class MovieEntity extends BaseEntity {

    constructor() {
        super();
        this.name = 'Movie';
        this.model = require('_data-access/models/Movie');
    }

    async create(payload) {
        return await this.model.create(payload);
    }

    async read(id) {
        if (id) {
            return await this.model.findById(id).lean();
        } else {
            return await this.model.find().lean();
        }
    }

    async update(id, payload) {
        return await this.model.findByIdAndUpdate(id, payload, { new: true });
    }
};

./entities/user/index.js

const BaseEntity = require('_entities/BaseEntity');

module.exports = class UserEntity extends BaseEntity {
    constructor() {
        super();
        this.name = 'User';
        this.model = require('_data-access/models/User');
    }

    async create(payload) {
        return await this.model.create(payload);
    }

    async read(id) {
        if (id) {
            return await this.model.findById(id).lean();
        } else {
            return await this.model.find().lean();
        }
    }

    async update(id, payload) {
        return await this.model.findByIdAndUpdate(id, payload, { new: true });
    }
};

Take note, both the entities are not supporting the delete method, so the expectation is that if someone (could be a hacker) hits the DELETE endpoint, they would see that it is not supported.

Step 4 - generate controllers and expose them for the new entities

In our ./index.js file, we have to make modifications to the setupApp function as follows:

....

    app.use(...getController(entities.getEntityOne()));
    app.use(...getController(entities.getMovieEntity()));
    app.use(...getController(entities.getUserEntity()));

    app.use('/', (req, res) => {
        res.send(`${req.originalUrl} can not be served`);
    });
....

And that's it! We have a fully functional persistent Node.js backend.

Next: Working with Node.js Entities and Mongoose Models - III

Working with Node.js Entities and Mongoose Models - I

Chiranjib — Sun, 19 Feb 2023 13:48:23 +0000

Previous: Integrate Node.js Backend with MongoDB

Let's reiterate the idea of our example application

Our application maintains a database of movies, and their ratings by people. So, we are dealing with three entities: User (you or me), Movie (the things that require our time, money and attention) and Rating (link between User & Movie).

Step 1 - define Schemas and Models

As can be deduced from above, let us create three (data-access) models:

./data-access/models/User.js

const { Schema } = require('mongoose');
const { Types } = Schema;
const { getDBConnection } = require('_data-access/ConnectionFactory');

const UserSchema = new Schema(
    {
        email: { type: Types.String, required: true, unique: true },
        firstName: { type: Types.String },
        lastName: { type: Types.String },
    },
    {
        timestamps: true,
    }
);

module.exports = getDBConnection().model('user', UserSchema, null, {});

./data-access/models/Movie.js

const { Schema } = require('mongoose');
const { Types } = Schema;
const { getDBConnection } = require('_data-access/ConnectionFactory');

const Movie = new Schema(
    {
        name: { type: Types.String, required: true, unique: true },
        imdbLink: { type: Types.String, required: true, unique: true },
        rottenTomatoesLink: { type: Types.String, required: true, unique: true },
    },
    {
        timestamps: true,
    }
);

module.exports = getDBConnection().model('movie', Movie, null, {});

./data-access/models/MovieRating.js

const { Schema } = require('mongoose');
const { Types } = Schema;
const { getDBConnection } = require('_data-access/ConnectionFactory');

const MovieRating = new Schema(
    {
        movie: { type: Types.ObjectId, ref: require('./Movie'), required: true },
        user: { type: Types.ObjectId, ref: require('./User'), required: true },
        rating: { type: Types.Number, enum: [1, 2, 3, 4, 5] },
    },
    {
        timestamps: true,
    }
);

module.exports = getDBConnection().model('movierating', MovieRating, null, {});


module.exports = getDBConnection().model('movierating', MovieRating, null, {});

Step 2 - define indices for faster data access

This step is easy to miss, and it seems unnecessary, but it is something essential that can come back to bite people in the middle of the night when a customer data can't be loaded or the database starts sending high severity alerts because it is experiencing high I/O or is running out of memory.

Now we're walking a slippery slope. Database indices are a two-edged sword. As the name suggests, think of indices as an index to a book. So, if there are edits to any page of a book or a new page gets added to the book, the index would have to be revised and recreated. That's why, while they enable faster data access, one too many indices will have I/O or memory implications of their own. Potentially, every document write will trigger refresh of the indices.

That's why, it is important to imagine and define the data-access patterns in the application. With our example, here are the potential data queries:

find user by email
find movie by name
find how many users have rated a movie
find how many movies a user has rated

Revisiting the Models and Schemas we have defined, let us define indices:

data-access/models/User.js

UserSchema.index({ firstName: 1 }); // enable searching user by first name

data-access/models/Movie.js

Movie.index({ name: 1 }); // enable searching a movie by name

data-access/models/MovieRating.js

MovieRating.index({ movie: 1, user: 1 }); // enable searching movie-ratings by movie first, and then by specific user if required
MovieRating.index({ user: 1, movie: 1 }); // enable searching movie-ratings by users first, and then by specific movie if required

At this point, we are well-equipped to take the next step and achieve full sufficiency in our backend.

Next: Working with Node.js Entities and Mongoose Models - II

Integrate Node.js backend with MongoDB

Chiranjib — Sun, 19 Feb 2023 13:09:43 +0000

Previous: Create modular routes with express

Now that we have set up a Node.js backend with express.js and understand how we want to define our controllers and entities, it's time to make our setup more realistic. Our backend right now is missing a very crucial aspect - persistence (or in simple words, it needs a database).

Staying true to our stack, let's integrate MongoDB. At this point, we should download and install MongoDB server. You may use this link: MongoDB Community Server

To understand things fully, we have to come up with an example. Here's what we are going with:

Our application maintains a database of movies, and their ratings by people. So, we are dealing with three entities: User (you or me), Movie (the things that require our time, money and attention) and Rating (link between User & Movie).

Step 1 - get mongoose

MongoDB is a schemaless database, which means you may store any JSON in any collection and a new onlooker can potentially be left absolutely mesmerized.
Enter mongoose - a library that provides Schemas and Models to make dealing with data on MongoDB a little more predictable.

Let's navigate to our Node.js project root, and execute this

npm i --save mongoose

Our package.json should have this new entry now:

    "dependencies": {
        "express": "4.18.2",
        "helmet": "6.0.1",
        "link-module-alias": "1.2.0",
        "mongoose": "6.9.2",
        "winston": "3.8.2"
    }

Step 2 - setting up for data-access

Going back to the drawing board a little bit, let us establish a data-access module (folder) within our project.

Create a data-access folder
mkdir ./data-access

Data-access is again something that you are probably going to need throughout your application code, even in nested places. Let's set it up for access with absolute path. Add an entry in the _moduleAliases section of package.json:

    "_moduleAliases": {
        "_controllers": "./controllers",
        "_entities": "./entities",
        "_utils": "./utils",
        "_config": "./config",
        "_data-access": "./data-access"
    }

Once done, we have to create the linkage of the new directory by executing
npm i

Notice in the console output:

> link-module-alias

link-module-alias: _controllers -> ./controllers, _entities -> ./entities, _utils -> ./utils, _config -> ./config, _data-access -> ./data-access

Step 3 - time to light it up

Let's understand something trivial - the application will be in an impaired state if the database connection doesn't work. So, before anything else, our Node.js process should connect to the database and then start the app.

Assuming that we are working with a local copy of MongoDB, let's modify the ./config/index.js file to include database details:

...
    DATABASE: {
        URL: 'mongodb://127.0.0.1',
        DB_NAME: 'moviemania'
    }

Our sample application is very simplistic, but real world applications may work with multiple databases and multiple models. With that in consideration, it is a good practice to set up a data connection factory. Our implementation makes use of the Factory Design Pattern and the Singleton Design Pattern. Create a file ./data-access/ConnectionFactory.js as:

const { DATABASE: { URL, DB_NAME } } = require('_config');
const mongoose = require('mongoose');
/**
 * @type {mongoose.Connection}
 */
let dbConnection;

module.exports = {
    getDBConnection: function () {
        if (!dbConnection) {
            dbConnection = mongoose.connect(URL, { dbName: DB_NAME, maxPoolSize: 10 });
        }
        return dbConnection;
    },
    closeConnection: function () {
        if (dbConnection && dbConnection.readyState === 1) {
            dbConnection.close();
        }
    }
};

We will have to refactor our index.js as:

one function to connect to the database
another subsequent function that starts the app

const logger = require('_utils/logger');

function connectToDB() {
    const { getDBConnection } = require('_data-access/ConnectionFactory');
    return new Promise((resolve, reject) => {
        getDBConnection()
        .then(connection => {
            let counter = 1;
            const timer = setInterval(function () {
                logger.info(`[${counter}] Checking connection..`);
                if (connection.readyState === 1) {
                    clearInterval(timer);
                    resolve();
                } else if (counter === 5) {
                    clearInterval(timer);
                    reject('Could not connect to database after 5 retries');
                }
                counter++;
            }, 1000);
        })
        .catch(e => {
            logger.error(e)
            reject('Could not connect to database')
        });
    });
}

function setupApp() {
    const {
        SERVER: { PORT, REQUEST_BODY_SIZE_LIMIT },
    } = require('_config');
    const { getController } = require('_controllers');
    const helmet = require('helmet');
    const express = require('express');
    const entities = require('_entities');
    const app = express();

    app.use((req, res, next) => {
        logger.log('info', `Received request [${req.method}] ${req.originalUrl}`);
        next();
    });

    app.use(helmet());
    app.use(express.json({ limit: REQUEST_BODY_SIZE_LIMIT }));
    app.use(express.urlencoded({ extended: true, limit: REQUEST_BODY_SIZE_LIMIT }));

    app.get('/healthcheck', (req, res) => {
        res.send('OK');
    });

    app.use(...getController(entities.getEntityOne()));
    app.use(...getController(entities.getMovieEntity()));
    app.use(...getController(entities.getUserEntity()));

    app.use('/', (req, res) => {
        res.send(`${req.originalUrl} can not be served`);
    });

    app.listen(PORT, () => {
        logger.log('info', `Listening on port ${PORT}`);
    });
}

connectToDB()
    .then(function () {
        logger.info('Connected to database, starting app now...');
        setupApp();
    })
    .catch(function (e) {
        logger.error('Failed to connect to database.', e);
    });

If you notice carefully, our connectToDB function returns a Promise. Only when the Promise is fulfilled would the next function setupApp be invoked.

So, we have set up a Node.js backend and connected to MongoDB. Next up, we will connect the dots between Controllers, Entities and Models.

Next: Working with Node.js Entities and Mongoose Models - I

Setting up a Node.js backend

Chiranjib — Tue, 14 Feb 2023 14:46:48 +0000

Previous: Learn to build with MERN stack

DISCLAIMER: This guide by no means aspires to be the holy grail. At the time of writing this I've been developing software for more than a decade and let's say I know of a few things that would make coding easy when it comes to basic principles like Readability, Maintainability, etc.
P.S. If you don't like my suggestions, please don't hurl abuses at me, I have my friends for doing that. If you have some constructive feedback, please do share!

Step 1 - start the project and install a package

Pick a folder on your device where you'd like to keep your code. It can be surprisingly easy to start a Node.js project. Open command terminal and navigate to the directory of your choice. Once there, execute this:

> npm init

Once you fill out the questionnaire, you should notice a package.json file in your directory. You may want to read more about npm, npmrc, package.json and package-lock.json here: About npm

To get our feet wet, let's install a package - winston

> npm install --save winston

Let's check our package.json

....
"dependencies": {
    "winston": "^3.8.2"
}
....

Notice the Caret (^) next to the version. This indicates winston should have any package compatible with 3.8.2. In some cases, this may present an unwanted problem. If we run npm update, it can potentially upgrade the package to the next compatible version. I like to exercise a bit more control on the package and the versions I have. There are many ways to do this, I prefer using npmrc for this.

Let's create a file named .npmrc as a sibling of package.json and put this line in it:

save-exact=true

If you want to check all the options available for this file, check out npmrc config options
Now, let's uninstall the package and install again

> npm uninstall --save winston
> npm install --save winston

Notice the package.json has this now

....
"dependencies": {
    "winston": "3.8.2"
}
....

Sweet! We have a solid grip on all the package versions.

Step 2 - installing a few essentials

> npm install --save express
> npm install --save-dev nodemon
> npm install --save-dev eslint

If you notice, we've installed express with --save and the other two with --save-dev.
The idea is to have some packages that are only required in our local setup while writing code. When it's time to release to production, the final version can be leaner to minimize the size of the shipment (more on that later).

About the packages:

express : popular library to define endpoints
nodemon : popular for development as it watches files for changes and restarts the node process
eslint : library required for enforcing code style guidelines

Step 3 - set up linting

It's essential to have a few rules to keep the code style consistent, eslint helps us do that. Let's set it up, run this command

npx eslint --init

Answer the questions, specify your style of coding, and we're good to go. You may observe, a .eslintrc.js file would have gotten created (provided you picked JavaScript as your option instead of JSON or YAML)

Step 4 - create a few folders

For the sake of sanity and keeping things modular, let's create the following directory structure to begin with. Don't worry about the content yet, we will focus on this as we move along.

/
  - config
  - controllers
  - entities
  - utils

Step 5 - preparing for env variables

It's a good practice to put all the variables in a single location, so that it's easier to refactor in the future. This may seem an overkill when we are just starting out, but as the application code grows and complexity increases, environment variables get messy and making changes can get tricky.
Let's create a file config/index.js and put the following content in it:

module.exports = {
    SERVER: {
        PORT: 3000,
        REQUEST_BODY_SIZE_LIMIT: '10mb'
    },
    LOGGING: {
        LEVEL: 'warn'
    }
};

Step 6 - setting up project for absolute path

One thing that quickly gets ugly, are relative file paths (require ../../../folder1/folder2/file1.js). Nowadays, there are many ways to get rid of it, here we'll explore one option.

npm i --save link-module-alias

Edit the package.json to add a postinstall script and the _moduleAliases section as shown below:

...
    "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1",
        "postinstall": "link-module-alias"
    },
    "_moduleAliases": {
        "_controllers": "./controllers",
        "_entities": "./entities",
        "_utils": "./utils",
        "_config": "./config"
    }
....

You may run npm i after this, and notice this in the console output:

....
> link-module-alias

link-module-alias: _controllers -> ./controllers, _entities -> ./entities, _utils -> ./utils, _config -> ./config
...

Now, we may use these as normal require statements like require('_controllers') in any file within our project structure.

In this post, I am only trying to highlight that you SHOULD enable something like this, by no means am I preaching that you use the same package.

Step 7 - specify start scripts

We choose to run Node.js projects with npm commands instead of a simple command like node index.js for the sake of consistency. Let's inject the following scripts in the package.json

....
"scripts": {
        "test": "echo \"Error: no test specified\" && exit 1",
        "start": "node index.js",
        "dev": "nodemon index.js",
        "postinstall": "link-module-alias"
    }
....

Step 8 - setting up the first endpoint

Let's try to set up a healthcheck endpoint that simply responds with text "OK". First, let's enable us to use the logger library we integrated earlier. Create a file utils/logger.js as shown below:

const winston = require('winston');
const { LOGGING: { LEVEL } } = require('_config');

const { combine, timestamp, prettyPrint, errors } = winston.format;

module.exports = winston.createLogger({
    format: combine(
        errors({ stack: true }),
        timestamp(),
        prettyPrint()
    ),
    level: LEVEL,
    transports: [
        new winston.transports.Console()
    ]
});

Now, we can focus on our route. Let's edit the index.js file as

const { SERVER: { PORT } } = require('_config');
const logger = require('_utils/logger');

const express = require('express');

const app = express();

app.use((req, res, next) => {
    logger.log('info', `Received request ${req.originalUrl}`);
    next();
});

app.get('/healthcheck', (req, res, next) => {
    res.send('OK');
});

app.use('/', (req, res) => {
    res.send(`${req.originalUrl} can not be served`);
});

app.listen(PORT, () => {
    logger.log('info', `Listening on port ${PORT}`);
});

Few highlights of the simple snippet above:

all the statements defined for app, get executed in sequence
we have defined a logger with winston that logs to the console for the sake of this example and restricted it to only warn level using an environment variable. The line logger.log('info'... won't print anything to the console until the level is set to 'info'. This comes in handy when in production we want to start printing more log statements to debug a problem. We can just change the environment variable on the fly.
app.use() statement in the beginning takes in a middleware that gets executed for any endpoint call that the app receives. This is useful for doing tasks like authentication, logging for audit, etc. The next() call is crucial here to pass on the request chain to the next applicable middleware or endpoint definition as applicable
app.get('/healthcheck') is the endpoint we intended to define, which simply responds with 'OK'
app.use('/') is the fallback endpoint, that would catch all the requests that could not be served, and we can show graceful error messages if we desire

To run the server, you need to execute either of the commands

npm run start
npm run dev

Voilà! We're all set! You may hit the endpoint we have just defined, and revel in the glory of your hard work.

So, we have our boilerplate ready. Let's shape it up to be more robust and introduce modularity for more routes.

Next: Create modular routes with express

Create modular routes with express

Chiranjib — Tue, 14 Feb 2023 14:46:12 +0000

Previous: Setting up a Node.js backend

DISCLAIMER: This guide by no means aspires to be the holy grail. At the time of writing this I've been developing software for more than a decade and let's say I know of a few things that would make coding easy when it comes to basic principles like Readability, Maintainability, etc.
P.S. If you don't like my suggestions, please don't hurl abuses at me, I have my friends for doing that. If you have some constructive feedback, please do share!

Assuming you have set up the Node.js backend boilerplate with express in accordance with the previous guide, let's go on and introduce some flavour.

Step 1 - enable body-parser

A crucial step before we dive into defining our routes, is to enable request body parsing in express. Mostly, the POST and PUT routes are expected to have a request body payload, and we need to be prepared to parse them.

Express.js includes the body-parser package now, but we need to plug this in to the base express router. Stating the obvious, this statement needs to be plugged in before any of the routes start processing, with the assumption that all routes would require this functionality.
One small detail to note here - we have introduced a new variable: REQUEST_BODY_SIZE_LIMIT. This helps control the maximum allowed request body limit. As application grows complex, this helps put restraints in place according to available infrastructure resources.

...
const { SERVER: { PORT, REQUEST_BODY_SIZE_LIMIT } } = require('_config');
...
app.use(express.json({ limit: REQUEST_BODY_SIZE_LIMIT }));
app.use(express.urlencoded({ extended: true, limit: REQUEST_BODY_SIZE_LIMIT }));

Step 2 (optional) - implement some basic security measures

Application security is non-negotiable. Thanks to the community, there's a package for this as well. The name is aptly selected - helmet. Put this on before you go for the ride 😉 😉

npm i --save helmet

And we plug this in our code towards the beginning.

...
const helmet = require('helmet');
...
app.use(helmet());
app.use(bodyParser.json({ limit: REQUEST_BODY_SIZE_LIMIT }));
...

Step 3 - decide on the approach

A very important but somewhat undervalued aspect of software development is to write code with a predefined structure. When the team size is small (<5), it seems manageable because communication is easier. But, as the team grows in size, people come in with different ideologies and beliefs. This has the potential of causing mayhem and a feeling like loss of identity.
An instrument to facilitate the same is Design Patterns. We will try to understand this step by step as we introduce our components.

This is what we will follow for the purposes of this example:

Entity: An object that supports CRUD operations
Controller: The mechanism of exposing the CRUD operations of an entity via endpoints of the application

Step 4 - define entities

We will implement the Decorator Design Pattern for our entities. The idea here is - our base entity will expose a few methods:

create()
read()
update()
delete()
getExtensions()

Each entity will have its name, which will be used to form the controller that exposes the routes associated with the entity. You may read debates on the internet about using classes versus functions in JavaScript, I would say use either that you like. We are defining entities with classes, just because I am comfortable this way.

Base Entity

Create this file entities/BaseEntity.js

const Router = require('express').Router;

/**
 * @class
 */
class BaseEntity {
    constructor() {
        this.name = 'BaseEntity';
        this.extensions = [];
    }

    /**
     * The child class is supposed to override this
     * @returns {Object}
     */
    create() {
        throw 'Not Supported';
    }

    /**
     * The child class is supposed to override this
     * @returns {Object | Array.<Object>}
     */
    read() {
        throw 'Not Supported';
    }

    /**
     * The child class is supposed to override this
     * @returns {Object}
     */
    update() {
        throw 'Not Supported';
    }

    /**
     * The child class is supposed to override this
     * @returns {Object}
     */
    delete() {
        throw 'Not Supported';
    }

    /**
     * The child class is supposed to override this
     * @returns {Array.<Router>}
     */
    getExtensions() {
        return this.extensions;
    }
}

module.exports = BaseEntity;

Entity One

We will set up our first Entity as a child of the BaseEntity. Create the file entities\entity1\index.js as:

const BaseEntity = require('_entities/BaseEntity');

module.exports = class EntityOne extends BaseEntity {
    constructor() {
        super();
        this.name = 'EntityOne';
        this.records = [
            { 'id': 1, 'key1': 'value1' },
            { 'id': 2, 'key2': 'value2' }
        ];
    }

    create(payload) {
        const lastObject = this.records[this.records.length - 1];
        this.records.push({ id: (lastObject?.id ?? 0) + 1, ...payload });
    }

    read(id) {
        if (id) {
            return this.records.find(record => record.id === parseInt(id));
        } else {
            return this.records;
        }
    }

    update(id, payload) {
        const record = this.records.find(record => record.id === parseInt(id));
        if (record) {
            const { _id, ...otherAttributes } = record;
            Object.assign(record, { ...otherAttributes, ...payload });
        }
    }

    delete(id) {
        const recordIndex = this.records.findIndex(record => record.id === parseInt(id));
        if (recordIndex) {
            this.records.splice(recordIndex, 1);
        }
    }
};

Let's take a moment to see what we've done above. The BaseEntity defines CRUD methods with an error. The child entity creates overrides of the same methods, with the correct implementation. If the child does not create the implementation (intentionally or unintentionally), then the child would still respond with the implementation of the BaseEntity.

Step 5 - enable controller generation

We will leverage the Builder Design Pattern for enabling our controllers. Let's define a file controllers\index.js as:

const BaseEntity = require('_entities').getBaseEntity();
const logger = require('_utils/logger');

/**
 * 
 * @param {BaseEntity} entityObject 
 * @returns {[string, Router]}
 */
function getController(entityObject) {
    const router = require('express').Router({ mergeParams: true });
    router.post('/', async (req, res, next) => {
        try {
            const response = await entityObject.create(req.body);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.get('/', async (req, res, next) => {
        try {
            const response = await entityObject.read();
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.get('/:entityObjectId', async (req, res, next) => {
        try {
            const response = await entityObject.read(req.params.entityObjectId);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.put('/:entityObjectId', async (req, res, next) => {
        try {
            const response = await entityObject.update(req.params.entityObjectId, req.body);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    router.delete('/:entityObjectId', async (req, res, next) => {
        try {
            const response = await entityObject.delete(req.params.entityObjectId);
            res.json(response);
        } catch(e) {
            logger.error(e);
            res.status(500).send('Oops! Something went wrong!');
        }
    });

    return [`/${entityObject.name}`, router];
}

module.exports = {
    getController
};

This file defines a consistent way of generating endpoints. The idea is - to define controllers, we will be forced to think in terms of entities, which in turn facilitates greater control on CRUD operations.

Step 5 - generate controllers with entities

We are good to have our first entity based controller. Let's modify the index.js as follows:

...
const { getController } = require('_controllers');
...
app.get('/healthcheck', (req, res, next) => {
    res.send('OK');
});

app.use(...getController(new EntityOne()));

app.use('/', (req, res) => {
    res.send(`${req.originalUrl} can not be served`);
});

Once we plug this in, EntityOne would expose four routes as:

POST /EntityOne
GET /EntityOne
PUT /EntityOne
DELETE /EntityOne

That's all for this one. In the next one, we will add more flavours to our setup.

Next: Integrate Node.js backend with MongoDB

Learn to build with MERN stack

Chiranjib — Mon, 16 Jan 2023 07:27:25 +0000

Assumption: You already know a few things, viz:

Basics of JavaScript, and that it has no connection to Java at all
What servers are, and the language(s) they speak. For this one, you only need to know about HTTP
Node.js exists, and that it uses JavaScript. Also, you should have installed it already. If not, here's the link that saves you a search: Node.js download
Facebook is not just good for mindless scrolling, it also gave us React.js
MongoDB is a database that loves JSON and isn't serious about relationships

This guide is a stepping stone for wannabe developers who have chosen to build with the MERN stack. I know that Typescript is a thing, but to keep things introductory I have gone with JavaScript. Maybe people will tell you that you are not cool enough if you don't use Typescript, but my response to such remarks usually is - Users of a website don't care if you have used JavaScript or C; what matters to them is whether the button does what it should do when they click it

Before beginning, I feel I should announce that the series of posts I am linking later, do not go into great detail of the concepts but do touch a lot of them. I have tried to mention the "why" in mostly, but even then if some part leaves you wondering about stuff in general, please do reach out and I'll try to answer your question.

Alright, enough preaching! Crack your knuckles and let's get into it. We will begin by setting up a Node.js backend

Next: Setting Up a Node.js Backend