In Part 0, we got an overview of the 12-factor app methodology and its importance in building a scalable and reliable backend. In this Part, we will learn in detail about the first 4 factors and simultaneously implement them by building a blogging platform using NodeJS
1. Codebase
Nowadays, everyone is familiar with version control systems aka VCS, such as Git, Mercurial, or Subversion, Git being the most popular one. It has become second nature to us that the first thing we do after initializing our project is run git init && git add . && git commit -m "Initial commit". To understand how important VCS is let's assume we don't know about VCS or something like VCS doesn't even exist.
Let's take 4 scenarios to understand how difficulty escalates exponentially
1) Working Solo on a Project with Single Deployment
2) Working Solo on a Project with Multiple Versions and Deployments
3) Working in a Team of 4 with a Single Version and Deployment
4) Working in a Team of 4 with Multiple Versions and Deployments
Here, deployment means a version of the codebase. Single Deployment means only one version of the codebase that will be deployed. Multiple Deployments means multiple versions(including client-specific) of codebase in multiple environments(staging, UAT, production)
Scenario 1: Working Solo on a Project with Single Deployment
As a solo developer without version control, working on a project with a single deployment becomes a risky situation. Any code changes made are irreversible, and there's no safety net in case of errors. If a bug is introduced, there's no straightforward way to revert to a previous, stable state. The lack of versioning makes it challenging to track changes over time, and it's easy to lose sight of the project's evolution.
Collaboration with past iterations is impossible, making it difficult to understand the reasoning behind certain decisions or to revisit successful implementations. The absence of branching and merging capabilities means that experimenting with new features or fixes could disrupt the entire codebase.
Deployments are a nerve-wracking process. If an issue arises during deployment, rolling back to a working state is a manual and error-prone task. Without a VCS, there are no efficient means of documenting the deployed versions or managing configurations for different environments.
Scenario 2: Working Solo on a Project with Multiple Versions and Deployments
In this scenario, the absence of version control worsens the challenges faced in the first scenario. Managing multiple versions and deployments without VCS becomes a logistical nightmare. Keeping track of different code states for various features or environments is practically impossible.
Each deployment is a high-stakes endeavor, as there is no systematic way to isolate changes specific to a version or deployment. Debugging becomes a herculean task, and the risk of introducing new bugs while fixing existing issues looms large. Without branching, trying out experimental features or implementing temporary fixes without affecting the main codebase becomes nearly impossible.
Documentation is limited to external notes or comments within the code, making it hard to understand the rationale behind specific versions or deployments. Coordinating changes across different branches or versions requires meticulous manual effort, leading to a high probability of errors.
Scenario 3: Working in a Team of 4 with Single Version and Deployment
Without version control in a team setting, collaboration turns chaotic. Coordinating efforts among team members becomes a communication-heavy process, relying on constant updates and manual file sharing. The risk of overwriting each other's work is ever-present, and resolving conflicts is an arduous manual task.
There's no clear history of changes, making it difficult to attribute modifications to specific team members, there will be blame games of faulty code as we won't be able to pinpoint who introduced the faulty code. Debugging and troubleshooting are cumbersome, as there's no easy way to identify when and why a particular change was made.
Deployment is a risky and unpredictable process, with no efficient rollback mechanism. Coordinating releases requires meticulous planning and synchronization among team members, increasing the likelihood of errors and downtime.
Scenario 4: Working in a Team of 4 with Multiple Versions and Deployments
In a team setting with multiple versions and deployments but without version control, the challenges escalate exponentially. Collaboration becomes a logistical nightmare, with each team member potentially working on a different version or deployment. Coordinating changes and ensuring a cohesive codebase is a constant struggle.
Without branching and merging capabilities, integrating features or fixes from different team members becomes an error-prone process. The risk of introducing conflicts and breaking the codebase during integration is high. Keeping track of changes across multiple branches or versions requires meticulous manual effort and is prone to oversight.
Deployments are high-stakes events, and rolling back to a stable state in case of issues is a manual and time-consuming task. The lack of versioning makes it challenging to manage configurations for different environments, leading to potential deployment errors.
It was just a Nightmare
To sum it up, without version control, it's like navigating a coding maze with a blindfold. Adding a version control system is like turning on the lights — suddenly, everything just clicks. I hope after reading this, your appreciation for VCS increases tenfold
Now that we know what it's like without VCS, let's create our project and initialize it with git. Our project will be a simple blog application that will be built using ExpressJS, Typescript and MySQL. We will name our project will be bloggo.
Let The Coding Begin
Create a Project folder with README.md
mkdir bloggo && cd bloggo && touch README.md && echo -e "# Bloggo \n\nSimple Blog Application" > README.md
Initialize Git Repo
git init && git add . && git commit -m "initial commit"
Link to commit for changes up to this section https://github.com/sumitbhanushali/bloggo/commit/a5f14218eab0fb5a84fdf1e31fe31965925e81d1
2. Dependencies
12 Factor app is easy to setup and run. This will be possible when all dependencies are declared in the dependency declaration manifest. These dependencies will be isolated i.e these dependencies will only apply to the current app and no global or system-wide package will be accessible inside it unless it is explicitly declared in the dependency declaration manifest.
This reduces friction to setup project on a developer's machine or even on production and leads to increased productivity and thus a better app will be delivered.
NodeJS has few popular package managers available like npm, pnpm and yarn. We will be using npm since it is the official package manager of NodeJS and most developers are already well-versed with it
This generates a default package.json file which will be used by npm to store metadata, dependencies and scripts
npm init -y
We will use express to create our app
npm i express
We don't want node_modules to be committed to git since it will unnecessarily increase our repo size which will make it longer to clone
echo "node_modules" > .gitignore
Our final package.json file will look like this
{
"name": "bloggo",
"version": "1.0.0",
"description": "Simple Blog Application",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"express": "^4.18.2"
}
}
Link to commit for changes in this post https://github.com/sumitbhanushali/bloggo/commit/a91c1424aa8a7279740338276af563bf1c0c5ee6
3. Config
An app’s config is everything likely to vary between deploys (staging, production, developer environments, etc) which are credentials to external services such as Database(MySQL, Redis), Cloud Providers or External APIs like Twitter, Dropbox, etc.
A beginner might hardcode such config in a codebase or might create separate versions of such file for each deployment. Such a person might not be aware of problems that can arise because of this. If such configs get in the wrong hands (hacker or fresher), that person can bring down the whole system by deleting the database or misconfigure something accidentally or purposely.
Configs should never be committed to VCS even if it is a private repository. A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.
Some teams maintain a config file with defaults and edit this file manually at the server level according to environment needs. This is a good approach but there is still room for errors.
Another approach is to batch config into named groups (often called “environments”) named after specific deploys, such as the development, test, and production environments. This method does not scale cleanly: as more deploys of the app are created, new environment names are necessary, such as staging or QA. As the project grows further, developers may add their special environments like joes-staging, resulting in a combinatorial explosion of config which makes managing deploys of the app very brittle.
The best way is to make use of environment variables which are provided by every OS out there. In NodeJS, you can access these environment variables using process.env.ENV_VARIABLE where ENV_VARIABLE can be any variable name. dotenv is a popular package to simplify using environment variables which is used by almost every NodeJS project out there. From Node v20.6.0, support for reading environment variables from .env file is added which removes the dependency from dotenv` package.
We can choose to add this file into .gitignore and pass this file manually across team members or provide a default file for staging and editing this file on sensitive(internet-facing) environments
environment variables in NodeJS are accessed from process.env
`ts
//index.js
console.log(process.env.foo);
`
node only supports reading environment variables from files having .env extension
`
//config.env
foo=bar
`
add file to .gitignore, as we don't want to commit this file in our git
bash
echo "config.env" >> .gitignore
we need to pass --env-file=config.env for NodeJS process to read our .env file
`bash
node --env-file=config.env index.js
`
Link to commit for changes in this post https://github.com/sumitbhanushali/bloggo/commit/56c3e46168a9146912db2814cb0030905d3d8300
references: https://nodejs.org/en/blog/release/v20.6.0
4. Backing Services
A backing service is any service the app consumes over the network as part of its normal operation. Examples include datastores (such as MySQL or MongoDB), messaging/queueing systems (such as RabbitMQ or Kafka), caching systems (such as Redis).
Make your app scalable
If you want your app to be scalable, you should keep components as decoupled as possible. Components like API Servers, Database, Message Brokers, Caching Servers, etc. This way you can independently scale them according to the application's needs. Our API Server should be prepared from the beginning for such changes to come Otherwise, It would take months of refactoring efforts depending on how big your codebase is.
Your App should make no distinction between local and the third party services. Each component must be pluggable where you can attach and detach them at your will. These components can be accessed via URL or some type of credentials which should be stored in the Config. This way, if your business needs require you to swap your local MySQL database instance to one managed by a third party service such as Amazon RDS, it would just require a change in URL stored in Config. The same applies to if you need to shard your database or need to replace a faulty database.
In previous posts, we initialized git repository, and did setup for adding dependencies and config. For this post, we require CRUD App Setup, scope of which is outside of this post hence I have created another post where I have added detailed setup instructions on how to setup Typescript with ExpressJS and Prisma as ORM which will communicate with our database. We will focus on connecting database using URL in this post using Prisma. You can checkout code at commit https://github.com/sumitbhanushali/bloggo/commit/03792df70faf7182c7ff88c62d9f5426d27627a9 and continue with below steps
bash
npm install prisma --save-dev
npx prisma init --datasource-provider mysql
This will generate prisma folder in our root directory where in schema.prisma will contain database connection details and prisma models which will be used to create/update tables in our database. We will add two models in schema.prisma
`ts
model User {
id Int @id @default(autoincrement())
email String @unique
name String?
blogs Blog[]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
model Blog {
id Int @id @default(autoincrement())
title String
content String?
published Boolean @default(false)
author User @relation(fields: [authorId], references: [id])
authorId Int
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
`
This model files are self-explanatory. You can check more about prisma model here: https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference
Next step is to execute prisma script to run migration which will create these model files inside our database. But before this we need to setup our mysql instance
`bash
docker pull mysql
docker run -d --name mysqldb -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql
`
This will pull mysql image from docker hub and create container from it, we have set container name as mysqldb and root password as password. Our db will listen on port 3306
`bash
docker exec -it mysqldb sh
mysql -u root -p
// enter "password" when prompted
create database bloggo;
`
This way we can attach to our db container and execute our bash command to connect to mysql and create database named "bloggo". type exit to exit from mysql and container respectively
We will update our environment variable to our credentials in config.env file
DATABASE_URL="mysql://root:password@localhost:3306/bloggo"
Next we will execute prisma script to run migrations, this command also generates prisma client files which comes in handy when using ORM
bash
npx prisma migrate dev --name init
You can now check that tables have been connected by connecting to mysql instance. Run this commands after connecting to mysql instance
`bash
use bloggo;
show tables;
describe Blog;
describe User;
`
Now we can use ORM to perform CRUD operations on our database. Let's update our route files to use prisma client to run operations.
`ts
// user.route.ts
import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();
router.get('/', async (req: Request, res: Response) => {
const users = await prisma.user.findMany();
res.send(users);
});
router.get('/:id', async (req: Request, res: Response) => {
const user = await prisma.user.findUnique({
where: {
id: parseInt(req.params.id),
},
})
res.send(user);
});
router.post('/', async (req: Request, res: Response) => {
const user = await prisma.user.create({
data: req.body
})
res.send(user);
});
router.delete('/:id', async (req: Request, res: Response) => {
await prisma.user.delete({
where: {
id: parseInt(req.params.id),
},
})
res.send(User with id: ${req.params.id} deleted);
});
`
`ts
//blog.route.ts
import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();
router.get('/', async (req: Request, res: Response) => {
const blogs = await prisma.blog.findMany();
res.send(blogs);
});
router.get('/:id', async (req: Request, res: Response) => {
const blog = await prisma.blog.findUnique({
where: {
id: parseInt(req.params.id),
},
})
res.send(blog);
});
router.post('/', async (req: Request, res: Response) => {
const createdBlog = await prisma.blog.create({ data: req.body });
res.send(createdBlog);
});
router.delete('/:id', async (req: Request, res: Response) => {
await prisma.blog.delete({
where: {
id: parseInt(req.params.id),
},
})
res.send(Blog with id: ${req.params.id} deleted);
});
`
Now, we can create user and their blog using your favorite tool like Postman.
`bash
// First we create a dummy user
curl --location 'localhost:3000/user' \
--header 'Content-Type: application/json' \
--data '{
"email": "abc@test.com"
}'
// Then blog from userId returned from previous command. In this case, authorId 1
curl --location 'localhost:3000/blog' \
--header 'Content-Type: application/json' \
--data '{
"title": "Title",
"content": "Content",
"authorId": 1
}'
`
Link to commit for changes in this post https://github.com/sumitbhanushali/bloggo/commit/a91c1424aa8a7279740338276af563bf1c0c5ee6
Up Next
This is where our Part 1 ends. We have learnt about VCS like git that how it can make a developer's life a lot simpler, about dependency declaration manifest that how it can make setting up project smoother, about storing config in env vars which makes deployment secure and benefits of using backing services as pluggable resources.
On next part we will continue with next 4 steps i.e 5. Build, Setup and Run, 6. Processes, 7. Port Binding, 8. Concurrency according to 12 Factor App's methodology
Top comments (0)