DEV Community: Rodel Talampas

Singleton Connection with Transactions in MongoDB

Rodel Talampas — Sat, 23 Sep 2023 00:05:47 +0000

I was looking at some patterns on how to use MongoDB Transactions but I want it to be a common wrapper and should be simple enough for nodejs beginners (like me) to understand. I saw some functions accepting another, function with multiple parameters, as a parameter, but you need to take care of overloading cases.

Thinking of a simple solution, why not just wrap all arguments into a single variable like a map. This may not work for others but certainly works on our case.

First, I need to make sure that I only have 1 DB Connection always. Hence I created a Singleton class.

const mongoose = require('mongoose');
...
...

// Define Singleton DB Connection Class
class Connection {
  constructor() {
    if (!Connection.instance) {
      mongoose.connect(config.mongoose.url, config.mongoose.options);
      this.conn = mongoose.connection;
      this.conn.on('error', () => logger.error.bind(console, 'connection error'));
      this.conn.on('open', () => logger.info(`Connected to MongoDB ${config.mongoose.url} for Transaction...`));
    }
  }
}

module.exports = new Connection();

I will use this class in every part of my systems and it will ensure that I won't be having multiple cases of DB connection.

Now to create a transaction wrapper, I do this. This is a self-managed transaction meaning I dont need to call commit and rollback. But if you want to manually handle each commit and rollback, you may do so here as well.

const db = require('./connection');

const withTransaction = async (fn) => async (map) => {
  const session = await db.conn.startSession();
  await session.withTransaction(fn(map), { session });
  session.endSession();
};

module.exports = withTransaction;

To call do something like this:

async function saveFunction(map) {
  //.. implementation
  const { var1, var2 } = map;
  // use the var1, var2 here ....
  // call mongodb CRUD here 
}

// service call
const save = async (map) => {
  withTransaction(saveFunction(map));
};

Hoping to get some comments so that I can improve this wrapper in the future.

Authorisation using NodeJS, AzureAD and MongoDB (Part 1 - Design)

Rodel Talampas — Tue, 17 Jan 2023 01:04:19 +0000

Application administrators use roles to group together permissions and assign them to individual users or groups of users. These permissions determine the actions that a user is able to take within a particular software. The roles that are assigned to a user depend on their tasks within the application itself. For instance, in a simple inventory application, users who are analysts might only need permissions to browse and download, but not to modify or change information. However, in an analyst group, a senior analyst may be granted full permissions to allow them to modify existing data or create/upload new inventory data.

In general, roles are used to make it easier to manage permissions by allowing administrators to assign a set of permissions to a user or group of users all at once, rather than having to assign individual permissions one at a time. This can make it more efficient to set up and manage access controls within an application.

A role is therefore defined as a collection of permissions that are grouped together and assigned to a user or group of users. A permission, on the other hand, is a specific action or set of actions that a user is allowed to take within a particular application.

I am designing an integration with AzureAD, and MongoDB collections using NodeJS. Roles/Groups will be created in AzureAD that will be synced in MongoDB for application usage. Each role will have its own permissions and Users can have one or more roles.

Data Definition

Permission Schema

const permissionSchema = mongoose.Schema(
  {
    permission: {
        type: String,
        unique: true,
        required: true
    }
  }
  {
    timestamps: true,
  }
);

const Permission = mongoose.model('Permission', permissionSchema);

Role Schema

const roleSchema = mongoose.Schema(
  {
    role: {
        type: String,
        unique: true,
        required: true
    },
    permissions: [
      {
        permissionId: {
          type: mongoose.Schema.Types.ObjectId,
          ref: 'Permission',
        },
      },
    ]
  }
  {
    timestamps: true,
  }
);

const Role = mongoose.model('Role', roleSchema);

User Schema

const userSchema = mongoose.Schema(
  {
    firstName: {
      type: String,
      required: true,
    },
    lastName: {
      type: String,
      required: true,
    },
    phone: {
      type: String,
      required: true,
    },
    email: {
      type: String,
      required: true,
      unique: true,
      lowercase: true,
    },
    roles: [
      {
        roleId: {
          type: mongoose.Schema.Types.ObjectId,
          ref: 'Role',
        }
      },
    ],
  },
  {
    timestamps: true,
  }
);

const User = mongoose.model('User', userSchema);

Management Process

Sync Process

On Load

GIT Flow

Rodel Talampas — Sun, 15 Jan 2023 03:58:58 +0000

Gitflow Workflow is a Git workflow that helps with continuous software development and implementing DevOps practices. It was first published and made popular by Vincent Driessen at nvie. The Gitflow Workflow defines a strict branching model designed around the project release. This provides a robust framework for managing larger projects. This is suited for projects that releases are based on a schedule.

Gitflow workflow is not for everybody. But for those who loves and using it below are some shortcuts that can help you understand what is this workflow.

Installing git-flow plugin for git client is plain and simple. For mac-os, easiest to install is brew install git-flow.

Basic Commands and Configuration

git flow init
git flow feature start/finish
git flow release start/finish
git flow hotfix start/finish
git merge

Basic Configuration Flow

Concurrent Configuration Flow

The overall flow of Gitflow is:

A develop branch is created from main.
A release branch is created from develop.
Feature branches are created from develop.
When a feature is complete it is merged into the develop branch.
When the release branch is done it is merged into develop and main.
If an issue in main is detected a hotfix branch is created from main.
Once the hotfix is complete it is merged to both develop and main and/or feature branches.

Git vs Git Flow Commands

These are base guidelines that corresponds to some of the git commands used by the git-flow wrapper client. These commands are not fixed in stone. Depending on situations, there are steps that can be altered or change within the course of the development.

Create a develop branch on your local if one doesn’t exists in the repo.

Initialisation (specific for git flow only)

$ git flow init
Which branch should be used for bringing forth production releases?
   - develop
   - master
Which branch should be used for integration of the "next release"?
   - master
   - develop
Branch name for "next release" development: [develop]

How to name your supporting branch prefixes?
Feature branches? [feature/]
Bugfix branches? [bugfix/]
Release branches? [release/]
Hotfix branches? [hotfix/]
Support branches? [support/]
Version tag prefix? []
Hooks and filters directory? [../../.git/hooks]

Creating feature branches

git flow feature start <branch>

git checkout develop                     # this switches the branch to the develop branch, do a `git pull` to make sure it is latest
git checkout -b feature/<branch>         # this creates a branch named 'feature/<branch>'
git push --set-upstream origin feature/<branch>   # push changes to git repo to a new 'feature/<branch>'

Completed a feature

git flow feature finish <branch>

git checkout feature/<branch>            # this switches the branch to the 'feature/<branch>' if you are not in it, do a `git pull` to make sure it is latest
git checkout develop                     # this switches the branch to the develop branch, do a `git pull` to make sure it is latest
git merge feature/<branch>               # merge feature/branch to develop
git push                                 # push changes to git repo in develop branch

Creating a release branch

git flow release start <branch>

git checkout develop                     # this switches the branch to the develop branch, do a `git pull` to make sure it is latest
git checkout -b release/<branch>         # this creates a branch named 'release/<branch>'
git push --set-upstream origin release/<branch>   # push changes to git repo to a new 'release/<branch>' and this will trigger the BUILD to QA

Release Completed

git flow release finish <branch>

git checkout release/<branch>            # this switches the branch to the 'release/<branch>' if you are not in it, do a `git pull` to make sure it is latest
git checkout develop                     # this switches the branch to the develop branch, do a `git pull` to make sure it is latest
git merge release/<branch>               # merge release/branch to develop
git push                                 # push changes to git repo in develop branch
git checkout main                        # this switches the branch to the main branch, do a `git pull` to make sure it is latest
git merge release/<branch>               # merge release/branch to main
git push                                 # push changes to git repo in main branch
git push --tags                          # push tags
git push -d origin release/<branch>      # delete remote branch

Creating a hotfix

git flow hotfix start <branch>

git checkout main                        # this switches the branch to the main branch, do a `git pull` to make sure it is latest
git checkout -b hotfix/<branch>          # this creates a branch named 'hotfix/<branch>'
git push --set-upstream origin hotfix/<branch>   # push changes to git repo to a new 'hotfix/<branch>' and this will trigger the BUILD to QA

Hotfix Completed

git flow hotfix finish <branch>

git checkout hotfix/<branch>             # this switches the branch to the 'hotfix/<branch>' if you are not in it, do a `git pull` to make sure it is latest
git checkout develop                     # this switches the branch to the develop branch, do a `git pull` to make sure it is latest
git merge hotfix/<branch>                # merge hotfix/branch to develop
git push                                 # push changes to git repo in develop branch
git checkout main                        # this switches the branch to the main branch, do a `git pull` to make sure it is latest
git merge hotfix/<branch>                # merge hotfix/branch to main
git push                                 # push changes to git repo in main branch

Enabling PlantUML in Visual Studio Code

Rodel Talampas — Sun, 15 Jan 2023 03:42:45 +0000

PlantUML is a component that allows you to quickly write the following diagrams in plain and simple intuitive language - a.k.a. Diagram-As-Code.

Sequence diagram
Usecase diagram
Class diagram
Object diagram
Component diagram
Deployment diagram

There are many similar tools out there. PlantUML is free to use and was created using Java and GraphViz. A simple way of using it is to download the plantuml.jar and run it to open PlantUML's graphical user interface. There is no need to unpack or install anything.

Most developers use Visual Studio Code, why not install its plugin:

Browse the plugin and click install. After installing the plugin, there is a need to install GraphViz binary. You can follow the link to install it. For mac users, easy to run brew install graphviz.

When all done, create a test file test.puml and add the following code



@startuml Basic Sample
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml

Person(admin, "Administrator")
System_Boundary(c1, "Sample System") {
    Container(web_app, "Web Application", "C#, ASP.NET Core 2.1 MVC", "Allows users to compare multiple Twitter timelines")
}
System(twitter, "Twitter")

Rel(admin, web_app, "Uses", "HTTPS")
Rel(web_app, twitter, "Gets tweets from", "HTTPS")
@enduml

Press Option + D in Mac or Alt-D for other OS's, should give the following output:

Blackbox - Secrets amongst your code

Rodel Talampas — Sun, 15 Jan 2023 03:29:21 +0000

One of the major concerns amongst developers is how to store shared secrets. Storing secrets in a config file along with your source code is problematic as it can compromise privacy. Storing it outside without proper process or documentation can tend to be forgotten (not saying this is not good as using SAAS tools like Vault is the way to go). But in case you have a limited budget and limited capability, using GPG is the way to go. Here comes Blackbox.

For any secrets you want to store alongside the source code, we should limit the risk of intentional or accidental sharing by encrypting secrets.

You should encrypt these data in your repositories:

SSH keys
Private keys
Usernames and passwords

Installation
Blackbox is a GPG-based encryption tool for Git. Assuming Homebrew is installed in a mac or linux machine, the command below is sufficient. Otherwise look for the Blackbox link above for more information

brew install blackbox

GPG Keys
GnuPG, also known as GPG, is a free, full implementation of the OpenPGP standard as outlined in RFC4880 (also known as PGP). It enables you to encrypt, sign and authenticate data and communication.

gpg --full-generate-key

When key has been generated, look through the messages that displayed on the screen and find your Key ID.

gpg: key <key_id> marked as ultimately trusted

Generate a revocation certificate for the primary public key. If your private key is compromised or lost, this revocation certificate may be published to notify others that the public key should no longer be used.

gpg --output revoke.asc --gen-revoke <key_id>

The revoke.asc file is stored in your user directory. Store the certificate key somewhere safe, ideally keypass or macpass.

Add your public key to the keys server so that others have verify it.

gpg --keyserver=pgp.key-server.io --armor --send-keys <key_id>

Browse to pgp.key-server.io and search for your name to see if your gpg key is there or not.

In case pgp.key-server.io is not registering your key, use other public known keyservers:

Using Blackbox

Make a repository a blackbox repo

$ blackbox_initialize 
Enable blackbox for this git repo? (yes/no) y
VCS_TYPE: git

NEXT STEP: You need to manually check these in:
      git commit -m'INITIALIZE BLACKBOX' .blackbox /talampas/development/repo/.gitignore

Add a new user in the blackbox repo

There is a need to look for the user's key to verify and add it in your keychain

gpg --keyserver keys.openpgp.org --search-key <personA@email.com | fingerprint>

Add the user as an admin of blackbox for that repo.

blackbox_addadmin <personA@email.com>

Update all secret files to incorporate the new key registered

blackbox_update_all_files

Do a commit / push for the repo

# the next two lines are not required if no other files where edited
git add .
git commit -m"Add new user - personA"
# Required to push changes
git push

Updating Blackbox Encrypted files

If there is a need to update the secrets in the encrypted file, follow the steps below

Open the file for editing

blackbox_edit_start <filename>.gpg

End the Editing process

blackbox_edit_end <filename>

Same as above, commit and push when you're done

Developer Setup for Mac

Rodel Talampas — Sun, 15 Jan 2023 02:49:44 +0000

You have a new MacBook! Congrats!

You need to setup your mac to be Developer Ready and don't know what to do. You are in the right place. I have compiled a list of installation scripts for a typical Developer to kick start a Developer Machine.

macOS Updates
Before installing any applications you require, it is a good idea to update the Operating System. As much as possible, I try to have the latest version of the MacOS with patches.

XCode
First step, you need to install XCode which is Apple's integrated development environment for MacOS. You can download it in the Apple Store free of charge. There is a command line instruction to install xcode. Open a new terminal and type the code below.

xcode-select --install

Homebrew
I pretty much use Homebrew or brew to install anything on my MacBook and Linux Machines. Run the following commands to download and install brew and upgrade it, though there should not be anything to upgrade. No harm doing though.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew upgrade

Python 3 and JQ
Next to Java, Python is probably my next go to programming language. I used python a lot to build client tools and serverless applications. JQ is a tool to parse JSON Strings and I predominantly use this with python and aws. VirtualEnv is my favorite environment configuration for python. Python virtualization is a must if you are going to develop using python. Python is also a requirement for some client tools like awscli.

brew install python3
curl https://bootstrap.pypa.io/get-pip.py | python3
pip install virtualenv
brew install jq

AWS
I work mostly with AWS Cloud hence I require AWS Client that uses Python. If you are not in AWS, you may skip this part. This command will help you install and upgrade your aws client. I have both added awscli and awscli-local.

brew install awscli
brew link awscli --overwrite 
pip install awscli-local

Docker Desktop
Since Mac-OS is not a linux distro, it is impossible to install docker base binaries. Don't fear though, docker created a desktop installer for mac here.

Localstack
Since AWS is a paid service, creating infrastructure usually cost $$$ if you are unable to use the free-tier. To test out my infrastructure-as-code scripts, I spin up a Mock AWS Service called localstack.

pip install localstack

Terminal
MacOS' terminal is nice, no complains about it. As a developer, sometimes I would like to work on multiple Terminals but within the same window. Hence, I require some Split functionality and iTerm2 has that capability. ZSH is the prefered console of MacOS terminal nowadays.

chsh -s /bin/zsh
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
brew install iterm2

NodeJS
As much as I don't want to tackle NodeJS, this programming language is pretty much use anywhere from backend processing to UI development using frameworks such as react or angular or vue. Somehow, I got the hang of this programming framework as well. I've been using this in my side projects. Yarn is my favourite package manager. You can install any PM you want.

brew install node
brew install nvm
brew install yarn

Serverless Framework
Between AWS Severless Application Model (SAM) and Serverless Framework (SLS), I prefer the latter. Both framework patterns their syntax from AWS Cloudformation, but I firmly believe Serverless Framework is more mature. Don't get me wrong, I love AWS. SLS is my own choice.

curl -o- -L https://slss.io/install | bash

Infrastructure As Code
I've been using Cloudformation ever since I had moved to the cloud. But when I've learned Terraform, I seldom touch Cloudformation unless I am working with SLS above.

brew install terraform

Git and Git Flow
In one way or another, you might be using a Source Code Versioning System like github or gitlab. A git client is required.I have added a plugin called git flow for the purpose of following a release framework.

brew install git
brew install git-flow

Database Connectivity Tool
We use databases to persist data. I've used a lot of databases in my career, from MS SQL Server to Oracle to Postgres or MySQL. I don't have a clear favourite but for this tutorial, I will go with the easiest postgres. Take note, this is only a connection tool, you may ask how do you install your database? As I've mentioned above, I would containerise my databases, read it here.

brew install libpq

Miscellaneous Tools
The rest of the apps in this code block are optional. Some are required due to the fact that I am using it like gpg for encryption and java being my favourite programming language. It is to your discretion if you will install them or not. Atom here is a good IDE but since Microsoft bought Github, it is not being developed anymore and Visual Studio Code Express is the preferred IDE.

brew install atom
brew install kubectx
brew install gpg
brew install blackbox
brew install java
brew cask install eclipse-java
brew cask install postman
brew install macpass
brew install credstash

Full Script

For all the apps that can be installed using the terminal, I've compiled a full list below.

# Install Homebrew - preferred apps installation manager
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew upgrade

# Install Python 3
brew install python3
# update pip 
curl https://bootstrap.pypa.io/get-pip.py | python3
# install virtualenv for python, if you dont use python no need to install this
pip install virtualenv
# install jq
brew install jq

# Install AWS Client using Homebrew
brew install awscli
# if there is already an install client, ovewrite its softlink
brew link awscli --overwrite 
# install aws client for localstack
pip install awscli-local


# install docker using docker installer
brew cask install docker

# install localstack
pip install localstack

# preferred mac terminal
brew install iterm2

# CHANGE TO zsh, if you want too.
chsh -s /bin/zsh
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

# all about NODEJS
brew install node
brew install nvm
brew install yarn
# if your office has certificate restrictions, you may want to remove ssl restriction
yarn config set strict-ssl false

# for AWS lambda deployment, we prefer Serverless Framework
curl -o- -L https://slss.io/install | bash

# for Infrastructure as Code
brew install terraform

#install git
brew install git

# install git flow
brew install git-flow

# postgres tool
brew install libpq 

# this is my preferred editor, if you dont like it dont install it
brew install atom

# some passwords used for OPs are in keepass and macpass is the keepass version for mac
brew install macpass

# we do use kubernetes, this is required to connect to it
brew install kubectx

# install gpg 
brew install gpg

# install Java
brew install java
brew cask install eclipse-java
brew cask install postman

# password tools
brew install blackbox
brew install credstash

Postgres Database in a Docker Container

Rodel Talampas — Sun, 15 Jan 2023 02:33:43 +0000

You want to install a Database in your machine for development purposes, correct? I would advise not to but instead install Docker and create a Dockerise Database, Postgres for example.

Why? Simple, you can spin off multiple instances of a database and you can easily drop the containers once you are done with it.

PostgreSQL is an open-source, object-relational database with strong capabilities. It allows for more sophisticated data types and object inheritance, but this increased functionality also increases complexity. It uses a single storage engine that guarantees ACID compliance.

docker run --name mydb -e POSTGRES_PASSWORD='mypassword' -d -p 5432:5432 postgres

Parameters	Description
name	The name of your container
-e POSTGRES_PASSWORD='mypassword'	This is the password of the admin user 'postgres'.
-p <local port>:<container port>	Port mapping between container and local machine
-d	Run the process in the background
postgres[:<version>	The container image to download from dockerhub. Add :<version> to specify the version of the container

Create the Database

Creating the database is as simple as running a createdb command. This command is part of the postgres client library libpq.
Make sure that an environment variable 'PGPASSWORD' has been set to have the value of the password
defined in the docker container command above.

export PGPASSWORD=mypassword
createdb -h localhost -U postgres mydatabase

-h <hostname>
This specifies the hostname of the database server. Since this is a docker container, it is default to be the localhost.
-U postgres
The default postgres admin user.
<database name>
The name of the database to be created, here it's 'mydatabase'

Create Database Users

You don't use the default user to connect to the database regularly, specially in a Production Environment. You need to create users. The psql command is part of the postgres client library 'libpq'.

psql -h localhost -U postgres -d mydatabase -c "create user db_admin superuser;"
psql -h localhost -U postgres -d mydatabase -c "create user write;"
psql -h localhost -U postgres -d mydatabase -c "create user readonly;"

-h <hostname>
This specifies the hostname of the database server. Since this is a docker container, it is default to be the localhost.
-U postgres
The default postgres admin user.
-d <databae name>
The database you want to connect to.
-c <sql statements>
The sql statement you want to run. In this case the create user statement.

Assign Permission

Creating user is not enough. You will not be able to use the user you created without giving them permission to access the database.

psql -h localhost -U postgres -d mydatabase -c "GRANT ALL PRIVILEGES ON DATABASE mydatabase TO write;"
psql -h localhost -U postgres -d mydatabase -c "GRANT SELECT ON ALL TABLES IN SCHEMA myschema TO readonly;"

Full Script

I've compiled all of the above commands into one script below.

  docker run --name mydb -e POSTGRES_PASSWORD='mypassword' -d -p 5432:5432 postgres    
  # add in `:<version>` to install a specific version of postgres (e.g. :11)
  brew install libpq # install postgres client libraries if havent done so

  # PGPASWORD env variable is used by psql command to authenticate a user with the -U option
  export PGPASSWORD=mypassword

  # Create a database based on your project
  createdb -h localhost -U postgres mydatabase

  # Create users based on your project
  psql -h localhost -U postgres -d mydatabase -c "create user admin superuser;"
  psql -h localhost -U postgres -d mydatabase -c "create user write;"
  psql -h localhost -U postgres -d mydatabase -c "create user readonly;"

  # Grant users
  psql -h localhost -U postgres -d mydatabase -c "GRANT ALL PRIVILEGES ON DATABASE mydatabase TO write;"
  psql -h localhost -U postgres -d mydatabase -c "GRANT SELECT ON ALL TABLES IN SCHEMA myschema TO readonly;"