DEV Community: AkankshaPriyadrshini

Node JS: How to access MySQL remotely using SSH.

AkankshaPriyadrshini — Tue, 16 Jun 2020 05:56:06 +0000

Recently I was working on a Data Migration Project. For one of the cases, I had to connect to a MySql database through an SSH Tunnel.

Here, I will show you how to get it done quickly.

We will be using ssh2 and mysql2 npm packages. So, make sure to install these.

Let’s get started by writing our database config file.

// dbConfig.js
// define connection config for the database
const dbServer = {
    host: process.env.DB_HOST,
    port: process.env.DB_PORT,
    user: process.env.DB_USERNAME,
    password: process.env.DB_PASSWORD,
    database: process.env.DB_DATABASE
}
// define connection config for the ssh tunnel
const tunnelConfig = {
    host: process.env.DB_SSH_HOST,
    port: 22,
    username: process.env.DB_SSH_USER,
    password: process.env.DB_SSH_PASSWORD
}

If you have to connect to SSH using a permission file, use privateKey key in place of password in the tunnelConfig object.

Example:

const tunnelConfig = {
    host: process.env.DB_SSH_HOST,
    port: 22,
    username: process.env.DB_SSH_USER,
    privateKey:
    require('fs').readFileSync('<path to your permission file>')
}

Now, we will specify the forward configuration for SSH.

// dbConfig.js
...
const forwardConfig = {
    srcHost: '127.0.0.1', // any valid address
    srcPort: 3306, // any valid port
    dstHost: dbServer.host, // destination database
    dstPort: dbServer.port // destination port
};

We are good to set up an SSH connection now.

// dbConfig.js
const mysql = require('mysql2');
const { Client } = require('ssh2');
// create an instance of SSH Client
const sshClient = new Client();
...
const SSHConnection = new Promise((resolve, reject) => {
    ssh.on('ready', () => {
        ssh.forwardOut(
        forwardConfig.srcHost,
        forwardConfig.srcPort,
        forwardConfig.dstHost,
        forwardConfig.dstPort,
        (err, stream) => {
             if (err) reject(err);

            // create a new DB server object including stream
            const updatedDbServer = {
                 ...dbServer,
                 stream
            };
            // connect to mysql
            const connection =  mysql.createConnection(updatedDbServer);
            // check for successful connection
           //  resolve or reject the Promise accordingly          
           connection.connect((error) => {
            if (error) {
                reject(error);
            }
            resolve(connection);
            });
       });
    }).connect(tunnelConfig.sshConfig);
});

Final Database Config File:

const mysql = require('mysql2');
const { Client } = require('ssh2');
const sshClient = new Client();
const dbServer = {
    host: process.env.DB_HOST,
    port: process.env.DB_PORT,
    user: process.env.DB_USERNAME,
    password: process.env.DB_PASSWORD,
    database: process.env.DB_DATABASE
}
const tunnelConfig = {
    host: process.env.DB_SSH_HOST,
    port: 22,
    username: process.env.DB_SSH_USER,
    password: process.env.DB_SSH_PASSWORD
}
const forwardConfig = {
    srcHost: '127.0.0.1',
    srcPort: 3306,
    dstHost: dbServer.host,
    dstPort: dbServer.port
};
const SSHConnection = new Promise((resolve, reject) => {
    ssh.on('ready', () => {
        ssh.forwardOut(
        forwardConfig.srcHost,
        forwardConfig.srcPort,
        forwardConfig.dstHost,
        forwardConfig.dstPort,
        (err, stream) => {
             if (err) reject(err);
             const updatedDbServer = {
                 ...dbServer,
                 stream
            };
            const connection =  mysql.createConnection(updatedDbServer);
           connection.connect((error) => {
            if (error) {
                reject(error);
            }
            resolve(connection);
            });
        });
    }).connect(tunnelConfig.sshConfig);
});

Now, SSHConnection Promise can be used wherever required.

Originally published on my website.

Spin up your first Kubernetes Cluster in under 30 mins.

AkankshaPriyadrshini — Fri, 22 May 2020 08:19:36 +0000

Kubernetes is an open-source system for automating deployment, scaling, and managing containerized applications.

It provides us with a platform to orchestrate different microservices or systems while deploying. It serves a load of features to manage our resources efficiently. To containerise an application, we need a base image which can be used while creating an instance of the container.

Docker Hub is the centralised container image registry where we can host our images. These images can then be used for development.

Scope of the Article:

Here, we will look at the following things:

Package application in a Docker container image.
Host the image on Docker Hub.
Set up a Kubernetes Cluster locally using MiniKube.
Deploy the image on a locally running Kubernetes cluster.

For the purpose of this article, let's quickly create a sample application in Node.js
Create a directory for the application where all the files will reside.

mkdir hello-world

Now, we will create package.json file to describe the app and its dependencies.

// package.json
{
    "name": "docker_sample_app",
    "version": "1.0.0",
    "description": "Node.js on Docker",
    "author": "akanksha priyadarshini",
    "main": "server.js",
    "scripts": {
        "start": "node server.js"
    },
   "dependencies": {
        "express": "^4.16.1"
   }
}

Run npm install to install all the dependencies for our app.

Create a server.js file that will define our application.

// server.js
'use strict';
const express = require('express');
// Constants
const PORT = 8080;
const HOST = '0.0.0.0';
// App
const app = express();
app.get('/', (req, res) => {
    res.send('Hello world');
});
app.listen(PORT, HOST);
console.log(`App is running on http://${HOST}:${PORT}`);

Now, let's create a file called Dockerfile.
A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.

touch Dockerfile

In this file, first we will define from what we want to build the image from. Here, we will use node version 10 available from the Docker Hub.

FROM node:10

Next, we will specify the directory where our application code will reside inside the image. If the directory doesn't exist it will be created.

WORKDIR /home/node/app

Copy package.json and package-lock.json and install the dependencies.

COPY package*.json ./
RUN npm install

To bundle the app's source code inside the docker image, we can use COPY command.

COPY . .

Since our app binds to port 8080. We will expose this port to be mapped by docker.

EXPOSE 8080

Lastly, specify the command to run the application.

CMD [ "node", "server.js" ]

Final Dockerfile:

// Dockerfile
FROM node:10
WORKDIR /home/node/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 8080
CMD [ "node", "server.js" ]

Let's create .dockerignore file.

// .dockerignore
node_modules
npm-debug.log

Before we move further, head on to Docker Hub, create an account if you don't already have one.

Let's create a repository named hello-world here to keep our application's images to be used and shared.

Build the docker image.

// sample app directory where Dockerfile is present
cd hello-world
docker build -t <your username>/hello-world .

The -t flag is used to tag the image so that it's easier to find.
Run the command docker images to see the list of images. It should contain the image we built above.

Run the image locally and verify that everything is working fine.

docker run -p 8000:8080 -d <your username>/hello-world

-p flag redirects a public port to a private port inside the container.
docker ps will give the list of running containers images.

In this example docker mapped the 8080 port inside of the container to the port 8000 on our machine.

Test:

curl -i localhost:8000

We have successfully built and tested our docker image.

Let's push the locally created image to the repo we created on Docker Hub using the following commands.

// docker tag <local-image>:<tagname> <new-repo>:<tagname>
docker tag <your-username>/hello-world:latest <your-username>/hello-world:latest
// docker push <new-repo>:<tagname>
docker push <your-username>/hello-world:latest

We can check our repo on Docker Hub to verify the container image upload. In the tags section, we can see the recently pushed tag.

We can use these images for development and sharing.

Now, we will look at how to set up a Kubernetes cluster locally and deploy our application's docker image.

Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit.

We will use Minikube to run Kubernetes cluster locally. You can check out how to install Minikube here.

Start minikube and create a cluster.

minikube start

You can check out more info about starting the cluster here.

To interact with the cluster, we need a command-line tool called kubectl . Install and set up kubectl from here.

The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.

Create a Kubernetes Deployment and use our application's docker image with it.

kubectl create deployment hello-kubernetes --image=<your-username>/hello-world

Output:

deployment.apps/hello-kubernetes created

To access the hello-kubernetes Deployment, we will expose it as a Service.

kubectl expose deployment hello-kubernetes --type=NodePort --port=8080

Output:

service/hello-kubernetes exposed

The pod hello-kubernetes is now launched. It takes a few seconds for the pod to get to the running state.

A Pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers.

You can run the following command to get the list of pods.

kubectl get pods

Get the url of the exposed service.

minikube service hello-kubernetes --url

Now, you can curl to the url and verify the deployment.
And we are done!

But you don't have to stop here. Continue to learn and test out different features supported by Kubernetes 😊

Originally Published on my website.

Handling Nested Db Transactions using CLS

AkankshaPriyadrshini — Thu, 21 May 2020 06:13:58 +0000

Before we start, let me briefly tell you about one of the issues we faced @Casaone. Once an order gets placed, we do a couple of associated actions like inventory allocation. Inventory Allocation performs a set of operations in a nested transactions. At times we encountered situations where the inventory allocation failed but other warehouse transactions (inside the main transaction) was successful. This led to a lot of confusion while debugging what went wrong.

What we really wanted is for the entire transaction to roll back if anything goes wrong while allocating the inventory. A change should reflect everywhere or nowhere at all.

Nested Database Transactions

function Func1() {
    Transaction1(() => {
        Func2()
        // other set of actions.
    });
};



function Func2() {
    Transaction2(() => {
       // set of actions
    });
};

Let’s assume we have some function Func1 and it has some code to perform a set of actions inside a Db Transaction (Transaction1). One of the actions inside it calls another function named Func2 which has another set of actions to be performed inside a transaction (Transaction2).

Now, imagine a case where the transaction inside Func2 gets committed but for some reason transaction inside Func1 rolls back. This could lead to different kinds of problem like data inconsistency.

One of the ways we could solve this is by passing the Transaction Object from parent to the child. We can then use the same transaction object instead of creating a new one. But, imagine having to do this at all the places. It’s quite cumbersome. This is where CLS comes at our rescue.

Continuation-Local Storage or CLS

In multithreaded languages like JAVA, global data can be associated or attached with each thread using Thread Local Storage. This however is not helpful in a single-threaded and asynchronous behaviour of Node JS. This is where CLS or Continuation-Local Storage comes into picture.

CLS enables us to attach the data to the current asynchronous execution context. Values in CLS are grouped into namespaces. These values are available to us until all the functions, called synchronously or asynchronously from the original function have finished executing.

We can automatically pass transaction object to all queries using CLS. Let’s look at how to do this. I will use Sequelize as an ORM for the purpose of this article.

cls-hooked is the module we will be using. Internally it uses async_hooks which keeps an eye on the context changes. Accordingly, it loads and unloads the data attached with it.

async_hooks module provides an API to track asynchronous resources. It helps in registering callbacks which is used to track the entire life of an asynchronous resource.

Let’s import the required packages.

const cls = require('cls-hooked');
const Sequelize = require('sequelize');

Now, we will create a namespace for keeping values in CLS using createNamespace method and ask Sequelize to use it. The level of grouping or the namespace depends on the use case.

const namespace = cls.createNamespace('your-namespace');
Sequelize.useCLS(namespace);

Now, whenever a db transaction will start, the details of that transaction will be stored against a key transaction in the particular namespace.

The idea is to execute a set of actions (task) under the same transaction object (if exists) else create a new one.

const sequelize = new Sequelize();
const db = {};
db.sequelize = sequelize();
// check if transaction object exists in the namespace else create a new one
db.transaction = (task) => {
    return namespace.get('transaction')
        ? task()
        : sequelize.transaction(task);
};

Now we can use our db.transaction wherever required.

const db = require('../db/models')
function Func1() {
    db.transaction(() => {
        Func2()
        // other set of actions.
    });
};
function Func2() {
    db.transaction(() => {
       // set of actions
    });
};

Notes:

async_hooks used with CLS might have some impact on performance. You should evaluate whether to go with it or not according to the use case. You can check out the demo by Benedikt Meurer on performance comparison with and without async_hooks.

Resources

Originally published on my website.