DEV Community: MD RAHIM IQBAL

Setting Up Kafka on MacOS M1 using Homebrew

MD RAHIM IQBAL — Sat, 29 Jul 2023 06:25:30 +0000

Introduction

This guide provides a step-by-step walkthrough of installing Apache Kafka on a MacOS M1 system. However, before we can install Kafka, we need to ensure Homebrew is installed on your system.

Homebrew Installation

Homebrew is a package manager for MacOS that simplifies the installation of software. To install Homebrew, open Terminal and run the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This will install Homebrew on your machine. For more information about Homebrew and how to use it, check out the official Homebrew documentation.

Kafka Installation

With Homebrew installed, we can now install Java and Kafka. Run the following commands in your Terminal:

brew install java
brew install kafka

These commands install both Java and Kafka on your system.

Running Kafka and Zookeeper

To start Kafka, we need to run Kafka and Zookeeper services separately. Open two Terminal windows for this purpose.

Start Zookeeper:

zookeeper-server-start /opt/homebrew/etc/kafka/zookeeper.properties

Start Kafka:

kafka-server-start /opt/homebrew/etc/kafka/server.properties

Make sure that both services are running successfully before proceeding.

Creating a Kafka Topic

Before interacting with Kafka using any producer or consumer API, you need to create a Kafka topic. A topic is essentially a category or feed name to which records get published. Here is how to create a topic named 'foobar':

kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic foobar

Note: We no longer use the --zookeeper flag because newer Kafka versions for MacOS do not require it.

Testing Kafka Producer and Consumer APIs

To validate your Kafka setup, let's test the Kafka producer and consumer APIs:

Open two new Terminal windows.

In the first terminal, initialize a producer console for the 'foobar' topic and send some test messages:

kafka-console-producer --broker-list localhost:9092 --topic foobar
> foo
> bar

In the second terminal, initialize a consumer console for the 'foobar' topic. This will listen to the bootstrap server on port 9092 and the 'foobar' topic:

kafka-console-consumer --bootstrap-server localhost:9092 --topic foobar --from-beginning
foo
bar

If you see these outputs, congratulations! Kafka is now set up and running smoothly on your MacOS M1 system.

Conclusion

Setting up Kafka on a MacOS M1 system is a straightforward process, especially with the help of Homebrew. With Kafka properly installed, you can now use it to manage real-time data pipelines and streams. Happy streaming!

Managing Multiple GitHub Accounts with SSH Keys on Your Device

MD RAHIM IQBAL — Fri, 28 Jul 2023 12:30:21 +0000

Introduction

Managing different GitHub accounts is a common scenario for many developers. Perhaps you maintain a personal GitHub account to showcase your school projects and a work account for your professional work. To switch between these accounts easily and securely on the same computer, you can use SSH keys. These keys serve as a secure method of authentication, much like a physical key can be used to lock or unlock a specific door.

Secure Shell (SSH) is a cryptographic protocol enabling secure remote access to servers. Widely used by developers and system administrators, SSH authenticates two parties (client and server) and encrypts data that passes between them using a pair of cryptographic keys: a public key for distribution and a private one kept secret.
You can generate your SSH keys using the ssh-keygen command:
ssh-keygen -t ed25519
This command creates a new set of keys using the ED25519 algorithm. You can specify where to save the keys and whether to use a passphrase for added security.
After generating your keys, distribute your public key to any system you need to access. When initiating an SSH session, the server uses the public key to encrypt messages in a way that can only be decrypted with the private key. The client uses the private key to authenticate its identity to the server.

Step 1: Generating SSH Keys

Generating SSH keys for each GitHub account involves creating a pair of keys – one private and one public – for each account. The private key is kept secret on your computer, and the public key is added to your GitHub account.

Open Terminal on your device and use the following commands to generate SSH keys for your personal and work GitHub accounts:

For your personal account:

ssh-keygen -t ed25519 -C "your_personal_email@example.com" -f ~/.ssh/github-personal

And for your work account:

ssh-keygen -t ed25519 -C "your_work_email@example.com" -f ~/.ssh/github-work

The -t flag specifies the type of key to create, ed25519 in this case. The -C flag lets you label the key with a comment, typically the associated email address. Finally, the -f flag lets you specify the filename of the key.

Step 2: Add SSH keys to SSH Agent

Now we have the keys but it cannot be used until we add them to the SSH Agent.

ssh-add -K ~/.ssh/github-personal
ssh-add -K ~/.ssh/github-work

Step 3: Adding SSH Keys to GitHub Accounts

The public keys (not the private ones!) generated in Step 1 need to be added to the respective GitHub accounts. This process is a way to introducing GitHub to your magic box and telling it that messages from these keys should go to your respective accounts.

To display the contents of your public key, run:

nano ~/.ssh/id_rsa_personal.pub

Copy the content and exit using ctrl+X
Go to personal GitHub account, click on your avatar > Settings > SSH and GPG keys > New SSH key.
Paste the key, give it a name and save it.

Repeat the process for your work account using the id_rsa_work.pub key.

Step 4: Configuring SSH for Different Accounts

With the SSH keys added to the respective GitHub accounts, we now need to create an SSH configuration file to map each key to the correct account.

Create and open a new config file in .ssh diectory by running:

touch ~/.ssh/config
nano ~/.ssh/config

And then add the following configurations:

# Work account
Host github.com-work
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_work

# Personal account
Host github.com-personal
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_personal

Save and exit the file (ctrl + X, then Y, then enter). SSH now knows which key to use for which GitHub account.

Step 5: Using the Configurations in Practice

With everything set up, you can now use these configurations to clone repositories, make commits, and push changes using each GitHub account.

For instance, if you wanted to clone a repository from your personal account, you would use:

git clone git@github.com-personal:username/repo.git

And to clone a repository from your work account:

git clone git@github.com-work:username/repo.git

Replace username and repo with your GitHub username and repository name, respectively.

Step 6: Git Local & Global Configurations

The next step involves telling Git who we are when we commit changes using these accounts. We do this by setting our name and email for each repositories. If you want to set this config for a specific repository only, navigate to that repository in your terminal and execute the below commands.

For the repositories have access with personal account:

git config user.name "Your Name"
git config user.email "your_personal_email@example.com"

And for the repositories have access with work account:

git config user.name "Your Work Name"
git config user.email "your_work_email@example.com"

These commands set your Git username and email respectively for a specific repository only. If you want same config globally(for all repositories) then the --global flag applies these settings across all repositories.

git config --global user.name "Your Name"
git config --global user.email "your_personal_email@example.com"

To push or pull to the correct account from the existing cloned repositories we need to add the remote origin to the project

//for repositories associated with personal account
git remote add origin git@github.com-personal:username/repo.git

//for repositories associated with work account    
git remote add origin git@github.com-work:username/repo.git

Conclusion

Managing multiple GitHub accounts on a single machine might seem complicated, but with the help of SSH keys and some simple Git configurations, you can easily switch between your accounts. This is a powerful and secure way to maintain separate identities for personal and professional projects. So go forth and code, knowing that you have a robust and flexible system for managing your GitHub accounts!

If you'd like, let's connect on LinkedIn and grow our community bigger.

Docker: The Ultimate Guide to Streamline Application Development

MD RAHIM IQBAL — Thu, 13 Apr 2023 20:34:48 +0000

In recent years, Docker has become a popular tool for developers looking to streamline their application development process. In this blog post, we will delve into the fundamentals of Docker, including its core components and how it compares to virtual machines. We will also touch on key terms and concepts that will help you understand how Docker can revolutionize your workflow.

Now what is docker?

Docker is a platform designed to simplify the process of building, running, and shipping applications. It achieves this by utilizing containerization technology, which allows developers to create isolated environments called containers for running applications.

Imagine you're moving to a new house, and you have a room with specific furniture, decorations, and settings that you want to recreate in your new place. Transporting everything individually, making sure nothing gets damaged, and setting it all up again in the new house can be a tedious and time-consuming process.

In this scenario, think of Docker as a portable room that you can use to pack all your belongings together. Docker allows you to put the furniture, decorations, and settings inside a container, which can then be sealed and transported to your new house without worrying about compatibility or damage. Upon arrival, you simply "unpack" the container, and your room is set up exactly as it was before.

In the world of software development, Docker works in a similar manner. Applications often depend on specific libraries, configurations, and runtime environments. Setting up these dependencies manually on different systems (e.g., development, testing, and production environments) can be complex and time-consuming, and may lead to inconsistencies or errors.

Docker containers encapsulate everything an application needs to run, including the operating system, libraries, dependencies, and application code. This ensures that the application runs consistently across different environments, regardless of the underlying system. With Docker, developers can build a container once and then deploy it to various stages of the development process (e.g., testing, staging, and production) without worrying about compatibility issues or discrepancies.

In summary, just as a portable room allows you to move your belongings easily and consistently between houses, Docker enables developers to build, run, and ship applications consistently across various environments, simplifying the deployment process and reducing potential errors.

Containers vs. Virtual Machines

While both containers and virtual machines (VMs) enable running applications in isolated environments, they differ significantly in their underlying architecture and resource consumption.

A virtual machine is an abstraction of hardware resources, created and managed by hypervisors such as VirtualBox, VMware, or Hyper-V (Windows-only). VMs emulate a complete system, including the operating system, and can be resource-intensive and slow to start.

On the other hand, containers are lightweight and launch quickly, as they do not include a full-fledged operating system and share the host's operating system, specifically the kernel and only require an operating-system process with its own file system. This results in more efficient resource utilization and faster startup times compared to VMs.

The kernel is the core component of an operating system, responsible for managing both applications and hardware resources.

Docker Architecture

Docker operates using a client-server architecture, consisting of a client component and a server component that communicate through a REST API. The server, also known as the Docker engine, runs in the background, handling the tasks of building and running containers. Docker does have the following components:

Docker Client: The Docker client is the primary interface through which users interact with Docker. It provides a command-line interface (CLI) and allows users to issue commands for building, running, and managing containers. The client communicates with the Docker daemon (or Docker Engine) via a RESTful API to perform various tasks, such as creating containers, pulling images, and managing container lifecycles. In addition to the CLI, Docker also provides a graphical user interface (GUI) called Docker Desktop for Windows and macOS users, which makes it easier to manage containers and images visually.

Docker Daemon (Engine): The Docker daemon, also known as the Docker Engine, is a background process that runs on the host machine and manages the entire lifecycle of containers. The daemon listens for API requests from the Docker client and performs the required tasks, such as building images, creating containers, and managing container lifecycles. It is responsible for performing the actual work in the Docker system, including managing container isolation, networking, and storage.

Docker Images: Docker images are the building blocks of containers. They are read-only templates that contain the operating system, runtime environment, libraries, and application code necessary to run an application. Images can be created from a Dockerfile, which is a script that contains instructions for building the image. Docker images can be stored and shared through Docker registries, such as Docker Hub or private registries, allowing users to easily distribute and deploy applications across different environments.

Docker Containers: Docker containers are lightweight, portable, and isolated runtime environments for running applications. They are created from Docker images and have a writable layer on top of the image, which allows them to store runtime data and maintain their state. Containers run in isolation from each other, sharing only the host's kernel, which makes them highly efficient compared to virtual machines. Containers can be easily managed using Docker commands, such as docker start, docker stop, and docker rm.

Docker Networking:
Docker provides a robust networking model that enables communication between containers and the outside world. It supports multiple network drivers, such as bridge, host, overlay, and Macvlan, which offer different levels of isolation and performance. By default, Docker creates a virtual network called a bridge network, which allows containers to communicate with each other and the host. Users can also create custom networks to isolate containers or connect them to external networks.

Docker Volumes and Storage:
Docker provides a flexible storage system for managing data within containers. It supports various storage drivers, such as overlay2, aufs, and btrfs, which determine how data is stored and managed on the host system. Docker also supports volumes, which are a way to persist data generated by containers and share it between containers. Volumes can be managed using Docker commands, such as docker volume create, docker volume ls, and docker volume rm.

Docker Registries: Repositories for storing and sharing Docker images. Docker Hub is the most popular registry, providing a platform for storing and distributing images.

Docker Containers vs. Images

A fundamental distinction in Docker terminology is the difference between containers and images. An image is a read-only template that includes the operating system, runtime environment, libraries, and application code necessary to run an application. A container, on the other hand, is a running instance of an image. When a container is created from an image, a writable layer is added on top of the image, allowing the container to store runtime data and maintain its state. Technically container is a operating system process which is special kind because it has its own file system that is provided by the image.

Installing Docker

To install Docker on your machine, follow the official installation guides for your operating system:

Docker Container Lifecycle

The lifecycle of a Docker container typically consists of several stages:

Create: A container is created from a Docker image using the docker create or docker run command. When created, the container has a unique ID assigned by the Docker daemon.

Start: A created container can be started using the docker start command, or it can be created and started in a single command using docker run. Once started, the container runs the entrypoint command specified in the Dockerfile (or a custom command provided in the docker run command).

Pause/Resume: Containers can be paused using the docker pause command, which temporarily suspends all processes inside the container. To resume a paused container, use the docker unpause command.

Stop: A running container can be stopped using the docker stop command. This command sends a SIGTERM signal to the main process inside the container, allowing it to perform a graceful shutdown. After a grace period, if the container has not exited, a SIGKILL signal is sent to forcefully terminate the container.

Restart: A stopped container can be restarted using the docker restart command, which stops and starts the container again.

Remove: A container that is no longer needed can be removed using the docker rm command. This permanently deletes the container and its writable layer, freeing up system resources.

Development Workflow

Write application code and create a Dockerfile: Start by writing your application code and creating a Dockerfile that defines the instructions for building the Docker image.

A Dockerfile is a text file containing a series of instructions used by Docker to build a new image. It automates the process of setting up an environment, installing dependencies, and configuring the application within a container. Docker reads the instructions from the Dockerfile and executes them in order, creating a new image as a result.

Build the Docker image: Use the docker build command to create a Docker image from the Dockerfile.
Run the container: Use the docker run command to start a container from the Docker image.
Test the application: Test the application running inside the container to ensure it functions as expected.
Publish the Docker image: If the application works as expected, publish the Docker image to a registry like Docker Hub.
Deploy the container: Deploy the container to your desired environment using tools like Docker Compose, Kubernetes, or other orchestration platforms.

Dockerizing a Node.js Application

Create a Node.js application: Start by creating a simple Node.js application with an app.js file and a package.json file.

Here's a simple Node.js application that serves a "Hello, World!" message on port 3000.

First, make sure you have Node.js installed on your system. If not, you can download and install it from the official Node.js website.
Create a new directory for your Node.js application, and navigate to it in your terminal or command prompt.
Initialize a new Node.js project by running the following command:

npm init -y

This will create a package.json file with default values.

Create a new file named app.js in your project directory and add the following code:

const http = require('http');

const hostname = '0.0.0.0';
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello, World!\n');
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

This code creates a basic HTTP server that listens on port 3000 and responds with "Hello, World!" to any incoming requests.

Update the package.json file to include a start script. Add the following line to the scripts section:

"start": "node app.js"

Your package.json file should now look like this:

{
  "name": "your-project-name",
  "version": "1.0.0",
  "description": "",
  "main": "app.js",
  "scripts": {
    "start": "node app.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

To run the Node.js application locally, execute the following command:
npm start
You should see the message "Server running at http://0.0.0.0:3000/" in your terminal. Open your web browser and navigate to http://localhost:3000/ to see the "Hello, World!" message.

Now that you have a simple Node.js application, you can proceed to create a Dockerfile and Dockerize the application.

Create a Dockerfile:

In the same directory as your Node.js application, create a Dockerfile with the following content:

# Set the base image
FROM node:latest

# Set the working directory
WORKDIR /app

# Copy package.json and package-lock.json to the working directory
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy the application code to the working directory
COPY . .

# Expose the application port
EXPOSE 3000

# Start the application
CMD ["npm", "start"]

This Dockerfile uses the latest official Node.js image as a base, sets the working directory to /app, copies the package.json and package-lock.json files, installs the required dependencies, copies the application code, exposes port 3000, and runs the npm start command.

Here is a brief overview of some common instructions used in a Dockerfile:

FROM: Specifies the base image to be used as a starting point for the new image. Examples include official images like ubuntu, alpine, or node. FROM node:latest

WORKDIR: Sets the working directory within the container. Any subsequent instructions that use relative paths will be executed relative to this directory. WORKDIR /app

COPY: Copies files or directories from the local machine (the build context) to the container's filesystem. COPY package.json .

ADD: Similar to COPY, but it can also download files from a URL and extract compressed files. ADD https://example.com/file.tar.gz /app

RUN: Executes a command within the container, typically used for installing packages or running build scripts. RUN npm install

CMD: Provides the default command to run when the container is started. If the user specifies a command when running the container, it will override this default command. There can be only one CMD instruction in a Dockerfile. CMD ["npm", "start"]

ENTRYPOINT: Similar to CMD, but it is not overridden when the user specifies a command when running the container. This is useful for defining a default executable for the container.
ENTRYPOINT ["npm"]
CMD ["start"]
ENV: Sets an environment variable within the container, which can be used by applications running inside the container. ENV NODE_ENV=production

EXPOSE: Informs Docker that the container listens on the specified network ports at runtime. This does not actually publish the port; it serves as documentation and a reminder to publish the port using the -p flag when running the container. EXPOSE 80

VOLUME: Creates a mount point for a volume to persist data outside of the container. This is useful for sharing data between containers or retaining data when a container is removed. VOLUME /app/data

Build the Docker image: Run the following command in the terminal:

docker build -t your-image-name .

Replace your-image-name with a descriptive name for your image.

Run the Docker container: Start the container using the following command:

docker run -p 3000:3000

Pushing this Docker Image to Docker Hub

Docker Hub is a public registry service that allows you to store and share Docker images. To push a Docker image to Docker Hub, you need to follow these steps:

Create a Docker Hub account: If you don't already have a Docker Hub account, sign up for one at . You'll need your Docker ID and password for subsequent steps.

Log in to Docker Hub: Open a terminal or command prompt and log in to Docker Hub using your Docker ID and password:

docker login

Enter your Docker ID and password when prompted. You should see a message indicating that you have successfully logged in.

Tag your Docker image: Before pushing the image to Docker Hub, you need to tag it with your Docker Hub username and a repository name. Use the following command to tag your image:

docker tag your-image-name your-docker-id/your-repository-name:your-tag

Replace your-image-name with the name of the image you built in step 4, your-docker-id with your Docker Hub username, your-repository-name with a desired repository name, and your-tag with a version or any descriptive tag (e.g., 'latest').

For example:

docker tag myapp-image johnsmith/myapp:latest

Push the Docker image to Docker Hub: Finally, push the tagged image to Docker Hub using the following command:

docker push your-docker-id/your-repository-name:your-tag

Replace your-docker-id, your-repository-name, and your-tag with the values you used in step 5. Docker will upload your image to your Docker Hub account.
For example:

docker push johndoe/myapp:latest

Verify the image on Docker Hub: Log in to your Docker Hub account and navigate to the "Repositories" section. You should see the newly pushed image listed under your repositories.
Now, your Docker image is successfully pushed to Docker Hub and can be easily pulled and run by others using the docker pull command, followed by docker run.

Benefits of Docker Containers

Docker containers offer several advantages over traditional deployment methods and virtual machines:

Consistency: Containers package all application dependencies and configurations, ensuring consistent behavior across various environments, from development to production.

Isolation: Containers run in isolated environments, preventing conflicts and ensuring that each application has access to its required resources.

Portability: Docker images can be easily shared and run on any system with Docker installed, making it easy to deploy applications across different platforms.

Resource Efficiency: Containers share the host operating system's kernel and resources, resulting in less overhead compared to virtual machines. This enables higher density and more efficient resource utilization.

Scalability: Containers can be easily scaled up or down to meet the changing demands of an application, making it easier to build and deploy microservices and distributed applications.

Version Control and Rollback: Docker images can be versioned, allowing for easy rollback to previous versions if needed. This can be particularly useful in case of application updates that introduce bugs or performance issues.

In summary, Docker containers provide an efficient, portable, and consistent environment for application deployment. By leveraging containerization, developers and operations teams can streamline the development, testing, and deployment process, ultimately leading to more reliable and maintainable software.

If you'd like, let's connect on LinkedIn and grow our community bigger.

Catching Unhandled Promise Rejections and uncaughtException in Node.js

MD RAHIM IQBAL — Sun, 02 Apr 2023 11:24:38 +0000

Node.js is an event-driven platform that executes code asynchronously. This means that errors thrown in a callback or a promise chain that are not caught will not be handled by the uncaughtException event-handler and will disappear without warning. Although recent versions of Node.js added a warning message when an unhandled rejection occurs, this does not constitute proper error handling.

This can be a significant issue for developers, who may overlook adding catch clauses to promise chains. Fortunately, Node.js provides a mechanism for catching these unhandled rejections, the unhandledRejection event. In this blog post, we will discuss how to catch unhandled promise rejections in Node.js and why it is essential.

Why Catching Unhandled Promise Rejections and uncaughtException is Important

Uncaught errors in Node.js can cause serious issues, such as memory leaks, unexpected termination, and server downtime. Therefore, it is essential to catch these errors in a controlled manner to minimize their impact on the system. By subscribing to the process.on('unhandledRejection', callback) and process.on('uncaughtException', callback) events, you can catch these errors and handle them appropriately.

Here's an example code snippet that could result in an unhandled promise rejection:

import { User } from './models/user.model';

async function getUserById(id: string) {
  return await User.findById(id);
}

async function updateUserName(id: string, name: string) {
  const user = await getUserById(id);
  user.name = name;
  return user.save();
}

// Call updateUserName function with invalid ID
updateUserName('invalidId', 'John Doe');

In this code snippet, the getUserById function returns a promise that resolves with the user object with the given ID. The updateUserName function uses this function to find the user with the given ID, updates the user's name, and saves the changes to the database.

However, if the updateUserName function is called with an invalid ID, the getUserById function will throw an error, resulting in an unhandled promise rejection.

Handling Unhandled Promise Rejections

The simplest way to handle unhandled promise rejections is to add a .catch clause within each promise chain call and redirect it to a centralized error handler. However, relying solely on developer discipline is a fragile way of building an error handling strategy. Hence, using a graceful fallback to subscribe to process.on('unhandledRejection', callback) is a recommended approach.

Here is an example of how to catch unhandled promise rejections:

process.on('unhandledRejection', (reason: string, p: Promise<any>) => {
  console.error('Unhandled Rejection at:', p, 'reason:', reason);
});

This code subscribes to the unhandledRejection event and prints the unhandled rejection's reason and promise to the console. By doing this, you can identify where the rejection occurred and handle it appropriately.

Handling Uncaught Exceptions

Uncaught exceptions occur when an error is thrown but not caught by any try-catch block or error handler. These exceptions can lead to the application's unexpected termination or cause memory leaks that can lead to server downtime. To handle these uncaught exceptions, you can subscribe to the process.on('uncaughtException', callback) event.

Here is an example of how to catch uncaught exceptions:

process.on('uncaughtException', (error: Error) => {
  console.error(`Caught exception: ${error}\n` + `Exception origin: ${error.stack}`);
});

This code subscribes to the uncaughtException event and prints the exception's stack trace to the console. By doing this, you can identify where the exception occurred and handle it appropriately.

Catching unhandled promise rejections and uncaught exceptions is crucial to avoid unexpected behavior in Node.js applications. By subscribing to the process.on('unhandledRejection', callback) and process.on('uncaughtException', callback) events and using graceful fallbacks, you can catch and handle these errors in a controlled manner. The examples provided in this article demonstrate how to catch these errors and handle them appropriately, thereby ensuring the stability and reliability of your Node.js applications.

Graceful Shutdown in Node.js: Handling Stranger Danger

MD RAHIM IQBAL — Sat, 01 Apr 2023 18:10:03 +0000

When a stranger comes to town, it's important to know how to handle them gracefully. The same goes for processes in Node.js - sometimes we need to shut them down gracefully when we detect a problem or when we receive a signal to terminate the process. In this article, we'll explore how to implement a graceful shutdown in a Node.js application using TypeScript.

What a graceful shutdown means

When a Node.js process is terminated, there might be some ongoing tasks or connections that need to be closed before the process can exit completely. A graceful shutdown ensures that these tasks are completed before the process is terminated. It also avoids any abrupt closing of connections, which can lead to data loss or corruption.

To implement a graceful shutdown, we need to handle the SIGINT and SIGTERM signals that are sent to the process when it's time to terminate. SIGINT and SIGTERM are signals used in Unix-based systems to interrupt or terminate a process.

SIGINT: This is a signal that is typically sent to a process when a user types Ctrl+C in the terminal. It is often used to request that a process terminate gracefully. When a process receives a SIGINT signal, it can catch it and perform any necessary cleanup operations before terminating.

SIGTERM: This is a signal that is typically sent to a process by the operating system to request that the process terminate. It is often used as a graceful way to ask a process to terminate, allowing it to perform any necessary cleanup operations before exiting. Processes can catch this signal and perform cleanup operations before terminating.

In Node.js, you can listen for these signals using the process.on method. We can define a function that handles these signals and performs the necessary cleanup operations before exiting the process.

Here is an example of how to handle SIGINT and SIGTERM signals:

import { Server } from 'http';

const server: Server = /* create your server here */;

function gracefulShutdown() {
  console.log('Shutting down gracefully...');

  server.close(() => {
    console.log('Server closed.');

    // Close any other connections or resources here

    process.exit(0);
  });

  // Force close the server after 5 seconds
  setTimeout(() => {
    console.error('Could not close connections in time, forcefully shutting down');
    process.exit(1);
  }, 5000);
}

process.on('SIGTERM', gracefulShutdown);
process.on('SIGINT', gracefulShutdown);

In this code snippet, we're defining a gracefulShutdown function that logs a message to the console and then closes the HTTP server. After the server is closed, any other connections or resources can be closed as well. Finally, we call process.exit(0) to exit the process with a success code.

We also set a timeout of 5 seconds to force close the server if it's not closed gracefully within that time. This ensures that the process exits even if there are ongoing connections or resources that cannot be closed gracefully.

A graceful shutdown is important to ensure that a Node.js application terminates cleanly and without any data loss or corruption. By handling SIGINT and SIGTERM signals, we can implement a graceful shutdown that closes ongoing tasks and connections before exiting the process. With this knowledge, we can handle stranger danger in our Node.js applications with grace and elegance.

Separating 'app' and 'server' in Express: Why it matters and how it benefits your application

MD RAHIM IQBAL — Fri, 31 Mar 2023 20:57:38 +0000

When building a Node.js application using Express, one of the key decisions you'll need to make is whether to keep your application logic in a single file, such as app.js, or to separate it out into multiple files, including a dedicated server.js file. While it may be tempting to keep everything in one place for the sake of simplicity, separating your code into distinct files can actually improve the quality and maintainability of your application in the long run.

While separating your code into different files is a good idea in general, there's also a specific reason why you should consider separating app.js and server.js. In this blog post, we'll explore why separating app.js and server.js is a good practice and provide examples and code snippets to illustrate the advantages.

Separation of concerns

One of the main reasons to separate app.js and server.js is to divide the application's concerns. app.js is responsible for defining the routes, middleware, and other application-level functionality. server.js, on the other hand, is responsible for creating the server, listening for incoming requests, and handling errors.

Thus we can keep our code organized and maintainable. If we need to make changes to the application logic or routes, we can do so in the app.js file without worrying about the server setup. Similarly, if we need to make changes to the server configuration, we can do so in the server.js file without affecting the application logic.

Here is an example of what a app.js and server.js file might look like:

app.js file:

const express = require('express');
const app = express();

app.get('/', (req, res) => {
  res.send('Hello World!');
});

module.exports = app;

server.js file:

const app = require('./app');

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

In this code snippet, the app.js file is responsible for defining the Express application and its routes, while server.js creates a new HTTP server instance and passes it to the app module. This approach separates the server configuration from the application's functionality.

Easier testing

Separating app.js and server.js not only draws a clean separation of concerns but also significantly eases mocking and testing the system by testing the API in-process, without performing network calls, with all the benefits that it brings to the table: fast testing execution and getting coverage metrics of the code.

For example, you can write unit tests for app.js to ensure that each route and middleware function is working as intended. You can also write integration tests for server.js to ensure that the server is properly configured and handling requests correctly.

Here is an example of a test file for app.js:

const request = require('supertest');
const app = require('../app');

describe('GET /', () => {
  it('should respond with "Hello, world!"', (done) => {
    request(app)
      .get('/')
      .expect(200)
      .end((err, res) => {
        if (err) return done(err);
        expect(res.text).toBe('Hello, world!');
        done();
      });
  });
});

In this code snippet, we're using the supertest library to make HTTP requests to our app module and test its responses.

Improved scalability

Finally, separating app.js and server.js can improve the application's scalability. By breaking up the code into smaller modules, it becomes easier to add new features or modify existing ones without having to touch the entire application codebase.

In summary, separating the app.js and server.js files in an Express.js application can improve code organization, maintainability, and testability.

How to Learn Anything Faster and More Effectively as a Software Developer

MD RAHIM IQBAL — Fri, 31 Mar 2023 19:06:46 +0000

When it comes to learning a new technology or skill, many of us often turn to courses or books to get started. However, this approach may not always yield the desired results. While there are certainly benefits to taking courses or reading tech books, there are also some potential drawbacks that are worth considering.

One of the main downsides of relying solely on courses or books is that they often only provide a theoretical understanding of the material. While this can be a great way to get started and gain a foundational understanding of a topic, it can also lead to a lack of practical experience. Without practical experience, it can be difficult to apply what you've learned to real-world situations and to troubleshoot problems that arise.

Another downside is that courses and books can be time-consuming and expensive. Depending on the course or book, it may take weeks or months to complete, and the cost can add up quickly. This can make it difficult for people with busy schedules or limited budgets to pursue their interests or advance their careers.

In addition, courses and books may not always provide the most up-to-date or relevant information. Technology is constantly evolving, and what was cutting-edge a few years ago may now be outdated or even irrelevant. This can be especially problematic if you're trying to keep up with the latest developments in your field.

Overall, while courses and books can be a valuable tool for learning, they should be supplemented with practical experience and other resources to ensure a well-rounded understanding of the topic at hand.

In this blog, we will explore a more practical and effective approach to learning, one that involves hands-on practice, problem-solving, and continuous exploration.

Part 1: Understanding the Purpose and Benefits of the Technology

Before diving into any new technology, it's important to understand its purpose and potential benefits. This can help you set clear goals and expectations for your learning journey. For example, if you want to learn Terraform, you could start by googling What is Terraform? and What problems it solves?. You could also read case studies or success stories of companies that have used Terraform to improve their infrastructure management.

Part 2: Getting Hands-On with Practice

Once you have a basic understanding of the technology, it's time to get hands-on with practice. This involves installing the software or tool on your device and trying out different commands and configurations. Don't be afraid to make mistakes or encounter errors – this is all part of the learning process. In fact, by troubleshooting errors and finding solutions through documentation and online resources, you can gain a deeper understanding of the technology and its inner workings.

Part 3: Building Your Knowledge Tree

As you practice and experiment with the technology, you will naturally start to build your own knowledge tree, with branches and leaves that connect different concepts and ideas. For example, when learning Terraform, you may also become more familiar with Linux commands, AWS services, and infrastructure design patterns.

After getting familiar with the basics of Terraform, the next step is to move on to more advanced topics. For example, you can try provisioning an EC2 instance with Terraform, which will give you hands-on experience with using variables, providers, and other advanced features. As you work through these tasks over the course of a week or so, you'll gain a deeper understanding of how Terraform works and how to apply it to real-world scenarios.

Keep a record of your progress and insights, whether it's through notes, diagrams, or code snippets. This can help you track your progress and identify areas that need more attention.

Part 4: Growing Your Knowledge with Real-World Examples

To truly master a technology or skill, you need to be able to apply it in real-world scenarios. This could involve building a project from scratch, contributing to an open-source project, or solving a complex problem using the technology.

To further cement your knowledge, it can be helpful to seek out Terraform interview questions and practice answering them. By doing so, you'll identify gaps in your knowledge and gain a deeper understanding of the concepts and techniques you need to master. Jotting down the questions and answers you encounter can also help you solidify your knowledge and make it easier to recall later on.

By testing your knowledge and skills in these scenarios, you can gain valuable experience and confidence in your abilities.

Part 5: Continuing Your Learning Journey

Learning is a continuous journey, and it doesn't stop once you've completed a course or mastered a technology. To stay current and relevant, you need to keep exploring and experimenting with new tools and techniques. This could involve attending meetups or conferences, reading blogs or articles, or taking on new projects that challenge you to learn new skills.

Learning a new technology or skill can be a daunting task, but by following a practical and hands-on approach, you can accelerate your learning and achieve your goals more effectively. Remember to focus on understanding the purpose and benefits of the technology, getting hands-on with practice, building your knowledge tree, testing your knowledge with real-world examples, and continuing your learning journey. With these tips and techniques, you'll be well on your way to becoming a lifelong learner and a master of your craft.

Software Development Engineer - Full Stack Interview Review

MD RAHIM IQBAL — Mon, 20 Mar 2023 11:05:03 +0000

As a software developer, there are few things more exciting than being invited to participate in a multi-stage technical interview. Recently, I had the opportunity to go through a three-stage interview process at Craftsmen Ltd. that not only challenged me technically, but also allowed me to get to know the company and their culture better.

First Stage

This was a technical assessment sent via email and time limit was 3 hours.

I was given a set of requirements of creating REST APIs that allows users to Confidential create, read, update, and delete blog posts. and asked to implement the solution using Node.js and the Express framework.

Then create a React app that allows the user to view and interact with the list of blog posts from the REST APIs created and asked to integrate redux to manage the state of the blog posts.

Also asked to write tests to ensure that the REST API, Redux store, and React app work as expected.

This was an intense experience, but it allowed me to showcase my technical skills and demonstrate my ability to work under pressure.

Second Stage

It was an onsite interview, which allowed me to meet the team in person and get a better understanding of their work culture.

During this stage, I was asked a range of technical questions, including ones related to AWS services, Lambda, S3, REST, Node.js, React, Redux, and lifecycle methods. The team was friendly, engaging, and clearly invested in getting to know me both as a developer and as a person.

One of the questions that particularly stood out to me was the one about why I used a custom error handler in my demo project. This question allowed me to demonstrate my understanding of the importance of handling errors properly in a production environment, and how a custom error handler can help to provide a better user experience.
Followings are a list of questions/topics we discussed there.

Can you introduce yourself?
What are the products you have worked on?
What are your strengths in frontend/backend development? (One interviewer mentioned that CSS is like black magic to him.)
What AWS services have you used?
If millions of files are uploaded to S3 and a Lambda function needs to be triggered, how would Lambda handle the workload? How would it work with SQS? [This was a dream question for me based on my work expectations]
Can you explain how dead letter queues work?
How does Lambda work, and what are its limitations?
Can you explain how to upload a file from your application to a private S3 bucket?
What are the advantages of REST, and how does it work?
Why did you use a custom error handler in your assignment project?
Are you satisfied with your demo project? If not, how would you improve it?
How often have you used unit tests in practice?
How is your code deployed, and from your answer, is it possible to run end-to-end tests before merging a pull request? If so, how?
Why are unit tests necessary?
Can you explain how a pure component works?
Have you encountered any scenarios where you needed to customize the Shadow DOM library in React?
Can you explain the lifecycle methods of a React class component?
How can you achieve the same behavior of lifecycle methods in a functional component?
What are the alternatives to Redux?
How can you share data between components?
Can you explain how the Context API works?
What is Kubernetes, and what problems does it solve?
In the event that a pod crashes in Kubernetes, who is responsible for fixing its state?
In one of your project, you used AWS ECS to deploy your application. Can you explain the reason behind this choice? Is it more expensive compared to other options?
Why did you leave your previous job, and what issues did you encounter? How did you try to communicate these issues to your superiors? If you encountered the same issues at this company, what would you do?
Are you interested in pursuing an MSC degree? If so, why? What are your goals after completing the degree?
How did you utilize your BSC credits in your practical experience, and how would pursuing an MSC degree for two more years help you?

Another highlight of the interview was when the team discussed their products and the kind of challenges they face and solve. This showed me that they are passionate about what they do, and that they have a deep understanding of the industry and the technologies they work with.

I appreciated the way they took the time to understand my experience, interests, and goals, and asked thoughtful questions that allowed me to showcase my skills and knowledge.

Final Stage

This was a non-technical interview where we discussed the company's facilities, work culture, and the negotiation round. This allowed me to get a better sense of the company's values, and to see whether I would be a good fit for their team.

Overall, the interview process was challenging but rewarding, and allowed me to showcase my skills and get to know the company and the team better. I appreciated the friendly and engaging nature of the interviews, and the way that the team took the time to understand my interests and goals.

Read/Download S3 files using Lambda Functions

MD RAHIM IQBAL — Wed, 15 Feb 2023 17:35:49 +0000

To get started with this tutorial, you'll first need to create an Amazon S3 bucket and upload a sample object to it.

Step 1 Create a bucket and upload a sample object:

Open the Amazon S3 console and choose "Create bucket".
Give your bucket a name (e.g. MyTestBucket07) and choose a region.
Click "Create bucket".
Select the bucket you just created and go to the "Objects" tab.
Choose "Upload" and select a test file (e.g. MyTestFile.txt) from your local machine. This file can be a text file containing anything you want.
Click "Upload".

Once your bucket is set up, you can create a Lambda function using a function blueprint. A blueprint is a sample function that demonstrates how to use Lambda with other AWS services.

Step 2 Create an IAM role:

Create an IAM role with a policy that provides read and write access to S3

Step 3 Create the Lambda Function:

Here are the steps to create a Lambda function in the console:

Open the Functions page on the Lambda console.
Choose "Create function".
Select "Use a blueprint".
Search for "s3" under Blueprints and choose the s3-get-object blueprint for Node.js.
Click "Configure".
Under "Basic information",

Enter a name for your function (e.g. ReadS3File)
Choose "Create a new role from AWS policy templates" for the execution role.
Enter a name for your role (e.g. S3ReadAccess).
From Policy template select Amazon S3 object read-only permissions
Under "S3 trigger", choose the bucket you created earlier.
Click "Create function". Next, you can review the function code, which retrieves the source S3 bucket name and the key name of the uploaded object from the event parameter it receives. The function uses the Amazon S3 getObject API to retrieve the content type of the object.

Step 3: To view the function code in the Lambda console:

Go to the "Code" tab while viewing your function.
Look under "Code source".
AWS provided code need some changes,

const AWS = require('aws-sdk');
const s3 = new AWS.S3({apiVersion: '2006-03-01'});

exports.handler = async (event, context) => {
    // Set the bucket name and object key based on the file name
    const bucketName = "MyTestBucket07";
    const objectKey = event.filePath;

    const params = {
        Bucket: bucketName,
        Key: objectKey
    };

    try {
        // Read the object
        const s3Object = await s3.getObject(params).promise();
        console.log(s3Object.Body.toString());

        return s3Object.Body.toString();
    } catch (err) {
        console.log(err);
        throw err;
    }
};

Step 4: Test the Lambda Function

On the Code tab, under Code source, choose the arrow next to Test, and then choose Configure test events from the dropdown list.
In the Configure test event window, do the following: Choose Create new test event named 'TestCase01'
Add the following code in Event JSON

{
  "filePath": "MyTestFile.txt"
}

Choose Create.

To invoke the function with your test event, under Code source, choose Test. The Execution results tab displays the response, function logs, and request ID