DEV Community: Yash Patil

Managing Storage and Packages in Linux: A Complete Guide

Yash Patil — Fri, 07 Mar 2025 07:11:29 +0000

Okay so lets get cracking on what could be the last part of this Linux Fundamentals series, and this is a exciting one to say the least !

Storage Management

The Importance of High Throughput and Low Latency Storage

In computing, the efficiency of storage systems significantly influences application performance. High throughput ensures that large volumes of data are read or written swiftly, which is crucial for tasks like data analysis and multimedia processing. Low latency guarantees immediate data retrieval, vital for real-time applications such as online gaming and financial transactions. Together, high throughput and low latency contribute to a seamless and efficient computing experience.
Understanding Block Storage

Block storage is a method where data is stored in fixed-sized chunks called blocks. Each block is assigned a unique address, allowing the operating system to retrieve data efficiently. For instance, when storing a 10GB file, the system divides it into smaller blocks (commonly 4KB each). Each block is then stored separately, and an index keeps track of these blocks, facilitating quick access and management. This method enhances performance and allows for flexible data management.

AWS EBS(Elastic Block Storage) is an example of Block Storage devices. Block Storage is suitable for transactional databases and more where multiple users access the data and where parallel processing is needed.

Exploring Block Devices

A block device is a hardware or virtual device that allows for reading and writing data in fixed-sized blocks. These devices include hard drives, SSDs, and virtual block devices. In Linux, block devices are represented as files and are typically found under the /dev directory. For example, /dev/sda might represent the first SATA drive in the system. These device files enable the operating system and applications to interact with the hardware seamlessly.

For our demo I will be using an EC2 instance from AWS with a default 8GB storage.

We have logged into our EC2 instance:

To list the available block devices in your system use the following command :

lsblk

What I can see is a xvda block device which has 4 partitions xvda1, xvda14, xvda15 and xvda16.

As I said all the block devices can be found under /dev directory.

Diving into Partitions and the GPT Scheme

Partitions are subdivisions of a storage device, allowing users to segment their storage for various purposes, such as separating system files from user data. The GUID Partition Table (GPT) is a modern partitioning scheme that offers several advantages over the older Master Boot Record (MBR) system. GPT supports larger disk sizes, allows for a nearly unlimited number of partitions, and includes redundancy and error-checking features.

Our xvda block device has 4 partitions xvda1, xvda14, xvda15 and xvda16 mounted on separate locations.

But we can also use our disk or block storage as is without necessarily creating any partitions out of it.

Filesystems and Mounting: Making Partitions Usable

Now lets attach another block storage to our instance in the form on an AWS EBS Volume.

Above is the default 8GB Volume (xvda) that was attached to our EC2 Instance when we created it.

Created a volume of 10GB :

Now attach this volume to our EC2 instance :

As you can see a new block storage is available under xvdf of 10GB, its the EBS volume we created just now.

Now lets create partitions out of it using gdisk and lets mount them to separate locations.

```
sudo gdisk /dev/partion_or_blockdevice 
```

Choose the n flag to create a new partition out of /dev/xvdf which was our 10GB EBS Block Storage Device.

Press n → Create a new partition.
Partition Number → Press Enter to accept the default.
First sector → Type 2048 (or press Enter for the default, usually aligned).
Last sector → Type +1G (this tells gdisk to create a 1GB partition).
Hex Code → Press Enter (default is 8300 for Linux filesystems).
Press w → Write the changes to the disk.
Confirm with y

Our partition of 1Gb is created with the name xvdf1.

Possible Actions for the Remaining 9GB:

Create More Partitions

* You can create additional partitions using the unallocated space.

* Example: You could create a second 5GB partition and a third 4GB 
  partition.

Expand an Existing Partition Later

* If needed, you can extend the **1GB partition** to use more space 
  later using tools like `gdisk`, `parted`, or `resize2fs` (for 
  filesystems like ext4).

Leave It Unallocated

* The remaining space will be available for future use but won’t be 
  accessible by the OS until partitioned and formatted.

After creating partitions, they remain raw spaces until formatted with a filesystem, which organizes data for storage and retrieval. A filesystem defines how data is stored and retrieved on a storage device. ext4 is a widely used filesystem in Linux, known for its robustness, support for large volumes, and journaling capabilities, which help in data recovery during unexpected shutdowns.

To format a partition with the ext4 filesystem and mount it:

Format the partition:
```
```
sudo mkfs.ext4 /dev/xvdf1
```
```

Replace /dev/xvdf1 with your specific partition identifier.

Create a mount point:
```
```
sudo mkdir /mnt/xvdf1Mount
```
```

Mount the partition:

```
sudo mount /dev/sdX1 /mnt/mydata
```

As you can see we have successfully mount our partition of a mount path in the Ubuntu OS of out EC2 Instance. Now you can create, read and write data in that mount path.

Persisting Mounts with /etc/fstab

To ensure that partitions mount automatically at boot, you can add entries to the /etc/fstab file specifying the device, mount point, filesystem type, and options :

Open /etc/fstab with a text editor:
```
```
sudo vi /etc/fstab
```
```
Add a line specifying the device, mount point, filesystem type, and options:
```
```
/dev/xvdf1 /mnt/xvdf1Mount ext4 rw 0 0
```
```
Save and close the file.

This configuration ensures that the system mounts the specified partitions automatically during startup.

So I just rebooted my EC2 Instance but as you can see the partition still exists and is mounted to that path :

And our file also exists :

Package Management

Understanding Package Management

Package management is a fundamental aspect of Linux systems,
facilitating the installation, upgrade, configuration, and removal
of software packages. It ensures that software dependencies are
handled automatically, maintaining system stability and
consistency.
1. Package Managers in Debian and RPM Families

   * **Debian Family:**
     - **APT (Advanced Package Tool):**

A command-line tool that simplifies the process of managing software packages on Debian-based distributions like Ubuntu. It handles package installation, updates, and removal, resolving dependencies automatically.

     - **APT-GET:** An older command-line tool for handling packages, still widely used for script-based operations.

   * **RPM Family:**

     - **YUM (Yellowdog Updater Modified):** A package manager for RPM-based distributions like CentOS and Fedora. It manages package installations, updates, and removals, resolving dependencies automatically.

     -  **DNF (Dandified YUM):** The next-generation version of YUM, offering improved performance and better dependency resolution.

Operations with APT and APT-GET

* **Difference Between APT and APT-GET:**

    While both tools serve similar purposes, `apt` is designed to provide a more user-friendly experience, combining functionalities of `apt-get` and `apt-cache`. It's recommended for interactive use, whereas `apt-get` is preferred for scripting purposes.

* **Common Commands:**

    * **Update Package Lists:**

        ```
        sudo apt update
        ```

    * **Upgrade Installed Packages:**

        ```
        sudo apt upgrade
        ```

    We usually run the above commands when we launch a new VM to update and upgrade its packages.

    * **Install a Package:**

        ```bash
        sudo apt install package_name
        ```

    * **Search for a Package:**

        ```bash
        apt search package_name
        ```

    * **Remove a Package:**

        ```bash
        sudo apt remove package_name
        ```

    * **Edit Sources List:**

        The package sources are listed in `/etc/apt/sources.list`. To edit:

        ```bash
        sudo nano /etc/apt/sources.list
        ```

List Installed Packages:
```
    apt list --installed
```

How Package Managers Work:

Package managers like APT work by connecting to remote repositories specified in the `/etc/apt/sources.list` file. They download package information and updates, ensuring that software installations and upgrades are consistent and reliable.

Overview

I hope this blog post helped you understand the standard partitioning scheme, storage is divided into fixed partitions during disk setup. Each partition is formatted with a filesystem and mounted for use and the Package Management in the Linux Systems. If you have any more questions or need further clarification on any topic, feel free to ask!

Next up is Traditional Partitioning vs. Logical Volume Management (LVM): Why LVM is the Better Choice. I know I said this could be the last of the series but this just got more interesting than I anticipated it to be. Also this post is getting too big! :)

Linux User Management and File Permissions

Yash Patil — Fri, 07 Mar 2025 06:36:24 +0000

Linux Users, Groups, and File Permissions: A Deep Dive

Why Do We Need User Accounts and File Permissions in Linux?

Imagine a company office where multiple employees share a workspace. Each employee has their own desk (personal files) and shared meeting rooms (public directories). To maintain order, employees should only have access to their own desks and specific shared spaces. Similarly, in Linux, user accounts and file permissions ensure that only authorized users can access certain files and directories, maintaining security and privacy.

Every Linux user is associated with an account that determines their identity and access levels.

And What is a Group?

A group is a collection of users, used to organize users based on common attributes such as roles and functions.

Remember, if a user is not a part of any group while its creation, it has a primary group assigned to itself with the same name as the username.

User accounts come with several attributes:

User ID (UID): A unique identifier for each user.
Group ID (GID): The primary group associated with the user.
Home Directory: The default location where user files and configurations are stored.
Default Shell: The command-line interface the user interacts with upon logging in.

Types of Accounts in Linux

Linux accounts fall into different categories:

Regular Users: Standard accounts with limited access, meant for everyday use.
Superuser (Root): The administrator with unrestricted system access. UID= 0
System Accounts: Used for system processes and background services. UID (500-1000)
Service Accounts: Created for applications that require specific permissions.

Understanding the `sudo` Group

In Linux, direct root access can be risky. Instead, users are given sudo privileges to execute administrative commands temporarily. The sudo group allows users to perform actions that typically require root access.

To add an existing user to the sudo group:

usermod -aG sudo username

Replace username with the actual username.

Managing Users and Groups

Creating a new group:

groupadd fsociety

Adding a new user and assigning them to a specific group:

useradd -m -d /home/john -s /bin/bash -G fsociety elliot

This creates a user john with a home directory, a default shell, and assigns them to the developers group.

Key System Files for User Management

/etc/passwd - Stores user account information such as UID, GID, home directory, and shell.

And here i am :

/etc/shadow - Contains encrypted passwords and expiration details for user accounts.

Ohhh look, **PERMISSION DENIED!!**

Good thing i have **sudo** privilages huh?!

Also if you see, my password is stored using a cryptographic hash using modern cryptographic hashing algorithms.

/etc/group - Lists all system groups and their associated users.

You can see a **yashp** group with **GID=1000**, it is the primary group that was created with the same name as my account username.

File Permissions: Why They Matter

Restricting file access prevents unauthorized modifications and data leaks. For example, SSH private keys need strict permissions; otherwise, SSH will refuse to use them.

Understanding the Octal Permission System

Each file in Linux has three types of permissions:

Read (r) = 4
Write (w) = 2
Execute (x) = 1

Permissions apply to three categories:

Owner (the file creator)
Group (users in the same group as the owner)
Others (everyone else)

Lets see what i actually mean..

Created these files using touch command with default permissions. You can see the files have read and write permissions for the owner i.e yashp user account, and only read permissions for members of the group yashp and other global users.

Setting permissions using chmod:

chmod 777 filename

This sets:

Owner: rwx (7 = 4+2+1)
Group: rwx (5 = 4+0+1)
Others: rwx (4 = 4+0+0)

As you can see Read, Write and Execute permissions are allocated to the Owner, Members of the Group and other Global users. Try and experiment different combinations yourself to get a good idea about this.

File Ownership and the `chown` Command

Each file has an owner and an associated group. Changing ownership:

chown newowner:newgroup filename

Example:

chown alice:developers project.txt

This makes alice the owner and assigns the developers group.

Conclusion

User and file management in Linux ensures security, privacy, and system integrity. Understanding these concepts helps prevent unauthorized access and system misuse, making Linux both powerful and secure.

Introduction to Linux: The Open-Source Revolution

Yash Patil — Fri, 07 Mar 2025 06:30:31 +0000

Let’s explore the history, architecture, and core concepts of Linux in this deep dive.

Would 100% recommend you to watch Linus’s TED talk, where he has talked about his journey in creating Linux and changing the open-source community completely!

1. The History of Linux: From UNIX to Today

The Birth of UNIX (1960s-1970s)

The journey of Linux began in 1964 when Bell Labs, MIT, and General Electric collaborated on the Multics project—a powerful, multi-user OS. However, Multics was overly complex, leading Bell Labs to withdraw from the project.

In 1969, two brilliant minds at Bell Labs—Dennis Ritchie and Ken Thompson—decided to build a simpler alternative. They created UNIX, originally called UNICS (Uniplexed Information and Computing Service), which was designed for multitasking and multi-user functionality. UNIX was small, efficient, and written in assembly language.

By 1973, Ritchie rewrote UNIX in C, making it portable across different hardware. This was a game-changer, as it allowed UNIX to spread rapidly in universities, research institutions, and corporations.

MINIX and the Rise of Linux (1980s-1991)

In the 1980s, UNIX became proprietary, leading to limited access. However, Andrew Tanenbaum, a professor, developed MINIX, a lightweight, Unix-like OS for educational purposes.

In 1991, Linus Torvalds, a Finnish computer science student, wanted a free and open OS like UNIX but with more flexibility than MINIX. So, he built his own kernel and announced it in a Usenet post:

“I'm doing a (free) operating system (just a hobby, won't be big and professional like GNU) for 386(486) AT clones.”

This quote holds massive AURA!

Thus, Linux was born. With the contributions of developers worldwide, what started as a personal project by Linus Torvalds, Linux quickly evolved into a powerful, open-source OS used in personal computers, servers, smartphones, and embedded systems.

Why Did We Need Linux?

UNIX was expensive and closed-source – Linux became the free alternative.
Flexibility and customization – Linux allowed users to modify and tailor the OS.
Security and stability – Linux provided a more secure and stable computing environment.

Key Features of Linux

Open-source: Anyone can modify and improve it.
Multi-user & multitasking: Supports multiple users and processes simultaneously.
Security: Built-in permissions, encryption, and firewall capabilities.
Portability: Runs on various hardware platforms.
Scalability: Powers everything from small devices to supercomputers.

2. Understanding the Linux Kernel

What is the Kernel?

The Linux kernel is the core of the operating system—the bridge between hardware and software. It controls everything, from memory management to process execution and device handling.

Why is the Kernel Necessary?

Without a kernel, your computer wouldn't know how to talk to the hardware.

Memory Management : Keeps track of how much memory is used to store what and where.
Process Management : Determines which process can use the CPU and for how long.
Device Drivers : Acts as a mediator between hardware and processes
System calls and Security : Receives requests from processes for services

Components of an Operating System

A complete OS consists of:

Kernel – Handles core system functions (process scheduling, hardware interaction).
GNU Utilities – Provides essential tools (shell, compilers, libraries).

Together, Linux Kernel + GNU create a fully functional OS.

Kernel Space vs. User Space

Kernel Space – It is the portion of memory where the kernel operates and provides its services, with full access to system resources.
User Space – Where applications and user processes run, with limited access for security reasons.

Conceptual Diagram of a UNIX system

Utilities and Applications interact with the Kernel via the Shell which is written in ‘C‘, to perform tasks. Hence users interact with the kernel via shell or system calls which then gives access to hardware and software resources for the applications to execute tasks.

uname -r

Use this command to get basic Kernel information.

3. The Linux Boot Sequence

The Linux boot process is how the system starts when you power it on:

1. BIOS/UEFI Initialization

BIOS (Basic Input/Output System) performs hardware checks using BIOS POST(Power on Self Test).
It looks for a bootable disk and hands over control to the boot loader.

2. Boot Loader (GRUB 2)

After a successful POST test, BIOS loads and executes the boot code from the boot device, located in the first sector of the hard drive.
GRUB (Grand Unified Bootloader) loads the Linux kernel into memory.
It allows you to select the OS (if dual-booting).

3. Kernel Initialization

The kernel decompresses, loads into the memory and executes tasks such as initializing hardware and memory management and mounts the root filesystem.
It looks for an init process to initialize user space and sets up the processes needed to run in user environment.

4. Init Process (systemd)

The init process then calls the systemd daemon.
systemd starts system services and targets, bringing the OS to a working state.
Essential services (networking, logging, display) are initialized.
Actually systemd the first process with its process id (PID) 1.

5. Shells and Environment Variables

Types of Shells in Linux

Bash (Bourne Again Shell) – Most widely used, powerful scripting features.
Zsh (Z Shell) – Extended features over Bash (auto-completion, themes).
Fish (Friendly Interactive Shell) – User-friendly, with syntax highlighting.

Features of Bash Shell

Command history
Aliases & functions
Script execution (.sh files)
Alias for commands

Environment Variables

Environment variables store system-wide settings. To check all the Environmental variables in the systems use :

env

How to create a custom environment variable and make it persistant across logins ?

export VARAIBLE_NAME=value

To make it persistent, in the .bashrc file add this export VARAIBLE_NAME=value line at the end of the file using a VIM editor and reboot the system.

6. Linux File Types and Filesystem Structure

Every object in Linux can be considered to be a type of file. This is a generic statement which in principle is actually true.

How to check file types?

ls -ltr

The first character such as d, -, etc. specifies its file type.

Linux File Types

Regular Files (-) – Documents, programs, scripts.
Directories (d) – Folders containing files. /home/user is a type of folder every linux OS has.
Character Files (c) – Represent devices under /dev filesystem that allows OS to communicate with I/O devices like keyboards.
Block Files (b) – Represent storage devices like hard drives.
Symbolic Links (l) – Shortcuts to other files.
Sockets (s) – Allow inter-process communication.
Pipes (p) – Enable data transfer between processes. Used to connect one process’s input to other process and this is a unidirectional flow.

Linux Filesystem Architecture

The Linux filesystem is a hierarchical structure starting from root (/):

Final Thoughts

Linux has come a long way from being a hobby project to powering the internet. Whether you're a beginner or an advanced user, understanding its history, architecture, and core components is key to mastering it.

In the next blog, we’ll dive deeper into user and file access management, package management, storage, networking and much more. Stay tuned! 💙🐧

Leveraging efficient workflows of GitHub Actions for Seamless Automation

Yash Patil — Sat, 25 Jan 2025 06:50:27 +0000

Introduction

Hello again !! In this blog, lets explore GitHub Actions and their role in automating CI/CD workflows. We’ll walk through a detailed explanation of a GitHub Actions YAML file I created for automating the CI/CD pipeline of a MERN stack application. Additionally, lets also compare GitHub Actions with Jenkins to know perks and cons of both.

What Are GitHub Actions?

GitHub Actions is a CI/CD and automation platform directly integrated into GitHub. It allows developers to automate their workflows—such as testing, building, and deploying—by writing simple YAML configuration files. These workflows are triggered by events such as pushes, pull requests, or even manual intervention.

GitHub Actions goes beyond just DevOps and lets you run workflows when other events happen in your repository. For example, you can run a workflow to automatically add the appropriate labels whenever someone creates a new issue in your repository.

Why Use GitHub Actions?

Seamless Integration: It’s built into GitHub, so there’s no need for separate configuration. If any organization uses Github for its codebase, it just makes sense of use Github Actions for their automation and SDLC workflows.
Automation: Automate repetitive tasks like testing, building, and deploying code.
Flexibility: Trigger workflows based on a variety of events such as commits to certain branches, issues created to the repository, PRs created, manual triggering with input parameters and many more.
Cost-Effectiveness: Its free for public repositories and has generous limits for private repositories.

Key Components:

You can configure a GitHub Actions workflow to be triggered when an event occurs in your repository, such as a pull request being opened or an issue being created. Your workflow contains one or more jobs which can run in sequential order or in parallel. Each job will run inside its own virtual machine runner, or inside a container, and has one or more steps that either run a script that you define or run an action, which is a reusable extension that can simplify your workflow.

Workflows:

A Workflow is a configurable automated process that will run one or more jobs. Workflows are defined by a YAML file checked in to your repository and will run when triggered by an event in your repository, or they can be triggered manually, or at a defined schedule.

Workflows are defined in the .github/workflows directory in a repository. A repository can have multiple workflows, each of which can perform a different set of tasks such as:
- Building and testing pull requests
- Deploying your application every time a release is created
- Adding a label whenever a new issue is opened
Events:

An event is a specific activity in a repository that triggers a workflow run. For example, an activity can originate from GitHub when someone creates a pull request, opens an issue, or pushes a commit to a repository. You can also trigger a workflow to run on a schedule, by posting to a REST API, or manually.
Jobs:

A job is a set of steps in a workflow that is executed on the same runner. Each step is either a shell script that will be executed, or an action that will be run. Steps are executed in order and are dependent on each other. Since each step is executed on the same runner, you can share data from one step to another. For example, you can have a step that builds your application followed by a step that tests the application that was built.
Steps: Individual tasks in a job, like running a script or executing an action.
Runners: A runner is a server that runs your workflows when they're triggered. Each runner can run a single job at a time. GitHub provides Ubuntu Linux, Microsoft Windows, and macOS runners to run your workflows. Each workflow run executes in a fresh, newly-provisioned virtual machine.

So before going forward, check out the MERN Stack Application we will be working with at this blog and checkout this project’s repository.

CI/CD Workflow for the MERN Stack Application

Below is the complete GitHub Actions YAML file which i have written for this project from scratch. Let’s break it down step by step.

Full YAML File:

name: Build and Deploy a MERN Stack Application

on:
    workflow_dispatch:

env:
    DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
    DOCKERHUB_PASSWORD: ${{ secrets.DOCKERHUB_PASSWORD }}
    VERSION: ${{ github.sha }}
    INSTANCE_IP: ${{ secrets.INSTANCE_IP }}
    SSH_KEY: ${{ secrets.PRIVATE_KEY }}

jobs:
    Build-Push:
        name: Build and Push Docker Images
        runs-on: ubuntu-latest

        steps:
        - name: Checkout code
          uses: actions/checkout@v2

        - name: Login to DockerHub
          run: echo $DOCKERHUB_PASSWORD | docker login -u $DOCKERHUB_USER --password-stdin

        - name: Build and push Frontend Image
          run: |
            cd ./mern/frontend
            docker build -t $DOCKERHUB_USER/mern-frontend:$VERSION .
            docker push $DOCKERHUB_USER/mern-frontend:$VERSION

        - name: Build and push Backend Image
          run: |
            cd ./mern/backend
            docker build -t $DOCKERHUB_USER/mern-backend:$VERSION .
            docker push $DOCKERHUB_USER/mern-backend:$VERSION

    Deploy:
        name: Deploy the application
        runs-on: ubuntu-latest
        needs: Build-Push

        steps:
        - name: Checkout code
          uses: actions/checkout@v2

        - name: Add SSH Key
          run: |
            mkdir -p ~/.ssh
            echo "${{ secrets.PRIVATE_KEY }}" > ~/.ssh/id_rsa
            chmod 600 ~/.ssh/id_rsa

        - name: Add EC2 to known hosts
          run: |
            ssh-keyscan -H $INSTANCE_IP >> ~/.ssh/known_hosts

        - name: Copy startup-script and docker-compose to EC2
          run: |
            scp -i ~/.ssh/id_rsa startup-script.sh ubuntu@$INSTANCE_IP:/home/ubuntu
            scp -i ~/.ssh/id_rsa docker-compose.yml ubuntu@$INSTANCE_IP:/home/ubuntu

        - name: Run deployment script on EC2
          run: |
            ssh -i ~/.ssh/id_rsa ubuntu@$INSTANCE_IP "bash /home/ubuntu/startup-script.sh $VERSION"

Step-by-Step Explanation

Workflow Trigger: `on: workflow_dispatch`

Purpose: This allows manual triggering of the workflow from the GitHub Actions interface.
Why: Useful for deployments where you want to ensure readiness before running the CI/CD pipeline.
We can also configure these triggers to execute our workflows by events such as commits to certain branches, issues raised to the repository, PRs created, manual triggering with input parameters and many more.

`env` Section:

This defines environment variables used across jobs. We have to configure these secrets under repository Settings/Secrets and Variables/Actions/Repository Secrets.

I have created these variables or secrets to avoid hardcoding values of repetitive and secrets information in the pipeline script.

DOCKERHUB_USER and DOCKERHUB_PASSWORD are pulled from GitHub Secrets for secure access to my Dockerhub account while pushing and pulling built images.
VERSION uses github.sha to dynamically tag Docker images with the commit SHA. This voids the need to hardcode image tags.
INSTANCE_IP specifies the target EC2 instance IP for deployment.
SSH_KEY contains the private key for SSH access to the EC2 instance.

Job 1: `Build-Push`

Runs-on: ubuntu-latest

Purpose: Specifies the environment where the job will run. GitHub-hosted runners (like ubuntu-latest) provide a pre-configured environment. When a workflow starts that is meant to be “run-on“ one of these GitHub Hosted Runners, GitHub spins up a fresh Virtual Machine(or a container) based on the specified environment(here ubuntu-latest). After the Job is executed, the VM is destroyed which ensures a clean environment for every new job or a workflow.
This Runner comes with pre-configured environment, that’s the reason why i am able to run Docker commands without explicitly installing it.
These Runners are managed and maintained entirely by GitHub and our project has no ownership over it.

Steps in Build-Push Job:

Checkout Code:

- name: Checkout code
  uses: actions/checkout@v2

* **Purpose:** `actions/checkout@v2` is a pre-built action from GitHub’s marketplace. Retrieves the code from the repository to the runner.

Login to DockerHub:

- name: Login to DockerHub
  run: echo $DOCKERHUB_PASSWORD | docker login -u $DOCKERHUB_USER --password-stdin

* **Purpose:** Logs into DockerHub to allow pushing images.

*

    In this step I used the Environment Variables created at the start of the script.

Build and Push Frontend Image:

- name: Build and push Frontend Image
  run: |
    cd ./mern/frontend
    docker build -t $DOCKERHUB_USER/mern-frontend:$VERSION .
    docker push $DOCKERHUB_USER/mern-frontend:$VERSION

* **Purpose:** Builds and tags the frontend image, then pushes it to DockerHub. The folder already contains the custom DockerFile for the frontend service.

* Thing to notice is how i use `VERSION` variable that uses `github.sha` to dynamically tag Docker images with the commit SHA. This voids the need to hardcode image tags.

Build and Push Backend Image:

- name: Build and push Backend Image
  run: |
    cd ./mern/backend
    docker build -t $DOCKERHUB_USER/mern-backend:$VERSION .
    docker push $DOCKERHUB_USER/mern-backend:$VERSION

* **Purpose:** Similar to the frontend step but for the backend application.

Here our First Job is completed where we Checked out the repository codebase, built the Docker Images for the application services and pushed them to the DockerHub.

After the execution of this Job, the GitHub hosted runner (ubuntu-latest) is terminated.

Job 2: `Deploy`

For Deploying our application I will be using a AWS EC2 Instance. So stick with me, I will also show the execution of the entire pipeline, this is just the explanation of the pipeline script.

Needs: Build-Push

Ensures this job starts only after the Build-Push job completes successfully.

Steps in Deploy Job:

Checkout Code:

* Same as the previous step; ensures code is available on the runner.

Add SSH Key:

- name: Add SSH Key
  run: |
    mkdir -p ~/.ssh
    echo "${{ secrets.PRIVATE_KEY }}" > ~/.ssh/id_rsa
    chmod 600 ~/.ssh/id_rsa

* **Purpose:** Configures the private SSH key for accessing the EC2 instance securely. I am using the `${{ secrets.PRIVATE_KEY }}` which stores the private key to the deployment server (EC2 Instance).

Add EC2 to Known Hosts:

- name: Add EC2 to known hosts
  run: |
    ssh-keyscan -H $INSTANCE_IP >> ~/.ssh/known_hosts

* **Purpose:** Prevents host authenticity issues during SSH commands.

Copy Deployment Files to EC2:

- name: Copy startup-script and docker-compose to EC2
  run: |
    scp -i ~/.ssh/id_rsa startup-script.sh ubuntu@$INSTANCE_IP:/home/ubuntu
    scp -i ~/.ssh/id_rsa docker-compose.yml ubuntu@$INSTANCE_IP:/home/ubuntu

* **Purpose:** This transfers my custom startup shell script and the docker-compose file for deployment to the target EC2 instance.

Run Deployment Script on EC2:

- name: Run deployment script on EC2
  run: |
    ssh -i ~/.ssh/id_rsa ubuntu@$INSTANCE_IP "bash /home/ubuntu/startup-script.sh $VERSION"

    **Purpose:** Executes the deployment script remotely on the EC2 instance to start the application using Docker Compose. I export the `$VERSION` variable to the EC2 Instance using the Shell Script which i have referenced in the Docker-Compose file to pull and run the correct versions of application service containers.

Pipeline Execution

Lets execute this pipeline shall we ?!?!

We have configured our workflow to be manually triggered. Click on Run workflow to execute the pipeline.

So the Workflow for our first job Build and Push Docker Images is started and lets checkout the logs of each step.

This sets up the GitHub hosted Runner for our job :

Checks out the codebase of our Repository:

Logs In to my DockerHub Account inside the Runner, Builds the images and pushes them to DockerHub:

As you can see our First Job is successfully executed and our application service’s docker images are built and pushed to Docker hub with dynamic tags.

As soon as our first job Build-Push i=has completed its execution, our deployment job(which i have configured as a separate job) starts its execution.

For the deployment of our application I have created an AWS EC2 Instance. NOTE : Make sure Docker and Docker-Compose are installed on the instance and set permissions to execute docker daemon. Also enable Inbound rule at port 5173 of the instance, because this is where we will access our application.

Job starts its execution.. and we can checkout its logs :

This copies the bash Startup-script and the docker-compose file to the EC2 Instance.

And with this command :

It SSH’s into the server and executes the custom shell script which deploys the application.

Lets access it using the Instance IP and the mapped port number :

And there you go!! We have successfully deployed our 3-tier MERN Stack application to a remote EC2 Instance which acts like a Deployment server using GitHub Actions.

Jenkins vs GitHub Actions

Having built CI/CD pipelines using both I have felt quite a difference between both. GitHub Actions uses its compute resources more efficiently and is more fast and easy to begin with as a beginner. Perfect for projects hosted on GitHub, as it requires minimal setup and no external tools to start automating workflows.
Jenkins requires separate installation and setup, either on-premises or via cloud infrastructure. Pipelines are defined using Groovy scripts, which can have a steeper learning curve for beginners compared to YAML. But it is best for advanced users who need fine-grained control over their CI/CD pipelines and want to access highly extensible UI with a rich plugin ecosystem (over 1,800 plugins), allowing integration with nearly any tool or platform.

Checkout my Jenkins pipeline setup for this same application, check out [this blog].

Connect with me on LinkedIn.

Building a Robust CI/CD Pipeline using Jenkins for a MERN Stack Application

Yash Patil — Sat, 25 Jan 2025 06:33:45 +0000

As modern software development emphasizes agility and reliability, CI/CD (Continuous Integration/Continuous Deployment) pipelines have become indispensable. In this blog, we’ll delve into the implementation of a CI/CD pipeline for a MERN stack application (React.js frontend, Node.js/Express backend, and MongoDB database) which we have already worked on (check out this blog to know about the application), highlighting semantic versioning, why versioning is needed in modern-day applications, Docker image versioning, automated deployment, and more. Let’s walk through each stage of the pipeline with detailed explanations and examples.

First let’s talk about what are CI/CD pipelines and how my built pipeline fits into the picture..

What is a CI/CD Pipeline?

A CI/CD pipeline automates the integration and deployment of code, ensuring that changes are validated, built, tested, and deployed efficiently. It minimizes manual intervention, reduces errors, and accelerates delivery. In our project, the pipeline not only automates these processes but also increments the application version for each build.

Your pipeline embodies a real-world DevOps best practice: Continuous Integration and Continuous Deployment (CI/CD). Let’s break this down in the context of industry-standard workflows and explain why each step is necessary.

Versioning in Real-World Practices

Semantic Versioning

Semantic Versioning is the most common standard for versioning applications in the industry. It follows the format:

MAJOR.MINOR.PATCH

PATCH: Incremented for bug fixes or minor improvements (e.g., 1.0.1 → 1.0.2).
MINOR: Incremented for new features that are backward-compatible (e.g., 1.0.0 → 1.1.0).
MAJOR: Incremented for changes that are backward-incompatible (e.g., 1.0.0 → 2.0.0).

Is It Necessary to Increment the Version for Every Commit or a pipeline build job?

No, in many cases, a version bump only happens when a release is being prepared for deployment to production or for public consumption.
Yes, if every commit directly impacts a deliverable (e.g., in a Continuous Deployment (CD) environment where every commit leads to production changes).

When to Increment Versions in the Real World

Version Bumps Are Typically Tied to Releases

Teams often work in branches (e.g., feature, development, or staging) and only merge into main when a feature or fix is complete.
A version bump occurs when a deployable release is ready, not for every commit.

Scenarios Where Version Increments Happen:

Feature Releases:

* A feature is complete, tested, and merged into `main`. The version is incremented (e.g., `1.1.0`) before releasing it to production.

Bug Fixes:

* A critical bug is fixed, and the version's **PATCH** number is incremented before deployment.

Hotfixes:

* Emergency fixes often lead to quick **PATCH** bumps (e.g., `1.2.1` → `1.2.2`).

Performance Improvements:

* Minor performance updates might warrant a **PATCH** bump, or a **MINOR** bump if they involve significant new optimizations.

Commits That Do Not Trigger Version Increments:

Work-in-progress changes.
Internal refactoring without user-facing impacts.
Development or experimental changes not yet merged to main.

In practice, incrementing versions should be done automatically. In our pipeline, to simulate a continuous deployment pipeline such that every build increments the patch version, signaling minor updates. Builds that are directly deliverable to production environments, version increments are necessary every time.

For this pipeline, we are using Jenkins, a widely popular automation server known for its flexibility and extensive plugin ecosystem. Jenkins excels in orchestrating tasks like building, testing, and deploying applications. Its robust community support, ease of configuration, and compatibility with a wide range of tools make it a go-to choice for CI/CD workflows.

To set up Jenkins, I have created a Jenkins container using the following Docker command:

docker run -p 8080:8080 -p 50000:50000 -d \
    -v jenkins_home:/var/jenkins_home \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $(which docker):/usr/bin/docker \
    --group-add $(stat -c '%g' /var/run/docker.sock) \
    jenkins/jenkins:lts

Explanation of the Arguments:

-p 8080:8080: Maps Jenkins’ web interface to port 8080 on the host machine.
-p 50000:50000: Maps the Jenkins agent communication port to the host machine.
-v jenkins_home:/var/jenkins_home: Persists Jenkins data on the host for durability.
-v /var/run/docker.sock:/var/run/docker.sock: Grants Jenkins access to the Docker daemon, enabling it to manage Docker containers.
-v $(which docker):/usr/bin/docker: Provides Jenkins access to the Docker CLI.
--group-add $(stat -c '%g' /var/run/docker.sock): Adds the Jenkins user to the Docker group for permission to run Docker commands.
jenkins/jenkins:lts: Specifies the Jenkins Long-Term Support (LTS) image.

This setup ensures Jenkins is fully equipped to handle Docker-based workflows, making it an integral part of our CI/CD process.

Also make sure that the Jenkins container has Node and NPM installed on it.

As you can see our Jenkins container is up and running and accessible at localhost:8080 :

I have opted to use a simple pipeline job for our project and written a pipeline script in the Jenkinsfile.

Check out the project repository that contains the codebase of the application, the pipeline script under Jenkinsfile, our docker-compose file and a bash script which is used for deployment of our application but we’ll get to that later.

STAGE ONE : Increment application versions

Our Frontend and Backend services use NPM as their package managers and dependencies handlers. Every package manager tool keeps track of a version in its main build file. package.json file we write for our application has a field version which denotes the current application version. This is also where build information, dependencies and startup scripts are listed.

Build tools have commands to increment the versions of the applications. In this pipeline, every build automatically increments the patch version, simulating the scenario of minor updates.

npm version patch --no-git-tag-version: Updates the patch version in package.json without creating a Git tag.

I have used JQ a lightweight and flexible command-line utility to parse, filter, transform, and process JSON data.

In our case it parses the updated version from package.json files of both frontend and backend services and stores them in environmental variables env.FRONTEND_VERSION & env.BACKEND_VERSION respectively followed by the BUILD_NUMBER which is also an environmental variable of Jenkins ecosystem out of the box.

STAGE TWO : Building Docker Images and pushing them to Dockerhub.

Building Docker Images for Every New Version

When changes are committed, rebuilding the Docker images ensures that the application is packaged with the latest code and dependencies.

Why is this important in industry?

Immutable Deployments:

* Docker images are snapshots of your application and its environment. By creating a new image for every version, you guarantee the environment for that version is consistent, regardless of where or when it is deployed.

Eliminates "Works on My Machine" Issues:

* All developers, testers, and production environments use the same image, avoiding discrepancies between local setups and production.

Supports Rollbacks:

* If something breaks, you can redeploy an older image/version without rebuilding the application.

Pushing Images to a Centralized Repository

By pushing images to Docker Hub (or another registry):

Centralized Access:

* Teams and systems can pull the latest images without needing access to the source code.

Supports Distributed Teams:

* Developers across different locations can pull the same image, ensuring consistency.

Versioned History:

* The registry acts as a timeline of all your application versions, making it easy to trace or rollback.

In industry, container registries like Docker Hub, AWS ECR, or Azure Container Registry are used to store and manage these images.

We will tag our image with ${env.FRONTEND_VERSION} and ${env.BACKEND_VERSION} for precise versioning.

Sorry for the small image.

We have already created Dockerfiles for the frontend and backend services at paths ./mern/frontend and ./mern/backend respectively. We build the images and tag them using our dockerhub username, the image repository for the individual service and the updated versions stored in ${env.FRONTEND_VERSION} and ${env.BACKEND_VERSION}.

We have created a environment variable to store value for the dockerhub username as to not hardcode it.

After the images are built we need to push them to Dockerhub, but to do so we need to login to DockerHub user from our Jenkins ecosystem. I had created usernamePassword type credentials in Jenkins.

And accessed them in the Pipeline script using the withCredentials() function. The --password-stdin flag avoids exposing sensitive data in logs while docker login.

STAGE THREE : Committing Version Updates to Git Repository

Why Commit Version Changes?

Now that we have built and pushed our images with the correct and latest version tags, committing the version bump ensures the repository reflects the current state of the application, keeping the development history consistent and traceable. This is crucial because:

Maintains Accurate State in the Repository:

* The `package.json` or equivalent file reflects the actual version of the deployed application.

Avoids Conflicts:

* Without this step, multiple contributors could unknowingly use the same old version number, leading to conflicts.

Collaboration:

* Other contributors always pull the latest version with updated dependencies, reducing confusion about which version is in production.

Again for this stage I have setup Github Access credentials so that the Jenkins User can commit the version bump and changes to the Main Branch.

Git Config: Sets up Jenkins as the Git user.
Remote URL Update: Includes the GitHub PAT (Personal Access Token) for secure authentication.
Commit and Push: Tracks version changes and pushes them to the main branch.

STAGE FOUR : Deploying the Application with the Latest Version to the Server

As for deploying the application I had used an EC2 instance from AWS which acts as a Deployment Server in our pipeline workflow.

There are few Pre-Requisites :

Add SSH rules for Jenkins Server to the EC2 Instance in the security group.
Add an Inbound rule for port 5173 which is where we have to access our application.
Docker and Docker-compose must be installed on the Instance Beforehand the execution of the pipeline.

I have used a sshagent plugin to SSH to the EC2 Instance using “SSH Username with private key” type credentials.

I have also used a startup-script which will be executed once the Jenkins user will SSH into the EC2 Instance.

I am passing ${env.FRONTEND_VERSION} and ${env.BACKEND_VERSION} variables are arguments to the script which it stores as Environmental Variables once we SSH to the EC2 instance and Docker-compose files references these variables for its image tags.

First we copy the script and the docker-compose file to the deployment server(EC2 Instance).
And we have instructed the pipeline to execute this script once it SSH to the EC2 Instance.

We have opted for a Pipeline Job to which we provide the Git Repository to use, the credentials to access it and provide the path to the Pipeline Script written in the JenkinsFile.

PIPELINE EXECUTION

Now lets execute the pipeline and see the console output for its execution of its each stage.

It checks-out the latest codebase from the repository and branch mentioned to build.

It executes the first stage of our script and increments the versions of the services.

Builds the images based on the latest version and code changes and pushes them to Dockerhub.

4. Commit the Version Updates to the Git Repository.

As for the Final stage of the pipeline, it starts a SSH agent for the Jenkins user.

6. Copies the startup-script and the docker-compose file to the deployment server.

7. SSH into the server and executes the startup-script which starts up our application using docker-compose.

As you can see, our application is up and running in the deployment server and our pipeline has been successfully executed.

And Finally lets access our application using the server’s public IP at port 5173.

Well that’s it, this CI/CD pipeline demonstrates a practical approach to automating the development and deployment lifecycle of a MERN stack application. From incrementing application versions and building Docker images to committing changes and deploying on an EC2 instance, every stage reflects a real-world workflow followed in the industry. While this example focuses on a specific setup, such as using Docker for containerization and Jenkins for pipeline orchestration, the foundational concepts remain the same. Depending on project requirements, additional stages like automated testing, security scans, or performance monitoring might be included, and deployment environments may vary—ranging from Kubernetes clusters to other cloud platforms. This flexibility and modularity make pipelines like this a cornerstone of modern DevOps practices.

Setting Up an NGINX Reverse Proxy with a Node.js Cluster Using Docker

Yash Patil — Fri, 10 Jan 2025 20:14:54 +0000

In this blog, I’ll walk you through a project where I set up an NGINX server as a reverse proxy to handle requests for a Node.js application cluster. The setup uses Docker to containerize both the NGINX server and the Node.js applications, enabling seamless scaling and management. By the end of this, you'll understand why NGINX is an essential tool for modern web development and how to configure it for such use cases.

What is NGINX?

NGINX (pronounced "engine-x") is a high-performance web server and reverse proxy server. It is widely used for its speed, reliability, and ability to handle concurrent connections. Here are some of its key functionalities:

Web Server: NGINX can serve static files like HTML, CSS, and JavaScript with exceptional speed.

Apache web server also provides this functionality but Nginx is favored for its high performance, low resource consumption, and ability to handle a large number of concurrent connections.
Reverse Proxy: It can forward incoming client requests to backend servers or upstream servers and return the responses to the client. This improves scalability and security. How? In this scenario the end user dont directly send request to the backend servers, instead Nginx acts like a mediator and handles this task.
Load Balancing: NGINX can distribute incoming traffic across multiple backend(upstream) servers using algorithms like round-robin or least connections(Default being the round-robin algorithm).

Kubernetes Ingress Controller: NGINX is often used as an ingress controller in Kubernetes clusters. In this role, NGINX receives requests from a cloud load balancer and routes them to services inside the cluster. This ensures that the cluster remains secure and only the load balancer is exposed to the public.

Bonus: NGINX in Kubernetes

When used as a Kubernetes ingress controller, NGINX takes on a similar role but within a cluster. Here’s how it works:

A cloud load balancer forwards requests to the NGINX ingress controller.
The ingress controller routes the requests to the appropriate Kubernetes service.
The service forwards the requests to the pods (application instances).

This setup ensures that the Kubernetes cluster remains secure, with only the cloud load balancer exposed to external traffic.

In this project, we use NGINX as a reverse proxy and load balancer for a Node.js cluster serving a web-page.

Here is the Github link for the project which consists of the source code, custom nginx configuration created by me and the Docker-Compose file used to containerize the whole setup.

Project Overview

The project consists of the following components:

NGINX Server: Listens on port 8080 and forwards incoming HTTP requests to a Node.js cluster.
Node.js Cluster: Comprises three Docker containers, each running a Node.js application on port 3000.
Docker Compose: Orchestrates the deployment of all containers.

Here’s how the setup works:

A client sends an HTTP request to the NGINX server on port 8080.
NGINX, acting as a reverse proxy, forwards the request to one of the Node.js containers using a round-robin load-balancing strategy.
The Node.js container processes the request and returns the response via NGINX.

Custom NGINX Configuration

Below is the NGINX configuration file written by me used in this project:

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    include mime.types;

    # Upstream block to define the Node.js backend servers
    upstream nodejs_cluster {
        server  app1:3000;
        server  app2:3000;
        server  app3:3000;
    }

    server {
        listen 8080;  # Listen on port 8080 for HTTP
        server_name localhost;

        # Proxying requests to the Node.js cluster
        location / {
            proxy_pass http://nodejs_cluster;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

Explanation:

So basically we have blocks like server, http, events, etc. And inside these block we have directives which decide the behavior of the server.
Nginx setups worker_processes which do the work of getting and processing the requests from the browsers. These handle several concurrent requests from the users in a single threaded event loop. The worker processes influences how well Nginx handles the traffic. This should be tuned according to the server’s hardware and expected traffic load. At production level its always advised to set the worker processes with the equivalent no. of CPU cores.
Then we setup worker_connections directive in the events block. This configures how many concurrent connections each worker process should handle.
The main logic of the server is defined in the http block. We can also configure Nginx to listen for requests using HTTPS protocol and SSL encryption but for this simple setup i have not configured it. So the http block defines at what port nginx handles user requests, where to forward it to for particular domains or IP addresses.
The server block listens on port 8080 and forwards requests to the upstream servers defined.
The upstream block defines the Node.js backend servers (three containers in our case). When Nginx acts as a reverse proxy, the request coming to the backend servers originates from Nginx, not directly from the client. AS a result, backend servers would see the IP address of the Nginx server as the source of request.
Why i have mentioned :

well docker’s internal **DNS** service resolves app1, app2 and app3 to the nodejs containers services we created in the docker compose file.

We also want to forward information from the original client requests. This provides useful info for logging purposes. The proxy_pass directive sends requests to the upstream cluster, while headers like Host and X-Real-IP preserve client information.
Another important configuration must be include mime.types; in response to the client. When Nginx returns the response from upstream servers, it can include the type of file Nginx is serving which helps the browser to process and render the files.

So that’s pretty much it about Nginx configuration and this will be foundational while setting up other projects as the logic pretty much remains the same.

Docker Compose File

Here’s the docker-compose.yml file that defines the entire setup:

version: '3'
services:
  app1:
    build: ./app
    environment:
      - APP_NAME=App1
    image: yashpatil16/nginx-app1:latest
    ports:
      - "3001:3000"

  app2:
    build: ./app
    environment:
      - APP_NAME=App2
    image: yashpatil16/nginx-app3:latest
    ports:
      - "3002:3000"

  app3:
    build: ./app
    environment:
      - APP_NAME=App3
    image: yashpatil16/nginx-app3:latest
    ports:
      - "3003:3000"

  nginx:
    build: ./app/nginx
    image: yashpatil16/nginx:nodejs-app
    ports:
      - "8080:8080"
    depends_on:
      - app1
      - app2
      - app3

Explanation:

The app1, app2, and app3 services each build and run a Node.js application, exposing port 3000 internally.
The nginx service builds the NGINX image, exposing port 8080 to the host.
The depends_on directive ensures that the Node.js containers start before NGINX.

I have also written a custom DockerFile for the Nginx server container to instruct it to use my configuration instead of the default configuration file.

Running the Project

Build and start the containers:
```
docker-compose up --build
```
Access the application in your browser:
```
http://localhost:8080
```
NGINX will forward the request to one of the Node.js containers and return the response.

To verify the Round-Robin loadbalancing approach check the logs :

As you can requests are served by different containers, in our case App1,2,3 as mentioned in the Docker Compose file.

Conclusion

Well that’s pretty much it. We understood what Nginx is, what functionalities it offers and how to setup a Nginx server as a reverse-proxy for our upstream servers. This was a simple setup, but for any other projects the logic remains the same with more configurations here and there.

And connect with me at LinkedIn : https://www.linkedin.com/in/yash-patil-24112a258/

Containerizing a MERN Stack Application!

Yash Patil — Fri, 03 Jan 2025 20:06:30 +0000

In this blog, I’ll walk you through a simple yet powerful project that demonstrates the MERN stack in action. We will go through what MERN is but also how it works together to create a robust application. Plus, we’ll explore how Docker helps containerize and orchestrate everything seamlessly.

Here is the Github link for the project which consists of the source code and the Docker-Compose file used to containerize the application.

NOTE: This application is an open-source project to showcase the process of containerizing existing applications.

Before beginning to containerize the application lets first understand what is MERN framework is and its architecture and structure and how it works.

What is MERN?

MERN stands for MongoDB, Express.js, React, and Node.js—a popular stack for building full-stack JavaScript applications. Here’s how each component contributes:

MongoDB: A NoSQL database that stores data in JSON-like documents, and acts as the database layer.
Express.js: A lightweight web framework for Node.js, used to build business logic of the application and manage HTTP requests and define API endpoints.
React: A frontend library for building dynamic and responsive user interfaces and client side logic. It fetches data from the backend via REST API or graphQL.
Node.js: A JavaScript runtime environment that powers the backend, enabling JavaScript to run on the server. It acts a virtual server on top of our servers or host system on which the application might be hosted. Hence it hosts the application, allowing it to run and be accessed over the internet.

Together, MERN allows developers to write the entire application in JavaScript, streamlining the development process.

For this basic MERN stack application which allows us to Create, View, Edit and Delete employee records, we have a frontend and a backend folder.

The backend of this MERN stack application is designed to handle data operations for managing employee records. Here’s a breakdown of how the connection.js, records.js, and server.js files interact to make everything function seamlessly.

1. Database Connection: `/db/connection.js`

This file establishes a connection to the MongoDB database.

Client : In this case, the database server is named mongodb (as defined in the docker-compose.yml file), and it listens on port 27017. This the database connection string and we can also read this connection string using a environment variable or a config file.

Database Reference: The employees database is referenced using client.db("employees"), and this reference is exported as db. This enables other parts of the application, such as API routes, to interact with the database.

2. API Routes: `records.js`

The records.js file defines a set of RESTful API endpoints for CRUD operations (Create, Read, Update, Delete). These routes use the express.Router to group and manage endpoints under the /record path and defines RESTful routes to handle requests for employee data using the db object.

3. Server Initialization: `server.js`

The server.js file is the entry point for the backend application.

Express App: An Express application is created to handle HTTP requests and responses.
Middleware:
- cors: Enables Cross-Origin Resource Sharing, allowing the frontend (on a different port) to communicate with the backend.
- express.json: Parses incoming JSON payloads in request bodies.
Route Mounting: The records routes are mounted at the /record path. For example, a request to /record is handled by the logic defined in records.js.
Server Startup: The app listens on port 5050 (or another port if specified in the PORT environment variable). A message is logged to confirm the server is running.

So basically that sums up the backend logic of the application.

When a request is made from the browser using the REACTjs frontend application, the Nodejs listens for the incoming HTTP requests on a specified port.
Passes the request to Express.js which matches the request to the defined route.
It executes the corresponding logic, like querying(CRUD operations) the database or prepare a response, etc.
Then Node takes the response by Express.js and sends it back to the Browser where it is displayed to the user via the React.js frontend application.

And that’s the actual workflow of a MERN stack application.

Containerizing the Application

What is Docker Compose?

Docker Compose is a tool that simplifies the management of multi-container applications. Using a docker-compose.yml file, you can define and configure services, networks, and volumes needed for your application. With a single command, Docker Compose can build, start, and orchestrate all the containers defined in the file.

Now lets begin to containerize the application. The docker-compose.yml file orchestrates the three services (frontend, backend, and MongoDB) to work together seamlessly. I have created Dockerfiles for both the frontend and the backend which we will run as separate containers and create a MongoDB container using a image from the DockerHub.

services:
  frontend:
    build: ./mern/frontend
    ports:
      - "5173:5173"
    networks:
      - mern

  backend:
    build: ./mern/backend
    ports:
      - "5050:5050"
    networks:
      - mern
    depends_on:
      - mongodb

  mongodb:
    image: mongo
    ports:
      - "27017:27017"
    networks:
      - mern
    volumes:
      - mongo-data:/data/db

networks:
  mern:
    driver: bridge

volumes:
  mongo-data:
    driver: local

Frontend Service:

- build: ./mern/frontend`: Specifies the Dockerfile for the frontend 
   application located in the `mern/frontend` directory.

- ports: "5173:5173"`: Maps port 5173 on the host to port 5173 in the container, making the React app accessible.

- networks: `mern`: Connects the service to the `mern` network for inter-service communication.

Backend Service:

* `build: ./mern/backend`: Specifies the Dockerfile for the backend application located in the `mern/backend` directory.

* `ports: "5050:5050"`: Maps port 5050 on the host to port 5050 in the container for backend access.

* `depends_on: mongodb`: Ensures MongoDB starts before the backend.

* `networks: mern`: Connects the service to the `mern` network.

MongoDB Service:

* `image: mongo`: Uses the official MongoDB image to run the database.

* `ports: "27017:27017"`: Maps port 27017 on the host to port 27017 in the container for database access.

* `volumes: mongo-data:/data/db`: Mounts a Docker volume named `mongo-data` to `/data/db` inside the container for persistent storage. The volume is mounted to `/data/db` inside the MongoDB container because `/data/db` is the default directory used by MongoDB to store its database files.

* `networks: mern`: Connects the service to the `mern` network.

Networks:

* `mern`: A custom bridge network allowing the frontend, backend, and MongoDB services to communicate.

Volumes:

* `mongo-data`: A named volume to persist MongoDB data between container restarts. This volume is created under `/var/lib/Docker/volumes`

Now make sure you are in the directory where the docker-compose.yml is present and use to following command to spin up all the containers :

docker-compose up

Note: For the first time it may take some time to build the images and run the containers.

Now access the frontend application at localhost:5173 where out React frontend application is running.

Now even if i stop and recreate all the containers, the records will still be there since we have used a host volume and mounted it to the mongoDB container. This allows the data persistence.

Stop all containers using :

docker-compose down

And that’s how easy it is to work with docker compose in a multi container environment.

Summary

Frontend: A React app running on port 5173, providing a user-friendly interface for managing employee records.
Backend: A Node.js API on port 5050, processing business logic and interacting with MongoDB.
Database: MongoDB, configured with a connection URL and persisting data via a Docker volume.
Docker Compose: Orchestrates the entire stack, making it easy to spin up the application.

Going forward the entire workflow of a MERN stack application is pretty much similar with a few teaks here and there but the logic remains the same.

Always make sure the to go through the application codebase and understand how the application works before proceeding forward which will make it easy to use tools like Docker or Docker-Compose.

Feel free to checkout my blog : yashpatilofficial.hashnode.dev & dev.to/yash_patil16

And connect with me at LinkedIn : https://www.linkedin.com/in/yash-patil-24112a258/

Containerizing a Django Web Application: Serving Static Pages with Docker

Yash Patil — Fri, 03 Jan 2025 16:51:54 +0000

In this blog post, we'll walk through the process of containerizing a Python Django web application that serves a simple static HTML page. We will cover the basics of Django project structure, how the an Django application works, and how we can use Docker to create an isolated environment for the application.

Here is the Github link for the project which consists of the source code and the Dockerfile used to containerize the application.

NOTE: This application is an open-source project to showcase the process of containerizing existing applications.

Before beginning to containerize the application lets first understand what is Django framework and its architecture and structure and how it works.

What is Django?

Django is a high-level Python web framework that promotes rapid development and clean, pragmatic design. It simplifies the creation of web applications by providing tools for common tasks like handling URLs, interacting with databases, and managing authentication.

Django Project Structure

A typical Django project consists of several important components. Let's break down the structure of the project we will be working with.

Using django-admin utility that is provided by django to perform administrative tasks for Django projects.

django-admin startproject devops

Using the above command a folder structure for a new Django project is created. This command creates the main project folder (devops), which contains the core settings and configurations.

manage.py is a command-line utility that helps you interact with your Django project. For example, you can run the development server with:

python manage.py runserver

This command starts the Django development server, allowing you to preview your web app on http://127.0.0.1:8000/.

Inside devops you will find :

devops/settings.py: Contains all project settings such as project configuration, middleware,database configuration, static files, etc.

devops/urls.py: The URL configuration file that maps URLs to views. It is responsible for serving the context route. So basically whoever tries to hit the context path mentioned in the file, it will be configured to serve the rendered frontend application.For our project, here's a simple urls.py file that includes routes for the demo app and the Django admin interface:

Now in Django, an application is a modular component of your project. Each application handles specific functionality. Again in our case its serving a static web page.

To create an app, which serves as the core of your project. This can be done using the following command:

python manage.py startapp demo

This creates the demo app folder. Inside this folder, you will find:

demo/urls.py: Manages URLs for the demo app, mapping them to views. We mentioned this file in our base devops project’s urls.py file.
demo/views.py: Contains the core logic of your application, including views that handle requests and return responses. In our case it renders and serves our static web page.

As u can see it renders and serves a demo_site.html present under /demo/templates.

What Happens in the Browser?

The user accesses localhost:8000/demo
Django matches the URL to the view.
The view uses render() to combine the demo_site.html template and the context data.
Django sends the final HTML page to the user's browser.

Dockerizing the Django Web App

Now, let's move on to containerizing the Django project. We'll use Docker to encapsulate the application and its dependencies into a single container.

Here’s a simple Dockerfile i have written to build the container:

Breaking Down the Dockerfile

FROM python:3.14.0a3-alpine3.21: Specifies the base image for the container. We're using an Alpine-based Python image to keep the image small.
WORKDIR /app: Sets the working directory in the container. So going forward every ENTRYPOINT or CMD instruction will be executed inside /app directory.
COPY requirements.txt /app: Copies the requirements.txt file, which lists the necessary Python dependencies, into the container.
COPY devops /app: Copies the entire devops directory (which includes the Django project) into the container.
RUN pip install -r requirements.txt : Installs the required Python packages specified in requirements.txt.
RUN cd /devops takes control inside the main directory where our demo applications and configuration files are present.
ENTRYPOINT ["python3"]: Specifies the default command to run when the container starts. We're using Python 3 to run the application.
CMD ["manage.py", "runserver", "0.0.0.0:8000"]: The default command that will run when the container starts. This runs the Django development server on all available network interfaces (0.0.0.0) at port 8000.

Running the Container

Once the Dockerfile is ready, you can build and run the Docker container with the following commands:

Build the Docker image:
```
docker build -t web-app .
```

I have already built the image:

Run the Docker container:

docker run -d -p 8000:8000 web-app:python

This create and runs a container based on our web-app image we created from the docker file. The above command runs the container on the port 8000 of our host which is mapped to 8000 port of the container where we have configured our Django application(development server) to run.

As you can see the container is running is the background which we specified using the -d flag.

And now lets access our application in the browser.

As u can see we accessed http://localhost:8000/demo/. which was configured to serve our static web page.

Conclusion

And there we go, we’ve containerized a simple Django web application that serves a static HTML page. We covered the basic project structure, how django works and how Docker can help us package and deploy the application in a self-contained environment. Now next time even if its a complex 3 tier Django project, the workflow and logic will be the same.

Connect with me at LinkedIn : https://www.linkedin.com/in/yash-patil-24112a258/

Docker 101: A Guide to Docker Commands, Terminologies & Dockerfile

Yash Patil — Fri, 03 Jan 2025 12:27:34 +0000

Introduction

Docker is a powerful tool for creating, deploying, and running applications in lightweight, portable containers. Containers allow developers to package an application along with its dependencies, making it easy to move and run across different environments. If you're just getting started with Docker, this blog will guide you through the basics, essential commands and how to create a Dockerfile to build your first containerized application.

To get a deeper understanding of Docker and its evolution, check out my first article on Docker.

Installation Guide

Before diving into the basics of Docker, let's first ensure Docker is installed on your machine. Docker can be installed on Windows, macOS, and Linux. For the most up-to-date instructions and installation files, visit the official Docker documentation.

Initially if we are using an EC2 instance, then by default the user doesn’t have permission to execute docker binaries. We have to add the user to the Docker group using :

sudo usermod -aG docker <name of user on EC2>

Creating Docker Image , Docker Container Using Docker.

Firstly check if the docker daemon is running in your system using :

systemctl status docker

Make sure that status is active(running), else there might be some issue with the installation.

Now lets go over basic commands before diving deep.

# Check Docker version
docker --version

# Pull an image from Docker Hub
docker pull <image_name>

# List all containers (running or stopped)
docker ps -a

# Run a container from an image
docker run <image_name>

# Build an image from a Dockerfile
docker build -t <image_name> <path_to_dockerfile>

# List all images
docker images

# Stop a running container
docker stop <container_id>

# Remove a stopped container
docker rm <container_id>

# Remove an image
docker rmi <image_name>

DockerFile

Now lets create our first image using a very basic dockerfile for a basic python script.

The python script is a simple script to print “Hello World!” message.

Make sure the DockerFile and the application, in this case Python Script are in the same directory.

Now lets go over the Dockerfile. A Dockerfile is a template to create images for our applications and its consists of instructions to define the application’s environment, dependencies, and how to access the application.

FROM python:3.14.0a3-alpine3.21
WORKDIR /app
COPY . /app
CMD ["python3", "app.py"]

Let’s go through each instruction in the Dockerfile:

FROM: This instruction sets the base image for your container.
Here, we're using the python:3.14.0a3-alpine3.21 image, which
is a lightweight Alpine-based image with Python 3.14.0a3
installed.
WORKDIR: This sets the working directory inside the container to
/app. All subsequent instructions will be executed in this
directory.
COPY: This command copies all the files from your local directory
(where your Dockerfile resides) into the /app directory
inside the container.
CMD: This command specifies the default command to run when the
container starts. Here, it tells Docker to run the app.py
script using Python 3.

Build the Docker image:

With the Dockerfile and app.py in the same directory, open a terminal and run the following command to build the image:

docker build -t python-hello-world .

This will create a Docker image tagged as python-hello-world.

Check for the built image using “docker images” command.

As we can see from the output an image tagged “python-hello-world” has been created.

Run the container:

After building the image, you can run the container with this command:

docker run python-hello-world

This confirms that your Python script is running inside a Docker container.

Conclusion

In this blog, we've covered the basics of Docker: basic commands, and how to write a Dockerfile. We also demonstrated how to use a simple Dockerfile to run a Python "Hello World" script inside a container. Docker makes it easier to manage and deploy applications by encapsulating everything needed to run the application in a container.

As you continue learning Docker, you’ll discover more advanced techniques and tools. Happy containerizing!

Exploring Docker: The Revolutionary Tool for Modern Application Development

Yash Patil — Fri, 03 Jan 2025 11:02:27 +0000

Introduction

In today’s fast-paced tech landscape, developers and organizations are on a constant quest to make applications faster, more efficient, and easier to manage. One tool that has been instrumental in transforming the way we build, ship, and deploy software is Docker. Let’s delve into Docker’s journey, its architecture, and the reasons why it’s a game-changer for modern development.

Before Microservices: The Era of Monolithic Architecture

Before microservices entered the picture, applications were often built as monolithic entities. In a monolithic architecture, every component of an application—from the user interface to the backend logic and database—was bundled together in a single codebase. While this approach was straightforward and easy to deploy initially, it came with significant challenges:

Scalability Issues: Scaling specific features independently was impossible since the entire application had to scale as a whole.
Maintenance Challenges: Updating or fixing a bug in one part of the application risked breaking the entire system.
Long Deployment Cycles: Even minor updates required thorough testing and redeployment of the entire application.

These limitations highlighted the need for a more modular approach to application design, paving the way for microservices.

The Shift to Microservices and Docker’s Role

Microservices architecture emerged as a solution to the challenges posed by monolithic systems. In this approach, an application is broken down into smaller, independently deployable services. Each service is responsible for a specific business function and communicates with others through APIs.

While microservices solved many issues, they introduced complexities in deployment, communication, and scaling. This is where Docker revolutionized the game. Docker provides a containerization platform that packages an application and its dependencies into a lightweight, portable container. These containers run consistently across different environments, making them ideal for microservices.

The Problem Docker Solves

Traditionally, applications were hosted and deployed directly on servers, often using hardware-level virtualization through virtual machines (VMs). VMs directly claims the hardware resources from the bare metal hardware servers, that is allocated by the hypervisor.

Virtual Machines holds these resources regardless of the usage and these resources once allocated, cannot be used by any other virtual machine.

Docker introduced OS-level virtualization, where containers share the host OS kernel but operate in isolated user spaces. This approach offers host and the Docker engine to allocate hardware resources to containers dynamically. And hence none of the containers permanently hold compute resources of the host system.

How Docker Came Into Being

Docker’s journey began with a company called dotCloud, a platform-as-a-service (PaaS) provider. In 2013, Solomon Hykes and Sebastian Pahl unveiled Docker as an open-source project. The concept of packaging applications in containers wasn’t new, but Docker simplified it to a level that made it accessible and powerful. Its initial release gained rapid adoption, and Docker Inc. soon pivoted to focus solely on container technology.

Docker Architecture

Docker’s architecture is designed to simplify containerization. Here’s an overview of its core components:

Images: Immutable snapshots of an application, its dependencies and its environment, created from Dockerfiles. These are templates used to create containers.
Containers: Lightweight, standalone executables built from Docker images. Containers include everything needed to run an application (the application code, system dependencies, libraries,etc).
Dockerfile: A text file that contains instructions to build a Docker image. It defines the application’s environment, dependencies, and how to access the application.
Docker Engine/Deamon: The runtime that builds, runs, and manages containers. It includes a server (daemon), a REST API, and a command-line interface (CLI). It runs on the host OS.
Docker Client: Users interact with the Docker engine/daemon through the CLI. Clients use commands and REST API to communicate with Docker Engine/Daemon.
Docker Hub: A public registry for sharing and storing Docker images. Developers can pull official or community images or push their own.
Ecosystem: Includes tools like Docker Compose (for managing multi-container applications) and Docker Swarm (for orchestration).

Advantages of Docker

Docker has transformed application development and deployment by offering several key advantages:

Consistency Across Environments: “It works on my machine” is no longer a problem as it solves the dependencies based issues.

Efficiency: Containers are lightweight compared to VMs, leading to better resource utilization.

Speed: Faster builds, startups, and deployments.

Portability: Containers can run on any system with Docker installed, from local machines to cloud environments.

Scalability: Containers make it easy to scale applications horizontally by spinning up additional instances.

Disadvantages of Docker

While Docker is a powerful tool, it’s not without its drawbacks:

It is difficult to manage large amounts of containers without a orchestration tool (Kubernetes).
Containers share the host OS kernel, which can lead to vulnerabilities if not managed properly.
Docker does not provide cross-platform compatibility, means if an application is designed to run in a docker container on windows, then it cant run on Linux or vice-versa. Containers share host OS kernel. A windows container cant run on a Linux kernel and vice-versa, because the required system calls, libraries and APIs differ between OS.
While lightweight, Docker containers still introduce some overhead compared to bare-metal systems.

Conclusion

Docker has undeniably transformed the way we develop, deploy, and scale applications. By addressing the limitations of traditional deployment methods and simplifying the complexities of microservices, Docker has become an indispensable tool for modern developers. Whether you’re building your first containerized app or managing a fleet of microservices, Docker offers a robust foundation to achieve your goals.

So, as you embark on your journey with Docker, remember: it’s not just about containers; it’s about redefining possibilities in application development.

DEV Community: Yash Patil

Managing Storage and Packages in Linux: A Complete Guide

Package Management

Overview

Linux User Management and File Permissions

Why Do We Need User Accounts and File Permissions in Linux?

Types of Accounts in Linux

Understanding the sudo Group

Managing Users and Groups

Key System Files for User Management

File Permissions: Why They Matter

Understanding the Octal Permission System

File Ownership and the chown Command

Conclusion

Introduction to Linux: The Open-Source Revolution

1. The History of Linux: From UNIX to Today

The Birth of UNIX (1960s-1970s)

MINIX and the Rise of Linux (1980s-1991)

Why Did We Need Linux?

Key Features of Linux

2. Understanding the Linux Kernel

What is the Kernel?

Why is the Kernel Necessary?

Components of an Operating System

Kernel Space vs. User Space

Conceptual Diagram of a UNIX system

3. The Linux Boot Sequence

1. BIOS/UEFI Initialization

2. Boot Loader (GRUB 2)

3. Kernel Initialization

4. Init Process (systemd)

5. Shells and Environment Variables

Types of Shells in Linux

Features of Bash Shell

Environment Variables

6. Linux File Types and Filesystem Structure

Linux File Types

Linux Filesystem Architecture

Final Thoughts

Leveraging efficient workflows of GitHub Actions for Seamless Automation

Introduction

What Are GitHub Actions?

Why Use GitHub Actions?

Key Components:

CI/CD Workflow for the MERN Stack Application

Full YAML File:

Step-by-Step Explanation

Workflow Trigger: on: workflow_dispatch

env Section:

Job 1: Build-Push

Runs-on: ubuntu-latest

Steps in Build-Push Job:

Job 2: Deploy

Needs: Build-Push

Steps in Deploy Job:

Pipeline Execution

Jenkins vs GitHub Actions

Building a Robust CI/CD Pipeline using Jenkins for a MERN Stack Application

What is a CI/CD Pipeline?

Versioning in Real-World Practices

Semantic Versioning

Is It Necessary to Increment the Version for Every Commit or a pipeline build job?

When to Increment Versions in the Real World

Version Bumps Are Typically Tied to Releases

Scenarios Where Version Increments Happen:

Commits That Do Not Trigger Version Increments:

Explanation of the Arguments:

STAGE ONE : Increment application versions

STAGE TWO : Building Docker Images and pushing them to Dockerhub.

Building Docker Images for Every New Version

Why is this important in industry?

Pushing Images to a Centralized Repository

STAGE THREE : Committing Version Updates to Git Repository

Why Commit Version Changes?

STAGE FOUR : Deploying the Application with the Latest Version to the Server

PIPELINE EXECUTION

Setting Up an NGINX Reverse Proxy with a Node.js Cluster Using Docker

What is NGINX?

Bonus: NGINX in Kubernetes

Project Overview

Understanding the `sudo` Group

File Ownership and the `chown` Command

Workflow Trigger: `on: workflow_dispatch`

`env` Section:

Job 1: `Build-Push`

Job 2: `Deploy`

1. Database Connection: `/db/connection.js`

2. API Routes: `records.js`

3. Server Initialization: `server.js`