DEV Community: Vishal

🚀 Wrapping Up My GitLab CI/CD Journey with 2 Real Projects

Vishal — Wed, 09 Jul 2025 17:59:37 +0000

🛠️ My Hands-On Dive into GitLab CI/CD

Instead of just learning CI/CD concepts in theory, I decided to put them into practice.I built two real pipelines for real apps — one with a Go Backend, another with a Nodejs — using GitLab CI/CD.

🔧 What I Tried to Do

I worked on two projects to understand GitLab CI/CD in action:

1: Built a 3-stage pipeline (Build → Test → Deploy) for a React frontend and Go backend app. This helped me automate the entire flow from code to container.
2: Set up and used a self-hosted GitLab Runner to run jobs on my own system. It gave me hands-on experience with job execution and custom runner configurations.

Project 1 – Full Stack App Pipeline

Work done on gitlab_link

For this pipeline setup, I used an existing open-source project from GitHub. It’s a full-stack application with a React frontend and a Go backend — a great combo to test out CI/CD workflows.The project simulates a basic web app architecture, making it perfect for understanding how continuous integration and deployment work across both frontend and backend.By using this real-world project, I was able to apply CI/CD concepts practically and troubleshoot challenges that commonly appear in production environments.The project that i have referred - Github_link

🐞 Issues I Faced

At first, I wrote the CI job script assuming it would just execute like a normal shell environment. But I didn’t know that GitLab’s SaaS runners use Docker images to execute each job.That meant:- The runner didn’t know how to run my commands — because it had no idea what environment or dependencies were needed.
I wanted to use docker build inside my pipeline job, but it kept failing. What I didn’t realize is that: You can’t directly use Docker inside a container unless you set it up properly.GitLab SaaS runners run jobs inside containers, and to use Docker commands there, you need a setup called Docker-in-Docker (DinD).

What I Learned

To fix this, I learned about something called Docker-in-Docker (DinD).

DinD allows Docker to run inside a Docker container by connecting to a Docker daemon.
By enabling privileged: true and using the official docker:latest image along with a service like docker:dind, I was able to use Docker commands inside my pipeline.

Here’s a snippet from the working .gitlab-ci.yml:

image: docker:latest

services:
  - docker:dind

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_TLS_CERTDIR: ""

before_script:
  - docker info

script:
  - docker compose build
  - docker compose up -d

Below is the .gitlab-ci.yml file screenshot

Project 2 – Using Custom GitLab Runners

Work done on Gitlab_link

For my second project, I wanted to explore how GitLab CI/CD can work across multiple environments by using two custom GitLab runners. I set up one runner on my local machine (Runner dev) and another on an AWS EC2 instance (Runner aws). The goal was to simulate a real-world DevOps workflow where the application is built locally and store the image of application on gitlab artifacts and then deployed remotely.

On Runner dev (my local system), I built the Docker image of the Nodejs application and then pushed that image as a build artifact or stored it temporarily. GitLab CI/CD allowed me to pass this artifact between jobs. In the next stage, the job assigned to Runner aws (on AWS) picked up this artifact. There, I used the built image to deploy the app by running the container on the EC2 instance.

Issue I Faced

One of the most confusing problems I ran into was that even after modifying my application code and rebuilding the Docker image on Runner dev (my local machine), those changes weren’t showing up when I deployed the app on Runner aws (EC2). I thought deleting the old Docker image and rebuilding it would ensure the latest version got deployed, but somehow the updates weren’t reflected.

How I Solved the Issue

✅ How I Solved It
After a lot of trial and error, I realized something important GitLab CI with Docker Compose generates images with names like:
<gitlab_folder_name>_<service_name> by default

Initially, I was deleting the container using the name I gave in docker-compose.yml (via container_name), but that doesn't remove the actual image. So even though I ran:

docker-compose build --no-cache

…the image still didn’t reflect my recent code changes — because Docker was still using the existing image with the same name.
Solution? I listed the images with docker images, found the correct one (named like todo_app_from_tws_web), and deleted it using:

docker image rm <name_of_the_image>

Then I rebuilt the image and re-ran the pipeline — and this time, the new changes were reflected correctly on the deployment side (AWS runner).

.gitlab-ci.yml file of Todo app

Stages of Pipeline

Artifacts of the project

🟢 And finally... it’s alive!

Resources That Helped Me Along the Way

I didn't figure it all out on my own - these resources were super helpful during the process:

gitlab docs - From Errors to Execution!
Youtube Tutorial - To visually understand the concept.
Google - For quick searches.

If you're starting out, I highly recommend using these - they make learning much easier!

🙌 Wrapping Up

Working with GitLab CI/CD pipelines across two different setups taught me more than just YAML and jobs — it showed how real DevOps systems behave in action. From understanding how GitLab SaaS runners execute jobs, to handling image build issues across custom self-hosted runners, this hands-on experience gave me clarity on what goes on behind automated deployments.

💬 Over to You

Have you worked with GitLab CI/CD pipelines or tried setting up your own runners? Faced any weird issues like Docker not reflecting changes or environment configs acting up?
I’d love to hear how you tackled them — or even if you're just starting out, feel free to reach out!

Let’s connect and grow together in this DevOps journey:
Connect on LinkedIn
Say hi on X (Twitter)

Running Django & Java Apps in Containers — My Docker Week Recap

Vishal — Tue, 01 Jul 2025 12:29:32 +0000

In my DevOps learning journey, I spent this past week diving into Docker — not just understanding the commands, but applying them in real projects.

This blog summarizes what I built, what I learned, and the small hurdles I faced while containerizing and deploying two full-stack applications.

🚧 What I Tried to Build

I worked on two different projects to get hands-on Docker experience:

✅ A Python Django Web Application
✅ A Java Maven-Based Application

Project -1: Python Django Web Application

🔍 What I Tried to Do

For the first project, I took a basic Django app and containerized it using Docker. My goal was to:

Write a Dockerfile that sets up Python, dependencies, and the Django project.
Use docker-compose.yml to run both the Django app and a MYSQL database together.
Set up a reverse proxy using NGINX to expose the app on a clean port (like 8080).
Make sure everything can be started with just one command using docker-compose up.

This project helped me understand how multi-container apps work, how services interact inside a Docker network, and how to troubleshoot startup issues in web apps.

🐞 Issues I Faced

1. Forgot to Add Containers to the Same Network

At first, my app and NGINX couldn’t talk to each other.I didn’t realize that all related containers need to be on the same Docker network to communicate.This tiny thing cost me a lot of time — everything looked okay, but nothing worked until I added the correct network in the docker-compose.yml.

2. Database Connection Problems

My Django app kept throwing errors when trying to connect to MySQL. After a lot of digging, I found out it was due to small things like incorrect hostnames or missing environment variables.One wrong word in the DB settings was enough to break everything.Lesson learned: always double-check your database config.

3. Used Wrong Name in NGINX Config

In the NGINX config, I accidentally used the container name instead of the service name from the Docker Compose file.This gave me “bad gateway” errors. I didn’t know at the time that NGINX looks for the service name, not the container name, when it tries to connect. It seems obvious now, but it wasn’t when I first started — and that’s okay.

✅ How I Solved Them

1. Docker Network Confusion

Now, whenever I write a docker-compose.yml, I make sure that all services are assigned to a common network — even if it seems optional at first.This way, containers can talk to each other smoothly without throwing “connection refused” errors.

Here's a small reminder I use now:

services:
  web:
    ...
    networks:
      - mynetwork
  nginx:
    ...
    networks:
      - mynetwork
networks:
  mynetwork:

2. MySQL Connection Troubles
A small typo in your environment variable (like MYSQL_USER, MYSQL_PASSWORD, or MYSQL_HOST) can ruin your day. I spent hours figuring out that my Django app couldn’t talk to MySQL just because I had a wrong key in the env file. Now I double-check every setting and make sure all configs match between app and database.

3. NGNIX Configuration Issue
In the beginning, I was giving the service name inside the NGINX config, like:

proxy_pass http://service_name:8000;

Which didn’t work.
So I changed it to:

proxy_pass http://container_name:8000;

Here’s how the app looks running inside Docker containers:

Project -2: Java Maven-Based Application

🔧 What I Tried to Do (Java Maven App)

After successfully containerizing my Django app, I wanted to push myself a little further — this time by working with a Java-based Maven application. The idea was to set up everything inside Docker and manage it through Docker Compose, just like I did with the Django app.But this project came with its own set of new learning curves. Unlike Python, Java applications involve build steps using Maven, so I had to:

Write a custom Dockerfile that builds the app using Maven and then runs the JAR file.
Handle MySQL integration inside containers (again), but with a slightly different config structure.
Configure NGINX to reverse proxy requests to the Java backend.

Issue I Faced

One of the most confusing parts was figuring out where the database settings were written in the Java Maven project. In Django, it's easy to find — it's in settings.py. But here, I had to dig around the project folders to finally find the right file.
Even after I updated the database details correctly, the app still crashed — and the logs didn’t help much. Everything seemed fine: all containers were running, and the database was connected. But the app just wouldn't work, and there was no clear error telling me what went wrong.
And Issue was like this:

Exception in thread "main" java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed at
     com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:108) at 
     com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95) at
     com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at     
     com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:862) at 
     com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:444) at
     com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:230) at
     com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:226) at
     com.mysql.cj.jdbc.MysqlDataSource.getConnection(MysqlDataSource.java:438) at
     com.mysql.cj.jdbc.MysqlDataSource.getConnection(MysqlDataSource.java:146) at
     com.mysql.cj.jdbc.MysqlDataSource.getConnection(MysqlDataSource.java:119) at
     ConnectionManager.getConnection(ConnectionManager.java:28) at
     Main.main(Main.java:8)

✅ How I Solved It

After spending quite some time googling and going through Stack Overflow posts, I finally discovered that the database configuration was located in a file called application.properties inside the path:

src/main/resources/

This file had keys like:

spring.datasource.url=jdbc:mysql://mysql:3306/expenses_tracker?useSSL=false&serverTimezone=UTC
spring.datasource.username=root
spring.datasource.password=Test@123
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

So, I added these database credentials under the environment section of the Java service in the docker-compose.yml file.
But… even after that, the app still wouldn’t come up.
Eventually, I found a Stack Overflow post that pointed out an important issue — Java MySQL drivers block public key retrieval by default, which can silently break DB connections.
So, I updated the URL to:

spring.datasource.url=jdbc:mysql://mysql:3306/expenses_tracker?allowPublicKeyRetrieval=true&useSSL=false

Note: Setting allowPublicKeyRetrieval=true is helpful in development environments but can be a security risk in production. Use it with caution.

After making this change and rebuilding the containers, the Java app finally started working! 🎉
Here’s how the app looks running inside Docker containers:

📚 Resources That Helped Me Along the Way

I didn’t figure it all out on my own — these resources were super helpful during the process:

💡 Stack Overflow – For finding answers to weird bugs and errors.
🎥 YouTube Tutorial – To visually understand Docker concepts and how to structure files.
🔍 Google – My go-to for quick searches, Docker docs, and troubleshooting steps.

If you're starting out, I highly recommend using these — they make learning much easier!

🙌 Wrapping Up

This project taught me a lot about how Docker actually works when you're building and running real apps. From setting up services to fixing bugs that didn’t even show clear errors — it was frustrating at times, but also really fun to solve.
I made mistakes, looked things up, and slowly started to understand how all the pieces fit together. And honestly, that’s how real learning happens. If you're just starting out like me — keep going, you're doing great! 😄

❓ Over to You

Have you tried using Docker to run your own apps? Did you also face any weird issues that took hours to figure out?
I’d love to hear your story! Feel free to drop a comment — and let’s connect on X or LinkedIn if you’re also learning Docker, DevOps, or backend stuff.

Thanks for reading - see you in the next blog!

🔒 My EC2 Instance Froze — No SSH, No Web CLI. Here's What Caused It (and How I Fixed It)

Vishal — Tue, 24 Jun 2025 18:57:28 +0000

Today I want to share something unexpected I faced — a stuck EC2 instance that wouldn’t respond to SSH or even the AWS Web CLI. It took me a while to figure out what was wrong, but I learned a lot through the process. If you’re new to AWS or DevOps like me, this might save you some stress!

🧩 The Problem

Everything seemed fine at first. But suddenly:

I couldn’t SSH into my EC2 instance
The Web CLI also wouldn’t connect
The EC2 instance showed as “running” in the console
Docker containers were running inside from the last few days

At first, I thought it was just a temporary glitch. But even after stopping and starting the instance again, nothing worked. I was stuck.

💡 The Root Cause

After digging deeper (and asking ChatGPT!), I learned that:

My EC2 instance was a t2.micro, which uses CPU credits to manage performance.
I had left Docker containers running for several days.
This drained my CPU credit balance to 0.
When CPU credits are gone, AWS throttles the CPU to 10% — which makes SSH and Web CLI unusable.

This completely locked me out of my instance. I couldn’t even access it to shut down Docker.

🔧 The Solution That Worked

Here’s how I fixed it without terminating the instance:

1. Launch a Temporary EC2 Instance

I created a new t2.micro with the same OS as my original instance.

2. Detach the Volume

From the AWS Console, I detached the root volume from my stuck instance.

3. Attach It to the New Instance

I then attached the volume to the temporary instance as a secondary disk (e.g. /dev/xvdf).

4. Mount the Volume

I mounted the volume and created a rescue directory:

sudo mkdir /mnt/rescue
sudo mount /dev/xvdf1 /mnt/rescue

Then mounted other required filesystems:

sudo mount --bind /dev /mnt/rescue/dev
sudo mount --bind /sys /mnt/rescue/sys
sudo mount --bind /proc /mnt/rescue/proc

5. Enter chroot Environment

This allowed me to "enter" the original system:

sudo chroot /mnt/rescue

6. Disable Docker

Inside the chroot environment, I disabled Docker from auto-starting:

systemctl disable docker

Then I exited the chroot:

exit

7. Reattach the Volume

I detached the volume from the temporary instance and re-attached it to the original EC2 instance.

8. Reboot and Success 🎉

Now, when I started the instance, SSH worked again because Docker didn’t start, and the system wasn’t overloaded.

✅ What I Learned

Always monitor CPU credits if you're using t2.micro or burstable instances.
Don’t keep containers running forever unless you have a strong reason.
It's possible to rescue a stuck instance without deleting it.
AWS chroot rescue method is a powerful trick.

Thanks to ChatGPT, I saved hours and didn’t lose my data.

Resources

Over to You

Have you ever faced something like this?
How do you monitor or avoid issues like this with EC2 or Docker?

Let’s discuss below or connect with me:
🔗 x
🔗 LinkedIn

Thanks for reading — see you in the next blog! 👋

7 Days of Shell Scripting: My First Step into DevOps Automation

Vishal — Sat, 14 Jun 2025 06:34:44 +0000

🚀 Getting Started

On June 7, 2025, I took my first real step into the world of DevOps.I had just wrapped up a crash course in Linux, and I was curious: "What if I could automate the boring stuff I was doing manually?"
That question led me to Shell Scripting. Instead of watching endless tutorials, I gave myself a challenge: 1 week. 2 real projects. Let’s see what I can build.

This blog captures that journey — the good, the broken, and the lessons.

🛠️ Project 1: Deploying a Django App Using Shell Script

🗂️ GitHub Link: View on GitHub

🔍 What I Tried to Build

I wanted to automate the deployment process of a Django app using a single Shell script.

The goal was to:

Clone a GitHub repo
Set up Docker and Docker Compose
Build and run the app containers with a single command

💡 What I Learned

The importance of breaking down a task into smaller steps using functions — like cloning a repo, checking dependencies, building containers, and running them.
How to use Shell commands like cd, git clone, if conditions, and echo for logging inside scripts.
Learned how to add logic like.

   if [ -d "myapp" ]; then
    echo "Directory already exists. Skipping clone."
   else
    git clone <repo-url>
   fi

Most important thing i learned while deploying the django app is Error Handling.

❗ Issues I Faced

1. Script didn’t execute because I forgot to give it execute permission using chmod +x deploy.sh.

2. Directory already existed when I ran the script a second time, which caused git clone to fail. I had to learn how to add checks to prevent duplicate actions.

3. Docker daemon wasn't running, so the docker build command failed with errors. I didn’t realize Docker needs to be started manually or as a service.

4. Docker image naming mistakes I initially wrote docker build -t . myapp instead of the correct docker build -t myapp ..

5. No error handling at first my script would continue even if a step failed. For example, if git clone or docker build failed, the next steps still ran, which made debugging harder.I didn’t realize how important it is to stop the script when something breaks, or at least handle the failure properly.I also forgot to check if commands actually succeeded before moving on, like.

 git clone <repo-url>

no check here — even if it fails, the script moves forward.

✅ How I Solved Them

1. Missing Execute Permission

I realized that shell scripts need execute permission. I ran:
```
 chmod +x deploy.sh
```
This allowed the script to run directly with ./deploy.sh.

2. Directory Already Exists Error

I added a check to see if the folder was already present before cloning:

if [ -d "django-notes-app" ]; then
echo "Directory already exists. Skipping clone."
else
git clone <repo-url>
fi

3. Docker Daemon Not Running

I learned to start the Docker daemon manually using:

sudo systemctl start docker

4. Incorrect Docker Build Command

Initially, I mixed up the syntax. Corrected it to:

docker build -t notes-app .

5. No Error Handling

Started using this format to catch and report errors:

git clone <repo-url> || { echo "Git clone failed"; exit 1; }

🛠️ Project 2: Automating AWS EC2 Instance Creation Using Shell Script

🗂️ GitHub Link: View on GitHub

🔧 What I Tried to Build

In this project, I wanted to automate the process of launching an EC2 instance on AWS using a simple Bash script — no manual clicks on the AWS Console.

The goal was to:

Install AWS CLI (if not already installed)
Create an EC2 instance using required parameters like AMI ID, instance type, key pair, subnet ID, and security group
Wait until the instance is in a running state
Handle basic error checking and output helpful logs throughout the process

I treated this as a real-world mini DevOps task — automating infrastructure provisioning using shell scripting and AWS CLI. This helped me understand how cloud resources can be managed programmatically and how to handle dynamic situations in automation.

📘 What I Learned

Validate Before You Automate

Skipping checks for things like empty AMI IDs or missing subnets led to broken scripts. I learned to always validate user inputs before running any cloud commands.
IAM Permissions Can Make or Break Automation

Even if the script is correct, missing permissions can silently block EC2 creation. Understanding IAM policies helped me debug smarter and faster.
Clarity in Output Makes Debugging Easier

Adding clear and structured log messages ([INFO], [ERROR], etc.) helped me understand what the script was doing at each step — and where it was going wrong.

🐞 Issues I Faced

1. Running Script Without Setting Required AWS Parameters In the initial version of the script, I left variables like AMI_ID, SUBNET_ID, and SECURITY_GROUP_IDS as empty strings. This caused AWS CLI commands to fail with confusing or unhelpful error messages.

An error occurred (InvalidAMIID.Malformed) when calling the RunInstances operation: Invalid id: ""

2. Missing IAM Permissions for EC2 Actions Even though the script was syntactically correct, it failed to create an instance because the AWS user didn’t have the necessary EC2 permissions.

You are not authorized to perform this operation.

3. No Status Feedback During Execution The script initially lacked logs or progress messages, which made it hard to trace which part was executing or where exactly it failed.

4. Script Continued Execution After Failure When AWS CLI wasn’t installed or a command failed, the script still continued, which caused additional errors and wasted time.

✅ How I Solved Them

1. Empty or Incorrect Parameters
I added clear variable assignments in the main function of the script and make sure to never leave them empty.

AMI_ID="ami-0abcd1234example"        
SUBNET_ID="subnet-0123example"
SECURITY_GROUP_IDS="sg-0123example"

2. Missing IAM Permissions
I updated the IAM user by attaching the AmazonEC2FullAccess managed policy in the AWS Console.
Then I tested access with a simple command:

aws ec2 describe-instances

3. Lack of Logging or Debug Output

I introduced [INFO], [ERROR], and progress messages before and after major steps.

echo "[INFO] AWS CLI is installed."
echo "[INFO] Creating EC2 instance..."

4. Script Continued Even After Failure

I used proper condition checks and exit 1 to safely stop the script if something wasn’t right.

check_aws_cli(){
    if ! which aws &> /dev/null; then
        echo "[ERROR] AWS CLI is not installed."
        return 1
    else
        echo "[INFO] AWS CLI is installed."
    fi
}

main() {
    if ! check_aws_cli; then
        install_aws_cli
        if ! check_aws_cli; then
            echo "[ERROR] AWS CLI install failed. Exiting..."
            exit 1
        fi
    fi
}

📚 Resources That Helped Me Along the Way

I didn’t figure everything out on my own — here are some awesome resources that guided me through this project:

AWS EC2 Documentation

The official AWS documentation was my go-to source for understanding how EC2 instances work and the parameters needed to launch one.
AWS CLI User Guide

Helped me structure commands properly and understand how the CLI interacts with various AWS services.
YouTube: Shell Scripting in One Shot

Gave me a clear foundation on how to use conditionals, functions, and error handling in shell scripts.

Feel free to explore these if you're starting out or building something similar.

🙌 Wrapping Up

This was a fun and challenging week exploring how powerful Bash can be when combined with AWS CLI. I learned a lot by building, breaking, and fixing things — and I’m just getting started on this DevOps journey.

❓ Over to You

Have you ever tried automating cloud tasks with Bash or any other scripting language? What was the biggest challenge you faced, or what tools made it easier for you?
Let me know in the comments or connect with me on X/Twitter or LinkedIn — I’d love to hear your experience!

Thanks for reading — see you in the next blog! 👋