DEV Community: Arun SD

A Developer’s Journey to the Cloud 6: My Path to Kubernetes and IaC

Arun SD — Mon, 18 Aug 2025 11:56:43 +0000

From Herding Servers to Building Worlds with Code

I had done it. I had achieved high availability. My application was running on a fleet of two identical servers, managed by a smart load balancer. If one server went down, the other would seamlessly take over. My application was resilient. It was professional. I felt invincible.

That feeling lasted until it was time to deploy a new feature.

My beautiful, simple CI/CD pipeline was now obsolete. It was designed to update one server. How was I supposed to update a whole fleet? My first attempt was a clumsy bash script—a for loop that would SSH into each server, one by one, pull the latest code, and restart the container.

The first time I ran it, my heart was in my throat. I watched the logs scroll by, praying that server #1 would come back online before server #2 went down. It was a "rolling update" in the most literal, terrifying sense of the word. My fleet wasn't a clean, unified entity; it was a messy collection of individuals that I had to wrangle personally. I wasn't a developer anymore; I was the stressed-out admiral of a small, complicated armada, and I was spending all my time just keeping the ships sailing in the same direction.

Chapter 1: The Fleet Commander

My life was no longer about building features. It was about managing the fleet. My evenings were spent writing and debugging deployment scripts. My anxieties shifted. "What if the deployment script fails halfway through?" "How do I roll back an update across all servers at once?" "What happens when I need to scale from two servers to five? Or ten?" I was back to micromanaging machines, and it felt like a huge step backward.

This constant, low-grade fear of things getting out of sync led me down a late-night research rabbit hole of "container orchestration." My first stop was my cloud provider's own solution, Amazon ECS (Elastic Container Service). It seemed like the logical next step—simple, deeply integrated, and less complex than the other options. It felt like the "easy" path.

But then I hesitated. A familiar feeling crept in—the same feeling I had when I chose the "easy" path of running Redis in a Docker container. Was I about to tie my entire application's fate to a single cloud provider's proprietary system? What if I wanted to move to another cloud in the future? Or run a hybrid setup? All my knowledge of ECS would be useless. I would be locked in. I had learned my lesson: the easy path is often a trap.

This time, I decided to invest in the long term. I chose the other path, the one that was known for being more complex, but also more powerful and universal: Kubernetes.

Learning Kubernetes felt like learning a new language. The initial tutorials were a flood of new concepts: Pods, Services, Deployments, ReplicaSets. It wasn't a tool you could master in an afternoon. But as I pushed through, a fundamental, game-changing idea began to crystallize.

With my bash script, I was giving the servers a list of imperative commands: "Go here. Stop this. Pull that. Start this." I was the micromanager.

Kubernetes didn't want my instructions. It wanted my intent.

I stopped telling my servers what to do. Instead, I wrote a configuration file that declared the state I wanted, and Kubernetes worked tirelessly, like a powerful robot, to make that state a reality.

I no longer commanded; I declared.

Instead of a script that says "update server 1, then update server 2," I now wrote a Deployment manifest—a simple YAML file that acted as the sheet music for my application's orchestra.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-awesome-app
spec:
  replicas: 3 # <-- I declare I want 3 copies.
  template:
    spec:
      containers:
      - name: app-container
        image: myusername/my-awesome-app:v1.1 # <-- I declare which version to run.
        ports:
        - containerPort: 3001

To perform an update, I just changed the image tag in this file and applied it. Kubernetes handled the zero-downtime rolling update. If a server died, Kubernetes would just reschedule its containers elsewhere. The individual machines had become an invisible, abstract resource. I had finally stopped being an admiral and could go back to being an architect.

Chapter 2: The Fragile Ground Beneath My Feet

I had done it. My application was now managed by a powerful, automated fleet commander. I felt unstoppable. I decided to create a staging environment—a perfect replica of production for testing. So I went to my cloud provider's console to start building it all again.

And that's when a quiet, sinking feeling set in.

How did I create my production Kubernetes cluster in the first place? I had clicked through dozens of web UI forms. I had configured VPCs, subnets, security groups, and IAM roles manually. It had taken me a whole day. I had no record of what I did. I couldn't remember every setting. My entire production environment, the ground upon which my perfect Kubernetes setup stood, was a fragile, hand-made artifact. How could I ever hope to recreate it perfectly? What if I accidentally deleted something? The whole thing felt like a house of cards.

I had automated my application, but the infrastructure itself was still a manual, brittle mess. This led me to my next discovery: Infrastructure as Code (IaC). The idea is to do for your infrastructure what Docker did for your application environment: define it all in code. For this, I found a powerful duo: Terraform for provisioning the infrastructure, and Ansible for configuring it.

With Terraform, I could write files that described my entire cloud setup—the "what."

# main.tf

# Define the Virtual Private Cloud (VPC)
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
}

# Define a managed Kubernetes cluster within our VPC
resource "aws_eks_cluster" "production" {
  name     = "my-awesome-app-cluster"
  role_arn = aws_iam_role.eks_cluster_role.arn

  vpc_config {
    subnet_ids = [ ... ]
  }
}

# Provision a separate EC2 instance to be our secure "bastion" host
resource "aws_instance" "bastion" {
  ami           = "ami-0c55b159cbfafe1f0" # An Amazon Linux 2 AMI
  instance_type = "t2.micro"
  subnet_id     = # ...
}

Terraform was brilliant at creating the empty house, but how did I install the specific tools I needed on that bastion host? That's where Ansible came in. It handled the "how"—the configuration of the software on the machines Terraform built. I wrote an Ansible "playbook" to define the desired state of my bastion server.

# playbook.yml
---
- hosts: bastion_hosts # This group is defined in an inventory file
  become: yes
  tasks:
    - name: Ensure standard monitoring tools are installed
      apt:
        name:
          - htop
          - ncdu
        state: latest
        update_cache: yes

    - name: Create a specific user for developers
      user:
        name: dev_user
        state: present
        shell: /bin/bash

I spent a week converting my entire hand-clicked setup into these declarative files. When I was done, I could destroy and recreate my entire production network, Kubernetes cluster, and all its supporting services from scratch with a single command: terraform apply, followed by ansible-playbook playbook.yml.

My infrastructure was no longer a fragile artifact; it was now a set of version-controlled text files living in my Git repository. Creating an identical staging environment was now as simple as running the same commands with a different variable.

I had finally reached a new level of automation. Everything, from the virtual network cables up to the application replicas, was now code. The system felt truly robust. With a few keystrokes, I could scale my application containers, and with a few more, I could scale the very cluster they ran on. The system felt unstoppable. And as more users flocked to the app, I saw my cluster effortlessly adding more resources to meet the demand. But all this traffic, all these new users, were all being funneled to one place. The bottleneck had moved again. My application servers and infrastructure were an army, but they were all trying to get through a single door: my database.

Stay tuned for the next post: A Developer’s Journey to the Cloud 7: Advanced Database Scaling.

A Developer’s Journey to the Cloud 5: Load Balancers & Multiple Servers

Arun SD — Mon, 18 Aug 2025 11:42:10 +0000

My Server Was a Superhero, and That Was the Problem

I had finally done it. My application was a well-oiled machine.

The database and cache were offloaded to managed services, so they could scale on their own.

My deployments were a one-command, automated dream.

My single server was humming along, its memory usage was stable, and the app was faster than ever.

For the first time, I felt like I had a truly professional setup.

And then, one Tuesday morning, AWS sent a routine email:

"Scheduled maintenance for hardware upgrades in your server's host region. Expect a brief reboot..."

My blood ran cold.

A reboot. A brief reboot. My entire application, my whole online presence, was going to just… turn off.

Sure, maybe it would only be for five minutes, but in that instant the fragility of my architecture hit me.

Everything I had built—every feature, every user account, every bit of hard work—depended entirely on one single machine staying on.

My server wasn't just a server; it was a superhero, single-handedly holding up my entire digital world.

And even superheroes have to sleep.

The Vertical Scaling Trap

My first instinct?

"Maybe I just need a better server."

It’s an appealing idea—click a button, pay more money, and upgrade to a machine with more CPU cores and RAM.

That’s vertical scaling: making your one thing bigger and stronger.

But the maintenance email proved a brutal truth: even the biggest, most expensive server is still just one server.

It still has to be rebooted. Its hard drive can still fail. Its power supply can still die.

Scaling vertically is like buying an “unsinkable” ship—it feels safe, but you’re still betting everything on a single vessel.

It doesn’t fix the real flaw: the single point of failure.

I didn’t need a bigger boat.

I needed a fleet.

The Power of More, Not Bigger

The only way to survive the failure of one thing is to have more than one of it.

That’s horizontal scaling.

Instead of one big server, what if I had two smaller, identical ones?

If one went down for maintenance or failed unexpectedly, the other could keep running—and my users would never even know.

This was the path to true resilience.

But it raised a new question:

“If I have two servers, which one do my users connect to? And how is the traffic split?”

That led me to AWS’s dashboard, where I met my new best friend: the Application Load Balancer (ALB).

Building the Fleet

Surprisingly, the plan was straightforward.

Launch a Twin

I spun up a second, identical VM and deployed my same Dockerized application to it.

Now I had two servers, side-by-side, each capable of running the whole app.
Hire the Traffic Cop

The ALB doesn’t just point at individual server IPs.

First, I created a Target Group—a logical container for my servers.

I set up a health check that pinged /health every 30 seconds.

If it got a 200 OK, the server was marked healthy.

(Think of it like a backstage manager making sure every performer is ready before sending them on stage.)
Set Up the Listener

On the ALB itself, I configured a Listener for port 80.

Its rule was simple:

“When a request comes in, send it to a healthy server in my Target Group.”

The ALB would automatically distribute requests evenly.
Update the Address

The magic moment—updating my DNS.

Instead of pointing myapp.com to my server’s IP, I pointed it to the public DNS name of the ALB.

The First Test

I shut down one server manually, refreshed my site… and nothing happened.

It stayed online, smooth as ever.

Behind the scenes, the load balancer had noticed the outage during a health check and was silently routing all traffic to the surviving server.

When I restarted the downed server, the ALB welcomed it back into rotation without any downtime.

That was it—I had built high availability.

No single point of failure.

A system that could take a punch and keep running.

The New Problem

As I admired my two-server fleet, a thought crept in.

My CI/CD pipeline was perfect for one server.

But now? Two servers meant two deployments.

What if I needed five servers?

Or ten?

How would I update them all at once without breaking things?

I could already picture the nightmare:

half my servers running old code, the other half on a new version, users getting inconsistent results.

I had solved the single-point-of-failure problem…

and opened the door to the complexity-at-scale problem.

Next up:

A Developer’s Journey to the Cloud 6: Managing Complexity with Kubernetes

A Developer’s Journey to the Cloud 4: Caching with Redis

Arun SD — Mon, 18 Aug 2025 11:38:30 +0000

My App Was Getting Popular, and It Was Starting to Hurt

For the first time in this journey, I felt a sense of true peace. My deployments were fully automated. I could push a new feature, walk away to make a cup of tea, and return to find it live in production.

No manual checklists. No “Did I forget to restart that service?” anxiety. The high-wire act of deploying by hand was gone. For a developer, this was bliss. I could finally focus purely on the application itself.

And that’s when I started to notice things.

The main dashboard took just a hair longer to load.

A user emailed me to say their profile page “felt sticky.”

Nothing was crashing. Nothing was broken. But a new kind of unease began to creep in — the quiet, creeping dread of a system that is silently starting to buckle under its own weight.

The Investigation: A Different Kind of Broken

My first instinct was the usual: check the server’s health.

I SSH’d in, ran my standard CPU and memory checks… all green. No spikes. No memory leaks. So why did everything feel sluggish?

I dug deeper — one layer down — into my managed database’s monitoring dashboard. And that’s when the story changed. The CPU utilization graph looked like an EKG for a hummingbird: constant, jagged peaks. My database was working incredibly hard.

I enabled query logging, leaned back in my chair, and watched the flood of requests pour in.

And then I saw it: my application was asking my database the exact same questions over and over.

It was like sending the same intern to the library hundreds of times a minute to fetch the same book.

The database, bless its heart, dutifully sprinted to the shelves every single time — never pausing to wonder if maybe it could just keep a copy on its desk.

My app wasn’t broken. It was just… tired. And it was tiring out my database.

The "Easy" Fix — and the Mistake I Didn’t See Coming

The solution seemed obvious: give my app a short-term memory.

In other words — caching.

Redis was the obvious choice. It’s an in-memory, high-speed store designed for exactly this problem.

I already had my docker-compose.yml set up. What’s one more service?

It felt clean. Simple. No meetings required.

# docker-compose.yml (The "easy" but flawed approach)
version: '3.8'
services:
  app:
    # ... my app config ...
    depends_on:
      - db
      - cache # <-- Added this dependency

  db:
    # ... my managed db is external now, so this is gone ...

  cache: # <-- The new service
    image: redis:6-alpine
    ports:
      - "6379:6379"

I wired the caching logic into my code, redeployed, and the results were instant.

The app was flying. The once frantic database CPU graph became a calm, glass-smooth line.

For a few days, I walked taller. I’d done it again — problem solved.

The Crash That Felt Familiar

Then, one afternoon, the alerts started firing.

The site wasn’t just slow — it was timing out.

My stomach tightened as I SSH’d into the server and ran:

top
The truth stared back:  
MEM --- 98.7% used

My little server’s RAM was choking. Sometimes Redis was the culprit. Other times, my Node.js process. Either way, the system was suffocating.

And there it was — the wave of déjà vu.

Just months ago, I’d been losing sleep over my database. Now I was losing sleep over my cache.

I hadn’t really solved the problem. I’d just moved the stress around, like shifting a heavy box from one arm to the other.

The lesson was starting to crystalize:

The goal isn’t just to use the right tool — it’s to use it in a way that reduces your operational anxiety, not just relocates it.

The Real Fix: Outsourcing My Anxiety (Again)

Humbled, I shut down the Redis container.

Then I went shopping for the right kind of Redis.

My cloud provider had exactly what I needed: Amazon ElastiCache, a fully managed Redis service.

A few clicks later, I had a production-grade cache that:

Didn’t touch my app server’s RAM
Was scalable and secure
Was patched and monitored by people whose full-time job was making Redis run perfectly

The migration was almost embarrassingly simple. All I had to do was swap the connection string in my .env:

From this:

REDIS_URL=redis://cache:6379

To this:

REDIS_URL=redis://my-app-cache.x1y2z.ng.0001.aps1.cache.amazonaws.com:6379

I redeployed.

The app was still fast.

My server’s RAM sat at a comfortable 30%.

And for the first time in weeks, I wasn’t worried about my cache exploding at 2 a.m.

The Next Problem, Already Knocking

But as I watched my healthy server hum along, a new thought crept in.

I’d offloaded my database.

I’d offloaded my cache.

But my application code — the heart of the product — still lived on one single server.

What happens when even without extra baggage, my app needs more CPU or RAM than one box can give?

What happens when the process itself becomes the bottleneck?

Stay tuned for the next post:

A Developer’s Journey to the Cloud — Part 5: Load Balancers & Multiple Servers

A Developer’s Journey to the Cloud 3: Building a CI/CD Pipeline

Arun SD — Wed, 13 Aug 2025 19:03:45 +0000

My Deployments Were a Ritual, Not a Process

We've come so far. Our application is neatly containerized in Docker, and our data is safe and sound in managed cloud services. I had eliminated the "works on my machine" curse and outsourced my 3 AM data-loss fears. I should have been happy.

But a new kind of dread was creeping in—a dread that arrived every time I had to ship a new feature: the deployment ritual.

The Manual Dance

It was a clunky, manual dance that I had perfected:

git push my changes.
Open a terminal and SSH into my server.
Navigate to the project folder.
Run docker-compose down, git pull, and finally docker-compose up --build.

Every. Single. Time.

It worked, but it felt wrong. It was slow. It was nerve-wracking—what if I accidentally typed the wrong command on my production server?

Most importantly, it was a bottleneck. I was the only one who could do it. I had become the deployment guy, a title I never wanted.

The Day I Locked Myself Out

The breaking point came in a coffee shop on shaky Wi-Fi. I SSH'd into my server to deploy a critical hotfix, but my connection dropped midway through.

The application stopped.

The server never got the command to bring it back up.

The site was down.

It took me ten frantic minutes to get a stable connection and fix it, but the damage was done.

I realized my manual process wasn’t just inefficient—it was fragile. It relied on me, my laptop, and a stable internet connection.

What If Deployment Just… Happened?

That night, I asked myself:

What if deploying wasn't a ritual I had to perform? What if it was just… something that happened automatically when the code was ready?

That question led me to CI/CD (Continuous Integration / Continuous Deployment)—an assembly line for code.

Tools like GitHub Actions or GitLab CI act as robots on this assembly line: you give them a recipe, and they execute it perfectly, every time.

Giving the Robot My Ritual

I built a recipe so that GitHub itself would:

Build and push my Docker image.
Tell my server to pull and run the new version.

After a day of tinkering, I had my workflow:

# .github/workflows/deploy.yml
name: Deploy to Production

on:
  push:
    branches: [ main ] # Run on every push to main branch

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and Push Docker Image
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: myusername/my-awesome-app:${{ github.sha }}

      - name: Deploy to Server
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.SERVER_HOST }}
          username: ${{ secrets.SERVER_USER }}
          key: ${{ secrets.SSH_PRIVATE_KEY }}
          script: |
            cd /home/user/app
            export IMAGE_TAG=${{ github.sha }}
            docker-compose up -d --no-build

The first time the pipeline ran successfully, it felt like magic.

I pushed my code, and a few minutes later—changes were live.

I hadn’t even opened my terminal. My server was no longer a sacred place I had to log into. It was just a machine that ran containers. The “keys” now lived securely in GitHub’s secrets, not in my pocket.

From Ritual to Repeatable Code

My deployment process was no longer trapped in my head—it was now code in my repository. Version-controlled. Repeatable. Secure.

Deployments went from being a 10-minute, high-anxiety event to a complete non-event. They just… happened.

A New Problem Emerges

For the first time, I felt real peace. I could focus entirely on the application itself.

But then I noticed:

A slight lag when the dashboard loaded.
A user emailing to say their profile page "felt sticky".

Nothing was crashing. Nothing was broken. But the system was straining.

The irony? My new, efficient deployments made it easier for more users to sign up—creating the very load that was slowing things down.

Diving into the logs, I saw the truth:

My database was getting hammered with the same queries over and over.

The app wasn’t broken. It was tired.

It was time to give it a break.

Stay tuned for the next post: A Developer’s Journey to the Cloud 4: Caching with Redis

A Developer’s Journey to the Cloud 2: My Database Lived in a Shoebox, and I Didn’t Even Know It

Arun SD — Wed, 13 Aug 2025 19:03:25 +0000

My Database Lived in a Shoebox, and I Didn’t Even Know It

We did it. In the last post, we took our application, boxed it up with Docker, and shipped it to a server. It was running, stable, and consistent. The "works on my machine" curse was broken. I felt like I had conquered the cloud.

For about a week, I was a DevOps king, basking in the glory of my perfectly containerized world.

Then, one evening, as I was about to shut my laptop, a cold thought washed over me:

Where does my data actually live?

The Shoebox Realization

It hit me like a bad database query: my entire database — every user, every post, every precious row of information — was running inside that same Docker container, on that same single server.

And it wasn’t just the database. My user-uploaded images? Just sitting in a /uploads folder on that same hard drive, quietly piling up like old photos in a forgotten attic.

The whole thing was one fragile digital shoebox. If the lid blew off (or the drive failed), it would all scatter into the void.

The 3 AM Fear

That night I lay in bed thinking about rm -rf / nightmares and spinning disks giving their last click of life.

What if the server’s hard drive failed? It’s just a machine, after all. Everything would be gone. Instantly.

What about backups? Sure, I could write a script, maybe a cron job:

pg_dump mydb > backup.sql

But… where would that backup go?
Another folder? On the same server?
That’s like hiding your spare house key under the doormat of a house that’s on fire.

The more I thought about it, the more absurd it became.

Googling Myself Into DBA Territory

I started Googling “how to back up a database properly” and promptly fell into a black hole: replication strategies, point-in-time recovery, WAL archiving, security patching.

I wasn’t just a developer anymore — I was now an unwilling, unqualified, and mildly terrified part-time Database Administrator and Storage Manager.

This wasn’t the dream.
The dream was building my app, not babysitting a database and a pile of user images like some digital hoarder.

The Cloud’s Best-Kept Secret

Defeated, I wandered through my cloud provider’s dashboard, clicking through services with names I didn’t fully understand.

And then I saw them — two shiny lifeboats in a sea of uncertainty:

Relational Database Service (RDS): “A managed relational database service... handles provisioning, patching, backup, recovery, failure detection, and repair.”
Simple Storage Service (S3): “Object storage designed to store and retrieve any amount of data... with 99.999999999% durability.”

It was almost comical. Of course the cloud companies were good at this. This is their entire business!

Here I was, ready to script a janky nightly backup, while they had teams of engineers whose only job was to make sure data never disappears.

Handing Over the Keys

The next day, I stopped being stubborn and started migrating.

Database Migration

With a few clicks, I spun up an RDS instance. Automatic backups? done
High availability? done
Security patches? done

I just updated my app’s connection string:

DATABASE_URL=postgres://user:password@database-1.abcdefghij12.us-east-1.rds.amazonaws.com:5432/mydb

File Storage Migration

Instead of saving files locally, I integrated the S3 SDK and changed my upload logic:

s3.upload({
  Bucket: "my-app-bucket",
  Key: `uploads/${file.name}`,
  Body: file.data
});

Suddenly, my images weren’t trapped in /uploads; they were in a globally redundant, highly durable vault.

A Stronger Foundation

From the outside, my app looked exactly the same.
But beneath the surface, the foundation had gone from a shoebox on a wobbly shelf to a bank vault inside a fortress.

I was no longer the single point of failure. I could finally focus on writing code without the looming fear of catastrophic data loss.

But One Problem Remained…

Even with the data safe, I still had to deploy my code the old-fashioned way: SSH into the server, run some commands, cross my fingers, and hope nothing broke.

It felt clunky. Slow. Archaic. There had to be a better way.

Next up: A Developer’s Journey to the Cloud 3: Building a CI/CD Pipeline.

A Developer’s Journey to the Cloud 1: From Localhost to Dockerized Deployment

Arun SD — Wed, 13 Aug 2025 19:02:32 +0000

About This Series

Over the past 8 years, I’ve built and deployed a variety of applications—each with its own unique set of challenges, lessons, and occasionally, hard-earned scars. Instead of presenting those experiences as isolated technical write-ups, I’ve woven them into a single, continuous narrative: A Developer’s Journey to the Cloud.

While the “developer” in this story is fictional, the struggles, breakthroughs, and aha-moments are all drawn from real projects I’ve worked on—spanning multiple tech stacks, deployment models, and problem domains. Each post captures the why and what behind key decisions and technologies, without drowning in step-by-step tutorials.

Think of it as a mix between a memoir and a guide—part storytelling, part practical insight—chronicling the messy, funny, and sometimes painful path of learning to build in the cloud.

That Time I Thought localhost Was Lying to Me

It all starts with that beautiful, electric feeling. The final line of code clicks into place, and your application—your glorious, bug-free masterpiece—purrs to life on localhost. You've tested every button, every form, every feature. It's perfect. All that's left is to share it with the world.

"How hard can that be?" I thought, basking in the glow of my monitor. And so began my glorious, agonizing, and unintentionally hilarious journey into the wild.

The First Launch

My plan was simple: rent the cheapest virtual server I could find. For a few hundred rupees, I was the proud owner of a blank command line with a public IP address. It felt like being handed the keys to the internet.

With the confidence of someone who had successfully used npm install more than once, I manually installed Node.js and PostgreSQL on the server. Then came the big moment: getting my code onto the server. The process was painfully manual.

First, I prepared my project files locally:


# On my laptop
zip -r my-awesome-app.zip .

Then, I used scp (Secure Copy) to upload the zipped file to the server:


# On my laptop
scp my-awesome-app.zip user@your_server_ip:/home/user/

Finally, I logged into the server to unpack and run everything:


# On the server
ssh user@your_server_ip
unzip my-awesome-app.zip -d app
cd app
npm install
npm start

I typed the IP address into my browser. It worked. My app was alive. I was, for all intents and purposes, a genius.

The first crack appeared a day later. I found a tiny typo. "Easy fix," I thought. I corrected the text, zipped everything up again, and repeated the entire upload-unzip-restart dance. I restarted the app.

And the entire thing crashed.

Somehow, in that simple process, something had gone terribly wrong. I had no history, no undo button. It took me an hour of frantic re-uploading to get it back online. I decided the "F" in FTP stood for "fragile."

Getting Smarter, a Little

Okay, no more zip files. I was a professional, and professionals use Git. I SSH'd into my server and set up a "bare" repository, a special repo just for receiving pushes.


# On the server
git init --bare /var/repos/app.git

I configured a hook that would automatically check out the code into my live directory whenever I pushed to it. My deployment process was now a sleek and sophisticated git push production main. I had leveled up.

This new system worked beautifully for weeks. I built a major new feature, a complex image upload and processing tool. To get access to some new performance improvements, I developed it locally using the latest version of Node.js. As always, it ran like a dream on my laptop. I pushed the code, and the deployment hook ran. I restarted the app with a confident smirk.

It crashed. Instantly.

The error message was a nightmare. A function I was using simply didn't exist. But... I had just used it. It was right there in my code. I spent the next six hours in a state of pure disbelief.

Then, it hit me. My server, being a server, had been prudently set up with the stable, Long-Term Support (LTS) version of Node.js. My feature, built with the shiny new tools of the "latest" version, was trying to call a function that hadn't been introduced in the "stable" release yet. It was then I understood that the most soul-crushing phrase in our industry, "but it works on my machine," isn't a joke. It's a curse. My code wasn't the problem; the entire universe it was running in was fundamentally, fatally different.

Shipping the Entire Universe

Defeated, I started searching for answers, and I kept stumbling upon the same word: Docker. The promise was simple: what if, instead of just shipping your code, you could ship your code's entire world along with it?

The idea was to define your exact environment in a text file—a Dockerfile. This file acts as a blueprint to create a "container," a lightweight, standardized box holding your app and its perfect environment.

I spent a weekend tinkering. My Dockerfile looked something like this:


# Dockerfile
# Use the 'latest' Node.js version we need
FROM node:latest

# Set the working directory in the container
WORKDIR /app

# Copy package files and install dependencies
COPY package*.json ./
RUN npm install --only=production

# Copy the rest of our application code
COPY . .

# Expose the port and start the server
EXPOSE 3001
CMD [ "node", "index.js" ]

To run my database alongside my app and manage secrets properly, I created a docker-compose.yml file and a separate .env file for my credentials.

This is the .env file, which should never be committed to Git:

Code snippet

# .env
# Database credentials
POSTGRES_USER=myuser
POSTGRES_PASSWORD=mysecretpassword
POSTGRES_DB=myapp_db
And this is the `docker-compose.yml` that uses it:


# docker-compose.yml
version: '3.8'
services:
  app:
    build: .
    ports:
      - "3001:3001"
    depends_on:
      - db
    # Load environment variables from the .env file
    env_file:
      - .env
    # Pass necessary variables to the app container
    environment:
      PGHOST: db
      PGUSER: ${POSTGRES_USER}
      PGPASSWORD: ${POSTGRES_PASSWORD}
      PGDATABASE: ${POSTGRES_DB}
      PGPORT: 5432

  db:
    image: postgres:13
    # Use the same .env file to configure the database
    env_file:
      - .env

I ran docker-compose up on my laptop. It worked. But this was the real test. I installed Docker on my server, copied my project over (including the .env file), and ran the exact same command.

And it just... worked. No errors. No version conflicts. No drama.

It wasn't magic. It was simply that for the first time, the environment on my server was not just similar to my laptop's; it was identical. I hadn't just deployed my code. I had shipped its entire universe in a box, and it didn't care where it was opened.

That's when it clicked. The goal was never just to get the code onto a server. It was to get a predictable, repeatable result, every single time. And my journey, I realized, was just getting started.

Our code is now safe, but our data is living dangerously. Let's fix that.

Stay tuned for the next post: A Developer’s Journey to the Cloud 2: Managed Databases & Cloud Storage.

AI vs(and?) Software Engineers

Arun SD — Thu, 19 Dec 2024 12:36:39 +0000

Artificial intelligence is revolutionizing industries across the board, and software engineering is no exception. From automating repetitive tasks to assisting in the design of user interfaces, AI tools are becoming integral to developers’ daily workflows. As these tools evolve, they prompt critical questions about the future of the profession and the skills needed to stay ahead in an increasingly AI-driven world.

The AI Advantage: A Powerful Ally

Let’s start by recognizing the undeniable advantages AI brings to software development. Tasks that were once time-consuming—like writing boilerplate code, generating basic UI elements, creating database schemas, or drafting initial logic—are now much quicker and easier with AI assistance. By handling these repetitive tasks, AI enables developers to dedicate more time to the complex, creative aspects of their work.

Personally, I used AI for in past few months:

Boilerplate Reduction: Generating repetitive code structures, saving both time and mental energy.
Rapid UI Prototyping: Quickly creating basic UI layouts for faster iteration and experimentation.
Database Schema Generation: Automating database table creation, ensuring consistency and reducing potential errors.
Logic Drafting: Offering initial code drafts, which I can then refine and optimize.
Code Refactoring: Automatically suggesting improvements in existing code to enhance readability and performance.
Unit Test Generation: Creating basic test cases to validate functionality, reducing the effort of manual test creation.
Documentation Assistance: Auto-generating docstrings or comments based on code functionality, which helps in maintaining better documentation.
This increased efficiency accelerates development cycles, reduces costs, and ultimately leads to more innovative software solutions.

The Trust Factor: A Double-Edged Sword

However, with AI’s ability to generate code comes a crucial question: How much should we trust the code it produces? While AI can often provide functional and well-structured code, it’s not perfect. Bugs, inefficiencies, and security vulnerabilities can easily slip through the cracks. This raises an important issue, do we, as software engineers, have the expertise to thoroughly review, refine, and adapt AI-generated code for our specific needs?

AI’s speed and convenience are incredibly helpful, but it’s essential that we remain vigilant. Blindly trusting AI-generated code without proper scrutiny risks building software on an unstable foundation. We must retain the ability to understand the underlying logic, identify potential flaws, and adapt the code accordingly.

The "Black Box" (like credit scores :D) Problem

One big problem is that we don't always know how AI makes decisions. It's like a "black box"—we see the output (the code), but not the process. This makes it harder to find and fix problems in the code. We need better ways to understand how AI writes code.
Created the image below with plantUML(great tool to creat UML diagrams try it).

The New Engineer Dilemma: Learning vs. Reliance

One of the most pressing concerns about AI’s role in development is its potential impact on new engineers entering the field. If a recent graduate starts relying heavily on AI tools right away, will they truly learn the fundamental principles of software engineering? Will they develop the critical thinking and problem-solving skills necessary to debug complex issues and design effective solutions?

The concern here is that over-reliance on AI could result in:

Reduced "Learning by Doing": Hands-on experience is key to mastering software development. If AI does much of the work, new engineers might miss out on this essential learning process.
Diminished Debugging Skills: Debugging is a core competency for software engineers. If AI takes over the debugging process, new engineers may not develop the diagnostic skills needed to tackle complex issues.
Stunted Problem-Solving Abilities: Software engineering is about solving problems. If AI supplies ready-made solutions, engineers may not hone their critical thinking skills and might struggle to tackle challenges independently.
The Path Forward: Collaboration, Not Replacement

It’s crucial to emphasize that AI is not here to replace software engineers. Rather, it’s a powerful tool that can enhance our capabilities. The goal is to find a balance—leveraging AI’s strengths while maintaining our expertise and creativity.

Here are a few key takeaways:

Embrace AI as a Tool: Use AI to automate repetitive tasks, boost productivity, and free up time for more creative and high-value work.
Maintain Critical Thinking: Always review and understand the code AI generates. Don’t rely on it blindly.
Focus on Fundamentals: Ensure a solid grasp of core software engineering principles—these will always be the foundation of great development, regardless of technological advances.
Prioritize Continuous Learning: Stay current with both AI developments and the latest software engineering techniques. The two will evolve together, and staying informed ensures we remain competitive.
Foster Collaboration: Rather than viewing AI as a threat, embrace it as a partner that can augment human creativity and problem-solving. Together, we can innovate faster and more effectively.
Stay Cautious with Dependencies: AI can automate many tasks, but it’s essential to understand when and where it’s appropriate to rely on it, ensuring we don’t become overly dependent on AI without understanding its limitations.
The future of software engineering lies in collaboration—human expertise combined with AI-powered efficiency. By embracing this partnership thoughtfully, we can unlock new possibilities, drive innovation, and build software that is not only more efficient but also more impactful.

*PS: AI tools helped a lot in writing this post like sounding more professional, using passive tone and generating the header image😉 *