DEV Community: Vivian Chiamaka Okose

I Locked Myself Out of My Own Cloud Server. Here Is How I Got Back In.

Vivian Chiamaka Okose — Wed, 29 Jul 2026 14:35:00 +0000

I did something this week that sounds like a mistake and was actually the most useful thing I have learned in months.

I locked myself out of my own server. On purpose. A server that was running fine, serving a live website, with no way back in. No SSH, no password, and no keyboard to plug in, because it lives in an Amazon data centre I will never enter.

Then I recovered it. And now I understand something most engineers only read about.

Logged back into the server that had locked me out.

Let me walk through the whole thing, because the recovery only makes sense once you see how the server was built.

Renting a computer by the hour

The cloud is someone else's computer that you rent by the hour. You get a slice of a machine in a warehouse, use it, and hand it back when you are done. Amazon calls this service EC2, and one rented machine is an "instance."

The mental shift that matters: you pay for the machine existing, not for using it. A server you forgot about costs the same as one serving thousands of people. That single fact shapes every decision in this module, and it is why cost discipline shows up at the end.

Locking the front door before building anything

The very first step had nothing to do with servers. It was locking down the account itself.

When you sign up for AWS, you get a "root" account that can do anything, including run up unlimited charges if the credentials leak. There is a whole economy of bots scanning the internet for leaked AWS keys, and they spin up expensive servers within minutes of finding one.

So the first move is to lock root away behind multi-factor authentication and create a weaker, everyday identity to actually work with. That everyday identity is an IAM user. Root is the master key you lock in a drawer; the IAM user is the normal key you carry.

Root account secured with multi-factor authentication.

An everyday IAM user, in an Administrators group, so I never work as root.

The server that would not let me in

With the account secured, I launched my first instance. And immediately hit a wall.

SSH timed out. Not "permission denied," which would mean I reached the server but was not allowed in. A timeout, which means the knock got no answer at all.

My security group, AWS's firewall, was correct: SSH open to my IP, web traffic open to everyone. So why the silence?

I worked through it one layer at a time. Was my key wrong? A wrong key gets rejected, not ignored. Had my home IP changed? It matched. Was the subnet missing its route to the internet? The gateway was there.

Then I found it. A network ACL, a second firewall sitting above the security group, set to deny all traffic. Someone had configured this account's network in a non-standard way long before I arrived. My rules were perfect; the layer above them blocked everything.

I made a call that felt like the professional one: stop patching someone else's confusing setup, and build my own clean network from scratch. A fresh VPC, one public subnet, a proper gateway, all wired correctly. I launched into that, and SSH worked instantly.

The instance running in my own clean network, finally reachable.

The lesson stuck harder than any happy-path tutorial could have taught it. A timeout is not one problem, it is a checklist: security group, your IP, the route, the network ACL. And when you inherit a mess, rebuilding clean often beats untangling.

Hardening, against real threats this time

I had hardened Linux servers before, but always on my own laptop where the threats were hypothetical. This one had a public address, which means bots start probing it within minutes of it going live.

Same baseline, real stakes: patch everything first, put up a firewall that denies everything by default and opens only SSH and the web ports, and run fail2ban to jail anything that hammers the login repeatedly.

The firewall: everything denied by default, only SSH and the web ports open.

fail2ban standing guard, ready to jail repeat login offenders.

Giving it a name and a padlock

A raw IP address is no way to reach a website. So I pointed a subdomain of my own domain, app.viviancloud.site, at the server, while carefully leaving my main domain on its existing site. One domain, different subdomains for different things, which is exactly how real companies organise this.

Then HTTPS. For years, certificates cost money and were fiddly. A nonprofit called Let's Encrypt made them free and automatic. One tool, one command, and my site loaded with a padlock and a certificate that renews itself. The plain-HTTP postcard became a sealed envelope.

My own page, served from the EC2 instance.

Reachable by name at app.viviancloud.site.

The padlock: HTTPS live, certificate valid and auto-renewing.

Data on its own disk

I attached a second disk, separate from the one the operating system lives on. The reason is practical: keep your data apart from your OS, and you can rebuild the OS whenever you like without touching the data. Format it, mount it, and add it to the system's startup config so the mount survives a reboot. I proved it by actually rebooting and watching the disk come back on its own.

The separate data disk, still mounted after a full reboot.

The lockout, and the way back

Now the part I opened with.

To learn recovery for real, I created the disaster. I emptied the file that lists which keys are allowed to log in, disconnected, and confirmed I was locked out. Permission denied, on my own running server.

But before breaking anything, I took a snapshot: a full backup of the disk, my undo button.

A snapshot first. Never break something risky without a backup.

Here is the recovery, and the single idea that makes it possible: the file that decides who can log in lives on the disk, and a disk is a movable object.

So I stopped the server, detached its disk, and attached that disk to a second "rescue" instance as a spare drive. From the rescue instance, I could reach into the locked server's files, and I wrote my key back onto its login list, with the exact ownership and permissions SSH insists on. Then I detached the disk, reattached it to the original server as its boot drive, and started it up.

I logged in. The same key that got "permission denied" ten minutes earlier now worked.

It is exactly like being locked out of your house, unbolting the front door, carrying it to a locksmith who fits a new lock you have a key for, and rehanging it. The house never changed. You fixed the lock by taking the door somewhere you could work on it.

The deeper realisation: a server accepts any key whose public half is written in its login file. So a lost key is never a dead end. You do not recover the old key, you generate a new one and install it. The lock is always replaceable.

There are faster recovery routes when you set them up in advance, but the disk-swap always works, with no agent and no preparation. It is the universal fallback, which is why it is the one worth learning first.

Building carefully, tearing down completely

The module ended where cloud work always should: accounting for the cost and knowing how to remove everything.

Total spend for all of this: zero dollars, by staying on free-tier resources and stopping the instance between sessions. I set a billing alarm as a tripwire, and wrote a teardown runbook listing every resource and how to remove it in order, because a resource you forget is a resource you keep paying for.

A billing alarm: emails me if estimated charges ever cross five dollars.

What I actually learned

Not just the steps. The principles underneath them.

That cloud bills on existence, not use. That you default everything to closed and open only what each service needs. That a lost key is a lock to replace, not a catastrophe. And that knowing how to cleanly destroy infrastructure matters as much as knowing how to build it.

I locked myself out of a server and got back in by hand. I will not forget how that works.

Your Build Output Is Not an Artifact Yet. Here Are the Four Things It Needs.

Vivian Chiamaka Okose — Thu, 23 Jul 2026 12:24:56 +0000

In December 2021, a flaw was found in a Java logging library called Log4j. Not a famous library. The boring thing that writes messages to a log file.

Then every company on earth asked the same question at once: are we using it?

Most couldn't answer. Not out of carelessness, but because nobody keeps a list of what is actually inside their software. Their app pulled in a library, which pulled in another, which quietly pulled in Log4j four levels down. Teams spent entire weekends grepping through servers trying to work out whether they were exposed.

The companies that answered in ten minutes had one thing the others didn't. A list.

That list is called an SBOM, and it's one of four things I learned to attach to my own artifacts this week.

First, what a build actually does

Source code is text. UserService.java is a file full of words like public class and return. A processor cannot run that. It speaks something much lower level.

Building is what closes that gap. It's the cooking, and source code is the recipe.

Nobody serves a recipe card to a customer. They serve the dish. Nobody deploys source code to a production server either; they deploy the built artifact.

I built two of them. A Java service with Maven, which produced a 4.8K JAR file. A Node service with npm, which produced a 1.7K .tgz. Different languages, same idea: source in, shippable thing out.

Every build tool is the same tool wearing different clothes

Java uses Maven. Node uses npm. Python uses pip. Three tools, three sets of commands, and it looks like three separate things to learn.

It isn't. They all answer the same five questions:

A config file. What is this project and what does it need? (pom.xml, package.json, requirements.txt)
A lock file. Which exact versions did we actually install? (package-lock.json, pinned versions)
A registry. Where do the libraries come from? (Maven Central, npmjs.com, PyPI)
A cache. Where do downloaded libraries live locally? (~/.m2, node_modules/)
An artifact. What comes out at the end? (.jar, .tgz, .whl)

Config, lock, registry, cache, artifact. Once you see those five boxes, learning the third tool takes minutes, because it's just the first tool with different filenames.

One genuine difference worth knowing. Java compiles everything upfront, translating the whole book before anyone reads it. Node and Python translate live, line by line, as the program runs. That's why my first Maven build took 64 seconds and npm install took 4.

Four questions that turn a build output into an artifact

Here's the part most tutorials skip. A file sitting in a target folder isn't ready to ship. A real artifact answers four questions.

What's in it?

An SBOM, a Software Bill of Materials, is a complete itemised list of every component inside your artifact. It's the ingredients label on food packaging.

I generated mine with a tool called Syft. My Node service listed two entries: my own code, and one library called dotenv.

Small, but the version was interesting. My config file asked for ^16.4.5, meaning "16.4.5 or anything compatible." What actually got installed was 16.6.1. Fuzzy request, specific reality. That gap is precisely why lock files exist.

Real applications run this and get three hundred entries. Most of which nobody on the team chose.

Is it safe?

Knowing what's inside is step one. Step two is checking whether any of it is known to be broken. Security researchers maintain public databases of flaws found in software libraries, and a tool called Grype cross-checks your ingredients list against them.

It's a food safety inspection. You have the ingredients; now someone checks them against the recall list.

Both my artifacts came back clean, which was true and slightly boring. So I deliberately installed an old version of a popular library called lodash to see what a real finding looks like.

Six findings from one library. Three rated High. Read one row as a sentence: you have lodash 4.17.15, it has a high severity flaw, version 4.17.21 fixes it.

Two columns earn their place. EPSS estimates how likely the flaw is to be exploited in the wild, and the top result sat at 22.4%, the 97th percentile. That's not theoretical; attackers are actively using it. RISK blends severity with that likelihood so you know what to fix on Monday morning.

And every single one had a fix already available. Most vulnerabilities are not unsolvable mysteries. They're someone forgetting to update a library for two years.

Is it the one I built?

In 2016, attackers broke into the servers of Linux Mint and swapped the download file for their own version with a backdoor in it. Same filename, same page, same site. Anyone downloading that day got malware.

Underneath the download link sat a line of gibberish characters. Anyone who ran one command against their file and compared it would have known instantly.

That line is a checksum: a fingerprint calculated from a file's exact contents. It's the tamper-evident seal on a medicine bottle.

I made one for my JAR, verified it passed, then appended a single character to the file and checked again.

One character. FAILED, immediately. There is no such thing as small tampering; any change at all breaks the seal, which is exactly what makes it worth publishing.

Who made it?

A checksum proves the file didn't change. It doesn't prove who made it. An attacker who controls the download page can replace both the file and the checksum, and everything matches perfectly. It's just their seal now.

Signing closes that gap. A signed release carries cryptographic proof it came from a specific person, and it cannot be forged without their private key.

I signed my release tag with the same SSH key I set up months ago for signed commits, and got back: Good "git" signature for vivian-signing-2026. On GitHub it shows a green Verified badge.

A checksum is a seal anyone can apply. A signature is your handwriting, notarised.

Then I stopped relying on my memory

Producing all of that took eight separate commands in the right order with the right flags. Which is fine on a Tuesday morning and useless at 5pm on a Friday when you need to ship a fix and the scan feels optional.

So I wrapped the whole path in one script. Build, SBOM, scan, checksum, in that order, stopping dead if anything fails.

![The full build and verify run]

One command, four steps, three files at the end: the artifact, its ingredients list, and its seal. You can't skip a step because there are no steps to skip.

That's also the bridge to what comes next. A CI pipeline is essentially this script running on a server, triggered by a push. Write it now, hand it to a robot later.

Three things that caught me

A command finishing without error is not proof it did the right thing. My Node tests ran green while testing nothing at all, because I'd pointed the test runner at the wrong path. Five tests written, one thing ran. I only caught it because the count said 1 and I expected 5.

Node's package doesn't contain its dependencies. Scanning the .tgz found zero packages, which looked broken. It wasn't. npm deliberately excludes node_modules, so the box holds the recipe and the ingredients get fetched on arrival. Java bakes them in; Node doesn't.

Build output does not belong in git. My first commit attempt swept in compiled classes, test reports, and the JAR itself. All of it reproducible from source with one command. Commit the recipe, not the meal.

The question I can answer now

If someone hands me a JAR with no SBOM, what should I assume about it?

That nobody knows what's inside it. Including the person who built it.

It might be perfectly fine. But there's no way to answer the Log4j question about it, no way to know whether it's carrying something with a public exploit, and no way to prove it's the file they think it is.

That's not ready to ship. It's just bytes.

I Deleted My Database on Purpose. It Was Back in 47 Seconds.

Vivian Chiamaka Okose — Fri, 17 Jul 2026 06:11:01 +0000

There's a sentence I can't stop thinking about:

A backup you have never restored is a wish, not a backup.

Most engineers set up backups and never test them. The backup job runs every night, everyone feels safe, and then one day a real disaster hits, they reach for the backup, and discover it's been silently broken for months. Now the data is genuinely gone, at the worst possible moment.

So this week I stopped wishing. I destroyed my own database on purpose and proved I could bring it back.

First, I learned before I built

I did something different this module: I asked to understand the concepts before touching a single command. What a database actually is. Why SQL and NoSQL both exist. What containers really are. What migrations do.

It changed everything. Instead of copying steps and hoping, I knew why each command mattered. If you're learning DevOps and you find yourself typing commands you don't understand, stop and go learn the idea underneath. The labs land completely differently after that.

The stack, from one command

I built a four-container stack that starts with a single command: Postgres (the strict, organized store for important data), Redis (the fast, flexible one), Adminer (a web view of the data), and pgBouncer (connection pooling).

The idea that made containers click for me: a container is a sealed lunchbox holding software plus everything it needs to run. It works identically on my laptop, your laptop, or a server in a data center. No more "but it works on my machine."

And one crucial piece: a volume. Containers are disposable, you throw them away and make new ones. A volume is the storage box that survives that. Without one, your data dies with the container. That distinction matters more than anything else in this setup.

Version control, but for your database's shape

Your database's structure changes constantly as an app grows. Add a column, add a table, add an index. The tempting way is to just log in and type the change. The problem: nobody has a record of what changed or when, and now your database, your teammate's, and production are all subtly different.

Migrations fix that. They're numbered, recorded files, V1, V2, V3, applied in order, with the tool keeping a history of exactly what ran and when.

They are, quite literally, git for your database structure. Which is why the concept felt instantly familiar after last month's Git module.

One detail worth stealing: I added a column in V2, and backfilled its values in V3, deliberately two separate migrations, not one. Because if an app is live while you change the database, adding and filling in one step means a moment where the app and database disagree. Additive first, destructive later. That staging is how you change a database without breaking the thing using it.

The refusal that impressed me most

Here's my favourite moment. In the real world, you almost never get to start a database from scratch. You inherit one that's been running for years, full of customer data you cannot wipe. So I simulated that: an existing database, real rows, no migration history.

I pointed the migration tool at it, and it refused:

Found non-empty schema but no schema history table.

It was saying: there's data here I didn't create, I don't know what version this is, and I won't guess and risk breaking it. That refusal is a feature. The tool protecting two years of data from a stranger's assumptions.

The fix is called baselining: you tell it "whatever exists now, call that version 1, count my changes from version 2." Like joining a company mid-project and saying "I'm not recreating what happened before I arrived. Today is my starting line."

That's the version of this skill you'll actually use at work.

Then I destroyed everything

The main event. Here's the drill:

Count the rows: 3 authors, 3 posts
Take a backup (8.2K file)
Drop the entire database
Restore it
Verify the counts match
Time it

Step 3 feels wrong to type. You're deleting the thing. And then:

FATAL: database "lumberyard_dev" does not exist

That's the message on someone's screen at 3am when a room goes quiet. My data was genuinely gone.

Then I restored it from that 8.2K file.

3 authors, 3 posts. Exact match. Nothing lost. 47 seconds.

That number matters more than it looks. In a real incident, everyone asks "how long until we're back?" Most people guess. I can answer, because I measured it.

What I'd tell past me

Three things I actually learned, beyond the commands:

Check the file size. A zero-byte backup file is a silent catastrophe that only surfaces when you desperately need it. Always look.

"No errors" is not "it worked." Verify the row counts. A restore that finishes quietly but incompletely is worse than one that fails loudly.

Backups don't belong in git. I committed my dump file, then realized: backups contain real data, they bloat the repo forever, and in a real job that's publishing customer data. They belong in object storage. I removed it and gitignored *.dump. A small fix, but the instinct matters.

The real takeaway

Code is replaceable. You lose it, you redeploy from git. Data is not. Lose it and it may be gone forever.

Which is why the most valuable thing I did this week wasn't building the stack or writing migrations. It was deleting everything and proving I could get it back. Most junior engineers have never restored a backup. They've just trusted one exists.

Don't trust. Drill.

Next up: containers, properly.

The Week Git Stopped Being Scary: Secrets, Conflicts, and Recovery

Vivian Chiamaka Okose — Mon, 13 Jul 2026 06:41:37 +0000

Most people learn just enough Git to add, commit, and push. I used to be one of them. This module was about crossing the gap between "I use Git" and "I trust Git with my work", the gap that, on most teams, separates a junior from a mid-level engineer. Here is everything I built, including the part where I got completely lost and how I found my way back.

Making it impossible to commit a secret

The single most valuable thing I set up: a pre-commit hook chain that scans every commit before it happens. I tested it by deliberately trying to commit a fake AWS key.

The commit never happened. The scanner (gitleaks) spotted the key, pointed to the exact file and line, and blocked it cold. Nothing leaked. Nothing to rotate.

This matters because leaked cloud keys get found by attackers within seconds of hitting a public repo, and the cleanup is genuinely painful. The whole philosophy here: turn discipline into automation. Not "remember to check for secrets," but "make it physically impossible to commit one." The same chain also enforces good commit messages, so a lazy "fixed stuff" gets rejected at the door.

Surviving my first merge conflict

I used to think merge conflicts were a disaster. They're not. A conflict is just Git saying "two people changed the same spot, you decide which wins."

I created one on purpose in a throwaway sandbox: changed the same line on two branches, then tried to combine them. Git stopped and marked the clash with <<<<<<< and >>>>>>> markers.

Resolving it was simpler than the reputation suggests: edit the file to what I actually want, delete the marker lines, tell Git it's resolved. That's the entire skill. Once I'd done it once, conflicts stopped being frightening.

The concept that unlocked everything

Halfway through, I got completely lost. Moving a commit between branches made no sense to me. I stopped and admitted I didn't understand, rather than pretending.

The fix was reframing what a branch actually is. A branch is not a copy of your code. It's a sticky note stuck on one commit. "Moving a commit to another branch" is really just sliding sticky notes around while the commit itself stays put. The moment that clicked, branching, resetting, and recovery all suddenly made sense.

Admitting I was lost was the best decision I made all module.

Recovering deleted work

Then I broke things on purpose. I ran git reset --hard and wiped three commits, including a merge. They vanished. In a real panic, this is the stomach-drop moment.

Except they weren't gone. Git keeps a diary called the reflog, a record of everywhere I've been. My "deleted" commits were still in it. One command, and everything came back.

This is the calm that experienced engineers have and beginners don't. git log only shows my current path; git reflog remembers the paths I left, including the lost work. Knowing that safety net exists changes how it feels to make a mistake.

Locking down main like a real team

Finally, I protected my main branch: no direct pushes, changes must come through a pull request, commits must be signed, history must stay linear. I proved it worked by trying to push straight to main, and watching GitHub reject it.

Then I did the full professional flow: made a branch, pushed it, opened a pull request, watched my automated checks run on GitHub's servers, and merged through the proper door.

That flow, branch, push, PR, pass checks, merge, is exactly what I'll do every day in a DevOps job. Building it on my own repo, with no pressure, is the best practice I could ask for.

What I actually learned

The commands matter, but the mindset matters more. Git isn't a thing to fear; it's a safety net that remembers almost everything. The real skills are: make good habits automatic so they don't depend on memory, and know your recovery moves before you need them.

And the most important lesson had nothing to do with Git: when I was lost, saying so out loud, instead of nodding along, was what let me actually learn. That's a habit worth keeping.

Next up: databases.

From Blank VM to Bastion: Everything I Learned Hardening Linux

Vivian Chiamaka Okose — Thu, 09 Jul 2026 08:49:25 +0000

A full walkthrough of a hands-on Linux module: hardening a server, building a multi-user permission system, writing a diagnostic toolkit, and setting up a real jump host. Plus the seven times the docs lied and the delete command that scared me.

Every server I will ever touch in DevOps runs Linux. Every container is a Linux process. Every CI runner shells out to bash. So I spent a module getting genuinely fluent at the command line, not copy-pasting my way through, but understanding every step. Here is everything I built, and more importantly, everything that went wrong on the way, because that is where the real learning was.

What I built

A hardened Ubuntu 24.04 server. Starting from a blank VM, I brought it to a production baseline: a deny-by-default firewall (UFW) with only SSH allowed, fail2ban guarding against brute-force login attempts, automatic security updates, a correct timezone, and SSH locked down to key-only auth with no root login.

The order matters more than it looks. Enable the firewall before allowing SSH, and you lock yourself out of your own server. Allow SSH first, then enable. That sequence has saved more careers than any clever command.

A multi-tenant permission system. Four users, each with a different level of access: two full developers with sudo constrained to specific commands, a read-only user who can look but not touch, and a contractor sealed into her own sandbox with an account set to auto-expire in 30 days. Then I tested every boundary by becoming each user and confirming the walls held.

This is where Linux permissions stopped being abstract. Watching a read-only user get "Permission denied" when they tried to write, and watching a "restricted" admin get blocked from reading the password file, made access control real in a way no diagram could.

A four-tool diagnostic toolkit. I got tired of typing the same six commands every time something felt slow, so I wrote tools that do it for me:

sysnap: a one-screen snapshot of a server's health (load, memory, top processes, disk, open ports, failed logins)
logscan: search the system logs for a pattern, with context, scoped to a time or service
diskmap: find what's actually eating the disk, biggest-first
netcheck: confirm I'm really online and reaching the things I depend on

Each is 30 to 45 lines of bash, all shellcheck-clean, all covered by a smoke test.

A real jump host (bastion). Production servers don't sit on the open internet. They hide behind a gateway you have to hop through. I built that with three VMs: a client, a bastion, and a private server with no direct route from outside. With one line of SSH config (ProxyJump) and one command, ssh bedrock-private, SSH tunneled through the bastion to reach the hidden box.

The proof it worked: SSH printed <no hostip for proxy command> during the handshake, its way of saying "I reached this through a proxy, not directly." That phrase is the fingerprint of a working jump host.

The seven times the docs lied

Here is what the tutorials never prepare you for: the documentation and my actual system disagreed seven separate times. Every time, the system was right. Learning to read what the machine tells me, instead of trusting what the guide claims, was the single most valuable skill of the module.

The OS version was wrong. I launched "22.04" and got 24.04. lsb_release -a is the source of truth, not the launch command.
A find-and-replace silently did nothing. My sed pattern didn't match, so it changed nothing, with no error. I only caught it by verifying with grep afterward. Silent no-ops are the most dangerous kind.
A command flag didn't exist. The guide used adduser --expiredate; my version had no such flag. The fix was a different tool (chage -E).
A missing package. setfacl didn't exist until I installed acl. "Command not found" is not always a typo.
My restricted user wasn't restricted. A user I'd limited to three commands still had full admin power, because sudo group membership silently overrode my rule. Checking effective permissions, not intended ones, revealed it.
Mutually exclusive flags. df -T and df --output couldn't be combined on my build.
grep broke my test. My smoke test failed because grep returns a failure exit code when it finds zero matches, which my set -euo pipefail correctly caught. Zero failed logins is good news, but grep calls it a failure. Guard it with || true.

The mistake that taught me the most

Near the end, I ran a recursive delete with a folder path accidentally on the list, and wiped my entire screenshots folder. No recycle bin. No undo.

Except the work survived, because it lived in two other places: committed in git, and backed up separately. One git checkout and a re-copy, and everything was back in under a minute.

That scare taught me more than any smooth step. Three habits I now hold without exception:

Commit often. Recovery took a minute instead of hours because the work was already saved somewhere safe.
ls before you rm. Look at exactly what you are about to delete.
Verify, don't assume. A command running without error does not mean it did what you intended.

The real takeaway

The gap between "works in the tutorial" and "works on my actual machine" is where the real skill lives. Run the command, read the error, understand what the system is telling you, adapt, repeat. Tutorials cannot teach that loop, because tutorials never fail. Your real system will, and that is a gift.

Next up: version control done properly. On to Git.

From Code to Cloud: Deploying 11 Microservices to AWS EKS with Terraform and ArgoCD

Vivian Chiamaka Okose — Sat, 06 Jun 2026 08:40:56 +0000

The CI pipeline was passing. Images were in ECR. Everything looked ready.

Then I ran kubectl get pods and saw this:

emailservice        0/1    CrashLoopBackOff   12 (3m ago)    24m
recommendationservice  0/1    ImagePullBackOff    0              5m

That was just the beginning. Before the app fully loaded on a live AWS URL, I hit authentication failures, image pull errors, a Kubernetes version mismatch in Terraform, and a pod scheduling limit I didn't know existed.

This is the full story of deploying to AWS EKS — what broke, why, and how each problem got solved.

The Stack

Before getting into what broke, here is what the deployment stack looks like:

Terraform — provisions the VPC, EKS cluster, node groups, and IAM roles on AWS
ArgoCD — watches the Git repo and syncs the Helm chart to the cluster automatically
Helm — packages all 11 Kubernetes manifests into one deployable chart
AWS EKS — managed Kubernetes on AWS
Amazon ECR — private container registry where CI pushes the Docker images

The flow is: Terraform creates the infrastructure, kubectl connects to the cluster, ArgoCD is installed, and from that point on every Git push triggers an automatic deployment.

Step 1 — Provisioning the Infrastructure with Terraform

The Terraform config creates everything the cluster needs: a VPC with public and private subnets across two availability zones, a NAT gateway for private subnet egress, an EKS cluster, and a managed node group with EC2 worker nodes.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    main = {
      instance_types = [var.node_instance_type]
      min_size       = var.node_min_count
      max_size       = var.node_max_count
      desired_size   = var.node_desired_count
      ami_type       = "AL2_x86_64"
    }
  }
}

Running terraform apply takes about 15 minutes. When it completes it outputs the cluster endpoint, certificate data, and ECR registry URL.

terraform init
terraform plan
terraform apply

After apply, connect kubectl to the new cluster:

aws eks update-kubeconfig \
  --name online-boutique-cluster \
  --region us-east-1

Problem 1 — kubectl Authentication Failure

The first thing that happened after the cluster was created was this:

error: You must be logged in to the server
(the server has asked for the client to provide credentials)

Running aws eks update-kubeconfig had added the cluster to the local kubeconfig but kubectl get nodes kept failing with credential errors.

The cause was that the cluster was created by one IAM identity but the local AWS CLI was authenticated as a different IAM user. EKS only grants automatic access to the identity that created the cluster. Everyone else has to be explicitly granted access.

The fix was creating an access entry for the current user:

aws eks create-access-entry \
  --cluster-name online-boutique-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::164885464623:user/crystal \
  --type STANDARD

aws eks associate-access-policy \
  --cluster-name online-boutique-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::164885464623:user/crystal \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

After that, kubectl get nodes returned the two worker nodes as expected.

Problem 2 — Kubernetes Version Mismatch in Terraform

When I later ran terraform apply to scale the node group, Terraform tried to downgrade the cluster from Kubernetes 1.30 to 1.29.

Error: updating EKS Cluster (online-boutique-cluster) version:
Unsupported Kubernetes minor version update from 1.30 to 1.29

The cluster_version variable in variables.tf said 1.29 but the actual cluster was already running 1.30 — AWS had auto-upgraded it. Terraform noticed the mismatch and tried to make reality match the config, which meant a downgrade. AWS doesn't allow Kubernetes downgrades.

The fix was updating the variable to match the actual running version:

variable "cluster_version" {
  description = "Kubernetes version for EKS"
  type        = string
  default     = "1.30"
}

This is a good reminder to always keep Terraform variables in sync with the actual state of your infrastructure, especially after AWS performs automatic maintenance upgrades.

Step 2 — Installing ArgoCD on EKS

With the cluster running and kubectl authenticated, the next step was installing ArgoCD:

kubectl create namespace argocd
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

One error came up during the install:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: must have at most 262144 bytes

This is a known issue with large CRDs and kubectl apply. The manifest is too big to fit in the annotation that kubectl apply uses to track changes. The fix is to use kubectl create for that specific CRD:

kubectl create -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/crds/applicationset-crd.yaml

After that all ArgoCD pods came up healthy:

kubectl get pods -n argocd

Step 3 — Deploying the App via ArgoCD

With ArgoCD running, the next step was creating the Application manifest that tells ArgoCD where to find the Helm chart and where to deploy it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: online-boutique
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/pawsible-cloud/online-boutique-platform.git
    targetRevision: main
    path: helm-chart
    helm:
      releaseName: online-boutique
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: online-boutique
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Applying this triggered the first sync:

kubectl apply -f argocd/application.yaml

ArgoCD pulled the Helm chart from Git, rendered the manifests, and started creating pods in the online-boutique namespace.

Problem 3 — ImagePullBackOff on Two Services

The first kubectl get pods -n online-boutique showed most services running but two stuck:

emailservice          0/1   ImagePullBackOff   0   5m
recommendationservice 0/1   ImagePullBackOff   0   5m

Running kubectl describe pod on one of them showed:

Failed to pull image "emailservice": pull access denied,
repository does not exist or may require 'docker login'

The image reference in the deployment was just emailservice — no registry, no tag. Kubernetes didn't know where to pull from so it tried Docker Hub, got denied, and gave up.

The Helm values file had the image field set incorrectly. The correct reference needed the full ECR URI:

164885464623.dkr.ecr.us-east-1.amazonaws.com/online-boutique/emailservice:v0.10.5

After fixing the values file and pushing to Git, ArgoCD detected the change and redeployed with the correct image references.

Problem 4 — The 500 Error on the Live Site

The app got a load balancer URL. Opening it in the browser showed this:

HTTP Status: 500 Internal Server Error
rpc error: code = Unavailable
dial tcp 172.20.197.197:7070: connect: connection refused
could not retrieve cart

The frontend was up but cartservice was unreachable. Checking the pods showed cartservice was in Pending state — it had never scheduled onto a node.

That led to Problem 5.

Problem 5 — Pod Scheduling Failures (The Unexpected One)

Three pods were stuck in Pending. Running kubectl describe pod showed:

0/2 nodes are available: 2 Too many pods.

Not out of memory. Not out of CPU. Too many pods.

Here is what I didn't know: AWS calculates the maximum number of pods a node can run based on the number of network interfaces and IP addresses the instance type supports, not just available resources. A t3.medium node maxes out at 11 pods per node.

With 11 app services plus ArgoCD plus Prometheus all competing for slots across 2 nodes (22 slots total), and system pods also taking space, there weren't enough slots for every pod to schedule.

kubectl describe nodes | grep -A 5 "Allocatable"
# pods: 11
# pods: 11

The first attempt was adding a third node by updating Terraform:

variable "node_desired_count" {
  default = 3
}

But the AWS account had a restriction blocking t3.medium On-Demand instance launches on a free tier account.

The practical fix was scaling down the load generator — the least critical service — to free up exactly enough slots:

kubectl scale deployment loadgenerator \
  -n online-boutique --replicas=0

That freed one slot. The remaining pending pods scheduled immediately. Every service came up. The app loaded.

The App Live on AWS

After all of that — the authentication fix, the version mismatch, the image pull errors, the scheduling limit — the app finally loaded on the AWS load balancer URL.

Every service running. Every pod healthy. ArgoCD showing Synced and Healthy.

What This Project Taught Me About EKS

A few things that aren't obvious until you actually run into them:

EKS access control is IAM-based. The identity that creates the cluster gets access automatically. Everyone else needs an explicit access entry. If you're working in a team or switching between IAM users, set up access entries from the start.

Pod limits on AWS are network-based, not resource-based. t3.medium maxes at 11 pods per node regardless of how much CPU or memory is free. If you're running a lot of services, plan your node count accordingly or use a larger instance type.

Terraform and AWS can drift. AWS performs automatic upgrades and maintenance. If your Terraform config doesn't reflect the actual state, the next apply will try to reconcile the difference — sometimes in a direction you don't want.

GitOps makes debugging easier. Because every deployment comes from Git, you always know exactly what version of the config is running. When something breaks you know where to look.

What Comes Next

With the app running on EKS and ArgoCD managing deployments, the next step was setting up monitoring. Prometheus and Grafana give visibility into what the cluster is actually doing — pod health, resource usage, and alerts when things go wrong.

The next post covers the full monitoring setup, what the ServiceMonitor limitation taught me about how the Online Boutique is instrumented, and how to build useful Grafana dashboards with the metrics you actually have.

This is part of an ongoing series documenting a full DevOps project built on Google's Online Boutique microservices demo, deployed to AWS EKS with Terraform, GitHub Actions, ArgoCD, Helm, Prometheus, and Grafana.

Repo: github.com/pawsible-cloud/online-boutique-platform

Why My CI Pipeline Kept Failing (And What I Had to Redesign to Fix It)

Vivian Chiamaka Okose — Fri, 05 Jun 2026 17:43:42 +0000

My CI pipeline kept rejecting pushes. Same error, every single run.

error: failed to push some refs to 'https://github.com/...'
Updates were rejected because the remote contains work that
you do not have locally.

The first time it happened I thought it was a simple git issue. Pull, rebase, push. Done.

It wasn't done. It happened again on the next run. And the one after that.

This is the story of what was actually broken, why a retry loop didn't fix it, and the architectural change that finally solved it.

What the Pipeline Was Doing

The CI pipeline builds 11 Docker images — one per microservice — and pushes them to Amazon ECR. After each build, it updates a values.yaml file in the repo with the new image tag, commits the change, and pushes it back to Git. ArgoCD watches that file and deploys automatically when it changes.

Here is a simplified version of what each job was doing:

- name: Build, tag, and push image to ECR
  run: |
    docker build -t "$IMAGE_URI" ./src/${{ matrix.service }}/
    docker push "$IMAGE_URI"

- name: Update image tag in Helm values
  run: |
    yq -i ".${{ matrix.service }}.image.tag = \"${IMAGE_TAG}\"" \
      helm-chart/values.yaml
    git add helm-chart/values.yaml
    git commit -m "ci: update ${{ matrix.service }} image to ${IMAGE_TAG}"
    git push

Looks reasonable. The problem is that matrix strategy runs all 11 jobs in parallel.

The Actual Problem: A Race Condition

Here is what happens when 11 jobs all run at the same time:

All 11 jobs check out the repo at the same commit
All 11 build their image and push to ECR
All 11 try to update values.yaml and push at roughly the same time
The first job to push wins
Every other job gets rejected because the remote has moved forward

! [rejected] vivian -> vivian (fetch first)
error: failed to push some refs
hint: Updates were rejected because the remote contains work
hint: that you do not have locally.

This is a classic race condition. Multiple writers, one file, no coordination.

The First Fix Attempt: Retry with Rebase

The first thing I tried was adding a retry loop. Fetch the latest, rebase, try again. If it fails, wait a random number of seconds and retry up to 5 times.

- name: Update image tag in Helm values
  run: |
    for i in 1 2 3 4 5; do
      git fetch origin
      git rebase origin/${{ github.ref_name }}

      yq -i ".${{ matrix.service }}.image.tag = \"${IMAGE_TAG}\"" \
        helm-chart/values.yaml

      git add helm-chart/values.yaml
      git diff --staged --quiet && echo "No changes to commit" && exit 0
      git commit -m "ci: update ${{ matrix.service }} image to ${IMAGE_TAG}"

      git push && echo "Push succeeded" && exit 0

      echo "Push failed, attempt $i of 5. Retrying..."
      sleep $((RANDOM % 10 + 5))
    done

    echo "All push attempts failed"
    exit 1

This helped but didn't fully solve it. During the rebase on retry, two jobs would hit a merge conflict on the same file because both had modified values.yaml in different ways.

CONFLICT (content): Merge conflict in helm-chart/values.yaml
error: could not apply fad9a46...

A retry loop treats the symptom. The real problem is the design.

The Real Fix: One Writer, Not Eleven

The root cause is that 11 jobs should never be writing to the same file at the same time. The solution is to separate the concerns completely.

Build jobs do one thing: build and push the image.

A single downstream job, running only after all 11 builds finish, updates values.yaml once with all the new tags in a single commit.

Here is the redesigned workflow:

jobs:
  build-and-push:
    name: Build ${{ matrix.service }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        service:
          - frontend
          - cartservice
          - productcatalogservice
          - currencyservice
          - paymentservice
          - shippingservice
          - emailservice
          - checkoutservice
          - recommendationservice
          - adservice
          - loadgenerator
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, tag, and push image to ECR
        env:
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t "$IMAGE_URI" ./src/${{ matrix.service }}/
          docker push "$IMAGE_URI"

  update-helm-values:
    name: Update Helm values
    runs-on: ubuntu-latest
    needs: build-and-push
    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/vivian'
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          fetch-depth: 0

      - name: Install yq
        run: |
          sudo wget -qO /usr/local/bin/yq \
            https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
          sudo chmod +x /usr/local/bin/yq

      - name: Update all image tags in Helm values
        env:
          IMAGE_TAG: ${{ github.sha }}
        run: |
          services=(
            frontend cartservice productcatalogservice currencyservice
            paymentservice shippingservice emailservice checkoutservice
            recommendationservice adservice loadgenerator
          )
          for service in "${services[@]}"; do
            yq -i ".$service.image.tag = \"${IMAGE_TAG}\"" \
              helm-chart/values.yaml
            echo "Updated $service to ${IMAGE_TAG}"
          done

      - name: Commit and push updated values
        env:
          IMAGE_TAG: ${{ github.sha }}
        run: |
          git config user.name "GitHub Actions Bot"
          git config user.email "actions@github.com"
          git add helm-chart/values.yaml
          git diff --staged --quiet && echo "No changes to commit" && exit 0
          git commit -m "ci: update all images to ${IMAGE_TAG}"
          git push origin ${{ github.ref_name }}

The key line is needs: build-and-push. This tells GitHub Actions to wait until every single build job has completed successfully before running the update job.

What Changed and Why It Works

Before: 11 jobs, each writing to values.yaml as soon as their build finished. No coordination. First push wins, rest fail.

After: 11 jobs build and push images. They do nothing else. One job runs after all of them finish, loops through all services, updates every tag in a single pass, commits once, pushes once.

One writer. No conflicts. Clean history.

The full workflow history tells the story clearly — the red X runs on the left where the old design kept failing, and the green checkmarks after the fix went in.

The Broader Lesson

Race conditions in CI pipelines are easy to miss because each individual step looks correct. The build step is correct. The push step is correct. The commit step is correct. But the system design is wrong.

When you have parallel jobs touching shared state, you need to ask: who owns this resource? In this case, values.yaml should have exactly one writer. Once I framed it that way, the fix was obvious.

If you are building a similar GitOps pipeline with GitHub Actions and Helm, separate your build jobs from your manifest update job from the start. It saves you a frustrating debugging session later.

What Comes Next

This pipeline feeds directly into ArgoCD, which watches values.yaml and syncs the cluster automatically when the file changes. In the next post I'll walk through setting up ArgoCD on EKS, connecting it to the repo, and the pod scheduling problem that showed up once everything was deployed.

Repo: github.com/pawsible-cloud/online-boutique-platform

I Deployed 11 Microservices Locally With kind, Then Broke One on Purpose published

Vivian Chiamaka Okose — Thu, 28 May 2026 18:10:47 +0000

Online Boutique running locally on kind. 11 microservices, 5 languages, one kubectl apply command.

I want to be honest with you: before this deployment, "microservices" was just a word I had read about. I knew the definition. I could explain it in theory. But I had never actually run a microservices application and watched it breathe.

This changed that.

Google's Online Boutique is a demo e-commerce app made up of 11 separate services, written in five different programming languages, all talking to each other over gRPC. I ran all of it on my laptop using kind, and then I did something that is not in any tutorial: I injected a CPU fault into one of the services just to see what would happen.

What is Online Boutique and Why Does It Matter?

Online Boutique is an open-source cloud-native reference application built by Google. It is not a toy. Google uses it internally to demo Kubernetes, gRPC, distributed tracing, and service mesh patterns.

What makes it interesting for learning is the architecture. Every part of the app is a completely separate service with its own codebase, its own container, and its own runtime:

frontend is Go. It is the HTTP entry point — the thing your browser talks to.
cartservice is C#. It stores your cart items in Redis.
checkoutservice is Go. It calls six other services to complete one order.
recommendationservice is Python. It suggests related products.
adservice is Java. It returns ads based on page context.

They all run at the same time. They all depend on each other. That is exactly what a microservices production environment looks like.

Why I Used kind Instead of minikube

kind stands for Kubernetes IN Docker. Instead of spinning up a virtual machine (like minikube does), kind runs your entire Kubernetes cluster inside Docker containers. That means:

No hypervisor needed
Faster cluster startup
Lower resource overhead
Clean teardown: kind delete cluster removes everything

If you have Docker running, you can have a Kubernetes cluster in under a minute.

Setting It Up

Install kind and kubectl, then create your cluster:

kind create cluster --name online-boutique

Clone the repo and deploy:

git clone https://github.com/GoogleCloudPlatform/microservices-demo.git
cd microservices-demo
kubectl apply -f ./release/kubernetes-manifests.yaml

One command creates 22 Kubernetes resources across 11 services.

Now watch the pods come up:

kubectl get pods --watch

The pull phase. Docker is downloading container images for all 11 services. This is the real process — not just the happy ending.

First run takes 5–10 minutes because Docker has to pull images. After that it is fast because images are cached locally.

All 11 pods Running. This is the screenshot I had been working towards.

Accessing the Application

kind does not give you a LoadBalancer (that requires a cloud provider). Use port-forward instead:

kubectl port-forward deployment/frontend 8080:8080

Open http://localhost:8080 in your browser.

An e-commerce store running entirely on my laptop. 11 services, all talking to each other over gRPC.

I clicked through the whole app. Added items to the cart. Completed a checkout. Every action crossed multiple service boundaries: the frontend called the productcatalogservice, the cartservice wrote to Redis, the checkoutservice called the paymentservice, emailservice, and shippingservice in sequence.

The Thing That Surprised Me: Live Traffic Was Already Running

After everything was up, I ran this out of curiosity:

kubectl logs -l app=frontend --tail=30

The logs were already full of requests. Not from me.

The loadgenerator service — which is part of the deployment — had been quietly simulating real user traffic the whole time. It browses the store, adds items to carts, and checks out orders on a loop, automatically. I did not configure it. It just happened.

That was the moment the architecture clicked for me: in a real production system, observability tools would already be watching this traffic, flagging anomalies, measuring latency. The loadgenerator is a built-in simulation of production load.

Then I Broke It on Purpose

Once I confirmed everything was healthy, I injected a CPU fault into the recommendationservice:

kubectl exec -it deployment/recommendationservice -- sh -c "while true; do true; done"

What this does: it opens a shell inside the running container and starts an infinite loop that burns CPU as fast as possible. The container is now consuming all the CPU it can get.

CPU usage on recommendationservice spiking after fault injection. Other services remain stable.

What I expected: Kubernetes would notice, restart the pod, and everything would be fine.

What actually happened: Nothing. Kubernetes did not restart the pod. It just kept running, getting slower and slower, because the container was not crashing. It was degrading.

Kubernetes only restarts containers that fail health checks or crash outright. A container that is alive but overwhelmed is invisible to the default scheduler.

The recommendation feature slowed down. The rest of the app kept working fine — cart, checkout, payment, all normal.

To recover, I deleted the pod manually:

kubectl delete pod -l app=recommendationservice

Kubernetes immediately created a fresh one. Clean slate. Thirty seconds later, recommendations were back to normal.

What I Took Away From This

kubectl apply -f is more powerful than it looks. One command, 22 resources, 11 services. The manifest is the source of truth.
Microservices isolate failure well. A broken recommendationservice did not take down the cart or checkout. That is the point of the architecture.
Kubernetes does not save you from everything. A degraded-but-not-dead service is an invisible problem without resource limits, liveness probes, and readiness probes properly configured.
kind is excellent for local Kubernetes experimentation. Fast, clean, no cloud cost.
The loadgenerator is underrated. It gives you something to observe immediately without setting up synthetic monitoring yourself.

What's Next

Next step for me is deploying the same application on AWS EKS and Azure AKS. The Kubernetes manifests will not change at all. That is the whole point of cloud-native architecture: the workload does not care where it runs. The infrastructure is the variable.

I will document both cloud deployments the same way I documented this one.

If you are learning DevOps or cloud engineering and have not tried running Online Boutique locally yet — do it. Then break something. That is where the real learning is.

GitHub: github.com/vivianokose
LinkedIn: linkedin.com/in/okosechiamaka

I Ran 9 Kubernetes Labs on a Local KIND Cluster — Here Is Everything I Learned

Vivian Chiamaka Okose — Sun, 26 Apr 2026 19:20:50 +0000

By Vivian Chiamaka Okose | DevOps Engineer

If you have been putting off learning Kubernetes because it feels overwhelming, this post is for you.

This week I completed a 9 hands-on labs covering the core building blocks of Kubernetes: Pods, ReplicaSets, Deployments, Horizontal Pod Autoscaler, health probes, and all three Service types. I ran everything locally on a KIND (Kubernetes IN Docker) cluster inside WSL Ubuntu on Windows, with zero cloud costs.

Here is exactly what I did, what broke, and what I now understand that I did not before.

The Setup: KIND on WSL

Before I could run a single kubectl command, I needed a cluster. I chose KIND because it runs Kubernetes entirely inside Docker containers on your local machine. If you already have Docker, you are most of the way there.

# Install KIND
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.23.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

# Create the cluster
kind create cluster --name k8s-labs
kubectl get nodes

Within two minutes I had a real Kubernetes node showing Ready. That felt good.

Assignment 1: Pods, ReplicaSets, and Deployments

Lab 1: Your First Pod

A Pod is the atomic unit in Kubernetes. Everything else in the system is built around it. I created one two ways.

The imperative way (fast, not repeatable):

kubectl run nginx-pod --image=nginx
kubectl get pods

The declarative way (YAML, production standard):

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: nginx
spec:
  containers:
    - name: nginx-container
      image: nginx
      ports:
        - containerPort: 80

kubectl apply -f nginx-pod.yaml
kubectl describe pod nginx-pod
kubectl logs nginx-pod
kubectl exec -it nginx-pod -- /bin/bash

The key insight: YAML is how you work in production because it is version-controllable, reviewable, and reproducible. The imperative method is fine for quick experiments but you would never use it to manage a real system.

Lab 2: ReplicaSets — the First Sign of Real Resilience

A single Pod has a problem: if it crashes or is deleted, it is gone. You would have to manually recreate it. That is not acceptable in production.

A ReplicaSet fixes this by ensuring a fixed number of Pod replicas are always running.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-replicaset
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.21.1

I applied this, confirmed 3 Pods were running, then deleted one:

kubectl delete pod nginx-replicaset-xxxxx
kubectl get pods

Before I could even blink, Kubernetes had already created a replacement. That moment is when auto-healing stops being a concept and becomes something you have actually seen.

I also scaled from 3 to 5 by changing replicas: 3 to replicas: 5 and reapplying. Two extra Pods appeared immediately.

The catch: ReplicaSets do not support rolling updates. That is where Deployments come in.

Lab 3: Deployments — Zero Downtime Updates

A Deployment manages a ReplicaSet and adds the features you actually need for production: rolling updates and rollback.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

This config means: spin up 1 new Pod before killing an old one, never leave any Pods unavailable. That is zero downtime.

To update the nginx image:

kubectl set image deployment/nginx-deployment nginx=nginx:1.23.0
kubectl rollout status deployment/nginx-deployment

To roll back instantly:

kubectl rollout undo deployment/nginx-deployment
kubectl rollout history deployment/nginx-deployment

The rollout history command showed me the revision log like a git blame for infrastructure. That is powerful.

Assignment 2: Auto-Scaling and Health Management

Lab 4: HPA — Let Kubernetes Decide How Many Pods You Need

The Horizontal Pod Autoscaler reads CPU (or memory) metrics and scales your Deployment up or down automatically.

For this to work on KIND, I needed to install the metrics-server with a flag for local clusters:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch deployment metrics-server -n kube-system --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

Then I created an HPA:

kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=5

And in a second terminal, I hammered it with a load generator:

kubectl run -i --tty load-generator --rm --image=busybox:1.28 \
  --restart=Never -- /bin/sh -c \
  "while sleep 0.01; do wget -q -O- http://php-apache; done"

Watching kubectl get hpa --watch while the load generator ran was one of those moments that makes DevOps genuinely fun. The Pod count went from 1 to 5. When I killed the load generator, it scaled back down on its own.

This is why autoscaling matters: you pay for what you use, and your app handles spikes without anyone manually touching it.

Lab 5: Readiness Probes — the Traffic Gate

A readiness probe answers the question: "Is this Pod ready to receive user traffic?"

readinessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

Without a readiness probe, Kubernetes sends traffic to a Pod the moment it starts, even if the application inside is still warming up. That causes 502 errors during deployments.

With a readiness probe, the Pod only enters the Service's endpoint list after passing the health check. You get clean deployments with no cold-start errors reaching users.

Lab 6: Liveness Probes — the Self-Healing Mechanism

A liveness probe answers a different question: "Is this running container still healthy?"

livenessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3

To see it in action, I deleted the nginx index.html file inside a running Pod:

kubectl exec -it <pod-name> -- rm /usr/share/nginx/html/index.html
kubectl get pods --watch

After 3 consecutive failed probes (30 seconds), Kubernetes restarted the container. The RESTARTS counter went from 0 to 1. No manual action required.

The real-world case for this: imagine an app that hits a deadlock or runs out of memory and becomes unresponsive but the process is still technically running. Without a liveness probe, that Pod sits there broken forever. With one, Kubernetes restarts it within minutes.

Assignment 3: Services and Networking

This assignment was the one that changed how I think about Kubernetes networking.

Lab 7: ClusterIP — Stable Internal Communication

Pod IPs are ephemeral. Every time a Pod restarts or is replaced, it gets a new IP. If another service is hard-coding that IP, it breaks.

A ClusterIP Service solves this by providing:

A stable virtual IP that never changes
A stable DNS name (e.g. nginx-svc.default.svc.cluster.local)
Automatic load balancing across all Ready Pods

apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

I proved it works by launching a busybox Pod and making requests from inside the cluster:

kubectl exec -it tester -- sh -c "wget -qO- http://nginx-svc | head -5"
kubectl exec -it tester -- nslookup nginx-svc

The DNS lookup resolved to the ClusterIP and the request succeeded. That is microservice communication working exactly as it should.

Lab 8: NodePort — Opening the Door from Outside

NodePort opens a specific port (between 30000-32767) on every node, making your app reachable from outside the cluster.

spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080

On KIND, since there is no real node IP accessible from the host, I used port-forward:

kubectl port-forward svc/nginx-nodeport 8080:80

Opening http://localhost:8080 in the browser showed the NGINX welcome page. External access confirmed.

Lab 9: LoadBalancer — the Cloud-Native Way to Go External

In a real cloud environment (Azure, AWS, GCP), a LoadBalancer Service automatically provisions a public IP address and routes internet traffic to your Pods. In KIND, there is no cloud behind it, so I used MetalLB to simulate this.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml

After configuring an IP address pool from the Docker bridge subnet, I created a LoadBalancer Service and got an external IP assigned from that pool:

kubectl get svc nginx-loadbalancer
# NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)
# nginx-loadbalancer   LoadBalancer   10.96.xxx.xxx  172.18.255.200   80:xxxxx/TCP

A curl against that external IP returned the NGINX welcome page. In Azure, that IP would be a real public internet address. MetalLB let me understand the behaviour without spending a penny on cloud resources.

What I Would Tell Anyone Starting Kubernetes

Start with Pods. Not because they are what you use in production (you use Deployments), but because understanding what a Pod is makes every other concept click faster.

Break things on purpose. Delete a Pod. Break a readiness probe. Kill a container. The fastest way to understand Kubernetes is to watch it recover from failure.

Health probes are not optional. Every production Deployment needs both a readiness probe and a liveness probe. Not having them is like deploying blind.

Services exist because Pod IPs are unreliable. Once you understand that one sentence, all three service types make sense immediately.

KIND is excellent for local learning. You get a real Kubernetes cluster with no cloud bill. The only gotchas are the metrics-server TLS flag and needing MetalLB for LoadBalancer behaviour.

What Is Next

Next up: Kubernetes Ingress, ConfigMaps, Secrets, and persistent storage. Each week I am writing up what I actually did, what broke, and what I learned from it.

If you are on a similar path, follow along. And if you are already deep into Kubernetes, I would love to know what concept took the longest to click for you.

Vivian Chiamaka Okose
DevOps Engineer
LinkedIn | GitHub

How I Deployed a Full-Stack Bookstore with Docker Compose on Azure (Capstone Project)

Vivian Chiamaka Okose — Tue, 21 Apr 2026 20:37:28 +0000

What I Deployed

The EpicBook is an online bookstore with a Node.js + Express backend, MySQL database, Handlebars frontend, and Nginx reverse proxy. It has real data: 54 books and 53 authors.

The architecture:

Internet (port 80)
        |
        v
Nginx (reverse proxy)
        |  frontend_network
        v
Node.js App (port 8080, internal)
        |  backend_network
        v
MySQL (port 3306, internal only)

Only Nginx faces the internet. MySQL is never reachable from outside the VM.

The .env File (Secrets First)

Before writing a single Dockerfile, I created a .env file:

MYSQL_ROOT_PASSWORD=<strong-password>
MYSQL_DATABASE=bookstore
MYSQL_USER=epicbook
MYSQL_PASSWORD=<strong-password>
NODE_ENV=production
PORT=8080
DB_HOST=db
DB_USER=epicbook
DB_PASSWORD=<strong-password>
DB_NAME=bookstore

The original config.json had hardcoded credentials pointing to a Heroku database URL. I updated it to use direct credentials pointing to the db Docker Compose service name. Secrets stay in .env, not in code.

The Multi-Stage Dockerfile

# Stage 1 - install production dependencies
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Stage 2 - production runtime
FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY . .
EXPOSE 8080
CMD ["node", "server.js"]

The builder stage installs only production dependencies. The runtime stage gets clean node_modules and source code. No dev tools in the final image.

The Nginx Config

upstream epicbook_app {
    server app:8080;
}

server {
    listen 80;
    server_name _;

    location / {
        proxy_pass http://epicbook_app;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

app is the Docker Compose service name. Docker's internal DNS resolves it automatically. No IP addresses needed.

The docker-compose.yml

Three services, two networks, two volumes:

version: '3.8'

networks:
  frontend_network:
    driver: bridge
  backend_network:
    driver: bridge

volumes:
  db_data:
  nginx_logs:

services:
  db:
    image: mysql:8.0
    networks:
      - backend_network
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      interval: 10s
      retries: 5
      start_period: 30s

  app:
    build: .
    depends_on:
      db:
        condition: service_healthy
    networks:
      - frontend_network
      - backend_network

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    depends_on:
      app:
        condition: service_healthy
    networks:
      - frontend_network

The depends_on with condition: service_healthy is the key. Nothing starts before the thing it depends on is ready.

What Went Wrong (And How I Fixed It)

Problem 1: Books not loading after deployment

The site came up but showed "no books available." The seed SQL files ran without errors but inserted no data.

The issue was that docker exec with stdin redirection reads the file from the VM host but executes inside the container. The file was not inside the container.

Fix: Use docker cp to copy the files into the container first, then execute them with source:

docker cp db/author_seed.sql epicbook_db:/tmp/author_seed.sql
docker exec epicbook_db mysql -u epicbook -pPassword bookstore -e "source /tmp/author_seed.sql"

Problem 2: Nginx healthcheck showing unhealthy

The healthcheck used wget which is not in nginx:alpine by default.

Fix: Removed the Nginx healthcheck entirely. The site was serving traffic correctly and the app healthcheck was already ensuring startup order.

Problem 3: docker-compose.yml corrupted by heredoc paste

Special characters in the YAML caused the heredoc to break mid-paste.

Fix: Used a Python script to write the file content instead of shell heredoc:

python3 -c "
content = '''..yaml content..'''
with open('docker-compose.yml', 'w') as f:
    f.write(content)
"

The Persistence Test

Before restart:

SELECT COUNT(*) FROM Book; --> 54

Run:

docker-compose down && docker-compose up -d

After restart:

SELECT COUNT(*) FROM Book; --> 54

Every container was destroyed and recreated. The data survived because it lives in the db_data named volume, not in the container.

GitHub

https://github.com/vivianokose/the-epicbook_capstone.git

Multi-Stage Docker Builds: How I Cut a React Image from 760MB to 94MB

Vivian Chiamaka Okose — Fri, 17 Apr 2026 15:07:31 +0000

I built two Docker images for the same React app this week.

One was 760MB. The other was 94MB. Both loaded the exact same website in the browser.

That 87.6% difference is the story of this post.

The Setup

This is Week 14 of my DevOps Micro Internship. The project: containerize a React app two ways, compare the results, and explain what changed and why it matters.

I am running everything on an Azure VM (Ubuntu 24.04 LTS) with Docker auto-installed via cloud-init.

The React app: https://github.com/pravinmishraaws/my-react-app

First: The .dockerignore

Before writing a single Dockerfile, I created a .dockerignore to keep things that should never be in an image out of the build context:

node_modules
build
.dockerignore
.git
.gitignore
*.md

This is especially important for node_modules. If you do not exclude it, Docker copies your entire local node_modules into the build context, which defeats the whole purpose of running npm ci inside the container.

Approach 1: Single-Stage Baseline (Dockerfile.single)

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
RUN npm install -g serve
EXPOSE 3000
CMD ["serve", "-s", "build", "-l", "3000"]

This image does everything in one go. Install dependencies, build the app, serve it. Simple.

The problem is that everything stays. Node.js, npm, all 1,342 packages, build tools. None of that is needed to serve a built React app. But it is all sitting there in the image.

Result: 760MB

Build command:

docker build -f Dockerfile.single -t react-single:latest .

Run command:

docker run -d --name react-single -p 3000:3000 --restart unless-stopped react-single:latest

Approach 2: Multi-Stage Build (Dockerfile)

# Stage 1 - build React app
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2 - serve with nginx
FROM nginx:alpine
COPY --from=builder /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Two stages. Stage 1 builds the app. Stage 2 starts completely fresh with nginx:alpine and picks up only the finished build/ folder from Stage 1.

Node.js never makes it into the final image. Neither do npm or any of those 1,342 packages.

Result: 94MB

Build command:

docker build -t react-multi:latest .

Run command:

docker run -d --name react-multi -p 80:80 --restart unless-stopped react-multi:latest

The Comparison

docker images

Image	Size
react-single:latest	760 MB
react-multi:latest	94 MB
Reduction	87.6%

Both containers ran simultaneously. Both loaded the same React app. The only difference was what was inside each image.

Why This Matters Beyond the Numbers

Security: Every package not in the final image cannot be exploited. The multi-stage image has no Node.js, no npm, no build tools. An attacker who somehow gets into that container finds a bare Nginx server. Nothing else.

CI/CD Speed: Smaller images push and pull faster. If your pipeline deploys 10 times a day and each deployment pulls a 760MB image instead of a 94MB one, that is a significant amount of wasted time over weeks and months.

Layer Caching: Notice that both Dockerfiles copy package.json before the rest of the source code. This is intentional. Docker caches each layer. If your dependencies have not changed, Docker skips the npm ci step entirely on the next build and jumps straight to copying your source. This alone can shave minutes off build times.

Running Both Simultaneously

One of the most satisfying parts of this project was running both containers at the same time on the same VM:

docker ps

CONTAINER ID   IMAGE                 PORTS                    NAMES
837fb6d5def2   react-multi:latest    0.0.0.0:80->80/tcp       react-multi
66efa6b350bf   react-single:latest   0.0.0.0:3000->3000/tcp   react-single

Opening both in the browser showed the same React app on two different ports, proving the multi-stage approach produces an identical result in a fraction of the space.

Full Project

GitHub: https://github.com/vivianokose/cloud-vm-docker-deploy

See you in the next one.

How I Deployed a Live Website Using Docker on Azure (And Let Cloud-Init Do the Heavy Lifting)

Vivian Chiamaka Okose — Thu, 16 Apr 2026 20:39:44 +0000

What I Built

1 Linux VM on Azure (Ubuntu 24.04 LTS, Standard D2lds v6)
A cloud-init script that installed Docker automatically on first boot
A Dockerized static website served by Nginx
A live URL accessible from anywhere in the world

Step 1: Provisioning the Azure VM

I created the VM through the Azure Portal with these settings:

Field	Value
Resource Group	docker-project-rg
VM Name	docker-vm
Region	UK South
Image	Ubuntu 24.04 LTS
Size	Standard D2lds v6
Authentication	SSH public key

Under the Networking tab, I opened port 22 for SSH and port 80 for HTTP traffic by adding inbound rules to the Network Security Group.

Step 2: The Cloud-Init Script (This Is the Magic Part)

Before launching the VM, I pasted this script under the Advanced tab in the Custom Data field:

#cloud-config
package_update: true
package_upgrade: true
packages:
  - apt-transport-https
  - ca-certificates
  - curl
  - gnupg
  - lsb-release

runcmd:
  - apt-get update -y
  - apt-get install -y docker.io
  - systemctl enable docker
  - systemctl start docker
  - usermod -aG docker azureuser

This script runs automatically the moment the VM boots. Docker installs itself, starts itself, and adds my user to the docker group. I had not even SSH'd in yet.

When I later ran:

sudo cat /var/log/cloud-init-output.log | grep -i docker

The log confirmed everything. Docker was installed by the startup script, not by me. That is infrastructure automation doing exactly what it is supposed to do.

Step 3: SSH Into the VM

Once deployment completed, I grabbed my public IP from the Azure Portal and connected:

chmod 400 docker-vm_key.pem
ssh -i docker-vm_key.pem azureuser@4.234.163.212

Then I verified Docker was running:

docker --version
# Docker version 29.1.3

docker ps
# Empty, no containers yet. But Docker is alive.

Step 4: Clone the Static Website

git clone https://github.com/pravinmishraaws/Azure-Static-Website.git
cd Azure-Static-Website
ls
# README.md  index.html

Simple. Just an index.html. That is all we need.

Step 5: Write the Dockerfile

FROM nginx:alpine
RUN rm -rf /usr/share/nginx/html/*
COPY . /usr/share/nginx/html
EXPOSE 80

Let me break this down:

FROM nginx:alpine — we are using a lightweight version of Nginx as our base image. Alpine Linux is tiny, which keeps our container small and fast.

RUN rm -rf /usr/share/nginx/html/* — wipes out the default Nginx welcome page so our site shows instead.

COPY . /usr/share/nginx/html — copies everything in our current folder (including index.html) into the Nginx web root.

EXPOSE 80 — tells Docker this container will accept traffic on port 80.

Step 6: Build the Image

docker build -t static-site:latest .

Docker pulls nginx:alpine, runs each step in the Dockerfile, and produces an image called static-site:latest. The whole process takes about a minute.

After building, I checked the image sizes:

Image	Size
nginx:alpine	93.5 MB
static-site:latest	92.9 MB

Our image is actually slightly smaller than the base because we replaced Nginx's default content with our own lighter files.

Step 7: Run the Container

docker run -d --name static-site \
  -p 80:80 \
  --restart unless-stopped \
  static-site:latest

Breaking down the flags:

-d runs the container in the background
--name static-site gives it a friendly name
-p 80:80 maps port 80 on the VM to port 80 inside the container
--restart unless-stopped means it comes back automatically after a VM reboot

Step 8: Verify and Open in Browser

docker ps

Output:

CONTAINER ID   IMAGE                PORTS                    NAMES
83cb16a6cb44   static-site:latest   0.0.0.0:80->80/tcp       static-site

Then I opened http://4.234.163.212 in my browser.

The site loaded.

I cannot fully explain how good that felt. Three months into learning DevOps, and I just served a live website from inside a Docker container running on a cloud VM I spun up myself.

What I Learned

Cloud-init is a game changer. The idea that a VM can configure itself on boot, without any human touching it, is what separates manual setups from real infrastructure automation.

Containers are not complicated. A Dockerfile is just a recipe. Build the recipe, get an image. Run the image, get a container. That is it.

nginx:alpine is perfect for static sites. Small, fast, and it just works.

The --restart flag matters. In production, you want your containers to survive reboots. Always add --restart unless-stopped.

Full Project

GitHub: https://github.com/vivianokose/cloud-vm-docker-deploy

See you in the next one.