DEV Community: POTHURAJU JAYAKRISHNA YADAV

How a Linux Kernel Upgrade Broke MongoDB on an Azure VM — And How We Fixed It

POTHURAJU JAYAKRISHNA YADAV — Wed, 21 Oct 2026 23:55:04 +0000

It was a normal morning. I woke up, grabbed my coffee, and before I'd even properly sat down my phone was already lighting up — alerts, messages, the whole thing. The app was down. Users couldn't connect, services were throwing errors, and nobody had the faintest idea why.

The confusing part? Nobody had touched anything. No deploy the night before, no config change, no code push. I'd left everything green and healthy when I logged off, and I woke up to a dead server. That's the kind of outage that really gets under your skin, because there's nothing obvious to point at.

So I SSH'd into the VM, half expecting the whole box to be gone, and instead found MongoDB sitting there flat-out refusing to start. That was the beginning of a morning I hadn't planned for — and the culprit turned out to be something none of us had on our radar: Azure had quietly upgraded the kernel overnight while we were all asleep.

This one caught us completely off guard, which is exactly why I wanted to write it down. Here's the whole story, including the stuff we tried that didn't work, because honestly the dead ends are half the lesson.

The incident

It started like most outages do — out of nowhere. One of our apps running on an Azure VM just stopped responding. Users couldn't connect, services were down, and the first thing we did was SSH in and check whether anything was even alive.

sudo systemctl status mongod

MongoDB was in a failed state — Result: exit-code. And the weird part was how fast it was dying. The service would try to start and then fall over within milliseconds, somewhere around 35-48ms. That's not a database struggling to load data. That's something rejecting it before it even gets going.

Hunting for the cause

Logs first, obviously.

sudo cat /var/log/mongodb/mongod.log | tail -50

Useless. The log only showed a clean shutdown from two days ago. Nothing about why it was refusing to start now. So we went to the system journal instead and filtered for fatal messages:

sudo journalctl -u mongod --since "today" --no-pager | grep -i '"s":"F"'

And there it was, the line that made the whole thing click:

MongoDB cannot start: Linux kernel versions 6.19 and newer has a known 
incompatibility with this version of MongoDB. 
See https://jira.mongodb.org/browse/SERVER-121912 for more information.

A kernel incompatibility. Not corruption, not permissions, not a config mistake. The kernel.

What actually happened

Turns out Azure had auto-upgraded the Linux kernel on the VM overnight during a maintenance window. The box jumped to 7.0.0-1014-azure, which under the hood is based on Linux kernel 6.19+.

MongoDB has a known problem with kernel 6.19 and up — something to do with the rseq system call. And rather than risk running in a broken state, MongoDB just refuses to start at all when it detects that kernel. A hard safety block.

We confirmed what we were dealing with:

uname -r
# Output: 7.0.0-1014-azure

uname -a
# Linux 7.0.0-1014-azure ... x86_64

And the MongoDB version:

mongod --version
# db version v8.0.29

So: MongoDB 8.0.29 sitting on a kernel it explicitly refuses to run on. That's the entire bug in one sentence.

The stuff we tried that didn't work

Before we figured out the kernel was the issue, we burned time on the usual suspects. Leaving these in because this is the part where you learn what the problem isn't.

First we blamed permissions.

sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown -R mongodb:mongodb /var/log/mongodb

Nothing changed.

Then we figured it was a stale lock file. Classic MongoDB-won't-start cause.

sudo rm -f /var/lib/mongodb/mongod.lock
sudo rm -f /tmp/mongodb-27017.sock

Still nothing.

Then we thought maybe upgrading MongoDB would fix it. If it's a known issue, surely a newer version handles it, right?

sudo apt-get update
sudo apt-get install -y mongodb-org

MongoDB upgraded fine, straight to 8.0.32 — and failed with the exact same error. The kernel check is in 8.0.32 too. So no, upgrading doesn't help; the block is intentional.

Then we tried forcing it with --fork.

sudo mongod --config /etc/mongod.conf --fork

This one actually worked for a bit, but only with the old binary. Once we'd upgraded MongoDB, even --fork ran into the kernel check and died. So that was a dead end too, just a slower one.

At this point it was clear the problem wasn't anything on the MongoDB side at all. It was the kernel, full stop. Which meant the fix had to be at the kernel level.

The real fix: boot into the older kernel

Here's the thing that saved us — Azure doesn't throw away the old kernel when it upgrades. It leaves the previous ones installed on disk. The compatible kernel was still right there, just not the one being booted.

We listed what was actually installed:

dpkg --list | grep linux-image-.*-azure

rc  linux-image-6.17.0-1021-azure   # rc = removed, can't use
ii  linux-image-6.17.0-1022-azure   # ii = installed ✅
ii  linux-image-7.0.0-1014-azure    # ii = installed, broken ❌

6.17.0-1022-azure was fully installed (ii), and since 6.17 is below 6.19, it was compatible with MongoDB. We just needed to tell the server to boot into it instead of the new broken one.

A quick word on GRUB

If you've never had a reason to touch this, GRUB (the Grand Unified Bootloader) is the thing that runs the instant your server powers on, before the OS loads. It picks which kernel to boot. Think of it as the startup selector for your Linux box.

By default it grabs the newest kernel — which was exactly our problem, because the newest one was the broken 7.0.0-1014-azure.

The config lives at:

/etc/default/grub

Pointing GRUB at the good kernel

We edited the GRUB config to default to the older compatible kernel, and gave ourselves a boot menu timeout while we were at it:

sudo sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-1022-azure"/' /etc/default/grub
sudo sed -i 's/GRUB_TIMEOUT=0/GRUB_TIMEOUT=5/' /etc/default/grub

What those two changes actually do:

GRUB_DEFAULT=0 just boots whatever the first (newest) kernel is. Changing it to the full "Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-1022-azure" string tells GRUB to boot that specific kernel by name instead.
GRUB_TIMEOUT=0 means GRUB shows no menu and boots instantly. Bumping it to 5 gives you a five-second window to manually pick a kernel if you ever need to.

Then — and this part is important — you have to regenerate the actual boot config and reboot:

sudo update-grub
sudo reboot

Don't skip update-grub. Editing /etc/default/grub on its own does nothing. update-grub reads that file and regenerates the real boot config at /boot/grub/grub.cfg. If you forget it, you'll reboot straight back into the broken kernel and wonder why nothing changed.

After the reboot

Server came back up, first thing we checked was the kernel:

uname -r
# Output: 6.17.0-1022-azure ✅

Then we cleaned up the leftover junk from all our failed attempts:

sudo rm -f /var/run/mongodb/mongod.pid
sudo rm -f /tmp/mongodb-27017.sock
sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown -R mongodb:mongodb /var/log/mongodb

And started MongoDB:

sudo systemctl start mongod
sudo systemctl status mongod
# Active: active (running) ✅

Running. The app was back online immediately.

Making sure it doesn't happen again

Getting it running was great, but we weren't out of the woods. Azure could just upgrade the kernel again on the next maintenance window and put us right back where we started. So we pinned the kernel packages to stop that:

sudo apt-mark hold linux-azure
sudo apt-mark hold linux-image-azure
sudo apt-mark hold linux-image-7.0.0-1014-azure
sudo apt-mark hold linux-image-6.17.0-1022-azure

What apt-mark hold does is tell the package manager to never automatically upgrade or remove those packages. The important ones are linux-azure and linux-image-azure — those are meta-packages that always point at the latest kernel, so holding them stops any new kernel from sneaking in automatically. This keeps us parked on 6.17.0-1022-azure until MongoDB ships an official fix.

Double-check what's held with:

sudo apt-mark showhold

What I took away from this

A few things worth remembering:

Cloud providers — Azure here, but it's not just Azure — will upgrade your kernel during maintenance windows without asking. Something can break overnight with zero changes on your end.
MongoDB 8.x has a hard block against Linux kernel 6.19+ because of the rseq incompatibility (SERVER-121912). It won't start, and that's on purpose.
A Linux box can have several kernels installed at once. You just have to tell GRUB which one to use.
GRUB is the bootloader. Edit /etc/default/grub, run update-grub, reboot — that's how you control the default kernel.
apt-mark hold is genuinely useful when you need to freeze a package and keep auto-upgrades from undoing your fix.

Quick reference for next time

If this bites us (or you) again on any VM, here's the short version:

# 1. Confirm what's broken
sudo journalctl -u mongod --no-pager | grep '"s":"F"'

# 2. List installed kernels
dpkg --list | grep linux-image-.*-azure

# 3. Pick the highest kernel below 6.19 that shows "ii"

# 4. Set it as the default and reboot
sudo sed -i 's/GRUB_DEFAULT=.*/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux <your-kernel>"/' /etc/default/grub
sudo update-grub
sudo reboot

# 5. After reboot, start Mongo
sudo systemctl start mongod
sudo systemctl status mongod

# 6. Freeze the kernels
sudo apt-mark hold linux-azure linux-image-azure

Wrapping up

What looked like a mysterious outage was really just a kernel compatibility issue triggered by an automatic cloud update. No code changes, no reinstall, no data loss — the whole thing came down to understanding how Linux kernel management and GRUB fit together.

The real lesson for me: know what kernels are installed on your VMs, and remember your cloud provider can swap them out from under you whenever it feels like it. A quick dpkg --list | grep linux-image and a basic grasp of GRUB would've saved us a good chunk of the time we spent chasing permissions and lock files.

If you've been burned by an auto-upgrade like this, I'd like to hear how it showed up for you. These silent infrastructure changes are sneaky, and the more stories floating around, the faster the next person figures it out.

Stop Googling Command Syntax — Your Terminal Already Knows the Answer

POTHURAJU JAYAKRISHNA YADAV — Sun, 18 Oct 2026 08:55:52 +0000

I'll admit something a little embarrassing. For years, every time I forgot the exact flags for tar, or couldn't remember whether it was -C or -c to change directory, my reflex was to alt-tab to a browser and search for it. Same with Kubernetes — I'd go hunting through the docs website every time I needed to remember what fields go into a Pod spec.

At some point it clicked that this was ridiculous. The machine right in front of me already knows all of this. It ships with the answers built in. I just wasn't asking it.

So this post is about the handful of tools that killed my habit of Googling command syntax: tldr, apropos, and info for everyday Linux commands, and kubectl explain for writing Kubernetes manifests without guessing. None of these are new or fancy. They're just underused, and once they're part of your muscle memory you stop breaking your flow to go look things up.

The problem with `man`

Everyone knows man. You type man tar and you get... the entire manual. Every flag, every edge case, every historical note, all in one dense scroll. It's thorough, and that's exactly the problem. When I've forgotten how to create a gzipped archive, I don't want a 600-line reference. I want the one line that does the thing.

man is a reference manual. Most of the time what you actually need is an example or a reminder. That's where the other tools come in.

`tldr` — examples, not essays

tldr is the one I reach for most. The name is literally "too long; didn't read," and that's the whole philosophy. Instead of the full manual, it gives you a short list of the most common real-world uses of a command, with copy-pasteable examples.

Here's tldr tar on my machine:

tar
Archiving utility. Often combined with a compression method, such as gzip or bzip2.

 - [c]reate an archive and write it to a [f]ile:
   tar cf path/to/target.tar path/to/file1 path/to/file2 ...

 - [c]reate a g[z]ipped archive and write it to a [f]ile:
   tar czf path/to/target.tar.gz path/to/file1 path/to/file2 ...

 - E[x]tract a (compressed) archive [f]ile into the current directory [v]erbosely:
   tar xvf path/to/source.tar.ext

 - E[x]tract a (compressed) archive [f]ile into the target directory:
   tar xf path/to/source.tar.ext -C path/to/directory

 - Lis[t] the contents of a tar [f]ile [v]erbosely:
   tar tvf path/to/source.tar

Look at how readable that is. See the little brackets — [c]reate, g[z]ipped, [f]ile? That's a genuinely clever touch. The letters in brackets map directly to the flags, so czf suddenly makes sense: create, g*zipped, **f*ile. You stop memorizing flags and start understanding them.

That tar example alone covers maybe 95% of what anyone ever does with tar. No scrolling, no essay, just the thing you came for.

Getting it

It's not always installed by default, but it's one package away:

# Debian / Ubuntu
sudo apt install tldr

# macOS
brew install tldr

# Via npm (the original client)
npm install -g tldr

First run might fetch the pages, and you can refresh the local cache whenever you want:

tldr --update

Use it for literally anything you half-remember — tldr ssh, tldr find, tldr git rebase, tldr ffmpeg. The ffmpeg one in particular has saved me more times than I can count, because nobody on earth remembers ffmpeg syntax.

`apropos` — when you don't even know the command's name

tldr is great when you know which command you want. But sometimes you don't. You know what you want to do, you just can't remember what it's called.

That's apropos. It searches the names and descriptions of all the man pages on your system for a keyword and shows you everything that matches. It's basically "search the manual by topic."

First, a gotcha that got me

The very first time I tried this, I got nothing but errors:

$ apropos compress
compress: nothing appropriate.

If that happens to you, don't panic — apropos isn't broken and the commands aren't missing. It relies on a prebuilt search index, and on a fresh or minimal install (this bites people on WSL and slim container images especially) that index just doesn't exist yet. Build it once:

sudo mandb

You'll see it churn through your man pages:

Processing manual pages under /usr/local/man...
7612 manual pages were added.

Now apropos has something to search, and it works.

Now it actually does something

With the index built, apropos compress returns a wall of genuinely useful results. Here's a trimmed slice of what came back on my machine:

bunzip2 (1)     - a block-sorting file compressor, v1.0.8
bzip2 (1)       - a block-sorting file compressor, v1.0.8
gunzip (1)      - compress or expand files
gzip (1)        - compress or expand files
lz4 (1)         - lz4, unlz4, lz4cat - Compress or decompress .lz4 files
lzma (1)        - Compress or decompress .xz and .lzma files
pigz (1)        - compress or expand files
unzip (1)       - list, test and extract compressed files in a ZIP archive
xz (1)          - Compress or decompress .xz and .lzma files
zcat (1)        - compress or expand files

You went in knowing only the word "compress" and came out with every compression tool on the box. From there, tldr gzip or man xz gets you the actual usage.

The real output is actually much longer than that — apropos matches the keyword anywhere in the description, so you also get things like zgrep, zless, library functions, and perl modules. Which brings us to the useful trick for cutting the noise.

Narrowing it down

Man pages are split into numbered sections, and section 1 is user commands. So if you only care about actual commands you can run, scope the search to section 1:

apropos -s 1 copy

cp (1)                  - copy files and directories
cpio (1)                - copy files to and from archives
dd (1)                  - convert and copy a file
docker-cp (1)           - Copy files/folders between a container and the local filesystem
install (1)             - copy files and set attributes
objcopy (1)             - copy and translate object files

That's much cleaner than the unfiltered version, which also drags in library calls and config file formats you don't care about when you're just trying to copy a file.

Think of the pairing like this: apropos finds the name of the command, tldr tells you how to use it. Together they cover the full "I forgot" spectrum.

`info` — the deep dive when you actually need it

Sometimes you do need the full story — not an example, not a one-liner, but the real, complete documentation with all the context. That's where info comes in.

info reads GNU Info documents, which are often far more detailed and better structured than the equivalent man page. For GNU tools especially (coreutils, gcc, make, tar, grep), the info manual is the authoritative, book-length version with chapters, sections, and cross-links you can navigate like a hyperlinked document.

info coreutils
info tar
info grep

A few navigation basics, because info trips people up the first time:

Arrow keys scroll.
n goes to the next node (section), p to the previous one.
u goes up a level.
Enter on a highlighted link follows it.
q quits.

Honestly I don't live in info the way I live in tldr. But when I genuinely need to understand why a tool behaves a certain way, or I'm chasing some obscure option, the info manual is usually where the real explanation lives.

So the mental model for the three is:

tldr — "just remind me how to do the common thing."
apropos — "I forgot what the command is even called."
info — "I need to actually understand this properly."

Now the Kubernetes version of the same problem: `kubectl explain`

Here's where this gets really useful, because the exact same frustration shows up when writing Kubernetes manifests. You're staring at a blank YAML file trying to remember: is it containers or container? Does image go under the pod spec or inside the container? What fields does a probe even take?

Most people's instinct is to go to the Kubernetes docs website. But kubectl has the entire API schema baked in, and kubectl explain is the self-documentation tool that mirrors exactly what tldr/man do for shell commands — except here it's generating the structure of your manifest for you.

The best part: it reads the schema straight from your cluster, so it matches your Kubernetes version. No guessing whether a field exists in your version or not.

Start at the top

kubectl explain pod

KIND:       Pod
VERSION:    v1

DESCRIPTION:
    Pod is a collection of containers that can run on a host.

FIELDS:
  apiVersion    <string>
  kind  <string>
  metadata  <ObjectMeta>
  spec  <PodSpec>
  status    <PodStatus>

Right away this tells you the four top-level keys of any Pod manifest: apiVersion, kind, metadata, and spec. That's your skeleton:

apiVersion: v1
kind: Pod
metadata:
spec:

Drill down with dot notation

Now, what goes inside spec? Just ask:

kubectl explain pod.spec

KIND:       Pod
VERSION:    v1

FIELD: spec <PodSpec>

DESCRIPTION:
    Specification of the desired behavior of the pod.

FIELDS:
  activeDeadlineSeconds <integer>
  affinity  <Affinity>
  automountServiceAccountToken  <boolean>
  containers    <[]Container> -required-
    List of containers belonging to the pod. There must be at least one
    container in a Pod.
  dnsPolicy <string>
  enum: ClusterFirst, ClusterFirstWithHostNet, Default, None
  ...

This output is doing a lot of work for you. Look at the details:

containers <[]Container> -required- — the [] means it's a list, and -required- means you can't skip it. So you know containers is a YAML list and it's mandatory.
dnsPolicy <string> even shows you the valid enum values, so you don't have to guess what strings are allowed.
Each field shows its type, which tells you how to shape the YAML — a <string> is a plain value, an <Affinity> is a nested object, a <[]Container> is a list of objects.

Keep going until you hit plain values

containers is a []Container, so let's see what a container looks like:

kubectl explain pod.spec.containers

KIND:       Pod
VERSION:    v1

FIELD: containers <[]Container>

DESCRIPTION:
    A single application container that you want to run within a pod.

FIELDS:
  args  <[]string>
  command   <[]string>
  env   <[]EnvVar>
  image <string>
  imagePullPolicy   <string>
  name  <string> -required-
  ports <[]ContainerPort>
  resources <ResourceRequirements>
  volumeMounts  <[]VolumeMount>
  ...

Now you can see that a container needs a name (required), takes an image, and that ports, env, and volumeMounts are all lists of their own object types. If you want to know what goes in a port, you just keep drilling:

kubectl explain pod.spec.containers.ports

You follow the types down the tree until everything bottoms out in <string>, <integer>, or <boolean> — plain scalar values you can actually type. That's your signal that you've reached the leaves and the manifest is complete.

Putting it together into a real manifest

Walking the schema with kubectl explain — pod → pod.spec → pod.spec.containers → pod.spec.containers.ports — gives you everything you need to assemble this by hand, confidently, without the docs website:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:                 # []Container, required
    - name: nginx             # string, required
      image: nginx:1.27       # string
      ports:                  # []ContainerPort
        - containerPort: 80   # integer

Every single field in that file maps back to something kubectl explain told us: containers is a required list, each item needs a name, image is a string, ports is a list, and containerPort is an integer. Nothing memorized, nothing guessed.

A couple of handy variations

Dump the entire nested tree in one shot instead of drilling level by level:

kubectl explain pod --recursive

That's a firehose, but it's great when you want to see the full shape of a resource at a glance.

And this works for every resource type, not just pods:

kubectl explain deployment.spec.template.spec.containers
kubectl explain service.spec
kubectl explain ingress.spec.rules
kubectl explain configmap

The deployment.spec.template.spec path is the one I use constantly, because that's where the pod template lives inside a Deployment and it's easy to lose track of how deep it is.

Why this matters more than it looks

None of these tools are impressive on their own. But together they change a habit, and the habit is what matters.

When you Google command syntax, three things happen: you break your focus by leaving the terminal, you risk landing on a Stack Overflow answer from eight years ago that's subtly wrong for your version, and you never actually learn the tool — you just copy and move on. When you ask your own machine instead, the answer is version-accurate (especially with kubectl explain, which reads your real cluster schema), it's instant, and you stay in flow.

The bracket notation in tldr and the type annotations in kubectl explain also quietly teach you why the syntax is the way it is, so over time you need them less. That's the real win. The goal isn't to depend on these forever — it's to stop context-switching to a browser for things your terminal can answer in half a second.

Quick reference

# Forgot how to use a command you know the name of
tldr <command>

# Forgot the command's name, know what it does
apropos <keyword>
sudo mandb              # run once if apropos finds nothing

# Need the full, detailed GNU manual
info <command>

# Building a Kubernetes manifest, forgot the fields
kubectl explain <resource>
kubectl explain <resource>.spec
kubectl explain <resource>.spec.<field>
kubectl explain <resource> --recursive

Wrapping up

I stopped Googling tar flags and Pod spec fields the day I realized the answers were already sitting on my machine, offline, version-correct, one command away. tldr for quick examples, apropos to find a command you've forgotten the name of, info for the deep dive, and kubectl explain to build manifests field by field straight from the API schema.

Next time you catch yourself reaching for the browser to look up syntax, try asking your terminal first. It knows. It's been waiting the whole time.

If you've got a self-documentation trick I didn't mention here, drop it in the comments — I'm always happy to retire another browser tab.

Building a SIP Calling Lab on AWS: Talking Between My Phone and Laptop Through the Cloud

POTHURAJU JAYAKRISHNA YADAV — Thu, 15 Oct 2026 04:38:24 +0000

I wanted to actually understand SIP, not just read about it. So I set myself a hands-on goal: run my own SIP server in the cloud and make a real voice call between my phone and my laptop, with the audio flowing through that server. This post is the full story, including the dead ends I hit, because the mistakes taught me more than the happy path did.

What you'll have at the end: one small AWS EC2 instance running Kamailio (the SIP server) and rtpengine (the audio relay), two softphones registered to it, and a working two-way call.

What SIP actually is (the mental model I needed)

Before touching anything, here's the model that made it click for me.

SIP (Session Initiation Protocol) is just the "who's calling whom and are they picking up" part of a call. It's signaling. It does not carry your voice. Think of it like the ringing and the "hello?" negotiation.

The actual audio travels over a separate protocol called RTP on a different set of ports. This split is the single most important thing to understand, because it's the source of 90% of the problems people hit:

SIP → port 5060, sets up and tears down the call
RTP → a range of ports (I used 10000–20000), carries the voice

There are two roles in the system:

A server (the registrar/proxy) that knows where everyone is and routes calls. I used Kamailio.
Clients (softphones) that have a mic and speaker. These live on devices with audio hardware — my phone and my laptop. Not on the server. A cloud server has no microphone, so a softphone running on it couldn't capture my voice. That realization saved me from a wrong turn early on.

   My phone                   EC2 (one instance)              My laptop
 [softphone 1001] <--SIP--> [Kamailio + rtpengine] <--SIP--> [softphone 1002]
      mic/spk     <------ RTP audio relayed through EC2 ------>   mic/spk

What is what: every piece in plain English

Before the step-by-step, let me explain each piece the way I wish someone had explained it to me. The big picture first: I built a tiny phone company in the cloud. When you make a normal call, a phone carrier's servers connect you to the other person. This lab is my own miniature version of that carrier, running on a single AWS server, connecting my phone to my laptop.

The two halves of every call

This is the one concept everything else hangs on. A phone call is really two separate conversations happening at the same time:

Signaling — "the arranging." Who's calling, who's being called, is it ringing, did they answer, did they hang up. It's the paperwork of the call. The protocol is SIP, and it rides on port 5060.
Media — "the actual voice." The real sound of your voice streaming to the other person. The protocol is RTP, and it rides on a range of ports (I used 10000–20000).

Why split them? Because they're different jobs. Signaling is a few small text messages ("ring this person"). Media is a continuous stream of audio. Keep this one line in your head and most problems make sense: SIP sets up the call, RTP carries the voice. The infamous "it rings but I hear nothing" bug is almost always RTP ports not being open — because people forget the voice travels separately from the signaling.

Kamailio — the switchboard operator

Kamailio is the SIP server, and its whole job is signaling. It does two things:

Registrar: it keeps a live list of who's online and where. When my phone says "Hi, I'm 1001, here's my address," Kamailio writes it down. That's registration — the green checkmark on the phone.
Proxy: when 1001 calls 1002, Kamailio looks up where 1002 is and forwards the call. It's the operator patching two lines together.

Crucially, Kamailio never touches your voice. It only handles the "who's calling whom" messages.

rtpengine — the audio relay in the middle

rtpengine handles the media (RTP) — the actual voice. Why do you need it on a cloud server? Because my phone and laptop are on different networks, both behind NAT (home router, mobile carrier). They can't send audio straight to each other because neither can see the other's real address. So rtpengine sits in the middle on the public server and relays the audio: phone → rtpengine → laptop, and back. Both sides only ever have to reach the server, which has a public address everyone can see.

The database (DBTEXT) — the user list

This stores the accounts: 1001 and its password, 1002 and its password. When a phone tries to register, Kamailio checks this list to confirm the password. I used DBTEXT, which is just plain text files on disk — the simplest option, no separate database server to run.

The softphones — the actual phones

A softphone is a phone app with a mic and speaker. I used Zoiper on my phone (extension 1001) and MicroSIP on my laptop (extension 1002). They register to Kamailio and place the calls. This is exactly why they live on my devices and not on the server — the server has no microphone.

The AWS pieces

EC2 instance — just a rented Linux computer in Amazon's datacenter. It runs Kamailio + rtpengine. Nothing magic, a normal Ubuntu box I reach over SSH.
Public IP vs Private IP — the instance has two addresses: a private one (172.31.x) it knows itself by inside AWS, and a public one (52.91.x) the outside world uses to reach it. This split is the root of the NAT problem: the server thinks it's 172.31.x, but phones must talk to 52.91.x. If the server naively tells a phone "send audio to 172.31.x," the phone fails because that address is meaningless on the internet. That's why I had to explicitly tell both Kamailio and rtpengine to advertise the public IP.
Security Group — AWS blocks all incoming traffic by default. The Security Group is where I poked holes for the ports I needed: SIP (5060) and the RTP range. Forget the RTP ports and calls connect but have no sound — the classic trap.

How one call actually flows

Putting it together, here's what happens when I dial 1002 from 1001:

Both phones register. Zoiper (1001) and MicroSIP (1002) each tell Kamailio "I'm online, here's my address." Kamailio checks their passwords against the user list and records where they are.
I dial. Phone 1001 sends an INVITE ("I want to call 1002") to Kamailio.
Kamailio routes it. It looks up where 1002 is, forwards the INVITE, and tells rtpengine to set up an audio relay for this call.
The laptop rings (180 Ringing).
I answer. MicroSIP sends 200 OK — the call is connected.
Voice flows. RTP audio streams phone → rtpengine → laptop and back. We're talking.
I hang up. One side sends BYE, the other replies 200 OK, and rtpengine tears down the relay.

That exact sequence is what shows up in the Kamailio log later — the INVITE ... 200 OK and BYE ... 200 OK lines are steps 2–7 happening for real.

One-line cheat sheet

Piece	Plain-English job
SIP	the language for arranging calls (ring, answer, hang up)
RTP	the stream that carries your actual voice
Kamailio	the switchboard — tracks who's online, connects callers
rtpengine	the audio relay in the middle; fixes the NAT problem
DBTEXT	the file holding users and passwords
Softphone (Zoiper / MicroSIP)	the phone app with a mic and speaker
EC2	the rented cloud computer running it all
Public / Private IP	two addresses for one server; the NAT headache
Security Group	the firewall; I opened the SIP + RTP ports

Why I moved from WSL to AWS

My first attempt was on my Windows laptop using WSL (Ubuntu). It technically works, but WSL2 puts Linux behind a NAT'd virtual network with a 172.x address, so reaching the SIP server from another device meant fighting with mirrored networking and port forwarding. On top of that, Ubuntu 20.04's Kamailio control tool had an old Python 2 vs 3 bug that broke user creation.

A disposable EC2 instance sidestepped all of it: a clean public IP, no WSL networking quirks, and a newer Ubuntu with a working toolchain. And when I'm done, I just terminate it.

Lesson learned: for anything involving inbound network traffic, a cloud VM with a real public IP is far less fiddly than WSL.

Step 1 — Launch the EC2 instance

I launched a small Ubuntu 24.04 instance (a t3a.medium is plenty for a lab).

Public IP (what the softphones connect to): 52.91.109.231
Private IP (what the instance sees on itself): 172.31.42.238

That public/private split is the NAT situation I'd have to handle later. Keep both numbers handy; everything references them.

Here's my instance in the EC2 console. Note the public IPv4 address — that's the one every softphone will connect to.

Step 2 — Open the ports (AWS Security Group)

This is where cloud SIP most often "silently fails." The firewall is the Security Group, and SIP needs more than one port family open. My inbound rules:

Type	Protocol	Port range	Source	Why
All UDP	UDP	0–65535	0.0.0.0/0	covers SIP 5060 and the RTP media range in one rule
Custom TCP	TCP	5060	0.0.0.0/0	SIP fallback if a network blocks UDP
SSH	TCP	22	my IP	so I can log in

The gotcha: if you only open 5060 and forget the RTP range, calls will connect and ring, but you'll hear silence. Opening All UDP is the lazy-but-reliable lab choice.

Security note: All UDP open to the whole internet means SIP scanners will start probing within minutes. Fine for a short-lived lab — just terminate the instance when done. Don't leave it running for days.

My inbound rules. The All-UDP rule is the important one — it covers both SIP signaling (5060) and the RTP media ports in a single line, which is what lets the audio through.

Step 3 — Install Kamailio and rtpengine

sudo apt update
sudo apt install -y kamailio kamailio-extra-modules sqlite3
sudo apt install -y rtpengine

Ubuntu 24.04 gave me Kamailio 5.7.4 and rtpengine 11.5.1 straight from the repos — no building from source. rtpengine is the piece that makes audio work across NAT: it relays the RTP stream and rewrites the media addresses so each side sends voice to a reachable IP.

Confirming both packages installed cleanly. Ubuntu 24.04 ships recent-enough versions that I didn't have to compile anything.

Step 4 — Configure rtpengine (the NAT fix)

The whole audio problem comes down to one line in /etc/rtpengine/rtpengine.conf:

interface = 172.31.42.238!52.91.109.231
port-min = 10000
port-max = 20000

That interface = PRIVATE!PUBLIC syntax is the magic. rtpengine binds on the private IP the OS actually has (172.31.x), but advertises the public IP (52.91.x) in the call setup, so remote phones know where to send audio. Without this, you get one-way or no audio every time.

Then:

sudo systemctl restart rtpengine-daemon
sudo systemctl status rtpengine-daemon

rtpengine up and running. "active (running)" is what you want to see here — if it's not, the audio relay is dead and calls will be silent.

Step 5 — Point the control tool at the database

I used DBTEXT, a flat-file database. Zero extra services to run, which is perfect for a lab. In /etc/kamailio/kamctlrc:

SIP_DOMAIN=52.91.109.231
DBENGINE=DBTEXT
DB_PATH="/etc/kamailio/dbtext"

I used the public IP as the SIP domain so the softphones don't need any DNS — they just point at the IP.

Step 6 — Turn on the features in kamailio.cfg

Kamailio's default config is modular; you switch features on with #!define flags at the very top of /etc/kamailio/kamailio.cfg:

#!define WITH_AUTH
#!define WITH_USRLOCDB
#!define WITH_NAT
#!define WITH_RTPENGINE
#!define DBURL "text:///etc/kamailio/dbtext"

WITH_AUTH — require a password to register
WITH_USRLOCDB — remember where users registered
WITH_NAT + WITH_RTPENGINE — enable NAT handling and route media through rtpengine
DBURL — use the flat-file DB

I also added one listen line so Kamailio advertises its public IP in SIP headers (otherwise phones try to reply to the unreachable private IP):

listen=udp:172.31.42.238:5060 advertise 52.91.109.231:5060

The very top of my kamailio.cfg. These five define lines are the switches that turn on auth, NAT handling, and the rtpengine integration. They must appear exactly once — more on that mistake below.

Step 7 — Create the two users

sudo kamdbctl create          # y to presence tables, n to extra tables
sudo kamctl add 1001 test1001
sudo kamctl add 1002 test1002
sudo kamctl db show subscriber

1001 is my phone, 1002 is my laptop. The db show listed both with hashed passwords, so I knew they were stored.

Both users stored in the database. You can see 1001 and 1002, each with a hashed password — Kamailio never keeps the plaintext.

The three mistakes I had to fix (this is where I learned the most)

Here's the honest part. Kamailio refused to start three separate times, and each failure taught me something.

Mistake 1 — Duplicate defines

I'd added the #!define lines in the editor, then re-added them with a command. Result:

CRITICAL: pp_define(): already defined: WITH_AUTH

Kamailio rejects a flag defined twice. I deleted the duplicate block and kept exactly one set.

Takeaway: kamailio -c -f /etc/kamailio/kamailio.cfg checks syntax before you start the service. Use it every time.

Mistake 2 — The database module wasn't loaded

Config parsed fine, but the service still died. Running it in the foreground (sudo kamailio -f ... -D -E) finally showed the real error:

ERROR: db_bind_mod(): Module db_text not found. Missing loadmodule?

I'd told Kamailio to use DBTEXT but never loaded the db_text module. The default config only loads the MySQL one. Fix:

sudo sed -i '/^loadmodule "jsonrpcs.so"/i loadmodule "db_text.so"' /etc/kamailio/kamailio.cfg

Takeaway: systemctl status hides the real error. Running the daemon in the foreground shows exactly why it won't start. That one trick unblocked me twice.

Mistake 3 — File permissions

The phone was now reaching the server (I could see REGISTER requests in the log — huge milestone), but every one failed with:

db_text: failed to open file [/etc/kamailio/dbtext/subscriber]
auth_db: get_ha1(): failed to query database

The subscriber file was owned by root with -rw------- permissions, but the Kamailio service runs as the kamailio user, which couldn't read it. kamctl had created it as root with locked-down perms because it holds password hashes. Fix:

sudo chown -R kamailio:kamailio /etc/kamailio/dbtext
sudo chmod 640 /etc/kamailio/dbtext/subscriber
sudo systemctl restart kamailio

Takeaway: "it works when I run it with sudo but not as a service" almost always means a permission or user mismatch.

Step 8 — Register the softphones

Phone (1001): I used Zoiper (Linphone works too). On the account setup screen that asks about an authentication username / outbound proxy, I left it unchecked and tapped Skip — my auth username is the same as my SIP username. The account settings:

Field	Value
Username	`1001`
Password	`test1001`
Domain / host	`52.91.109.231`
Transport	UDP

A green checkmark and "Account is ready" meant it had registered to the cloud server.

The moment it clicked: the green check and "Account is ready" mean my phone successfully registered to the server running in AWS.

Laptop (1002): I installed MicroSIP on Windows (lightweight, simple) and added an account with the same shape but username 1002 / password test1002. It showed "Online" at the bottom.

The laptop side, registered as 1002. "Online" here is the Windows equivalent of the phone's green check — both endpoints are now live on the server.

Step 9 — Make the call

I kept the live log running on the server the whole time:

sudo journalctl -u kamailio -f

Then I dialed 1002 from my phone. The laptop rang, I answered, and the log showed the money line:

INVITE ... code=200;reason=OK;src_user=1001;...;dst_user=1002
BYE    ... code=200;reason=OK ...

A 200 OK on the INVITE means the call connected end to end. The BYE with 200 OK means it hung up cleanly. Two-way audio, through my own server in the cloud.

Tip: if the phone and laptop are in the same room, use headphones on at least one of them or you'll get awful echo/feedback as each picks up the other's speaker.

The call connected on the phone — live and counting.

And the same call on the laptop side in MicroSIP. Both ends are talking through the cloud server.

The proof in the server log: the INVITE got a 200 OK (call answered) and the BYE got a 200 OK (clean hang-up). This is exactly what a successful call looks like from Kamailio's side.

Reading the SIP call flow

Once it worked, the log became a great teacher. A successful call looks like this:

INVITE — caller asks to start a call
100 Trying — server acknowledges it's working on it
180 Ringing — the other phone is ringing
200 OK — the callee answered
ACK — caller confirms, media starts flowing via rtpengine
BYE / 200 OK — someone hangs up

I also saw a few 486 Busy Here responses — those were just calls I declined or didn't pick up. Not errors, just normal call outcomes.

What I'd do differently / next steps

Try SIP over TCP or TLS. Some networks block UDP 5060; TCP got through reliably, and TLS would encrypt the signaling.
Tighten the Security Group. All-UDP-open is fine for an afternoon, not for anything persistent.
Add a third user and try a group scenario, or point a desk phone at it.
Watch the raw packets with ngrep -d any port 5060 or sngrep to see the actual SIP messages, not just the accounting log.

The whole thing in one glance

Piece	What it does	Where it runs
Kamailio	SIP registrar + proxy (signaling)	EC2
rtpengine	relays the voice (RTP), fixes NAT	EC2
DBTEXT	stores users 1001 / 1002	EC2 (flat files)
Zoiper (1001)	softphone with a mic	my phone
MicroSIP (1002)	softphone with a mic	my laptop

One cloud server, two phones, a real call. The hard parts weren't the concepts — they were the duplicate config line, the unloaded module, and the file permission. Each one failed loudly, and reading the error (especially in the foreground) was always the way out.

Don't forget: when you're finished, stop or terminate the EC2 instance so it isn't billing or sitting open to the internet.

Learning Ansible Without Opening the Docs Website

POTHURAJU JAYAKRISHNA YADAV — Wed, 14 Oct 2026 09:58:41 +0000

How I used ansible-doc, local collection files, and a few terminal commands to
understand Ansible instead of blindly copying YAML.

I've been poking at Ansible lately, and somewhere along the way I made a small
decision that changed how I learn it: I stopped reaching for the browser.

Every time I didn't know how something worked, my reflex was to Google it, land on
the official docs, copy a snippet, move on. It worked, but I never really kept
anything. So this time I tried something different. I wanted to see how far I could
get using only what's already sitting on my machine. Turns out, pretty far.

Here's the trail I followed.

It started with a broken command

I ran a simple scaffolding command and got slapped with this:

ERROR! Unexpected Exception, this is probably a bug: module 'lib' has no attribute 'SSL_load_error_strings'

Not exactly welcoming. The traceback pointed at pyOpenSSL calling into
cryptography, so I checked what versions were actually installed — making sure I
queried the same Python that runs Ansible:

python3 -m pip show pyOpenSSL cryptography

The version check revealed a compatibility problem: an older pyOpenSSL was running
alongside a much newer cryptography package. The older library expected an OpenSSL
binding that was no longer available in that combination. This kind of drift is easy
to end up with when system packages and pip installs get mixed together.

Rather than modifying system-managed Python packages, I created a virtual
environment and installed Ansible in isolation:

python3 -m venv ~/.venvs/ansible
source ~/.venvs/ansible/bin/activate
pip install --upgrade pip ansible

This gave me a cleaner, more predictable environment and avoided changing
dependencies the operating system relies on. One ansible --version later and I had
a modern ansible-core with everything matched up. Lesson one, re-learned the hard
way: when your Python environment is a mess, don't patch it, isolate it.

The command that does most of the work

The thing I kept coming back to is ansible-doc. It ships with Ansible, works
completely offline, and documents basically everything.

Want to know what you can do? List it all:

ansible-doc -l

On my setup that printed thousands of available plugins and modules, because I'd
installed the full community bundle — the exact number depends on which Ansible
packages and collections you have installed. Searching through them is just grep:

ansible-doc -l | grep -i copy

And to actually learn a module, you read its page:

ansible-doc copy

The part I now rely on most is the EXAMPLES section that many module pages carry
near the bottom. That's where you see real, working YAML you can adapt. No website
needed. For a quick skeleton of just the options, there's a snippet mode too:

ansible-doc -s copy

The confusion that finally made things click

Here's where I got stuck for a bit, and honestly it was the most useful stumble of
the whole thing.

I tried to read up on loop:

ansible-doc loop

[WARNING]: loop was not found

Not found? I use loops all the time. Then I tried tasks, when, same thing.
Nothing.

The penny dropped when I realised Ansible has a few distinct pieces, and I'd been
treating them as one. The mental model that helped me was simple:

Modules perform actions, such as copying files, installing packages, or managing services. These are what ansible-doc <name> looks up.
Module arguments specify what the module should do, such as the package name or the desired state.
Keywords control how Ansible organises and executes those actions, such as when, loop, become, and register. These live under a different flag.

So keywords need this:

ansible-doc -t keyword loop
ansible-doc -t keyword when

And suddenly it told me exactly what I wanted:

loop:
  applies_to: [Task, Handler]
  description: Takes a list for the task to iterate over, saving each element into
    the item variable
  type: list

Once I understood those three pieces — module, arguments, keywords — reading an
Ansible task became much easier. A task is a module invocation, with its arguments,
optionally wrapped in task-level keywords. Plays add their own play-level keywords on
top of that:

tasks:                          # play-level keyword
  - name: Install nginx         # task-level keyword
    ansible.builtin.package:    # module
      name: nginx               # module argument
      state: present            # module argument
    when: ansible_os_family == "Debian"   # task-level keyword

Every line traces back to one of those categories. Once I saw that, I didn't feel
like I was guessing anymore.

A rough edge, and how I got past it

A couple of times ansible-doc just fell over on me:

ERROR! ... CLI.pager_pipe() takes 1 positional argument but 2 were given

It was intermittent, which made it annoying to pin down. The fix that worked for me
was to force a plain pager for that invocation:

PAGER=cat ansible-doc -s copy

That disables interactive paging for the command. I now just prefix everything with
PAGER=cat — it does nothing when it isn't needed and saves me when it is. If the
error shows up for you, it's worth checking your installed Ansible version and
whether it reproduces with your default pager, since the behaviour can depend on
both. If you'd rather not type it every time, bury it in an alias:

alias adoc='PAGER=cat ansible-doc'

Keywords don't have examples, so I went looking for them

One frustration with keywords: their doc pages describe what they do but never show
them in use. There's no EXAMPLES section for when or loop.

But the collections I'd installed came with lots of real YAML files bundled right on
disk. So instead of searching the internet for "ansible when example", I searched my
own machine.

First, find where the collections actually live (the path varies by machine and
Python version, so don't hardcode it):

ansible-galaxy collection list

That prints the collection paths. Then grep the relevant directory for the keyword
you want to see used — searching the whole tree, not just an examples/ folder,
since not every collection keeps one:

grep -rl "when:" <collections-path> --include="*.yml"

That surfaced real examples and reusable patterns from the installed collections.
Worth knowing: these files can also include tests, fixtures, and other development
content, so not everything you find is a polished playbook. But plenty of it is
genuinely useful to read. Here's the kind of thing that came back:

- name: Add command completion
  ansible.builtin.include_tasks: command_completion.yml
  when: gcloud_command_completion

Reading other people's real tasks, right there offline, taught me more about idiom
and style than any tutorial had.

Putting it together in one small playbook

All of this only clicked once I assembled the pieces into something that runs. Here's
a tiny playbook that installs nginx and makes sure it's running:

---
- name: Install and start nginx
  hosts: webservers
  become: true

  tasks:
    - name: Install nginx
      ansible.builtin.package:
        name: nginx
        state: present

    - name: Ensure nginx is running
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

Writing it, I leaned entirely on the offline docs to get the module arguments right:

ansible-doc ansible.builtin.package
ansible-doc ansible.builtin.service

And before running anything, I checked the file:

ansible-playbook site.yml --syntax-check

The one thing that playbook assumes is an inventory. hosts: webservers only means
something if a group called webservers exists, so I wrote a tiny inventory file to
go with it:

# inventory.ini
[webservers]
192.168.1.10

I didn't have to guess the format, either — the INI inventory plugin documents
itself, examples included:

ansible-doc -t inventory ansible.builtin.ini

Then I pointed the playbook at it:

ansible-playbook -i inventory.ini site.yml --syntax-check

A couple of honest caveats here. --syntax-check only validates the playbook's
structure — it doesn't confirm the hosts are reachable or that the tasks will
actually succeed. It just catches the silly mistakes before you waste a real run. If
you want to see how Ansible reads your inventory back, ansible-inventory -i inventory.ini --graph prints the groups and hosts as a tree, which is a nice offline
sanity check before you run anything.

Where I've landed

After all this, my loop for learning any new piece of Ansible is pretty tight:

Find the module: ansible-doc -l | grep -i <topic>
Read its arguments and examples: ansible-doc <module>
Look up any structure keywords I need: ansible-doc -t keyword <word>
Borrow patterns from the bundled collection files with a quick grep
Check my own file before running it: ansible-playbook file.yml --syntax-check

The whole thing happens in a terminal, nothing opens in a browser, and because I'm
reading and assembling instead of copy-pasting, it actually sticks.

I think that's the real takeaway for me. The docs were never the thing holding me
back. Slowing down enough to understand why a command is shaped the way it is, and
using the tools already in front of me, is what finally made Ansible feel less like
magic and more like something I can just write.

And the method isn't really about Ansible. Investigating the error instead of
pasting it into a search box, inspecting the references already on disk, reading real
examples, validating my own work — that same loop works just as well for Terraform,
Docker, or Kubernetes.

Still a lot to explore.

Learn RAID the Hard Way: Building 0/1/5/6/10 by Hand on EC2

POTHURAJU JAYAKRISHNA YADAV — Sat, 10 Oct 2026 05:10:54 +0000

I wanted to actually understand RAID instead of memorizing a comparison table. So I launched an Ubuntu 24.04 EC2 instance, attached a few spare EBS volumes, and built every common RAID level by hand with mdadm: RAID 0, 1, 5, 6, and 10. For each one I created the array, wrote a file, deliberately failed a disk to watch what happens, then rebuilt it.

This article is the full walkthrough, in plain language, and it has three parts:

The concepts — what RAID and an "array" actually are, and the storage vocabulary around them.
The hands-on build — every level with the real commands and real output, including the warnings you'll hit.
Production reality — the errors you face on live systems and exactly how to handle each one.

Because this runs entirely on AWS, I've kept EC2 and EBS specifics inline where they matter. If you've ever been confused about why RAID 5 and RAID 10 give different usable space, or frozen at the word "degraded" in a 2 AM alert, this is for you.

1. The concepts

What is RAID, in one minute?

RAID stands for Redundant Array of Independent Disks. You combine several physical disks into one logical disk. Depending on how you combine them, you trade between three things:

Speed — spread data across disks so reads and writes happen in parallel.
Redundancy — keep extra copies or parity so you survive a disk dying.
Capacity — how much usable space you actually get.

You cannot maximize all three at once. Each RAID "level" is a different compromise. That is the entire concept.

The Linux tool for software RAID is mdadm (multiple-device admin). It builds a virtual device like /dev/md0 out of your real disks.

What is an "array"?

You will see the word array on nearly every line below, so let's define it clearly.

An array is the single logical disk that RAID creates out of your group of physical disks. "Array" just means "an organized collection" — here, a collection of disks organized to behave as one.

The flow:

You have 4 separate physical disks: nvme1n1, nvme2n1, nvme3n1, nvme4n1.
You tell mdadm to combine them.
mdadm hands you back one new device: /dev/md0.

That /dev/md0 is the array. Your OS treats it like a normal single disk — you put a filesystem on it, mount it, read and write files — with no idea there are really 4 disks underneath. mdadm quietly handles spreading data (and copies or parity) across the members.

Vocabulary for the rest of the article:

Member / component disk — one real physical disk in the group (nvme1n1, etc.).
Array — the combined virtual disk (/dev/md0) those members form.
RAID level — the rule for how data is spread across the members (stripe, mirror, parity).

So "create the array" means "fuse these physical disks into one /dev/md0," and "the array is degraded" means "the combined disk still works, but one member is missing."

Do I need to partition the disks first (fdisk)?

Short answer: no, not for this. You will notice there is no fdisk anywhere below — the arrays are built directly on whole raw disks.

A partition divides one physical disk into separate logical sections, each acting like its own disk. Tools like fdisk, parted, or gdisk write a small partition table at the start of the disk recording where each section begins and ends. You partition when you need:

separate areas for different purposes (OS vs data vs swap) on one disk,
boot-specific partitions (the EFI System Partition, /boot/efi),
or multiple filesystems / operating systems on one disk.

But a filesystem or a RAID array can live directly on a whole raw disk — /dev/nvme1n1 and /dev/nvme1n1p1 (its first partition) are both just block devices. On EC2, each EBS volume is already a clean, single-purpose disk, so using whole disks is simplest and common. (On physical servers, some admins still partition first so a slightly-smaller replacement drive from a different vendor still fits the array.)

The three layers, kept straight:

fdisk = divide a disk into sections (optional).
mkfs = put a filesystem on a section, a whole disk, or a RAID array.
mdadm = combine disks into a RAID array.

A word on filesystems: ext4 vs xfs

A RAID array is just a block device — it needs a filesystem before it can hold files. The two common choices:

	ext4	xfs
Best at	General purpose, small-to-medium files	Large files, high parallelism, big filesystems
Can grow?	Yes	Yes (online only)
Can shrink?	Yes	No — grow only
Default on	Debian, Ubuntu	RHEL, Fedora, Amazon Linux 2

Grow means enlarging a filesystem to use more space (common and safe — e.g. after expanding an EBS volume you still run resize2fs or xfs_growfs to make the new space usable). Shrink means reducing it to free space (rarer, riskier, and xfs cannot do it at all).

⚠️ mkfs always erases. Reformatting a device — even to switch from ext4 to xfs — destroys everything on it. There is no in-place ext4-to-xfs conversion. To change filesystem type without losing data: copy the data off, mkfs the new type, then copy it back. In this lab, reformatting between RAID levels was fine only because the test files were disposable.

I used ext4 throughout. For most workloads either is fine; pick xfs for very large files and heavy concurrency, ext4 for everything else or when you might need to shrink.

2. The hands-on build

The setup

On EC2 I used:

1 × 30 GiB root volume — the OS. Never touched.
4 × 10 GiB EBS volumes — the RAID practice disks.

Modern EC2 instances present disks as NVMe devices. Here is lsblk:

nvme0n1      30G  disk              <- ROOT. Has /, /boot. NEVER touch.
├─nvme0n1p1  29G  part /
├─nvme0n1p15 106M part /boot/efi
└─nvme0n1p16 913M part /boot
nvme1n1      10G  disk              <- RAID disk
nvme2n1      10G  disk              <- RAID disk
nvme3n1      10G  disk              <- RAID disk
nvme4n1      10G  disk              <- RAID disk

The one rule: the root disk is the one with partitions mounted at /. Every RAID command below names only nvme1n1–nvme4n1. If you ever see nvme0n1 in a format or RAID command, stop.

Install the tool and learn two status commands:

sudo apt update && sudo apt install -y mdadm

cat /proc/mdstat — quick live status of all arrays.
sudo mdadm --detail /dev/md0 — full report on one array.

How to read the status symbols (this is 90% of understanding mdadm):

[UU], [UUUU] — each U is a disk that is Up. An _ is down or missing.
[2/2], [4/3] — (disks expected / disks present).
(F) beside a disk — it is marked Failed.
resync — syncing a freshly created array.
recovery — rebuilding a disk you re-added after a failure.
State : clean = healthy; State : degraded = running but missing a disk.

RAID 0 — Striping (speed, zero safety)

RAID 0 splits data across disks in chunks — a write goes to disk 1, then disk 2, then disk 1, so both work at once. Fast. But no redundancy: lose either disk and everything is gone.

Capacity: full sum of all disks. 2 × 10 GiB = ~20 GiB.

sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1
cat /proc/mdstat

md0 : active raid0 nvme2n1[1] nvme1n1[0]
      20953088 blocks super 1.2 512k chunks

512k chunks is the stripe size (how much data lands on one disk before moving to the next). 20953088 blocks ≈ 20 GiB — full capacity, no redundancy overhead.

Add a filesystem and mount it:

sudo mkfs.ext4 /dev/md0
sudo mkdir -p /mnt/raid0
sudo mount /dev/md0 /mnt/raid0
echo "RAID 0 test" | sudo tee /mnt/raid0/test.txt

There is no failure drill for RAID 0 — if a disk dies, the array is simply gone. That is the point.

Teardown (frees the disks for the next level):

sudo umount /mnt/raid0
sudo mdadm --stop /dev/md0
sudo mdadm --zero-superblock /dev/nvme1n1 /dev/nvme2n1

The superblock is metadata mdadm writes on each disk saying "I belong to array X." --zero-superblock wipes it so the disk looks fresh. Skip this and mdadm may re-assemble a ghost array later.

RAID 1 — Mirroring (safety, half the space)

RAID 1 writes identical data to both disks — twins. One dies, the other has a complete copy.

Capacity: one disk's worth. 2 × 10 GiB = ~10 GiB. The 50% mirror tax.

sudo mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1

First scary-but-harmless message:

mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device. ...
Continue creating array? y

It is only saying this array cannot boot an OS (its metadata sits at the disk start). We use it for data, so type y.

cat /proc/mdstat

md0 : active raid1 nvme2n1[1] nvme1n1[0]
      10476544 blocks super 1.2 [2/2] [UU]
      [==>..................]  resync = 11.8% (1241088/10476544) ...

That progress bar is a resync — mdadm copying disk 1 onto disk 2 so they start identical. The array is usable immediately; you do not wait. [UU] = both up. 10476544 ≈ 10 GiB — half of RAID 0.

sudo mkfs.ext4 /dev/md0
sudo mkdir -p /mnt/raid1 && sudo mount /dev/md0 /mnt/raid1
echo "RAID 1 test" | sudo tee /mnt/raid1/test.txt

Warning you'll see: mkfs.ext4 says /dev/md0 contains a ext4 file system ... Proceed anyway? (y,N). That is leftover from the RAID 0 filesystem on these reused disks. Type y to overwrite.

The failure drill:

sudo mdadm /dev/md0 --fail /dev/nvme2n1
cat /proc/mdstat

md0 : active raid1 nvme2n1[1](F) nvme1n1[0]
      10476544 blocks super 1.2 [2/1] [U_]

(F) failed, [2/1] one of two present, [U_] one up and one down → degraded. But:

cat /mnt/raid1/test.txt        # RAID 1 test   <- still works

A disk "died" and you lost nothing. Rebuild it:

sudo mdadm /dev/md0 --remove /dev/nvme2n1
sudo mdadm --zero-superblock /dev/nvme2n1
sudo mdadm /dev/md0 --add /dev/nvme2n1
cat /proc/mdstat        # now shows: recovery = X%

recovery = re-mirroring onto the fresh disk. Back to [UU] when done. Then tear down.

RAID 5 — Striping + distributed parity (the popular one)

RAID 5 stripes like RAID 0 (fast) and computes parity (a checksum) spread across all disks. Lose one disk and mdadm recalculates its data from parity plus the survivors. There is no full copy — it is math.

Capacity: you lose exactly one disk's worth to parity, regardless of disk count. 3 × 10 GiB = ~20 GiB. Cheap redundancy (1/N overhead). Minimum 3 disks.

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
cat /proc/mdstat

md0 : active raid5 nvme3n1[3] nvme2n1[1] nvme1n1[0]
      20953088 blocks ... level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      [=>...................]  recovery =  8.4% ...

Looks broken but isn't. [3/2] [UU_] plus "recovery" on a brand-new array (and mdadm --detail showing spare rebuilding) is a deliberate optimization: mdadm builds the last disk via recovery instead of syncing all three from scratch. It ends at [3/3] [UUU]. Don't panic.

~20 GiB from three disks → one disk's worth went to parity.

sudo mkfs.ext4 /dev/md0
sudo mkdir -p /mnt/raid5 && sudo mount /dev/md0 /mnt/raid5
echo "RAID 5 test" | sudo tee /mnt/raid5/test.txt

Failure drill:

sudo mdadm /dev/md0 --fail /dev/nvme3n1
cat /mnt/raid5/test.txt        # RAID 5 test   <- reconstructed from parity, live

There is no copy of the dead disk anywhere — every read rebuilds the missing bytes from parity plus the survivors on the fly.

Hard limit: RAID 5 survives exactly ONE failure. A second disk down = total loss. That limitation is why RAID 6 exists. Rebuild with the same remove → zero → add pattern, then tear down all three disks.

RAID 6 — Double parity (survive TWO failures)

RAID 6 is RAID 5 with a second, independent parity block. Any two disks can die at once and you lose nothing. Critical for large arrays, where a second disk often fails during the stressful rebuild of the first.

Capacity: lose two disks' worth. 4 × 10 GiB = ~20 GiB. Minimum 4 disks.

sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
# then mkfs.ext4, mount at /mnt/raid6, write test.txt

The headline drill — fail two at once:

sudo mdadm /dev/md0 --fail /dev/nvme3n1 --fail /dev/nvme4n1
cat /mnt/raid6/test.txt        # RAID 6 test   <- still works!
cat /proc/mdstat

md0 : active raid6 nvme4n1[3](F) nvme3n1[2](F) nvme2n1[1] nvme1n1[0]
      ... [4/2] [UU__]

Two disks dead at once, data survives. RAID 5 would be destroyed by this. Rebuild both:

sudo mdadm /dev/md0 --remove /dev/nvme3n1
sudo mdadm /dev/md0 --remove /dev/nvme4n1
sudo mdadm --zero-superblock /dev/nvme3n1 /dev/nvme4n1
sudo mdadm /dev/md0 --add /dev/nvme3n1
sudo mdadm /dev/md0 --add /dev/nvme4n1

RAID 10 — Mirror + Stripe (speed and safety)

RAID 10 is "nested": it makes mirror pairs, then stripes across them. You get RAID 1's safety and RAID 0's speed, with fast rebuilds (copy from the mirror partner — no parity math). The database workhorse.

Capacity: half to mirroring. 4 × 10 GiB = ~20 GiB.

sudo mdadm --create --verbose /dev/md0 --level=10 --raid-devices=4 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
cat /proc/mdstat

md0 : active raid10 nvme4n1[3] nvme3n1[2] nvme2n1[1] nvme1n1[0]
      20953088 blocks ... 512K chunks 2 near-copies [4/4] [UUUU]

2 near-copies = every block lives on 2 disks (the mirror part).

The crucial detail — which disks are partners?

sudo mdadm --detail /dev/md0

0  ... active sync set-A  /dev/nvme1n1
1  ... active sync set-B  /dev/nvme2n1
2  ... active sync set-A  /dev/nvme3n1
3  ... active sync set-B  /dev/nvme4n1

The set-A / set-B labels are the mirror groups:

set-A: nvme1n1 ⇄ nvme3n1
set-B: nvme2n1 ⇄ nvme4n1

Notice the pairing is not 1+2 / 3+4 as you might guess. Always read the set- labels. This controls survival:

Lose 1 disk → always fine.
Lose 2 in different sets → fine.
Lose 2 in the same set → array dead.

RAID 6 survives any 2; RAID 10's two-disk survival is conditional — but it rebuilds much faster.

Failure drill:

sudo mdadm /dev/md0 --fail /dev/nvme3n1
cat /proc/mdstat          # [4/3] [UU_U]
cat /mnt/raid10/test.txt  # still works — partner nvme1n1 serves it
sudo mdadm /dev/md0 --remove /dev/nvme3n1
sudo mdadm --zero-superblock /dev/nvme3n1
sudo mdadm /dev/md0 --add /dev/nvme3n1

Rebuild speed is noticeably faster (~159 MB/s in my run) — a straight copy from the partner, no parity to compute.

3. Production reality — what actually goes wrong, and how to handle it

The lab is clean. Production is not. Here are the failures you will genuinely face on live systems, roughly in order of how often they bite.

1. A disk fails for real (the everyday case)

What you see: an email from mdadm --monitor, or cat /proc/mdstat shows [U_UU] and State : degraded. The array still serves data.

How to handle:

Don't panic — degraded means working but no safety margin. Reads and writes continue.
Identify the dead disk and its serial so you pull the right one from the chassis:

   sudo mdadm --detail /dev/md0
   sudo smartctl -a /dev/sdX | grep -i serial

Fail + remove it (if not already failed), swap the physical disk, then add the new one:

   sudo mdadm /dev/md0 --fail /dev/sdX --remove /dev/sdX
   # physically replace the drive, then:
   sudo mdadm /dev/md0 --add /dev/sdY
   watch cat /proc/mdstat   # watch recovery climb to [UUUU]

Pull the correct disk. Yanking a healthy disk from a degraded RAID 5 destroys the array. Match by serial number every time.

Set up alerting so you even know a disk died:

sudo nano /etc/mdadm/mdadm.conf   # add: MAILADDR you@example.com
sudo mdadm --monitor --scan --daemon

Without monitoring, a disk can sit failed for weeks until a second one dies and takes everything. This is the number one cause of "RAID didn't save us" stories.

2. The rebuild kills a second disk (the RAID 5 killer)

What happens: a disk fails, you add a replacement, and during the long, intense full-disk rebuild another aging disk hits a bad sector and drops out. On RAID 5 that is total array loss — the classic reason RAID 5 is discouraged for large modern drives.

How to prevent:

Use RAID 6 (or RAID 10) for anything large or important — a second failure during rebuild is survivable.
Run periodic scrubs to find bad sectors before a rebuild forces you to read them:

  echo check | sudo tee /sys/block/md0/md/sync_action
  cat /sys/block/md0/md/mismatch_cnt     # 0 is good

Keep backups. RAID is not a backup — it protects against disk failure, not deletion, corruption, ransomware, or a controller frying every disk at once.

3. "Array won't assemble after reboot"

What you see on boot: the array is missing, you're dropped to an emergency shell, or /mnt/data is empty.

Root cause: the array config was never persisted, so the OS does not know to assemble it. (In this lab we skipped that on purpose, which is exactly why nothing survived a reboot.)

The fix — do this for any array you want to keep:

# 1. Save the array definition
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

# 2. Rebuild the initramfs so the array assembles early at boot
sudo update-initramfs -u          # Debian/Ubuntu
# (RHEL/CentOS: sudo dracut -f)

# 3. Mount by UUID in /etc/fstab (never by /dev/md0 — numbers can change)
sudo blkid /dev/md0               # grab the UUID
# add to /etc/fstab:
# UUID=xxxx /mnt/data ext4 defaults,nofail 0 2

The nofail option matters: without it, a missing array can hang the entire boot. With it, the box boots and you fix the array live.

If it is already not assembling:

sudo mdadm --assemble --scan                  # try auto-assemble
sudo mdadm --assemble /dev/md0 /dev/sd[a-d]   # or name the members
sudo mdadm --examine /dev/sdX                 # inspect each disk's superblock

4. The device name changed (`/dev/md0` became `/dev/md127`)

What you see: the array shows up as /dev/md127 instead of /dev/md0, and fstab entries pointing at /dev/md0 fail.

Why: when mdadm cannot match the array to a config entry, it auto-numbers from 127 downward. Almost always means mdadm.conf is missing or stale.

Fix: regenerate the config (step 3 above) with the correct ARRAY line, run update-initramfs -u, and always mount by UUID, not device name.

5. Split brain after a crash (Events counter mismatch)

What you see: --assemble refuses because the disks disagree about who has the newest data. mdadm --examine /dev/sdX shows different Events counts per disk.

How to handle:

Assemble using the disks with the highest, matching Events count:

  sudo mdadm --examine /dev/sd[a-d] | grep -E 'Events|/dev'
  sudo mdadm --assemble /dev/md0 /dev/sda /dev/sdb /dev/sdc   # the good ones

As a last resort, --assemble --force lets a slightly-behind disk rejoin. Use it only when you understand the risk — it can resurrect stale data. Take a dd image of each disk first if the data matters.

6. Silent data corruption / bit rot

What you see: files are subtly wrong, or a scrub reports a non-zero mismatch_cnt.

How to handle: mdadm RAID does not checksum data, so it cannot detect silent corruption on its own — it only knows a disk is present or gone. Scrubs catch mismatches between data and parity but can't always say which side is right. For true bit-rot protection, use a checksumming filesystem (ZFS or Btrfs) on top of, or instead of, mdadm.

7. Rebuild is crawling and tanking performance

What you see: a rebuild says finish=600min and your app is slow because disk I/O is saturated.

How to handle — tune the rebuild rate:

cat /proc/sys/dev/raid/speed_limit_min   # current limits (KB/s)
cat /proc/sys/dev/raid/speed_limit_max

echo 200000 | sudo tee /proc/sys/dev/raid/speed_limit_min   # speed up in a window
echo 10000  | sudo tee /proc/sys/dev/raid/speed_limit_min   # throttle for production

Trade-off: faster rebuild means less time at risk but heavier I/O load. On a busy box, throttle during peak and open it up at night.

8. Out of space on a "full capacity" array

What you see: writes fail with No space left on device sooner than expected.

Reminder: RAID 0/5/6 report combined capacity, but filesystem metadata, ext4's ~5% reserved blocks, and parity all eat into it. Plan for ~90–95% usable and alert well before 100% — a full filesystem can corrupt in-flight writes.

The lab warnings (harmless — don't confuse them with the above)

Message	Meaning	Action
`may not be suitable as a boot device`	Can't boot an OS from this array; fine for data	Type `y`
`contains a ext4 file system ... Proceed anyway?`	Leftover FS from reusing a disk	Type `y`
`[3/2] [UU_]` + `spare rebuilding` right after creating RAID 5	Normal creation optimization	Wait for `[UUU]`
`No such file or directory` on a test file	You're looking at an unmounted mountpoint	Remount

Cleanup (and stop the AWS bill)

sudo umount /mnt/raidXX
sudo mdadm --stop /dev/md0
sudo mdadm --zero-superblock /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
cat /proc/mdstat        # should say: unused devices: <none>

We never wrote /etc/fstab or mdadm.conf, so nothing persists across reboot — a clean slate.

On AWS: EBS volumes bill while they exist, attached or not. Detach the 4 extra volumes, delete them, and terminate the instance if you are done.

The one-page summary

RAID	Min disks	Usable (4×10 GiB)	Survives	Overhead	Rebuild	Best for
0	2	20 GiB	nothing	0%	n/a (data lost)	scratch, max speed
1	2	10 GiB	any 1	50%	copy from mirror	OS/boot, small critical data
5	3	20 GiB (3 disks)	any 1	~33%	parity math (slow)	balanced general storage
6	4	20 GiB	any 2	50%	parity math (slowest)	large arrays, long rebuilds
10	4	20 GiB	1 always; 2 if different sets	50%	copy from partner (fast)	databases, write-heavy

Mental model:

Stripe (0) = speed, no safety.
Mirror (1) = safety, half capacity.
Parity (5/6) = safety at lower capacity cost; 6 adds a second parity for two-failure tolerance.
10 = mirror + stripe; speed and safety, 50% capacity, conditional two-disk survival.

Three production truths worth remembering:

RAID is not a backup. It survives disk failure, not deletion, corruption, or human error.
A degraded array with no monitoring is a time bomb — set up mdadm --monitor on day one.
Persist the config (mdadm.conf + update-initramfs + fstab by UUID), or your array vanishes on the next reboot.

Wrapping up

Building each level by hand — and failing disks on purpose — made RAID click in a way no diagram ever did. But the real lesson is in Part 3: the lab always rebuilds cleanly, while production throws you degraded arrays at 2 AM, rebuilds that saturate I/O, and boxes that won't boot. Knowing the handful of commands above turns those from emergencies into routine.

AWS makes this experiment cheap: a small EC2 instance plus a few tiny EBS volumes for an hour costs almost nothing, and you tear it all down when you're done. Spin one up and try it yourself — then go set up monitoring on whatever RAID you already run.

"It Works on My Machine" — How a Shared Redis Database Broke Authentication Across Environments

POTHURAJU JAYAKRISHNA YADAV — Sat, 03 Oct 2026 03:02:23 +0000

We've all heard the classic developer line: "But it works on my local machine!" Usually it's a punchline. This time, it was the starting point of a debugging journey that taught me a lesson I won't forget about environment isolation, Redis, and the subtle ways shared infrastructure can betray you.

In this post I'll walk through a real production-adjacent incident I ran into recently: the application ran perfectly on local, worked fine when I ran it myself, but kept failing in the Dev and QA environments with authentication errors coming from a third-party vendor. The root cause turned out to be something that looked completely innocent until the very end.

If you run multiple environments that share backing services, this one is worth reading to the end.

The Setup

Our application is a fairly standard web service. Without going into proprietary details, the moving parts relevant to this story are:

The application — runs in multiple environments: local (developer machines), Dev, and QA.
MySQL — our primary relational data store.
Redis — used for caching and, importantly, for storing session/authentication tokens.
A third-party vendor server — we authenticate against it, and it issues tokens we store and reuse.

The flow that mattered: a user logs in, we obtain an authentication token from the third-party vendor, and we cache that token in Redis so we don't have to re-authenticate on every request.

Simple enough. Until it wasn't.

The Symptom

The report came in like this:

"The dev team runs the application locally and everything works. But in Dev and QA, it's not working."

That phrasing is a trap. "Not working" can mean a hundred things. The actual symptom, once I dug in, was an authentication error returned by the third-party vendor server — but only in the shared Dev and QA environments. Local was perfectly happy.

The confusing part: the error was intermittent and seemed to have nothing to do with what any individual developer was doing. One minute a user was authenticated, the next minute they were getting kicked out, and the vendor was rejecting tokens that should have been valid.

Classic "works on my machine" territory. Which is exactly why I started where most of us start.

The Investigation (and Every Dead End Along the Way)

I want to be honest about the order I actually went through this, because the dead ends are the most instructive part. Debugging is rarely a straight line, and each thing I ruled out narrowed the search.

Step 1: "Did someone change the code?"

The first instinct when something works in one place and breaks in another: what changed?

I checked the recent commits and diffs between what was running locally and what was deployed to Dev and QA. I was looking for any code change — a new feature, a refactor, a tweak to the auth flow, anything.

Result: No code changes. The code running locally was effectively the same code running in Dev and QA. If the code is identical, the code probably isn't the problem.

That's an important deduction: when the same code behaves differently in different places, the difference is almost always in the environment, not the code.

Step 2: "Is it a dependency version mismatch?"

The next usual suspect: package versions. Maybe local had a different version of a library than the deployed environments — a different Redis client, a different HTTP client, a different auth SDK. Version drift between environments is a very common cause of "works here, not there."

So I compared the dependency versions across environments. Lockfiles, installed package versions, the whole list.

Result: Same versions everywhere. Another dead end, but another data point: it's not a dependency problem.

Step 3: "Let me reproduce it myself"

At this point I did the thing you should always do — I stopped theorizing and ran the application myself, locally, trying to reproduce the exact failing flow.

Result: It worked perfectly for me. No authentication errors. Everything flowed as expected.

This is the most frustrating stage of any bug hunt. The thing refuses to break when you're watching it. But it's also a huge clue: if it works in isolation on local but fails in the shared environments, the problem lives in something the shared environments have in common that local doesn't.

Step 4: "Maybe the backing services are misconfigured"

Now I turned my attention to the infrastructure. The application depends on Redis and MySQL. My hypothesis: maybe Redis or MySQL in Dev/QA was misconfigured — wrong settings, wrong memory limits, wrong eviction policy, something.

I went through the server-side configuration for these services across environments.

Result: The server-side configurations were all the same. The Redis and MySQL servers themselves looked correctly and identically configured. Yet another dead end for the "something is broken" theory.

By now a pattern was forming in my head. Nothing was broken. Everything was configured the same. And yet behavior differed. That usually means something is being shared when it shouldn't be.

Step 5: "Let me look at the application environment config"

The final place to look was the application's own environment configuration — the .env-style config that tells each environment which Redis, which MySQL, which vendor endpoint to talk to.

And here's where it clicked.

I looked at where the application was pointed for Redis, and I saw that Dev and QA were pointed at the same Redis instance and the same Redis database.

That was the smoking gun.

The Root Cause

Here's what was actually happening.

Redis supports multiple logical databases within a single instance (numbered 0, 1, 2, and so on by default). Our Dev and QA environments were both pointed at:

the same Redis host, and
the same Redis logical database.

Because Redis stores data as keys, and both environments used the same key naming for authentication tokens, the two environments were writing to and reading from the same slots.

Now layer the authentication flow on top of that:

A developer logs into Dev. The app authenticates against the third-party vendor, gets a token, and stores it in Redis under some key — say, a key derived from the user's identity.
The same user (or the same key) logs into QA. The app authenticates again, gets a new token from the vendor, and overwrites the same Redis key.
Back in Dev, the next request pulls the token from Redis — but now it's holding the QA session's token, or a token that the vendor has effectively superseded.
The third-party vendor sees a token that doesn't match the session it expects and rejects it with an authentication error.

In other words: logging into one environment was effectively logging the user out of the other. The two environments were stomping all over each other's tokens because they shared the same Redis database.

That perfectly explained every observation:

Why no code change mattered — the code was correct; the data store was shared.
Why versions didn't matter — it was never a dependency issue.
Why it worked on local — local pointed at its own isolated Redis, so nothing else was overwriting its tokens.
Why it worked when I ran it myself — I was the only one touching those keys at that moment.
Why it was intermittent — it only broke when activity in Dev and QA overlapped on the same user/keys.
Why the vendor threw the auth error — the token it received had been replaced by the other environment's token.

The services weren't misconfigured in the sense of being broken. They were misconfigured in the sense of being shared across environments that should have been isolated.

The Fix

The fix was refreshingly small once the cause was understood: give each environment its own Redis database.

Redis makes this easy because a single instance exposes multiple numbered logical databases. By changing the Redis database index (and ideally the host) per environment in the application's environment config, each environment got its own isolated keyspace.

Conceptually, the configuration went from this:

# Dev
REDIS_HOST=shared-redis
REDIS_DB=0

# QA
REDIS_HOST=shared-redis
REDIS_DB=0      # <-- same instance AND same DB as Dev

to this:

# Dev
REDIS_HOST=shared-redis
REDIS_DB=0

# QA
REDIS_HOST=shared-redis
REDIS_DB=1      # <-- separate logical database

After separating the Redis databases per environment, the token collisions stopped. Logging into Dev no longer touched QA's tokens, the vendor stopped rejecting tokens, and the authentication errors disappeared entirely.

A note on logical databases vs. separate instances: Using separate Redis logical databases (REDIS_DB) solved our immediate problem and is a perfectly valid quick fix. For stronger isolation, the better long-term approach is a completely separate Redis instance per environment, since logical databases still share the same memory, CPU, eviction policy, and FLUSHALL blast radius. Another robust option is key prefixing / namespacing (for example, prefixing every key with the environment name) so collisions are impossible even on a shared database. Pick the level of isolation that matches your risk tolerance.

Why This Was So Easy to Miss

This is the kind of bug that hides in plain sight, and I think it's worth reflecting on why.

Nothing was broken. Every service was up. Every config "looked" correct in isolation. There was no stack trace pointing at Redis — the error surfaced at the third-party vendor, which is as far downstream from the real cause as you can get.
The error was misleading. An authentication failure from an external vendor makes you suspect credentials, tokens, network, or the vendor itself — not your cache layer.
It only manifested under concurrency. When one environment was quiet, the other worked fine. That's why it never reproduced on demand.
Shared infrastructure is often invisible in day-to-day work. You rarely stop and ask, "Wait, are these two environments literally the same Redis database?" You assume isolation that was never actually there.

The real lesson is that the error message pointed at the symptom (vendor auth rejection), while the cause was several layers upstream (shared cache state). Good debugging means following the chain backward past the first plausible-looking culprit.

Lessons Learned

Here are the takeaways I'm carrying forward, and that I'd offer to anyone running multi-environment systems.

1. Environments must be isolated — all the way down

It's not enough to have separate application deployments. Every backing service — Redis, MySQL, message queues, object storage, search indexes — needs to be isolated per environment, or at minimum namespaced so there's no chance of cross-environment collision. Shared state between environments is a latent bug waiting for enough concurrent traffic to surface.

2. "Same code, different behavior" means look at the environment

When identical code behaves differently across environments, the bug is almost never in the code. It's in configuration, data, infrastructure, or state. Save yourself time by ruling out code quickly and moving to the environment.

3. Follow the error upstream, past the obvious suspect

The vendor threw the authentication error, but the vendor wasn't the problem. The first component that reports an error is frequently just the victim of something further up the chain. Keep asking "but why did that happen?" until you reach a cause that explains all your observations, not just some of them.

4. The config file deserves the same scrutiny as the code

I checked code, versions, and server-side service configuration before I looked carefully at the application's environment config — which is exactly where the problem was. Environment/config files are code. Diff them across environments as a first-class debugging step.

5. Reproduce in isolation to localize the problem

The fact that it worked on local and worked when I ran it alone was itself the key insight: the problem only existed where state was shared. "Can I reproduce it in isolation?" is a powerful question — the answer tells you whether the bug is in the component or in its surroundings.

6. Design keyspaces defensively

Even if you believe your environments are isolated today, prefix your cache keys with the environment name. It costs almost nothing and makes an entire category of cross-environment collisions structurally impossible. Defense in depth applies to cache keys too.

A Simple Mental Checklist for "Works Here, Not There" Bugs

Next time you hit a bug that works in one place but fails in another, run through this:

Did the code change? Diff it. If it's identical, move on.
Did the dependencies change? Compare versions and lockfiles. If identical, move on.
Can you reproduce it in isolation? If it only breaks in a shared environment, suspect shared state.
Are the backing services misconfigured? Check server-side config.
Where is each environment actually pointed? Check the application's env/config — host, port, database index, credentials, endpoints.
Is anything being shared that shouldn't be? Same database, same keyspace, same bucket, same queue.
Follow the error upstream until you find a cause that explains every symptom.

In my case, the answer was hiding at step 5 and 6 the whole time.

Closing Thoughts

The bug felt enormous while I was chasing it — authentication failing in Dev and QA, a third-party vendor rejecting tokens, nothing reproducing locally. The fix was one line of configuration: point each environment at its own Redis database.

That contrast is the whole story of debugging distributed systems. The symptoms are loud and scattered; the cause is quiet and specific. The work is in patiently ruling things out until the quiet cause has nowhere left to hide.

If you take one thing away: isolate your environments completely, and never assume that two environments pointed at "Redis" are pointed at different Redis. Verify it. Because the day your traffic overlaps, that shared database will find you.

Have you run into a similar shared-infrastructure gotcha? I'd love to hear how it surfaced for you and how you tracked it down. Drop it in the comments.

How I Eliminated Lambda Deployment Downtime with Terraform and Blue-Green Deployments

POTHURAJU JAYAKRISHNA YADAV — Sat, 06 Jun 2026 11:31:09 +0000

I Got Tired of Lambda Deployment Headaches, So I Built My Own Blue-Green Deployment System

We've all had that moment.

You push a new Lambda version to production, hit deploy, and then spend the next few minutes refreshing dashboards and logs hoping nothing breaks.

I've been there more times than I'd like to admit.

Sometimes deployments worked perfectly. Other times I would see unexpected 503 errors, traffic routing issues, or situations where rolling back wasn't as fast as I wanted. The Lambda deployment itself wasn't usually the problem. The challenge was everything around it—traffic routing, validation, and rollback.

After hitting the same deployment issues multiple times, I decided to build a deployment model that matched how I actually wanted releases to work.

The result was a Terraform-driven blue-green deployment framework for AWS Lambda that gives me:

Zero-downtime deployments
IP-based canary testing
Weighted traffic shifting
Instant rollback
No CodeDeploy
No Route53 dependency
No hardcoded VPC IDs

Everything is controlled through Terraform.

The Problem I Wanted to Solve

AWS already provides deployment options for Lambda through aliases and CodeDeploy.

For many teams, that's enough.

But in my case, there were a few limitations that kept bothering me.

1. I Wanted to Test Production Before Users Did

When deploying a new version, I wanted a way to access the new release from my own machine while everyone else continued using the stable version.

With standard Lambda deployments, that wasn't straightforward.

2. Rollbacks Needed To Be Instant

If something goes wrong after deployment, I don't want to wait for another deployment process.

I want traffic to switch back immediately.

3. I Wanted Full Traffic Control

Sometimes I want:

10% rollout
25% rollout
50% rollout
75% rollout

Not just predefined deployment strategies.

4. I Wanted Less Moving Parts

CodeDeploy works, but it introduces additional services, configurations, hooks, and deployment definitions.

For this project, I wanted something simpler.

The Idea

Instead of thinking about Lambda deployments as version updates, I started thinking about them the same way ECS blue-green deployments work.

What if two Lambda environments always existed?

One would be stable.

One would contain the new release.

Traffic could simply move between them.

That led to this architecture:

Blue Lambda (current production)
Green Lambda (new version)
Blue Target Group
Green Target Group
ALB Listener Rules controlling traffic

The important detail is that both environments always exist.

Nothing is destroyed.

Nothing is recreated.

Traffic is simply redirected.

That's what eliminates deployment interruptions.

Everything Is Controlled Through Terraform

The deployment workflow is driven through a simple Terraform configuration.

nginx-lambda = {
  family          = "nginx-svc"

  blue_image      = ".../nginx-lambda:blue"
  green_image     = ".../nginx-lambda:green"

  active_color    = "blue"

  activate_canary = false
  green_weight    = 0
}

Whenever I want to change deployment behavior, I update the values and run:

terraform apply

That's it.

No deployment pipelines.

No manual traffic switching.

No complicated release process.

Phase 1: Canary Testing

The first thing I do after deploying a new version is test it myself.

I enable canary mode:

activate_canary = true
green_weight    = 0

This creates a higher-priority ALB rule.

If requests come from my whitelisted IP address:

My IP → Green Version

Everyone else still reaches:

Users → Blue Version

This allows me to validate:

Application functionality
API responses
Headers
Authentication flows
Database interactions

All in production.

Without exposing the new version to actual users.

Phase 2: Gradual Traffic Shifting

Once canary testing looks good, I begin moving real traffic.

For example:

green_weight = 50

Now:

50% → Blue
50% → Green

The Application Load Balancer handles traffic distribution automatically.

Because both target groups already exist and already have Lambda functions attached, traffic shifting happens smoothly.

No downtime.

No target registration delays.

No deployment gaps.

Phase 3: Full Promotion

When confidence is high enough, I move all traffic to Green.

green_weight = 100

At this point:

100% Users → Green

But here's the part I like most.

Blue still exists.

It's sitting there untouched.

Ready for rollback.

Phase 4: Instant Rollback

Every deployment system claims to support rollback.

The real question is:

How fast?

With this setup, rollback is literally:

green_weight = 0

Apply Terraform.

Traffic immediately returns to Blue.

No redeployment.

No waiting for Lambda versions.

No waiting for target group registration.

No waiting for containers to start.

The previous stable environment never disappeared.

What About The Next Release?

This was actually one of the hardest design problems.

Let's say:

Blue = v1
Green = v2

After promoting v2, what happens when v3 arrives?

The answer is surprisingly simple.

Rotate the images.

blue_image  = ".../nginx-lambda:green"
green_image = ".../nginx-lambda:yellow"

Now:

Blue  = v2 (stable)
Green = v3 (candidate)

The deployment cycle repeats.

Canary.

Weighted rollout.

Promotion.

Rollback if needed.

Forever.

Why I Stopped Seeing 503 Errors

Before building this, I experimented with several approaches.

Lambda Alias Switching

This occasionally introduced timing issues during updates.

Dynamic Target Groups

Using conditional resources sometimes caused Terraform to destroy resources before listener rules updated.

Permission Propagation Delays

Lambda permissions occasionally weren't fully propagated before attachments occurred.

The common problem was simple:

Resources were being created and destroyed during deployment.

Eventually I realized the best deployment is often the one that changes the least.

So I stopped changing infrastructure during releases.

Instead:

Blue always exists
Green always exists
Target groups always exist
Attachments always exist

The only thing that changes is traffic routing.

That's why the process is so reliable.

GitHub Repository

https://github.com/jayakrishnayadav24/lambda-blue-green-deployment

Final Thoughts

The biggest lesson from building this wasn't about Lambda or Terraform.

It was about reducing moving parts.

Instead of making deployments smarter, I made them simpler.

Both environments stay alive.

Traffic moves gradually.

Rollback is immediate.

And I can personally validate a release before customers ever touch it.

After several production deployments using this pattern, it's become one of my favorite deployment strategies for Lambda workloads.

If you've been fighting deployment-related downtime or complicated release workflows, you might find this approach useful too.

What If You Need Two ArgoCD Instances in One EKS Cluster?

POTHURAJU JAYAKRISHNA YADAV — Sat, 30 May 2026 08:26:58 +0000

Most ArgoCD tutorials start the same way:

Deploy ArgoCD.

Connect a Git repository.

Create an Application.

Done.

But what happens when multiple teams start sharing the same Kubernetes cluster?

Recently I was exploring a scenario where a single Amazon EKS cluster needed to support two different ArgoCD environments:

A Custom ArgoCD instance for the Platform Team
A Managed ArgoCD instance for Application Teams

The Custom ArgoCD would manage infrastructure components such as:

cert-manager
external-dns
monitoring
ingress controllers

While the Managed ArgoCD would be used by application teams to deploy:

APIs
frontends
microservices
business applications

The challenge was figuring out how to keep both environments isolated while still sharing the same EKS cluster, Cognito User Pool, and AWS Application Load Balancer.

At first, I thought:

Why not just use a single ArgoCD instance?

The more I thought about it, the more questions came up.

How do we separate responsibilities?
How do we avoid accidental changes?
How do we provide SSO?
How do we keep costs under control?

That led me down an interesting path:

Can we run two ArgoCD instances inside the same EKS cluster, authenticate both using Cognito SSO, and still share the same ALB?

Turns out we can.

This article walks through the architecture, challenges, lessons learned, and a few surprises along the way.

The Scenario

Imagine you have a Kubernetes cluster used by multiple teams.

The Platform Team owns:

cert-manager
external-dns
monitoring
ingress controllers
cluster add-ons

The Application Teams own:

APIs
frontends
microservices
business applications

A single ArgoCD instance technically works.

But should both groups operate from the same GitOps control plane?

For smaller environments, probably yes.

For larger environments, things start getting messy.

Different permissions.

Different ownership boundaries.

Different operational responsibilities.

That was the motivation behind splitting ArgoCD into two instances.

The Goal

The architecture I wanted looked like this:

EKS Cluster

├── ArgoCD (Platform Team) - Custom argocd
│     ├── cert-manager
│     ├── external-dns
│     └── monitoring
│
└── ArgoCD (Application Teams) - Managed argocd
      ├── frontend
      ├── backend
      └── business applications

And on top of that:

Single Sign-On
Fully automated provisioning
Infrastructure as Code
Minimal AWS cost

The Final Architecture

The final design consisted of:

Developer
    |
    | HTTPS
    v

Shared AWS ALB
(group.name = argocd)

    |
    +---------------------+
    |                     |
    v                     v

argocd              argocd2
Namespace:          Namespace:
argocd              argocd-managed

Platform Team       Application Teams

    |                     |
    +----------+----------+
               |
               v

AWS Cognito User Pool

Both ArgoCD instances run inside the same EKS cluster.

The difference is that each lives in its own namespace.

argocd
argocd-managed

This gives logical separation without requiring another cluster.

Why Cognito?

Before choosing Cognito, I looked at AWS IAM Identity Center.

Identity Center is generally the preferred enterprise solution.

However, for this learning exercise I wanted everything automated using Terraform.

My ideal deployment process was:

terraform apply

and automatically create:

User Pool
App Clients
Groups
Users
Passwords

without opening the AWS Console.

Cognito made that surprisingly easy.

Building the Authentication Layer

The first step was creating a Cognito User Pool.

Each ArgoCD instance gets its own App Client.

Visually, it looked like this:

Cognito User Pool

├── App Client
│     └── ArgoCD Platform
│
└── App Client
      └── ArgoCD Managed

Here's what it looked like in AWS.

One thing I particularly liked was eliminating manual user creation.

Terraform creates:

User Pool
Domain
App Clients
Groups
Admin User

Everything becomes Infrastructure as Code.

No temporary passwords.

No manual console work.

No forgotten setup steps.

Connecting ArgoCD to Cognito

ArgoCD supports OIDC natively.

That means Cognito can act as the identity provider.

The authentication flow becomes:

User
  |
  v
ArgoCD
  |
  v
Cognito Hosted UI
  |
  v
Authentication
  |
  v
OIDC Callback
  |
  v
ArgoCD Dashboard

Once logged in, ArgoCD reads the Cognito groups from the token and maps them to ArgoCD roles.

That means administrators are controlled centrally through Cognito instead of managing users inside ArgoCD itself.

The Shared ALB Experiment

This was probably the most interesting part.

Initially I considered creating two separate Application Load Balancers.

One for each ArgoCD instance.

That would work.

But it also means:

More AWS resources
More management
More cost

Then I remembered the AWS Load Balancer Controller supports Ingress Groups.

That changed everything.

Instead of:

ALB #1 → ArgoCD
ALB #2 → ArgoCD Managed

I could use:

One ALB
Two Hostnames
Two ArgoCD Instances

The ALB creates listener rules based on the hostname.

And from Kubernetes:

Both ingress resources point to the same ALB.

The routing happens automatically based on the Host header.

A nice cost optimization with very little additional complexity.

The Problem Nobody Warned Me About

The first ArgoCD deployment worked perfectly.

The second one failed immediately.

The error looked something like this:

customresourcedefinitions.apiextensions.k8s.io already exists

At first it wasn't obvious.

Then it clicked.

ArgoCD installs CRDs.

CRDs are cluster-scoped resources.

The first installation already created them.

The second installation attempted to create them again.

Kubernetes refused.

The fix was simple:

crds:
  install: false

Once I disabled CRD installation for the second ArgoCD instance, everything worked.

It was one of those issues that takes a while to understand but only seconds to fix.

The Login Experience

After deployment, the user experience was exactly what I wanted.

First, users see the ArgoCD login screen.

Selecting the Cognito login option redirects users to the Hosted UI.

After successful authentication, users land directly inside ArgoCD.

No local accounts.

No shared admin credentials.

No Kubernetes secret lookups.

Just SSO.

Another Small Gotcha

While testing Applications, I ran into this error:

app path is not a directory

I had accidentally pointed ArgoCD to a YAML file instead of a directory.

ArgoCD expects:

path: manifests/

not:

path: manifests/app.yaml

A small detail, but one that can waste a surprising amount of time when you're learning.

What I Learned

The most valuable lesson wasn't about Cognito.

It wasn't about Terraform.

It wasn't even about ArgoCD.

It was about ownership.

As environments grow, separating responsibilities becomes more important than simply making things work.

Running a Custom ArgoCD for platform operations and a Managed ArgoCD for application teams gave:

Better operational boundaries
Cleaner RBAC
Clear ownership between teams
Reduced risk of accidental changes
Independent GitOps workflows

The platform team can safely manage infrastructure applications while application teams operate in their own ArgoCD environment without interfering with cluster-level services.

And because both instances share a single ALB, the cost impact remained minimal.

Would I Do This Again?

For a small team?

Probably not.

A single ArgoCD instance is easier to manage.

For organizations with dedicated platform teams and application teams?

Absolutely.

The pattern scales nicely, provides cleaner ownership boundaries, and integrates well with SSO.

Most importantly, the entire environment can be created with a single:

terraform apply

which is exactly what I wanted from the beginning.

If you've experimented with multiple ArgoCD instances or different GitOps multi-tenancy approaches, I'd love to hear how you've solved it.

Happy GitOps 🚀

GitHub Repository

https://github.com/jayakrishnayadav24/eks-argocd-cognito-sso

Zero-Downtime Blue-Green and IP-Based Canary Deployments on ECS Fargate

POTHURAJU JAYAKRISHNA YADAV — Sat, 23 May 2026 08:36:49 +0000

Most ECS blue-green deployment tutorials eventually lead to the same stack:

AWS CodeDeploy
Deployment groups
AppSpec files
Lifecycle hooks
Weighted traffic shifting
Complex rollback orchestration

And while CodeDeploy works, I kept running into one practical limitation during real deployments:

I couldn’t let my internal team validate a new release on the actual production URL before exposing it to customers.

That became the entire motivation behind this setup.

I didn’t want:

separate staging domains
duplicate ALBs
temporary preview environments
“almost production” testing

I wanted something much simpler:

Internal users should see the new version first
Customers should continue seeing the stable version
Both should use the same production domain
Rollback should be immediate
Deployments should remain fully zero downtime

So I built a Terraform-driven deployment workflow using:

ECS Fargate
Application Load Balancer (ALB)
ALB listener priorities
Source IP routing
Terraform

without using CodeDeploy.

After running this setup in practice, I ended up preferring it for many ECS workloads.

The Core Idea

Both BLUE and GREEN environments run behind the same ALB.

Internal office/VPN IPs get routed to GREEN first.

Everyone else continues hitting BLUE.

That means QA and internal teams can validate the new release directly on the real production infrastructure before public rollout begins.

Same:

domain
SSL certificate
ALB
authentication flow
redirects
networking path

No “staging surprises” later.

A lot of deployment issues only appear on the real production routing path.

Real Example

Internal users open:

https://nginx.jayakrishnayadav.cloud

…and immediately see the GREEN version.

Meanwhile, public users continue seeing BLUE.

No DNS switching.

No duplicate infrastructure.

Just ALB listener routing.

Architecture Overview

The deployment flow looks like this:

                ┌────────────────────┐
                │   Application LB   │
                └─────────┬──────────┘
                          │
         ┌────────────────┴────────────────┐
         │                                 │
 Internal Office/VPN IPs             Public Users
         │                                 │
         ▼                                 ▼
   GREEN Target Group               BLUE Target Group
         │                                 │
    ECS GREEN Tasks                  ECS BLUE Tasks

The canary routing rule gets evaluated first.

If the request source IP matches internal CIDRs, traffic goes to GREEN.

Everything else falls back to BLUE.

Terraform Structure

I kept the Terraform layout modular so it could be reused across multiple services.

.
├── main.tf
├── variables.tf
├── outputs.tf
├── env/
│   ├── backend.hcl
│   └── terraform.tfvars
├── modules/
│   ├── vpc/
│   ├── iam/
│   ├── alb/
│   ├── ecs-cluster/
│   └── ecs-blue-green-service/
└── scripts/
    └── zero-downtime-test.sh

Each ECS service gets:

BLUE ECS service
GREEN ECS service
BLUE target group
GREEN target group
production listener rule
optional canary listener rule

ALB Listener Rule Logic

The entire deployment behavior depends on ALB listener priorities.

The canary listener rule gets evaluated first.

If the request source IP matches internal CIDRs, traffic gets forwarded to GREEN.

resource "aws_lb_listener_rule" "canary" {
  count    = var.activate_canary ? 1 : 0
  priority = 99

  condition {
    source_ip {
      values = var.canary_source_ips
    }
  }

  condition {
    host_header {
      values = ["nginx.jayakrishnayadav.cloud"]
    }
  }

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.green.arn
  }
}

The production rule remains below it:

resource "aws_lb_listener_rule" "production" {
  priority = 100

  condition {
    host_header {
      values = ["nginx.jayakrishnayadav.cloud"]
    }
  }

  action {
    type             = "forward"
    target_group_arn = local.active_target_group
  }
}

That’s it.

No weighted routing.

No lifecycle hooks.

Just listener priorities.

Real Deployment Workflow

This wasn’t built as a theoretical architecture exercise.

I tested the rollout flow directly from Terraform while continuously validating traffic behavior against live ECS Fargate services.

Terraform initialization:

terraform init -backend-config=env/backend.hcl

Deployment apply:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

During canary validation, I continuously verified my public IP:

curl ifconfig.me

That mattered because the ALB source-IP rule decides whether traffic reaches:

BLUE
GREEN

Once my IP matched the configured canary CIDRs, traffic immediately started routing to GREEN.

Deployment Flow

The nice part about this setup is that everything becomes variable-driven.

Step 1 — Normal Production State

BLUE handles all production traffic.

GREEN remains scaled down.

enable_canary   = false
activate_canary = false
promote_to_all  = false

Apply:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

Result:

BLUE active
GREEN inactive
minimal Fargate cost

Step 2 — Start GREEN Tasks

Now we start the GREEN environment.

enable_canary   = true
activate_canary = false
promote_to_all  = false

Apply again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

At this stage:

GREEN tasks start
ECS health checks complete
ALB target registration completes
no production traffic reaches GREEN yet

Users never hit partially starting containers.

Step 3 — Internal Canary Validation

Now we enable canary routing.

enable_canary   = true
activate_canary = true
promote_to_all  = false

Apply again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

Now:

internal office/VPN users hit GREEN
public users continue hitting BLUE

This became the most valuable phase of the deployment workflow.

Because now:

QA validates production behavior
developers inspect logs
authentication flows get tested
sessions and redirects get verified

while customers remain completely unaffected.

Internal Canary Routing

This is the ALB listener rules view while canary routing is enabled.

The priority 99 rule matches internal source IPs and forwards them to GREEN, while everyone else continues hitting BLUE.

Step 4 — Promote GREEN to Production

Once validation looks good:

enable_canary   = true
activate_canary = false
promote_to_all  = true

Apply again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

Now:

production listener switches to GREEN
BLUE scales down
all users see the new version

No downtime occurs.

Traffic simply moves from one target group to another.

Verifying Zero Downtime

I didn’t want to assume the deployment was safe.

I wanted to verify it continuously during rollout.

So I used a simple curl-based validation script that continuously hit both applications while traffic shifted between BLUE and GREEN.

for i in {1..100}
do
  for url in \
    "https://nginx.jayakrishnayadav.cloud/" \
    "https://apache.jayakrishnayadav.cloud/"
  do
    response=$(curl -k -s -w " HTTPSTATUS:%{http_code}" "$url")

    body=${response% HTTPSTATUS:*}
    status=${response##*HTTPSTATUS:}

    if [[ $body == *"BLUE - v"* ]]; then
      color="BLUE"
    elif [[ $body == *"GREEN - v"* ]]; then
      color="GREEN"
    else
      color="UNKNOWN"
    fi

    echo "Run: $i | URL: $url | Status: $status | Version: $color"
  done
done

Output during deployment:

You can clearly see:

HTTP 200 responses throughout deployment
no failed requests
no 503s
clean traffic movement from BLUE to GREEN

That confirmed the deployment was genuinely zero downtime.

Production Promotion View

After promotion:

the canary rule disappears
the production listener points directly to GREEN
all traffic reaches the new version
BLUE scales down to zero

Clean and simple.

Rollback

Rollback became extremely simple.

I just reverted the Terraform variables:

enable_canary   = false
activate_canary = false
promote_to_all  = false

Apply Terraform again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

ALB immediately routes traffic back to BLUE.

The rollback process stays predictable because traffic switching is entirely controlled through ALB listener rules.

HTTPS Configuration

The ALB uses ACM certificates for HTTPS.

Listeners:

Port 80 → redirect to HTTPS
Port 443 → production traffic
optional internal listener → restricted to internal CIDRs

Example:

test_listener_allowed_cidrs = [
  "160.30.39.198/32"
]

That keeps internal preview traffic private while still using the same production infrastructure.

Cost Optimization

One thing I specifically wanted to avoid was permanently doubling infrastructure cost.

Normal state:

only BLUE tasks run

Deployment window:

BLUE + GREEN both run temporarily

After promotion:

BLUE scales down again

So infrastructure cost only increases briefly during deployments.

Final Thoughts

This project started because I wanted a very practical deployment workflow:

Internal users should validate the new version on the actual production URL before customers ever see it.

Once I implemented that using ALB listener priorities and source IP routing, I realized I no longer really needed CodeDeploy for this workflow.

The end result became:

simpler
easier to operate
easier to rollback
easier to debug
easier to reason about
fully zero downtime

And because everything is Terraform-driven, the deployment process stays reproducible and predictable.

GitHub Repository

Full Terraform implementation:

https://github.com/jayakrishnayadav24/ecs-blue-green-deployment/tree/canary

I Replaced 47 DevOps Scripts With One AI Agent — Here’s What Happened

POTHURAJU JAYAKRISHNA YADAV — Fri, 17 Apr 2026 10:15:34 +0000

The Hook: I Was Wrong About Automation

I thought I automated DevOps.

I had 47 deployment scripts.

Then I started replacing them with an AI agent —
and most scripts became unnecessary.

Not by following instructions.
By making decisions.

And the 2 AM debugging stopped.

Note: The code shown here is simplified for clarity.
The GitHub repo contains a more modular implementation.

That's when I realized:

I wasn't automating.
I was hardcoding decisions.

I gave Claude a list of AWS tools and one instruction: "Deploy this app."

No hardcoded logic.
No decision trees.

Just: describe the goal, Claude figures out how.

Deployment time: 3 hours → in minutes for most cases.

(Built this while deploying real workloads on AWS: Docker, ECS, EC2, IAM.)

🎯 Is This For You?

This post is for:

DevOps engineers with 10+ deployment scripts
Platform engineers building internal developer platforms
Anyone exploring AI agents beyond chatbots
Teams on AWS (Docker, EC2, ECS, IAM)

If manual deployment takes > 30 minutes, read this.

✨ What's Possible

Old way (manual):

aws ecs create-cluster --cluster-name prod
aws ecs register-task-definition --family myapp ...
# [50+ commands, 3 hours, manual debugging]

New way (agent):

agent.run("Deploy FastAPI with PostgreSQL, auto-scale to 20, "
          "minimal IAM perms, CloudWatch monitoring")

# ✅ Done in minutes for most cases, with significantly reduced debugging

No scripts.

Just Claude thinking out loud about your infrastructure.

🚀 How It Actually Works

You describe the goal (natural language):

"Deploy my app on 5 ECS tasks, auto-scale to 20 on high CPU"

Claude gets these tools:

dispatcher = {
    "ecs__create_cluster": ecs.create_cluster,
    "ecs__register_task": ecs.register_task_definition,
    "ecs__create_service": ecs.create_service,
    "iam__create_role": iam.create_role,  # For permissions
}

Claude reasons autonomously:

"User wants ECS deployment.
 I need to:
 1. Check if cluster exists
 2. Register task definition
 3. Create service with 5 tasks
 4. Setup auto-scaling
 5. Verify it's running"

It executes:

Call ecs__create_cluster() → Get cluster ARN
Call ecs__register_task() → Get task definition
Call ecs__create_service() → Get running tasks
Call ecs__setup_autoscaling() → Confirmed
Return: "All 5 tasks running, auto-scaling 5-20"

** Minimal manual intervention in most cases. Just reasoning + execution + feedback loops.**

⚠️ The Problem I Started With

I had:

deploy_docker.py — 150 lines of Docker logic
deploy_ec2.py — 200 lines of EC2 logic
deploy_ecs.py — 300 lines of ECS logic
3 routers trying to chain them together
0 way to handle "deploy to both EC2 AND ECS"

Every new service = new script. Every new workflow = rewrite everything.

Solution: Reduce rigid scripts — let the agent handle orchestration logic dynamically.

Instead of routing to specific tools, give Claude all available tools and let it decide which ones to use, in which order, adapting as it goes.

# All tools in one place
tools = {
    "docker__run": run_container,
    "ec2__create": create_instance,
    "ecs__deploy": create_service,
    "iam__create_role": create_role,
    # ... more tools
}

# Let Claude orchestrate
agent.run("Deploy with Docker locally, then scale to ECS production")
# Claude figures out: Docker first, then ECS, then IAM for permissions

🏗️ The Architecture (3 Layers)

┌──────────────────────────────┐
│ User Goal (natural language) │  
│ "Deploy scalable production  │
│  stack with auto-scaling"    │
└──────────┬───────────────────┘
           │
           ▼
    ┌─────────────────────────────────────────────┐
    │  Claude (Bedrock)                           │
    │  ← Reads goal + available tools             │
    │  ← Decides sequence of actions              │
    │  ← Adapts when things fail                  │
    └──────────────┬────────────────────────────┘
                   │
    ┌──────────────┴──────────────┐
    │                             │
    ▼                             ▼
┌─────────────┐  ┌──────────────────┐
│ AWS APIs    │  │ Conversation     │
│ (via boto3) │  │ Memory (DynamoDB)│
│ ← Executes  │  │ ← Recalls setup  │
│   decisions │  │   from last week │
└─────────────┘  └──────────────────┘
    │                    │
    └────────┬───────────┘
             │
    ┌────────▼─────────┐
    │ Actual Resources │
    │ EC2, ECS, Docker │
    │      IAM, etc    │
    └──────────────────┘

3 moving parts:

Claude — Reasons about the task
Tools — Execute AWS API calls
Memory — Remember past deployments for coherence

🔧 The Setup (Code Foundation)

Here's the entire base system in ~100 lines. Full code on GitHub.

import boto3
from agents.memory import load_history, save_message

MODEL = "apac.amazon.nova-lite-v1:0"  # Fast, cheap 

class BaseAgent:
    """Foundation for all agents (Docker, EC2, ECS, IAM)"""
    AGENT_KEY = None  # "docker", "ec2", etc.
    SYSTEM_PROMPT = ""  # Agent's personality
    CAPABILITIES = []  # What this agent can do

    def __init__(self, name: str, region: str = "us-east-1"):
        self.name = name
        self.session = boto3.Session(region_name=region)
        self._bedrock = None

    @property
    def bedrock(self):
        """Lazy load Bedrock connection (only when needed)"""
        if not self._bedrock:
            self._bedrock = self.session.client("bedrock-runtime")
        return self._bedrock

    def run(self, task: str, session_id: str = None) -> dict:
        """The agentic loop: think → decide → act → repeat"""
        tools = self.get_tools()
        dispatcher = self.get_dispatcher()
        messages = load_history(session_id) + [{"role": "user", "content": [{"text": task}]}]

        for _ in range(10):  # Max 10 iterations to prevent infinite loops
            # Ask Claude what to do
            response = self.bedrock.converse(
                modelId=MODEL,
                system=self.SYSTEM_PROMPT,
                messages=messages,
                tools=tools,
                toolUseDepth=2,
            )

            # Is Claude done?
            if response["stopReason"] == "endTurn":
                return {"status": "SUCCESS", "message": response["content"]}

            # Execute tools Claude wants to call
            tool_results = []
            for tool_use in response["content"]:
                tool_name = tool_use["toolUse"]["name"]
                try:
                    result = dispatcher<a href="tool_use["toolUse"]["input"]">tool_name</a>
                except Exception as e:
                    result = {"error": str(e), "status": "failed"}

                tool_results.append({"toolUseId": tool_use["toolUse"]["toolUseId"],
                                    "content": [{"json": result}]})

            # Add Claude's decision + results to conversation
            messages.append({"role": "assistant", "content": response["content"]})
            messages.append({"role": "user", "content": [{"toolResult": tr} for tr in tool_results]})

            # Save for future reference
            if session_id:
                save_message(session_id, messages[-1])

        return {"status": "FAILED", "reason": "Max iterations reached"}

What's happening:

Lazy loading (@property bedrock): Don't connect to Claude until needed
Tool loop: Call tools, capture results, show Claude the output
Error handling: Don't crash—tell Claude what failed, let it adapt
Memory: Save each turn so Claude remembers next week

🚀 Agents in Action: 3 Real Examples

DockerAgent: Deploy Locally

class DockerAgent(BaseAgent):
    AGENT_KEY = "docker"
    SYSTEM_PROMPT = """You manage Docker containers.
Rules: Pull image first, check if container exists, use sensible defaults."""

    CAPABILITIES = [
        {"name": "list_containers", "description": "List running containers"},
        {"name": "run_container", "description": "Pull and run image"},
        {"name": "stop_container", "description": "Stop a container"},
    ]

Usage:

docker_agent = DockerAgent()
docker_agent.run("Deploy FastAPI on port 8000 with health check")
# Claude calls: list_containers → run_container → confirms it's running

EC2Agent: Scale to Cloud

class EC2Agent(BaseAgent):
    AGENT_KEY = "ec2"
    SYSTEM_PROMPT = """You manage EC2 instances.
Rules: Tag for organization, verify security groups, create new→test→retire old."""

    CAPABILITIES = [
        {"name": "describe_instances", "description": "List instances"},
        {"name": "create_instance", "description": "Launch new instance"},
        {"name": "stop_instance", "description": "Stop instance"},
    ]

Usage:

ec2_agent = EC2Agent()
ec2_agent.run("Create 2 t2.micro instances, tag as app-server")
# Claude: checks if instances exist → creates 2 → returns IPs → confirms running

ECSAgent: Production Auto-Scaling

class ECSAgent(BaseAgent):
    AGENT_KEY = "ecs"
    SYSTEM_PROMPT = """You manage ECS services at production scale.
Rules: Task definition first, use Fargate, always configure auto-scaling."""

    CAPABILITIES = [
        {"name": "create_cluster", "description": "Create ECS cluster"},
        {"name": "register_task_definition", "description": "Register blueprint"},
        {"name": "create_service", "description": "Deploy service"},
        {"name": "setup_autoscaling", "description": "Configure scaling rules"},
    ]

Usage:

ecs_agent = ECSAgent()
ecs_agent.run("Deploy with 5 tasks, auto-scale to 20 on high CPU, health monitoring")
# Claude orchestrates: cluster → task definition → service → autoscaling → verification

Real Usage: From Local to Production

Scenario: Deploy FastAPI from laptop to production in minutes for most cases, with significantly reduced debugging.

Stage 1: Test locally

docker_agent.run("Run myapp:latest on port 8000", session_id="prod_001")
# ✅ Docker container running

Stage 2: Scale to cloud

ec2_agent.run("Create 2 instances for load balancing", session_id="prod_001")
# ✅ 2 EC2 instances up (same session = Claude remembers port 8000)

Stage 3: Auto-scaling production

ecs_agent.run("Deploy with 5-20 auto-scaling, CloudWatch monitoring", session_id="prod_001")
# ✅ 5 ECS tasks running, auto-scaling 5-20 based on CPU

All with the same session_id: Claude remembers the image name, port, and configuration from Stage 1. Everything connects seamlessly.

📚 Memory: Agents Remember

# Day 1
agent.run("Deploy on Docker with port 8000", session_id="user_123")

# Week later
agent.run("Move this to ECS for production", session_id="user_123")
# Claude reads history from DynamoDB:
# "I remember this app used port 8000. I'll keep that for ECS too."

DynamoDB stores every conversation turn. Claude recalls past decisions and uses them for coherence.

🛡️ Production-Ready Code Patterns

This is what the user feedback emphasized. Real code needs safety.

Pattern 1: Error Handling (Don't Crash, Recover)

# ❌ Wrong: crashes on error
result = dispatcher[tool_name](tool_input)

# ✅ Right: tell Claude about the error
try:
    result = dispatcher[tool_name](tool_input)
except Exception as e:
    result = {
        "error": str(e),
        "status": "failed",
        "request_id": request_id,  # For debugging
    }
# Claude sees the error and tries a different approach

Why this matters: If Docker pull fails, don't give up. Tell Claude: "Pull failed, but let me check if the image is locally cached." Claude adapts.

Pattern 2: Timeouts (Prevent Hanging)

from functools import wraps
import signal

def with_timeout(seconds=30):
    """Prevent tools from running forever"""
    def decorator(func):
        def wrapper(*args, **kwargs):
            def handler(signum, frame):
                raise TimeoutError(f"Tool execution exceeded {seconds}s")

            signal.signal(signal.SIGALRM, handler)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
                signal.alarm(0)  # Cancel timeout
                return result
            except TimeoutError as e:
                return {"error": str(e), "status": "timeout"}
        return wrapper
    return decorator

@with_timeout(seconds=60)
def create_instance(params):
    """If EC2 creation hangs, timeout after 60 seconds"""
    ec2 = boto3.client("ec2")
    return ec2.create_instances(**params)

Pattern 3: Input Validation (Prevent Bad Requests)

def create_instance(params):
    """Validate before executing"""
    # Check required fields
    required = ["ImageId", "InstanceType"]
    for field in required:
        if field not in params:
            return {"error": f"Missing required field: {field}", "status": "failed"}

    # Validate instance type
    valid_types = ["t2.micro", "t2.small", "t3.medium"]
    if params["InstanceType"] not in valid_types:
        return {
            "error": f"InstanceType must be one of {valid_types}",
            "status": "failed"
        }

    # Now safe to execute
    ec2 = boto3.client("ec2")
    return ec2.create_instances(**params)

Pattern 4: Audit Logging (Proof for Compliance)

import json
from datetime import datetime

def log_action(request_id: str, user: str, action: str, params: dict, result: dict):
    """Log everything for audits"""
    log_entry = {
        "request_id": request_id,
        "timestamp": datetime.utcnow().isoformat(),
        "user": user,
        "action": action,
        "params": json.dumps(params),  # What was requested
        "result_status": result.get("status"),
        "result_code": result.get("error"),
    }

    # Save to CloudWatch Logs or DynamoDB
    dynamodb.put_item(TableName="agent_audit", Item=log_entry)

# In the tool dispatcher:
result = dispatcher[tool_name](tool_input)
log_action(request_id, user_id, tool_name, tool_input, result)

Pattern 5: Cost Guards (Don't Deploy Expensive Mistakes)

MONTHLY_BUDGET = 1000  # $1000/month limit

def estimate_cost(action: str, params: dict) -> float:
    """Estimate AWS cost before executing"""
    if action == "create_instance":
        instance_type = params.get("InstanceType", "t2.micro")
        hourly_rates = {
            "t2.micro": 0.012,
            "t2.small": 0.023,
            "t3.medium": 0.042,
        }
        months_running = 1
        return hourly_rates.get(instance_type, 0) * 730 * months_running

    elif action == "create_rds":
        # RDS: ~$400/month for 20GB
        return 400

    return 0

# In the runnable:
estimated = estimate_cost(tool_name, tool_input)
if estimated > MONTHLY_BUDGET:
    return {
        "error": f"Cost ${estimated:.2f} exceeds budget ${MONTHLY_BUDGET}",
        "status": "rejected",
    }
# Only execute if under budget
result = dispatcher[tool_name](tool_input)

Pattern 6: Role-Based Access (Security Boundaries)

# Define who can do what
USER_PERMISSIONS = {
    "admin": ["create_instance", "delete_instance", "create_role", "delete_role"],
    "developer": ["create_instance", "stop_instance", "deploy_to_ecs"],
    "readonly": ["describe_instances", "list_containers", "get_logs"],
}

def check_permission(user_role: str, action: str) -> bool:
    """Prevent unauthorized actions"""
    allowed = USER_PERMISSIONS.get(user_role, [])
    if action not in allowed:
        return False
    return True

# Before executing:
if not check_permission(user_role, tool_name):
    return {
        "error": f"User role '{user_role}' cannot perform '{tool_name}'",
        "status": "permission_denied",
    }
result = dispatcher[tool_name](tool_input)

✅ Why Agents > Scripts

	Scripts	Agents
New workflow	Rewrite code	Claude adapts
Error recovery	Crashes	Tries alternatives
Reasoning	None (hardcoded)	Full decision log
Maintenance	Grows linearly	One framework
Learning	Manual proof	Automatic audit trail

🎯 Getting Started

Just 3 commands:

git clone https://github.com/jayakrishnayadav24/ai-agents
cd ai-agents
pip install -r requirements.txt

Then try:

from agents.docker_agent import DockerAgent

agent = DockerAgent()
response = agent.run(
    "Deploy FastAPI app on port 8000 with health check"
)
print(response)

That's it. Claude handles the rest.

📚 What's Next (Part 2)

This was the concept. Github:

✅ Full working code (all agents)
✅ DynamoDB setup for memory
✅ Deploying agents to Lambda
✅ Real production patterns (request tracing, cost estimation)
✅ Demo with actual deployments

⚠️ Where This Breaks

Complex compliance environments (manual approvals still needed)
Cost estimation is approximate
Requires well-defined tools (garbage in → garbage out)

The Bottom Line

Before: 47 scripts, 3 hours/deployment, constant debugging

After: 1 agent framework, in minutes for most cases, with significantly reduced debugging/deployment.

Why? Because Claude doesn't follow scripts. Claude plans and executes the steps based on the goal.

It adapts. It learns. It remembers.

That's not automation anymore. That's the future of infrastructure.

How many deployment scripts are you still maintaining?

10+? 20+? 50+?

I want to see how bad this problem is. Drop a comment 👇

Your feedback shapes Part 2.

The User Interface & Ground-Truth Testing

POTHURAJU JAYAKRISHNA YADAV — Tue, 14 Apr 2026 10:49:18 +0000

Part 5 of 5 | ← Part 4 | Complete Series

Streamlit Overview

Streamlit lets you build data apps in pure Python—no HTML, CSS, or JavaScript needed.

import streamlit as st
from datetime import datetime
import pandas as pd
import requests

# Page config
st.set_page_config(
    page_title="IPL AI Assistant",
    page_icon="🏏",
    layout="wide",
)

# Title
st.title("🏏 IPL AI Assistant")
st.subtitle("Predictions + Q&A Powered by ML")

# Tabs
tab1, tab2, tab3 = st.tabs(["💬 Chat", "🎯 Predict", "📊 Metrics"])

Tab 1: Chat Interface

with tab1:
    st.header("Ask Anything")

    # Initialize session state
    if "messages" not in st.session_state:
        st.session_state.messages = []

    # Display chat history
    for message in st.session_state.messages:
        with st.chat_message(message["role"]):
            st.markdown(message["content"])

    # User input
    if user_input := st.chat_input("Ask about IPL..."):
        st.session_state.messages.append({"role": "user", "content": user_input})

        with st.chat_message("user"):
            st.markdown(user_input)

        # Call backend
        try:
            response = requests.post(
                "http://localhost:8000/chat",
                json={"message": user_input},
                timeout=5,
            )
            response.raise_for_status()

            result = response.json()
            assistant_message = result.get("message", "I couldn't understand that.")

            st.session_state.messages.append({
                "role": "assistant",
                "content": assistant_message,
            })

            with st.chat_message("assistant"):
                st.markdown(assistant_message)

        except Exception as e:
            st.error(f"❌ Backend error: {str(e)}")

Key Concepts:

Session State: st.session_state.messages persists across reruns
- When user submits a message, Streamlit reruns the entire script
- Session state preserves conversation history
- Without it: chat history disappears on each input
Chat Message: st.chat_message() renders messages with role-based styling
- "user" = right-aligned, blue background
- "assistant" = left-aligned, gray background
Chat Input: st.chat_input() provides a textbox with submission handling
- Returns None until user submits
- Automatically clears after submission

Tab 2: Prediction Interface

with tab2:
    st.header("Match Prediction Simulator")

    col1, col2 = st.columns(2)

    with col1:
        st.subheader("Teams")
        batting_team = st.selectbox(
            "Batting Team:",
            [
                "Mumbai Indians",
                "Chennai Super Kings",
                "Royal Challengers Bangalore",
                "Kolkata Knight Riders",
                # ... all 10 teams
            ],
        )
        bowling_team = st.selectbox(
            "Bowling Team:",
            options=["All teams except batting team"],
            index=0,
        )

    with col2:
        st.subheader("Venue")
        venue = st.text_input("Ground name:", "Wankhede")

    st.subheader("Pre-Match Form")
    col1, col2, col3, col4 = st.columns(4)

    with col1:
        h2h_rate = st.slider(
            "H2H Win Rate (Batting Team)",
            0.0, 1.0, 0.5, 0.05,
        )

    with col2:
        overall_rate = st.slider(
            "Overall Win Rate",
            0.0, 1.0, 0.5, 0.05,
        )

    with col3:
        venue_rate = st.slider(
            "Venue Win Rate",
            0.0, 1.0, 0.5, 0.05,
        )

    with col4:
        rolling_rate = st.slider(
            "Last 5 Matches Win Rate",
            0.0, 1.0, 0.5, 0.05,
        )

    st.subheader("Toss")
    col1, col2 = st.columns(2)

    with col1:
        toss_win = st.radio("Who won toss?", ["Batting Team", "Bowling Team"])
        toss_win = 1 if toss_win == "Batting Team" else 0

    with col2:
        toss_choice = st.radio("Toss choice?", ["Bat", "Field"])
        toss_choice = toss_choice.lower()

    # Predict button
    if st.button("🎯 Predict Match Outcome", use_container_width=True):
        try:
            response = requests.post(
                "http://localhost:8000/predict",
                json={
                    "batting_team": batting_team,
                    "bowling_team": bowling_team,
                    "venue": venue,
                    "h2h_rate": h2h_rate,
                    "overall_rate": overall_rate,
                    "venue_rate": venue_rate,
                    "rolling_rate": rolling_rate,
                    "toss_win": toss_win,
                    "toss_choice": toss_choice,
                },
                timeout=5,
            )
            response.raise_for_status()

            result = response.json()
            winner = result["winner"]
            confidence = result["confidence"]

            st.success(
                f"### 🏆 {winner} wins!\n"
                f"**Confidence:** {confidence:.1%}"
            )

            # Show prediction breakdown
            st.info(
                f"**Model:** {result['model']}\n\n"
                f"**Reasoning:**\n"
                f"- H2H: {h2h_rate:.0%}\n"
                f"- Form: {rolling_rate:.0%}\n"
                f"- Venue: {venue_rate:.0%}\n"
            )

        except Exception as e:
            st.error(f"❌ Prediction failed: {str(e)}")

Key UI Patterns:

st.selectbox() — Dropdown selector
st.slider() — Range input (0.0-1.0)
st.radio() — Single-choice radio buttons
st.columns() — Grid layout (col1, col2, etc.)
st.button() — Form submission
st.success/error/info() — Colored alerts

Tab 3: Metrics & Transparency

with tab3:
    st.header("Model Performance")

    # Load metrics
    import json
    with open("models/metrics.json") as f:
        metrics = json.load(f)

    col1, col2, col3 = st.columns(3)
    col1.metric("Test Accuracy", f"{metrics['accuracy']:.1%}")
    col2.metric("Precision", f"{metrics['precision']:.1%}")
    col3.metric("Recall", f"{metrics['recall']:.1%}")

    st.subheader("Confusion Matrix")
    st.image("models/confusion_matrix.png", use_column_width=True)

    st.subheader("Feature Importance")
    importance_df = pd.DataFrame({
        "Feature": ["h2h_rate", "rolling_rate", "venue_rate", ...],
        "Importance": [0.32, 0.28, 0.18, ...],
    }).sort_values("Importance", ascending=False)

    st.bar_chart(importance_df.set_index("Feature"))

    st.subheader("Q&A Engine")
    st.info(
        f"**Total Q&A Pairs:** 42,523\n\n"
        f"**Vocabulary Size:** 18,394\n\n"
        f"**Match Strategy:** TF-IDF + Cosine Similarity (threshold: 0.15)\n\n"
        f"**Coverage:** {(42523 / 50000 * 100):.1f}% of expected cricket topics"
    )

Testing: The Foundation of Trust

Good tests = confidence in deployment.

Test Structure

# tests/test_qa.py

import pytest
import pandas as pd
from src.build_qa_model import answer_question
from joblib import load

# Load test data
test_df = pd.read_csv("datasets/ipl_2008_2024_complete.csv")
qa_model = load("models/qa_model.joblib")

# Extract Q&A components
tfidf = qa_model["tfidf"]
Q_matrix = qa_model["Q_matrix"]
answers = qa_model["answers"]

Test 1: Specific Match Facts

def test_match_lookup():
    """Can we answer specific match questions?"""
    questions = [
        "Who won the match on 2024-04-01 between MI and RR?",
        "How many runs were scored by MI in 2024-04-01?",
        "What was the result of MI vs RR on 2024-04-01?",
    ]

    for question in questions:
        answer, score = answer_question(
            question, tfidf, Q_matrix, answers, threshold=0.15
        )

        assert answer is not None, f"Failed on: {question}"
        assert len(answer) > 10, f"Answer too short: {answer}"
        assert score > 0.15, f"Confidence too low: {score}"

Why data-driven?

Not hardcoded: "answer == 'Mumbai Indians wins'"
CSV-based: Pulls real facts from dataset
Robust: Works even if answer phrasing changes

Test 2: Aggregate Statistics

def test_most_wins():
    """Can we retrieve aggregate stats?"""
    questions = [
        "Which team has won most matches?",
        "Who has most IPL titles?",
        "Team with highest win percentage?",
    ]

    for question in questions:
        answer, score = answer_question(
            question, tfidf, Q_matrix, answers, threshold=0.15
        )

        # Verify answer is a valid team name
        assert answer is not None
        valid_teams = ["Mumbai Indians", "CSK", "RCB", ...]
        assert any(team in answer for team in valid_teams)

Test 3: Head-to-Head

def test_head_to_head():
    """Can we answer H2H questions?"""
    questions = [
        "Head-to-head record between MI and CSK?",
        "Does MI have winning record vs KKR?",
        "Who dominates MI vs RR?",
    ]

    for question in questions:
        answer, score = answer_question(
            question, tfidf, Q_matrix, answers, threshold=0.15
        )

        assert answer is not None
        # H2H answers contain numbers (win counts)
        assert any(char.isdigit() for char in answer)

Test 4: Threshold Behavior

def test_threshold_protects_low_confidence():
    """Low-confidence matches are rejected."""
    nonsense = "xyzabc qwerty asdfgh"  # Gibberish

    answer, score = answer_question(
        nonsense, tfidf, Q_matrix, answers, threshold=0.15
    )

    # Model shouldn't hallucinate
    assert answer is None, f"Got answer for nonsense: {answer}"
    assert score < 0.15

Test 5: ML Model Predictions

def test_model_predictions():
    """Can we predict match winners?"""
    from src.train import normalize_teams, engineer_features
    from joblib import load

    ml_model = load("models/model.joblib")
    pipeline = ml_model["pipeline"]

    # Create a test case
    test_row = test_df.iloc[0].copy()  # Use real match
    test_row["date"] = "2023-05-01"  # Test on recent data

    # Engineer features
    features_df = engineer_features(
        test_df[test_df["date"] < "2023-01-01"],  # Historical
        test_row,
    )

    # Predict
    prediction = pipeline.predict(features_df)
    prob = pipeline.predict_proba(features_df)

    assert prediction[0] in [0, 1]  # Binary classification
    assert 0 <= max(prob[0]) <= 1  # Valid probability
    assert abs(sum(prob[0]) - 1.0) < 0.01  # Probabilities sum to 1

Test 6: Feature Sanity

def test_feature_ranges():
    """Are features in valid ranges?"""
    from src.features import engineer_features

    # Get one match
    test_match = test_df.iloc[0]

    # Engineer
    features = engineer_features(test_df, test_match)

    # Check rates (should be 0-1)
    assert 0 <= features["h2h_rate"].values[0] <= 1
    assert 0 <= features["overall_rate"].values[0] <= 1
    assert 0 <= features["venue_rate"].values[0] <= 1
    assert 0 <= features["rolling_rate"].values[0] <= 1

    # Check binary fields
    assert features["toss_win"].values[0] in [0, 1]

Test 7: No Data Leakage

def test_future_data_not_used():
    """Ensure no future data affects past predictions."""
    from src.features import engineer_features

    # Engineer a 2023 match using only 2022 history
    before_date = "2023-01-01"
    hist_data = test_df[test_df["date"] < before_date]

    test_match = test_df[
        (test_df["date"] >= before_date) & 
        (test_df["date"] < "2023-02-01")
    ].iloc[0]

    features = engineer_features(hist_data, test_match)

    # Features should only use hist_data, not test_match
    # (This is enforced in engine_features with before_date guards)
    assert features is not None

    # Verify: no 2023 data leaked into 2022 rates
    assert all(hist_data["date"] < before_date)

Running Tests

# Install test dependencies
pip install pytest

# Run all tests
pytest tests/

# Run with verbose output
pytest tests/ -v

# Run specific test
pytest tests/test_qa.py::test_match_lookup -v

# Coverage report
pytest tests/ --cov=src --cov-report=html

Deployment

Option 1: Streamlit Cloud

# Push to GitHub
git push origin main

# Link in Streamlit Cloud (streamlit.io/cloud)
# - Select repository
# - Select app.py
# - Auto-deploys on push

Live in 2 minutes, updates automatically.

Option 2: Docker

FROM python:3.11

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

EXPOSE 8501

CMD ["streamlit", "run", "src/streamlit_app.py", "--server.port=8501"]

docker build -t ipl-app .
docker run -p 8501:8501 ipl-app

Visit http://localhost:8501

Option 3: Heroku / Railway

# Deploy with one command
heroku create ipl-ai-assistant
git push heroku main

App runs at ipl-ai-assistant.herokuapp.com

Performance Monitoring

# Log predictions to dataframe
import logging
from datetime import datetime

logging.basicConfig(filename="predictions.log", level=logging.INFO)

@st.cache_data
def get_prediction_logs():
    return pd.read_csv("predictions.log")

# Dashboard
st.line_chart(
    get_prediction_logs()
    .groupby("hour")["confidence"]
    .mean()
)

Track:

Average confidence over time
Common queries
Backend latency

Common Issues & Solutions

Issue	Solution
"Connection refused"	Backend not running on localhost:8000
Chat history disappears	Use st.session_state, not regular variables
Predictions slow	Enable model caching, use lazy loading
Tests fail on new data	Read from CSV, not hardcoded values
Threshold too strict	Lower from 0.15 to 0.10 for more results

Summary: The Complete System

Component	Purpose	Technology
Feature Engineering	Calculate pre-match metrics	pandas, numpy
ML Model	Predict match winners	scikit-learn
Q&A Engine	Answer cricket questions	TF-IDF, cosine similarity
FastAPI Backend	Intelligent routing, lazy loading	FastAPI, uvicorn
Streamlit Frontend	Chat, prediction, metrics	Streamlit
Testing	Verify correctness	pytest, CSV-based assertions

You now have:

✅ Production-ready predictions (61.8% accuracy)

✅ Intelligent Q&A with 42K learning pairs

✅ Low-latency API (<20ms per request)

✅ Smooth UI with session persistence

✅ 22 tests ensuring reliability

✅ Multiple deployment options

What's Next?

Deploy this and message me your results! 🚀

Repository: https://github.com/jayakrishnayadav24/ipl-ai-assistant

Series Complete 🎉

← Part 4: Backend Routing | ← Return to Series

Built with 💚 for cricket fans. Questions? DM me!

Smart Backend Routing — Predictions vs Q&A Intelligently

POTHURAJU JAYAKRISHNA YADAV — Tue, 14 Apr 2026 10:47:09 +0000

Part 4 of 5 | ← Part 3 | Part 5 → | View Series

The Hook: Two Systems, One Decision

I had two systems:

One predicts match winners.

One answers 42,000 questions.

Now the real problem:

How do you decide which one to use?

This is where most systems fail, Not because of bad models. Because of bad decisions.

Users call /predict for Q&A. They call /qa for predictions. Chaos.

⚠️ Mistake I Almost Made: Separate APIs

At first, I built two separate endpoints:

/predict  → Only for predictions
/qa       → Only for Q&A

Users had to decide which one to call.

It was confusing. Unintuitive. Wrong.

That's when I realized:

The system should decide — not the user.

The Architecture

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI(
    title="IPL AI Assistant",
    description="Predictions + Q&A for cricket",
    version="1.0"
)

# Allow frontend to call backend
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

Lazy Loading: The Secret Sauce

Stop here for a second.

This optimization seems small.

But it changes everything about how your system scales.

Instead of loading models at startup:

# ❌ Wrong way (startup)
model = joblib.load("models/model.joblib")  # 500ms
qa_index = joblib.load("models/qa_model.joblib")  # 400ms
# Server ready: 900ms

# ✅ Right way (lazy)
BUNDLE = None
QA_BUNDLE = None

def get_bundle():
    global BUNDLE
    if BUNDLE is None:
        BUNDLE = joblib.load("models/model.joblib")  # On first use
        print("[INFO] ML model loaded")
    return BUNDLE

def get_qa():
    global QA_BUNDLE
    if QA_BUNDLE is None:
        QA_BUNDLE = joblib.load("models/qa_model.joblib")  # On first use
        print("[INFO] Q&A index loaded")
    return QA_BUNDLE

Benefits:

Faster startup (150ms vs 900ms)
Serverless-friendly (AWS Lambda, Google Cloud)
Optional loading (if users never ask, models never load)

Cost savings: If you deploy 100 servers, save 100 × 750ms = 75 seconds total startup.

Endpoint 1: Health Check

@app.get("/health")
def health_check():
    """Called by load balancers every 10 seconds."""
    return {
        "status": "ok",
        "version": "1.0",
        "timestamp": datetime.utcnow().isoformat()
    }

If this returns 200, traffic flows. If timeout, traffic reroutes. Fast failure detection.

Endpoint 2: Metadata

@app.get("/model-info")
def model_info():
    """Tell clients what models are running."""
    bundle = get_bundle()  # First call triggers load

    return {
        "ml_model": {
            "type": "GradientBoostingClassifier",
            "test_accuracy": 0.618,
            "features": 13,
        },
        "qa_engine": {
            "qa_pairs": 42523,
            "vocabulary": 18394,
            "threshold": 0.15,
        },
    }

Clients call this before predictions: "Are you ready? What accuracy?"

Endpoint 3: Raw Prediction

from pydantic import BaseModel

class PredictionRequest(BaseModel):
    batting_team: str
    bowling_team: str
    venue: str
    h2h_rate: float        # 0.0-1.0
    overall_rate: float    # 0.0-1.0
    venue_rate: float      # 0.0-1.0
    rolling_rate: float    # 0.0-1.0
    toss_win: int          # 0 or 1
    toss_choice: str       # "bat" or "field"

@app.post("/predict")
def predict_winner(request: PredictionRequest):
    """Raw ML model prediction."""
    bundle = get_bundle()

    # Create feature vector
    df = pd.DataFrame([{
        "batting_team": request.batting_team,
        "bowling_team": request.bowling_team,
        "venue": request.venue,
        "h2h_rate": request.h2h_rate,
        # ... other rates
        "toss_win": request.toss_win,
        "toss_choice": request.toss_choice,
    }])

    # Predict
    prediction = bundle["pipeline"].predict(df)[0]
    confidence = bundle["pipeline"].predict_proba(df)[0]
    winner = bundle["label_encoder"].inverse_transform([prediction])[0]

    return {
        "winner": winner,
        "confidence": float(max(confidence)),
        "model": "GradientBoostingClassifier",
    }

Example:

POST /predict
{
  "batting_team": "Mumbai Indians",
  "bowling_team": "Chennai Super Kings",
  "venue": "Wankhede",
  "h2h_rate": 0.54,
  "overall_rate": 0.55,
  "venue_rate": 0.60,
  "rolling_rate": 0.52,
  "toss_win": 1,
  "toss_choice": "bat"
}

Response:
{
  "winner": "Mumbai Indians",
  "confidence": 0.62
}

Endpoint 4: Intelligent Chat Router

Stop for a moment.

This endpoint is doing something powerful.

It's not just forwarding requests. It's making decisions for the user.

This is the magic endpoint:

class ChatRequest(BaseModel):
    message: str

@app.post("/chat")
def handle_chat(request: ChatRequest):
    """
    Routes to prediction or Q&A automatically.
    """
    message = request.message.lower()

    # Step 1: Detect intent
    is_prediction = detect_prediction_intent(message)

    # Step 2: Extract teams
    teams = extract_teams(message)

    # Step 3: Route
    if is_prediction and len(teams) == 2:
        return handle_prediction(teams[0], teams[1])
    else:
        return handle_qa(message)

Intent Detection

def detect_prediction_intent(message: str) -> bool:
    """Is user asking for a prediction?"""
    keywords = [
        "will", "would", "who will", "predict",
        "who wins", "vs", "against", "beat",
    ]
    return any(kw in message for kw in keywords)

Examples:

"Will MI beat KKR?" → True (has "will")
"MI vs CSK?" → True (has "vs")
"How many sixes?" → False
"Does toss matter?" → False

Team Extraction

from difflib import get_close_matches

def extract_teams(message: str) -> list:
    """Find team names, even with typos."""
    bundle = get_bundle()
    team_names = list(bundle["team_index"].keys())

    found_teams = []
    words = message.split()

    # Exact matching
    for team in team_names:
        if team.lower() in message.lower():
            found_teams.append(team)

    # Fuzzy matching (for abbreviations like "mi" → "Mumbai Indians")
    for word in words:
        matches = get_close_matches(word, team_names, n=1, cutoff=0.8)
        if matches and matches[0] not in found_teams:
            found_teams.append(matches[0])

    return found_teams[:2]  # At most 2 teams

Examples:

"Will mi beat kkr?" 
→ Fuzzy match "mi" → "Mumbai Indians"
→ Fuzzy match "kkr" → "Kolkata Knight Riders"
→ Return ["Mumbai Indians", "Kolkata Knight Riders"]

"who wins, royal challengers bangalore vs sunrisers?"
→ Exact match "royal challengers bangalore"
→ Fuzzy match "sunrisers" → "Sunrisers Hyderabad"
→ Return ["RCB", "SRH"]

Prediction Handling

def handle_prediction(team1: str, team2: str) -> dict:
    """Compute features, run model, format response."""
    bundle = get_bundle()
    history_df = bundle["history_df"]

    # Compute live rates from historical data
    h2h = compute_h2h_rate(team1, team2, history_df)
    overall1 = compute_overall_rate(team1, history_df)
    overall2 = compute_overall_rate(team2, history_df)

    # Create prediction request
    pred_req = PredictionRequest(
        batting_team=team1,
        bowling_team=team2,
        venue="TBD",
        h2h_rate=h2h,
        overall_rate=overall1,
        venue_rate=0.5,
        rolling_rate=compute_rolling_rate(team1, history_df),
        toss_win=1,
        toss_choice="bat",
    )

    # Call /predict
    prediction = predict_winner(pred_req)

    # Format as chat response
    return {
        "type": "prediction",
        "message": f"🏆 **{prediction['winner']}** likely wins ({prediction['confidence']:.0%})",
        "confidence": prediction['confidence'],
    }

Q&A Handling

def handle_qa(message: str) -> dict:
    """Call Q&A retrieval system."""
    qa = get_qa()

    answer, score = answer_question(
        message,
        qa["tfidf"],
        qa["Q_matrix"],
        qa["answers"],
        threshold=0.15,
    )

    if answer is None:
        return {
            "type": "qa",
            "message": "🤔 I'm not confident about that.",
            "confidence": score,
        }

    return {
        "type": "qa",
        "message": answer,
        "confidence": score,
    }

Error Handling

@app.post("/chat")
def handle_chat(request: ChatRequest):
    try:
        message = request.message.lower()

        # ... routing logic ...

        return result

    except ValueError as e:
        return {"error": "Invalid input"}

    except Exception as e:
        # Log to monitoring (Sentry, DataDog)
        logger.error(str(e))
        return {"error": "Internal server error"}

Never return stack traces. Log errors, return friendly messages.

Performance Characteristics

Endpoint	Latency	Queries/sec
/health	<1ms	10,000+
/model-info	50ms	200
/predict	10ms	1,000
/chat (ML)	20ms	500
/chat (Q&A)	5ms	2,000

One $5 server handles 10,000 queries/day easily.

Deployment Options

Option 1: Traditional Server

pip install fastapi uvicorn scikit-learn pandas

uvicorn app:app --host 0.0.0.0 --port 8000

API available at http://localhost:8000

Option 2: Docker Container

FROM python:3.11

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

docker build -t ipl-api .
docker run -p 8000:8000 ipl-api

Option 3: AWS Lambda (Serverless)

# With Lambda, lazy loading is critical
# Models load on first invocation, persist across requests

from mangum import Mangum

handler = Mangum(app)

Deploy with SAM or Serverless Framework.

Testing the Backend

# Health check
curl http://localhost:8000/health
# {"status": "ok"}

# Model info
curl http://localhost:8000/model-info
# {"ml_model": {...}, "qa_engine": {...}}

# Prediction
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "batting_team": "Mumbai Indians",
    "bowling_team": "Chennai Super Kings",
    ...
  }'

# Chat
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Will MI beat CSK?"}'

Scaling Tips

Use gunicorn for multiple workers:

   gunicorn -w 4 -k uvicorn.workers.UvicornWorker app:app

Enable caching for Q&A (results don't change):

   from functools import lru_cache

   @lru_cache(maxsize=10000)
   def answer_question_cached(question):
       return answer_question(question, ...)

Load balance across multiple servers:
- Send requests to different servers
- Each loads models independently
- Scales to 1000s of requests/sec

💡 Key Insight: Routing is Intelligence

AI isn't just about models.

It's about deciding which model to use, when to use it, and when NOT to use it.

Your ML algorithm is 61.8% accurate. That's good.

But your routing logic? That's 100% accurate.

Because it knows the user's intent before they do.

🎤 The Mic Drop Lesson

The smartest part of this system isn't the model.

It's the decision of when NOT to use it.

When a user asks "How many sixes?", the system says: "This is Q&A, not ML."

When they ask "Will MI beat CSK?", it says: "This is prediction, not lookup."

That's what makes it feel intelligent.

What's in Part 5 (Frontend + Testing)

Final post: Making it all visible and reliable:

✅ Streamlit frontend (3 tabs)

✅ Session state (conversation persistence)

✅ 22 ground-truth tests

✅ Deployment (Streamlit Cloud, Docker)

✅ Common pitfalls

Sneak preview: Your tests should pull data from CSV, not be hardcoded. Here's why it matters...

This is Part 4 of 5. Subscribe for the finale! 🏏

← Part 3: Q&A Engine | Part 5: Frontend + Testing →

DEV Community: POTHURAJU JAYAKRISHNA YADAV

How a Linux Kernel Upgrade Broke MongoDB on an Azure VM — And How We Fixed It

The incident

Hunting for the cause

What actually happened

The stuff we tried that didn't work

The real fix: boot into the older kernel

A quick word on GRUB

Pointing GRUB at the good kernel

After the reboot

Making sure it doesn't happen again

What I took away from this

Quick reference for next time

Wrapping up

Stop Googling Command Syntax — Your Terminal Already Knows the Answer

The problem with man

tldr — examples, not essays

Getting it

apropos — when you don't even know the command's name

First, a gotcha that got me

Now it actually does something

Narrowing it down

info — the deep dive when you actually need it

Now the Kubernetes version of the same problem: kubectl explain

Start at the top

Drill down with dot notation

Keep going until you hit plain values

Putting it together into a real manifest

A couple of handy variations

Why this matters more than it looks

Quick reference

Wrapping up

Building a SIP Calling Lab on AWS: Talking Between My Phone and Laptop Through the Cloud

What SIP actually is (the mental model I needed)

What is what: every piece in plain English

The two halves of every call

Kamailio — the switchboard operator

rtpengine — the audio relay in the middle

The database (DBTEXT) — the user list

The softphones — the actual phones

The AWS pieces

How one call actually flows

One-line cheat sheet

Why I moved from WSL to AWS

Step 1 — Launch the EC2 instance

Step 2 — Open the ports (AWS Security Group)

Step 3 — Install Kamailio and rtpengine

Step 4 — Configure rtpengine (the NAT fix)

Step 5 — Point the control tool at the database

Step 6 — Turn on the features in kamailio.cfg

Step 7 — Create the two users

The three mistakes I had to fix (this is where I learned the most)

Mistake 1 — Duplicate defines

Mistake 2 — The database module wasn't loaded

Mistake 3 — File permissions

Step 8 — Register the softphones

Step 9 — Make the call

Reading the SIP call flow

What I'd do differently / next steps

The whole thing in one glance

Learning Ansible Without Opening the Docs Website

It started with a broken command

The command that does most of the work

The confusion that finally made things click

A rough edge, and how I got past it

Keywords don't have examples, so I went looking for them

Putting it together in one small playbook

Where I've landed

Learn RAID the Hard Way: Building 0/1/5/6/10 by Hand on EC2

1. The concepts

What is RAID, in one minute?

What is an "array"?

Do I need to partition the disks first (fdisk)?

A word on filesystems: ext4 vs xfs

2. The hands-on build

The setup

RAID 0 — Striping (speed, zero safety)

RAID 1 — Mirroring (safety, half the space)

RAID 5 — Striping + distributed parity (the popular one)

RAID 6 — Double parity (survive TWO failures)

The problem with `man`

`tldr` — examples, not essays

`apropos` — when you don't even know the command's name

`info` — the deep dive when you actually need it

Now the Kubernetes version of the same problem: `kubectl explain`

4. The device name changed (`/dev/md0` became `/dev/md127`)