DEV Community

POTHURAJU JAYAKRISHNA YADAV
POTHURAJU JAYAKRISHNA YADAV

Posted on

How a Linux Kernel Upgrade Broke MongoDB on an Azure VM — And How We Fixed It

#devops #azure #mongodb #linux

It was a normal morning. I woke up, grabbed my coffee, and before I'd even properly sat down my phone was already lighting up — alerts, messages, the whole thing. The app was down. Users couldn't connect, services were throwing errors, and nobody had the faintest idea why.

The confusing part? Nobody had touched anything. No deploy the night before, no config change, no code push. I'd left everything green and healthy when I logged off, and I woke up to a dead server. That's the kind of outage that really gets under your skin, because there's nothing obvious to point at.

So I SSH'd into the VM, half expecting the whole box to be gone, and instead found MongoDB sitting there flat-out refusing to start. That was the beginning of a morning I hadn't planned for — and the culprit turned out to be something none of us had on our radar: Azure had quietly upgraded the kernel overnight while we were all asleep.

This one caught us completely off guard, which is exactly why I wanted to write it down. Here's the whole story, including the stuff we tried that didn't work, because honestly the dead ends are half the lesson.

The incident

It started like most outages do — out of nowhere. One of our apps running on an Azure VM just stopped responding. Users couldn't connect, services were down, and the first thing we did was SSH in and check whether anything was even alive.

sudo systemctl status mongod
Enter fullscreen mode Exit fullscreen mode

MongoDB was in a failed state — Result: exit-code. And the weird part was how fast it was dying. The service would try to start and then fall over within milliseconds, somewhere around 35-48ms. That's not a database struggling to load data. That's something rejecting it before it even gets going.

Hunting for the cause

Logs first, obviously.

sudo cat /var/log/mongodb/mongod.log | tail -50
Enter fullscreen mode Exit fullscreen mode

Useless. The log only showed a clean shutdown from two days ago. Nothing about why it was refusing to start now. So we went to the system journal instead and filtered for fatal messages:

sudo journalctl -u mongod --since "today" --no-pager | grep -i '"s":"F"'
Enter fullscreen mode Exit fullscreen mode

And there it was, the line that made the whole thing click:

MongoDB cannot start: Linux kernel versions 6.19 and newer has a known 
incompatibility with this version of MongoDB. 
See https://jira.mongodb.org/browse/SERVER-121912 for more information.
Enter fullscreen mode Exit fullscreen mode

A kernel incompatibility. Not corruption, not permissions, not a config mistake. The kernel.

What actually happened

Turns out Azure had auto-upgraded the Linux kernel on the VM overnight during a maintenance window. The box jumped to 7.0.0-1014-azure, which under the hood is based on Linux kernel 6.19+.

MongoDB has a known problem with kernel 6.19 and up — something to do with the rseq system call. And rather than risk running in a broken state, MongoDB just refuses to start at all when it detects that kernel. A hard safety block.

We confirmed what we were dealing with:

uname -r
# Output: 7.0.0-1014-azure

uname -a
# Linux 7.0.0-1014-azure ... x86_64
Enter fullscreen mode Exit fullscreen mode

And the MongoDB version:

mongod --version
# db version v8.0.29
Enter fullscreen mode Exit fullscreen mode

So: MongoDB 8.0.29 sitting on a kernel it explicitly refuses to run on. That's the entire bug in one sentence.

The stuff we tried that didn't work

Before we figured out the kernel was the issue, we burned time on the usual suspects. Leaving these in because this is the part where you learn what the problem isn't.

First we blamed permissions.

sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown -R mongodb:mongodb /var/log/mongodb
Enter fullscreen mode Exit fullscreen mode

Nothing changed.

Then we figured it was a stale lock file. Classic MongoDB-won't-start cause.

sudo rm -f /var/lib/mongodb/mongod.lock
sudo rm -f /tmp/mongodb-27017.sock
Enter fullscreen mode Exit fullscreen mode

Still nothing.

Then we thought maybe upgrading MongoDB would fix it. If it's a known issue, surely a newer version handles it, right?

sudo apt-get update
sudo apt-get install -y mongodb-org
Enter fullscreen mode Exit fullscreen mode

MongoDB upgraded fine, straight to 8.0.32 — and failed with the exact same error. The kernel check is in 8.0.32 too. So no, upgrading doesn't help; the block is intentional.

Then we tried forcing it with --fork.

sudo mongod --config /etc/mongod.conf --fork
Enter fullscreen mode Exit fullscreen mode

This one actually worked for a bit, but only with the old binary. Once we'd upgraded MongoDB, even --fork ran into the kernel check and died. So that was a dead end too, just a slower one.

At this point it was clear the problem wasn't anything on the MongoDB side at all. It was the kernel, full stop. Which meant the fix had to be at the kernel level.

The real fix: boot into the older kernel

Here's the thing that saved us — Azure doesn't throw away the old kernel when it upgrades. It leaves the previous ones installed on disk. The compatible kernel was still right there, just not the one being booted.

We listed what was actually installed:

dpkg --list | grep linux-image-.*-azure
Enter fullscreen mode Exit fullscreen mode 
rc  linux-image-6.17.0-1021-azure   # rc = removed, can't use
ii  linux-image-6.17.0-1022-azure   # ii = installed ✅
ii  linux-image-7.0.0-1014-azure    # ii = installed, broken ❌
Enter fullscreen mode Exit fullscreen mode

6.17.0-1022-azure was fully installed (ii), and since 6.17 is below 6.19, it was compatible with MongoDB. We just needed to tell the server to boot into it instead of the new broken one.

A quick word on GRUB

If you've never had a reason to touch this, GRUB (the Grand Unified Bootloader) is the thing that runs the instant your server powers on, before the OS loads. It picks which kernel to boot. Think of it as the startup selector for your Linux box.

By default it grabs the newest kernel — which was exactly our problem, because the newest one was the broken 7.0.0-1014-azure.

The config lives at:

/etc/default/grub
Enter fullscreen mode Exit fullscreen mode

Pointing GRUB at the good kernel

We edited the GRUB config to default to the older compatible kernel, and gave ourselves a boot menu timeout while we were at it:

sudo sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-1022-azure"/' /etc/default/grub
sudo sed -i 's/GRUB_TIMEOUT=0/GRUB_TIMEOUT=5/' /etc/default/grub
Enter fullscreen mode Exit fullscreen mode

What those two changes actually do:

  • GRUB_DEFAULT=0 just boots whatever the first (newest) kernel is. Changing it to the full "Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-1022-azure" string tells GRUB to boot that specific kernel by name instead.
  • GRUB_TIMEOUT=0 means GRUB shows no menu and boots instantly. Bumping it to 5 gives you a five-second window to manually pick a kernel if you ever need to.

Then — and this part is important — you have to regenerate the actual boot config and reboot:

sudo update-grub
sudo reboot
Enter fullscreen mode Exit fullscreen mode

Don't skip update-grub. Editing /etc/default/grub on its own does nothing. update-grub reads that file and regenerates the real boot config at /boot/grub/grub.cfg. If you forget it, you'll reboot straight back into the broken kernel and wonder why nothing changed.

After the reboot

Server came back up, first thing we checked was the kernel:

uname -r
# Output: 6.17.0-1022-azure ✅
Enter fullscreen mode Exit fullscreen mode

Then we cleaned up the leftover junk from all our failed attempts:

sudo rm -f /var/run/mongodb/mongod.pid
sudo rm -f /tmp/mongodb-27017.sock
sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown -R mongodb:mongodb /var/log/mongodb
Enter fullscreen mode Exit fullscreen mode

And started MongoDB:

sudo systemctl start mongod
sudo systemctl status mongod
# Active: active (running) ✅
Enter fullscreen mode Exit fullscreen mode

Running. The app was back online immediately.

Making sure it doesn't happen again

Getting it running was great, but we weren't out of the woods. Azure could just upgrade the kernel again on the next maintenance window and put us right back where we started. So we pinned the kernel packages to stop that:

sudo apt-mark hold linux-azure
sudo apt-mark hold linux-image-azure
sudo apt-mark hold linux-image-7.0.0-1014-azure
sudo apt-mark hold linux-image-6.17.0-1022-azure
Enter fullscreen mode Exit fullscreen mode

What apt-mark hold does is tell the package manager to never automatically upgrade or remove those packages. The important ones are linux-azure and linux-image-azure — those are meta-packages that always point at the latest kernel, so holding them stops any new kernel from sneaking in automatically. This keeps us parked on 6.17.0-1022-azure until MongoDB ships an official fix.

Double-check what's held with:

sudo apt-mark showhold
Enter fullscreen mode Exit fullscreen mode

What I took away from this

A few things worth remembering:

  • Cloud providers — Azure here, but it's not just Azure — will upgrade your kernel during maintenance windows without asking. Something can break overnight with zero changes on your end.
  • MongoDB 8.x has a hard block against Linux kernel 6.19+ because of the rseq incompatibility (SERVER-121912). It won't start, and that's on purpose.
  • A Linux box can have several kernels installed at once. You just have to tell GRUB which one to use.
  • GRUB is the bootloader. Edit /etc/default/grub, run update-grub, reboot — that's how you control the default kernel.
  • apt-mark hold is genuinely useful when you need to freeze a package and keep auto-upgrades from undoing your fix.

Quick reference for next time

If this bites us (or you) again on any VM, here's the short version:

# 1. Confirm what's broken
sudo journalctl -u mongod --no-pager | grep '"s":"F"'

# 2. List installed kernels
dpkg --list | grep linux-image-.*-azure

# 3. Pick the highest kernel below 6.19 that shows "ii"

# 4. Set it as the default and reboot
sudo sed -i 's/GRUB_DEFAULT=.*/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux <your-kernel>"/' /etc/default/grub
sudo update-grub
sudo reboot

# 5. After reboot, start Mongo
sudo systemctl start mongod
sudo systemctl status mongod

# 6. Freeze the kernels
sudo apt-mark hold linux-azure linux-image-azure
Enter fullscreen mode Exit fullscreen mode

Wrapping up

What looked like a mysterious outage was really just a kernel compatibility issue triggered by an automatic cloud update. No code changes, no reinstall, no data loss — the whole thing came down to understanding how Linux kernel management and GRUB fit together.

The real lesson for me: know what kernels are installed on your VMs, and remember your cloud provider can swap them out from under you whenever it feels like it. A quick dpkg --list | grep linux-image and a basic grasp of GRUB would've saved us a good chunk of the time we spent chasing permissions and lock files.

If you've been burned by an auto-upgrade like this, I'd like to hear how it showed up for you. These silent infrastructure changes are sneaky, and the more stories floating around, the faster the next person figures it out.

Top comments (0)