DEV Community

Cover image for 100 Days of DevOps and Cloud (AWS), Day 9: A Database That Won't Start Is Usually a Permissions Problem
Nnamdi Felix Ibe
Nnamdi Felix Ibe

Posted on

100 Days of DevOps and Cloud (AWS), Day 9: A Database That Won't Start Is Usually a Permissions Problem

When a service refuses to start, the instinct is to blame the service. Usually, the service itself is fine, and something around it is broken. Day 9's dead MariaDB was a clean example of exactly that.

One Linux task, one AWS task. Bring a failed MariaDB service back to life by reading the logs rather than guessing, then enable termination protection for an EC2 instance so it cannot be deleted on a bad day. The tasks come from the KodeKloud Engineer platform, the same as the rest of this series.

MariaDB: read the logs, don't guess

A database that will not start feels opaque, and that is what makes it stressful. It is not actually opaque. MariaDB, like most services, tells you exactly why it failed if you look in the right place, so the discipline is looking before you start changing things.

# Where does it stand right now
sudo systemctl status mariadb

# Read the service's own log, newest lines first
sudo tail -30 /var/log/mariadb/mariadb.log

# Narrow the systemd journal to the failure window
sudo journalctl -u mariadb --since "2025-11-09 12:43:00" --until "2025-11-09 12:45:00"
Enter fullscreen mode Exit fullscreen mode

systemctl status gives you the headline, the last few log lines and whether systemd even tried to start it. The service log and journalctl give you the details. Reading the tail first is the shortcut, the newest lines are almost always the ones that matter.

In this case, the log pointed at the runtime directory. MariaDB runs as the mysql user, and it needs to write its socket and PID file into /run/mariadb. That directory was owned by root, so the mysql user could not write to it, and the service died on startup. The fix is one line:

# Who owns the runtime directory
ls -ld /run/mariadb

# Hand it to the user the service actually runs as
sudo chown mysql:mysql /run/mariadb

# Restart and confirm
sudo systemctl restart mariadb
sudo systemctl status mariadb
Enter fullscreen mode Exit fullscreen mode

That is the whole lesson in miniature. The database engine was never broken. A directory it depended on had the wrong owner, and everything downstream fell over. While you are in there, confirm the service is set to come back after a reboot too:

# Make sure the fix survives a reboot
sudo systemctl is-enabled mariadb
Enter fullscreen mode Exit fullscreen mode

EC2 termination protection: a guard against the irreversible

The AWS task is the safety net for the worst kind of mistake, deleting a server you still needed. Terminating an EC2 instance is permanent, so termination protection exists to make it hard to do by accident. Turn it on with one flag:

# Enable termination protection
aws ec2 modify-instance-attribute \
  --instance-id i-1234567890abcdef0 \
  --disable-api-termination
Enter fullscreen mode Exit fullscreen mode

Same naming quirk I hit with stop protection on Day 8. The flag is --disable-api-termination, which sounds destructive but is the switch that protects the instance. It disables the terminate API for that box. To lift it later, you use --no-disable-api-termination.

The note verified this by actually calling terminate-instances and watching it get refused. That works, but if the protection had not applied, you would have deleted the instance to find out. Ask AWS directly instead:

# Confirm protection is on without risking the instance
aws ec2 describe-instance-attribute \
  --instance-id i-1234567890abcdef0 \
  --attribute disableApiTermination
Enter fullscreen mode Exit fullscreen mode

If it returns true, you are protected, and terminate calls will bounce with an error. Two things to keep in mind. Termination protection and stop protection are separate switches, so a box you want fully guarded needs both. And this only blocks the API path, it does nothing about someone deleting the attached volumes, so treat it as one layer, not the whole wall.

The thread

Both halves of Day 9 were about the same instinct, slowing down before you act. The database rewarded a careful read of the logs over a panicked restart. The EC2 flag made the destructive action deliberately awkward to perform. Speed feels productive, but the operators who last are the ones who look first.

So here is the Day 9 question. Would you rather move fast and hope the thing you are about to break is not the thing you needed, or spend two minutes confirming before you commit?

Day 9 down. Ninety-one to go.

Top comments (0)