DEV Community

Janak Shrestha
Janak Shrestha

Posted on

MariaDB Troubleshooting

#linux #mariadb #debugging

There is a critical issue going on with the Nautilus application in Stratos DC. The production support team identified that the application is unable to connect to the database. After digging into the issue, the team found that mariadb service is down on the database server.

Look into the issue and fix the same.

Step 1: Connect to the Database Server

Access the database server using SSH.

ssh peter@stdb01
Password: Sp!dy
Enter fullscreen mode Exit fullscreen mode

Switch to root to perform administrative tasks.

sudo su -
Password: Sp!dy
Enter fullscreen mode Exit fullscreen mode

Step 2: Check the MariaDB Service Status

The first step is to check whether the service is running.

systemctl status mariadb
Enter fullscreen mode Exit fullscreen mode

The output showed the service was inactive and dead.

○ mariadb.service - MariaDB 10.5 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
Enter fullscreen mode Exit fullscreen mode

The service was not running and was also disabled, meaning it would not start on boot.

Step 3: Attempt to Start and Enable the Service

Try to start and enable the service to see what error occurs.

systemctl enable --now mariadb
Enter fullscreen mode Exit fullscreen mode

The output showed the service failed to start.

Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xeu mariadb.service" for details.
Enter fullscreen mode Exit fullscreen mode

Step 4: Check the Logs

Check the system logs for MariaDB.

journalctl -u mariadb -n 50
Enter fullscreen mode Exit fullscreen mode

The logs showed the initialization process but did not immediately reveal the root cause.

Sep 22 11:41:00 stdb01 systemd[1]: Starting MariaDB 10.5 database server...
Sep 22 11:41:00 stdb01 mariadb-prepare-db-dir[22278]: Initializing MariaDB database
...
Sep 22 11:41:01 stdb01 systemd[1]: mariadb.service: Main process exited, code=exited, st...
Sep 22 11:41:01 stdb01 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Sep 22 11:41:01 stdb01 systemd[1]: Failed to start MariaDB 10.5 database server.
Enter fullscreen mode Exit fullscreen mode

Step 5: Check the MariaDB Error Log

The MariaDB error log contains more detailed information.

cat /var/log/mariadb/mariadb.log
Enter fullscreen mode Exit fullscreen mode

The error log clearly identified the root cause.

2026-09-22 11:41:01 0 [Note] Server socket created on IP: '::'.
2026-09-22 11:41:01 0 [ERROR] mariadbd: Can't create/write to file '/run/mariadb/mariadb.pid' (Errcode: 13 "Permission denied")
2026-09-22 11:41:01 0 [ERROR] Can't start server: can't create PID file: Permission denied
Enter fullscreen mode Exit fullscreen mode

Root Cause Identified

MariaDB cannot create its PID file at /run/mariadb/mariadb.pid because of a permission issue. The directory /run/mariadb/ has incorrect ownership or permissions.

Step 6: Check the Directory Permissions

Inspect the /run/mariadb/ directory.

ls -la /run/mariadb/
Enter fullscreen mode Exit fullscreen mode

The output showed the directory was owned by root:mysql with permissions 755.

drwxr-xr-x 2 root mysql 40 Sep 22 10:23 .
Enter fullscreen mode Exit fullscreen mode

MariaDB runs as the mysql user. While the group is correct, the directory is owned by root, and the mysql user cannot create files in it because the owner permissions do not include write for the mysql user (the owner is root, and root has write, but mysql is not root).

Wait, 755 means the owner has rwx, group has r-x, and others have r-x. Since the owner is root, the mysql user only has group permissions (r-x), which do not include write. Therefore, mysql cannot create the PID file.

Step 7: Fix the Directory Ownership

Change the ownership of the directory to mysql:mysql.

chown -R mysql:mysql /run/mariadb
chmod 755 /run/mariadb
Enter fullscreen mode Exit fullscreen mode

Verify the change.

ls -la /run/mariadb/
Enter fullscreen mode Exit fullscreen mode

Expected output:

drwxr-xr-x 2 mysql mysql 40 Sep 22 10:23 .
Enter fullscreen mode Exit fullscreen mode

Now the mysql user owns the directory and has write permission.

Step 8: Start the MariaDB Service

Start the service again.

systemctl start mariadb
systemctl status mariadb
Enter fullscreen mode Exit fullscreen mode

The service started successfully.

● mariadb.service - MariaDB 10.5 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-09-22 11:43:18 UTC; 18ms ago
   Main PID: 23135 (mariadbd)
     Status: "Taking your SQL requests now..."
Enter fullscreen mode Exit fullscreen mode

Step 9: Verify the Service

Check that the service is listening on port 3306.

ss -tlnp | grep 3306
Enter fullscreen mode Exit fullscreen mode

Expected output:

LISTEN 0 80 *:3306 *:* users:(("mariadbd",pid=23135,fd=19))
Enter fullscreen mode Exit fullscreen mode

Test the database connection.

mysql -u root -e "SELECT 1;"
Enter fullscreen mode Exit fullscreen mode

Expected output:

+---+
| 1 |
+---+
| 1 |
+---+
Enter fullscreen mode Exit fullscreen mode

Step 10: Ensure the Service is Enabled

Confirm the service is enabled to start on boot.

systemctl is-enabled mariadb
Enter fullscreen mode Exit fullscreen mode

Expected output:

enabled
Enter fullscreen mode Exit fullscreen mode

Complete Troubleshooting Workflow 

# 1. Connect to the database server
ssh peter@stdb01
sudo su -

# 2. Check service status
systemctl status mariadb

# 3. Attempt to start and enable
systemctl enable --now mariadb

# 4. Check logs
journalctl -u mariadb -n 50

# 5. Check MariaDB error log
cat /var/log/mariadb/mariadb.log

# 6. Check directory permissions
ls -la /run/mariadb/

# 7. Fix ownership
chown -R mysql:mysql /run/mariadb
chmod 755 /run/mariadb

# 8. Start service
systemctl start mariadb
systemctl status mariadb

# 9. Verify port
ss -tlnp | grep 3306

# 10. Test connection
mysql -u root -e "SELECT 1;"

# 11. Check enabled
systemctl is-enabled mariadb
Enter fullscreen mode Exit fullscreen mode

Summary of the Fix

Issue Cause Fix
MariaDB fails to start Permission denied on /run/mariadb/mariadb.pid chown -R mysql:mysql /run/mariadb
Directory owned by root MariaDB runs as mysql user Change ownership to mysql:mysql
Service disabled Not enabled on boot systemctl enable mariadb

Common MariaDB Issues and Fixes

Issue Symptom Fix
Permission denied on PID file Can't create/write to file '/run/mariadb/mariadb.pid' chown -R mysql:mysql /run/mariadb
InnoDB corruption ib_logfile0 not found Remove ib_logfile* and restart
Port already in use Address already in use Find and stop conflicting process
Data directory permissions Can't open file chown -R mysql:mysql /var/lib/mysql
Configuration errors Unknown variable Check my.cnf files

Verification Checklist

Check Command Expected
Service Status systemctl status mariadb active (running)
Port Listening `ss -tlnp \ grep 3306`
Connection Test mysql -u root -e "SELECT 1;" Returns 1
Enabled on Boot systemctl is-enabled mariadb enabled
Error Log tail /var/log/mariadb/mariadb.log No errors

Best Practices

  1. Always check the service status first.
  2. Review both system logs and application-specific logs.
  3. Identify the exact error message before making changes.
  4. Fix permissions based on the user the service runs as.
  5. Verify the service after starting it.
  6. Ensure the service is enabled for boot persistence.
  7. Document the fix for future reference.

Conclusion

This guide covered a complete MariaDB troubleshooting workflow. The root cause was a permission issue on the /run/mariadb/ directory, which prevented MariaDB from creating its PID file. By changing the ownership to mysql:mysql, the service started successfully.

The key steps were checking the service status, reviewing logs, identifying the permission error, fixing the directory ownership, and verifying the service is running and listening on port 3306.

Top comments (0)