TLDR
The firewall config is not in /etc/pve/firewall/cluster.fw but in a SQLite Database in /var/lib/pve-cluster/config.db. You need to reboot your system into rescue mode, edit the value enable: 1 to enable: 0 and reboot into Proxmox.
Context
I made a noob mistake and locked myself out of my server. Luckily Hetzner allows me to reboot into rescue mode. This is what happened and how I managed to get my access back.
In other words, this tutorial is for situations where you've accidentally locked yourself out of your Proxmox server due to a firewall misconfiguration (like I did). In my case, I enabled the firewall (enable: 1) with an incorrect configuration, preventing access to the server. The solution involves booting into a rescue system, mounting the Proxmox partition, and manually editing the firewall configuration in the SQLite database.
Prerequisites
- Access to a rescue system (e.g., Hetzner Rescue System)
- Basic knowledge of Linux commands and SQLite, although you can copy and paste these commands and it should work.
Disclaimer : I am not responsible for data loss or anything else for that matter. The following commands worked for me and nothing bad happened. I out them here in case they help someone else, as I had to research a few hour before solving this (specially the issue of not finding the config).
Step 1: Boot into Rescue System
Boot your server into the rescue system provided by your hosting provider (e.g., Hetzner Rescue System).
Step 2: Identify the Proxmox Partition
Use the lsblk command to list all block devices:
lsblk
Identify the partition where Proxmox is installed. It's often part of a RAID array or LVM setup.
In my case the output was like this:
loop07:003.1G1 loop
nvme1n1259:00 476.9G0 disk
├─nvme1n1p1259:10256M0 part
│ └─md09:00 255.9M0 raid1
├─nvme1n1p2259:201G0 part
│ └─md19:101022M0 raid1
└─nvme1n1p3259:30 475.7G0 part
└─md29:20 475.6G0 raid1
├─vg0-root 253:0064G0 lvm
├─vg0-swap 253:108G0 lvm
└─vg0-data 253:20402G0 lvm
nvme0n1259:40 476.9G0 disk
├─nvme0n1p1259:50256M0 part
│ └─md09:00 255.9M0 raid1
├─nvme0n1p2259:601G0 part
│ └─md19:101022M0 raid1
└─nvme0n1p3259:70 475.7G0 part
└─md29:20 475.6G0 raid1
├─vg0-root 253:0064G0 lvm
├─vg0-swap 253:108G0 lvm
└─vg0-data 253:20402G0 lvm
There I saw that I should mount vg0, and that is was in a raid md2
Step 3: Assemble RAID Array (if applicable)
If your Proxmox partition is part of a RAID array, assemble it:
mdadm --assemble --scan
Step 4: Activate Volume Group
Activate the volume group (usually named vg0 in Proxmox):
vgchange -ay vg0
Step 5: Mount the Proxmox Partition
Create a mount point and mount the Proxmox root partition:
mkdir /mnt/proxmox
mount /dev/vg0/root /mnt/proxmox
Verify the mount:
ls /mnt/proxmox/
Here you should see some files and directories.
Step 6: Locate the Configuration Database
The Proxmox configuration is stored in an SQLite database. Locate it:
ls -la /mnt/proxmox/var/lib/pve-cluster
You should see a file named config.db.
Step 7: Access the SQLite Database
Open the SQLite database:
sqlite3 /mnt/proxmox/var/lib/pve-cluster/config.db
sqlite3 is already installed in the rescue system of Hetzner. You need to install it if it's not available in your system.
Step 8: Check the Current Firewall Configuration
View the current firewall configuration:
SELECT \* FROM tree WHERE name = 'cluster.fw';
Note : Initially I didn't know where this was, so I used the following to find where the entry was and if there was any.
SELECT \* FROM tree WHERE name = 'cluster.fw';
Step 9: Update the enable Option
Change the enable option from 1 to 0 to disable the firewall:
UPDATE tree
SET data = replace(data, 'enable: 1', 'enable: 0')
WHERE name = 'cluster.fw';
Step 10: Verify the Change
Confirm that the change was made successfully:
SELECT \* FROM tree WHERE name = 'cluster.fw';
Step 11: Exit SQLite
Exit the SQLite prompt:
.quit
Step 12: Unmount and Reboot
Unmount the Proxmox partition and reboot the server:
umount /mnt/proxmox
reboot
Important Notes
- Disabling the Firewall: This process disables the firewall cluster-wide. Re-enable it after properly configuring it once you regain access.
- Security Risks: A disabled firewall may expose your system to security risks. You have been warned.
- Backup: Always create backups before making significant changes. I have my proxmox configs in a git repository for reference.
- Alternative Methods: When possible, use the Proxmox web interface or CLI tools for configuration changes. At least that's what I've read. I like to use config files, but I also locked myself out of my server.
References
Several sites, but I cannot longer remember all of them.
Some of the sites I visited are:
Top comments (0)