Ceph Standalone: RADOS, RBD, CephFS Without Proxmox

A monitor got wedged at 03:14 on a Tuesday. The Proxmox cluster was fine — quorum held, VMs were running — but ceph -s from the host returned a HEALTH_WARN nobody on the team could interpret, because nobody on the team had ever run ceph -s outside the GUI. The pveceph wrapper had hidden the monmap, the wrapper had baked opinionated defaults into a config file that was actually a symlink into /etc/pve, and the recovery procedure in the vendor KB started with "edit ceph.conf" on a file that was being regenerated by a daemon every thirty seconds. We lost four hours to a problem that a standalone cluster would have surfaced in a single ceph mon dump.

This guide is for the admin who has outgrown the hypervisor-managed storage layer — or is about to build storage from scratch and wants to do it properly. By the end you'll have bootstrapped a standalone Ceph cluster with cephadm, built RADOS pools with sane placement, served RBD images to clients via systemd units, mounted CephFS with a standby MDS, and wired up the monitoring and backup paths that actually matter. No Proxmox, no wrappers, no config-file roulette.

:::note[TL;DR]

• Proxmox's pveceph integration hides the RADOS layer you need when things break; standalone cephadm gives you the real control surface.
• Three nodes is the floor, not the target — plan for at least four with a dedicated cluster network.
• RADOS is the foundation: get pools, PGs, and CRUSH right before you touch RBD or CephFS.
• RBD and CephFS are clients — tune them like clients, back them up like clients.
• Replication is not backup. Write the rbd export script before you need it. :::

Prerequisites

• Three or more physical or VM nodes running a modern Debian/Ubuntu or RHEL-family release, all on the same L2 network segment for the public and cluster networks.
• Root or passwordless sudo on every node.
• At least one dedicated block device per node for OSDs (not a partition on the OS disk).
• Two network interfaces per node (public + cluster), or a single interface with VLAN separation if you're brave and documented.
• chrony installed and pointed at a reliable time source.
• SSH key-based access from the bootstrap node to every other node.

Why You Should Stop Letting Proxmox Drive Your Ceph Cluster

Proxmox's Ceph integration is genuinely good at one thing: making a hyperconverged cluster boot in an afternoon. It wraps ceph commands in pveceph, generates a ceph.conf that lives as a symlink under /etc/pve/, and applies defaults that assume you want a small cluster with minimal tuning. That's fine until the defaults aren't fine, and then you're debugging through a layer that was never designed to be debugged.

Here's the telltale fingerprint of a Proxmox-generated config:

# /etc/pve/ceph.conf — note this is a symlink, not a regular file
[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.15.0.0/24
     fsid = 4f0d2a8e-...
     mon_allow_pool_delete = true
     mon_host = 10.15.0.11 10.15.0.12 10.15.0.13
     ms_bind_ipv4 = true
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 10.15.0.0/24

That mon_allow_pool_delete = true is a Proxmox default you probably don't want on a production cluster. The symlink means any hand-edit gets clobbered. And the comparison between the wrapper and the real thing is stark:

---

⚠️ TRUNCATED VERSION
This is an abbreviated cross-post. Full article (all config files, architecture diagrams, images): valtersit.com

---

🛠️ Partner Tools for Developers

ValtersIT curates partner deals for developers and sysadmins — VPS hosting, security tools, monitoring platforms and dev productivity gear. No filler, no affiliate spam.

➜ valtersit.com/deals/