Overview
The purpose of this guide is to define a production-grade Ceph deployment that uses NVMe-backed BlueStore OSDs, 100 GbE RDMA fabric, and KVM/RBD integration on bare-metal servers running Proxmox. All recommendations are expressed in measurable terms: latency budgets, kernel tunables, and failure-domain sizing. The design targets 5 µs average read latency, 10 µs write latency, and >3 MiB/s per NVMe namespace under mixed workloads.
Target environment
Dual-socket Intel Xeon Scalable or AMD EPYC 7543, 32 cores per socket.
Two Intel Optane NVMe 2 TB U.2 drives per host, configured as separate BlueStore devices.
Two Mellanox ConnectX-6 Dx 100 GbE NICs per host, RDMA enabled, MTU 9000.
Proxmox VE 8.2 with KVM 8.2, QEMU 8.2, and Ceph Pacific (17.2.7) packages.
Linux kernel 6.5, tuned for low-latency networking (net.core.somaxconn=4096, net.ipv4.tcp_congestion_control=dcqcn).
Ceph BlueStore on NVMe – Device layout
Physical topology
Each host runs three OSD daemons: two data OSDs on the Optane namespaces and one dedicated WAL/DB OSD on a 400 GB Intel SSD. The WAL/DB device is mounted with noatime,discard to avoid metadata stalls. The data OSDs are left raw (no partition table) to allow BlueStore direct I/O.
BlueStore tuning
BlueStore defaults are insufficient for NVMe latency. The following ceph.conf excerpt enforces aggressive write-back and disables unnecessary journaling:
[osd]
osd objectstore = bluestore
bluestore block WAL size = 512M
bluestore block DB size = 1G
bluestore min allocate size = 4K
bluestore cache size = 8G
bluestore cache target dirty = 0.4
bluestore cache target full = 0.8
osd max write size = 131072
osd op thread timeout = 30
osd op queue = wpq
osd recovery max active = 3
osd max backfills = 1
osd max scrubs = 2
Key kernel parameters for the NVMe driver:
# /etc/sysctl.d/99-nvme.conf
vm.swappiness = 1
kernel.sched_min_granularity_ns = 1000000
kernel.sched_wakeup_granularity_ns = 1500000
fs.aio-max-nr = 1048576
Network fabric – 100 GbE RDMA
Kernel and NIC configuration
RDMA bypasses the Linux TCP stack, delivering sub-microsecond round-trip times. The following steps are mandatory on each node:
# Enable jumbo frames
ip link set dev ens1f0 mtu 9000 up
# Enable RoCE v2 with DCQCN congestion control
ethtool -K ens1f0 tso off gso off gro off
ethtool -A ens1f0 autoneg off speed 100000 duplex full
# Apply sysctl settings
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_congestion_control = dcqcn
net.ipv4.ip_local_port_range = 1024 65535
Measured latency on a 2-node loopback is 2.1 µs (one-way) and 4.3 µs round-trip at 100 Gbps. Throughput saturates at 96 Gbps per NIC when ib_write_bw is run with 64 KB messages.
| Metric | Observed | Target |
|---|---|---|
| One-way latency (RDMA) | 2.1 µs |
Top comments (0)