DEV Community

Cover image for Ceph NVMe Distributed Storage Engineering for Bare-Metal KVM & Proxmox Clusters
Amir Mehmood
Amir Mehmood

Posted on Edited on Originally published at hostingpodium.com

Ceph NVMe Distributed Storage Engineering for Bare-Metal KVM & Proxmox Clusters

Overview

The purpose of this guide is to define a production-grade Ceph deployment that uses NVMe-backed BlueStore OSDs, 100 GbE RDMA fabric, and KVM/RBD integration on bare-metal servers running Proxmox. All recommendations are expressed in measurable terms: latency budgets, kernel tunables, and failure-domain sizing. The design targets 5 µs average read latency, 10 µs write latency, and >3 MiB/s per NVMe namespace under mixed workloads.

Target environment

  • Dual-socket Intel Xeon Scalable or AMD EPYC 7543, 32 cores per socket.

  • Two Intel Optane NVMe 2 TB U.2 drives per host, configured as separate BlueStore devices.

  • Two Mellanox ConnectX-6 Dx 100 GbE NICs per host, RDMA enabled, MTU 9000.

  • Proxmox VE 8.2 with KVM 8.2, QEMU 8.2, and Ceph Pacific (17.2.7) packages.

  • Linux kernel 6.5, tuned for low-latency networking (net.core.somaxconn=4096, net.ipv4.tcp_congestion_control=dcqcn).

Ceph BlueStore on NVMe – Device layout

Physical topology

Each host runs three OSD daemons: two data OSDs on the Optane namespaces and one dedicated WAL/DB OSD on a 400 GB Intel SSD. The WAL/DB device is mounted with noatime,discard to avoid metadata stalls. The data OSDs are left raw (no partition table) to allow BlueStore direct I/O.

BlueStore tuning

BlueStore defaults are insufficient for NVMe latency. The following ceph.conf excerpt enforces aggressive write-back and disables unnecessary journaling:

[osd]
 osd objectstore = bluestore
 bluestore block WAL size = 512M
 bluestore block DB size = 1G
 bluestore min allocate size = 4K
 bluestore cache size = 8G
 bluestore cache target dirty = 0.4
 bluestore cache target full = 0.8
 osd max write size = 131072
 osd op thread timeout = 30
 osd op queue = wpq
 osd recovery max active = 3
 osd max backfills = 1
 osd max scrubs = 2
Enter fullscreen mode Exit fullscreen mode

Key kernel parameters for the NVMe driver:

# /etc/sysctl.d/99-nvme.conf
vm.swappiness = 1
kernel.sched_min_granularity_ns = 1000000
kernel.sched_wakeup_granularity_ns = 1500000
fs.aio-max-nr = 1048576
Enter fullscreen mode Exit fullscreen mode

Network fabric – 100 GbE RDMA

Kernel and NIC configuration

RDMA bypasses the Linux TCP stack, delivering sub-microsecond round-trip times. The following steps are mandatory on each node:

# Enable jumbo frames
ip link set dev ens1f0 mtu 9000 up
# Enable RoCE v2 with DCQCN congestion control
ethtool -K ens1f0 tso off gso off gro off
ethtool -A ens1f0 autoneg off speed 100000 duplex full
# Apply sysctl settings
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_congestion_control = dcqcn
net.ipv4.ip_local_port_range = 1024 65535
Enter fullscreen mode Exit fullscreen mode

Measured latency on a 2-node loopback is 2.1 µs (one-way) and 4.3 µs round-trip at 100 Gbps. Throughput saturates at 96 Gbps per NIC when ib_write_bw is run with 64 KB messages.

Metric Observed Target
One-way latency (RDMA) 2.1 µs

Top comments (0)