DEV Community

Cover image for Ceph NVMe Distributed Storage Engineering for Bare-Metal KVM & Proxmox Clusters
Amir Mehmood
Amir Mehmood

Posted on Originally published at hostingpodium.com

Ceph NVMe Distributed Storage Engineering for Bare-Metal KVM & Proxmox Clusters

#kubernetes #devops #cloud #linux

Originally published on the HostingPodium Enterprise Engineering Blog.

Free SRE & Developer Diagnostics: Test your stack live with our DMARC Record Checker, Generator & SPF/DKIM Validator, Free Website Speed & Core Web Vitals Checker, or explore all 9 Free Webmaster & Cloud Engineering Tools.

This guide details a production Ceph deployment using NVMe-backed BlueStore, 100 GbE RDMA, and KVM/RBD integration on Proxmox. It covers tuning, network configuration, data protection trade-offs, and operational runbooks.

Overview

The purpose of this guide is to define a production-grade Ceph deployment that uses NVMe-backed BlueStore OSDs, 100 GbE RDMA fabric, and KVM/RBD integration on bare-metal servers running Proxmox. All recommendations are expressed in measurable terms: latency budgets, kernel tunables, and failure-domain sizing. The design targets 5 µs average read latency, 10 µs write latency, and >3 MiB/s per NVMe namespace under mixed workloads.

Target environment

  • Dual-socket Intel Xeon Scalable or AMD EPYC 7543, 32 cores per socket.

  • Two Intel Optane NVMe 2 TB U.2 drives per host, configured as separate BlueStore devices.

  • Two Mellanox ConnectX-6 Dx 100 GbE NICs per host, RDMA enabled, MTU 9000.

  • Proxmox VE 8.2 with KVM 8.2, QEMU 8.2, and Ceph Pacific (17.2.7) packages.

  • Linux kernel 6.5, tuned for low-latency networking (net.core.somaxconn=4096, net.ipv4.tcp_congestion_control=dcqcn).

Ceph BlueStore on NVMe – Device layout

Physical topology

Each host runs three OSD daemons: two data OSDs on the Optane namespaces and one dedicated WAL/DB OSD on a 400 GB Intel SSD. The WAL/DB device is mounted with noatime,discard to avoid metadata stalls. The data OSDs are left raw (no partition table) to allow BlueStore direct I/O.

BlueStore tuning

BlueStore defaults are insufficient for NVMe latency. The following ceph.conf excerpt enforces aggressive write-back and disables unnecessary journaling:

[osd]
 osd objectstore = bluestore
 bluestore block WAL size = 512M
 bluestore block DB size = 1G
 bluestore min allocate size = 4K
 bluestore cache size = 8G
 bluestore cache target dirty = 0.4
 bluestore cache target full = 0.8
 osd max write size = 131072
 osd op thread timeout = 30
 osd op queue = wpq
 osd recovery max active = 3
 osd max backfills = 1
 osd max scrubs = 2
Enter fullscreen mode Exit fullscreen mode

Key kernel parameters for the NVMe driver:

# /etc/sysctl.d/99-nvme.conf
vm.swappiness = 1
kernel.sched_min_granularity_ns = 1000000
kernel.sched_wakeup_granularity_ns = 1500000
fs.aio-max-nr = 1048576
Enter fullscreen mode Exit fullscreen mode

Network fabric – 100 GbE RDMA

Kernel and NIC configuration

RDMA bypasses the Linux TCP stack, delivering sub-microsecond round-trip times. The following steps are mandatory on each node:

# Enable jumbo frames
ip link set dev ens1f0 mtu 9000 up
# Enable RoCE v2 with DCQCN congestion control
ethtool -K ens1f0 tso off gso off gro off
ethtool -A ens1f0 autoneg off speed 100000 duplex full
# Apply sysctl settings
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_congestion_control = dcqcn
net.ipv4.ip_local_port_range = 1024 65535
Enter fullscreen mode Exit fullscreen mode

Measured latency on a 2-node loopback is 2.1 µs (one-way) and 4.3 µs round-trip at 100 Gbps. Throughput saturates at 96 Gbps per NIC when ib_write_bw is run with 64 KB messages.

Metric Observed Target
One-way latency (RDMA) 2.1 µs

Free Webmaster, SRE & Cloud Diagnostic Tools (No Signup Required)

Verify and benchmark your production infrastructure directly from your browser using the HostingPodium Free Webmaster & Developer Suite:

Free Engineering Utility Primary Diagnostic Use Case
Global DNS Propagation & SSL Checker Verify live A, AAAA, CNAME, MX, TXT, NS & CAA records across 21+ global resolvers + TLS certificate audits
Website Speed Test & Core Web Vitals Auditor Measure server TTFB, LCP, INP, CLS, and HTTP/2 compression latency in real time
Rich Snippets & Schema.org JSON-LD Generator Generate & validate structured JSON-LD markup for TechArticles, FAQs, Products & Organizations
HTTP Redirect Chain & Status Code Tracer Trace 301/302 redirect hops, inspect response headers, and verify load balancer health checks
DMARC, SPF & DKIM Email Security Validator Audit domain email authentication policies and generate compliant DMARC records
Google SERP Simulator & Meta Tag Previewer Preview desktop/mobile Google snippets, OpenGraph cards, and pixel-width truncation
Free WebP Image Converter & Optimizer Client-side PNG/JPG to WebP compression for faster Core Web Vitals LCP

Related Enterprise Architecture & Benchmarks

Top comments (0)