DEV Community

Cover image for Deploying Lustre as a Self-Hosted AWS FSx Alternative
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya • Originally published at docs.vultr.com

Deploying Lustre as a Self-Hosted AWS FSx Alternative

#hpc #storage #linux #selfhosted

Lustre is an open-source, POSIX-compliant parallel distributed file system for HPC, ML training, and large-scale data processing, it separates metadata from data across dedicated server nodes, scaling aggregate throughput linearly as you add storage targets. AWS FSx for Lustre is the fully-managed equivalent, with per-TiB pricing and lock-in to AWS. This guide deploys a self-hosted Lustre cluster (1 Metadata Server + 2 Object Storage Servers + a client), configures striping/PFL, LNET, quotas/ACLs, Prometheus+Grafana monitoring, Pacemaker HA, IOR/mdtest benchmarks, and covers migrating off FSx.

Architecture

Lustre separates metadata (filenames, directory structure, permissions, stripe layout) from file data. The Metadata Server (MDS) hosts the Metadata Target (MDT); Object Storage Server (OSS) nodes host Object Storage Targets (OSTs) holding actual file data. Clients talk directly to both MDS and OSS over the LNET network protocol, so I/O flows in parallel across OSTs without a central gateway.

AWS FSx Feature Self-Hosted Lustre Equivalent
FSx for Lustre File System Lustre file system (MDS + OSS cluster)
FSx Metadata MDS + MDT
FSx Data Storage OSS + OSTs
FSx SSD Storage Class Lustre with LDISKFS on NVMe/SSD
FSx Intelligent-Tiering Lustre HSM + Progressive File Layout (PFL)
FSx S3 Data Repository Lustre HSM with an S3-compatible backend
FSx Auto Import/Export Robinhood Policy Engine or lhsmtool_posix
FSx Throughput Scaling Adding OSS/OST pairs
FSx Client Mount Lustre kernel client module
CloudWatch Metrics Lustre Jobstats + Prometheus exporter

Prerequisites: 3 Linux servers (1 MDS + 2 OSS) plus a separate client server, all non-root sudo; a dedicated block storage volume on the MDS (≥10GB) and each OSS (≥20GB each); all nodes connected over a private network (10GbE minimum, InfiniBand recommended for production HPC); comfort with kernel modules and matching kernel-devel to the running kernel.

Install Lustre Packages

Lustre 2.15 LTS needs a Whamcloud-patched kernel. This uses the Enterprise Linux 8 build (Rocky/AlmaLinux/RHEL 8), check the Whamcloud download site for other distros.

Add the Repository

On every MDS, OSS, and client node:

$ sudo nano /etc/yum.repos.d/lustre.repo
Enter fullscreen mode Exit fullscreen mode 
[lustre-server]
name=Lustre Server
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.8/el8.10/server/
gpgcheck=0
enabled=1

[lustre-client]
name=Lustre Client
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.8/el8.10/client/
gpgcheck=0
enabled=1

[e2fsprogs-wc]
name=Lustre e2fsprogs
baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/
gpgcheck=0
enabled=1
Enter fullscreen mode Exit fullscreen mode

Install Server Packages

On the MDS and both OSS nodes:

$ sudo dnf install -y epel-release
$ sudo dnf install -y kernel-4.18.0-553.82.1.el8_lustre kernel-devel-4.18.0-553.82.1.el8_lustre --nogpgcheck
$ sudo grubby --set-default /boot/vmlinuz-4.18.0-553.82.1.el8_lustre.x86_64
$ sudo dnf install -y e2fsprogs --enablerepo=e2fsprogs-wc --nogpgcheck
$ sudo dnf install -y lustre-osd-ldiskfs-mount lustre kmod-lustre-osd-ldiskfs --nogpgcheck
$ echo "exclude=kernel-4.18.0*" | sudo tee -a /etc/dnf/dnf.conf
Enter fullscreen mode Exit fullscreen mode

The kernel pin is critical — Lustre modules build against a specific kernel, and a routine distro kernel update will break the cluster on next reboot.

$ sudo reboot
Enter fullscreen mode Exit fullscreen mode

Wait ~60s, reconnect:

$ uname -r
Enter fullscreen mode Exit fullscreen mode

Should show 4.18.0-553.82.1.el8_lustre.x86_64.

$ sudo modprobe lustre
$ sudo lsmod | grep lustre
Enter fullscreen mode Exit fullscreen mode

Confirms lustre, lnet, obdclass, etc. loaded.

Install Client Packages

Client node only — uses the stock distro kernel:

$ sudo dnf install -y epel-release
$ sudo dnf install -y kmod-lustre-client lustre-client --nogpgcheck
$ sudo modprobe lustre
$ sudo lsmod | grep lustre
$ echo "exclude=kernel-4.18.0*" | sudo tee -a /etc/dnf/dnf.conf
Enter fullscreen mode Exit fullscreen mode

Open the LNET Port

On every node — firewalld blocks TCP 988 by default:

$ sudo firewall-cmd --permanent --add-port=988/tcp
$ sudo firewall-cmd --reload
Enter fullscreen mode Exit fullscreen mode

Important: LNET binds to each server's primary interface by default — typically the public interface on a cloud instance. The MGS registers its network ID (NID) on that interface, so whatever MDS-IP-ADDRESS you use to format OSTs and mount clients must match it. To route Lustre traffic over a private network instead, configure LNET on every node before formatting any target: create /etc/modprobe.d/lustre.conf with options lnet networks=tcp0(INTERFACE) (replace INTERFACE with your private-network interface name from ip addr, e.g. enp8s0), then reboot. Use that node's private address as MDS-IP-ADDRESS throughout.

Deploy the Metadata Server

The MDS hosts both MGS (coordinates file system registration) and MDT (stores namespace/stripe layout) roles.

$ lsblk
Enter fullscreen mode Exit fullscreen mode

Identify the dedicated block device — commonly /dev/vdb on virtio-based cloud block storage, with the OS on /dev/vda. Use whatever your lsblk output shows.

$ sudo mkfs.lustre --fsname=lustrefs --mgs --mdt --index=0 /dev/vdb
$ sudo mkdir -p /mnt/mdt
$ sudo mount -t lustre /dev/vdb /mnt/mdt
$ sudo lctl dl
Enter fullscreen mode Exit fullscreen mode

Should list mgs, mds, mdt, osd-ldiskfs as active.

Deploy Object Storage Servers

First OSS (index 0) 

$ lsblk
$ sudo mkfs.lustre --fsname=lustrefs --ost --index=0 --mgsnode=MDS-IP-ADDRESS@tcp /dev/vdb
$ sudo mkdir -p /mnt/ost0
$ sudo mount -t lustre /dev/vdb /mnt/ost0
$ sudo lctl dl
Enter fullscreen mode Exit fullscreen mode

Should show obdfilter, ost, and an mgc connection to the MGS.

Second OSS (index 1) 

$ sudo mkfs.lustre --fsname=lustrefs --ost --index=1 --mgsnode=MDS-IP-ADDRESS@tcp /dev/vdb
$ sudo mkdir -p /mnt/ost1
$ sudo mount -t lustre /dev/vdb /mnt/ost1
Enter fullscreen mode Exit fullscreen mode

From the MDS, confirm both OSTs registered:

$ sudo lctl get_param osc.*.ost_server_uuid
Enter fullscreen mode Exit fullscreen mode

Both should show FULL state.

Mount on Clients 

$ sudo mkdir -p /mnt/lustre
$ sudo mount -t lustre MDS-IP-ADDRESS@tcp:/lustrefs /mnt/lustre
$ df -h /mnt/lustre
$ lfs df -h /mnt/lustre
Enter fullscreen mode Exit fullscreen mode

lfs df breaks down capacity per MDT/OST — useful for spotting imbalances.

$ sudo dd if=/dev/zero of=/mnt/lustre/testfile bs=1M count=100
$ sudo rm /mnt/lustre/testfile
Enter fullscreen mode Exit fullscreen mode

Configure Striping and PFL

Striping spreads a file across multiple OSTs for aggregate throughput; Progressive File Layout (PFL) applies different stripe counts per file-size range — the self-hosted equivalent of FSx Intelligent-Tiering. Run on the client.

Uniform striping:

$ sudo mkdir -p /mnt/lustre/striped
$ sudo lfs setstripe -c 2 -S 1M /mnt/lustre/striped
$ sudo lfs getstripe -d /mnt/lustre/striped
Enter fullscreen mode Exit fullscreen mode

-c 2 spreads each file across 2 OSTs; -S 1M sets a 1MiB stripe size.

Progressive layout (3-tier):

$ sudo mkdir -p /mnt/lustre/pfl
$ sudo lfs setstripe -E 4M -c 1 -E 64M -c 2 -E -1 -c 2 /mnt/lustre/pfl
$ sudo lfs getstripe -d /mnt/lustre/pfl
Enter fullscreen mode Exit fullscreen mode

Files ≤4MiB use 1 stripe (avoids overhead on small files), ≤64MiB use 2, larger use 2 for remaining extents.

LNET Networking 

$ sudo lnetctl net show
$ sudo lnetctl net show --verbose
$ sudo lctl ping MDS-IP-ADDRESS@tcp
Enter fullscreen mode Exit fullscreen mode

A successful ping confirms LNET routes to the MDS. Multi-rail bonds multiple interfaces for redundancy/bandwidth. InfiniBand deployments use o2ib instead of tcp.

Quotas and ACLs

Enable and Apply Quotas

From the MDS:

$ sudo lctl conf_param lustrefs.quota.mdt=ug
$ sudo lctl conf_param lustrefs.quota.ost=ug
Enter fullscreen mode Exit fullscreen mode

From the client:

$ sudo lfs setquota -u USER-NAME -b 100M -B 200M -i 10000 -I 20000 /mnt/lustre
$ sudo lfs quota -u USER-NAME /mnt/lustre
Enter fullscreen mode Exit fullscreen mode

-b/-B are soft/hard block limits; -i/-I are soft/hard inode (file count) limits.

POSIX ACLs 

$ sudo mkdir -p /mnt/lustre/acltest
$ sudo setfacl -m u:nobody:rwx /mnt/lustre/acltest
$ sudo getfacl /mnt/lustre/acltest
Enter fullscreen mode Exit fullscreen mode

Monitoring with Prometheus + Grafana

The self-hosted equivalent of CloudWatch for FSx.

Enable Jobstats

On MDS and client:

$ sudo lctl set_param jobid_var=procname_uid
Enter fullscreen mode Exit fullscreen mode

Tags each I/O op with process name + UID (e.g. dd.0).

Build and Run the Lustre Exporter

On the MDS:

$ sudo dnf install -y golang git wget --nogpgcheck
$ git clone https://github.com/GSI-HPC/lustre_exporter.git ~/lustre_exporter
$ cd ~/lustre_exporter
$ GOTOOLCHAIN=auto go build
$ ls -la lustre_exporter
$ sudo cp ~/lustre_exporter/lustre_exporter /usr/local/bin/
$ sudo nano /etc/systemd/system/lustre_exporter.service
Enter fullscreen mode Exit fullscreen mode 
[Unit]
Description=Lustre Prometheus Exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/lustre_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode 
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now lustre_exporter
$ curl -s http://localhost:9169/metrics | grep -E "^lustre_" | head -20
Enter fullscreen mode Exit fullscreen mode

Deploy Prometheus 

$ wget https://github.com/prometheus/prometheus/releases/download/v3.13.1/prometheus-3.13.1.linux-amd64.tar.gz
$ tar -xzf prometheus-3.13.1.linux-amd64.tar.gz
$ sudo mv prometheus-3.13.1.linux-amd64 /opt/prometheus
$ sudo nano /opt/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode 
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'lustre'
    static_configs:
      - targets: ['localhost:9169']
Enter fullscreen mode Exit fullscreen mode 
$ sudo nano /etc/systemd/system/prometheus.service
Enter fullscreen mode Exit fullscreen mode 
[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data
Restart=on-failure

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode 
$ sudo chcon -t bin_t /opt/prometheus/prometheus
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now prometheus
$ curl -s http://localhost:9090/api/v1/targets | grep '"health"'
Enter fullscreen mode Exit fullscreen mode

Should show "health":"up".

Install Grafana 

$ sudo nano /etc/yum.repos.d/grafana.repo
Enter fullscreen mode Exit fullscreen mode 
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
Enter fullscreen mode Exit fullscreen mode 
$ sudo dnf install -y grafana --nogpgcheck
$ sudo systemctl enable --now grafana-server
$ sudo grafana cli --homepath /usr/share/grafana admin reset-admin-password ADMIN-PASSWORD
$ sudo firewall-cmd --permanent --add-port=3000/tcp
$ sudo firewall-cmd --reload
Enter fullscreen mode Exit fullscreen mode

Open http://MDS-IP-ADDRESS:3000, log in as admin:

  1. Connections → Data sources → Add data source → Prometheus, URL http://localhost:9090, Save & test.
  2. Dashboards → New → New dashboard → Add visualization, Prometheus source, Code mode, query lustre_capacity_kibibytes, Run queries, Save.

High Availability with Pacemaker

Active-passive MDS failover via Pacemaker/Corosync — requires a second MDS and a block device both nodes can reach.

Shared-storage failover needs a block device attached to both MDS nodes simultaneously (SAN, iSCSI target, or multi-attach volume). Most cloud block storage attaches to one instance at a time, so treat the shared MDT resource here as illustrative unless your infra supports multi-attach — the cluster stack itself still deploys and forms quorum as shown, ready to pair with shared storage where available.

Install Cluster Packages

On both MDS nodes:

$ sudo dnf config-manager --set-enabled ha
$ sudo dnf install -y pacemaker pcs corosync fence-agents-all --nogpgcheck
$ sudo firewall-cmd --permanent --add-service=high-availability
$ sudo firewall-cmd --reload
$ sudo systemctl enable --now pcsd
$ echo "hacluster:CLUSTER-PASSWORD" | sudo chpasswd
Enter fullscreen mode Exit fullscreen mode

Important: Corosync needs each node's hostname to resolve to one consistent IPv4 address on the cluster network — not loopback. Cloud-init often maps hostnames to 127.0.0.1/::1 in /etc/hosts, which breaks Corosync with a "different IP families" error. Fix: set manage_etc_hosts: false in /etc/cloud/cloud.cfg, remove loopback entries for the node hostnames, and add one IPv4 entry per node pointing at its private-network address (e.g. 10.42.0.4 lustre-mds, 10.42.0.7 lustre-mds2).

Form the Cluster

From the primary MDS:

$ sudo pcs host auth MDS-PRIMARY-HOSTNAME MDS-SECONDARY-HOSTNAME -u hacluster -p CLUSTER-PASSWORD
$ sudo pcs cluster setup lustre-cluster MDS-PRIMARY-HOSTNAME MDS-SECONDARY-HOSTNAME --force
$ sudo pcs cluster start --all
$ sudo pcs cluster enable --all
$ sudo pcs status
Enter fullscreen mode Exit fullscreen mode

Both nodes should show Online with quorum.

Define the MDT Resource 

$ sudo pcs resource create mdt-resource ocf:heartbeat:Filesystem device=/dev/SHARED-DEVICE directory=/mnt/mdt fstype=lustre op monitor interval=30s
Enter fullscreen mode Exit fullscreen mode

Production needs STONITH fencing matched to your infrastructure — the default stonith-enabled=false is for initial verification only.

Verify with IOR and mdtest

Build 

$ sudo dnf install -y gcc gcc-c++ make openmpi openmpi-devel git autoconf automake libtool --nogpgcheck
$ source /etc/profile.d/modules.sh
$ module load mpi/openmpi-x86_64
$ git clone https://github.com/hpc/ior.git ~/ior
$ cd ~/ior
$ git checkout 4.0.0
$ ./bootstrap
$ ./configure
$ make
Enter fullscreen mode Exit fullscreen mode

Run 

$ sudo mkdir -p /mnt/lustre/iortest-dir
$ sudo chmod 777 /mnt/lustre/iortest-dir
$ cd ~/ior/src
$ ./ior -w -r -t 1M -b 100M -F -o /mnt/lustre/iortest-dir/testfile
Enter fullscreen mode Exit fullscreen mode

-w -r write+read test, -t 1M transfer size, -b 100M block size/process, -F file-per-process.

$ sudo mkdir -p /mnt/lustre/mdtest-dir
$ sudo chmod 777 /mnt/lustre/mdtest-dir
$ ./mdtest -n 100 -i 2 -d /mnt/lustre/mdtest-dir
Enter fullscreen mode Exit fullscreen mode

Reports directory/file creation, stat, and removal rates. PSM3 warnings like Failed to get eth0 (unit 0) cpu set on cloud Ethernet are harmless — just OpenMPI falling back from InfiniBand-style transport.

Migrating from AWS FSx for Lustre

Data: FSx exports to S3 via a Data Repository Association (aws fsx create-data-repository-task --type EXPORT_TO_REPOSITORY); pull it down on a Lustre-mounted client with aws s3 sync --endpoint-url <your-s3-endpoint>, which works against any S3-compatible store. Parallelize across clients/prefixes for large datasets, or use lfs migrate + rsync between two mounted file systems directly.

Clients: unmount FSx, mount -t lustre MDS-IP-ADDRESS@tcp:/lustrefs in its place — same POSIX interface, no app code changes.

S3 tiering: Lustre HSM replaces the FSx S3 Data Repository. HSM coordinator runs on the MDS (mdt.*.hsm_control), a copytool (lhsmtool_posix) runs on a client, backend is any S3-compatible bucket exposed via FUSE (e.g. s3fs). lfs hsm_archive copies to S3, lfs hsm_release frees the local copy, reads transparently restore from S3. Robinhood automates archive/release policy. Note: s3fs backends log harmless fsetxattr ... Operation not supported during archive (no xattr support) — archive/restore still succeed.

Sizing: ~2KiB MDT space per file (budget 1GiB MDT per million files); sum OST capacity + 20% overhead; add capacity by formatting new OSTs with the next sequential index — the MGS picks them up automatically.

Watch for:

  • Scaling: FSx auto-scales. Self-hosted needs manual OST additions — size for projected growth upfront
  • S3 sync: FSx syncs bidirectionally automatically. Self-hosted HSM needs explicit archive/restore or a policy engine
  • Kernel compat: client modules must match the client kernel; pin server kernels
  • LNET: default TCP is fine for most cloud setups. InfiniBand needs matching config on every node
  • Metrics: swap CloudWatch for the Prometheus exporter + Grafana
  • Backups: no automated snapshots like FSx — use lfs find + tar/rsync for data, LVM/LDISKFS snapshots for metadata
  • Cost: FSx is per-TiB. Self-hosted cost is compute+storage — calculate your breakeven

Next Steps

The cluster delivers parallel I/O across OSTs with the full Lustre POSIX interface FSx workloads expect. From here:

  • Explore Distributed Namespace (DNE) for scaling metadata across multiple MDS nodes
  • Tune network striping for multi-rail LNET configurations
  • Automate HSM archive/release policy with Robinhood instead of manual lfs hsm_* calls

For the full guide, visit the original article on Vultr Docs.

Top comments (1)

Collapse
 
topstar_ai profile image
Luis Cruz •

I was particularly interested in the section on installing Lustre packages, specifically the requirement for a Whamcloud-patched kernel, which adds an extra layer of complexity to the setup process. The use of a specific kernel version, such as 4.18.0-553.82.1.el8_lustre, and the need to pin it to prevent routine distro kernel updates from breaking the cluster, highlights the importance of careful planning and configuration. Have you considered using a configuration management tool like Ansible to automate and simplify the deployment process for larger-scale clusters, and what are your thoughts on the trade-offs between using a self-hosted Lustre cluster versus a managed service like AWS FSx?