DEV Community

Stefor07
Stefor07

Posted on

Shared Filesystems (NFS) on OpenStack without Manila

#openstack #nfs #linux #devops

Need shared storage between OpenStack instances, but Manila isn't available in your deployment? You're not out of luck. With a Cinder volume and a plain Ubuntu VM running NFS, you can get a solid shared filesystem up and running in a few minutes.

In this post I'll walk through the whole setup, from creating a dedicated project to mounting the share on client VMs.

TL;DR

  • Create a dedicated project and user (shares)
  • Spin up an Ubuntu 22.04 VM as the NFS server
  • Attach a Cinder volume, format it, and mount it on /srv/nfs
  • Export /srv/nfs/share over NFSv4 (TCP 2049)
  • Lock it down with Security Groups
  • Mount it from any client VM with nfs-common

The architecture 

OpenStack
│
├── Project: shares
│   ├── User: shares
│   ├── NFS Server VM (Ubuntu 22.04)
│   │   ├── System Disk
│   │   ├── Cinder Volume
│   │   │   └── /srv/nfs/share
│   │   └── NFS Server
│   │       └── TCP 2049
│   │
│   └── Client VM(s)
│       └── /mnt/shared-storage
│
└── Neutron Network
    └── Private connectivity between instances
Enter fullscreen mode Exit fullscreen mode

The idea is simple: Cinder gives us persistent block storage, NFS turns it into shared file storage. The size of the Cinder volume defines the capacity of the share.

⚠️ Heads up: this is not a Manila replacement feature-for-feature. There are no share quotas, and a single NFS server is a single point of failure. More on that at the end.

Prerequisites

  • Access to an OpenStack deployment with the openstack CLI installed
  • Permissions to create projects and users (admin) for step 1
  • The member role available in your deployment
  • Instances that can reach each other over a Neutron network

1. Create a dedicated project and user

I like keeping storage infrastructure separate from general-purpose workloads. A dedicated project makes resource management much easier.

# Create the project
openstack project create \
  --description "Dedicated project for shared storages" \
  shares

# Create the user (you'll be prompted for a password)
openstack user create \
  --description "User for managing storage instances" \
  --password-prompt \
  shares

# Give the user the member role in the project
openstack role add \
  --project shares \
  --user shares \
  member

# Set the default project
openstack user set \
  --project shares \
  shares
Enter fullscreen mode Exit fullscreen mode

Verify everything:

openstack project show shares
openstack user show shares

openstack role assignment list \
  --project shares \
  --user shares \
  --names
Enter fullscreen mode Exit fullscreen mode

The member role must exist in your deployment, and your OpenStack policies must allow the operations we're about to run.

2. Create the RC file

Create shares-openrc.sh with the credentials for the new user:

#!/usr/bin/env bash

export OS_AUTH_URL="https://<KEYSTONE_ENDPOINT>:5000/v3"
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_TYPE="password"

export OS_PROJECT_NAME="shares"
export OS_PROJECT_DOMAIN_NAME="Default"

export OS_USERNAME="shares"
export OS_USER_DOMAIN_NAME="Default"

export OS_PASSWORD="<USER_PASSWORD>"
Enter fullscreen mode Exit fullscreen mode

Adjust the Keystone endpoint, port, and domains to match your environment. Then lock the file down and load it:

chmod 600 shares-openrc.sh
source shares-openrc.sh

# Sanity check
openstack token issue
openstack project show shares
Enter fullscreen mode Exit fullscreen mode

🔐 This file contains a plaintext password. Protect it, or use a supported alternative authentication method (such as application credentials) where available.

3. Upload a cloud image

Grab a qcow2 cloud image for your architecture, for example:

This guide uses Ubuntu 22.04. Upload it to Glance:

source shares-openrc.sh

openstack image create "Ubuntu-22.04" \
  --file jammy-server-cloudimg-amd64.img \
  --disk-format qcow2 \
  --container-format bare \
  --private
Enter fullscreen mode Exit fullscreen mode

Replace the filename with the one you actually downloaded, then verify:

openstack image list
openstack image show Ubuntu-22.04
Enter fullscreen mode Exit fullscreen mode

4. Create an SSH key pair 

mkdir -p ~/.ssh
chmod 700 ~/.ssh

openstack keypair create \
  --private-key ~/.ssh/shares-key.pem \
  shares-key

chmod 600 ~/.ssh/shares-key.pem
Enter fullscreen mode Exit fullscreen mode

Verify it:

openstack keypair show shares-key
openstack keypair list
Enter fullscreen mode Exit fullscreen mode

Keep that private key safe, since you'll need it to SSH into the server.

5. Launch the NFS server instance

First, check that everything you need exists:

openstack image list
openstack flavor list
openstack network list
openstack keypair list
openstack security group list
Enter fullscreen mode Exit fullscreen mode

Then create the VM:

openstack server create \
  --image Ubuntu-22.04 \
  --flavor <FLAVOR_NAME> \
  --network <NETWORK_NAME> \
  --key-name shares-key \
  --security-group <SECURITY_GROUP_NAME> \
  nfs-server
Enter fullscreen mode Exit fullscreen mode

Wait for the status to become ACTIVE, and grab its addresses:

openstack server show nfs-server \
  -c status \
  -c addresses
Enter fullscreen mode Exit fullscreen mode

6. Floating IP (only if you need it)

Instances on the same private network can already talk to each other, so you don't necessarily need a Floating IP. You'll need one if you want to reach the server from an external or administrative network (for SSH, for example).

# Find the external network
openstack network list --external

# Allocate a Floating IP
openstack floating ip create <EXTERNAL_NETWORK_NAME>

# Associate it
openstack server add floating ip \
  nfs-server \
  <FLOATING_IP_ADDRESS>

# Verify
openstack server show nfs-server
openstack floating ip list
Enter fullscreen mode Exit fullscreen mode

💡 For NFS traffic between private instances, always prefer the server's private IP. Don't expose storage services through a Floating IP unless you really have to.

7. Open SSH in the Security Group

Find the port and Security Group attached to the server:

openstack port list --server nfs-server
openstack port show <PORT_ID>
openstack security group rule list <SECURITY_GROUP_ID>
Enter fullscreen mode Exit fullscreen mode

If there's no SSH rule, add one restricted to your admin network:

openstack security group rule create \
  --ingress \
  --protocol tcp \
  --dst-port 22 \
  --remote-ip <ADMIN_CIDR> \
  <SECURITY_GROUP_ID>
Enter fullscreen mode Exit fullscreen mode

For a single admin host, use a /32 such as 192.168.10.15/32.

🚫 Avoid 0.0.0.0/0 for SSH unless you have a specific, justified reason.

Now connect (the default Ubuntu user is ubuntu):

# Via private IP
ssh -i ~/.ssh/shares-key.pem ubuntu@<NFS_SERVER_PRIVATE_IP>

# Or via Floating IP
ssh -i ~/.ssh/shares-key.pem ubuntu@<FLOATING_IP_ADDRESS>
Enter fullscreen mode Exit fullscreen mode

From here on, most of the work happens inside the VM.

8. Create and attach the Cinder volume

This is where the storage capacity is defined. Unlike Manila, where you pick the size at share creation, here the Cinder volume size is the capacity.

openstack volume create \
  --size 100 \
  nfs-share-volume
Enter fullscreen mode Exit fullscreen mode

Change 100 to whatever you need (in GB), then check it:

openstack volume show nfs-share-volume
openstack volume list
Enter fullscreen mode Exit fullscreen mode

Wait for the available status, then attach it:

openstack server add volume \
  nfs-server \
  nfs-share-volume
Enter fullscreen mode Exit fullscreen mode

Verify the attachment:

openstack server show nfs-server
openstack volume show nfs-share-volume
Enter fullscreen mode Exit fullscreen mode

9. Format and mount the volume (inside the VM)

⚠️ Formatting destroys all data on the target device. Double-check the device name, and never format the OS disk.

Identify the new disk:

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINTS
Enter fullscreen mode Exit fullscreen mode

In this example it's /dev/vdb, but yours may differ.

Format it with one filesystem of your choice.

ext4:

sudo mkfs.ext4 /dev/vdb
Enter fullscreen mode Exit fullscreen mode

or XFS:

sudo apt update
sudo apt install -y xfsprogs
sudo mkfs.xfs /dev/vdb
Enter fullscreen mode Exit fullscreen mode

Create the mount point and mount it:

sudo mkdir -p /srv/nfs
sudo mount /dev/vdb /srv/nfs

df -h /srv/nfs
lsblk -f
Enter fullscreen mode Exit fullscreen mode

Make it persistent

Grab the filesystem UUID:

sudo blkid /dev/vdb
Enter fullscreen mode Exit fullscreen mode 
/dev/vdb: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="ext4"
Enter fullscreen mode Exit fullscreen mode

Add an entry to /etc/fstab:

sudo nano /etc/fstab
Enter fullscreen mode Exit fullscreen mode

For ext4:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /srv/nfs ext4 defaults,nofail 0 2
Enter fullscreen mode Exit fullscreen mode

For XFS:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /srv/nfs xfs defaults,nofail 0 0
Enter fullscreen mode Exit fullscreen mode

Test it without rebooting:

sudo mount -a
findmnt /srv/nfs
Enter fullscreen mode Exit fullscreen mode

🧠 nofail lets the VM boot even if the volume is missing. In production, make sure that failure can't cause NFS to export the underlying system filesystem by accident.

10. Install and configure the NFS server 

sudo apt update
sudo apt install -y nfs-kernel-server

sudo systemctl enable --now nfs-kernel-server
sudo systemctl status nfs-kernel-server
Enter fullscreen mode Exit fullscreen mode

Create the shared directory on the mounted volume:

sudo mkdir -p /srv/nfs/share
Enter fullscreen mode Exit fullscreen mode

For a first connectivity test, you can use broad permissions:

sudo chown root:root /srv/nfs/share
sudo chmod 0777 /srv/nfs/share
Enter fullscreen mode Exit fullscreen mode

⚠️ Don't keep 0777 in production. Set ownership and permissions based on your workloads and their UID/GID requirements.

Check that the directory really lives on the Cinder volume:

df -h /srv/nfs/share
Enter fullscreen mode Exit fullscreen mode

Configure the export 

sudo nano /etc/exports
Enter fullscreen mode Exit fullscreen mode

Add (replacing <CLIENT_CIDR> with the subnet allowed to access the share):

/srv/nfs/share <CLIENT_CIDR>(rw,sync,no_subtree_check,root_squash)
Enter fullscreen mode Exit fullscreen mode

For example:

/srv/nfs/share 192.168.10.0/24(rw,sync,no_subtree_check,root_squash)
Enter fullscreen mode Exit fullscreen mode

What this does:

  • rw: read/write access
  • sync: synchronous writes
  • no_subtree_check: skips subtree checking
  • root_squash: remote root doesn't get root privileges on the server

Apply and verify:

sudo exportfs -rav
sudo exportfs -v
sudo ss -lntup | grep ':2049'
Enter fullscreen mode Exit fullscreen mode

11. Open the right ports

Port Protocol Needed for
2049 TCP NFS (NFSv4)
445 TCP SMB/CIFS, only if you also run Samba

Port 445 is not required for NFS.

Allow NFS from your client subnet:

openstack security group rule create \
  --ingress \
  --protocol tcp \
  --dst-port 2049 \
  --remote-ip <CLIENT_CIDR> \
  <SECURITY_GROUP_ID>
Enter fullscreen mode Exit fullscreen mode

Only if you're also deploying SMB/CIFS:

openstack security group rule create \
  --ingress \
  --protocol tcp \
  --dst-port 445 \
  --remote-ip <CLIENT_CIDR> \
  <SECURITY_GROUP_ID>
Enter fullscreen mode Exit fullscreen mode

Opening 445 alone doesn't give you SMB. You still need Samba (or similar) installed and configured.

Verify:

openstack security group rule list <SECURITY_GROUP_ID>
Enter fullscreen mode Exit fullscreen mode

This guide assumes NFSv4 over TCP 2049. If you use NFSv3, additional RPC-related ports and services come into play.

12. Mount the share from a client VM

On an Ubuntu client:

sudo apt update
sudo apt install -y nfs-common

sudo mkdir -p /mnt/shared-storage

sudo mount -t nfs <NFS_SERVER_PRIVATE_IP>:/srv/nfs/share /mnt/shared-storage
Enter fullscreen mode Exit fullscreen mode

For example:

sudo mount -t nfs 192.168.10.20:/srv/nfs/share /mnt/shared-storage
Enter fullscreen mode Exit fullscreen mode

Verify it works:

df -h /mnt/shared-storage
findmnt /mnt/shared-storage

touch /mnt/shared-storage/test-file.txt
ls -l /mnt/shared-storage
Enter fullscreen mode Exit fullscreen mode

If you mount the same export from a second client, you should see the file there too. That's your shared storage working. 🎉

Mount automatically at boot

Add this to /etc/fstab on the client:

<NFS_SERVER_PRIVATE_IP>:/srv/nfs/share /mnt/shared-storage nfs defaults,_netdev,nofail 0 0
Enter fullscreen mode Exit fullscreen mode

And test it:

sudo mount -a
findmnt /mnt/shared-storage
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

On the server:

sudo systemctl status nfs-kernel-server
sudo exportfs -v
sudo ss -lntup | grep ':2049'
df -h /srv/nfs/share
Enter fullscreen mode Exit fullscreen mode

From the client:

ping <NFS_SERVER_PRIVATE_IP>
nc -zv <NFS_SERVER_PRIVATE_IP> 2049
showmount -e <NFS_SERVER_PRIVATE_IP>
Enter fullscreen mode Exit fullscreen mode

A few notes on those checks:

  • ping may fail if ICMP is blocked, even when NFS works fine. The nc check on port 2049 is the more meaningful one.
  • showmount relies on mountd and may fail on an NFSv4-only setup. A failure there doesn't mean NFSv4 is down.

Common issues

Symptom What to check
Connection timeout Neutron routing, Security Group rules, server IP
Permission denied /etc/exports, export options, filesystem ownership, client UID/GID
Mount fails after reboot /etc/fstab, filesystem UUID, whether the Cinder volume is attached
Directory appears empty Make sure the volume is mounted on /srv/nfs before NFS starts exporting
Cannot write files Directory permissions and export configuration
Out of space df -h and the Cinder volume size

Final thoughts

You now have shared file storage across OpenStack instances without Manila: a Cinder volume, exported by an NFS server, and mounted by your clients over a private network.

Before you take this to production, keep these points in mind:

  • A dedicated project is organization, not isolation. It doesn't replace proper network isolation and access control.
  • Back up the Cinder volume according to your operational requirements.
  • Restrict SSH and NFS to authorized source networks only.
  • Keep UID/GID mappings consistent across clients, and set proper filesystem permissions.
  • Single point of failure: one NFS server means one thing that can go down. Production setups may need HA, recovery, and monitoring.
  • No quotas: the Cinder volume size sets the filesystem capacity. It's not a Manila-style share quota.

Have you built something similar, or found a better way to handle HA for this setup? Let me know in the comments! 👇

Top comments (0)