DEV Community

Cover image for Raspberry Pi Kubernetes Cluster: NFS Storage (Issue #5)
Jules
Jules

Posted on • Originally published at juliantellez.com

Raspberry Pi Kubernetes Cluster: NFS Storage (Issue #5)

#kubernetes #homelab #raspberrypi

Welcome to the Raspberry Pi Kubernetes home lab series. A bite sized informative guide to help you provision a cluster from scratch. By now we should have our k8s cluster up and running; great job everyone.

Today's issue will walk you through installing and creating a Network File System (NFS) server so it can be used as our default cluster storage.

1. Requirements

2. Locate the hard drive

Plug your hard drive into one of your slave nodes and SSH into it to find its location.

sudo lsblk -o UUID,NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,MODEL
Enter fullscreen mode Exit fullscreen mode

3. Format the hard drive with mkfs.ext3

We can see that our hard drive is located at /dev/sda2. Let's go ahead and format the hard drive's filesystem so it can be used by the Linux system. Other examples include /dev/sdb2.

sudo mkfs.ext3 -L k8s-volume /dev/sda2
Enter fullscreen mode Exit fullscreen mode

4. Create a directory for the volume and mount it

This could be in any location. We are going to "namespace" it under k8s-mount and label it k8s-volume-1.

sudo mkdir -p /k8s-mount/k8s-volume-1
sudo mount /dev/sda2 /k8s-mount/k8s-volume-1 # mount disk
Enter fullscreen mode Exit fullscreen mode

5. Install the NFS server 

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install nfs-kernel-server -y
Enter fullscreen mode Exit fullscreen mode

6. Start the server 

sudo systemctl start nfs-server
sudo systemctl status nfs-server # verify it is running
Enter fullscreen mode Exit fullscreen mode

7. Allocate permissions

In order to have access to your NFS server we need to create some read/write permissions.

# owner (read/write) group (read)
sudo find /k8s-mount/k8s-volume-1/ -type d -exec chmod 755 {} \;
sudo find /k8s-mount/k8s-volume-1/ -type f -exec chmod 644 {} \;
sudo find /k8s-mount/k8s-volume-1/ -type f -exec chmod 777 {} \;
Enter fullscreen mode Exit fullscreen mode

8. Set up remote access

The line below will give permissions to all hosts in the network, so be careful!

sudo nano /etc/exports
Enter fullscreen mode Exit fullscreen mode 
/k8s-mount/k8s-volume-1/ *(rw,sync,no_subtree_check,insecure)
Enter fullscreen mode Exit fullscreen mode

We can set the permissions to be restrictive per host. Let me know if you would like to see how this is done. In the meantime, here is a walk through of this setup and the official docs too.

9. Export the etc config 

sudo exportfs -ra # export config
sudo exportfs -v  # see current config
showmount -e      # see what is being exported
Enter fullscreen mode Exit fullscreen mode

10. Install the NFS client on all the other worker nodes 

# ssh ubuntu@192.168.0.xx # WORKER_NODE
sudo apt install nfs-common -y
Enter fullscreen mode Exit fullscreen mode

11. Create volume manifests 

# system-volume.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: system-volume
  labels:
    name: system-volume
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-k8s-volume-1
  labels:
    type: local
spec:
  storageClassName: k8s-volume-1
  capacity:
    storage: 1Ti
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.26
    path: /k8s-mount/k8s-volume-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-k8s-volume-1
spec:
  storageClassName: k8s-volume-1
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Ti
Enter fullscreen mode Exit fullscreen mode

12. Apply the manifests 

kubectl apply -k path-to-manifest.yaml
kubectl --namespace system-volume get pv,pvc
Enter fullscreen mode Exit fullscreen mode

Bonus

This isn't a required section but more of an educational addendum. Since we have given global access to our drive, we can now access it anywhere from within the network.

1. Find nodes 

nmap -sP 192.168.0.1/24 # list nodes

WORKER_NODE_1=192.168.0.24
WORKER_NODE_2=192.168.0.26 # hard drive lives here
Enter fullscreen mode Exit fullscreen mode

2. Find the volume on the network 

ssh ubuntu@$WORKER_NODE_1 # a worker node

sudo apt install nfs-common

showmount -e $WORKER_NODE_2 # see what is being exported

# output
Export list for 192.168.0.26:
/k8s-mount/k8s-volume-1 *
Enter fullscreen mode Exit fullscreen mode

3. Access the hard drive 

sudo mkdir -p /test
sudo mount -t nfs $WORKER_NODE_2:/k8s-mount/k8s-volume-1 /test/
ls -al /test/ # inspect drive
Enter fullscreen mode Exit fullscreen mode

4. End of test

Clear the test area by removing the files created previously.

sudo umount /test
sudo rm -rf /test
Enter fullscreen mode Exit fullscreen mode

We are now ready to start using our local storage. It is important that you know how to access the nodes independently and that you understand how to manipulate the NFS server across the network.

Leave a comment if you have any questions. We will be integrating with MetalLB's load balancer next. Happy coding.

First published in my LinkedIn newsletter, Built from Scratch.

This was originally published on juliantellez.com. It is part of my newsletter on video, streaming, and building reliable systems from scratch: the architecture, the tradeoffs, and the occasional 3am incident.

If it was useful, subscribe for the next one. No spam, unsubscribe anytime. If you are building something similar and want to compare notes, say hello. I read every message.

Built from scratch, like everything here.

Top comments (0)