DEV Community

Cover image for Deploying CephFS as a Self-Hosted GCP Filestore Alternative
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya • Originally published at docs.vultr.com

Deploying CephFS as a Self-Hosted GCP Filestore Alternative

#kubernetes #storage #ceph #selfhosted

CephFS is a POSIX-compliant distributed file system built on Ceph, deployable on Kubernetes via the Rook operator with dynamic CSI provisioning and optional NFS-Ganesha export for external clients, a self-managed, Kubernetes-native alternative to GCP Filestore. This guide deploys Rook + Ceph, creates a CephFS filesystem, provisions PVCs, exports over NFS, configures snapshots/backups, sets up CephX auth and tenant isolation, adds monitoring, and covers migrating off Filestore.

Concept Mapping

GCP Filestore CephFS Equivalent Description
Filestore Instance CephFilesystem Distributed shared filesystem via Rook/Ceph
Filestore NFS Export CephNFS + NFS-Ganesha Exposes CephFS over standard NFS
Filestore NFSv3/v4.1 NFS-Ganesha NFSv4 NFS client compatibility
Filestore Capacity Tiers Ceph pools + StorageClasses Performance/placement via pool settings
Filestore Snapshots VolumeSnapshot CSI point-in-time snapshots
Filestore Backups Snapshot export to object storage Backup via snapshots + external storage
Filestore Multi-share CephFS subvolumes Isolated shared volumes via CSI
Filestore Performance Tiers OSD device classes SSD/HDD OSDs with pool tuning
Cloud Monitoring Ceph Dashboard + Prometheus + Grafana Metrics + visualization

Components: Rook Operator (deploys/manages Ceph), MON/MGR (quorum + monitoring), OSDs (data on block storage), MDS (CephFS metadata), Ceph CSI driver (dynamic provisioning), NFS-Ganesha (NFS export), Prometheus + Grafana (monitoring).

Prerequisites: a Kubernetes cluster with 3+ worker nodes, each with a raw unformatted block storage volume attached; kubectl configured; Helm 3; basic Kubernetes storage familiarity.

Install the Rook Ceph Operator 

$ mkdir -p ~/rook-ceph
$ cd ~/rook-ceph
$ kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

Confirm all nodes are Ready.

$ helm repo add rook-release https://charts.rook.io/release
$ helm repo update
$ helm install rook-ceph rook-release/rook-ceph --namespace rook-ceph --create-namespace --version v1.19.7
Enter fullscreen mode Exit fullscreen mode

This installs the operator + CRDs. Pinned to v1.19.7 — a tested release that auto-provisions the rook-ceph.cephfs.csi.ceph.com CSI driver. If you use a newer version, confirm that driver exists before provisioning PVCs.

$ kubectl get pods -n rook-ceph
$ kubectl get crds | grep ceph.rook.io
Enter fullscreen mode Exit fullscreen mode

Deploy the Ceph Cluster 

$ nano ceph-cluster.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19.2.3
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 1
    allowMultiplePerNode: false
  dashboard:
    enabled: true
    ssl: false
  storage:
    useAllNodes: true
    useAllDevices: false
    deviceFilter: "^vd[b-z]"
  resources:
    mgr:
      requests:
        cpu: "100m"
        memory: "1Gi"
      limits:
        memory: "2Gi"
    mon:
      requests:
        cpu: "250m"
        memory: "512Mi"
      limits:
        memory: "1Gi"
  healthCheck:
    daemonHealth:
      mon:
        interval: 45s
      osd:
        interval: 60s
Enter fullscreen mode Exit fullscreen mode

deviceFilter: "^vd[b-z]" tells Rook to use only additional virtio block devices (vdb, vdc, etc.) for OSDs, excluding the primary system disk (typically vda). Check your actual device names with lsblk on each node and adjust if needed.

$ kubectl apply -f ceph-cluster.yaml
$ kubectl get pods -n rook-ceph -w
Enter fullscreen mode Exit fullscreen mode

Deployment can take several minutes while MON/MGR/OSD pods initialize — see the Rook troubleshooting docs if pods stay pending.

$ kubectl get pods -n rook-ceph | grep osd
$ kubectl get cephcluster -n rook-ceph
Enter fullscreen mode Exit fullscreen mode

Should show PHASE: Ready, HEALTH: HEALTH_OK.

Create a CephFS Filesystem 

$ nano ceph-filesystem.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: cephfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true
    resources:
      requests:
        cpu: "100m"
        memory: "256Mi"
      limits:
        memory: "512Mi"
Enter fullscreen mode Exit fullscreen mode

3x replication on both metadata and data pools, 1 active + 1 standby MDS for failover, filesystem data survives resource deletion.

$ kubectl apply -f ceph-filesystem.yaml
$ kubectl get pods -n rook-ceph -w
$ kubectl get pods -n rook-ceph | grep mds
$ kubectl get cephfilesystem -n rook-ceph
Enter fullscreen mode Exit fullscreen mode

Should show PHASE: Ready.

Provision CephFS Volumes 

$ nano cephfs-storageclass.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cephfs-storage
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: cephfs
  pool: cephfs-data0
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f cephfs-storageclass.yaml
$ kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

You'll see cephfs-storage alongside whatever block-storage StorageClasses your cluster ships with by default.

Create a PVC and test pod:

$ nano cephfs-pvc.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: cephfs-storage
  resources:
    requests:
      storage: 5Gi
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f cephfs-pvc.yaml
$ kubectl get pvc
Enter fullscreen mode Exit fullscreen mode 
$ nano cephfs-test-pod.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: v1
kind: Pod
metadata:
  name: cephfs-test-pod
spec:
  containers:
    - name: cephfs-test-container
      image: busybox
      command: ["/bin/sh", "-c"]
      args:
        - while true; do sleep 30; done
      volumeMounts:
        - name: cephfs-storage
          mountPath: /mnt/cephfs
  volumes:
    - name: cephfs-storage
      persistentVolumeClaim:
        claimName: cephfs-pvc
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f cephfs-test-pod.yaml
$ kubectl get pod cephfs-test-pod
$ kubectl exec -it cephfs-test-pod -- sh
Enter fullscreen mode Exit fullscreen mode

Inside the container:

$ echo "CephFS volume test successful" > /mnt/cephfs/test.txt
Enter fullscreen mode Exit fullscreen mode 
$ kubectl exec -it cephfs-test-pod -- cat /mnt/cephfs/test.txt
Enter fullscreen mode Exit fullscreen mode

Export CephFS via NFS 

$ nano ceph-nfs.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: ceph.rook.io/v1
kind: CephNFS
metadata:
  name: ceph-nfs
  namespace: rook-ceph
spec:
  rados:
    pool: .nfs
  server:
    active: 1
    resources:
      requests:
        cpu: "100m"
        memory: "256Mi"
      limits:
        memory: "512Mi"
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f ceph-nfs.yaml
$ kubectl get pods -n rook-ceph | grep nfs
Enter fullscreen mode Exit fullscreen mode

Deploy the toolbox to manage exports:

$ kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/toolbox.yaml
$ kubectl get pods -n rook-ceph | grep tools
Enter fullscreen mode Exit fullscreen mode

Create the export:

$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export create cephfs ceph-nfs /cephfs cephfs --path=/
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export ls ceph-nfs
$ kubectl get svc -n rook-ceph | grep nfs
Enter fullscreen mode Exit fullscreen mode

Mount from a Linux Client

The NFS service is ClusterIP by default — reachable from cluster nodes/pods only. Expose it via LoadBalancer/NodePort for external access.

$ sudo mkdir -p /mnt/cephfs-nfs
Enter fullscreen mode Exit fullscreen mode

Install NFS client tools (Ubuntu/Debian: sudo apt install nfs-common -y; Rocky/AlmaLinux: sudo dnf install nfs-utils -y), then:

$ sudo mount -t nfs NFS-SERVICE-IP:/cephfs /mnt/cephfs-nfs
$ echo "NFS export test successful" | sudo tee /mnt/cephfs-nfs/test.txt
$ cat /mnt/cephfs-nfs/test.txt
Enter fullscreen mode Exit fullscreen mode

Storage Pools and Performance

Beyond the CephFS-backing pools, create dedicated block pools for other workloads:

$ nano ceph-blockpool.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: app-block-pool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
  parameters:
    compression_mode: none
Enter fullscreen mode Exit fullscreen mode

failureDomain: host spreads replicas across nodes; compression is off here for predictable performance.

$ kubectl apply -f ceph-blockpool.yaml
$ kubectl get cephblockpool -n rook-ceph
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status
Enter fullscreen mode Exit fullscreen mode

Snapshots and Backups

Install CSI Snapshot CRDs 

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
$ kubectl get pods -n kube-system | grep snapshot
Enter fullscreen mode Exit fullscreen mode

Create a VolumeSnapshotClass 

$ nano cephfs-snapshotclass.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: cephfs-snapshotclass
driver: rook-ceph.cephfs.csi.ceph.com
deletionPolicy: Delete
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f cephfs-snapshotclass.yaml
$ kubectl get volumesnapshotclass
Enter fullscreen mode Exit fullscreen mode

Snapshot and Restore 

$ nano cephfs-snapshot.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: cephfs-snapshot
spec:
  volumeSnapshotClassName: cephfs-snapshotclass
  source:
    persistentVolumeClaimName: cephfs-pvc
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f cephfs-snapshot.yaml
$ kubectl get volumesnapshot
Enter fullscreen mode Exit fullscreen mode

Restore to a new PVC (must be ≥ source size):

$ nano cephfs-restore-pvc.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-restore-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: cephfs-storage
  resources:
    requests:
      storage: 5Gi
  dataSource:
    name: cephfs-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f cephfs-restore-pvc.yaml
$ kubectl get pvc
Enter fullscreen mode Exit fullscreen mode

Scheduled Snapshots via CronJob

Kubernetes doesn't schedule snapshots natively — use a CronJob with RBAC to manage VolumeSnapshot resources.

$ nano cephfs-snapshot-rbac.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-snapshot-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-snapshot-role
  namespace: default
rules:
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-snapshot-rolebinding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: cephfs-snapshot-sa
    namespace: default
roleRef:
  kind: Role
  name: cephfs-snapshot-role
  apiGroup: rbac.authorization.k8s.io
Enter fullscreen mode Exit fullscreen mode 
$ kubectl apply -f cephfs-snapshot-rbac.yaml
$ nano cephfs-snapshot-cronjob.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cephfs-snapshot-cronjob
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: cephfs-snapshot-sa
          restartPolicy: OnFailure
          containers:
            - name: snapshot-creator
              image: alpine/k8s:1.36.2
              command:
                - /bin/sh
                - -c
                - |
                  SNAPSHOT_NAME="cephfs-snapshot-$(date +%s)"
                  cat <<EOF | kubectl apply -f -
                  apiVersion: snapshot.storage.k8s.io/v1
                  kind: VolumeSnapshot
                  metadata:
                    name: ${SNAPSHOT_NAME}
                  spec:
                    volumeSnapshotClassName: cephfs-snapshotclass
                    source:
                      persistentVolumeClaimName: cephfs-pvc
                  EOF
Enter fullscreen mode Exit fullscreen mode

Every 6 hours, timestamped snapshot names.

$ kubectl apply -f cephfs-snapshot-cronjob.yaml
$ kubectl get cronjob
Enter fullscreen mode Exit fullscreen mode

Authentication and Access Control

POSIX Permissions 

$ kubectl exec -it cephfs-test-pod -- sh
Enter fullscreen mode Exit fullscreen mode 
$ mkdir -p /mnt/cephfs/tenant-a /mnt/cephfs/tenant-b
$ chown 1001:1001 /mnt/cephfs/tenant-a
$ chmod 700 /mnt/cephfs/tenant-a
$ ls -ld /mnt/cephfs/tenant-a
$ exit
Enter fullscreen mode Exit fullscreen mode

Restricted CephFS Client Users 

$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- bash
Enter fullscreen mode Exit fullscreen mode 
$ ceph fs authorize cephfs client.tenant-a /tenant-a rw
$ ceph auth get-key client.tenant-a
$ ceph auth get client.tenant-a
$ exit
Enter fullscreen mode Exit fullscreen mode

The path in ceph fs authorize resolves from the CephFS filesystem root, not the pod's PVC mount point. If the CSI driver provisions a subvolume rather than the root, get its actual path with ceph fs subvolume getpath first.

Verify restriction by mounting directly with the kernel client:

$ sudo mount -t ceph MON-IP:6789:/tenant-a /mnt/tenant-a-test -o name=tenant-a,secret=CLIENT-KEY
$ ls /mnt/tenant-a-test
Enter fullscreen mode Exit fullscreen mode

A permission error outside the authorized path confirms CephX is enforcing correctly.

Monitoring

Ceph Dashboard 

$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status
$ kubectl get pods -n rook-ceph | grep mgr
$ kubectl get svc -n rook-ceph | grep dashboard
$ kubectl port-forward -n rook-ceph svc/rook-ceph-mgr-dashboard 7000:7000
Enter fullscreen mode Exit fullscreen mode

Open http://127.0.0.1:7000.

$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
Enter fullscreen mode Exit fullscreen mode

Log in as admin with that password.

Prometheus + Grafana 

$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph mgr module enable prometheus
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph mgr module ls | grep prometheus
$ kubectl get svc -n rook-ceph | grep mgr
Enter fullscreen mode Exit fullscreen mode 
$ helm repo add grafana-community https://grafana-community.github.io/helm-charts
$ helm repo update
$ helm install grafana grafana-community/grafana --namespace monitoring --create-namespace
$ kubectl get pods -n monitoring
$ kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
$ kubectl port-forward -n monitoring svc/grafana 3000:80
Enter fullscreen mode Exit fullscreen mode

Open http://127.0.0.1:3000, log in as admin with the retrieved password.

Verify the Deployment 

$ kubectl get pvc
$ kubectl get pod cephfs-test-pod
$ kubectl exec -it cephfs-test-pod -- sh -c 'echo "CephFS deployment verification successful" > /mnt/cephfs/verify.txt'
$ kubectl exec -it cephfs-test-pod -- cat /mnt/cephfs/verify.txt
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export ls ceph-nfs
$ kubectl get svc -n rook-ceph | grep nfs
$ kubectl get volumesnapshot
$ kubectl get pvc cephfs-restore-pvc
$ kubectl get svc -n rook-ceph | grep dashboard
$ kubectl get pods -n monitoring
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status
Enter fullscreen mode Exit fullscreen mode

Migrating from GCP Filestore

Data: mount both the Filestore NFS export and the CephFS NFS export on the same client, then rsync -avh between them (preserves ownership/permissions/symlinks/timestamps). Parallelize across clients/subdirectories for large datasets, or copy directly into a pod with the CephFS PVC mounted.

Applications: swap each workload's PVC storageClassName from your Filestore class (e.g. filestore-rwx) to cephfs-storage, recreate the PVC, restart the workload — CephFS gives the same POSIX + ReadWriteMany semantics, so no code changes needed. Non-Kubernetes apps just repoint their NFS mount at the CephFS export.

Snapshots/backups: VolumeSnapshot resources replace Filestore snapshots (as configured above). Recreate any scheduled-backup workflow with a CronJob or external tool, storing long-term copies in an S3-compatible object storage backend.

Sizing: CephFS defaults to 3x replication — budget raw storage at ~3x usable capacity. Add capacity by attaching more block storage to worker nodes and letting Rook provision new OSDs via the device filter; Ceph rebalances automatically. Use separate pools/device classes to match Filestore's capacity tiers.

Watch for:

  • NFS version: some Filestore setups use NFSv3; NFS-Ganesha defaults to NFSv4 — check client compatibility
  • Performance tuning: Filestore tiers are automatic; CephFS performance depends on OSD type, replication, network, and pool config — you own the tuning
  • Multi-share: maps to CephFS subdirectories or separate subvolumes for tenant isolation
  • Auth: GCP IAM → CephX + POSIX permissions/ACLs
  • Monitoring: Cloud Monitoring → Ceph Dashboard + Prometheus + Grafana
  • Capacity planning: factor in the 3x replication overhead before migrating production data

Next Steps

CephFS is running with dynamic provisioning, NFS export, snapshots, CephX-enforced tenant isolation, and full monitoring. From here:

  • Add more CephBlockPools with different device classes (SSD vs HDD) for tiered performance
  • Automate scheduled snapshot pruning alongside the creation CronJob
  • Explore Ceph's multi-site replication for cross-region DR

For the full guide, visit the original article on Vultr Docs.

Top comments (0)