Backing Up Qdrant Snapshots Correctly: From emptyDir to NFS Persistent Backups

The Qdrant snapshot API returns 200 OK with a JSON body containing a filename, a size, and a creation timestamp. That response confirms a tar archive got written somewhere inside the container. It says nothing about whether that somewhere survives a pod restart, and with the manifest most people start from, it doesn't.

That's the whole problem in one sentence. The API is honest about what it did. It has no opinion about where your storage lives, and neither does your monitoring, which is why a broken Qdrant backup can report success for a very long time before anyone finds out.

Who should care

If you're running Qdrant in Kubernetes as the vector store behind a RAG pipeline, an agent memory layer, or a semantic search index, the embeddings in it are expensive. Not expensive to store, expensive to regenerate. Re-embedding a few hundred thousand chunks means re-chunking the source documents, re-running the embedding model, and hoping the model version you used originally is still pinned somewhere. A restore from a 20 MB snapshot file takes ninety seconds. A rebuild takes an afternoon and a stack of GPU hours.

This post walks the progression from the naive setup to one that actually holds: emptyDir → PersistentVolumeClaim → external NFS target, plus the retention logic that keeps the whole thing from eating your cluster.

Stage 1: the emptyDir trap

Here's the deployment shape you'll find in a dozen blog posts and at least two Helm chart examples:

volumes:
  - name: qdrant-storage
    emptyDir: {}
containers:
  - name: qdrant
    image: qdrant/qdrant:latest
    volumeMounts:
      - name: qdrant-storage
        mountPath: /qdrant/storage

Nothing about that manifest is wrong for a demo. It's wrong for anything you care about, and the failure mode is worse than "I lose my data on restart" because of where Qdrant puts snapshots by default.

Qdrant writes snapshots to ./snapshots relative to its working directory, which in the official image resolves to /qdrant/snapshots. That path is not /qdrant/storage. So even if you got the storage volume right and skipped emptyDir for the data, an unmounted snapshots path means your snapshots land on the container's writable layer, which is destroyed on every pod recreation. Node drains during a cluster upgrade, an OOM kill, a rollout from a changed config map, all of it wipes the snapshot directory silently.

The check takes one command:

kubectl exec -n vectordb deploy/qdrant -- df -h /qdrant/storage /qdrant/snapshots

If either path shows overlay as the filesystem, that directory is ephemeral. If /qdrant/snapshots shows the same device as /qdrant/storage, you have a different problem, which is stage 2.

Stage 2: the PVC that isn't a backup

The obvious fix is to put both paths on a PersistentVolumeClaim. Data survives restarts, snapshots survive restarts, everyone goes home. Except now your backups live on the same block device as the thing they're backing up.

I wrote about this failure shape in Your Vector DB Snapshots Are Landing on the Same Disk That Will Fail, and it's the single most common mistake in self-hosted vector database setups. A snapshot on the same volume protects you against exactly one class of event: application-level corruption. Someone drops a collection, an ingestion job writes garbage vectors, a schema migration goes sideways. Fine, restore from the local snapshot.

It protects you against nothing else. Replica corruption, a Longhorn volume that won't attach, a filesystem that mounts read-only after an unclean shutdown, an accidental kubectl delete pvc with a Delete reclaim policy. In every one of those cases the snapshot goes down with the ship.

There's a second, sneakier problem with co-locating snapshots on the data PVC: capacity. Qdrant snapshots are not incremental and they are not deduplicated. Creating a snapshot of a collection requires roughly as much free space as the collection itself, because Qdrant is tarring up the segment files. A 20 GiB PVC holding 12 GiB of vectors cannot take a snapshot. You get a failed request and a partially written archive that still occupies space until something cleans it up.

Nothing cleans it up. Qdrant has no built-in retention for collection snapshots. Every call to POST /collections/{name}/snapshots creates a new file with a fresh timestamp and leaves every previous one exactly where it was. Wire that to an hourly CronJob, forget about it for a few weeks, and you end up with north of a hundred snapshot archives on a volume sized for the live data. The PVC fills, Qdrant's write path fails, and the failure looks like an application bug rather than a storage one.

Stage 3: snapshots leave the cluster's failure domain

The pattern that holds separates three things that people tend to collapse into one:

Layer	Lives on	Protects against
Qdrant data	Longhorn RWO PVC	nothing, this is the live copy
Qdrant snapshot (transient)	same PVC, deleted after export	app-level corruption, fast rollback
Exported snapshot	external NFS share	disk failure, volume loss, cluster loss

The transient snapshot is the important nuance. You still create the snapshot locally, because that's the only interface Qdrant gives you, but you treat the local copy as a temporary artifact with a lifetime of a few minutes rather than as the backup.

Do not mount NFS as Qdrant's storage path

Before the manifests, the caveat that will save you a week: Qdrant's storage directory needs a local block device. Qdrant memory-maps segment files and uses RocksDB for payload storage, and neither of those behaves well over NFS. File locking semantics differ, mmap over NFS has a very different consistency model, and the performance drop on HNSW search is severe enough that you'll notice it in p99 latency immediately. Qdrant's own documentation recommends against network storage for the data path.

NFS is fine as a destination for finished snapshot archives. Those are sequential writes of an immutable tar file, which is the workload NFS is actually good at.

The data volume

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: qdrant-data
  namespace: vectordb
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn
  resources:
    requests:
      storage: 40Gi   # 2x expected collection size, snapshots need headroom

That sizing rule matters. If your collections total 15 GiB, a 20 GiB PVC will fail to snapshot. Size for double, or move the snapshots path to its own smaller volume.

The NFS target

An NFS-backed PV, statically provisioned, mounted only by the backup job:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: qdrant-backups-nfs
spec:
  capacity:
    storage: 200Gi
  accessModes: [ReadWriteMany]
  persistentVolumeReclaimPolicy: Retain   # never let K8s delete the backups
  nfs:
    server: 10.0.0.50
    path: /export/backups/qdrant
  mountOptions:
    - nfsvers=4.1
    - hard          # block on server outage instead of returning EIO mid-write
    - timeo=600
    - retrans=2

hard over soft is deliberate. A soft mount returns an I/O error after the timeout, which means a partially written snapshot archive that looks like a file and restores like a coaster. A hard mount blocks, the CronJob hits its activeDeadlineSeconds, and you get a clean failure you can alert on.

Retain on the reclaim policy is the other non-negotiable. If someone deletes the PVC, you want the PV to go to Released and the data on the NFS server to stay exactly where it is.

The export job

The CronJob runs in its own pod and never touches Qdrant's PVC. That's not just hygiene, it's a hard constraint: a Longhorn RWO volume can only be attached to one node at a time, so a backup pod scheduled somewhere else physically cannot mount it. Everything moves over the HTTP API instead.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: qdrant-snapshot-export
  namespace: vectordb
spec:
  schedule: "0 */6 * * *"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      activeDeadlineSeconds: 3600
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: export
              image: curlimages/curl:8.11.0
              command: ["/bin/sh", "/scripts/export.sh"]
              env:
                - name: QDRANT_URL
                  value: "http://qdrant.vectordb.svc.cluster.local:6333"
                - name: QDRANT_API_KEY
                  valueFrom:
                    secretKeyRef: { name: qdrant-api, key: api-key }
              volumeMounts:
                - { name: backups, mountPath: /backups }
                - { name: scripts, mountPath: /scripts }
          volumes:
            - name: backups
              persistentVolumeClaim: { claimName: qdrant-backups }
            - name: scripts
              configMap: { name: qdrant-export-script }

concurrencyPolicy: Forbid prevents a slow export from stacking on top of the next scheduled run, which is how you accidentally create three concurrent snapshots and fill the data PVC.

The script itself does four things: create, download, verify, delete the local copy.

#!/bin/sh
set -eu

AUTH="api-key: ${QDRANT_API_KEY}"
DEST="/backups/$(date -u +%Y-%m-%d)"
mkdir -p "$DEST"

COLLECTIONS=$(curl -sf -H "$AUTH" "$QDRANT_URL/collections" \
  | grep -o '"name":"[^"]*"' | cut -d'"' -f4)

for C in $COLLECTIONS; do
  # wait=true blocks until the archive is fully written
  SNAP=$(curl -sf -X POST -H "$AUTH" \
    "$QDRANT_URL/collections/$C/snapshots?wait=true" \
    | grep -o '"name":"[^"]*"' | cut -d'"' -f4)

  curl -sf -H "$AUTH" \
    "$QDRANT_URL/collections/$C/snapshots/$SNAP" -o "$DEST/$SNAP"

  # a truncated tar is worse than no backup, so check before pruning
  if ! tar -tf "$DEST/$SNAP" >/dev/null 2>&1; then
    echo "corrupt archive for $C, keeping remote snapshot" >&2
    exit 1
  fi
  sha256sum "$DEST/$SNAP" >> "$DEST/SHA256SUMS"

  curl -sf -X DELETE -H "$AUTH" \
    "$QDRANT_URL/collections/$C/snapshots/$SNAP"
done

The tar -tf check is the part people skip. A snapshot that downloaded halfway because the NFS server hiccuped produces a file with a plausible name and a plausible-ish size, and you will not find out it's unreadable until the day you need it. Validating the archive before deleting the source means a failed export leaves the local snapshot in place and the job retries.

Retention, on the NFS side

Qdrant won't prune for you and neither will NFS. A second, tiny CronJob handles it:

# keep 14 days of dailies, plus the first snapshot of each month forever
find /backups -maxdepth 1 -type d -mtime +14 \
  ! -name "$(date -u +%Y-%m)-01" -exec rm -rf {} +

Six-hourly snapshots at 14 days retention on a 3 GiB collection set works out to about 170 GiB before compression, which is why the PV above is sized at 200 GiB. Run the arithmetic for your own collection sizes before you pick a schedule, because the number gets ugly fast with hourly snapshots.

Why the separation actually matters

The word to keep in your head is blast radius. Every backup design is really a claim about which failures it survives, and the honest way to evaluate one is to name the failure and trace where the bytes are.

A Longhorn recurring snapshot job is not a substitute here, and the reason is worth understanding. Longhorn snapshots are block-level, they live inside the same replicas as the volume data, and they form a chain. They're excellent for fast rollback and they cost almost nothing to create. They are not off-box copies. If a replica's underlying disk dies, the snapshot chain on that disk dies with it. Longhorn backups (the ones that target S3 or NFS) are the off-box tier, and those are a different resource type entirely. Conflating the two is common enough that I'd call it the default misunderstanding. Longhorn Volume Health: The Gap Between 'Healthy' and Actually Working goes deeper on how the replica state and the dashboard's idea of "healthy" diverge.

There's a capacity wrinkle too. Deleting a Longhorn snapshot does not immediately return the space. The snapshot gets marked for removal and the actual coalescing happens when the purge runs against each replica, so you'll see a volume that reports 40 GiB of usage against 12 GiB of live data for a while after cleanup. Recent Longhorn versions also cap snapshots per volume (250 by default), and a retain value on a recurring job that's too generous will hit that ceiling and start failing jobs. Retention that looked reasonable when the collection was small stops being reasonable at ten times the size.

Application-level snapshots solve a problem block-level snapshots can't touch: portability. A Qdrant snapshot archive restores into any Qdrant instance, on any storage class, on any cluster, with a single API call:

curl -X POST -H "api-key: $KEY" \
  -F "snapshot=@/backups/2026-07-29/docs-1753747200.snapshot" \
  "http://qdrant.example.local:6333/collections/docs/snapshots/upload?priority=snapshot"

A Longhorn volume backup restores into a Longhorn cluster. That's a meaningfully smaller set of options on the day you're rebuilding.

Distributed mode changes the math

One caveat that catches people scaling from a single pod to a StatefulSet: in a distributed Qdrant deployment, a collection snapshot taken against one node contains only the shards that live on that node. Hitting the API through a Service round-robins your request to a random pod and gives you a partial backup that looks complete.

For multi-node setups you need to enumerate the pods and snapshot each one, addressing them by their StatefulSet DNS names rather than the Service:

for i in 0 1 2; do
  NODE="qdrant-$i.qdrant-headless.vectordb.svc.cluster.local:6333"
  curl -sf -X POST -H "$AUTH" "$NODE/collections/$C/snapshots?wait=true"
done

Restore is correspondingly per-node and needs the same shard topology on the target. If you're running distributed Qdrant as the memory layer for an agent system, that recovery procedure is worth writing down and rehearsing before you need it, which is exactly the kind of thing I end up doing in infrastructure and AI agent consulting work.

Cleaning up what's already there

If you're retrofitting this onto a running deployment, there's probably a pile of accumulated snapshots and a few orphaned PVCs. Find the snapshots first:

kubectl exec -n vectordb deploy/qdrant -- \
  sh -c 'du -sh /qdrant/snapshots && ls -1 /qdrant/snapshots | wc -l'

Then the detached volumes, which are the ones nobody remembers creating:

kubectl get pvc -A -o json | jq -r '
  .items[] | select(.status.phase == "Bound")
  | "\(.metadata.namespace)/\(.metadata.name) \(.spec.resources.requests.storage)"'

Cross-reference against pods that actually mount them. Anything bound with no consumer is either a deliberate cold spare or dead weight, and in my experience it's usually dead weight left behind by a Helm release someone uninstalled without --cascade. Snapshot the volume before you delete it, confirm nothing breaks for a week, then remove it. Space won't come back instantly for the reasons above.

What I'd tell someone starting over

Test the restore before you trust the backup. An untested backup is a hypothesis. Pick a snapshot at random once a month, upload it into a throwaway collection named restore-drill, run a search against it, compare the point count to the source, then delete it. Three minutes of work that converts a guess into a fact.

Alert on backup freshness, not on job success. A CronJob that exits 0 while writing zero bytes is a green checkmark and an empty directory. The metric that matters is the age of the newest file on the NFS share, and if it exceeds two schedule intervals, page someone.

Size the data PVC assuming snapshots will temporarily double your footprint, because they will. This is the single most common way a Qdrant deployment falls over, and the error you get back is generic enough that it sends people looking in entirely the wrong place.

NFS is the middle tier, not the last one. It's off-box and it's simple, but it's still one server in one rack. The natural next step is pushing those archives to S3-compatible object storage, either through the same job or by pointing Qdrant's snapshots_config at S3 directly if your version supports it (check the docs for your specific release, this landed relatively recently and the config shape has moved). Velero + MinIO covers the object storage side of that, and it composes cleanly with everything above.

The thing that stuck with me most from working through this: the Qdrant snapshot API is a perfectly good primitive that quietly assumes you've solved a problem it can't see. It writes a file. Where that file lands, how long it lives, and whether anything ever reads it back are decisions the API can't make for you, and they're the only decisions that determine whether you have a backup at all.