alexey.zh

Posted on Jun 7

High-Speed File Transfer to and from Kubernetes PVCs

#kubernetes #devops #backup

When working with Kubernetes, Persistent Volume Claims (PVCs) are the backbone of many stateful workloads.
However, transferring data to and from a PVC is surprisingly hard to do efficiently.

Sure, there are plenty of tools that look like they can help - tools that mount PVCs locally, migrate data between PVCs,
or provide backup/restore functionality.

But when I needed a fast and reliable way to copy large files into and out of PVCs, especially for PostgreSQL WAL archiving workflows, I ran into a big roadblock:

kubectl cp became painfully slow - I gave up waiting on a 100 GiB copy
kubectl exec requires shell tools inside your container

And CSI-backed PVCs (like Ceph RBD) don't easily expose the underlying storage for direct access.

That’s why I built kubectl-syncpod.

What is kubectl-syncpod?

It’s a CLI tool that enables high-speed file transfers between your local machine and Kubernetes PVCs - even if the workload uses minimal container images (distroless, scratch) or CSI-backed block storage.

Unlike kubectl cp, it leverages:

✅ A temporary helper Pod that mounts the PVC
✅ An ephemeral SSHD server for fast and secure file transfer
✅ A parallel SFTP client that transfers files concurrently
✅ Automatic cleanup (no leftover Pods or Services)

About

kubectl cp and kubectl exec degrade in performance with a large set of files (~100 GiB).
They also require additional tools (tar, sh) and fail with distroless/scratch images.
Most importantly, they do not support concurrent reading/writing.

Features

Upload local files/directories to a pod volume
Download the pod volume files to the local
Safe overwrite protection
Auto-rename remote directories
Concurrent file transfer with worker pool
Preserves directory structure
Optional chown of uploaded files inside the pod
Fully based on SFTP + Kubernetes Exec API

Typical Use Cases

✅ Download backup from PVC for verification/restore
✅ Sync files between PVCs and local
✅ Testing PVC mount behavior
✅ CI/CD pipelines for volume data prep

How It Works

Spins up a helper pod mounting the PVC
Runs SSHD server with ephemeral key
Local SFTP client transfers files
Auto cleans up pod/service

Installation

Homebrew

brew tap hashmap-kz/homebrew-tap
brew install kubectl-syncpod

Manual

Download the latest release and place it in PATH.

Installation script for UNIX-based OS:

(
set -euo pipefail

OS="$(uname | tr '[:upper:]' '[:lower:]')"
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')"
TAG="$(curl -s https://api.github.com/repos/hashmap-kz/kubectl-syncpod/releases/latest | jq -r .tag_name)"

curl -L "https://github.com/hashmap-kz/kubectl-syncpod/releases/download/${TAG}/kubectl-syncpod_${TAG}_${OS}_${ARCH}.tar.gz" |
tar -xzf - -C /usr/local/bin && \
chmod +x /usr/local/bin/kubectl-syncpod
)

Example Usage

Upload with safe rename and chown

kubectl-syncpod upload \
  --namespace my-db \
  --pvc postgres-data \
  --mount-path=/var/lib/postgresql/data \
  --src=backups \
  --dst=pgdata-new \
  --allow-overwrite \
  --owner="999:999"

Download the remote directory

kubectl-syncpod download \
  --namespace my-db \
  --pvc postgres-data \
  --mount-path=/var/lib/postgresql/data \
  --src=pgdata-new \
  --dst=backups-copy

Comparison Table

Feature	`kubectl cp`	`kubectl exec`	`kubectl-syncpod` (SFTP mode)
Uses sidecar or helper pod	❌	❌	✅
Works with PVCs	⚠️ Only if mounted in container	⚠️ Manual path required	✅ Helper pod mounts PVC
Requires tools in container (`tar`, `sh`)	✅	✅	❌ (uses `sshd` in helper pod)
Supports `readOnlyRootFilesystem` pods	❌	❌	✅
Works on `distroless`/`scratch` images	❌	❌	✅
Affects main application container	✅	✅	❌
Requires container to run as root	Often yes	Often yes	❌ or configurable via helper pod spec
Safe for production workloads	⚠️ Risky	⚠️ Risky	✅ (safe for read)
Auto-cleans after sync	❌	❌	✅
Supports concurrent transfers	❌	❌	✅ (parallel SFTP workers)
Performance on large file trees	🐢 Slow	🐢 Slow	🚀 Fast (streaming + concurrency)

Conclusion

kubectl-syncpod fills a small but important gap in the Kubernetes toolchain:

👉 Efficient and fast file transfer to/from PVCs
👉 No need to modify your main containers
👉 Works great with CSI-backed volumes
👉 Production-safe and automated

The project is open source and evolving:

👉 https://github.com/hashmap-kz/kubectl-syncpod

If you find it useful - contributions, feedback, and GitHub ⭐️ stars are very welcome!

DEV Community