DEV Community

Cover image for Feeding GPUs from Object Storage: A Self-Hosted S3 Pipeline for Kubernetes MLOps
Ethan Carter
Ethan Carter

Posted on

Feeding GPUs from Object Storage: A Self-Hosted S3 Pipeline for Kubernetes MLOps

#s3 #kubernetes #mlops #rustfs

Most AI training pipelines waste GPU cycles waiting on data. The fix is to serve datasets, checkpoints, and model artifacts from an S3-compatible object store that runs inside your own Kubernetes cluster. RustFS gives you that S3 tier under Apache 2.0, with no per-GB egress bill from the store and no vendor lock. This applies to any CNCF-conformant cluster, including OpenShift, EKS, GKE, or a local kind cluster.

Key Stats

Dimension RustFS
License Apache 2.0
API surface S3-compatible (HTTP REST), built in Rust
Kubernetes deployment K8s Helm Charts Available — install per the official Helm docs, credentials via values or an existing Secret
Credentials Docker quick start: rustfsadmin / rustfsadmin (rotate); the Helm chart rejects well-known default credentials
File access No POSIX/FUSE mount in the Feature & Status table; built-in FTPS and WebDAV (default build), SFTP (opt-in build feature), or third-party s3fs-fuse
Preview features S3 Tables (Iceberg REST), MinIO on-disk compatibility

What does feeding GPUs from object storage mean?

Feeding GPUs from object storage means treating your S3 bucket as the single source of truth for everything a training job touches: the raw corpus, the shuffled shards, the mid-epoch checkpoints, and the final model weights. The GPU node pulls an object over the network, decodes it in RAM, and never needs a POSIX filesystem underneath. In practice this looks like a PyTorch or TensorFlow worker calling the S3 API through boto3 or fsspec rather than opening a local path. The mental shift is from "mount a volume" to "address a key." RustFS is an Apache 2.0, S3-compatible server built in Rust, so your existing S3 tooling — aws s3, mc, the Python SDKs — works unchanged. We treat the object store as the data plane and leave POSIX concerns to a gateway or a FUSE shim.

Why not just use NFS or local NVMe for training data?

Local NVMe is fast but local: it does not follow the pod when the scheduler reschedules a failed worker, and it turns every node into a data silo you must sync by hand. NFS gives you shared, ReadWriteMany files, but random reads of millions of small shards hammer the filer, and a single appliance becomes the throughput ceiling and the single point of failure. Object storage sidesteps both: data lives behind a key, scales horizontally, and any pod in any zone can read the same object through the S3 endpoint. Object reads are network round-trips rather than local seeks, so you stage hot shards into RAM or a node-local cache instead of streaming every byte remotely. Pick the primitive that matches the workload: NVMe for scratch, object storage for the durable corpus.

Where RustFS fits: a self-hosted S3 tier

RustFS is the S3 layer you run yourself, in the same cluster as your training jobs. Its Feature & Status table lists S3 Core, Versioning, Object Lock (WORM), Lifecycle, Distributed Mode, IAM, SSE, KMS, and the K8s Helm Charts all as Available, so you get object versioning and lifecycle rules for free. It is Apache 2.0, so there is no per-GB egress fee and no contract locking you to one cloud. The honest boundary: RustFS has no POSIX or FUSE mount in its feature table — path-based access goes through its built-in FTPS and WebDAV gateways (both enabled in the default build), SFTP (an opt-in cargo feature), or a third-party FUSE shim like s3fs-fuse. S3 Tables (Iceberg REST) and MinIO on-disk compatibility are still Preview. If your pipeline is Iceberg-native and needs a production catalog today, Apache Polaris and Project Nessie are mature, Apache-2.0 catalogs you can run separately; we treat RustFS's own Iceberg support as not-yet-GA and say so plainly.

How do you deploy RustFS on Kubernetes?

RustFS ships a Helm chart, marked Available in its Feature & Status table. Both documented chart-access routes in the official Helm installation docs (docs.rustfs.com) end by setting the chart reference the install commands use — don't skip that line, or the install command gets an empty variable:

helm repo add rustfs https://charts.rustfs.com
helm repo update
export RUSTFS_CHART=rustfs/rustfs
Enter fullscreen mode Exit fullscreen mode

One thing the chart itself makes non-negotiable: it refuses to install with the well-known rustfsadmin / rustfsadmin defaults. That guard lives in the chart's templates/secret.yaml, which fails rendering with "…must be set to non-default, non-empty values…" unless you set real keys or explicitly opt in with secret.allowInsecureDefaults=true (local development only). The docs' standalone example supplies real keys through a values file and installs with:

helm upgrade --install rustfs "$RUSTFS_CHART" \
  --namespace rustfs \
  --create-namespace \
  -f standalone-values.yaml
Enter fullscreen mode Exit fullscreen mode

That standalone-values.yaml (verbatim from the docs) enables one-Pod standalone mode, sets the Secret keys, pins StorageClass and sizes, and disables the Ingress — worth keeping, because the chart's default is ingress.enabled: true with className: "nginx", and a kind or OpenShift cluster without an nginx controller would just get an idle Ingress resource:

mode:
  standalone:
    enabled: true
  distributed:
    enabled: false
secret:
  rustfs:
    access_key: "<your-access-key>"
    secret_key: "<your-secret-key>"
storageclass:
  name: standard
  dataStorageSize: 10Gi
  logStorageSize: 1Gi
ingress:
  enabled: false
Enter fullscreen mode Exit fullscreen mode

For a quick local check before you touch a real cluster, the project's Docker quick start (verbatim from the README) runs the container as a non-root user with UID/GID 10001:10001:

mkdir -p data logs
chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
Enter fullscreen mode Exit fullscreen mode

The Docker image still answers to the README's documented default rustfsadmin / rustfsadmin — rotate them before anything shared. In Kubernetes, what pods actually dial is a Service name. Per the chart's templates/service.yaml, the ClusterIP Service is {{ include "rustfs.fullname" . }}-svc: the chart's fullname plus an -svc suffix. The fullname equals the release name only when the release name already contains the chart name — helm install rustfs ... gives rustfs-svc, but helm install my-store ... gives my-store-rustfs-svc. The Service exposes the S3 API on 9000 and the Console on 9001, with a headless -headless Service only in distributed mode. For release name rustfs in namespace rustfs, the in-cluster endpoint is http://rustfs-svc.rustfs.svc:9000. One correction worth passing on: the docs' verify step reads kubectl -n rustfs port-forward svc/rustfs 9000:9000 9001:9001, but this chart creates no Service named rustfs — use svc/rustfs-svc or the port-forward fails to resolve.

How do training pods actually read from RustFS?

Pods talk to RustFS over the S3 API, not through a volume. The cleanest pattern is to inject the access and secret key from a Kubernetes Secret as environment variables and point the AWS SDK at the in-cluster endpoint — boto3's default credential chain picks up AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, so no keys land in the image or the code. A minimal boto3 pull before the epoch looks like this:

import os, boto3
s3 = boto3.client(
    "s3",
    endpoint_url="http://rustfs-svc.rustfs.svc:9000",
    aws_access_key_id=os.environ["RUSTFS_ACCESS_KEY"],
    aws_secret_access_key=os.environ["RUSTFS_SECRET_KEY"],
)
s3.download_file("datasets", "imagenet-shards/train-0001.tar", "/tmp/train-0001.tar")
Enter fullscreen mode Exit fullscreen mode

For lazy, key-addressed reads without downloading first, fsspec plus s3fs (pip install fsspec s3fs) opens remote objects directly through the same endpoint:

import s3fs
fs = s3fs.S3FileSystem(
    key=os.environ["RUSTFS_ACCESS_KEY"],
    secret=os.environ["RUSTFS_SECRET_KEY"],
    client_kwargs={"endpoint_url": "http://rustfs-svc.rustfs.svc:9000"},
)
Enter fullscreen mode Exit fullscreen mode

Because RustFS is S3-compatible, you do not rewrite your data loader — you change the endpoint_url and the credentials. The operational rule we follow: keep credentials in a Secret, never in the image or the code, and prefer versioned buckets so a bad overwrite is one versionId away from recovery.

Checkpoints, artifacts, and the model registry bucket

A single bucket rarely fits every job, so we split the data plane by purpose. One versioned bucket holds the raw and preprocessed corpus; a second holds checkpoints, written every N steps so a preempted pod resumes instead of restarting; a third holds the model registry, where each published artifact is an immutable object keyed by version. RustFS Object Lock (WORM) is Available, so the registry bucket can be locked to prevent accidental deletion of a released model. Lifecycle rules (also Available) move stale checkpoints to cheaper tiers or expire them after a retention window, which matters once you are checkpointing every job on every node. The pattern scales because object storage has no per-node ceiling — grow capacity by adding RustFS nodes, not by buying a bigger filer for the training subnet.

What about a filesystem view? Built-in gateways first, s3fs second

Before reaching for FUSE, check whether the legacy tool speaks a protocol RustFS already serves: the Feature & Status table marks FTPS and WebDAV as Available (both enabled in the default build) and SFTP as Available (an opt-in cargo feature). A tool that can read ftp:// or webdav:// paths needs no FUSE at all. When something genuinely requires a POSIX-style bucket mount, s3fs-fuse earns its place. s3fs-fuse is a FUSE filesystem backed by Amazon S3 and other S3-compatible stores, licensed GPLv2; you mount a bucket as a local directory and point the legacy tool at it:

s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o url=http://rustfs-svc.rustfs.svc:9000 -o use_path_request_style
Enter fullscreen mode Exit fullscreen mode

Its own README is blunt about the cost: random writes or appends rewrite the entire object, directory listings suffer network latency, there are no atomic renames, no hard links, and no coordination between multiple clients mounting the same bucket. So s3fs is a read-mostly or write-once shim, not a POSIX replacement. Use it for the one tool that cannot speak S3; keep the hot training path on the native S3 API where the throughput actually is.

A reference pipeline you can copy: kind + RustFS + a shard-pull Job

You can stand up the whole loop on a laptop with kind. Create the cluster, add the chart repo, and install RustFS with credentials from a values file (see the deploy section above):

kind create cluster --wait 30s
helm repo add rustfs https://charts.rustfs.com
helm repo update
export RUSTFS_CHART=rustfs/rustfs
Enter fullscreen mode Exit fullscreen mode

Then create the Secret and a Job that pulls a shard from RustFS over S3 before training. This manifest is our own minimal working shape — it proves the connectivity, not the training loop. Uncomment the GPU lines on GPU nodes (scheduling needs the NVIDIA device plugin); adapt names and images to your cluster:

apiVersion: v1
kind: Secret
metadata:
  name: rustfs-creds
  namespace: default
stringData:
  RUSTFS_ACCESS_KEY: "<your-access-key>"
  RUSTFS_SECRET_KEY: "<your-secret-key>"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: shard-pull-train
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: train
          image: pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime  # or your preferred PyTorch image
          envFrom:
            - secretRef:
                name: rustfs-creds
          # Uncomment on GPU nodes (requires the NVIDIA device plugin):
          # resources:
          #   limits:
          #     nvidia.com/gpu: 1
          # nodeSelector:
          #   nvidia.com/gpu.present: "true"
          command: ["python", "-c"]
          args:
            - |
              import os, boto3
              s3 = boto3.client(
                  "s3",
                  endpoint_url="http://rustfs-svc.rustfs.svc:9000",
                  aws_access_key_id=os.environ["RUSTFS_ACCESS_KEY"],
                  aws_secret_access_key=os.environ["RUSTFS_SECRET_KEY"])
              s3.download_file("datasets",
                              "imagenet-shards/train-0001.tar",
                              "/tmp/train-0001.tar")
              print("shard ready, training starts here")
Enter fullscreen mode Exit fullscreen mode

The pattern is identical on OpenShift, EKS, or GKE — only the ingress, SCC, and node classes differ (see the OpenShift section below). RustFS Distributed Mode is Available, so when one node's disks fill up you add nodes instead of migrating. We keep the corpus in a versioned bucket, the checkpoints in a second bucket with a lifecycle rule, and the registry in a third with Object Lock on. That is the entire data plane: one S3 tier, three buckets, and no per-GB egress bill from the store itself.

What actually changes on OpenShift

The same chart and the same rustfs-svc endpoint pattern apply — OpenShift is Red Hat's CNCF-certified Kubernetes distribution. Two OpenShift-specific pieces do change, and neither is optional in the default configuration:

  1. SecurityContextConstraints (SCC). The chart hardcodes podSecurityContext with runAsUser / runAsGroup / fsGroup of 10001 (confirmed in the chart's values.yaml). OpenShift's default restricted-v2 SCC validates UIDs against the range assigned to the namespace (Run As User Strategy: MustRunAsRange) and validates fsGroup the same way (FSGroup Strategy: MustRunAs), so hardcoded 10001 values can fall outside those ranges and be rejected at admission. You have two supported ways out: override podSecurityContext / containerSecurityContext in your values file — dropping runAsUser, runAsGroup, and fsGroup together, so every range-based strategy can apply — or bind the RustFS service account to a nonroot-class SCC. Dropping only runAsUser is not enough; a stale fsGroup: 10001 gets the pod rejected on its own. Either way, the deployment on OpenShift is a values change, not an unchanged install.
  2. Route for external access. If clients outside the cluster need the S3 API or Console, expose them with an OpenShift Route (the chart's Ingress support covers in-cluster alternatives via ingress.className). Traffic that stays inside the cluster — training pods pulling shards — needs neither a Route nor an Ingress; it goes straight to rustfs-svc.

Deeper OpenShift AI integration — workbenches reading the bucket, KServe model servers pulling weights — follows the same endpoint pattern, but we have not exercised RHOAI specifically and do not claim it here.

FAQ

Is RustFS a drop-in for S3 in my training code?

Yes for the S3 API your data loaders use. Point boto3, fsspec, or the AWS CLI at the RustFS endpoint with its access key and secret, and your existing loaders work without code changes. The official S3 compatibility matrix names the boundaries: ACL authorization is intentionally unsupported, and server-side-encrypted objects are implementation-specific (MinIO-encrypted objects are not readable by RustFS in default builds), so check the matrix before migrating encrypted or ACL-heavy buckets. Tools that require a mounted path need one of the file-access routes above, not raw RustFS.

Can I run this on OpenShift specifically?

Yes, with two OpenShift-specific adjustments, detailed in the body: the chart's hardcoded runAsUser / runAsGroup / fsGroup of 10001 must be reconciled with the namespace's SCC ranges (override all three in values or bind a nonroot-class SCC), and external access uses a Route. The S3 endpoint pattern for pods (rustfs-svc) is unchanged.

Do I need a CSI driver to mount RustFS?

No. RustFS is reached over the S3 API, so there is no native volume type and no CSI driver required. Pods call the endpoint with SDKs or mc. If a specific tool needs a filesystem view, try RustFS's built-in FTPS or WebDAV gateways first, then s3fs-fuse as a sidecar rather than looking for a RustFS CSI driver.

How do I avoid egress and lock-in?

Because RustFS runs inside your own cluster under Apache 2.0, reads and writes between pods and the store never leave your cluster network, so the store adds no per-GB egress bill and no cloud contract. One honest caveat: if the cluster itself runs on a public cloud, cross-AZ or cross-node traffic still bills at the infrastructure layer — self-hosting the store removes the storage egress line, not the whole network bill.

What is still maturing on RustFS?

S3 Tables (Iceberg REST) and MinIO on-disk compatibility are Preview, not GA. The core S3 surface, Versioning, Object Lock, Lifecycle, Distributed Mode, IAM, SSE, and the K8s Helm Charts are all Available. Plan Iceberg-heavy workloads around a separate, mature catalog until RustFS's Iceberg support graduates.

A note on how this was verified

This article is a systematic reading of official documentation and the S3 compatibility matrix, plus direct inspection of the rustfs-1.0.4 Helm chart package from charts.rustfs.com (we unpacked it and read templates/service.yaml for the Service names and values.yaml for the securityContext values). It is not a production deployment report. Every AWS/OpenShift behavior quoted above was checked against the sources listed below (first pass 2026-10-21; the chart package, the Helm docs pages, and the SCC strategies re-checked 2026-10-27). What we did not do, and you should, before relying on this in production: we did not provision an actual RustFS instance, kind cluster, or Kubernetes training Job in the writing environment (no container runtime was available), so the commands were not executed end-to-end here; we did not run a distributed, multi-node RustFS deployment; we did not run a concurrent-throughput or failure-injection benchmark; and we did not exercise OpenShift AI (RHOAI) or KServe. The Job manifest and boto3/s3fs snippets are our own minimal working shapes, not copied from a vendor document — validate them in your cluster. Keep a rollback (versioned buckets make this cheap), and confirm the RustFS feature status against the README at your install time, since Preview features graduate.

Sources

Top comments (0)