How to Provision S3 Buckets in Kubernetes with COSI and VersityGW

#devops #opensource #tutorial #kubernetes

Kubernetes has had CSI for block and file storage for years. But if your app needs an S3 bucket, you're on your own — script some API calls, create an IAM user, write a bucket policy, inject the credentials. COSI changes that.

What is COSI?

COSI (Container Object Storage Interface) is a SIG-Storage project that adds two CRDs to Kubernetes:

BucketClaim — like a PersistentVolumeClaim, but for S3 buckets. Apply one and the driver creates the bucket on your storage backend.
BucketAccess — provisions a dedicated IAM user with a scoped bucket policy. The credentials are written to a Kubernetes Secret your app can mount.

Delete the CRs and the bucket, user, and policy get cleaned up. The spec is at v1alpha1 — functional, but expect changes.

What is VersityGW?

VersityGW is an open-source (Apache 2.0) S3-compatible gateway written in Go. Its POSIX backend takes a directory and exposes it as an S3 endpoint — no erasure coding or custom data format. If you have a filesystem (local disk, NFS, ZFS), VersityGW can serve it as S3.

This makes it a good fit for homelabs and on-prem setups where you already have storage and just need an S3 API in front of it.

Architecture

BucketClaim / BucketAccess
        │
        ▼
  COSI Controller
        │
        ▼
    COSI Sidecar
        │ gRPC (Unix Socket)
        ▼
  versitygw-cosi-driver
        │
   ┌────┴────┐
   ▼         ▼
S3 API   Admin API
   └────┬────┘
        ▼
    VersityGW
        │
        ▼
   Filesystem

The driver runs as a Deployment with two containers: the driver itself and the standard COSI sidecar. The sidecar watches Kubernetes CRs and dispatches gRPC calls to the driver over a shared Unix socket.

Prerequisites

A Kubernetes cluster (1.25+, when COSI was introduced)
Helm 3.x
A running VersityGW instance with IAM and Admin API enabled
kubectl configured

Step 1: Install the COSI Controller

The COSI controller manages the CRD lifecycle. Install it with:

kubectl create -k 'https://github.com/kubernetes-sigs/container-object-storage-interface//?ref=v0.2.2'

Verify it's running:

kubectl get pods -A | grep objectstorage

Step 2: Install VersityGW (if not already running)

If you're already running VersityGW, skip this. Otherwise, install it with its Helm chart:

helm install versitygw oci://ghcr.io/versity/versitygw/charts/versitygw \
  --set auth.accessKey=admin \
  --set auth.secretKey=admin123 \
  --set admin.enabled=true \
  --set gateway.iam.enabled=true

The chart creates a credentials Secret automatically (versitygw-versitygw-credentials).

Step 3: Install the COSI Driver

helm install versitygw-cosi-driver \
  oci://ghcr.io/isac322/charts/versitygw-cosi-driver \
  --set driver.name=versitygw.cosi.dev \
  --set versitygw.credentials.secretName=versitygw-versitygw-credentials

This deploys the driver and creates a default BucketClass and BucketAccessClass named versitygw.

Step 4: Create a Bucket

# bucket-claim.yaml
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClaim
metadata:
  name: my-bucket
  namespace: default
spec:
  bucketClassName: versitygw
  protocols:
    - s3

kubectl apply -f bucket-claim.yaml
kubectl get bucketclaim my-bucket

Step 5: Get Credentials

# bucket-access.yaml
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketAccess
metadata:
  name: my-bucket-access
  namespace: default
spec:
  bucketClaimName: my-bucket
  bucketAccessClassName: versitygw
  credentialsSecretName: my-bucket-credentials
  protocol: s3

kubectl apply -f bucket-access.yaml
kubectl get secret my-bucket-credentials -o jsonpath='{.data}' | jq 'map_values(@base64d)'

The Secret contains accessKeyID, accessSecretKey, endpoint, and region. Mount it in your app and you're connected.

Cleanup

kubectl delete -f bucket-access.yaml
kubectl delete -f bucket-claim.yaml

The driver removes the IAM user, bucket policy, and bucket automatically.

Wrapping Up

COSI is still early (v1alpha1), but if you're managing S3 buckets on Kubernetes, it's worth trying. The declarative model fits naturally into GitOps workflows — your bucket definitions live in the same repo as your app manifests, and tools like ArgoCD or Flux handle the rest.

The driver is open source (MIT):