DEV Community

Oleksandr Kuryzhev
Oleksandr Kuryzhev

Posted on Originally published at kuryzhev.cloud

Postgres Backup CronJob on Kubernetes: Setup and Restore Testing

#devops #kubernetes

Originally published on kuryzhev.cloud

The scenario

A Postgres backup CronJob is often the first line of defense for teams running self-managed databases inside Kubernetes. The usual story goes like this: someone runs pg_dump manually after a migration, promises to "automate that later," and then forgets. Weeks pass. Nobody notices the backups stopped until a restore is actually needed — and by then it's too late to fix retroactively.

This tutorial covers a self-managed Postgres instance running as a Deployment or StatefulSet inside a cluster, backed up on a schedule via pg_dump to object storage (S3, GCS, or a MinIO-compatible endpoint). This is not about operator-managed clusters with built-in backup CRDs, and it's not a substitute for continuous WAL archiving in a high-availability setup.

A Kubernetes CronJob is a reasonable fit here because it's transparent — you can read the YAML and know exactly what runs, when, and where the output goes. Heavier tools like Velero, pgBackRest, or cloud-managed snapshot services solve a broader set of problems (point-in-time recovery, cluster-wide backups, incremental WAL shipping), but they add operational surface area that a small-to-mid-size database may not need yet. If nightly full dumps comfortably finish inside your maintenance window, a CronJob is often the pragmatic starting point — what that means in gigabytes varies with schema complexity, disk and network throughput, and how long that window actually is, so measure it on your own hardware rather than assume a number. Official reference: the Kubernetes CronJob documentation.

Prerequisites

Before writing any YAML, confirm the following pieces are in place:

  • Working kubectl access scoped to the namespace where Postgres runs, along with the Service name and connection details (host, port, database name).
  • An object storage bucket with write access — S3, GCS, or a self-hosted MinIO endpoint — and a set of credentials scoped only to that bucket.
  • A container image that includes both pg_dump and an upload client. Neither the official postgres image nor its -alpine variant ships the AWS CLI, rclone, or mc — build a small custom image on top of the Debian-based postgres tag (not alpine, so /bin/bash is available) with one of those tools installed, or run the dump and upload as two separate containers in the same Pod sharing an emptyDir volume.
  • A Kubernetes version where the CronJob's timeZone field is available — it's been GA since 1.27, though that specific minor is now past its upstream support window, so target whichever currently supported minor your provider offers. Verify with kubectl version.

Watch out: a common mistake is pinning the backup image to postgres:latest. Pin the client's major version to match the server's instead. Newer pg_dump versions can dump from older PostgreSQL servers, but an older pg_dump run against a newer server will refuse to connect — the rule is client version at or above server version, not the other way around.

Step 1: Store credentials as Secrets, not env literals

Authentication is the part most likely to go wrong first, so get it right before touching the Job spec. Create one Secret for the database connection and a separate one for object storage credentials — separating them keeps blast radius smaller if one leaks.

# Database connection secret
kubectl create secret generic pg-backup-db-creds \
  --namespace data \
  --from-literal=PGHOST=postgres.data.svc.cluster.local \
  --from-literal=PGUSER=backup_reader \
  --from-literal=PGPASSWORD='use-a-strong-password-here' \
  --from-literal=PGDATABASE=app

# Object storage credentials, kept separate from DB creds
kubectl create secret generic pg-backup-s3-creds \
  --namespace data \
  --from-literal=AWS_ACCESS_KEY_ID=AKIA... \
  --from-literal=AWS_SECRET_ACCESS_KEY=${SECRET} \
  --from-literal=AWS_DEFAULT_REGION=us-east-1 \
  --from-literal=S3_ENDPOINT_URL=https://s3.amazonaws.com

Reference these Secrets in the Pod spec via envFrom or secretKeyRef — never as command-line arguments. Anything passed as an argument shows up in kubectl describe pod output and in the process list visible to ps inside the container, which defeats the purpose of using a Secret at all. If you're targeting MinIO or a GCS S3-compatible endpoint instead of AWS, set S3_ENDPOINT_URL to that endpoint — the AWS default won't reach them.

The Pod consuming these Secrets via envFrom needs no RBAC grant to do it — the kubelet retrieves Secret data referenced in the Pod spec using its own node identity, not the Pod's ServiceAccount permissions. RBAC controls a different thing entirely: which humans or CI pipelines can call kubectl get secret or read Secret objects through the API directly. Scope a Role to the data namespace and these two Secret names, and bind it only to whoever legitimately needs to view or rotate them.

Step 2: Write the CronJob manifest

Building the custom image is simple — pin the Postgres major version and add the AWS CLI:

FROM postgres:16

RUN apt-get update && apt-get install -y --no-install-recommends awscli \

    &&


kuryzhev.cloud



Related

Top comments (0)