DEV Community

Cover image for We Built a Self-Healing Registry Mirror (Because Docker Hub Rate Limits Are No Fun)
Maksym Trofimenko
Maksym Trofimenko

Posted on

We Built a Self-Healing Registry Mirror (Because Docker Hub Rate Limits Are No Fun)

#kubernetes #docker #automation

If you've ever stared at ImagePullBackOff in your cluster at 2 PM on a Tuesday, you know the pain. Docker Hub rate limits hit, your pods can't pull, and suddenly your perfectly fine deployment is stuck.

We decided to fix this properly — a local registry mirror that automatically copies images from remote registries and patches deployments to use local copies. No more rate limits. No more surprise outages.

Here's how we did it.

The Setup: Zot on GKE

We went with zot — a lightweight, OCI-native registry that runs nicely as a single StatefulSet. Install it with Helm:

helm repo add zot https://zotregistry.dev/helm-charts
Enter fullscreen mode Exit fullscreen mode

Here's our values.yaml:

persistence: true
pvc:
  storage: 20Gi
mountConfig: true
configFiles:
  config.json: |
    {
      "storage": {
        "rootDirectory": "/var/lib/registry",
        "dedupe": false,
        "gc": true,
        "gcDelay": "1h",
        "gcInterval": "6h"
      },
      "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "compat": ["docker2s2"]
      },
      "log": { "level": "info" },
      "extensions": {
        "search": { "enable": true },
        "scrub": { "enable": true, "interval": "24h" }
      }
    }
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: registry-mirror.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: registry-mirror-tls
      hosts:
        - registry-mirror.example.com
Enter fullscreen mode Exit fullscreen mode 
helm install zot zot/zot -n zot --create-namespace -f values.yaml
Enter fullscreen mode Exit fullscreen mode

Two things to watch for.

The docker2s2 gotcha

Zot is OCI-first, which means it rejects Docker V2 Schema 2 manifests by default. You'll see MANIFEST_INVALID or HTTP 415 when pushing images from Docker Hub.

That "compat": ["docker2s2"] in the http section is the fix. Without it, multi-arch images like minio/minio:latest will fail every time. Took us a few rounds to find this — it's under http, not storage.

Keep it internal

Since this mirror sits inside the cluster, there's no reason to expose it to the internet. The whitelist-source-range annotation locks access to your pod CIDR. Adjust the range to match your cluster. No auth needed — pods pull freely, nobody else gets in.

The Automation: Tiny Systems Flow

A registry without automation is just a fancy disk. We built a Tiny Systems flow that runs every 5 minutes and does this:

  1. Lists all deployments (filterable by namespace and labels)
  2. Skips anything already using the local registry, or scaled to zero
  3. Reads each deployment's own imagePullSecrets for source registry auth
  4. Copies the image from the source registry to the local mirror
  5. Patches the deployment to use the local copy

The whole thing is 9 nodes, no code to deploy, no CronJob YAML to maintain.

The flow handles edge cases you'd forget about in a script:

  • Deployments without pull secrets (public images) skip the secret read entirely
  • Multi-container pods get each container mirrored individually
  • Failed copies don't block other images — errors go to a debug sink and the loop continues

The Flow 

Ticker (5min) -> Deployment List -> JS (plan) -> Split -> Router -> HAS_SECRET -> Secret Get -> Registry Copy -> Update
                                                                 -> NO_SECRET  -> Registry Copy -> Update
Enter fullscreen mode Exit fullscreen mode

The Router splits based on whether the deployment has imagePullSecrets. Private images (ghcr.io, etc.) go through Secret Get first to grab credentials. Public images (Docker Hub) go straight to copy.

Results

After starting the ticker, every deployment in our namespace got mirrored within a couple of minutes. Images from Docker Hub, ghcr.io, quay.io — all living locally now.

Next time Docker Hub has a bad day, our cluster won't even notice.

Try It

The Image Mirror solution is ready to install. Set your local registry address in the Ticker settings, click Start. That's it.

You'll need:

  • A zot registry (or any OCI registry) accessible from your cluster
  • These modules installed in your Tiny Systems workspace:
    • common-module — ticker, router, array split, debug
    • kubernetes-module — deployment list/update, secret get
    • js-module — image planning logic
    • distribution-module — registry copy

Happy mirroring.

Top comments (0)