We Built a Self-Healing Registry Mirror (Because Docker Hub Rate Limits Are No Fun)

Maksym Trofimenko — Mon, 30 Mar 2026 16:11:35 +0000

If you've ever stared at ImagePullBackOff in your cluster at 2 PM on a Tuesday, you know the pain. Docker Hub rate limits hit, your pods can't pull, and suddenly your perfectly fine deployment is stuck.

We decided to fix this properly — a local registry mirror that automatically copies images from remote registries and patches deployments to use local copies. No more rate limits. No more surprise outages.

Here's how we did it.

The Setup: Zot on GKE

We went with zot — a lightweight, OCI-native registry that runs nicely as a single StatefulSet. Install it with Helm:

helm repo add zot https://zotregistry.dev/helm-charts

Here's our values.yaml:

persistence: true
pvc:
  storage: 20Gi
mountConfig: true
configFiles:
  config.json: |
    {
      "storage": {
        "rootDirectory": "/var/lib/registry",
        "dedupe": false,
        "gc": true,
        "gcDelay": "1h",
        "gcInterval": "6h"
      },
      "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "compat": ["docker2s2"]
      },
      "log": { "level": "info" },
      "extensions": {
        "search": { "enable": true },
        "scrub": { "enable": true, "interval": "24h" }
      }
    }
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: registry-mirror.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: registry-mirror-tls
      hosts:
        - registry-mirror.example.com

helm install zot zot/zot -n zot --create-namespace -f values.yaml

Two things to watch for.

The docker2s2 gotcha

Zot is OCI-first, which means it rejects Docker V2 Schema 2 manifests by default. You'll see MANIFEST_INVALID or HTTP 415 when pushing images from Docker Hub.

That "compat": ["docker2s2"] in the http section is the fix. Without it, multi-arch images like minio/minio:latest will fail every time. Took us a few rounds to find this — it's under http, not storage.

Keep it internal

Since this mirror sits inside the cluster, there's no reason to expose it to the internet. The whitelist-source-range annotation locks access to your pod CIDR. Adjust the range to match your cluster. No auth needed — pods pull freely, nobody else gets in.

The Automation: Tiny Systems Flow

A registry without automation is just a fancy disk. We built a Tiny Systems flow that runs every 5 minutes and does this:

Lists all deployments (filterable by namespace and labels)
Skips anything already using the local registry, or scaled to zero
Reads each deployment's own imagePullSecrets for source registry auth
Copies the image from the source registry to the local mirror
Patches the deployment to use the local copy

The whole thing is 9 nodes, no code to deploy, no CronJob YAML to maintain.

The flow handles edge cases you'd forget about in a script:

Deployments without pull secrets (public images) skip the secret read entirely
Multi-container pods get each container mirrored individually
Failed copies don't block other images — errors go to a debug sink and the loop continues

The Flow

Ticker (5min) -> Deployment List -> JS (plan) -> Split -> Router -> HAS_SECRET -> Secret Get -> Registry Copy -> Update
                                                                 -> NO_SECRET  -> Registry Copy -> Update

The Router splits based on whether the deployment has imagePullSecrets. Private images (ghcr.io, etc.) go through Secret Get first to grab credentials. Public images (Docker Hub) go straight to copy.

Results

After starting the ticker, every deployment in our namespace got mirrored within a couple of minutes. Images from Docker Hub, ghcr.io, quay.io — all living locally now.

Next time Docker Hub has a bad day, our cluster won't even notice.

Try It

The Image Mirror solution is ready to install. Set your local registry address in the Ticker settings, click Start. That's it.

You'll need:

A zot registry (or any OCI registry) accessible from your cluster
These modules installed in your Tiny Systems workspace:
- common-module — ticker, router, array split, debug
- kubernetes-module — deployment list/update, secret get
- js-module — image planning logic
- distribution-module — registry copy

Happy mirroring.

Cloud-Native Workflow Engine for Kubernetes - What Would You Use It For?

Maksym Trofimenko — Fri, 27 Mar 2026 11:14:01 +0000

Hey everyone, I've been building an open-source workflow engine that runs natively on Kubernetes called Tiny Systems (still working on the name). Not "deployed on Kubernetes" - actually native. Flows are CRDs. Nodes are CRDs. So entire state lives in etcd. No external database.

Think n8n or Node-RED, but designed for people who already run Kubernetes and want automation that feels like part of their cluster.

Problems I'm solving right now

These are ready-to-use solutions you can clone and deploy:

Cluster Cost Saver — scales down your deployments every evening, brings them back every morning. Dead simple, saves real money. No Lambda functions, no cronjobs, no external schedulers - just a flow inside your cluster.
Pod Failure Alerts — watches pods in real time, catches CrashLoopBackOff, OOMKilled, image pull errors, and sends a Slack message before your users start complaining. Yes, Alertmanager exists, but I get my slack message next second after the crash (not like 1-5min it takes for prometheus chain).
GitHub Deploy Bot — receives a webhook after CI builds an image, updates the Kubernetes deployment, posts the result to Slack. A lightweight GitOps pipeline without ArgoCD/FluxCD overhead for simpler setups. Using it for really tiny (the title!) clusters.
Firestore to ConfigMap Sync - listens to a Firestore collection and syncs changes to a Kubernetes ConfigMap in real time. Useful for feature flags. Got idea from app managements, thought why not to use that for K8s.
Service Status Page — a self-hosted status page that pings your endpoints and serves a page. No Statuspage.io subscription, no cron jobs, just a flow.

What workflow engine does today

You build flows visually - drag nodes, connect them, configure, deploy. Each component module runs as its own pod. Nodes in the same module talk over Go channels, across modules — gRPC. Everything stays inside your cluster.

There's also an AI assistant that can build flows from a prompt, but that's a story for another post.

Where I'm headed next

These are the solutions I'm planning to build:

CronJob failure alerter — Kubernetes CronJobs fail silently. This would watch them and alert on failure via Slack or PagerDuty.
TLS certificate expiry watcher — scan ingresses across namespaces, alert before certs expire (sometimes I miss emails from acme)
Slack command bot — slash commands that trigger Kubernetes actions: restart a pod, get logs, scale a deployment. ChatOps without writing a bot from scratch.
Webhook relay and transformer — receive webhooks from external services, reshape the payload, forward to internal services. A lightweight alternative to writing one-off HTTP handlers.

Why I'm writing this

I've been building in isolation for too long. The engine works. The solutions work. But I keep guessing what people actually need instead of asking.

So - if you run Kubernetes at work:

Which of the solutions above would you actually use?
What's a recurring ops task you've been meaning to automate but haven't?
What's keeping you on n8n / Airflow / bash scripts?

Thanks!

DEV Community: Maksym Trofimenko