Md Asaduzzaman Miah
July 21, 2026 • 8 min read
Introduction
Rhesis's infrastructure started off relying on a ready-to-use cloud deployment solution, GCP's Cloud Run, picked because it let a small team ship fast and run cheap without spending engineering time on infrastructure, with one cloud project covering every environment. That setup made sense for our size, and outgrowing it wasn't a failure, it was expected. Environments shared a trust boundary they had outgrown, and secrets management, everything sitting in GitHub Actions secrets with a 100-secret repository cap, had run into a ceiling of its own. Migrating to Kubernetes was hardening we did ahead of scale, closing that gap before it became relevant to the kind of customer relationships we were about to take on, while we still had the runway to do it on our own timeline.
Figure 1 below lays out where we landed: the network and cluster layers, with the application services running on top.
Architecture overview diagram showing platform tooling and Rhesis application services running inside a managed Kubernetes cluster
Figure 1: Architecture Overview. The platform tooling (ArgoCD, kGateway, cert-manager, External Secrets Operator, the observability stack) and Rhesis application services (Backend, Frontend, Worker, DB) running inside the managed Kubernetes cluster, the external systems they depend on (secrets engine, DNS provider, Let's Encrypt, S3, GitHub), and how admins reach the cluster through the WireGuard VPN while users reach it through the app gateways.
Final Architecture
Locking down the network
Rhesis now runs on private Kubernetes clusters, one per environment. The Kubernetes API server has no public endpoint, it's reachable only from an authorized network, and the only network authorized is a dedicated WireGuard VPN. There's no bastion host sitting in front of the cluster network either; the VPN peers directly into it. Peers are defined declaratively, each one tied to a static IP and an explicit list of which environments it's allowed to reach. Access isn't implicit: someone who can reach dev doesn't automatically get a path to staging or production.
The same lockdown extends to what used to be casually public. Every cluster exposes two separate ingress classes: external, backed by a public load balancer, and internal, backed by a private load balancer reachable only through the WireGuard network. Anything that doesn't need to face the internet, dashboards, deployment tooling, development and staging in most cases, goes behind internal. Figure 2 below shows that split: public traffic hitting the external gateway, admin and internal traffic tunneling in through WireGuard to the internal gateway, with cert-manager and external-dns wired into both. It's the direct fix for the problem we started with. Nothing is public by default now; it has to be deliberately routed there.
Figure 2: Traffic Routing, DNS and TLS. Admins reaching the internal gateway through the WireGuard VPN, public users reaching the external gateway over HTTPS, with cert-manager issuing TLS certs and external-dns managing DNS records for the external gateway path, and ZeroSSL/Let's Encrypt handling ACME validation via Cloudflare DNS.
Secrets and access, rebuilt
Secrets no longer live in Git or CI. They sit in an external secret engine, and the External Secrets Operator syncs them into the cluster as native Kubernetes Secrets through a ClusterSecretStore. Authentication happens through workload identity, not a static cloud key sitting in a repo somewhere waiting to be rotated or leaked.
Each environment gets its own cloud identity, scoped to only the secrets that the environment needs. That's the least privilege enforced by the plumbing rather than a policy document: a compromised dev credential has no path to a production secret, because the identity behind it was never granted one. Figure 3 below traces that path: the external secrets engine on one side, the operator syncing into Kubernetes Secrets on the other, and the backend and worker consuming those secrets directly.
Figure 3: External Secrets Operator, Rhesis Application & Databases. The external secrets engine syncing into the External Secrets Operator, which creates and updates Kubernetes Secrets; and separately, the app gateway routing to the Frontend and Backend, the Frontend calling the Backend over its API, and the Backend and Worker both reading from PostgreSQL and Valkey.
Keeping the live state honest
Every deployment now goes through ArgoCD. The desired state of each cluster is declared in Git, and ArgoCD continuously reconciles the live cluster against it, with pruning and self-healing turned on, so a manual kubectl change made outside of Git gets detected and reverted automatically rather than quietly drifting from what's documented.
That has a direct audit consequence: "what's running in production" and "what's committed to the deployment repo" are effectively the same question, with a shared history to prove it. Every deploy today leaves a commit behind, with a diff, an author, and a timestamp attached automatically. The diagram below shows the full path a change takes: from the application repo, through CI and the container registry, into the deployment repo, and from there into the cluster via ArgoCD.
Figure 4: Deployment Automation. CI builds and pushes the image to the container registry while also updating the image reference in the deployment repo; the cluster then pulls the image from the registry directly, while ArgoCD syncs and reconciles the cluster against what's declared in the deployment repo.
Monitoring and debugging
We added an observability stack that didn't exist before: Prometheus for metrics and alerting, Grafana Alloy as the collector, running as a DaemonSet on every node, and Loki as the log store Alloy ships to. Grafana itself, like everything else that doesn't need to be public, sits behind the internal ingress class, so visibility into the system didn't come at the cost of exposing it. The diagram below shows how the data flows: applications feeding metrics straight to Prometheus, Alloy picking up logs and forwarding them to Loki, and both landing in Grafana as the single place to look.
Figure 5: Observability. Applications feeding metrics to Prometheus, Grafana Alloy collecting logs and forwarding them to Loki, and both landing in Grafana as the shared view.
The Cost Picture
What it cost
Four months, with the work split across three people. We brought in Pit Wegner, a freelance DevOps engineer, to architect the environment, then handled implementation ourselves. One person led coordination and supervision alongside him. A three-person effort rebuilt the company's deployment pipeline and its entire access model in a single quarter, without pausing shipping in the meantime.
It also made day-to-day maintenance harder in one way. The private clusters and VPN layer, on top of the GitOps tooling, add up to more moving parts than a single Cloud Run deploy button, and a small team feels that overhead directly: every new engineer needs VPN access configured before they can even look at a cluster, not just a cloud login.
Application troubleshooting got easier. Before, there was no real observability stack, so debugging a production issue meant piecing together logs by hand across services. Now Grafana and Loki give a shared, queryable view that didn't exist before. Deployments got more predictable too: pushing a change through ArgoCD replaced a manual, per-service Cloud Run push with something we can reason about and roll back, with a diff to check first.
Lessons Learned
The trade-off
We didn't move to Kubernetes because Cloud Run couldn't scale, it scaled fine. We moved because "staging is public and any engineer can reach production" isn't a sentence you want to still be true a year from now, and because a 100-secret ceiling with no sensitivity tiering only gets harder to live with as the team grows. The cost was real: four months, three people, and a permanent increase in operational surface area for a small team.
The return was a system where network access is scoped and declared, secrets are centrally managed with per-environment identity, and every change is logged in Git and self-healing. Debugging, unexpectedly, ended up easier than before.
Wrapping Up
The version of this migration we wanted to avoid was the one that starts with an incident report instead of a blog post. Retiring a decision because the team has outgrown it, rather than because it failed in public, is what separates planning ahead from cleaning up afterward. Worth checking whether the environment and secrets setup that made sense for your team a year or two ago still makes sense for the team you have now.
FAQ
Why did Rhesis migrate from Cloud Run to Kubernetes?
Our infrastructure had outgrown a shared cloud project with a single trust boundary across environments, and secrets management had run into GitHub's 100-secret repository cap. Migrating to Kubernetes was hardening we did ahead of scale, closing that gap before it became relevant to the kind of customer relationships we were about to take on.
How long did the migration take, and who worked on it?
Four months. We brought in Pit Wegner, a freelance DevOps engineer, to architect the environment, then handled implementation ourselves, with one person coordinating and overseeing the effort.
How are secrets managed after the migration?
Secrets live in a managed secrets store and are synced into the cluster by the External Secrets Operator through a ClusterSecretStore, authenticated via workload identity, so no static cloud keys sit in CI or Git.
What is the only way to access the Kubernetes clusters?
A dedicated WireGuard VPN. The Kubernetes API server has no public endpoint, there's no bastion host, and access is only possible through declared WireGuard peers with explicit, per-environment permissions.
Did moving to Kubernetes make troubleshooting harder?
No, it got easier. The migration added an observability stack, Prometheus, Loki, and Grafana Alloy, that didn't exist under Cloud Run, giving the team a shared, queryable view of logs and metrics for the first time.
References
WireGuard — Official site for the VPN protocol used for cluster access.
Google Kubernetes Engine private clusters — Google Cloud documentation on private GKE clusters.
External Secrets Operator — Documentation for syncing external secrets into Kubernetes.
Argo CD — Documentation for the GitOps continuous delivery tool.
Prometheus — Documentation for the metrics and alerting toolkit.
Grafana Loki — Documentation for the log aggregation system.
Grafana Alloy — Documentation for the OpenTelemetry-based collector.
Top comments (0)