DEV Community: 3deep5me

Giving Kubernetes ServiceAccounts an identity in Forgejo with oauth2-proxy

3deep5me — Sun, 25 Oct 2026 11:01:11 +0000

In my little agent platform every coding agent is a Kubernetes Job: it gets a repository, a task and a deadline, works, pushes a branch and opens a pull request. Which means every one of those Jobs has to authenticate against Forgejo, our Git host - and my first version of that was exactly the thing you'd expect: a Personal Access Token per agent, sitting in a SealedSecret next to the agent definition.

It worked. I still didn't like it, and the reasons were never "it's a secret in Git", because it wasn't a secret in Git - it was sealed. The reasons were:

The credential outlives the work. An agent run lasts two hours. The token lasts until somebody remembers to revoke it.
It is a shared identity, not a per-run one. Anybody who can create a Job can mount the SealedSecret. That includes a fellow agent, and it means "who pushed this?" has exactly one answer for the whole platform.
It is a value somewhere. In a pod spec, in env, in a crash dump, in a diff of a debug session. A credential that never exists as a value cannot leak.

So instead of managing a token per agent, I made the agents log in as themselves. Kubernetes already knows who a pod is - a ServiceAccount - and it will happily mint a signed token for it. That token is a JWT with a sub claim. Forgejo accepts an HTTP header as an authenticated user if you ask it to. And oauth2-proxy is a small piece of software that verifies JWTs.

All three pieces exist. The interesting part is the glue - and the diagram below is that glue: an agent pod whose only credential is a projected token, a gate with two doors, and a Forgejo that nothing but the gate may reach.

The pods: an identity, not a secret

The pod gets one volume. That's the entire credential configuration of an agent:

volumes:
  - name: forgejo-identity
    projected:
      sources:
        - serviceAccountToken:
            path: token              # /var/run/secrets/forgejo/token
            audience: forgejo        # what makes it acceptable to the gate
            expirationSeconds: 86400 # longer than any run may be, and no longer

Two things matter here.

audience is the entire point. A normal ServiceAccount token is minted for the API server. A token with audience: forgejo is useless against the API server and only usable by whoever accepts that audience - which is the thing in front of Forgejo. It is a scoping mechanism, and it costs one line.

projected is the other half: the kubelet writes this file and rewrites it in place while the token approaches expiry. So nothing in the container may freeze its value. The entrypoint refuses to start without it:

if [ ! -s "${PI_FORGEJO_SA_TOKEN_FILE:-}" ]; then
  log "fatal: no projected ServiceAccount token - refusing to run unauthenticated"
  exit 1
fi

I'd rather have a run that does not happen than a run that silently talks to Forgejo unauthenticated.

The pod doesn't write the token anywhere either - git is authenticated by a wrapper that is put on PATH and reads the file on every call:

REAL_GIT=$(command -v git)
cat > /agent-bin/git <<WRAPPER
#!/bin/sh
exec "$REAL_GIT" -c "http.${PI_FORGEJO_URL}/.extraHeader=Authorization: Bearer \$(cat "$PI_FORGEJO_SA_TOKEN_FILE")" "\$@"
WRAPPER

Because the header is scoped to our Forgejo URL, the token is never offered to any other host; because it is read per call, a rotation in the middle of a two-hour run is invisible. Nothing lands in .git/config and no credential helper is involved - which also means a git command that fails cannot print a credential in its error message.

The cluster as an identity provider

This is the part that surprised me the most: the Kubernetes API server is a working OIDC issuer, and you can check the claims yourself without installing anything:

kubectl create token hermes-pke --audience=forgejo --duration=10m \
  | cut -d. -f2 | base64 -d 2>/dev/null; echo

You get iss (the API server), aud: ["forgejo"], an expiry, and

"sub": "system:serviceaccount:hermes-pke:hermes-pke"

That subject is the identity: namespace and ServiceAccount name, unforgeable, signed with the cluster's own keys, expiring on its own.

The catch is that the API server is a verification-only issuer. Its discovery document advertises no authorization and no token endpoint, because there is nothing to authorize - you don't log into Kubernetes, you present a token Kubernetes gave you. Every OIDC client library that insists on doing a full discovery will therefore fail against it, and you have to tell it to skip discovery and fetch the signing keys directly.

oauth2-proxy as the verifier

oauth2-proxy normally does "log in with GitHub/Google" for a web app. It also understands bearer tokens, and that mode is what we use: it takes the Authorization: Bearer <token> header, checks signature, issuer, audience and expiry against the API server's own keys, and forwards the verified username to its upstream.

args:
  - --provider=oidc
  - --oidc-issuer-url=https://kubernetes.default.svc.cluster.local
  - --skip-oidc-discovery=true
  - --oidc-jwks-url=https://kubernetes.default.svc.cluster.local/openid/v1/jwks
  - --provider-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  - --client-id=forgejo            # = the audience the token must carry
  - --user-id-claim=sub            # a ServiceAccount has no e-mail address
  - --email-domain=*
  - --skip-jwt-bearer-tokens=true  # accept the bearer token, no browser session
  - --bearer-token-login-fallback=false  # invalid token => 403, not a redirect
  - --pass-user-headers=true       # hand the subject on as X-Forwarded-User
  - --upstream=http://127.0.0.1:8081/
  - --http-address=0.0.0.0:4180

Four of those lines are the ones I'd keep if I could only keep four: --client-id (a token for another audience is rejected), --user-id-claim=sub (there is no e-mail to claim), --skip-jwt-bearer-tokens (this is what makes a Kubernetes token a login at all) and --bearer-token-login-fallback=false - with this issuer there is no interactive flow to redirect to, so an invalid token must be a hard 403.

Why oauth2-proxy and not Forgejo itself verifying the token? Because Forgejo, like most applications, does not grow custom token verification for my homelab. It has one standard mechanism for "something in front of me authenticated this request", and I'd rather use the standard mechanism with a dedicated verifier than patch the application. Which brings us to the interesting part.

Forgejo as the relying party

Forgejo supports reverse-proxy authentication: if a request arrives with a certain header, that header value is the username. Turned on in app.ini:

[service]
ENABLE_REVERSE_PROXY_AUTHENTICATION = true      # the web UI
ENABLE_REVERSE_PROXY_AUTHENTICATION_API = true  # and the API - agents use this
ENABLE_REVERSE_PROXY_AUTO_REGISTRATION = false  # never invent an account
ENABLE_REVERSE_PROXY_EMAIL = false
ENABLE_REVERSE_PROXY_FULL_NAME = false

[security]
REVERSE_PROXY_AUTHENTICATION_USER = X-Forwarded-User

Auto-registration off is deliberate: an account that the gate can present must already exist, with its rights declared in the repository that manages them. An unknown subject stays an unauthenticated request.

Now the part I measured rather than assumed, and the reason this post is worth writing: in Forgejo 15.0.10 there is no trusted-proxy check for this method. The reverse-proxy login code takes the header on faith. REVERSE_PROXY_TRUSTED_PROXIES exists, but in this version it only influences the source address used for logging - it does not gate the login.

Which means the header is a login credential, and reachability is part of the authentication: whoever can open a TCP connection to Forgejo's HTTP port can be whoever they like. Two consequences:

Forgejo's Service is a ClusterIP, and a Cilium network policy allows exactly one pod to speak to it - the gate. Re-opening that port for anything else is a hole, so the "extra allowed CIDRs" list in my values file is empty and documented as such.
My own gate must not be reachable in a way that lets someone skip the verifier.

Hence the gate is a pod with two doors:

:8080   the classic door   - browsers, the CI runner, Forgejo tokens
                            BLANKS every identity header a client sends
:8081   the identity door  - only this pod's own oauth2-proxy may reach it
                            (enforced by the network policy)

Both doors proxy to the same Forgejo. The classic door deletes the headers, always:

proxy_set_header X-Forwarded-User "";
proxy_set_header X-WEBAUTH-USER "";
proxy_set_header X-Auth-Request-User "";
proxy_set_header X-Forwarded-Email "";

Not "overwrites if trusted" - blanks. A client sending those headers is not lying more convincingly, it is simply ignored. Without this line, the whole design collapses back into "become anyone by sending a header".

The identity door is the one oauth2-proxy forwards to, and it strips the token on the way through, because the token has done its job by then:

proxy_set_header X-Forwarded-User $identity_user;
proxy_set_header Authorization "";

One header, many subjects

Now the cosmetic problem: the subject arriving from Kubernetes is system:serviceaccount:agent-reviewer:reviewer. Forgejo usernames cannot contain colons, and I don't want a name pattern to create accounts - an account must exist, and it must be the right account.

So the identity door maps the subject to a login, with a default that fails closed:

map $http_x_forwarded_user $identity_user {
  default "";
  "system:serviceaccount:agent-tofu:tofu"        "agent-tofu";
  "system:serviceaccount:hermes-pke:hermes-pke"  "agent-hermes-pke";
  "system:serviceaccount:agent-reviewer:reviewer" "agent-reviewer";
}

Anything not listed - including an empty header, including something that merely looks like a ServiceAccount - becomes an empty string, i.e. an unauthenticated request. No account is ever invented from a name pattern.

Adding an agent is therefore two reviewed lines in two repositories: one line in this table (which Kubernetes subject may act as which Forgejo account) and one row in the access repository (what that account may do in which repository). Revoking an agent is deleting the line. That asymmetry - the identity table is small, explicit and reviewable - is what I actually wanted from all of this, much more than "no secret".

What bit me

Because a post that only shows the working end state is useless, here are the failures, with the errors:

cookie_secret must be 16, 24 or 32 bytes, but is 44. Even in bearer-token mode oauth2-proxy validates the cookie config. My SealedSecret held a base64 string (44 chars) instead of the 32 bytes it decodes to.
502 from the gate. oauth2-proxy's upstream pointed at the Forgejo pod's port (:3000) instead of the Service (:80). Use the Service - it is what a chart should depend on, and it survives rescheduling.
host not found in upstream, but only on the node where the new pod landed. The network policy's DNS egress rule didn't match: toEndpoints.matchLabels matches inside the policy's own namespace, so k8s-app: kube-dns needs Cilium's own io.kubernetes.pod.namespace: kube-system label next to it.
nginx died silently in a read-only root filesystem. Every temp path it would create under /var/cache/nginx has to be redirected onto an emptyDir mounted at /tmp.
The web UI's live updates died at the gate until I forwarded Upgrade/Connection for websockets - and client_max_body_size 0, because a git push is not a small request and must not be buffered onto a small disk.
My first cut had the identity door open to the whole cluster. Anyone could have sent the header straight to :8081 and skipped oauth2-proxy entirely. The gate's own network policy now allows that port from its own pod only - which is the same rule as before, just applied to the other door.

Why not the obvious alternatives

A token per agent (PAT). Works, and it is how I started. Long-lived, shared by whoever can create a Job, present as a value, and revocation is a human task. The Git identity of an agent should not be something a human has to remember to take away.
Deploy keys. Nice for a push-only robot, but they are per repository and they are not a user: no attribution, no pull-request authorship, no reviews. With n agents and m repositories you also get n×m keys.
Username + password for the agent account. The same as a PAT, with a worse rotation story and a real login session.
Trusting the wire. That's the trap above, not an alternative.

What it costs, what it buys

Costs: one additional hop in front of Git (the gate), an identity table to maintain, and the discipline that only the gate may reach the instance. Both gates are ordinary Kubernetes objects, so all of it is in Git and reconciled.

Buys: no credential exists as a value anywhere - not in Git, not in a pod, not in a backup; each agent run is itself; a runaway agent is revoked by deleting one line; every commit, branch and pull request is attributable to the agent that made it; and the identity of a Job is the same identity the cluster already uses for its RBAC, which means there is exactly one story about "who is this" in the platform instead of two.

Next on my list: an agent that starts the runs of another agent (a reviewer that reviews every pull request), and what that costs in admission control. That's a post of its own. 🙂

Tested against Forgejo 15.0.10 (chart 17.3.1), a Cilium-based cluster and oauth2-proxy in bearer-token mode. The reverse-proxy behaviour without a trusted-proxy check is a measured fact about this version - if you rely on it, measure it on yours.

Thanks for reading! If you're doing something similar - or you know a Forgejo version that gates the reverse-proxy header properly - I'd love to hear it in the comments.

This post was completely written by LLM also the architecture and wiring behind this. It's only validated by a human if it works. (It works!)
But tbh I don't know if it's secure - so before you use it in production you should verify it by yourself - because rbac is nothing to play with.

From Zero to Scale: Proxmox Kubernetes Engine

3deep5me — Fri, 02 May 2025 13:53:31 +0000

If you are looking for a enterprise solution for Kubernetes on Proxmox check out my available subscriptions and support - which evolved out of the project ❤️.

https://www.proxmox-kubernetes-engine.eu/

Today, we'll take a beginner-friendly look at housing Kubernetes in Proxmox. But instead of the traditional SSH/Ansible approach, we'll explore a method akin to what you'd find with AWS, Azure, or GCP. This means we're talking about scaling from tens to hundreds of Kubernetes clusters in minutes, with automated, reproducible cluster creation and upgrades.

Does that sound like it requires heavy modifications to your Proxmox hosts or datacenter? I can reassure you: I dislike straying far from default settings, so you won't need to modify your Proxmox installation in any way. It's simply a virtual machine, allowing you to add and remove it like a plugin. (More on the architecture later.)

Why Do You Need This?

I don't need this personally — it's just a bit of fun to replicate big cloud provider functionality in my tiny homelab!

However, if your company lacks a scalable Kubernetes platform, you might find it tough to keep up in today's service-oriented world. With major cloud providers dominating, efficiently managing a private cloud is more crucial than ever, and Kubernetes is one of the most popular cloud tools. So, how can you compete? The answer lies in two of my favorite open-source projects: Cluster-API and Proxmox.

Proxmox

Proxmox, developed in Vienna, is widely known in the open-source and home-lab communities. It gained a huge boost in small to medium-scale private clouds after VMware's new pricing model alienated its customers. It's a simple yet powerful open-source hypervisor based on KVM, and it's been the core of my home lab for nearly a decade (I've been using it since PVE 4).

Cluster-API

While Cluster API might be less known in the home-lab community, it's highly valued in enterprises and by Kubernetes administrators and enthusiasts. So, what exactly is Cluster API?

Cluster API is a project under the Cloud Native Computing Foundation (CNCF), strongly supported by the Kubernetes community and various vendors, including VMware, Apple, and NVIDIA. This project offers a unified way to create and manage Kubernetes clusters across different "providers," such as Proxmox or VMware. For instance, VMware heavily leverages Cluster API in its commercial product, Tanzu.

Cluster API currently boasts over 30 infrastructure providers, with Proxmox being just one of them. In short, Cluster API provides a unified API and method for creating production-ready Kubernetes clusters across numerous providers. It has become, at least for me, the de facto standard for multi-cloud and on-premises Kubernetes deployments.

Architecture

Let me introduce you to the architecture we will use for our setup. Of course, we need at least one Proxmox host to rely on; I'll be using the newest version. On top of Proxmox, we will have our main component: the Management VM. The Management VM will house Cluster API.

The Management VM is then responsible for the Kubernetes clusters we want to create. For example, the Management VM automatically coordinates the creation and updates of Kubernetes clusters.

The Management VM achieves this by instructing Proxmox to automatically create and remove specially configured VMs on your Proxmox infrastructure. The process in detail would look like this:

We will give our Management VM the order to create, for instance, a 12-node Kubernetes cluster. The Management VM, or more precisely, Cluster API inside the Management VM, will communicate with Proxmox. Cluster API will then instruct Proxmox to provision 12 VMs from a special Kubernetes VM template. After the successful creation of the VMs, Cluster API configures the cluster. Additionally, Cluster API will monitor the VMs and can replace them automatically upon failure.

You can see that this setup is far more capable than statically creating clusters with tools like Terraform and Ansible. This is why setups like this are often referred to as a Kubernetes Platform - because you can order clusters just like you order a pizza from Lieferando.

In my opinion, it's also a less complicated way to set up multi-node Kubernetes clusters on Proxmox. We simply need to install Cluster API on our Management VM and have a Kubernetes VM template ready. No complex Ansible Playbooks or Terraform-states are required.

Our Path to a Scalable Kubernetes Platform on Proxmox

In this blog post, we'll follow these steps to achieve our goal of creating a scalable Kubernetes platform on Proxmox:

Create the Management VM as our pivotal point.
Create a Kubernetes VM Template for our Kubernetes Clusters.
Initialize Cluster API & Configure the "Caprox-Kubernetes-Engine."
Create our first Workload Cluster.

In my example, the Management VM and Kubernetes clusters will all reside on a single Proxmox host within one network that utilizes DHCP. DHCP is a requirement.

The Guide still works - but it is a bit outdated. I recommend to check out the GitHub Project which evolved out of this blogpost. Here is the updated Tutorial: Proxmox-Kubernetes-Engine/Quick-Start

Create the Management VM with k3sup

Disclaimer: All the steps shown here are for home-lab purposes only and should not be used for a production or even development environment. This guide is merely a starting point. If you're looking for a more production-ready setup, feel free to reach out to me.

Create a New VM for Our Management VM

We need a simple Linux VM for our Management VM with 30 GiB of disk space and at least 4 GiB of RAM.

For Proxmox beginners, I recommend this great guide. You'll also need an ISO image, which you can download here from the official Ubuntu website.

Of course, you can also use other methods such as Terraform, cloud-init or cloning an existing Linux VM or template.

Once you've logged into your new VM, we can proceed.

Install Kubernetes (K3s) on Our New VM

Cluster API is Kubernetes-exclusive, meaning it itself relies on Kubernetes to operate. Because of this, we need Kubernetes on our Management VM. In this guide, we'll use K3s for that. K3s is a Kubernetes distribution from Rancher that is now fully community-driven and under the CNCF. It's a popular Kubernetes distro for small to mid-sized clusters.

To install K3s, we'll use k3sup - a really simple CLI tool that allows you to create a K3s Kubernetes cluster within seconds on any Linux VM. The following commands will download k3sup and then create a simple cluster.

# install k3s-Kubernetes with k3sup
curl -sLS https://get.k3sup.dev | sh
sudo cp k3sup /usr/local/bin/k3sup
k3sup install --local --k3s-version v1.33.1+k3s1

You can test if the cluster is working with sudo k3s kubectl get nodes. It should show something like this:

sudo k3s kubectl get nodes
NAME        STATUS   ROLES                  AGE     VERSION
localhost   Ready    control-plane,master   6m30s   v1.33.1+k3s1

Congratulations! You've successfully set up a simple single-node Kubernetes cluster. This cluster will serve as our Cluster API Management VM.

Creating Your Kubernetes VM Template

Every Kubernetes cluster typically starts with a machine template, most often a VM template. This template is where all Kubernetes components are configured and downloaded. For Cluster API, we need VM templates that already include essential Kubernetes packages like the API server and the container runtime.

We'll achieve this with the help of the excellent Kubernetes Image Builder project. While it sounds fancy, it's essentially Packer and Ansible cleverly glued together to automate the image creation process. But as promised, you don't need to touch Ansible or Packer at all.

Create an API Token and a Secret with the Values

To get started, we need to create API access to our Proxmox Datacenter. You might wonder why. The image builder constructs the image directly on your Proxmox node, so it requires access to start a VM and later convert it into a template.

To begin, open a shell on your Proxmox node. You can do this by clicking on your Proxmox node in the interface and selecting "Shell"; a command-line interface will then open directly in your browser. Once the shell is open, execute the following commands:

pveum user add caprox@pve
pveum aclmod / -user caprox@pve -role PVEAdmin
pveum user token add caprox@pve capi -privsep 0

After that you should get all relevant information.

root@node01:~# pveum user token add caprox@pve capi -privsep 0
┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ caprox@pve!capi                      │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ 6e59df15-a2c9-4dc5-b293-367772950c68 │
└──────────────┴──────────────────────────────────────┘

We will also use our Management VM to coordinate the VM Template Builds. For that we will use the Kubernetes Job Resource and a Kubernetes Secret to pass the API-Key and some configuration.

First let us create the Secret with values we got from above.
You only need to change the first four Proxmox-Vars.

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: proxmox-image-build-config
  namespace: proxmox-build-infrastructure-system
type: Opaque
stringData:
  PROXMOX_URL: "https://your-proxmox-adress:8006/api2/json"
  PROXMOX_USERNAME: "full-tokenid"
  PROXMOX_TOKEN: "value"
  PROXMOX_NODE: "your proxmox node name"
  PROXMOX_ISO_POOL: "local" #this should be fine for the most users
  PROXMOX_BRIDGE: "vmbr0" #this should be fine for the most users
  PROXMOX_STORAGE_POOL: "local" #this should be fine for the most users
  PACKER_FLAGS: "--var memory=4096 --var 'kubernetes_rpm_version=1.33.1' --var 'kubernetes_semver=v1.33.1' --var 'kubernetes_series=v1.33' --var 'kubernetes_deb_version=1.33.1-1.1'"

Configure needed values and save the File.

Now we need also a Job which creates the Image. A Job is a Kubernetes Pod which is only executed once with a finite life - until the Job successfully complete. It uses Image-Builder Docker-Image with the required configuration to build Kubernetes Proxmox VM-Templates.

# job.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: proxmox-template-builder
  namespace: proxmox-build-infrastructure-system
spec:
  schedule: "* * * * *"
  suspend: true 
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          restartPolicy: OnFailure
          containers:
          - name: image-builder
            image: registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.45
            envFrom:
            - secretRef:
                name: proxmox-image-build-config
            args:
            - build-proxmox-ubuntu-2404
            env:
            # help for slow nodes
              - name: ANSIBLE_TIMEOUT
                value: "60"

Just copy and save the file.

Start the Build-Job and create a Kubernetes Image

Now you should have two file one Secret and one (Cron)-Job, the Cron-Job has the benefit of easier executing if we need new image or even do scheduled automatic builds. Now its build time!

# create a namespace for the build
sudo k3s kubectl create namespace proxmox-build-infrastructure-system
# apply secret & job
sudo k3s kubectl apply -f secret.yaml
sudo k3s kubectl apply -f job.yaml
# start the build
sudo k3s kubectl create job build-image --from cj/proxmox-template-builder -n proxmox-build-infrastructure-system

To Monitor the current state we can look at the Logs of the Builder-Pod. For that we need the name and the kubectl logs command.

sudo k3s kubectl get pods -n proxmox-build-infrastructure-system
NAME                READY   STATUS    RESTARTS   AGE
build-image-tdv78   1/1     Running   0          97s
sudo k3s kubectl logs -f -n proxmox-build-infrastructure-system build-image-tdv78
proxmox-iso.ubuntu-2204: output will be in this color.
==> proxmox-iso.ubuntu-2204: Retrieving ISO
...

You should see a new VM pop-up in your Proxmox UI. This is often the trickiest part of the guide. Sometimes it can take a long time, or the job might even require multiple attempts. Kubernetes will automatically restart the job if the build fails, so you can just relax and let it do its thing. In my experience, it occasionally took over 30 minutes to create the VM template.

Please ensure that you have DHCP configured in your network. If you encounter any issues, feel free to leave a comment below; I'm happy to help! Sometimes a Proxmox reboot also helps.

If you're lucky, you should see a new template in the UI after a few minutes.

And thats it - you completed the "most complicated" part!

Initialize Cluster-API

In the next step, we'll configure Cluster-API to our needs. I prefer to avoid keeping files, especially complex ones, directly on my PC. For this reason, we'll use ArgoCD with a GitOps workflow to install Cluster-API. New to GitOps and ArgoCD? Don't worry, it's straightforward!

Install ArgoCD and get cluster-api

ArgoCD will retrieve the YAML configurations for the various Cluster-API components. You can install ArgoCD using these two commands:

sudo k3s kubectl create namespace argocd
sudo k3s kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

You can check the status of argocd with this command sudo k3s kubectl get pods -n argocd if all pods are running you are ready for the next step.

We will install these Cluster-API components as ArgoCD Applications. An ArgoCD Application is similar to something you might know from your Google Play Store or Apple App Store – it's a packaged application or set of applications. Just as you download and install an app on your phone, ArgoCD manages the deployment and lifecycle of these Cluster-API components onto your Kubernetes clusters.

Cluster-API Operator
- The operator is responsible for installing Cluster-API and configuring it for Proxmox.
Cert-Manager.
- Cert-Manager is utilized by Cluster-API to automatically generate and manage certificates required for the Kubernetes clusters.
Caprox-Engine
- This application includes my opinionated ClusterClass, which has been designed with the needs of home labs and Small and Medium-sized Businesses (SMBs) in mind. A ClusterClass acts as a customizable template, significantly simplifying the process of creating and managing Kubernetes clusters.

# app-cert-manager.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-api-operator-cert-manager
  namespace: argocd 
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: capi-operator-system
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://charts.jetstack.io
    targetRevision: 1.17.2
    chart: cert-manager
    helm:
      values: |
        installCRDs: true
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    automated: 
      prune: true
      selfHeal: true

# app-cluster-api.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-api-operator-main
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: capi-operator-system
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://kubernetes-sigs.github.io/cluster-api-operator
    targetRevision: 0.19.0
    chart: cluster-api-operator
    helm:
      values: |
        manager:
          featureGates:
            proxmox:
              ClusterTopology: true
            core:
              ClusterTopology: true
            kubeadm:
              ClusterTopology: true
        core:
          cluster-api:
            enabled: true
            version: v1.10.2
        bootstrap:
          kubeadm: 
            enabled: true
            version: v1.10.2
        controlPlane: 
          kubeadm: 
            enabled: true
            version: v1.10.2
        infrastructure: 
          proxmox:
            enabled: true
            version: v0.7.1
        ipam:
          in-cluster:
            enabled: true
            version: v1.0.1
        addon:
          helm: 
            enabled: true
            version: v0.3.1
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    automated: 
      prune: true
      selfHeal: true

# app-caprox-engine.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-api-operator-caprox-engine
  namespace: argocd 
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: capi-operator-system
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://github.com/Caprox-eu/Proxmox-Kubernetes-Engine.git
    targetRevision: 99c87250802d886cfce28fe20a313637eae8a80a
    path: manifests/clusterclass-cilium-with-shared-ippool/base
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    automated: 
      prune: true
      selfHeal: true
  ignoreDifferences:
    - group: cluster.x-k8s.io
      kind: ClusterClass
      jsonPointers:
        - /spec

Save these files and apply them one by one. This sequential application is necessary because the applications depend on each other, and we need to ensure each one is ready before proceeding to the next.

Lets start with the Cert-Manager.

sudo k3s kubectl apply -f app-cert-manager.yaml

Wait until the App is Heathly and synced.

sudo k3s kubectl get apps -A
NAME                                 SYNC STATUS   HEALTH STATUS
cluster-api-operator-cert-manager    Synced        Healthy
...

Then go on with Cluster-API.

sudo k3s kubectl apply -f app-cluster-api.yaml

Wait until the App is Heathly and synced.

sudo k3s kubectl get apps -A
NAME                                 SYNC STATUS   HEALTH STATUS
cluster-api-operator-cert-manager    Synced        Healthy
cluster-api-operator-main            Synced        Healthy
...

Then with the caprox-engine.

sudo k3s kubectl apply -f app-caprox-engine.yaml

Wait until the App is Heathly and synced.

sudo k3s kubectl get apps -A
NAME                                 SYNC STATUS   HEALTH STATUS
cluster-api-operator-caprox-engine   Synced        Healthy
cluster-api-operator-cert-manager    Synced        Healthy
cluster-api-operator-main            Synced        Healthy
...

Nice! Now we need to configure cluster-api to our specific environment.

Connect Cluster-API to Proxmox

Cluster-API needs access to your Proxmox cluster to create VMs for Kubernetes clusters. For a homelab setup, we can simply reuse the Proxmox credentials you created in the "Create an API Token and Create a Secret with the Values" step.

Create a secret with the exact same structure, configure it for your environment, then save and apply it.

# secret-capmox.yaml
apiVersion: v1
stringData:
  secret: "6e59df15-a2c9-4dc5-b293-367772950c68"
  token: "caprox@pve!capi"
  # note: only the host - not the api-path
  url: "https://192.168.2.142:8006/"
kind: Secret
metadata:
  name: capmox-manager-credentials
  namespace: proxmox-infrastructure-system

Create the secret.

sudo k3s kubectl apply -f secret-capmox.yaml

Because we are overwriting the default secret we need to trigger a restart of the proxmox-provider to read the new secret.

sudo k3s kubectl rollout restart deploy capmox-controller-manager -n proxmox-infrastructure-system

Configure IP Range for the Cluster-API Caprox Kubernetes Engine

The Virtual Machines (VMs) provisioned by Proxmox/Cluster-API will require IP addresses. Currently, DHCP support is not available. However, we can specify an IP pool for our Kubernetes VMs. In my setup, I've disabled DHCP for a range of IPs within my FritzBox. (Note: It's recommended to disable DHCP for more IPs than you initially anticipate needing – more on this later.)

Copy this file, configure it for your network, then save and apply it.

# ip-pool.yaml
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: clusterclass-ipv4
  namespace: caprox-kubernetes-engine
spec:
  # Change the IP range to match your needs
  # These IPs will be used for the Kubernetes nodes
  # Also configure your network prefix and gateway accordingly
  addresses:
  - 192.168.2.150-192.168.2.199
  gateway: 192.168.2.1
  prefix: 24

sudo k3s kubectl apply -f ip-pool.yaml

That's it! We've successfully installed ArgoCD on our Management VM, used ArgoCD to install Cluster-API, and configured Cluster-API for our environment. Now we're finally ready to create our first multi-node Kubernetes cluster!

Create our first Workload Cluster

As always everthing is a file - same is true for a Kubernetes Cluster in Cluster-API. So yeah we will create a Kubernetes Cluster from Cluster YAML. The file will be later proccesed by Cluster-API within our Management VM.

The Cluster Resource

A cluster configuration which is compatible with our setup could look like this.

# cluster.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    caprox.eu/cni: cilium-v1.17.4
  name: manuels-k8s-cluster
  namespace: caprox-kubernetes-engine 
spec:
  topology:
    class: proxmox-clusterclass-cilium-v0.1.0
    version: 1.33.1
    controlPlane:
      replicas: 1
    workers:
      machineDeployments:
      - class: proxmox-worker
        name: proxmox-worker-pool
        replicas: 1
    variables:
    - name: cloneSpec
      value:
        vmTemplate:
          sourceNode: node01
          templateID: 114
    - name: controlPlaneEndpoint
      value:
        host: 192.168.2.201

The fields you must change are:

vmTemplate
- sourceNode: Set this to the name of the Proxmox node where your VM template is located.
- templateID: This is the ID of your Kubernetes template. You can find it in the Proxmox UI, listed as the number next to the template name.
controlPlaneEndpoint:
- This IP address will be used to access your Kubernetes cluster.
- Ensure it's outside your defined IP pool range.
- This is a floating IP, shared among different nodes. This ensures your cluster remains reachable even during node failures or maintenance, as long as at least one node is available.

The fields you can change are:

controlplane.replicas:
- This specifies the number of VMs for your control plane (master) nodes. These nodes host Kubernetes control plane components like the Kubernetes API and etcd (the Kubernetes database).
- Set this to 1 for a non-HA (High Availability) setup, or 3 or 5 for an HA setup.
workers.replicas:
- This determines the number of VMs that will run your workloads.
- Set this to at least 1.
metadata.name:
- This is the name of your Kubernetes cluster.

For more configuration options, including memory and CPU settings for your VMs, refer to the available variables here.

After you've set the required fields, remember to save the file.

Scaling and New Clusters

If you want to scale out your current cluster, simply adjust the node replica counts in your configuration file. Then, reapply the changes using kubectl apply -f cluster.yaml.

To create an additional cluster, create a new file with a different metadata.name and set your desired node replica counts.

Create the Workload Cluster and Retrieve the Kubeconfig

To create the cluster, we need to "send" our cluster configuration to our Management VM. This action will kick off the cluster creation process, which is often referred to as reconciliation. After the first control plane VM is created and started, we can retrieve the kubeconfig for our new cluster. The kubeconfig file contains all the necessary login information to access the created cluster.

# Create the Cluster
sudo k3s kubectl apply -f cluster.yaml

# Retrieve and save the kubeconfig
sudo k3s kubectl get secrets -n caprox-kubernetes-engine manuels-k8s-cluster-kubeconfig -o jsonpath='{.data.value}' | base64 --decode > kubeconfig.yaml

# Connect to the new cluster
export KUBECONFIG=./kubeconfig.yaml

You can verify the connection by listing the nodes in your new cluster:

# This command should now show your multiple nodes, depending on your configuration.
kubectl get nodes

kubectl get nodes
NAME                                            STATUS   ROLES           AGE     VERSION
manuels-k8s-cluster-control-plane-4z458-ghx2z   Ready    control-plane   11m     v1.33.1
manuels-k8s-cluster-worker-6vhjx-c2nv2-v4dbw    Ready    node            5m23s   v1.33.1

If you want to interact with the Management VM/Cluster API again, you'll need to unset the environment variable:

unset KUBECONFIG

Congratulations! You've successfully completed the whole process. You can now dynamically create Kubernetes clusters in your Proxmox datacenter in an API-driven way.

If you'd like a guide on how to update your existing cluster, please let me know in the comments!

Troubleshooting

There are many reasons why cluster creation might fail. I've tried to address all common errors within the template and reduce the setup steps to make the process as lean as possible. All versions and dependencies are fixed. I've tested this guide on multiple Proxmox datacenters and also asked friends to test it, and it has been successful every time.

However, if you encounter problems, it's likely that you either missed a step or didn't follow the guide exactly as described. Please feel free to leave a comment if you struggle or don't understand any steps.

What's Next?

Now that your Kubernetes engine is configured, what can you do with it? Here are some common next steps I like to take on newly created Kubernetes clusters:

Set up 1Password with a external secret manager for secure secret management.
Install an ingress controller to manage external access to cluster services.
Configure load balancing in Cilium for multi-node load balancing.
Install applications like databases or Pi-hole using Helm.

Trivia

For this blog post, the following PRs/comments/issues were contributed:

Using 1Password with External Secrets Operator in a GitOps way

3deep5me — Tue, 30 Jul 2024 15:35:57 +0000

I recently created a free Kubernetes cluster on Oracles Always Free Tier. But how to handle secrets in a secure way, especially in a GitOps scenario?

This post will describe the full journey how to integrate the External Secrets Operator with 1Password to automatically inject secrets from 1Password to your cluster. For managing the different components, I will use argocd, but you can use also just helm install or FluxCD.

My Goal was to have a enterprise-known solution which integrates with different secret-managment tools, in my case it's 1Password.

External Secrets Operator

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.
-external-secrets.io

I like the fact that i can theoretically change my secret-management-tool without touching my secrets. Its also possible to have the same Secret-File in all my deployments over different stages. Because the secret-value gets filled regarding to the current stage. These are all things which were often a pain for me with sealed-secrets.

1Password

Really comfy Password-manager.
I have it so why not also using it for K8s?

Architecture

Let us have a look on the architecture especially the mechanics between external secret operator with Kubernetes and 1Password with the external secret operator.

External Secret Operator

The External Secret Operator uses in general two Custom Resources. The first is the SecretStore which holds information, how to connect to the "Secret Backend". The "Secret Backend" can be something like AWS Secrets Manager, Scaleway or like in our case 1Password. It holds for example the URL and the authentication for it. You can have different SecretStores if you have more than one provider. The other resource is the ExternalSecret which is a blueprint for the secret which should be created. So in practice if you create an ExternalSecret a normal K8s-Secret follows. The ExternalSecret holds information about which entry in 1Password to fetch and in which way to populate the values in a Secret. (More later)

1Password Connect

1Password-Connect is a kind-of Proxy which caches Secrets from the 1Password Cloud-Service, so you are able to retrieve secrets even 1Password is down, it also reduces the amount of requests to 1Password.
Its needed for the External Secrets Operator to work.
In my approach, it is right next to the External Secret Operator in the same namespace.

All together the workflow looks similar to that.

External Secret is created
External Secret Operator uses the information's in SecretStore to fetch the needed password/login from 1Password Connect
1Password connect itself fetch the needed password/login from 1Password directly
External Secret Operator gets the password/login & creates the normal Kubernetes Secret with the requested values

Don't worry if you don't understand the whole process, we'll do a full hands-on walk-through at the end.

Install needed Helm-Charts

For the Installation of the helm-charts i will use argocd-apps, because i find this approach really portable. If i have a new cluster i just install argocd and then a kubectl apply . and i have all my needed apps installed. Beyond that i can use kustomize or helm and i have a auto-update feature of the apps - if i want to. Just set the targetRevision field to something like this 1.x.x.

But you can also use helm or fluxcd.

Install 1Password-Connect

Install the helm-chart

I modified the default values of the official Chart to the following:

# values.yaml
connect:
  serviceType: ClusterIP

The default value is NodePort which is IMO dangerous if you publish your Secret-management-api to the local network or even the Internet. ClusterIP is suffient because the External Secrets Operator is in the cluster - not outside.

To install the Chart with the predefined values you can just apply this argocd-app. (kubectl apply -f app-1password-connect.yaml)

# app-1password-connect.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: 1password-connect
  namespace: argocd 
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://1password.github.io/connect-helm-charts
    targetRevision: 1.x.x
    chart: connect
    helm:
      values: |
        connect:
          serviceType: ClusterIP
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

After that you should have a failed Pod in the external-secrets namespace.

$ kubectl get pods -n external-secrets
onepassword-connect-5fcbd4c68b-fgx68                         0/2     CreateContainerConfigError   0             4d20h

The reason is Error: secret "op-credentials" not found (kubectl get event -n external-secrets). This Secret contains the token which is used for the connect-server to connect to 1Password.

To create the Secret install the 1Password CLI and execute following commands.

# Create a new Vault for the Secrets
$ op vault create "K8s"
# Create a connect server in 1Password
op connect server create "Kubernetes" --vaults "K8s"

The location of the credentials file can be seen in the output.

With the 1password-credentials.json in place we can create the needed secret.

# The connect service expects the 1password-credentials.json in base64
$ kubectl create secret generic op-credentials -n external-secrets --from-literal=1password-credentials.json="$(cat /path/to/1password-credentials.json | base64)"

After a while the pod should be running

$ kubectl get pod -n external-secrets
onepassword-connect-5fcbd4c68b-chmfg                         2/2     Running   0
  21m

Congratulations you completed the first step - installed the 1Password-Connect Server in Kubernetes!

Install the External Secret Operator

For installation you can use again an argocd-app.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets-operator
  namespace: argocd 
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.x.x
    chart: external-secrets
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

After that your namespace should look like this.

$ kubectl get pods -n external-secrets
NAME                                                         READY   STATUS    RESTARTS      AGE
external-secrets-operator-5db95c68b8-j6pzj                   1/1     Running   1 (18d ago)   18d
external-secrets-operator-cert-controller-65f74dc7bf-784hg   1/1     Running   1 (18d ago)   18d
external-secrets-operator-webhook-5475bddd67-p9vq9           1/1     Running   0             18d
onepassword-connect-5fcbd4c68b-chmfg                         2/2     Running   0             31m

Then we can establish the connection to the connect server (höhö).

# Create the token and save it in the OP_CONNECT_TOKEN environment variable 
export OP_CONNECT_TOKEN=$(op connect token create "external-secret-operator" --server "Kubernetes" --vault "K8s")
# Create secret with the token which is used by the External-Secret-Operator ClusterSecretStore
kubectl create secret -n external-secrets generic onepassword-connect-token --from-literal=token=$OP_CONNECT_TOKEN

Create the ClusterSecretStore with kubectl apply -f

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: k8s
spec:
  provider:
    onepassword:
      connectHost: http://onepassword-connect:8080
      vaults:
        K8s: 1  # look in this vault first
      auth:
        secretRef:
          connectTokenSecretRef:
            name: onepassword-connect-token
            key: token
            namespace: external-secrets

To see if it the connection was successful you can check the status of the ClusterSecretStore.

kubectl get -n external-secrets clustersecretstores.external-secrets.io k8s
NAME   AGE    STATUS   CAPABILITIES   READY
k8s    103m   Valid    ReadOnly       True

Congratulations you also installed the external secret operator and established a connection with 1password connect.

End-to-End-Test the Solution

Theoretically you should be able to retrieve secrets from 1Password - but let us test it with a pragmatical example.

First let us create a secret in 1Password with the 1Password CLI.

# This creates a new entry named "Scaleway Credentails" with two properties/fields. 
op item create --vault="K8s" --title="Scaleway Credentials" --category="login" accessKeyId="token-xyz" secretKey="xyz"

In the next step we can create our ExternalSecret which then creates our normal Kubernetes Secret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  # name of the ExternalSecret & Secret which gets created
  name: scaleway-credentials
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: k8s
  target:
    creationPolicy: Owner
  data:
  - secretKey: accessKeyId
    remoteRef:
      # 1password-entry-name
      key: "Scaleway Credentials"
      # 1password-field
      property: accessKeyId
  - secretKey: secretKey
    remoteRef:
      # 1password-entry-name
      key: "Scaleway Credentials"
      # 1password-field
      property: secretKey

I tried my best to explain the references with comments.
To summarize the name of the ExternalSecret do not count. But its important that the secretKey is and the property field has the same value. The secretKey and property field represent the "Field" in 1Password e.g. password or user. The key field represents the entry in 1Password e.g. dev.to.

You can check the status of the external secret and also if the normal K8s secret was created.

$ kubectl get externalsecret -n default
NAME                   STORE   REFRESH INTERVAL   STATUS         READY
scaleway-credentials   k8s     1h                 SecretSynced   True
$kubectl get secret -n default
NAME                   TYPE     DATA   AGE
scaleway-credentials   Opaque   2      7m13s

You can see the secret is created. You can also have a deeper look into the values.

$ kubectl get secret -n default -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    accessKeyId: dG9rZW4teHl6
    secretKey: eHl6
  immutable: false
  kind: Secret
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"external-secrets.io/v1beta1","kind":"ExternalSecret","metadata":{"annotations":{},"name":"scaleway-credentials","namespace":"default"},"spec":{"data":[{"remoteRef":{"key":"Scaleway Credentials","property":"accessKeyId"},"secretKey":"accessKeyId"},{"remoteRef":{"key":"Scaleway Credentials","property":"secretKey"},"secretKey":"secretKey"}],"secretStoreRef":{"kind":"ClusterSecretStore","name":"k8s"},"target":{"creationPolicy":"Owner"}}}
      reconcile.external-secrets.io/data-hash: f3d79dd2ad3055723d0bebe3481f2372
    creationTimestamp: "2024-07-29T15:36:04Z"
    labels:
      reconcile.external-secrets.io/created-by: c66895ad2c04350c73c512c0175cee24
    name: scaleway-credentials
    namespace: default
    ownerReferences:
    - apiVersion: external-secrets.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: ExternalSecret
      name: scaleway-credentials
      uid: 1df5f0f5-ec44-4cb7-859a-3d3ef60b1d45
    resourceVersion: "689603"
    uid: 9543092a-7bdc-4695-9c83-bc35c36188c0
  type: Opaque
kind: List
metadata:
  resourceVersion: ""

Under data you can see the both keys and the corresponding values encoded in base64.

Congratulations you successfully created a secret in 1Password and pulled it into a normal Kubernetes Secret which can now be used as usual in deployments and other resources. 🥳🥳🥳

Thanks for reading my first post!
I hope you had fun and that it helped you. I'd love to hear your thoughts or feedback, so feel free to leave a comment below.