DEV Community: 3deep5me

From Zero to Scale: Proxmox Kubernetes Engine

3deep5me — Fri, 02 May 2025 13:53:31 +0000

Today, we'll take a beginner-friendly look at housing Kubernetes in Proxmox. But instead of the traditional SSH/Ansible approach, we'll explore a method akin to what you'd find with AWS, Azure, or GCP. This means we're talking about scaling from tens to hundreds of Kubernetes clusters in minutes, with automated, reproducible cluster creation and upgrades.

Does that sound like it requires heavy modifications to your Proxmox hosts or datacenter? I can reassure you: I dislike straying far from default settings, so you won't need to modify your Proxmox installation in any way. It's simply a virtual machine, allowing you to add and remove it like a plugin. (More on the architecture later.)

Why Do You Need This?

I don't need this personally — it's just a bit of fun to replicate big cloud provider functionality in my tiny homelab!

However, if your company lacks a scalable Kubernetes platform, you might find it tough to keep up in today's service-oriented world. With major cloud providers dominating, efficiently managing a private cloud is more crucial than ever, and Kubernetes is one of the most popular cloud tools. So, how can you compete? The answer lies in two of my favorite open-source projects: Cluster-API and Proxmox.

Proxmox

Proxmox, developed in Vienna, is widely known in the open-source and home-lab communities. It gained a huge boost in small to medium-scale private clouds after VMware's new pricing model alienated its customers. It's a simple yet powerful open-source hypervisor based on KVM, and it's been the core of my home lab for nearly a decade (I've been using it since PVE 4).

Cluster-API

While Cluster API might be less known in the home-lab community, it's highly valued in enterprises and by Kubernetes administrators and enthusiasts. So, what exactly is Cluster API?

Cluster API is a project under the Cloud Native Computing Foundation (CNCF), strongly supported by the Kubernetes community and various vendors, including VMware, Apple, and NVIDIA. This project offers a unified way to create and manage Kubernetes clusters across different "providers," such as Proxmox or VMware. For instance, VMware heavily leverages Cluster API in its commercial product, Tanzu.

Cluster API currently boasts over 30 infrastructure providers, with Proxmox being just one of them. In short, Cluster API provides a unified API and method for creating production-ready Kubernetes clusters across numerous providers. It has become, at least for me, the de facto standard for multi-cloud and on-premises Kubernetes deployments.

Architecture

Let me introduce you to the architecture we will use for our setup. Of course, we need at least one Proxmox host to rely on; I'll be using the newest version. On top of Proxmox, we will have our main component: the Management VM. The Management VM will house Cluster API.

The Management VM is then responsible for the Kubernetes clusters we want to create. For example, the Management VM automatically coordinates the creation and updates of Kubernetes clusters.

The Management VM achieves this by instructing Proxmox to automatically create and remove specially configured VMs on your Proxmox infrastructure. The process in detail would look like this:

We will give our Management VM the order to create, for instance, a 12-node Kubernetes cluster. The Management VM, or more precisely, Cluster API inside the Management VM, will communicate with Proxmox. Cluster API will then instruct Proxmox to provision 12 VMs from a special Kubernetes VM template. After the successful creation of the VMs, Cluster API configures the cluster. Additionally, Cluster API will monitor the VMs and can replace them automatically upon failure.

You can see that this setup is far more capable than statically creating clusters with tools like Terraform and Ansible. This is why setups like this are often referred to as a Kubernetes Platform - because you can order clusters just like you order a pizza from Lieferando.

In my opinion, it's also a less complicated way to set up multi-node Kubernetes clusters on Proxmox. We simply need to install Cluster API on our Management VM and have a Kubernetes VM template ready. No complex Ansible Playbooks or Terraform-states are required.

Our Path to a Scalable Kubernetes Platform on Proxmox

In this blog post, we'll follow these steps to achieve our goal of creating a scalable Kubernetes platform on Proxmox:

Create the Management VM as our pivotal point.
Create a Kubernetes VM Template for our Kubernetes Clusters.
Initialize Cluster API & Configure the "Caprox-Kubernetes-Engine."
Create our first Workload Cluster.

In my example, the Management VM and Kubernetes clusters will all reside on a single Proxmox host within one network that utilizes DHCP. DHCP is a requirement.

The Guide still works - but it is a bit outdated. I recommend to check out the GitHub Project which evolved out of this blogpost. Here is the updated Tutorial: Proxmox-Kubernetes-Engine/Quick-Start

Create the Management VM with k3sup

Disclaimer: All the steps shown here are for home-lab purposes only and should not be used for a production or even development environment. This guide is merely a starting point. If you're looking for a more production-ready setup, feel free to reach out to me.

Create a New VM for Our Management VM

We need a simple Linux VM for our Management VM with 30 GiB of disk space and at least 4 GiB of RAM.

For Proxmox beginners, I recommend this great guide. You'll also need an ISO image, which you can download here from the official Ubuntu website.

Of course, you can also use other methods such as Terraform, cloud-init or cloning an existing Linux VM or template.

Once you've logged into your new VM, we can proceed.

Install Kubernetes (K3s) on Our New VM

Cluster API is Kubernetes-exclusive, meaning it itself relies on Kubernetes to operate. Because of this, we need Kubernetes on our Management VM. In this guide, we'll use K3s for that. K3s is a Kubernetes distribution from Rancher that is now fully community-driven and under the CNCF. It's a popular Kubernetes distro for small to mid-sized clusters.

To install K3s, we'll use k3sup - a really simple CLI tool that allows you to create a K3s Kubernetes cluster within seconds on any Linux VM. The following commands will download k3sup and then create a simple cluster.

# install k3s-Kubernetes with k3sup
curl -sLS https://get.k3sup.dev | sh
sudo cp k3sup /usr/local/bin/k3sup
k3sup install --local --k3s-version v1.33.1+k3s1

You can test if the cluster is working with sudo k3s kubectl get nodes. It should show something like this:

sudo k3s kubectl get nodes
NAME        STATUS   ROLES                  AGE     VERSION
localhost   Ready    control-plane,master   6m30s   v1.33.1+k3s1

Congratulations! You've successfully set up a simple single-node Kubernetes cluster. This cluster will serve as our Cluster API Management VM.

Creating Your Kubernetes VM Template

Every Kubernetes cluster typically starts with a machine template, most often a VM template. This template is where all Kubernetes components are configured and downloaded. For Cluster API, we need VM templates that already include essential Kubernetes packages like the API server and the container runtime.

We'll achieve this with the help of the excellent Kubernetes Image Builder project. While it sounds fancy, it's essentially Packer and Ansible cleverly glued together to automate the image creation process. But as promised, you don't need to touch Ansible or Packer at all.

Create an API Token and a Secret with the Values

To get started, we need to create API access to our Proxmox Datacenter. You might wonder why. The image builder constructs the image directly on your Proxmox node, so it requires access to start a VM and later convert it into a template.

To begin, open a shell on your Proxmox node. You can do this by clicking on your Proxmox node in the interface and selecting "Shell"; a command-line interface will then open directly in your browser. Once the shell is open, execute the following commands:

pveum user add caprox@pve
pveum aclmod / -user caprox@pve -role PVEAdmin
pveum user token add caprox@pve capi -privsep 0

After that you should get all relevant information.

root@node01:~# pveum user token add caprox@pve capi -privsep 0
┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ caprox@pve!capi                      │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ 6e59df15-a2c9-4dc5-b293-367772950c68 │
└──────────────┴──────────────────────────────────────┘

We will also use our Management VM to coordinate the VM Template Builds. For that we will use the Kubernetes Job Resource and a Kubernetes Secret to pass the API-Key and some configuration.

First let us create the Secret with values we got from above.
You only need to change the first four Proxmox-Vars.

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: proxmox-image-build-config
  namespace: proxmox-build-infrastructure-system
type: Opaque
stringData:
  PROXMOX_URL: "https://your-proxmox-adress:8006/api2/json"
  PROXMOX_USERNAME: "full-tokenid"
  PROXMOX_TOKEN: "value"
  PROXMOX_NODE: "your proxmox node name"
  PROXMOX_ISO_POOL: "local" #this should be fine for the most users
  PROXMOX_BRIDGE: "vmbr0" #this should be fine for the most users
  PROXMOX_STORAGE_POOL: "local" #this should be fine for the most users
  PACKER_FLAGS: "--var memory=4096 --var 'kubernetes_rpm_version=1.33.1' --var 'kubernetes_semver=v1.33.1' --var 'kubernetes_series=v1.33' --var 'kubernetes_deb_version=1.33.1-1.1'"

Configure needed values and save the File.

Now we need also a Job which creates the Image. A Job is a Kubernetes Pod which is only executed once with a finite life - until the Job successfully complete. It uses Image-Builder Docker-Image with the required configuration to build Kubernetes Proxmox VM-Templates.

# job.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: proxmox-template-builder
  namespace: proxmox-build-infrastructure-system
spec:
  schedule: "* * * * *"
  suspend: true 
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          restartPolicy: OnFailure
          containers:
          - name: image-builder
            image: registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.45
            envFrom:
            - secretRef:
                name: proxmox-image-build-config
            args:
            - build-proxmox-ubuntu-2404
            env:
            # help for slow nodes
              - name: ANSIBLE_TIMEOUT
                value: "60"

Just copy and save the file.

Start the Build-Job and create a Kubernetes Image

Now you should have two file one Secret and one (Cron)-Job, the Cron-Job has the benefit of easier executing if we need new image or even do scheduled automatic builds. Now its build time!

# create a namespace for the build
sudo k3s kubectl create namespace proxmox-build-infrastructure-system
# apply secret & job
sudo k3s kubectl apply -f secret.yaml
sudo k3s kubectl apply -f job.yaml
# start the build
sudo k3s kubectl create job build-image --from cj/proxmox-template-builder -n proxmox-build-infrastructure-system

To Monitor the current state we can look at the Logs of the Builder-Pod. For that we need the name and the kubectl logs command.

sudo k3s kubectl get pods -n proxmox-build-infrastructure-system
NAME                READY   STATUS    RESTARTS   AGE
build-image-tdv78   1/1     Running   0          97s
sudo k3s kubectl logs -f -n proxmox-build-infrastructure-system build-image-tdv78
proxmox-iso.ubuntu-2204: output will be in this color.
==> proxmox-iso.ubuntu-2204: Retrieving ISO
...

You should see a new VM pop-up in your Proxmox UI. This is often the trickiest part of the guide. Sometimes it can take a long time, or the job might even require multiple attempts. Kubernetes will automatically restart the job if the build fails, so you can just relax and let it do its thing. In my experience, it occasionally took over 30 minutes to create the VM template.

Please ensure that you have DHCP configured in your network. If you encounter any issues, feel free to leave a comment below; I'm happy to help! Sometimes a Proxmox reboot also helps.

If you're lucky, you should see a new template in the UI after a few minutes.

And thats it - you completed the "most complicated" part!

Initialize Cluster-API

In the next step, we'll configure Cluster-API to our needs. I prefer to avoid keeping files, especially complex ones, directly on my PC. For this reason, we'll use ArgoCD with a GitOps workflow to install Cluster-API. New to GitOps and ArgoCD? Don't worry, it's straightforward!

Install ArgoCD and get cluster-api

ArgoCD will retrieve the YAML configurations for the various Cluster-API components. You can install ArgoCD using these two commands:

sudo k3s kubectl create namespace argocd
sudo k3s kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

You can check the status of argocd with this command sudo k3s kubectl get pods -n argocd if all pods are running you are ready for the next step.

We will install these Cluster-API components as ArgoCD Applications. An ArgoCD Application is similar to something you might know from your Google Play Store or Apple App Store – it's a packaged application or set of applications. Just as you download and install an app on your phone, ArgoCD manages the deployment and lifecycle of these Cluster-API components onto your Kubernetes clusters.

Cluster-API Operator
- The operator is responsible for installing Cluster-API and configuring it for Proxmox.
Cert-Manager.
- Cert-Manager is utilized by Cluster-API to automatically generate and manage certificates required for the Kubernetes clusters.
Caprox-Engine
- This application includes my opinionated ClusterClass, which has been designed with the needs of home labs and Small and Medium-sized Businesses (SMBs) in mind. A ClusterClass acts as a customizable template, significantly simplifying the process of creating and managing Kubernetes clusters.

# app-cert-manager.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-api-operator-cert-manager
  namespace: argocd 
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: capi-operator-system
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://charts.jetstack.io
    targetRevision: 1.17.2
    chart: cert-manager
    helm:
      values: |
        installCRDs: true
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    automated: 
      prune: true
      selfHeal: true

# app-cluster-api.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-api-operator-main
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: capi-operator-system
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://kubernetes-sigs.github.io/cluster-api-operator
    targetRevision: 0.19.0
    chart: cluster-api-operator
    helm:
      values: |
        manager:
          featureGates:
            proxmox:
              ClusterTopology: true
            core:
              ClusterTopology: true
            kubeadm:
              ClusterTopology: true
        core:
          cluster-api:
            enabled: true
            version: v1.10.2
        bootstrap:
          kubeadm: 
            enabled: true
            version: v1.10.2
        controlPlane: 
          kubeadm: 
            enabled: true
            version: v1.10.2
        infrastructure: 
          proxmox:
            enabled: true
            version: v0.7.1
        ipam:
          in-cluster:
            enabled: true
            version: v1.0.1
        addon:
          helm: 
            enabled: true
            version: v0.3.1
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    automated: 
      prune: true
      selfHeal: true

# app-caprox-engine.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-api-operator-caprox-engine
  namespace: argocd 
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: capi-operator-system
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://github.com/Caprox-eu/Proxmox-Kubernetes-Engine.git
    targetRevision: 99c87250802d886cfce28fe20a313637eae8a80a
    path: manifests/clusterclass-cilium-with-shared-ippool/base
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    automated: 
      prune: true
      selfHeal: true
  ignoreDifferences:
    - group: cluster.x-k8s.io
      kind: ClusterClass
      jsonPointers:
        - /spec

Save these files and apply them one by one. This sequential application is necessary because the applications depend on each other, and we need to ensure each one is ready before proceeding to the next.

Lets start with the Cert-Manager.

sudo k3s kubectl apply -f app-cert-manager.yaml

Wait until the App is Heathly and synced.

sudo k3s kubectl get apps -A
NAME                                 SYNC STATUS   HEALTH STATUS
cluster-api-operator-cert-manager    Synced        Healthy
...

Then go on with Cluster-API.

sudo k3s kubectl apply -f app-cluster-api.yaml

Wait until the App is Heathly and synced.

sudo k3s kubectl get apps -A
NAME                                 SYNC STATUS   HEALTH STATUS
cluster-api-operator-cert-manager    Synced        Healthy
cluster-api-operator-main            Synced        Healthy
...

Then with the caprox-engine.

sudo k3s kubectl apply -f app-caprox-engine.yaml

Wait until the App is Heathly and synced.

sudo k3s kubectl get apps -A
NAME                                 SYNC STATUS   HEALTH STATUS
cluster-api-operator-caprox-engine   Synced        Healthy
cluster-api-operator-cert-manager    Synced        Healthy
cluster-api-operator-main            Synced        Healthy
...

Nice! Now we need to configure cluster-api to our specific environment.

Connect Cluster-API to Proxmox

Cluster-API needs access to your Proxmox cluster to create VMs for Kubernetes clusters. For a homelab setup, we can simply reuse the Proxmox credentials you created in the "Create an API Token and Create a Secret with the Values" step.

Create a secret with the exact same structure, configure it for your environment, then save and apply it.

# secret-capmox.yaml
apiVersion: v1
stringData:
  secret: "6e59df15-a2c9-4dc5-b293-367772950c68"
  token: "caprox@pve!capi"
  # note: only the host - not the api-path
  url: "https://192.168.2.142:8006/"
kind: Secret
metadata:
  name: capmox-manager-credentials
  namespace: proxmox-infrastructure-system

Create the secret.

sudo k3s kubectl apply -f secret-capmox.yaml

Because we are overwriting the default secret we need to trigger a restart of the proxmox-provider to read the new secret.

sudo k3s kubectl rollout restart deploy capmox-controller-manager -n proxmox-infrastructure-system

Configure IP Range for the Cluster-API Caprox Kubernetes Engine

The Virtual Machines (VMs) provisioned by Proxmox/Cluster-API will require IP addresses. Currently, DHCP support is not available. However, we can specify an IP pool for our Kubernetes VMs. In my setup, I've disabled DHCP for a range of IPs within my FritzBox. (Note: It's recommended to disable DHCP for more IPs than you initially anticipate needing – more on this later.)

Copy this file, configure it for your network, then save and apply it.

# ip-pool.yaml
apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: clusterclass-ipv4
  namespace: caprox-kubernetes-engine
spec:
  # Change the IP range to match your needs
  # These IPs will be used for the Kubernetes nodes
  # Also configure your network prefix and gateway accordingly
  addresses:
  - 192.168.2.150-192.168.2.199
  gateway: 192.168.2.1
  prefix: 24

sudo k3s kubectl apply -f ip-pool.yaml

That's it! We've successfully installed ArgoCD on our Management VM, used ArgoCD to install Cluster-API, and configured Cluster-API for our environment. Now we're finally ready to create our first multi-node Kubernetes cluster!

Create our first Workload Cluster

As always everthing is a file - same is true for a Kubernetes Cluster in Cluster-API. So yeah we will create a Kubernetes Cluster from Cluster YAML. The file will be later proccesed by Cluster-API within our Management VM.

The Cluster Resource

A cluster configuration which is compatible with our setup could look like this.

# cluster.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    caprox.eu/cni: cilium-v1.17.4
  name: manuels-k8s-cluster
  namespace: caprox-kubernetes-engine 
spec:
  topology:
    class: proxmox-clusterclass-cilium-v0.1.0
    version: 1.33.1
    controlPlane:
      replicas: 1
    workers:
      machineDeployments:
      - class: proxmox-worker
        name: proxmox-worker-pool
        replicas: 1
    variables:
    - name: cloneSpec
      value:
        vmTemplate:
          sourceNode: node01
          templateID: 114
    - name: controlPlaneEndpoint
      value:
        host: 192.168.2.201

The fields you must change are:

vmTemplate
- sourceNode: Set this to the name of the Proxmox node where your VM template is located.
- templateID: This is the ID of your Kubernetes template. You can find it in the Proxmox UI, listed as the number next to the template name.
controlPlaneEndpoint:
- This IP address will be used to access your Kubernetes cluster.
- Ensure it's outside your defined IP pool range.
- This is a floating IP, shared among different nodes. This ensures your cluster remains reachable even during node failures or maintenance, as long as at least one node is available.

The fields you can change are:

controlplane.replicas:
- This specifies the number of VMs for your control plane (master) nodes. These nodes host Kubernetes control plane components like the Kubernetes API and etcd (the Kubernetes database).
- Set this to 1 for a non-HA (High Availability) setup, or 3 or 5 for an HA setup.
workers.replicas:
- This determines the number of VMs that will run your workloads.
- Set this to at least 1.
metadata.name:
- This is the name of your Kubernetes cluster.

For more configuration options, including memory and CPU settings for your VMs, refer to the available variables here.

After you've set the required fields, remember to save the file.

Scaling and New Clusters

If you want to scale out your current cluster, simply adjust the node replica counts in your configuration file. Then, reapply the changes using kubectl apply -f cluster.yaml.

To create an additional cluster, create a new file with a different metadata.name and set your desired node replica counts.

Create the Workload Cluster and Retrieve the Kubeconfig

To create the cluster, we need to "send" our cluster configuration to our Management VM. This action will kick off the cluster creation process, which is often referred to as reconciliation. After the first control plane VM is created and started, we can retrieve the kubeconfig for our new cluster. The kubeconfig file contains all the necessary login information to access the created cluster.

# Create the Cluster
sudo k3s kubectl apply -f cluster.yaml

# Retrieve and save the kubeconfig
sudo k3s kubectl get secrets -n caprox-kubernetes-engine manuels-k8s-cluster-kubeconfig -o jsonpath='{.data.value}' | base64 --decode > kubeconfig.yaml

# Connect to the new cluster
export KUBECONFIG=./kubeconfig.yaml

You can verify the connection by listing the nodes in your new cluster:

# This command should now show your multiple nodes, depending on your configuration.
kubectl get nodes

kubectl get nodes
NAME                                            STATUS   ROLES           AGE     VERSION
manuels-k8s-cluster-control-plane-4z458-ghx2z   Ready    control-plane   11m     v1.33.1
manuels-k8s-cluster-worker-6vhjx-c2nv2-v4dbw    Ready    node            5m23s   v1.33.1

If you want to interact with the Management VM/Cluster API again, you'll need to unset the environment variable:

unset KUBECONFIG

Congratulations! You've successfully completed the whole process. You can now dynamically create Kubernetes clusters in your Proxmox datacenter in an API-driven way.

If you'd like a guide on how to update your existing cluster, please let me know in the comments!

Troubleshooting

There are many reasons why cluster creation might fail. I've tried to address all common errors within the template and reduce the setup steps to make the process as lean as possible. All versions and dependencies are fixed. I've tested this guide on multiple Proxmox datacenters and also asked friends to test it, and it has been successful every time.

However, if you encounter problems, it's likely that you either missed a step or didn't follow the guide exactly as described. Please feel free to leave a comment if you struggle or don't understand any steps.

What's Next?

Now that your Kubernetes engine is configured, what can you do with it? Here are some common next steps I like to take on newly created Kubernetes clusters:

Set up 1Password with a external secret manager for secure secret management.
Install an ingress controller to manage external access to cluster services.
Configure load balancing in Cilium for multi-node load balancing.
Install applications like databases or Pi-hole using Helm.

Trivia

For this blog post, the following PRs/comments/issues were contributed:

Using 1Password with External Secrets Operator in a GitOps way

3deep5me — Tue, 30 Jul 2024 15:35:57 +0000

I recently created a free Kubernetes cluster on Oracles Always Free Tier. But how to handle secrets in a secure way, especially in a GitOps scenario?

This post will describe the full journey how to integrate the External Secrets Operator with 1Password to automatically inject secrets from 1Password to your cluster. For managing the different components, I will use argocd, but you can use also just helm install or FluxCD.

My Goal was to have a enterprise-known solution which integrates with different secret-managment tools, in my case it's 1Password.

External Secrets Operator

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.
-external-secrets.io

I like the fact that i can theoretically change my secret-management-tool without touching my secrets. Its also possible to have the same Secret-File in all my deployments over different stages. Because the secret-value gets filled regarding to the current stage. These are all things which were often a pain for me with sealed-secrets.

1Password

Really comfy Password-manager.
I have it so why not also using it for K8s?

Architecture

Let us have a look on the architecture especially the mechanics between external secret operator with Kubernetes and 1Password with the external secret operator.

External Secret Operator

The External Secret Operator uses in general two Custom Resources. The first is the SecretStore which holds information, how to connect to the "Secret Backend". The "Secret Backend" can be something like AWS Secrets Manager, Scaleway or like in our case 1Password. It holds for example the URL and the authentication for it. You can have different SecretStores if you have more than one provider. The other resource is the ExternalSecret which is a blueprint for the secret which should be created. So in practice if you create an ExternalSecret a normal K8s-Secret follows. The ExternalSecret holds information about which entry in 1Password to fetch and in which way to populate the values in a Secret. (More later)

1Password Connect

1Password-Connect is a kind-of Proxy which caches Secrets from the 1Password Cloud-Service, so you are able to retrieve secrets even 1Password is down, it also reduces the amount of requests to 1Password.
Its needed for the External Secrets Operator to work.
In my approach, it is right next to the External Secret Operator in the same namespace.

All together the workflow looks similar to that.

External Secret is created
External Secret Operator uses the information's in SecretStore to fetch the needed password/login from 1Password Connect
1Password connect itself fetch the needed password/login from 1Password directly
External Secret Operator gets the password/login & creates the normal Kubernetes Secret with the requested values

Don't worry if you don't understand the whole process, we'll do a full hands-on walk-through at the end.

Install needed Helm-Charts

For the Installation of the helm-charts i will use argocd-apps, because i find this approach really portable. If i have a new cluster i just install argocd and then a kubectl apply . and i have all my needed apps installed. Beyond that i can use kustomize or helm and i have a auto-update feature of the apps - if i want to. Just set the targetRevision field to something like this 1.x.x.

But you can also use helm or fluxcd.

Install 1Password-Connect

Install the helm-chart

I modified the default values of the official Chart to the following:

# values.yaml
connect:
  serviceType: ClusterIP

The default value is NodePort which is IMO dangerous if you publish your Secret-management-api to the local network or even the Internet. ClusterIP is suffient because the External Secrets Operator is in the cluster - not outside.

To install the Chart with the predefined values you can just apply this argocd-app. (kubectl apply -f app-1password-connect.yaml)

# app-1password-connect.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: 1password-connect
  namespace: argocd 
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://1password.github.io/connect-helm-charts
    targetRevision: 1.x.x
    chart: connect
    helm:
      values: |
        connect:
          serviceType: ClusterIP
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

After that you should have a failed Pod in the external-secrets namespace.

$ kubectl get pods -n external-secrets
onepassword-connect-5fcbd4c68b-fgx68                         0/2     CreateContainerConfigError   0             4d20h

The reason is Error: secret "op-credentials" not found (kubectl get event -n external-secrets). This Secret contains the token which is used for the connect-server to connect to 1Password.

To create the Secret install the 1Password CLI and execute following commands.

# Create a new Vault for the Secrets
$ op vault create "K8s"
# Create a connect server in 1Password
op connect server create "Kubernetes" --vaults "K8s"

The location of the credentials file can be seen in the output.

With the 1password-credentials.json in place we can create the needed secret.

# The connect service expects the 1password-credentials.json in base64
$ kubectl create secret generic op-credentials -n external-secrets --from-literal=1password-credentials.json="$(cat /path/to/1password-credentials.json | base64)"

After a while the pod should be running

$ kubectl get pod -n external-secrets
onepassword-connect-5fcbd4c68b-chmfg                         2/2     Running   0
  21m

Congratulations you completed the first step - installed the 1Password-Connect Server in Kubernetes!

Install the External Secret Operator

For installation you can use again an argocd-app.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets-operator
  namespace: argocd 
spec:
  destination:
    namespace: external-secrets
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.x.x
    chart: external-secrets
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

After that your namespace should look like this.

$ kubectl get pods -n external-secrets
NAME                                                         READY   STATUS    RESTARTS      AGE
external-secrets-operator-5db95c68b8-j6pzj                   1/1     Running   1 (18d ago)   18d
external-secrets-operator-cert-controller-65f74dc7bf-784hg   1/1     Running   1 (18d ago)   18d
external-secrets-operator-webhook-5475bddd67-p9vq9           1/1     Running   0             18d
onepassword-connect-5fcbd4c68b-chmfg                         2/2     Running   0             31m

Then we can establish the connection to the connect server (höhö).

# Create the token and save it in the OP_CONNECT_TOKEN environment variable 
export OP_CONNECT_TOKEN=$(op connect token create "external-secret-operator" --server "Kubernetes" --vault "K8s")
# Create secret with the token which is used by the External-Secret-Operator ClusterSecretStore
kubectl create secret -n external-secrets generic onepassword-connect-token --from-literal=token=$OP_CONNECT_TOKEN

Create the ClusterSecretStore with kubectl apply -f

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: k8s
spec:
  provider:
    onepassword:
      connectHost: http://onepassword-connect:8080
      vaults:
        K8s: 1  # look in this vault first
      auth:
        secretRef:
          connectTokenSecretRef:
            name: onepassword-connect-token
            key: token
            namespace: external-secrets

To see if it the connection was successful you can check the status of the ClusterSecretStore.

kubectl get -n external-secrets clustersecretstores.external-secrets.io k8s
NAME   AGE    STATUS   CAPABILITIES   READY
k8s    103m   Valid    ReadOnly       True

Congratulations you also installed the external secret operator and established a connection with 1password connect.

End-to-End-Test the Solution

Theoretically you should be able to retrieve secrets from 1Password - but let us test it with a pragmatical example.

First let us create a secret in 1Password with the 1Password CLI.

# This creates a new entry named "Scaleway Credentails" with two properties/fields. 
op item create --vault="K8s" --title="Scaleway Credentials" --category="login" accessKeyId="token-xyz" secretKey="xyz"

In the next step we can create our ExternalSecret which then creates our normal Kubernetes Secret.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  # name of the ExternalSecret & Secret which gets created
  name: scaleway-credentials
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: k8s
  target:
    creationPolicy: Owner
  data:
  - secretKey: accessKeyId
    remoteRef:
      # 1password-entry-name
      key: "Scaleway Credentials"
      # 1password-field
      property: accessKeyId
  - secretKey: secretKey
    remoteRef:
      # 1password-entry-name
      key: "Scaleway Credentials"
      # 1password-field
      property: secretKey

I tried my best to explain the references with comments.
To summarize the name of the ExternalSecret do not count. But its important that the secretKey is and the property field has the same value. The secretKey and property field represent the "Field" in 1Password e.g. password or user. The key field represents the entry in 1Password e.g. dev.to.

You can check the status of the external secret and also if the normal K8s secret was created.

$ kubectl get externalsecret -n default
NAME                   STORE   REFRESH INTERVAL   STATUS         READY
scaleway-credentials   k8s     1h                 SecretSynced   True
$kubectl get secret -n default
NAME                   TYPE     DATA   AGE
scaleway-credentials   Opaque   2      7m13s

You can see the secret is created. You can also have a deeper look into the values.

$ kubectl get secret -n default -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    accessKeyId: dG9rZW4teHl6
    secretKey: eHl6
  immutable: false
  kind: Secret
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"external-secrets.io/v1beta1","kind":"ExternalSecret","metadata":{"annotations":{},"name":"scaleway-credentials","namespace":"default"},"spec":{"data":[{"remoteRef":{"key":"Scaleway Credentials","property":"accessKeyId"},"secretKey":"accessKeyId"},{"remoteRef":{"key":"Scaleway Credentials","property":"secretKey"},"secretKey":"secretKey"}],"secretStoreRef":{"kind":"ClusterSecretStore","name":"k8s"},"target":{"creationPolicy":"Owner"}}}
      reconcile.external-secrets.io/data-hash: f3d79dd2ad3055723d0bebe3481f2372
    creationTimestamp: "2024-07-29T15:36:04Z"
    labels:
      reconcile.external-secrets.io/created-by: c66895ad2c04350c73c512c0175cee24
    name: scaleway-credentials
    namespace: default
    ownerReferences:
    - apiVersion: external-secrets.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: ExternalSecret
      name: scaleway-credentials
      uid: 1df5f0f5-ec44-4cb7-859a-3d3ef60b1d45
    resourceVersion: "689603"
    uid: 9543092a-7bdc-4695-9c83-bc35c36188c0
  type: Opaque
kind: List
metadata:
  resourceVersion: ""

Under data you can see the both keys and the corresponding values encoded in base64.

Congratulations you successfully created a secret in 1Password and pulled it into a normal Kubernetes Secret which can now be used as usual in deployments and other resources. 🥳🥳🥳

Thanks for reading my first post!
I hope you had fun and that it helped you. I'd love to hear your thoughts or feedback, so feel free to leave a comment below.