Bootstrap cluster with FluxCD

That is the next iteration of the "5 AM Club" Kubernetes migration. As you can remember from other entries published in this series, I started playing with Kubernetes daily. Not on a daily basis, but literally every single day. To be honest, I'm pretty happy with the results. However, my plan has one challenge, that requires to be solved. Build time was long ~15 minutes every day, and sometimes ESO operator faced an issue regarding using kustomize for Helm-based deployment. And just due to random constraints, I wanted to use one tool for bootstrapping a cluster.

That is why I started with the layout presented last time. So logical split per operator with the base/overlay sub-group.

Then I thought, why not just use some standard solution for that?
The first idea was to extend the usage of Argo for my infrastructure. But... It requires setting ESO, Doppler secret, and ArgoCD installed. Then I can reconfigure apps from the git level.

The challenge was, to make it easier, faster, and more standardised, than it was currently.

Starting from beginning

What Flux is? And what is not?

Flux is a tool for keeping Kubernetes
clusters in sync with sources of configuration (like Git repositories),
and automating updates to the configuration when there is new code to deploy.

We can use Flux to keep the configuration of our whole cluster in sync, with the Git repository. The funny thing is that we can configure both apps and infra with the usage of Flux.

However, managing apps with Flux in my opinion is not the easiest and most comfortable solution. Especially, if we're changing versions quite often and we would like to have at least a few dependencies between apps and infra. For example, my Immich instance needs csi-driver-smb, which on the other hand requires external-secret-operator and external-secret-secret
(the actual link between in-cluster secret and ESO ClusterSecretStore). So, every new relese, needs to be built and checks that all kustomizations are in place.
Then actually deploy a new version. Very long process, also ArgoCD UI is just better, easier to use, and definitely more user-friendly - at least in my opinion.

Repository structure

So after a few initial rounds it's ended in the following state:

.
├── README.md
├── clusters
│   └── cluster0
│       └── flux-system
│           ├── gotk-components.yaml
│           ├── gotk-sync.yaml
│           ├── infrastructure.yaml
│           └── kustomization.yaml
└── infrastructure
    └── controllers
        ├── argocd-operator
        │   ├── configmap-patch.yaml
        │   ├── kustomization.yaml
        │   ├── namespace.yaml
        │   └── patch-argocd-server-annotations.yaml
        ├── argocd-operator-apps
        │   ├── applications.yaml
        │   ├── kustomization.yaml
        │   ├── projects.yaml
        │   └── repositories.yaml
        ├── csi-driver-smb
        │   ├── csi-driver-smb.yaml
        │   └── kustomization.yaml
        ├── external-secrets
        │   ├── external-secrets-operator.yaml
        │   └── kustomization.yaml
        ├── external-secrets-secret
        │   ├── cluster-secret-store.yaml
        │   └── kustomization.yaml
        ├── tailscale-operator
        │   ├── kustomization.yaml
        │   └── tailscale-operator.yaml
        ├── tailscale-operator-secrets
        │   ├── kustomization.yaml
        │   └── tailscale-operator-exteral-secret.yaml
        └── traefik
            ├── kustomization.yaml
            └── traefik-ext-conf.yaml

Now we need some explanation, right?

Flux configuration

clusters
└── cluster0
    └── flux-system
        ├── gotk-components.yaml
        ├── gotk-sync.yaml
        ├── infrastructure.yaml
        └── kustomization.yaml

Here we have the configuration of our cluster, which is an awesome idea. In one repository we can have configurations for multiple clusters, based on provider, environment, or location, and manage them in a very simple way. Then we have regular flux-system files, so flux-system/gotk-components.yaml and flux-system/gotk-sync.yaml. Next, let's talk about my simple Kustomization file

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - infrastructure.yaml

This just tells Flux, that after bootstrapping itself, install
manifests based on infrastructure.yaml file. So let's take a look at the most crucial part of the config.

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: csi-driver-smb
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/csi-driver-smb
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets-secret
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/external-secrets-secret
  dependsOn:
    - name: external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tailscale-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/tailscale-operator
  dependsOn:
    - name: external-secrets-secret
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tailscale-operator-secret
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/tailscale-operator-secrets
  dependsOn:
    - name: external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: traefik
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/traefik
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: argocd-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/argocd-operator
  dependsOn:
    - name: tailscale-operator
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: argocd-apps
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/argocd-operator-apps
  dependsOn:
    - name: argocd-operator
  prune: true
  wait: true

As you can see I heavily rely on dependsOn command. That is due Flux and Kubernetes archtecture design. When we're applying a Kustomization, we do not control the order of creating the resources. Of course, if we put in one file, service, ingress, and deployment, Kubernetes will know how to handle that. The problem is when we deploy an application that has dependencies on each other, that is not what Kubernetes understands. So we need to specify it directly. In my example csi-driver-smb, external-secrets, and traefik can be installed in parallel, but then we have some relations. At the end, we're installing Argo's app-of-apps, which requires in general all previous components, besides traefik - I'm routing my traffic through Tailscale there, not via the Internet.

Now you can think:

App-of-apps? You said infra only!

That is right, my logic was quite complex, to be honest here. Do you remember, when I wrote about use-case? Bootstraping the whole cluster so that I will be able to work with it fast. Apps are part of the overall cluster at the end. So my app-of-apps definition was very simple:

infrastructure/controllers/argocd-operator-apps
├── applications.yaml
├── kustomization.yaml
├── projects.yaml
└── repositories.yaml

Let's dive into it one, one by one.

1. applications.yaml
   

    ---
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: self-sync
      namespace: argocd
    spec:
      syncPolicy:
        automated:
          selfHeal: true
      project: non-core-namespaces
      source:
        repoURL: https://codeberg.org/3sky/argocd-for-home
        targetRevision: HEAD
        path: app-of-apps
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
   

   That is a very simple definition of my main Application that controls all apps running on my cluster. Nothing very special here, I was experimenting with codeberg, but let's talk about that later.

2. kustomization.yaml
   

     ---
     apiVersion: kustomize.config.k8s.io/v1beta1
     kind: Kustomization
   
     namespace: argocd
     resources:
       - repositories.yaml
       - projects.yaml
       - applications.yaml
   

   Kustomization is simple, just ordered in the logic path.

   Repository -> Projects -> Application

3. projects.yaml
   

     ---
     apiVersion: argoproj.io/v1alpha1
     kind: AppProject
     metadata:
       name: non-core-namespaces
       namespace: argocd
     spec:
       description: Allow argo deploy everywhere
       sourceRepos:
         - 'https://codeberg.org/3sky/argocd-for-home'
       destinations:
         - namespace: '*'
           server: https://kubernetes.default.svc
       namespaceResourceWhitelist:
         - group: '*'
           kind: '*'
       clusterResourceWhitelist:
         - group: '*'
           kind: '*'
   

   As I'm creating namespaces as part of my application setup, permission granted to Argo's Projects are very wide. In this case, I just trust the GitOps process.

4. repositories.yaml
   

     apiVersion: external-secrets.io/v1beta1
     kind: ExternalSecret
     metadata:
       name: gitops-with-argo-secret
       namespace: argocd
     spec:
       refreshInterval: 6h
       secretStoreRef:
         name: doppler-auth-argocd
         kind: ClusterSecretStore
       target:
         name: gitops-with-argo
         creationPolicy: Owner
         template:
           type: Opaque
           metadata:
             labels:
               argocd.argoproj.io/secret-type: repository
           data:
             type: git
             url: https://codeberg.org/3sky/argocd-for-home
             username: 3sky
             password: "{{ .password }}"
       data:
         - secretKey: password
           remoteRef:
             key: CODEBERG_TOKEN
   

   This is probably the most interesting part of the configuration. As we're unable to inject our password directly into argocd.argoproj.io/secret-type: repository, which is a regular Kubernetes Secret. We need to generate the whole object
   with ESO - details can be found here.

And that's it. Now let's talk about the actual bootstrap process.

Bootstraping environment

1. Spin up infrastructure with Terraform.
   

   terraform apply -var="local_ip=$(curl -s ifconfig.me)"
   

2. Installing K3S with local fork of k3s-ansible.
   

   ansible-playbook playbooks/site.yml -i inventory.yml
   

3. Load KUBECONFIG
   

   export KUBECONFIG=/home/kuba/.kube/config.hetzner-prod
   kubectl config use-context k3s-ansible
   

4. Doppler configuration (we need an initail secret somewhere)
   

   kubectl create namespace external-secrets
   kubectl create secret generic \
     -n external-secrets doppler-token-argocd \
     --from-literal dopplerToken=""
   

5. Bootstrap the cluster
   

   flux bootstrap github \
     --owner=3sky \
     --repository=flux-at-home \
     --branch=main \
     --path=./clusters/cluster0 \
     --personal
   

   Where GitHub is supported natively (via GitHub Apps). Solutions like Codeberg needs a more direct method.
   

   flux bootstrap git \
     --url=ssh://[e-mail]/3sky/flux-at-home.git \
     --branch=main   \
     --path=./clusters/cluster0   \
     --private-key-file=/home/kuba/.ssh/id_ed25519_git
   

6. Get ArgoCD password if needed
   

   kubectl --namespace argocd \
     get secret argocd-initial-admin-secret \
     -o json | jq -r '.data.password' | base64 -d
   

Summary

With this set of commands, I'm able to set fresh clusters, with Hetzner or whatever in literally 7 minutes. Starting from installing k3s to having applications exposed via Cloudflare tunnel, or Tailscale tailnet. To be honest I'm really satisfied with the result of this exercise. Besides a long-running cluster for my self-hosted apps. I can quite easily, fast, and with low costs test new versions of ArgoCD, or smb-driver operator, without harming my current setup! And that is great, to be honest. Especially for self-hosting, where it's good to have apps that are working in let's say production mode, even if there is only one user.