Introduction
As a Kubernetes user, I chose to use the default tooling available, named Kustomize, to manage my manifests and all modifications I need to do over them for each environment I want to deploy. For that, Kustomize is a powerful solution based on concepts of inheritance (resources) and composition (components).
But one thing I miss the most is the ability to distribute manifests to other people, letting them inherit or compose them as they want. This is one subject where helm is ahead of Kustomize now.
However, Kustomize provides a mechanism to use remote elements, for both resources and components using git under the hood. This solution is perfect, because we can use any git repository as a yaml registry for our reousrces and components 🔥.
But, the downside is FluxCD does not supports so well, for a lot of good reasons (especially when your repositories are private)… In this article, we will see how to bypass this limitation and use remote resources and components, the FluxCD way!
Cluster & FluxCD configuration
In this article, I won't detail so much the cluster & flux configuration… because there is already plenty of documentation and blogposts about that!
For this article, I have set up a cluster using Google Kubernetes Engine (aka GKE) using this documentation and installed FluxCD connected to a GitLab repository with the following command (extracted from the official documentation).
$ flux bootstrap gitlab \
--owner=davinkevin.fr/articles/flux-and-kustomize-remote-base \
--repository=clusters \
--branch=gke \
--path=fluxcd
NOTE: I don't like to use default branch, so I chose to create a branch for this cluster, so if I have another cluster, I just need another branch 😉. This is some kind of GitLab Flow for environment.
With this, I have a complete Kubernetes Cluster ready for GitOps, using clusters repository as source-of-truth. Every modification made in this repository will be automatically deployed by FluxCD 🤩.
bases repository
I have set up another git repository dedicated to resources and components. It hosts what we usually call base, because it contains the common core of an application and also many components to enable some extra features (at infrastructure or application level).
This repository is available in GitLab and is agnostic of my deployment environment. Again, I only host here the bare definition of my applications, we can compare this to an helm registry. Because it is managed in git, I can use branch & tags for management 🚀.
NOTE: Usually, this kind of yaml repository is only populated by CI from other projects automatically.
PodInfo base publication
I have chosen the well known podinfo application as an example, and I have developed the base and some components associated to it.
.
├── README.md
└── podinfo
└── base
├── components
│ ├── hpa
│ │ ├── hpa.yaml
│ │ └── kustomization.yaml
│ ├── ingress
│ │ ├── ingress.yaml
│ │ └── kustomization.yaml
│ └── redis
│ ├── kustomization.yaml
│ ├── redis.conf
│ └── redis.yaml
├── kustomization.yaml
└── podinfo.yaml
Core PodInfo
This part is the common part of all potential deployment. It contains:
- The
podinfoKubernetes Service - The
podinfoKubernetes Deployment
And nothing else… if you need to deploy any other part of the app, you have to use components 👇.
Components
In this components, we can find all optional part of our application. As an end-user, I need to declare those I want to use, because by default, they are not applied at all in the common core.
Here, we have:
-
hpawith a default preconfiguredHorizontal-Pod-Autoscalerif you want the application to scale up and down automatically -
ingressdefinition to expose the application to the outside world (see official documentation) -
rediscomponent to enable cache on the PodInfo application with default configuration.
It's interesting to note components can be of any sort, hpa and ingress are focused on infrastructure when redis is a feature of the application. Without this redis component, the application is able to work, just without any caching system. This redis component is here to deploy a redis server (of course 😅) but also to enable caching at podinfo level too.
NOTE: If you want a better description of this component feature, I advise you to discover this article from the official documentation
So, we have a complete, environment agnostic resource and components committed to our base repository. Now, It is time to use it!
Publish PodInfo to dev
Back to our cluster, I want to deploy this podinfo application, but of course I want to apply some extra customizations on it, like:
- Define
componentsI want to enable - Define a domain name for the
ingressdefinition - Define the required
tlsconfiguration to theingress - Define an alternative registry for the image
So, to do that, I need to have one (or multiple) kustomization.yaml to define this:
# <clusters-repo>/podinfo-dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo-dev
images:
- name: podinfo
newName: ghcr.io/stefanprodan/podinfo
resources:
- ../base
- ingress
components:
- ../base/components/redis
NOTE: The ingress modification is in its own file to keep this one simple enough.
But, what is this ../base folder? There is no base at all in cluster repository. And you are right… we will use FluxCD feature available in flux to include the base repository at the right location during reconciliation. To do that, we need to define multiple flux resources.
podinfo-base flux repository
First, we need a representation of our flux.GitRepository for bases, which is materialized in flux with:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: podinfo-base-dev
namespace: flux-system
spec:
interval: 5m
ref:
branch: podinfo
url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
secretRef:
name: flux-system
NOTE: Here, I have chosen to define spec.ref.branch: podinfo to follow every commit made to this branch. It is, for development, a continuous deployment system 🚀.
podinfo flux repository
Now, we need to create a flux.GitRepository for manifests in clusters repository and manifests from the bases repository created in the step before. To do that, we are going to use the include feature from FluxCD:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: podinfo-dev
namespace: flux-system
spec:
interval: 5m
include:
- repository:
name: podinfo-base-dev
fromPath: podinfo/base
toPath: base
ref:
branch: gke
url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters
secretRef:
name: flux-system
The important part is the following:
spec:
include:
- repository:
name: podinfo-base-dev # <1>
fromPath: podinfo/base # <2>
toPath: base # <3>
This means the folder podinfo/base (2) previously defined in the podinfo-base-dev flux.GitRepository (1) will be injected in the path base (3) of the current podinfo-dev flux.GitRepository during FluxCD reconciliation… making all resources and components available.
podinfo-dev flux Kustomization
This one is the standard flux.Kustomization you would use in any case. This one is just using the composite flux.GitRepository from the previous step as source (spec.sourceRef):
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: podinfo-dev
namespace: flux-system
spec:
suspend: false
interval: 10m0s
path: podinfo-dev # file path in the <clusters> repository
prune: true
sourceRef:
kind: GitRepository
namespace: flux-system
name: podinfo-dev
validation: client
If we deploy all those files (you can put them in the same one), you will see your application deployed by FluxCD with your provided custom configuration.
❯ kubectl get all -n podinfo-dev
NAME READY STATUS RESTARTS AGE
pod/redis-869ff7c78b-jjbbk 1/1 Running 0 103m
pod/podinfo-6dbf56d6d7-ctxxr 1/1 Running 0 103m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/podinfo ClusterIP 10.43.115.207 <none> 9898/TCP,9999/TCP 103m
service/redis ClusterIP 10.43.192.84 <none> 6379/TCP 103m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/redis 1/1 1 1 103m
deployment.apps/podinfo 1/1 1 1 103m
NAME DESIRED CURRENT READY AGE
replicaset.apps/redis-869ff7c78b 1 1 1 103m
replicaset.apps/podinfo-6dbf56d6d7 1 1 1 103m
Publish PodInfo to prod
Now, we want to use again the same base this time for a production environment.
Because we want something relatively stable, we chose to define spec.ref.tag to a specific tag in the bases repository (instead of the podinfo branch).
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: podinfo-base-prod
namespace: flux-system
spec:
interval: 5m
ref:
tag: podinfo-1.0
url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
secretRef:
name: flux-system
All others flux.Kustomization and flux.GitRepository are the same.
Because we are in production, we also want to enable different modules, here the HorizontalPodAutoscaler. So we have a different kustomization.yaml. Thanks to the full control we have over the kustomization.yaml per environment, we can modify this only in production 😍.
Once everything is published and FluxCD reconciliation is OK, we have:
❯ kubectl get all -n podinfo-prod
NAME READY STATUS RESTARTS AGE
pod/redis-869ff7c78b-lnll5 1/1 Running 0 113m
pod/podinfo-6dbf56d6d7-k8rjl 1/1 Running 0 113m
pod/podinfo-6dbf56d6d7-qldr8 1/1 Running 0 113m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/podinfo ClusterIP 10.43.109.213 <none> 9898/TCP,9999/TCP 113m
service/redis ClusterIP 10.43.252.54 <none> 6379/TCP 113m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/redis 1/1 1 1 113m
deployment.apps/podinfo 2/2 2 2 113m
NAME DESIRED CURRENT READY AGE
replicaset.apps/redis-869ff7c78b 1 1 1 113m
replicaset.apps/podinfo-6dbf56d6d7 2 2 2 113m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/podinfo Deployment/podinfo 4%/99% 2 4 2 113m
Conclusion
It was tough, with many yaml but we have achieved our original goal. Being able to publish bare manifests in a central place and allow other team / people to consume them instantaneously with all features available in Kustomize.
We are then able to enjoy all the feature of a GitOps model with a dedicated (yaml) registry, like helm already have 😇.
There is some follow-up improvements to come, like OCI registries (announced in FluxCD 0.32), to replace flux.GitRepository. We still need this issue to be solved to be able to use it with all features provided by Kustomize.
In a more distant future, some other solution like porch could be a good candidate to simplify again our deployment strategy… but it is currently in alpha stage.
I hope you liked this article, you can find all resources used in GitLab, and you can reproduce it in your own cluster! If you have any questions, don't hesitate to comment or ping me on twitter @davinkevin
Top comments (1)
Can you please also show how HelmRepository works and how can HelmRepo be integrated via FluxCD