DEV Community: Nigel Brown

Using Frp to Publicly Expose Services in a Local Kubernetes Cluster

Nigel Brown — Thu, 27 Jun 2024 09:04:15 +0000

I recently had cause to test out cert-manager using the Kubernetes Gateway API, but wanted to do this using a local cluster on my laptop, based on kind. I wanted cert-manager to automatically acquire an X.509 certificate on behalf of an application service running in the cluster, using the ACME protocol. This isn’t straightforward to achieve, as:

The cluster, hosting cert-manager as an ACME client, runs on a laptop on a private network behind a router, using NAT.
The certificate authority (CA) issuing the X.509 certificate, which provides the ACME server component, needs to present a domain validation challenge to cert-manager, from the internet.

Essentially, the problem is that the cluster is on a private network, but needs to be addressable via a registered domain name, from the internet. How best to achieve this?

Options

There are probably a million and one ways to achieve this, all with varying degrees of complexity. We could use SSH reverse tunnelling, or a commercial offering like Cloudflare Tunnel, or one of the myriad of open source tunnelling solutions available. I chose to use frp, “a fast reverse proxy that allows you to expose a local server located behind a NAT or firewall to the internet”. With 82,000 GitHub stars, it seems popular!

Frp

Frp uses a client/server model to establish a connection at either end of a tunnel; the server component at the end that is publicly exposed to the internet, and the client on the private network behind the router. Client traffic arriving at the frp server end is routed to the frp client through the tunnel, according to the configuration provided for the client and server components.

Frp Server Configuration

For my scenario, I chose to host the frp server on a DigitalOcean droplet, with a domain name set to resolve to the droplet’s public IP address. This is the domain name that will appear in the certificate’s Subject Alternative Name (SAN). The configuration file for the server looks like this:

# Server configuration file -> /home/frps/frps.toml

# Bind address and port for frp server and client communication
bindAddr = "0.0.0.0"
bindPort = 7000

# Token for authenticating with client
auth.token = "CH6JuHAJFDNoieah"

# Configuration for frp server dashboard (optional)
webServer.addr = "0.0.0.0"
webServer.port = 7500
webServer.user = "admin"
webServer.password = "NGe1EFQ7w0q0smJm"

# Ports for virtual hosts (applications running in Kubernetes)
vhostHTTPPort = 80
vhostHTTPSPort = 443

In this simple scenario, the configuration provides:

the interfaces and port number through which the frp client interacts with the server
a token used by the client and the server for authenticating with each other
access details for the server dashboard that shows active connections
the ports the server will listen on for virtual host traffic (80 for HTTP and 443 for HTTPS)

Because this setup is temporary, and to make things relatively easy, the frp server can be run using a container rather than installing the binary to the host:

docker run -d --restart always --name frps \
    -p 7000:7000 \
    -p 7500:7500 \
    -p 80:80 \
    -p 443:443 \
    -v /home/frps/frps.toml:/etc/frps.toml \
    ghcr.io/fatedier/frps:v0.58.1 -c /etc/frps.toml

A container image for the frp server is provided by the maintainer of frp, as a GitHub package. The Dockerfile from which the image is built, can also be found in the repo.

Kind Cluster

Before discussing the client setup, let’s just describe how the cluster is configured on the private network.

$ docker network inspect -f "{{json .IPAM.Config}}" kind | jq '.[0]'
{
  "Subnet": "172.18.0.0/16",
  "Gateway": "172.18.0.1"
}

Kind uses containers as Kubernetes nodes, which communicate using a (virtual) Docker network provisioned for the purpose, called ‘kind’. In this scenario, it uses the local subnet 172.18.0.0/16. Let’s keep this in the forefront of our minds for a moment, but turn to what’s running in the cluster.

$ kubectl -n envoy-gateway-system get pods,deployments,services
NAME                                            READY   STATUS    RESTARTS   AGE
pod/envoy-default-gw-3d45476e-b5474cb59-cdjps   2/2     Running   0          73m
pod/envoy-gateway-7f58b69497-xxjw5              1/1     Running   0          16h

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/envoy-default-gw-3d45476e   1/1     1            1           73m
deployment.apps/envoy-gateway               1/1     1            1           16h

NAME                                    TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)               AGE
service/envoy-default-gw-3d45476e       LoadBalancer   10.96.160.26   172.18.0.6    80:30610/TCP          73m
service/envoy-gateway                   ClusterIP      10.96.19.152   <none>        18000/TCP,18001/TCP   16h
service/envoy-gateway-metrics-service   ClusterIP      10.96.89.244   <none>        19001/TCP             16h

The target application is running in the default namespace in the cluster, and is exposed using the Envoy Gateway, acting as a gateway controller for Gateway API objects. The Kubernetes Gateway API supersedes the Ingress API. The Envoy Gateway provisions a deployment of Envoy, which proxies HTTP/S requests for the application, using a Service of type LoadbLanacer. The Cloud Provider for Kind is also used to emulate the provisioning of a ‘cloud’ load balancer, which exposes the application beyond the cluster boundary with an IP address. The IP address is 172.18.0.6, and is on the subnet associated with the ‘kind’ network. Remember, this IP address is still inaccessible from the internet, because it’s on the private network, behind the router.

If the frp client can route traffic received from the frp server, for the domain name, to this IP address on the ‘kind’ network, it should be possible to use cert-manager to request an X.509 certificate using the ACME protocol. Further, it’ll enable anonymous, internet-facing clients to consume applications running in the cluster on the private network, too.

Frp Client Configuration

Just as the frp server running on the droplet needs a configuration file, so does the frp client running on the laptop.

# Client configuration file -> /home/frpc/frpc.toml

# Address of the frp server (taken from the environment),
# along with its port
serverAddr = "{{ .Envs.FRP_SERVER_ADDR }}"
serverPort = 7000

# Token for authenticating with server
auth.token = "CH6JuHAJFDNoieah"

# Proxy definition for 'https' traffic, with the destination
# IP address taken from the environment
[[proxies]]
name = "https"
type = "https"
localIP = "{{ .Envs.FRP_PROXY_LOCAL_IP }}"
localPort = 443
customDomains = ["myhost.xyz"]

# Proxy definition for 'http' traffic, with the destination
# IP address taken from the environment
[[proxies]]
name = "http"
type = "http"
localIP = "{{ .Envs.FRP_PROXY_LOCAL_IP }}"
localPort = 80
customDomains = ["myhost.xyz"]

The configuration file content is reasonably self-explanatory, but there are a couple of things to point out:

For flexibility, the IP address of the frp server is configured using an environment variable rather than being hard-coded.
The file contains proxy definitions for both, HTTP and HTTPS traffic, for the domain myhost.xyz. The destination IP address for this proxied traffic is also taken from the environment (which evaluates to 172.18.0.6 in this particular scenario).

As the frp client is getting some of its configuration from the environment, the relevant environment variables need to be set. In this case, the frp server is running on a DigitalOcean droplet, which requires doctl in order to interact with the DigitalOcean API:

export FRP_SERVER_ADDR="$(doctl compute droplet list --tag-name kind-lab --format PublicIPv4 --no-header)"

We know the local target IP address already, but this may be different in subsequent test scenarios, so it’s best to query the cluster to retrieve the IP address and set the variable accordingly:

export FRP_PROXY_LOCAL_IP="$(kubectl get gtw gw -o yaml | yq '.status.addresses.[] | select(.type == "IPAddress") | .value')"

Just as the frp server was deployed as a container, so too can the frp client (Docker image for the client is here, and the Dockerfile here):

docker run -d --restart always --name frpc \
    --network kind \
    -p 7000:7000 \
    -v /home/frpc/frpc.toml:/etc/frpc.toml \
    -e FRP_SERVER_ADDR \
    -e FRP_PROXY_LOCAL_IP \
    ghcr.io/fatedier/frpc:v0.58.1 -c /etc/frpc.toml

The frpc client container is attached to the ‘kind’ network, so that the traffic that it proxies can be routed to the IP address defined in the FRP_PROXY_LOCAL_IP variable; 172.18.0.6. Once deployed, the frp server and client establish a tunnel that proxies HTTP/S requests to the exposed Service in the cluster.

This enables cert-manager to initiate certificate requests for suitably configured Gateway API objects using the ACME protocol. But, it also allows a CA (for example, Let’s Encrypt), to challenge cert-manager with an HTTP-01 or DNS-01 challenge for proof of domain control. In turn, cert-manager is able to respond to the challenge, and then establish a Kubernetes secret with the TLS artifacts (X.509 certificate and private key). The secret can then be used to establish secure TLS-encrypted communication between clients and the target application in the cluster on the private network.

Conclusion

Not everyone wants to spin up a cloud-provided Kubernetes cluster for testing purposes; it can get expensive. Local development cluster tools, such as kind, are designed for just such requirements. But, you’ll always need to satisfy that one scenario where you need to access the local cluster from the internet, and sometimes with an addressable domain name. Frp is just one solution available, but it’s a comprehensive solution with a lot more features that haven’t been discussed here. Just to be clear, you should read up on securing the connection between the client and server, to ensure no eavesdropping on the traffic flow.

Bootstrapping Argo CD

Nigel Brown — Mon, 19 Dec 2022 15:18:55 +0000

This article follows on from an introductory article that discussed the chicken or the egg paradox when it comes to bootstrapping GitOps agents into Kubernetes clusters. The article discusses how this relates to one of the popular GitOps solutions commonly used to automate application deployments to Kubernetes, Argo CD. Argo CD is one of the prominent GitOps solutions that pioneers this evolving approach to application delivery, and is part of a family of tools that co-exist under the Argo umbrella. Collectively, the Argo toolset is a recently graduated project of the Cloud Native Computing Foundation (CNCF).

Installing Argo CD

When it comes to bootstrapping Argo CD into a cluster to act as a GitOps agent, there are detailed instructions for installation according to the preferred setup (i.e. multi-tenant, high availability and so on). Kubernetes configuration files (including its custom resource definitions) are maintained at Argo CD's GitHub repo, and these can be applied directly with kubectl, or through a Kustomization definition. There's a Helm chart, too, for anyone who prefers to use the chart packaging metaphor for applications. Using one of these techniques gets Argo CD running in a target cluster.

So, we know how to install Argo CD, but, what's less clear is how or if Argo CD can manage itself according to GitOps principles. The project's documentation is a little opaque in this regard. But, if you look hard enough, you'll find a reference to managing Argo CD with Argo CD. It suggests using a Kustomization to define how Argo CD is configured to run in a Kubernetes cluster, with the config stored in a Git repo, which is monitored by the installed instance of Argo CD. There's even a live online example of Argo CD managing itself, alongside a bunch of other applications.

A deployed agent, with its configuration held in versioned storage, configured to fetch and reconcile that same configuration, fulfils the GitOps principles. What's missing from this tantalising glimpse of self-management, however, is how this works in practice. Let's see if we can unpick this.

Applications in Argo CD

GitOps agents work with instances of their own custom resources, that enable them to manage applications according to the GitOps principles. For Argo CD, the main custom resource it extends the Kubernetes API with, is the 'Application'. In defining an Application object, and having it applied to the cluster, we provide Argo CD with the information it needs to manage that application. Some of the information defined in the object is mandatory, and some is optional. For example, the source location of the application's remote Git or Helm repository that contains its configuration, and the coordinates of the target cluster where the application is to be deployed, are mandatory. Here's an example Application definition that could be used by Argo CD for managing itself;

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: argocd-bootstrap/argocd
    repoURL: https://github.com/nbrownuk/gitops-bootstrap.git
    targetRevision: HEAD
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true

This object definition would need to be applied to a cluster running Argo CD, which would then be enacted upon by Argo CD's application controller, resulting in periodic fetches of the configuration from the source repo. Any changes made to Argo CD's configuration in the repo (for example, a change in app version), would then be applied to the cluster, which would result in a rolling update of Argo CD. Hence, Argo CD is managing itself.

But, let's stop to think for a moment; isn't the definition of the Application part of the overall configuration? We've manually applied the Application object definition to the cluster, but it's not stored in our 'source of truth' in the Git repo. If we had to re-create the cluster, would we remember the Git reference to use for the targetRevision; was it the HEAD, a branch, a specific commit? What was the sync policy for imperative changes made inside the cluster? This is one of the typical scenarios that inspired the GitOps philosophy - to take away the ambiguity of configuration, by using declarative configuration residing in versioned storage. This includes the configuration that drives the actions of the GitOps agent itself.

App of Apps Pattern

Using a pattern called 'app of apps', Argo CD enables Argo CD to manage itself declaratively. The pattern is more general purpose in utility, in that a single 'root' Application definition serves as a pointer to a Git directory path containing one or more further Application definitions. Applying the 'root' definition results in the creation of all the subordinate Application objects in the cluster, which are then acted upon by Argo CD's application controller. One of the subordinate Application objects might be one that defines Argo CD itself.

.
├── managedapps
│  ├── argocd
│  │  ├── kustomization.yaml
│  │  └── namespace.yaml
│  └── podinfo
│     └── kustomization.yaml
├── rootapp
│  ├── kustomization.yaml
│  └── rootapp.yaml
└── subapps
   ├── argocd.yaml
   ├── kustomization.yaml
   └── podinfo.yaml

Here, the 'rootapp' Application definition (contained in rootapp.yaml) references the path, 'subapps', and each of the Application definitions at this location (argocd.yaml, podinfo.yaml) reference the corresponding sub-directories under 'managedapps'.

If there are just a few applications, this pattern works well. But, if there are a multitude of applications, then maintenance of the plethora of Application definitions can become burdensome. As an enhancement to this pattern, the Argo CD project introduced the ApplicationSet custom resource and controller. An ApplicationSet object defines one or more 'generators', which generate key/value pairs called parameters. The definition will also contain a 'template', which the ApplicationSet controller will render with the corresponding parameters. In this way, a single ApplicationSet object can automatically spawn numerous Application resources, saving on the administrative overhead of maintaining lots of Application definitions by hand.

An ApplicationSet definition can work equally as well as the app of apps pattern for managing Argo CD itself. Maybe you'd want to lean towards the ApplicationSet solution, as it's seen as an evolution of the app of apps pattern.

Argo CD Autopilot

One of the problems with setting up the 'app of apps' or ApplicationSet configuration for Argo CD self-management, is that there are a number of manual steps involved:

Establish Git repo with app configuration
Install Argo CD
Create secret for trusted access to Git repo
Apply 'root' Application or ApplicationSet object definition

None of these steps are particularly onerous, but where there are manual steps there is always the opportunity to introduce error and ambiguity. The more automated everything is, the better chance of a successful outcome. This is where Argo CD Autopilot adds some value.

Argo CD Autopilot is a command line tool for bootstrapping Argo CD and managed applications into Kubernetes clusters. It originated from engineers at Codefresh as an ancillary project to Argo CD. And, it seeks to take away some of the complexity of bootstrapping an Argo CD GitOps implementation, as well as to provide a sane repo structure for the configuration of the managed applications. It still uses ApplicationSet and Application objects under the covers, but hides the complexity behind its CLI. The price you pay for this simplified user experience, is an opinionated approach to the repo structure, and the way you might approach your deployment workflows.

In terms of bootstrapping Argo CD itself, however, it's as simple as issuing a command:

$ argocd-autopilot repo bootstrap \
    --repo https://github.com/nbrownuk/argocd-bootstrap \
    --git-token "$(< ./github-token)"

The net result is the creation of a Git repo (hence, the need to supply a personal access token with 'repo' privileges) containing a prescribed directory structure, with all of the necessary configuration elements to enable self-management of Argo CD. It also results in the deployment of Argo CD to the cluster addressed by your current context. From here you can use the CLI to create projects and applications for management by Argo CD, with the necessary configuration automatically created in the Git repo.

.
├── apps
│  ├── podinfo
│  │  ├── base
│  │  │  └── kustomization.yaml
│  │  └── overlays
│  │     └── prod
│  │        ├── config.json
│  │        └── kustomization.yaml
│  └── README.md
├── bootstrap
│  ├── argo-cd
│  │  └── kustomization.yaml
│  ├── argo-cd.yaml
│  ├── cluster-resources
│  │  ├── in-cluster
│  │  │  ├── argocd-ns.yaml
│  │  │  └── README.md
│  │  └── in-cluster.json
│  ├── cluster-resources.yaml
│  └── root.yaml
└── projects
   ├── prod.yaml
   └── README.md

The directory structure above is an example of a configured Git repo when using Argo CD Autopilot.

It's early days for the Argo CD Autopilot project, with some features still missing. For example, its CLI doesn't yet support the use of Helm charts as the embodiment of application configuration, although it's possible to work around this using the limited support provided by Kustomize. Argo CD Autopilot isn't necessarily for everybody, especially given its opinionated approach. But, if you're looking for a convenient method for bootstrapping Argo CD, it does the job perfectly. It even allows you to recover from the loss of a cluster using the CLI, by bootstrapping with reference to the original Git repo as the source of truth.

Bootstrapping Flux

Nigel Brown — Mon, 12 Dec 2022 12:13:28 +0000

Following on from an introductory article on the merits of bootstrapping GitOps agents into Kubernetes clusters, this article looks into how this can be achieved with Flux. Flux is a collective name to describe several discrete Kubernetes controllers (also known as the GitOps Toolkit), that each perform a separate, specific GitOps-oriented function. Flux is a graduated Cloud Native Computing Foundation (CNCF) project.

Installing Flux

Bootstrapping involves installation, and Flux can be installed into a Kubernetes cluster in a variety of ways. A set of YAML manifests are maintained in the project's GitHub repo:

$ kustomize build https://github.com/fluxcd/flux2/manifests/install?ref=v0.37.0 | \
    kubectl apply -f -

But, if Helm is your thing, then there's a packaged Helm chart maintained for installing Flux, too. There's even a command line tool, unsurprisingly called Flux, that can also be used to get Flux up and running in a cluster:

$ flux install

Performing any of these actions would provide us with a working Flux deployment in a cluster, consisting of the set of controllers that govern its GitOps actions.

$ kubectl -n flux-system get po
NAME                                          READY   STATUS    RESTARTS   AGE
helm-controller-7b85c84687-dzdbr              1/1     Running   0          81s
image-automation-controller-f45c4b86b-sqqlf   1/1     Running   0          81s
image-reflector-controller-59c894c647-lzb76   1/1     Running   0          80s
kustomize-controller-d88d76876-dqw2m          1/1     Running   0          80s
notification-controller-55df97fbb9-jbrqb      1/1     Running   0          80s
source-controller-5bb5c7b9bd-dsd6w            1/1     Running   0          80s

But, it doesn't provide us with a self-managed setup, using immutable, declarative configuration stored under version control.

It turns out that the Flux project considers that bootstrapping, self-management and recovery, are primary concerns of GitOps deployments based on Flux. And, as such, it provides an installation mechanism that speaks to each of these concerns, which is described prominently in the documentation. So, how does Flux go about bootstrapping?

Bootstrapping with the Flux CLI

The Flux CLI is a like a Swiss army knife, in that it provides a large number of useful, utilitarian features. As well as using it to install Flux's controllers into a cluster, you can use it to create the manifests for the custom resource objects that the controllers use to perform GitOps actions, for example. And a whole lot more, besides.

Perhaps one of its most important functions, however, is its ability to perform a bootstrap of a GitOps environment, using a single command:

$ flux bootstrap github \
    --repository flux-bootstrap \
    --owner nbrownuk \
    --personal true

This is a fairly vanilla example of the use of the flux bootstrap command, but there are a ton of other flags available to nuance the outcome. The command performs a lot of work behind the scenes; it:

creates a remote Git repo (GitHub, GitLab, AWS CodeCommit etc.)
clones the repo
generates the manifests for each of Flux's controllers
commits the manifests and pushes them to the remote repo
installs the controllers into the cluster using the generated manifests
creates an ssh key pair for trusted communication between Flux's source controller and the remote repo (if you don't bring your own)
creates a secret embodying the private key, and adds the public key to the remote Git repo as a 'deploy key'
generates manifests (based on Flux's custom resources) for the purposes of syncing desired state from the remote repo to the cluster
commits sync manifests and pushes them to the remote repo
applies the sync manifests to cluster, and waits for reconciliation

That's a lot of heavy lifting done on our behalf, and results in a repo structure that contains all that's needed for Flux to manage itself:

.
└── flux-system
   ├── gotk-components.yaml
   ├── gotk-sync.yaml
   └── kustomization.yaml

The 'gotk-components.yaml' manifest contains the custom resource definitions, and the Kubernetes object definitions for each of the controllers. And, the 'gotk-sync.yaml' manifest contains custom resource object definitions, providing Flux with the location of the repo, and the path within the repo that contains the configuration that is to be applied to the cluster. That's all Flux needs to manage itself. And, should you be unfortunate enough to lose you cluster, the bootstrap command can be run again to provision Flux to a new cluster, using the configuration in the repo.

Updating Flux

As I write, even though Flux has gained graduate status with the CNCF, and has acquired a sizeable community of adopters, the project is still working towards General Availability. This means that Flux gets updated on a regular basis, which begs the question, how does Flux get updated after bootstrap?

Firstly, with each release, a new version of the Flux CLI is provided¹. And so, to achieve an update to Flux after an initial bootstrap, all that's required is a re-run of the bootstrap command with the updated version of the CLI. Execution of the command results in an update to the controller definitions in the 'gotk-components.yaml' file in the remote Git repo, which is subsequently fetched and applied to the cluster by Flux. And this results in a rolling update of each controller running in the cluster.

It may be more preferable to gate this change in the source Git repo using a pull request. The revised manifest for the new controller versions could be generated in advance using the following command:

$ flux install --export > ./gotk-components.yaml

If you're using GitHub as the Git host provider, this could also be automated using GitHub Actions, which could easily be adapted for use in other CI/CD systems.

Bootstrapping and updating in this manner is all well and good, if we're happy to use a command line interface to achieve this. Sometimes, DevOps teams prefer to bundle the installation of core services into the infrastructure layer. How do we manage this for Flux?

Bootstrapping with Terraform

When provisioning an entire environment for running cloud-native applications, an interesting question arises. What delineates infrastructure from applications? This line can get quite blurred, and different people will have different answers as to where to draw the line. But, the question is an important one, because different tools are generally used for managing the different layers. Terraform is pretty ubiquitous when it comes to managing infrastructure, and as we know, Flux is used to manage application deployments to Kubernetes.

Flux sits on the boundary; you can't manage applications with Flux until it's provisioned, and you can't provision Flux until the Kubernetes cluster is provisioned. Many will choose to drop Flux into the infrastructure layer, and have Terraform handle its bootstrapping, after which, Flux is able to manage itself. Anticipating the need, and recognising the importance of catering for this scenario, the Flux project has provided a Flux Terraform provider for provisioning Flux.

Up to at least v0.21.0, the Flux provider consisted of two data sources; one for generating the YAML for the CRDs and deployment configuration, and one for generating the YAML for the sync activity. The code used to achieve this is the same code used in the CLI for bootstrapping Flux. But, generation of the Kubernetes configuration doesn't get Flux bootstrapped, and the user is required to make use of other Terraform providers to store the configuration in a Git source, and then to get it applied to a cluster.

The official Terraform Kubernetes provider, which is needed to apply the generated configuration to the cluster, comes with some drawbacks. And, further, the notion that the Flux provider doesn't itself handle the bootstrapping, might be considered a sub-optimal user experience. This has prompted the Flux project to overhaul the Flux provider, so that it performs the bootstrap process in its entirety. At the time of writing, a new bootstrap resource is in development for the Flux provider. With this important revision, the initial terraform apply will:

generate the necessary configuration,
push the configuration to the Git repo,
apply the configuration to the cluster, and wait for reconciliation

Subsequently, Terraform is responsible for managing the configuration in the repo, and Flux is responible for fetching and applying the configuration from the repo. This provides a clear delineation of responsibility, and a much improved user experience.

For the sake of clarity, the version of the CLI reflects the Flux release version, whereas the individual controllers have different, independent release versions. For example, in Flux v0.37.0, the source controller version is v0.32.1, whilst the helm controller is v0.27.0. ↩

Bootstrapping GitOps Agents

Nigel Brown — Sun, 04 Dec 2022 11:47:54 +0000

Since the term was originally coined back in 2017 by Alexis Richardson (CEO of Weaveworks), the GitOps philosophy has struggled to find a coherent, consensual definition. This fostered some ambiguity, and occasionally some contradiction. For example, push-based deployments were once categorised as a valid approach to GitOps deployments, but now they're not. There was going to be a common GitOps engine for the good of the community, and then there wasn't. Flux was in the vanguard of software tools that exemplified GitOps, and then it needed to be rewritten from the ground up. And, so on.

It's encouraging, then, that the last 12 months or so have seen a gradual maturing of the concept, with a common understanding of what we mean by the term 'GitOps'. And the community has started working together across competing technologies, all under the umbrella of the Cloud Native Computing Foundation (CNCF). Some of the tooling has even acquired 'graduated' project status with the CNCF. All is good in GitOps land.

As the work to define standards has unfolded, and the development of software tools to meet those standards has ensued, one aspect of GitOps has consistently bugged me. It's an implementation detail that doesn't seem to get a lot of attention.

GitOps Agents

First, let's briefly outline what a GitOps agent is in the context of a Kubernetes cluster. A GitOps agent is a software application that (at least in part) implements the Kubernetes controller pattern, and is responsible for reconciling the declared 'desired state' with the actual cluster state. That is, the Kubernetes configuration for an application stored in a version control system (VCS), is periodically fetched and applied to the cluster, resulting in automated deployments. Change is initiated through the VCS, and explicitly not through direct, imperative change in the cluster. This allows application deployments to be managed using the inherent control features of the hosting VCS (i.e. pull or merge requests).

Chicken or the Egg

So, GitOps agents allow us to automate application installs and updates in a Kubernetes cluster, according to the GitOps principles. But, aren't the agents themselves, software applications? They might be controllers, but they're still just software applications, requiring; installation, update, reconfiguration, recovery, and so on. This begs the question, how do GitOps agents get deployed to a cluster? Can they be deployed to a cluster according to GitOps principles, like the apps they manage? Can they be used to manage themselves? Seemingly, this is an example of the classic chicken or the egg paradox!

If we have to manually install or update a GitOps agent, then it's deployment is open to all of the issues that the GitOps principles seek to resolve; ambiguous state, configuration drift, unsolicited change, and much more. This effectively makes the state of the whole system as frail as if there were no GitOps agent at all. To that end, it's crucial that GitOps solutions address this problem head on, and provide a means for bootstrapping their agents, whilst allowing for their ongoing management, in line with the GitOps principles that underpin the whole approach.

This is the first article in a brief series looking into the different approaches taken to the bootstrapping conundrum.

Up Next

The two leading projects in the GitOps domain (arguably), Flux and ArgoCD, go about bootstrapping in different ways. First, we'll discuss how the Flux project goes about bootstrapping its numerous controllers, and then we'll lift the lid on ArgoCD, to see how it gets things done.