DEV Community: Apurv Kiri

Kubernetes GitOps with FluxCD - Part 3 - Automated Image Updates

Apurv Kiri — Tue, 25 Mar 2025 10:00:00 +0000

In our previous post, we explored how to manage secrets in FluxCD with SOPS. Building on our GitOps foundation from part 1 and secure secrets handling from part 2, this article will focus on image update automation - a powerful FluxCD feature that automatically updates your deployments when new container images are available, maintaining GitOps principles while eliminating manual image version updates.

1. Setup image repository

To demonstrate image automation, we'll create a sample application using Quarkus, a Kubernetes-native Java framework.

First, let's create a GitHub repository.

Let's implement a simple REST controller for our sample application.

package com.kiriyard;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/")
public class K8SampleApplication {

    private static final int VERSION = 1;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String index() {
        return "Greetings From K8S App : Version %d".formatted(VERSION);
    }
}

Now, let's create a GitHub Action workflow that builds and publishes this image to the GitHub Container Registry.

name: Build K8S App Image

on:
  push:
    branches:
      - main

jobs:
  build-image:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      checks: write
      id-token: write
      packages: write
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set up JDK 21
        uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
          cache: 'maven'

      - name: Generate build ID
        id: prep
        run: |
          branch=${GITHUB_REF##*/}
          sha=${GITHUB_SHA::8}
          ts=$(date +%s)
          echo "BUILD_ID=${branch}-${sha}-${ts}" >> $GITHUB_OUTPUT

      - name: Build and push Docker image
        run: |
          ./mvnw install \
            -Dquarkus.container-image.build=true \
            -Dquarkus.container-image.registry=ghcr.io \
            -Dquarkus.container-image.group="" \
            -Dquarkus.container-image.name=${{ github.repository }} \
            -Dquarkus.container-image.tag=${{ steps.prep.outputs.BUILD_ID }} \
            -Dquarkus.container-image.push=true

After pushing our changes, we can verify the image has been created in the GitHub Container Registry.

We'll note the full image path for future reference: ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397

2. Setup private repository pull secrets

Since we're using a private container registry, we need to configure GitHub authentication credentials as Kubernetes pull secrets for our deployment.

We'll generate a Personal Access Token (PAT) and configure a Kubernetes secret using SOPS, following the encryption approach we established in our previous post.

First, let's create the secret YAML manifest:

kubectl create secret docker-registry github-registry-secret \
  --docker-server=ghcr.io \
  --docker-username=** \
  --docker-password=** \
  --namespace=default \
  --dry-run=client -o yaml > github-registry-secret.yaml

Now, let's encrypt this secret using SOPS:

sops encrypt --in-place github-registry-secret.yaml

After pushing these changes to our Git repository, we'll wait for FluxCD to perform reconciliation:

flux get kustomizations flux-system --watch

NAME            REVISION                SUSPENDED       READY   MESSAGE                              
flux-system     main@sha1:893edbfc      False           True    Applied revision: main@sha1:893edbfc
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   Unknown Reconciliation in progress
flux-system     main@sha1:893edbfc      False   True    Applied revision: main@sha1:c2636b99
flux-system     main@sha1:c2636b99      False   True    Applied revision: main@sha1:c2636b99

Let's verify that our secret has been successfully created in the cluster:

kubectl get secrets

NAME                     TYPE                             DATA   AGE
github-registry-secret   kubernetes.io/dockerconfigjson   1      65s
samplesecret             Opaque                           1      20h

3. Create deployment

Now let's create a Kubernetes deployment using our container image.

First, let's create apps/sample-app/deployment.yaml

apiVersion: apps/v1
metadata:
  name: sample-app
  namespace: default
kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      imagePullSecrets:
        - name: github-registry-secret
      containers:
      - name: sample-app
        image: ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "128Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "50m"

Next, create a Kustomization file to manage our resources at apps/sample-app/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

Finally, create a FluxCD Kustomization to deploy our application at cluster/default/sample-app.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sample-app
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/sample-app
  prune: true
  retryInterval: 2m
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: default
  timeout: 3m
  wait: true

After pushing these changes, let's monitor the FluxCD reconciliation process:

flux get kustomizations sample-app --watch

NAME            REVISION        SUSPENDED       READY   MESSAGE                    
sample-app                      False           Unknown Reconciliation in progress
sample-app              False   Unknown Reconciliation in progress
sample-app              False   True    Applied revision: main@sha1:939bb8b3
sample-app      main@sha1:939bb8b3      False   True    Applied revision: main@sha1:939bb8b3

Let's verify that our deployment has been created:

kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
nginx-deployment-854f6c9499-wxdjx   1/1     Running   1 (11m ago)   45h
sample-app-9477cc684-fs876          1/1     Running   0             48s

To validate that our application is working correctly, let's set up port forwarding and test the endpoint:

kubectl port-forward deployments/sample-app 8080:8080 
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Let's send a request to verify our application is responding:

curl http://localhost:8080

Greetings From K8S App : Version 1

Perfect! Our application is working as expected.

4. Setup FluxCD Image Automation

Now that we have our application deployed, we'll configure FluxCD's image automation capabilities. This requires three key components:

An ImageRepository resource that monitors our container registry for new images
An ImagePolicy resource that defines the criteria for selecting new images
An ImageUpdateAutomation resource that configures the automation process

We'll configure ImageRepository and ImagePolicy in same cluster/default/sample-app.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sample-app
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/sample-app
  prune: true
  retryInterval: 2m
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: default
  timeout: 3m
  wait: true
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: sample-app
  namespace: default
spec:
  image: ghcr.io/kiriapurv/k8s-sample-app
  interval: 5m
  secretRef:
    name: github-registry-secret
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: sample-app-main-policy
  namespace: default
spec:
  imageRepositoryRef:
    name: sample-app
  filterTags:
    pattern: '^main-[a-fA-F0-9]+-(?P<ts>.*)'
    extract: '$ts'
  policy:
    numerical:
      order: asc

Understanding the ImagePolicy Configuration

The ImagePolicy resource is crucial for defining how FluxCD selects the latest version of an image. Let's break down its key components:

Tag Filtering with filterTags

The filterTags section uses regular expressions to identify and extract meaningful information from image tags:

pattern: Defines a regex pattern to match against image tags. In our case, '^main-[a-fA-F0-9]+-(?P<ts>.*)' means:
^main-: The tag must start with "main-"
[a-fA-F0-9]+: Followed by one or more hexadecimal characters (our Git commit SHA)
(?P<ts>.*): Captures the remaining part of the tag as a named group "ts" (timestamp)
extract: Specifies which part of the pattern to extract for version comparison. The '$ts' refers to the named capture group "ts" from our pattern, which contains the Unix timestamp.

This extraction is essential because the FluxCD automation needs to know which part of the tag contains the version information to compare.

Version Selection with policy

The policy section defines how FluxCD determines which image version is the "latest":

numerical: Indicates we're dealing with numeric versioning (as opposed to semantic versioning)
order: asc: Specifies that higher numbers are considered newer (ascending order)

In our case, we're using Unix timestamps in our image tags, which naturally increase over time. By setting order: asc, we ensure that FluxCD selects the image with the most recent timestamp.

More details on sortable image tag is available in official documentation - https://fluxcd.io/flux/guides/sortable-image-tags/

Next, we need to annotate our deployment manifest to indicate which image policy should be applied. Let's update

We'll update apps/sample-app/deployment.yaml

apiVersion: apps/v1
metadata:
  name: sample-app
  namespace: default
kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      imagePullSecrets:
        - name: github-registry-secret
      containers:
      - name: sample-app
        image: ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397 # {"$imagepolicy": "default:sample-app-main-policy"}
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "128Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "50m"

Note that we've added a special comment annotation after the image value: # {"$imagepolicy": "default:sample-app-main-policy"}.

This tells FluxCD which policy to use when determining image updates.

Finally, we need to create an ImageUpdateAutomation resource to orchestrate the update process.

Let's create cluster/default/image-update-automation.yaml:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: default-image-update
  namespace: default
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}}'
    push:
      branch: main
  update:
    path: ./
    strategy: Setters

This configuration tells FluxCD to:

Check for updates every 30 minutes
Commit changes using the FluxCD bot identity
Use a message template that shows which images were updated
Push changes to the main branch
Use the "Setters" strategy to update image references throughout the repository

Let's push these changes and monitor FluxCD events to verify our automation is working:

flux events -n default --watch
LAST SEEN               TYPE    REASON                  OBJECT                                  MESSAGE                                             
31s (x24 over 2m38s)    Warning DependencyNotReady      ImagePolicy/sample-app-main-policy      referenced ImageRepository has not been scanned yet
30s (x2 over 30s)       Normal  Succeeded       ImagePolicy/sample-app-main-policy      Latest image tag for 'ghcr.io/kiriapurv/k8s-sample-app' resolved to main-3cbb3902-1740556397
30s     Normal  Succeeded       ImageRepository/sample-app      successful scan: found 1 tags

Let's confirm that our image resources are properly configured:

flux get -n default images all
NAME                            LAST SCAN                       SUSPENDED       READY   MESSAGE                       
imagerepository/sample-app      2025-02-26T14:08:00+05:30       False           True    successful scan: found 1 tags

NAME                                    LATEST IMAGE                                                    READY   MESSAGE                                                                                      
imagepolicy/sample-app-main-policy      ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397       True    Latest image tag for 'ghcr.io/kiriapurv/k8s-sample-app' resolved to main-3cbb3902-1740556397

5. Updating Image

Now let's test our automation by updating our application code and pushing the change.

We'll increment the version number in our controller class:

package com.kiriyard;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/")
public class K8SampleApplication {

-    private static final int VERSION = 1;
+    private static final int VERSION = 2;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String index() {
        return "Greetings From K8S App : Version %d".formatted(VERSION);
    }
}

After committing and pushing this change, GitHub Actions will build a new container image. Once the image is published, FluxCD should detect it, update our Git repository, and then apply the changes to our cluster.
Let's monitor the FluxCD events:

flux -n default events 

6s                      Normal  Succeeded               ImageUpdateAutomation/default-image-update      pushed commit '91fce99' to branch 'main'                                                                                                   
                                                                                                        ghcr.io/kiriapurv/k8s-sample-app:main-3cbb3902-1740556397 -> ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130

We can also verify the changes in our Git repository:

Now let's check if the deployment in our cluster has been updated:

kubectl -n default describe pod sample-app-649d676845-gcpwk 
Name:             sample-app-649d676845-gcpwk
Namespace:        default
Priority:         0
Service Account:  default
Node:             192.168.1.19/192.168.1.19
Start Time:       Wed, 26 Feb 2025 16:18:50 +0530
Labels:           app=sample-app
                  pod-template-hash=649d676845
Annotations:      <none>
Status:           Running
IP:               10.42.0.15
IPs:
  IP:           10.42.0.15
Controlled By:  ReplicaSet/sample-app-649d676845
Containers:
  sample-app:
    Container ID:   containerd://c505519d6dd3d013fa2be3275e2b5085df4486a4c3ebe75a50688b37e49eb7ff
    Image:          ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130
    Image ID:       ghcr.io/kiriapurv/k8s-sample-app@sha256:8447a6a128b96474ee620dfc692c15d7c22309f973663917191d2b87b8f3db72
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 26 Feb 2025 16:18:57 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     50m
      memory:  128Mi
    Requests:
      cpu:        50m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l2rxk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-l2rxk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  4m42s  default-scheduler  Successfully assigned default/sample-app-649d676845-gcpwk to 192.168.1.19
  Normal  Pulling    4m41s  kubelet            Pulling image "ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130"
  Normal  Pulled     4m35s  kubelet            Successfully pulled image "ghcr.io/kiriapurv/k8s-sample-app:main-29e96d2a-1740565130" in 5.3s (5.3s including waiting). Image size: 166698164 bytes.
  Normal  Created    4m35s  kubelet            Created container sample-app
  Normal  Started    4m35s  kubelet            Started container sample-app

As we can see, the pod is now running with the new image tag: main-29e96d2a-1740565130.

Finally, let's test our application to confirm the version has been updated:

curl http://localhost:8080

Greetings From K8S App : Version 2

Success! Our application has been automatically updated to version 2, demonstrating the complete GitOps-based image automation flow with FluxCD.

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

Helm chart automation
Notification and alerting configuration

Stay tuned for each of these topics.

References

Official FluxCD Documentation - https://fluxcd.io/flux/guides/image-update/
GitOps Working Group - https://opengitops.dev
Kubernetes Documentation - https://kubernetes.io/docs/
Quarkus Container Image Guide - https://quarkus.io/guides/container-image

Kubernetes GitOps with FluxCD - Part 2 - Secret Management using SOPS

Apurv Kiri — Thu, 20 Mar 2025 10:00:00 +0000

In Kubernetes-based GitOps workflows, securely managing sensitive information presents a unique challenge. While GitOps principles encourage storing all cluster configurations in Git repositories, committing plaintext secrets creates significant security risks. SOPS (Secrets OPerationS) offers an elegant solution to this problem when integrated with FluxCD.

In our previous post, we explored how to perform the initial setup of FluxCD in a Kubernetes cluster. Building on that foundation, we'll now address one of the most critical aspects of GitOps implementation: Secret Management.

SOPS, developed by Mozilla, enables encrypting specific values within YAML, JSON, and other configuration formats while keeping the file structure intact. When combined with FluxCD's native support for decryption, this allows teams to safely store encrypted secrets directly in their Git repositories alongside other infrastructure definitions. The secrets are only decrypted at runtime within the Kubernetes cluster, maintaining security while preserving the GitOps workflow.

This article explores how to implement a robust secret management strategy using SOPS and FluxCD. We'll cover the setup process, encryption workflows, integration with different key management systems, and best practices for maintaining secure GitOps operations in production environments.

1. Install SOPS and GPG

Your installation method may differ depending on your operating system. Below is the command for openSUSE Tumbleweed:

sudo zypper in sops gpg2

For other distributions:

Ubuntu/Debian: sudo apt install sops gnupg2
macOS: brew install sops gnupg
Arch Linux: sudo pacman -S sops gnupg

2. Generate Keypair

Next, we'll generate a GPG keypair that will be used for encryption and decryption:

export KEY_NAME="cluster.local"
export KEY_COMMENT="For Flux Secrets"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

This creates a 4096-bit RSA key with no expiration date. The output will look similar to:

gpg: directory '/home/apurv/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/apurv/.gnupg/openpgp-revocs.d/A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4.rev'

Now retrieve the key fingerprint, which we'll need for subsequent steps:

gpg --list-secret-keys "${KEY_NAME}"

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
sec   rsa4096 2025-02-25 [SCEAR]
      A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
uid           [ultimate] cluster.local (For Flux Secrets)
ssb   rsa4096 2025-02-25 [SEA]
      8B514FAAC98DF16F61EE487A45E294E6CACB03D4

Take note of the fingerprint (in this example: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4).

3. Store Private Key as Kubernetes Secret

Now we'll export the private key and store it as a Kubernetes secret that FluxCD can access:

gpg --export-secret-keys --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" | kubectl create secret generic sops-gpg \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin

secret/sops-gpg created

Let's verify the secret was created properly:

apurv@oxygen:~> kubectl -n flux-system get secrets

NAME          TYPE     DATA   AGE
flux-system   Opaque   3      25h
sops-gpg      Opaque   1      38s

Since the local key is unprotected (we used %no-protection for demonstration purposes), we should delete it from our local machine:

gpg --delete-secret-keys "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4"

4. Configure In-Cluster Secrets Decryption

For production environments, the recommended approach is to store secrets in a separate repository with restricted access. However, for this tutorial, we'll patch the existing cluster manifest to add support for the decryption provider.

Edit cluster/default/flux-system/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
+patches:
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: flux-system
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-gpg

This patch configures FluxCD to use SOPS for decryption, referencing our previously created GPG key secret.
Let's commit and push our changes, then verify the deployment:

git commit -m "Configured SOPS decryption for main cluster repo" && git push origin

flux get kustomizations flux-system --watch

NAME            REVISION                SUSPENDED       READY   MESSAGE                              
flux-system     main@sha1:c35e4e45      False           True    Applied revision: main@sha1:c35e4e45

5. Export Public Key and Configure SOPS

We'll export the public key to the cluster directory:

gpg --export --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" > ./cluster/default/.sops.pub.asc

Next, create a SOPS configuration file to specify which parts of the YAML files should be encrypted:

cat <<EOF > ./cluster/default/.sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
EOF

This configuration tells SOPS to only encrypt the data and stringData sections of YAML files, leaving metadata intact for Kubernetes to process.
Commit these configuration files:

git add . && git commit -m "Added SOPS public key and config"

6. Validate the Setup

Let's verify our setup by creating a sample secret and ensuring it works properly:

Create ./cluster/default/samplesecret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: samplesecret
  namespace: default
type: Opaque
stringData:
  message: This is a secret message

Now encrypt this file with SOPS:

sops --encrypt --in-place samplesecret.yaml

After encryption the yaml file looks like below.

apiVersion: v1
kind: Secret
metadata:
    name: samplesecret
    namespace: default
type: Opaque
stringData:
    message: ENC[AES256_GCM,data:H0iMCWDGpqRSQZtjSXTbMlXaRvXQRxqs,iv:/r5ylcoqWd1wnOp1p9ksDUEs+kkPkdQz31I66LXrpxo=,tag:r2m9+dyo9tnBz+ty42cfrA==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2025-02-25T12:00:11Z"
    mac: ENC[AES256_GCM,data:AuJ9hHq2+Yars67Rw8QhkpjCu8iwqGVsWDJ2/1oEx430yaGSCNXXcZnntek9rsI3xuEzyVwTJcXIE/1yW1bCvPPdWUOel+dY+mLkLSUxio238FbDoGI8mv+7FLpDe4Tn9GZdBbaMyGF6FwXXfQJ8021tLSqK6IJQN8nW2XbT0L0=,iv:KkmRa+HzZ0lI0vsEdqX+ZY6HIpudA/0ABny6OruwuQ8=,tag:6gaSJssqOEQk53DvXCZT0Q==,type:str]
    pgp:
        - created_at: "2025-02-25T12:00:11Z"
          enc: |-
            -----BEGIN PGP MESSAGE-----

            hQIMA0XilObKywPUARAArblXLVn7VaJ+Sw2jNqptBPoPaDq7CA4V2LqUoGQUdIiX
            6pHZoZmfWCpvXAUTqLQ1xEnZ32XHl+lwEVxLzV6OckkYcZKxMGz1lHZGmK7QghtV
            Gm32lFvXVOnMfI9uZ96WH9WW+8Dng3VgyBAK/putNG+N3NLkeXa3vrWaaNaHY/Dj
            aBPK1FqzFAYLbFlGjadA1+1xpFfA4JnE5dLX5HTDIo9DKldJWxxqY6cGo6jTg1lO
            j+vmLHQicdKrApbMqdq1KyRLWTU21B0cRlfIIK37rX9xESXjUlwPmF27sew2KPsx
            iuLyofDu/fVD9kFUC/Zkrmog0IgBWaKuUniOBT1KkcPJoJ5AX9m3qyTGjTGNwvVU
            DWATppNCqnO2NhN2D13sOxL5/BSkJT4HZgdj8oIiQQjv0QwLW6nwlEHcmveiX6aK
            CRUOR51gp+ja+fAlZnevYUpJMJZFz4TX1LTVXqUd8dJeXePaAsCPzO38Ywrpe83D
            Z9n9ABYawCohwSb+Nd1/eoU0RBZwRfcD0PumTbWj8mbSMn6cPHJuAY1j1WZKH1/y
            PM583Vv1zA4NlUJwhAiqI/X0kl22+Qh4tq+tdrhPVLw7P+m7diHG2pyd6jKDZ/K9
            D3tY+eeQWcM3EV0hRBN7yA87Rs07lCy0m72PqAtD07qoUbO6jLXipIuE7TgPuZ3U
            aAEJAhBexKkWWprEJjk+jpt4h8aXyG7wUAeafCWr2kIWz0/kOSnG0STCjuL1kDbu
            +Ysah6EqMijU1sQBJv9Jn5oQ9eTAiHwN2Brh8F1nCPT+E6Ih6lbiJSLAD8duEa7V
            Nsgo8cYB0ebx
            =R3G0
            -----END PGP MESSAGE-----
          fp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4

Lets push these changes.

git add samplesecret.yaml && git commit -m "Added sample secret"
git push origin

Let's verify that the secret is created in the cluster:

kubectl get secrets
NAME           TYPE     DATA   AGE
samplesecret   Opaque   1      52s

Verify the content of the secret:

kubectl get secret samplesecret -o jsonpath='{.data.message}' | base64 --decode

This is a secret message

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

Helm chart automation
Image update automation
Notification and alerting configuration

Stay tuned for each of these topics.

References

Official FluxCD Documentation Manage Kubernetes secrets with SOPS
Mozilla SOPS - https://getsops.io/
GitOps Working Group - https://opengitops.dev
Kubernetes Documentation - https://kubernetes.io/docs/

Kubernetes GitOps with FluxCD - Part 1 - Initial Setup

Apurv Kiri — Mon, 17 Mar 2025 18:30:00 +0000

GitOps practice enables us to define our infrastructure as code in a declarative manner. It serves as an audited single source of truth for our cloud/cluster state, providing benefits like:

Version control and audit trails for infrastructure changes
Automated reconciliation between desired and actual states
Improved security through encrypted credentials and access control
Simplified rollback capabilities
Enhanced collaboration through Git workflows

FluxCD is a CNCF graduated project that enables GitOps practices in the Kubernetes ecosystem. It automates the deployment, monitoring, and reconciliation of cluster resources based on Git repositories.

This post covers the initial setup and core functionality of FluxCD in a Kubernetes cluster.

1. Setup Git dependencies

Lets start by creating git repository in Github.

Next, we need to generate a fine-grained access token that FluxCD will use to interact with our repository. These tokens provide more granular control over repository permissions compared to classic personal access tokens.

We'll keep Read and write permission for Administration, Contents access and Read-only for Metadata access limited to repository we created above and permissions as recommeded by official FluxCD documentation for this initial setup.

2. Setup Flux CLI

Lets start by installing FluxCD cli

sudo zypper in flux2-cli

Next step is to export github token and user details variables.

export GITHUB_TOKEN=******
export GITHUB_USER=*****

For this post we are using locally VM hosted k3s cluster, lets check if we have access to same with kubectl.

# 1. Point to specific cluster we are setting up
export KUBECONFIG=~/.kube/local.config
# 2. Check for access and version
kubectl version

Let's check same with Flux cli if it has access as well.

flux check --pre

3. Bootstrap FluxCD

We are going to install all components including extra for image automation.

flux bootstrap github \
    --owner=$GITHUB_USER \
    --repository=k8s-gitops \
    --branch=main \
    --path=./cluster/default \
    --read-write-key \
    --components source-controller,kustomize-controller,helm-controller,notification-controller \
    --components-extra image-reflector-controller,image-automation-controller \
    --personal

Lets verify Flux setup step by step.

Bootstrap has created one deployment key in Github repository and stored as flux-system/flux-secret

Let's verify deployment key in Github

Let's verify secret with kubectl -n flux-system get secrets

Flux has initialized the repository with initial commit having definition of components at cluster/default as specified bootstrap command.

FluxCD reconclided with Git repository and installed the components in flux-system namespace, lets verify same with kubectl

4. Verify installation

Lets verify our installation by creating a sample deployment.

apps/nginx/deployment.yaml

apiVersion: apps/v1
metadata:
  name: nginx-deployment
  namespace: default
kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "64Mi"
            cpu: "50m"

apps/nginx/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

cluster/default/nginx.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: nginx
  namespace: flux-system
spec:
  interval: 5m
  path: ./apps/nginx
  prune: true
  retryInterval: 2m
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: default
  timeout: 3m
  wait: true

The Kustomization resource has several important fields:

interval: How often Flux should reconcile this resource (5m = 5 minutes)
path: Directory containing the Kubernetes manifests
prune: Whether Flux should delete resources that are no longer defined in Git
retryInterval: How long to wait between retries if reconciliation fails
wait: Whether Flux should wait for resources to be ready before completing reconciliation
timeout: Maximum time to wait for reconciliation

Lets push these changes and wait for reconciliation to finish.

Lets verify if nginx is deployed

5. What if

One of the core goals of GitOps is ensuring that our cluster's state always matches the resources defined in our git repository. This continuous reconciliation process is what makes GitOps powerful for maintaining system consistency.

To demonstrate this functionality, we'll delete our nginx deployment using kubectl and observe how Flux's reconciliation process automatically restores it to the desired state defined in Git. This showcases the "self-healing" capability of the GitOps approach.

In screen above:

First step is to verify the deployment with kubectl get deployments to check if nginx is up and running
Second step is to delete deployment with kubectl delete deployment/nginx-deployment
3rd step to confirm nginx deployment is deleted with kubectl get deployment
4th step is to watch over FluxCD reconciliation to kick in
5th step to verify that nginx is re-deployed post Flux's reconciliation

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

Helm chart automation
Image update automation
Notification and alerting configuration
Secrets management with SOPS

Stay tuned for each of these topics.

References

Official FluxCD Documentation - https://fluxcd.io/flux/
GitOps Working Group - https://opengitops.dev
Kubernetes Documentation - https://kubernetes.io/docs/