DEV Community

Cover image for GitOps: Demo with Argo CD and KSOPS on GKE
Pierre-Alexandre Bardina for Stack Labs

Posted on • Updated on

GitOps: Demo with Argo CD and KSOPS on GKE

What is GitOps ?

It's a new concept introduced by Weave and seems to be the new trend for managing Kubernetes applications.

GitOps is a way to do Kubernetes cluster management and application delivery. It works by using Git as a single source of truth for declarative infrastructure and applications.

It is very similar to infrastructure as code, the main goal is to avoid developers or Ops to manage Kubernetes resources from their own kubectl. It could be resumed by Operations by Pull Request. No new stuff for now.

Some nice features from this concept :

  • Fast error recovery: Indeed, it is only a Git revert to restore your environment.
  • Deploy Faster: No need to wait for an OPS to be in the mood to prod.
  • Security: Developers have not access to clusters (and CI tools neither, we'll see why later).
  • Documentation: Because Git is the single source of truth, no need to connect to clusters to see which resources are present.

Two ways to implement this pattern

Push based

Classic way to deploy your application. Once committed, a trigger starts a job on your CI tool.

Pull based

In this case, an agent is running on the cluster and pull the modification. It avoids giving cluster access to CI tool.


There are many tools which implement this pattern. But in fact, a simple script could do the job :

git clone https://awesome-repo
cd awesome-repo

while true {
  git pull 
  kubectl apply -f .
  sleep 30
Enter fullscreen mode Exit fullscreen mode

Let's see some tools ready for production.

Flux CD

Flux CD seems to be the most famous product of the GitOps concept. It has been developed by Weaveworks. Probably the simplest GitOps tool to use though it only has a CLI and you need one agent per managed application.

Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

Argo CD tracks all the changes made in a repository to deploy the desired application state.

Some nice features :

  • Automated or manual deployments of applications.
  • Could manage multi-cluster.
  • Rollback to any Git commit.
  • Canary deployment.
  • Execute operations before or after sync.
  • Health status of existing applications resources.
  • Support multiple templating tools like Kustomize or Helm.

Focus on Argo CD

We will focus on Argo CD for this demo, it has more features and its Web UI is very powerful to easily see our managed applications.


Part 1 - Deployment

Requirements needed before starting :

  • GCP account
  • kubectl
  • kustomize
  • SOPS
  • argocd cli (optional: it's used to update admin password)

Secret Management with SOPS and Google KMS

I have discovered SOPS some months ago and I like to use it now to manage secrets in git. We are going to use it with GCP kms. For more information about SOPS, don't hesitate to read our previous article

Step 1: Create a KMS Key

gcloud kms keyrings create "gke-argocd-demo" \

gcloud kms keys create "gke" \
  --location "global" \
  --keyring "gke-argocd-demo" \
  --purpose "encryption"

gcloud kms keys list \
  --location "global" \
  --keyring "gke-argocd-demo"

KMS_ID=$(gcloud kms keys list --location "global" --keyring "gke-argocd-demo" --format 'get(name)')
Enter fullscreen mode Exit fullscreen mode

Step 2: Launch on GKE with workload Identity

Workload Identity is the recommended way to access Google Cloud services from within GKE due to its improved security properties and manageability.

# Create the GKE cluster with workload identity enabled

PROJECT_ID=$(gcloud config list --format 'value(core.project)')

gcloud beta container clusters create gke-argocd-demo \
  --release-channel regular \
Enter fullscreen mode Exit fullscreen mode

Step 3: Prepare Workload Identity

# Configure kubectl
gcloud container clusters get-credentials gke-argocd-demo

# Create iam service account
gcloud iam service-accounts create gke-argocd-demo

# Create kubectl namespace (we will deploy argo resources inside later)
kubectl create ns argocd

kubectl create serviceaccount --namespace argocd gke-argocd-demo 

# Link Kubernetes service account to GCP service account
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$[argocd/gke-argocd-demo]" \

# Add permissions to interact with KMS from pods
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:"gke-argocd-demo@$" \
  --role "roles/cloudkms.cryptoKeyEncrypterDecrypter"

kubectl annotate serviceaccount \
  --namespace argocd gke-argocd-demo \$

Enter fullscreen mode Exit fullscreen mode

Kustomize with SOPS

In this demo, argo CD uses Kustomize to deploy k8s resources. There is a SOPS plugin for Kustomize ( ). It is not mandatory to install this plugin locally though it is recommended to check your manifests before pushing to Git and avoid potential errors during deployment.

Step 4: Deploy Argo CD

kubectl apply -n argocd -f

# Mandatory with GKE
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user="$(gcloud config get-value account)"

# Optional: if you want to access to the Web UI with an external IP
kubectl patch svc argocd-server \
  -n argocd \
  -p '{"spec": {"type": "LoadBalancer"}}'

Enter fullscreen mode Exit fullscreen mode

Step 5: Get default password

Default Argo CD password is the name of the server pod. Save it somewhere, or change it with next step.

# get current password
kubectl get pods \
  -n argocd \
  -l \
  -o "jsonpath={.items[0]}"
Enter fullscreen mode Exit fullscreen mode

Step 6: Change default password

# forward port is needed (Argo server service is a clusterIP) unless you have patched the service with a LoadBalancer
kubectl port-forward svc/argocd-server -n argocd 8080:443

### Optional steps if you want to change the default password
# Install Argo CD CLI

# login
argocd login
argocd account update-password
Enter fullscreen mode Exit fullscreen mode

Step 7: Patch Argo CD to use SOPS

Edit config map to enable Kustomize with alpha plugins. For this, you need to add data part (see below).

kubectl -n argocd edit cm argocd-cm

apiVersion: v1
kind: ConfigMap
  name: argocd-cm
  labels: argocd-cm argocd
# add this :
  kustomize.buildOptions: "--enable_alpha_plugins"
Enter fullscreen mode Exit fullscreen mode

To integrate kustomize-sops with Argo CD, we need to patch the deployment server with an init-container and the new service account created. You can look the patch here.

# Patch command
kubectl patch \
  -n argocd deployment/argocd-repo-server \
  -p "$(curl"

# Watch for the new pod to be ready
kubectl -n argocd get pod \
  -l \
Enter fullscreen mode Exit fullscreen mode

Step 8: Launch UI

Open, connect with admin and password from last step and register an application.

You can fork my demo application: and use your fork repository url in Argo CD.

Alt TextAlt TextAlt Text

The application deployed:

Alt Text

Part 2 - Demo with secrets

Step 1: Clone locally your fork repository

git clone<your-username>/gitops-argocd-sops.git
cd gitops-argocd-sops
Enter fullscreen mode Exit fullscreen mode

Step 2: Add a secret

cat <<EOF > base/secret.yaml
apiVersion: v1
kind: Secret
  name: super-secret
  pass: c3VwZXItcGFzc3cwcmQ=
Enter fullscreen mode Exit fullscreen mode

Step 3: Encrypt the new secret with SOPS

# encrypt the file with sops
sops -e -i --gcp-kms $KMS_ID base/secret.yaml

# Create Kustomize generator

cat <<EOF > base/secret-generator.yaml
kind: ksops
  # Specify a name
  name: example-secret-generator
  - ./secret.yaml

# Update base/kustomization.yaml to add generator part

vi base/kustomization.yaml

kind: Kustomization
- deployment.yaml
- service.yaml
# add this :
- ./secret-generator.yaml
Enter fullscreen mode Exit fullscreen mode

Step 4: Valid your manifest and push to git

If you haven't installed ksops locally, you won't be able to do next command :

# Validate your manifest locally
kustomize build --enable_alpha_plugins base
Enter fullscreen mode Exit fullscreen mode

Push your secret to Git

git add . \
  && git commit -m "Improved architecture with secrets" \
  && git push
Enter fullscreen mode Exit fullscreen mode

In a minute, argo CD will detect the new commit and apply the changes. You will be able to see your new secret added to the architecture.

Alt Text


Argo CD is a powerful product, the SOPS integration is not that simple though. Even if some steps are required, it's cool to use this kind of tool.

Final step: Destroy resources

# kubernetes
gcloud container clusters delete gke-argocd-demo

# iam
gcloud projects remove-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:"gke-argocd-demo@$" \
  --role "roles/cloudkms.cryptoKeyEncrypterDecrypter"

gcloud iam service-accounts delete gke-argocd-demo@$
Enter fullscreen mode Exit fullscreen mode

Top comments (1)

rbabyuk profile image

I've tried this but I am getting encrypt/decrypt things inside argocd container "Failed to call GCP KMS encryption service: googleapi: Error 403: Request had insufficient authentication scopes.
More details:
Reason: insufficientPermissions, Message: Insufficient Permission

and I am using argocd with sops preinstalled, because I am trying to use helm-secrets with argocd where my keys are GCP KMS.
probably the last resort will be to 'mount' secret key for user account and use env. variable GOOGLE_APPLICATION_CREDENTIALS in each pod.
but it look weird at this point of time.