GitOps: Demo with Argo CD and KSOPS on GKE

What is GitOps ?

It's a new concept introduced by Weave and seems to be the new trend for managing Kubernetes applications.

GitOps is a way to do Kubernetes cluster management and application delivery. It works by using Git as a single source of truth for declarative infrastructure and applications.

It is very similar to infrastructure as code, the main goal is to avoid developers or Ops to manage Kubernetes resources from their own kubectl. It could be resumed by Operations by Pull Request. No new stuff for now.

Some nice features from this concept :

• Fast error recovery: Indeed, it is only a Git revert to restore your environment.
• Deploy Faster: No need to wait for an OPS to be in the mood to prod.
• Security: Developers have not access to clusters (and CI tools neither, we'll see why later).
• Documentation: Because Git is the single source of truth, no need to connect to clusters to see which resources are present.

Two ways to implement this pattern

Push based

Classic way to deploy your application. Once committed, a trigger starts a job on your CI tool.

Pull based

In this case, an agent is running on the cluster and pull the modification. It avoids giving cluster access to CI tool.

Tools

There are many tools which implement this pattern. But in fact, a simple script could do the job :

git clone https://awesome-repo
cd awesome-repo

while true {
  git pull 
  kubectl apply -f .
  sleep 30
}

Let's see some tools ready for production.

Flux CD

Flux CD seems to be the most famous product of the GitOps concept. It has been developed by Weaveworks. Probably the simplest GitOps tool to use though it only has a CLI and you need one agent per managed application.

Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

Argo CD tracks all the changes made in a repository to deploy the desired application state.

Some nice features :

• Automated or manual deployments of applications.
• Could manage multi-cluster.
• Rollback to any Git commit.
• Canary deployment.
• Execute operations before or after sync.
• Health status of existing applications resources.
• Support multiple templating tools like Kustomize or Helm.

Focus on Argo CD

We will focus on Argo CD for this demo, it has more features and its Web UI is very powerful to easily see our managed applications.

Demo

Part 1 - Deployment

Requirements needed before starting :

• GCP account
• kubectl
• kustomize
• SOPS
• argocd cli (optional: it's used to update admin password)

Secret Management with SOPS and Google KMS

I have discovered SOPS some months ago and I like to use it now to manage secrets in git. We are going to use it with GCP kms. For more information about SOPS, don't hesitate to read our previous article https://dev.to/stack-labs/manage-your-secrets-in-git-with-sops-g0a

Step 1: Create a KMS Key

gcloud kms keyrings create "gke-argocd-demo" \
  --location=global

gcloud kms keys create "gke" \
  --location "global" \
  --keyring "gke-argocd-demo" \
  --purpose "encryption"

gcloud kms keys list \
  --location "global" \
  --keyring "gke-argocd-demo"

KMS_ID=$(gcloud kms keys list --location "global" --keyring "gke-argocd-demo" --format 'get(name)')

Step 2: Launch on GKE with workload Identity

Workload Identity is the recommended way to access Google Cloud services from within GKE due to its improved security properties and manageability. https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity?hl=en

# Create the GKE cluster with workload identity enabled

PROJECT_ID=$(gcloud config list --format 'value(core.project)')

gcloud beta container clusters create gke-argocd-demo \
  --release-channel regular \
  --workload-pool=$PROJECT_ID.svc.id.goog

Step 3: Prepare Workload Identity

# Configure kubectl
gcloud container clusters get-credentials gke-argocd-demo

# Create iam service account
gcloud iam service-accounts create gke-argocd-demo

# Create kubectl namespace (we will deploy argo resources inside later)
kubectl create ns argocd

kubectl create serviceaccount --namespace argocd gke-argocd-demo 

# Link Kubernetes service account to GCP service account
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$PROJECT_ID.svc.id.goog[argocd/gke-argocd-demo]" \
  gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com

# Add permissions to interact with KMS from pods
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:"gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com" \
  --role "roles/cloudkms.cryptoKeyEncrypterDecrypter"

kubectl annotate serviceaccount \
  --namespace argocd gke-argocd-demo \
  iam.gke.io/gcp-service-account=gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com

Kustomize with SOPS

In this demo, argo CD uses Kustomize to deploy k8s resources. There is a SOPS plugin for Kustomize ( https://github.com/viaduct-ai/kustomize-sops ). It is not mandatory to install this plugin locally though it is recommended to check your manifests before pushing to Git and avoid potential errors during deployment.

Step 4: Deploy Argo CD

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Mandatory with GKE
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user="$(gcloud config get-value account)"

# Optional: if you want to access to the Web UI with an external IP
kubectl patch svc argocd-server \
  -n argocd \
  -p '{"spec": {"type": "LoadBalancer"}}'

Step 5: Get default password

Default Argo CD password is the name of the server pod. Save it somewhere, or change it with next step.

# get current password
kubectl get pods \
  -n argocd \
  -l app.kubernetes.io/name=argocd-server \
  -o "jsonpath={.items[0].metadata.name}"

Step 6: Change default password

# forward port is needed (Argo server service is a clusterIP) unless you have patched the service with a LoadBalancer
kubectl port-forward svc/argocd-server -n argocd 8080:443

### Optional steps if you want to change the default password
# Install Argo CD CLI https://argoproj.github.io/argo-cd/cli_installation

# login
argocd login 127.0.0.1:8080
argocd account update-password

Step 7: Patch Argo CD to use SOPS

Edit config map to enable Kustomize with alpha plugins. For this, you need to add data part (see below).

kubectl -n argocd edit cm argocd-cm


apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
# add this :
data:
  kustomize.buildOptions: "--enable_alpha_plugins"

To integrate kustomize-sops with Argo CD, we need to patch the deployment server with an init-container and the new service account created. You can look the patch here.

# Patch command
kubectl patch \
  -n argocd deployment/argocd-repo-server \
  -p "$(curl https://raw.githubusercontent.com/pabardina/gitops-argocd-sops/master/argo-cd-repo-server-ksops-patch.yaml)"

# Watch for the new pod to be ready
kubectl -n argocd get pod \
  -l app.kubernetes.io/name=argocd-repo-server \
  --watch=true

Step 8: Launch UI

Open http://127.0.0.1:8080, connect with admin and password from last step and register an application.

You can fork my demo application: https://github.com/pabardina/gitops-argocd-sops.git and use your fork repository url in Argo CD.

The application deployed:

Part 2 - Demo with secrets

Step 1: Clone locally your fork repository

git clone https://github.com/<your-username>/gitops-argocd-sops.git
cd gitops-argocd-sops

Step 2: Add a secret

cat <<EOF > base/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: super-secret
data:
  pass: c3VwZXItcGFzc3cwcmQ=
EOF

Step 3: Encrypt the new secret with SOPS

# encrypt the file with sops
sops -e -i --gcp-kms $KMS_ID base/secret.yaml

# Create Kustomize generator

cat <<EOF > base/secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  # Specify a name
  name: example-secret-generator
files:
  - ./secret.yaml
EOF


# Update base/kustomization.yaml to add generator part

vi base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
# add this :
generators:
- ./secret-generator.yaml

Step 4: Valid your manifest and push to git

If you haven't installed ksops locally, you won't be able to do next command :

# Validate your manifest locally
kustomize build --enable_alpha_plugins base

Push your secret to Git

git add . \
  && git commit -m "Improved architecture with secrets" \
  && git push

In a minute, argo CD will detect the new commit and apply the changes. You will be able to see your new secret added to the architecture.

Conclusion

Argo CD is a powerful product, the SOPS integration is not that simple though. Even if some steps are required, it's cool to use this kind of tool.

Final step: Destroy resources

# kubernetes
gcloud container clusters delete gke-argocd-demo

# iam
gcloud projects remove-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:"gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com" \
  --role "roles/cloudkms.cryptoKeyEncrypterDecrypter"

gcloud iam service-accounts delete gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com

Comments

[rbabyuk]

I've tried this but I am getting encrypt/decrypt things inside argocd container "Failed to call GCP KMS encryption service: googleapi: Error 403: Request had insufficient authentication scopes.
More details:
Reason: insufficientPermissions, Message: Insufficient Permission
]"

and I am using argocd with sops preinstalled, because I am trying to use helm-secrets with argocd where my keys are GCP KMS.
probably the last resort will be to 'mount' secret key for user account and use env. variable GOOGLE_APPLICATION_CREDENTIALS in each pod.
but it look weird at this point of time.