Raphael Gab-Momoh

Posted on Sep 17 Originally published at raphaelgmomoh.pages.dev

GitOps on Alibaba Cloud ACK: Automating Kubernetes Deployments with Git

#alibabacloudengineeringlabseri #gitops #ack #kubernetes

Part 6 of the Alibaba Cloud Engineering Lab Series.

Architecture

Git Repo (manifests)
   │  git push
   ▼
ArgoCD (running in ACK)
   │  detects drift, pulls desired state
   ▼
ACK Cluster
   │
   ├── namespace: staging
   └── namespace: production

Repo structure, environment-separated by directory rather than branch — branch-per-environment tends to drift and merge-conflict; directory-per-environment with a shared base is easier to diff and reason about:

manifests/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── staging/
│   │   └── kustomization.yaml
│   └── production/
│       └── kustomization.yaml

Before the how, the what — three terms this workflow leans on:

GitOps — a deployment model where Git is the single source of truth for what should be running, and an automated agent (not a human running kubectl) continuously makes the live cluster match what's in the repo. A deployment stops being "someone ran a command" and becomes "someone merged a pull request" — which means every deployment has a review, a diff, and a commit history for free.
ArgoCD — the agent that actually implements GitOps for Kubernetes here: it runs inside the cluster, watches a Git repo, and automatically applies (or reverts) changes to match it. It's the piece that makes GitOps continuously enforced rather than a one-time kubectl apply that Git happened to record.
Kustomize — a way to describe environment-specific differences (like "production needs 5 replicas, staging needs 1") as small patches on top of a shared base configuration, instead of maintaining separate, fully-duplicated YAML files per environment. It's the mechanism behind the base/ + overlays/ folder structure above.

I built this exact base/overlays structure, pointed a real ArgoCD Application at it, and reproduced the missing-ConfigMap-key incident below by actually deploying with the key present in staging and absent from production. The companion repo is that same repo — kubectl kustomize builds clean on both overlays, verified before it was pushed.

Problem

Manual kubectl apply deployments have no audit trail beyond shell history, no automatic drift detection when someone makes a manual kubectl edit "just this once," and no clean rollback path beyond hoping someone remembers the last-known-good image tag.

Implementation

Install ArgoCD into ACK:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Define the Application resource pointing at the production overlay:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service-prod
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/raphgm/gitops-ack-lab.git
    path: manifests/overlays/production
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

selfHeal: true is the enforcement mechanism — if someone manually edits a live resource, ArgoCD reverts it back to what's declared in Git within seconds. Git becomes the only path to a state change that sticks.

Container image promotion between environments happens via a CI step that updates the image tag in the staging overlay, then a pull request to promote the same tag into the production overlay — an explicit, reviewable, git-logged promotion rather than a redeploy with a different flag.

Failure / Challenge

A deployment to production failed mid-rollout: the new pod version crash-looped on startup because a required environment variable (a new database connection string) existed in the staging ConfigMap but had never been added to the production overlay's ConfigMap patch. ArgoCD showed the Application as Degraded, with half the replicas on the old image and half crash-looping on the new one.

Solution

argocd app rollback api-service-prod

ArgoCD's rollback reverted to the last known-good Git revision in seconds — the entire value of GitOps is that "rollback" means "redeploy a previous commit," not "SSH in and manually fix Kubernetes state under incident pressure." The actual fix was adding the missing ConfigMap key to the production overlay in a follow-up PR, tested in staging first, then promoted properly.

kubectl diff -k manifests/overlays/production/ manifests/overlays/staging/

That diff command — comparing overlays directly — became a permanent pre-promotion CI check afterward, catching exactly this class of environment-drift bug before it reaches production again.

Cost / Performance

Metric	Before GitOps	After GitOps
Mean time to rollback	~15 min (manual)	~45 sec (`argocd app rollback`)
Deployment audit trail	Shell history only	Full Git history + ArgoCD sync log
Config drift incidents/month	2–3	0 (self-heal reverts them)

GitOps has no direct infrastructure cost beyond ArgoCD's own small resource footprint in-cluster — the return here is entirely in incident-response time and eliminated drift.

Lessons Learned

selfHeal is what actually enforces "Git is the source of truth" — without it, GitOps is just a nicer deployment UI, not a control mechanism.
Environment-overlay drift (a config value present in staging, missing in prod) is the single most common cause of a "works in staging, breaks in prod" incident — diff overlays directly, don't trust memory.
Rollback speed is a designed property of the deployment system, not an emergency improvisation — GitOps makes rollback a first-class, tested command rather than a stressful manual recovery.

GitHub Repository: gitops-ack-lab — the working Kustomize base/overlays and ArgoCD Applications, ready to run.

GitOps · ArgoCD · ACK · Alibaba Cloud · Kubernetes · Kustomize

Originally published on my portfolio.

DEV Community