Originally published on kuryzhev.cloud
Your ArgoCD dashboard shows "OutOfSync" every ten seconds and you have no idea why, because nothing in Git actually changed. If you're running an ArgoCD game server deployment on top of Agones, this is almost always the controller doing exactly what it's supposed to do — and ArgoCD fighting it. I've spent enough nights debugging this exact flapping to write down what actually fixed it, plus a handful of other lessons from running multi-region Agones fleets through GitOps. These are independent tips, pick whatever's biting you right now.
Ignore Agones status fields or your sync will flap forever
The Agones controller mutates .status.state, .status.address, and .status.ports on every GameServer object continuously — that's just how it tracks server lifecycle. ArgoCD sees this as drift from the manifest in Git and, with selfHeal: true, tries to revert it. The result is a UI error like ComparisonError: resource GameServer/agones.dev "fleet-xyz" has diff at spec/status repeating every refresh cycle, and worse, ArgoCD occasionally re-applying stale state onto a live GameServer and knocking players off.
The fix is ignoreDifferences with jsonPointers scoped to the exact fields the controller owns, not the whole resource. Don't disable selfHeal globally — you still want it for everything else in the app. Scope it per-resource instead.
ignoreDifferences:
# Agones mutates these constantly - stop ArgoCD from fighting the controller
- group: agones.dev
kind: GameServer
jsonPointers:
- /status
- /spec.ports
- group: agones.dev
kind: Fleet
jsonPointers:
- /status/allocatedReplicas
- /status/readyReplicas
ArgoCD 2.9+ also supports jqPathExpressions if you need conditional matching rather than fixed pointers — useful if only some fleets need the exemption.
Sequence deploys with sync waves or fleets race their own dependencies
A Fleet spinning up before the matchmaker service exists is a classic race condition, and it's entirely avoidable with sync waves. Give the matchmaker/queue manifests argocd.argoproj.io/sync-wave: "-1" so they land before anything else — the default wave is "0", so anything unannotated deploys alongside your Fleet. Put the FleetAutoscaler in a later wave like "1", because it shouldn't be scaling a Fleet that isn't ready yet.
Watch out for one mistake I made early on: putting the Agones cluster-scoped CRD install in the same wave as namespaced Fleet objects. Kubernetes API server sometimes accepts both requests concurrently, and if the CRD isn't registered yet when the Fleet object lands, you get a rejected apply that ArgoCD retries blindly. Give the CRD install its own earlier wave, full stop.
Drain players with a PreSync hook before rolling new builds
Recreating a Fleet on every version bump kicks every connected player instantly — not acceptable for anything with active matches. A PreSync hook Job that calls the Agones SDK's Shutdown() (or flips the drain annotation) gives in-flight sessions a grace window before the new manifest applies.
apiVersion: batch/v1
kind: Job
metadata:
name: pre-sync-drain-players
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/sync-wave: "-1"
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
containers:
- name: drain
image: kuryzhev/agones-drain-cli:1.2.0
# calls Agones SDK Shutdown() on servers past a grace period
args: ["--fleet=game-fleet-eu-west", "--grace-period=120s"]
restartPolicy: Never
Pair this with Argo Rollouts (v1.6+ has native Agones fleet support) for canary-style gradual replacement instead of a full recreate. A PostSync hook is also worth adding — check active player count metrics before ArgoCD marks the sync as healthy, so a broken build doesn't get a green checkmark while nobody can actually connect.
Scope AppProjects per region or a bad manifest goes global
I've seen a single shared AppProject across all regions let a EU-only manifest change sync straight into US production. That's the kind of incident you only need once. Define a dedicated AppProject per region with sourceRepos, destinations, and namespaceResourceWhitelist locked to that region's namespace and cluster — for example game-eu-west can only ever target the game-eu-west namespace on the EU cluster.
Map your OIDC groups to project-scoped roles in the argocd-rbac-cm ConfigMap's policy.csv rather than handing everyone the default admin role. This is standard ArgoCD RBAC practice, but it's easy to skip when you're moving fast on a new project — and that's exactly when it bites you.
Fan out fleets across clusters with ApplicationSets, not copy-paste YAML
Once you're running the same fleet in three or four regions, maintaining separate ArgoCD Applications by hand becomes error-prone busywork. An ApplicationSet with a list generator keyed on region solves this cleanly — inject {{region}} into your Kustomize overlay path and let one Git source drive every cluster.
generators:
- list:
elements:
- region: eu-west
cluster: https://eu-west-cluster-api.internal
- region: us-east
cluster: https://us-east-cluster-api.internal
template:
spec:
source:
path: overlays/{{region}}
destination:
server: '{{cluster}}'
Keep node selectors, replica counts, and other per-region tuning in overlays/<region>/kustomization.yaml, with shared logic in base/fleet.yaml. The gotcha here: the ApplicationSet template's syncPolicy defaults can silently override the per-app ignoreDifferences you set up in tip one, unless you re-declare it explicitly inside the template. I lost an afternoon to this exact thing — the flapping came back the moment I moved a single-region app into an ApplicationSet.
Use spot nodes for cost, but never mid-match
Spot/preemptible nodes cut compute cost significantly for game server workloads, but Agones fleets need to respect active matches or you'll kill sessions on a preemption notice. Taint the spot pool with game-workload=spot:NoSchedule and only tolerate it on non-critical, low-stakes fleets — keep the matchmaker and any stateful queue service on on-demand nodes, full stop.
metadata:
annotations:
agones.dev/safe-to-evict: "false" # blocks node drain during active matches
One silent cost leak worth checking right now: oversized bufferSize in your Fleet manifest. A buffer of 50 idle servers for a 10-player game wastes roughly 4-5x the node capacity you actually need, and it's the kind of thing nobody notices until the AWS bill review. Check the Kubernetes docs on pod priority and preemption if you're tuning spot behavior further.
Never commit game server credentials as plain Secrets
Base64 is encoding, not encryption — I still see teams commit a raw Secret manifest to their GitOps repo because "it's base64'd, it's fine." It isn't. Anyone with repo read access, including that contractor who left six months ago, can decode it in one line of shell.
echo "cGxheWVyLWFwaS10b2tlbg==" | base64 -d
# player-api-token — not encryption, just formatting
Use Bitnami SealedSecrets (v0.24+) or the External Secrets Operator pulling from Vault or SSM instead. Only the encrypted SealedSecret CRD lives in Git — the controller inside the cluster decrypts it, and a leaked repo doesn't leak your matchmaker's auth tokens. This is basic hygiene for any ArgoCD game server deployment holding player auth or third-party API keys, and it's cheap to set up compared to the incident response after a leak.
For more on structuring GitOps repos and ArgoCD sync policies generally, our DevOps_DayS archive has related breakdowns worth a read.
Top comments (0)