DEV Community

Cover image for Building a Real GitOps Pipeline: Argo CD Canaries, Cosign Keyless Signing, and Prometheus Gates
cypher682
cypher682

Posted on

Building a Real GitOps Pipeline: Argo CD Canaries, Cosign Keyless Signing, and Prometheus Gates

#gitops #devops #kubernetes #cicd

Overview

Pushing to main starts 9 stages / 22 jobs in GitLab CI: lint, build, test, security scan, keyless signing, tag bump, dev sync, and smoke test. Staging promotion is a manual gate preceded by a cryptographic re-verification of the image signature; the Argo Rollouts canary then steps 10% → 50% → 100%, each step gated by a Prometheus app_error_rate AnalysisRun. No kubectl between commit and canary.

gitops-pipeline is an end-to-end GitOps delivery pipeline: GitLab CI builds, scans, and keyless-signs a container image; a Helm chart in the same repo is the source of truth; Argo CD reconciles two local Kubernetes environments (dev + staging) to it; staging releases are Argo Rollouts canaries gated by real Prometheus error-rate analysis. Cloud-hosted GitLab runners reach the local cluster through ngrok and cloudflared tunnels.

This writeup covers the architecture, the security decisions, and the roadblocks that shaped the system. The flow in one line: push → lint → build → test → scan → cosign+sbom → bump values → argocd sync dev → smoke test → manual approve → re-verify signature → sync staging → canary (10 → 50 → 100) with Prometheus gates. Signed end-to-end and gated by real metrics, with zero cloud infrastructure spend.

Verified Run Snapshot

Metric Recorded Value
Pipeline ID #2824754659 (9 stages, 22/22 jobs green)
Application Version v0.2.0 (Image tag: 0891d917)
GitOps Controller Argo CD v3.5.2 (App-of-Apps + ApplicationSet)
Environment State gitops-dev & gitops-staging (Healthy / Synced)
Canary Rollout Step 7/7 (ActualWeight: 100%, 2/2 AnalysisRuns Successful)
Canary Gate Metric app_error_rate{job="app-chart-staging"} == 0 (Threshold: ≤ 0.05)

What Makes This Different from a Basic CI/CD Pipeline

  1. Git is the Desired-State Authority: CI never applies manifests directly to Kubernetes. CI updates the desired state in git, and Argo CD reconciles actual cluster state against it.
  2. Closed-Loop Tag Updates: CI commits the built SHA back into the Helm chart's values.yaml ([skip ci]), ensuring the repository reflects exactly what is running.
  3. Cryptographic Promotion Gate: Staging promotion requires human approval AND an automated cryptographic re-verification of the Cosign image signature.
  4. Metric-Driven Canaries: Rollouts don't just rely on sleep timers; they query live Prometheus metrics (app_error_rate) to abort and self-heal on failure.
  5. Zero Plaintext Secrets: Secrets stay SOPS-encrypted at rest in git and are only decrypted in-memory during sync by an Argo CD ConfigManagementPlugin (CMP).

Architecture & Mental Model

Seven moving parts, one repo: GitLab CI drives the pipeline; Helm renders one chart with three value sets; Argo CD (app-of-apps → ApplicationSet) reconciles gitops-dev and gitops-staging; SOPS + age + a CMP sidecar decrypt secrets at apply time; Argo Rollouts stages releases as canaries; Prometheus passes judgement on them; ngrok + cloudflared let Cloud-hosted GitLab runners reach the local cluster.

[ git push main ]
       │
       ▼
GitLab CI/CD (.gitlab-ci.yml — 9 stages, 22 jobs)
 ├── validate   ── hadolint, kubeval (manifest validation)
 ├── build      ── Docker build → GitLab Registry (SHA tag: 0891d917)
 ├── test       ── Pytest with coverage threshold gate (70%)
 ├── scan       ── Trivy, Semgrep SAST, pip-audit, SOPS encryption detect
 ├── sign       ── Cosign keyless (Sigstore Fulcio) + Syft CycloneDX SBOM
 ├── push       ── tag :latest + commit image tag to helm/app-chart/values.yaml [skip ci]
 ├── deploy-dev ── Argo CD sync gitops-pipeline-dev (via ngrok tunnel)
 ├── smoke-test ── curl /health via cloudflared tunnel
 └── promote (manual gate → re-verify signature)
       └── deploy-staging ── Argo CD sync gitops-pipeline-staging
                                    │
                                    ▼
                         Argo Rollouts Canary
               10% → AnalysisRun (Prometheus) → 50% → AnalysisRun → 100%
Enter fullscreen mode Exit fullscreen mode

Key Technical Decisions & Deep Dive

One repository, one commit, one provenance trail: the app source, the CI definition, the Helm chart, the SOPS-encrypted secrets, and the Argo CD bootstrap manifests all share a single history — so every image tag traces back to a specific commit a human reviewed. Three decisions explain the rest:

1. Keyless Supply-Chain Security with Cosign & Sigstore

Rather than managing static private keys and rotation overhead, signing uses short-lived OIDC workload identity tokens issued by GitLab CI.

  • Cosign exchanges the GitLab OIDC token with Sigstore Fulcio to obtain an ephemeral X.509 certificate.
  • The signature and transparency log entry (Rekor) are attached directly to the registry image.
  • At the staging gate, promote:verify-signature re-validates: 
cosign verify \
  --certificate-identity-regexp ".*gitops-pipeline.*" \
  --certificate-oidc-issuer "https://gitlab.com" \
  registry.gitlab.com/cypher682/gitops-pipeline:0891d917
Enter fullscreen mode Exit fullscreen mode

2. Secrets Management: SOPS + age with CMP Decrypt-at-Apply

Secrets in secrets/app-secrets.enc.yaml are encrypted using age. Plaintext never exists in git.

  • An Argo CD sops-helm sidecar plugin runs inside argocd-repo-server.
  • During sync, the plugin executes helm template and pipes decrypted SOPS data into the manifest stream, dynamically adjusting the namespace to ${ARGOCD_APP_NAMESPACE}.

3. Progressive Canary Delivery with Prometheus

Staging deployments run via Argo Rollouts:

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause: {duration: 120s}
      - analysis:
          templates:
            - templateName: error-rate-check
      - setWeight: 50
      - pause: {duration: 120s}
      - analysis:
          templates:
            - templateName: error-rate-check
      - setWeight: 100
Enter fullscreen mode Exit fullscreen mode

Each AnalysisRun executes an in-cluster Prometheus query against the app's /metrics endpoint:

app_error_rate{job="app-chart-staging"}
Enter fullscreen mode Exit fullscreen mode

If app_error_rate > 0.05 at any check, the release immediately halts and rolls back. Two gated steps exist instead of one so a bad release can fail at 10% (minimal blast radius) and again at 50% (under real load) before the final cutover — the cost is a few extra minutes, the payoff is two independent chances to abort. Note that in this setup without a Service Mesh traffic router, weights apply across the ReplicaSet pod distribution.

Engineering Roadblocks & Solutions

Three infrastructure bugs cost the most time — each with a fix you'll reuse:

  1. Argo CD + ngrok 307 Redirect Loop: ngrok terminates TLS and forwards cleartext; Argo CD's TLS listener issued a 307 redirect loop. Fix: Set server.insecure: "true" in argocd-cmd-params-cm.
  2. The argocd CLI URL Parsing Trap: argocd login https://… dials a bogus tcp///host:port proxy; --server https://host becomes https://https//host. Fix: Use bare host:port + per-command --auth-token, bypassing persistent login.
  3. CMP Context Blindspot: With source.path: helm/app-chart, the repo server only shipped the chart subdirectory — breaking ../secrets/*.enc.yaml — and an AnalysisTemplate referenced outside the chart made the rollout instantly degrade. Fix: Set source.path: "." so the CMP sees the repo root, and bundle analysis-template.yaml into the chart templates.
  4. ngrok Free-Tier URL Collision: Free accounts only provide one static domain. Fix: Used cloudflared quick tunnels (accountless, ephemeral) for the second exposed app endpoint.

Try It (and Steal It)

Everything is one repo: app + CI + Helm chart + Argo CD bootstrap config + SOPS-encrypted secrets. git push → green pipeline → canary. Fork it, swap the namespace/image names, and follow the README Quickstart + docs/.

Curious what you'd add — a Service Mesh traffic router for true weighted HTTP splits? Kyverno for cluster-level signature enforcement? Multi-cluster Argo CD? Let me know in the comments.

Top comments (0)