Flo Comuzzi

Posted on Oct 3

PART 3 A Helm Chart for Ephemeral Environments

#devex #idp #devops #cloud

Part 3 — Ephemeral Environments with Helm and Argo CD (Starry IDP)

TL;DR: We package a complete, isolated preview stack (two frontends, one backend, Redis, two DBs) into a single Helm chart and reconcile it with Argo CD. Each preview lives in its own namespace, gets secrets from External Secrets, exposes stable ingress with managed TLS, and tears down cleanly via prune/TTL—shrinking feedback loops and avoiding collisions in shared dev/stage.

Note: I use preview and ephemeral environment terms interchangeably. The point is to emphasize that these are short-lived environment instances.

Who this is for

Platform teams adopting GitOps on GKE.
App teams wanting per-PR previews with minimal toil.

Prerequisites and assumptions

GKE with GCE Ingress, ManagedCertificate available.
Artifact Registry for images; Google Secret Manager for secrets.
Argo CD installed and reachable by the cluster.
Optional: External Secrets Operator (GSM integration), Workload Identity.

Why previews (recap)

Shared environments create collisions, noisy logs, version skew, and review friction. Previews isolate each change into its own namespace with predictable URLs, allowing fast, deterministic testing without blocking teammates.

Architecture (at a glance)

A preview environment has two frontends that connect to a backend. The backend connects to a Redis instance and two databases.

From PR to preview (sequence)

The Helm chart below is what Argo CD renders when creating an ephemeral/preview environment.

File hierarchy (Helm chart)

You can create a Git repository with a similar file hierarchy. Each file is a template for a Kubernetes resource needed for an ephemeral environment.

ephemeral-environment-helm/
├─ Chart.yaml
├─ values.yaml
├─ values.preview.yaml            # optional per-env defaults (e.g., TTL/resource caps)
├─ values.schema.json             # optional, validates user-provided values
├─ charts/                        # optional, vendored dependencies
├─ templates/
│  ├─ _helpers.tpl               # names/labels templates
│  ├─ configmap.yaml             # non‑secret settings
│  ├─ externalsecret.yaml        # pulls secrets from GSM (or your vault)
│  ├─ serviceaccount.yaml        # workload identity
│  ├─ rbac-role.yaml             # minimal namespace Role
│  ├─ rbac-rolebinding.yaml
│  ├─ ingress.yaml               # GCE Ingress with hosts per app/env
│  ├─ managedcertificate.yaml    # TLS for Ingress hosts (GKE)
│  ├─ service-backend.yaml
│  ├─ deployment-backend.yaml
│  ├─ hpa-backend.yaml           # optional autoscaling
│  ├─ service-frontend1.yaml
│  ├─ deployment-frontend1.yaml
│  ├─ hpa-frontend1.yaml         # optional
│  ├─ service-frontend2.yaml
│  ├─ deployment-frontend2.yaml
│  ├─ hpa-frontend2.yaml         # optional
│  ├─ redis-statefulset.yaml
│  ├─ redis-service.yaml
│  ├─ db1-statefulset.yaml       # ephemeral DB1 (init/fixtures optional)
│  ├─ db1-service.yaml
│  ├─ db2-statefulset.yaml       # ephemeral DB2
│  ├─ db2-service.yaml
│  ├─ cronjob-cleanup.yaml       # TTL enforcement / garbage collection
│  ├─ networkpolicy.yaml         # optional, if using NetworkPolicies
│  └─ NOTES.txt                  # optional Helm install notes
└─ README.md                     # chart overview and values reference

Minimal configuration snippets

Small, copy‑pasteable examples to get a preview running.

values.yaml (minimal)

global:
  ttlMinutes: 30
  labels:
    starry.env: preview-123

ingress:
  enabled: true
  className: gce
  hosts:
    - sample-be.preview-123.starry.mycompany.com
    - sample-fe-1.preview-123.starry.mycompany.com
    - sample-fe-2.preview-123.starry.mycompany.com
  tls:
    managedCertificate: true

backend:
  image:
    repository: us-docker.pkg.dev/myproj/sample-be
    tag: pr-123-abcd123
    pullPolicy: IfNotPresent
  service:
    port: 8080
  env:
    REDIS_URL: redis://starry-redis-master:6379
    DB1_URL: postgres://db1:5432/app
    DB2_URL: postgres://db2:5432/app

frontend1:
  enabled: true
  image:
    repository: us-docker.pkg.dev/myproj/sample-fe-1
    tag: pr-123-abcd123
  service:
    port: 8080
  env:
    VITE_API_BASE_URL: https://sample-be.preview-123.starry.mycompany.com

frontend2:
  enabled: true
  image:
    repository: us-docker.pkg.dev/myproj/sample-fe-2
    tag: pr-123-abcd123
  service:
    port: 8080
  env:
    VITE_API_BASE_URL: https://sample-be.preview-123.starry.mycompany.com

redis:
  enabled: true

databases:
  db1:
    enabled: true
  db2:
    enabled: true

secrets:
  externalSecret:
    enabled: true
    secretStoreRef:
      name: gsm-store
      kind: ClusterSecretStore

values.schema.json (guardrails excerpt)

{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "global": {
      "type": "object",
      "properties": {
        "ttlMinutes": { "type": "integer", "minimum": 5, "maximum": 240 }
      },
      "required": ["ttlMinutes"]
    },
    "backend": {
      "type": "object",
      "properties": {
        "image": {
          "type": "object",
          "properties": {
            "repository": { "type": "string", "minLength": 3 },
            "tag": { "type": "string", "minLength": 5 }
          },
          "required": ["repository", "tag"]
        }
      }
    }
  }
}

Argo CD Application (preview)

Starry can make a request to the Kubernetes API to create an Application resource that will manage a preview environment instance.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: preview-123
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: starry
  source:
    repoURL: https://github.com/myorg/environment-helm.git
    targetRevision: main
    path: .
    helm:
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: preview-123
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

ExternalSecret (GSM)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: preview-123
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gsm-store
    kind: ClusterSecretStore
  target:
    name: app-secrets
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: projects/123/secrets/db-password/versions/latest
    - secretKey: REDIS_PASSWORD
      remoteRef:
        key: projects/123/secrets/redis-password/versions/latest

Security hardening

Workload Identity on ServiceAccounts to access GSM; least‑privilege IAM on secrets.
Namespace‑scoped Roles/RoleBindings; deny cluster‑wide privileges by default.
No secrets in Git; inject via External Secrets only.
Optional NetworkPolicies to confine pod traffic and egress.

Cost and quotas

Defaults: ttlMinutes: 30, HPA minReplicas: 1, narrow requests/limits for density.
Cap concurrent previews per team/namespace with ResourceQuota/LimitRange.
Cleanup CronJob as a watchdog for stragglers.

Observability and troubleshooting

Standard labels (app.kubernetes.io/*, starry.env) and probes for all pods.
Common issues and quick checks:
- Cert Pending: DNS/host mismatch or quota; verify ManagedCertificate status.
- 404 after sync: NEG endpoints warming; check Service → Endpoints and pod readiness.
- Image pull back‑off: tag/registry typo; confirm Artifact Registry permissions.
- RBAC denied: verify namespace Role/RoleBinding and ServiceAccount name.
- Quota exceeded: review ResourceQuota and HPA limits.

Conclusion

By shipping previews as a Helm chart reconciled by Argo CD, we get isolation by default, secure secret handling, fast spin‑up/tear‑down, and a fully auditable GitOps trail. CI stays simple—build and push images on PR branches—while the platform provides predictable URLs, ephemeral data stores, and least‑privilege access.

In future articles, we’ll automate the PR lifecycle end‑to‑end: auto‑create on PR open, auto‑destroy on merge/close, add quotas and policies for cost control, layer optional E2E tests and data seeding for production‑like previews, and discuss Terraform for infrastructure setup.

DEV Community