DEV Community

Pavan Kumar Sadashiv
Pavan Kumar Sadashiv

Posted on

Production GitOps with GitLab: Continuous Delivery with the GitLab Agent for Kubernetes

#kubernetes #gitlab #gitops #devops

Production GitOps with GitLab: Continuous Delivery with the GitLab Agent for Kubernetes

For years, continuous delivery to Kubernetes from CI/CD pipelines relied on a push-based model:

  1. You created a long-lived kubeconfig or ServiceAccount token inside your cluster.
  2. You pasted that secret into your CI/CD repository settings.
  3. Your CI/CD runner reached through your firewall into the Kubernetes API server to run kubectl apply.

While push-based deployment is simple to set up, it introduces severe enterprise security liabilities:

  • Cluster Exposure: Your Kubernetes API server must be reachable from the internet or public CI runners.
  • Credential Sprawl: Storing cluster-admin credentials inside CI variables creates a high-value attack vector if a pipeline or dependency is compromised.
  • Configuration Drift: If someone manually changes a pod or service with kubectl edit, the cluster silently drifts away from Git.

To solve this, GitLab introduced the GitLab Agent for Kubernetes (KAS) — a modern, pull-based GitOps engine.

In this guide, I will show you how to configure the GitLab Agent to manage your Kubernetes deployments securely with zero open inbound firewall ports.

1. Push vs. Pull-Based GitOps Architecture 

+-------------------------------------------------------------+
|               TRADITIONAL PUSH-BASED DEPLOYMENT             |
|                                                             |
|   GitLab CI Runner  ---(Exposed API Port 6443)--->  K8s API |
|   * High attack surface; credentials stored in CI runner.   |
+-------------------------------------------------------------+

                              vs.

+-------------------------------------------------------------+
|              PULL-BASED GITOPS (GITLAB AGENT / KAS)         |
|                                                             |
|   GitLab Repository (GitOps Manifests / Kustomize)          |
|                          ^                                  |
|                          | Outbound Secure gRPC/WebSocket   |
|                          | (No inbound firewall openings)   |
|                          v                                  |
|   In-Cluster GitLab Agent (Polls & Syncs State Directly)    |
|   - Reconciles desired state vs. actual cluster state       |
|   - Prevents configuration drift automatically              |
+-------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

2. Setting Up the Agent Configuration in Your Repository

In GitLab, the Agent for Kubernetes is configured entirely via version-controlled files inside your repository.

Create the configuration file at the mandatory path:
.gitlab/agents/<agent-name>/config.yaml

# .gitlab/agents/production-agent/config.yaml
gitops:
  manifest_projects:
    - id: "my-org/cloud-infrastructure"
      paths:
        - glob: "manifests/**/*.{yaml,yml}"
      reconcile_timeout: "5m"
      prune: true # Automatically deletes resources removed from Git!

ci_access:
  projects:
    - id: "my-org/microservices-app"
Enter fullscreen mode Exit fullscreen mode

Key Parameters:

  • prune: true: Ensures declarative synchronization. If you delete a deployment.yaml from Git, the agent automatically deletes the deployment from the cluster.
  • ci_access: Allows authorized GitLab CI pipelines to securely interact with the cluster via the agent without storing raw cluster credentials.

3. Installing the Agent Inside Your Kubernetes Cluster

Step A: Register the Agent in GitLab UI

  1. Navigate to your GitLab project $\rightarrow$ Operate $\rightarrow$ Kubernetes clusters.
  2. Click Connect a cluster (agent).
  3. Select your agent name (production-agent) from the dropdown.
  4. GitLab displays a registration token and an auto-generated Helm command.

Step B: Install via Helm

Run the Helm command directly inside your cluster (or via your local kubectl admin context):

helm repo add gitlab https://charts.gitlab.io
helm repo update

helm upgrade --install production-agent gitlab/gitlab-agent \
  --namespace gitlab-agent-production \
  --create-namespace \
  --set image.tag="v17.5.0" \
  --set config.token="YOUR_AGENT_REGISTRATION_TOKEN" \
  --set config.kasAddress="wss://kas.gitlab.com"
Enter fullscreen mode Exit fullscreen mode

Within seconds, the agent pod connects outbound to GitLab over secure WebSockets (wss://). In your GitLab dashboard, the cluster status instantly turns Connected (Green).

4. Declarative Synchronization in Action

Create a simple deployment manifest in your Git repository under manifests/api-service.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
  namespace: production
  labels:
    app: api-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
    spec:
      containers:
        - name: web
          image: registry.gitlab.com/my-org/microservices-app:v2.1.0
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: "500m"
              memory: "512Mi"
            requests:
              cpu: "100m"
              memory: "128Mi"
Enter fullscreen mode Exit fullscreen mode

Commit and push this file to main:

git add manifests/api-service.yaml
git commit -m "feat(deploy): provision api-service with 3 replicas"
git push origin main
Enter fullscreen mode Exit fullscreen mode

The GitLab Agent detects the commit within seconds, validates the manifest, and applies it to the cluster.

5. Drift Detection & Self-Healing

What happens if an engineer enters the cluster and accidentally deletes a pod or modifies replicas?

# Simulating accidental drift
kubectl scale deployment api-service -n production --replicas=1
Enter fullscreen mode Exit fullscreen mode

Because GitOps treats Git as the single source of truth, the GitLab Agent detects that the cluster state (replicas: 1) diverges from Git (replicas: 3).

Within the configured reconcile_timeout window, the agent automatically scales the deployment back to 3 replicas, self-healing the cluster without human intervention!

6. Secure CI/CD Pipeline Access (Hybrid GitOps)

Sometimes a CI/CD job needs to run an imperative command against the cluster (like executing database migrations or running integration smoke tests).

With the agent configured under ci_access, your .gitlab-ci.yml can interact with the cluster securely:

deploy_migration:
  stage: deploy
  image:
    name: bitnami/kubectl:latest
    entrypoint: [""]
  script:
    # Set context to the registered GitLab Agent
    - kubectl config get-contexts
    - kubectl config use-context my-org/cloud-infrastructure:production-agent
    - kubectl run db-migration --image=registry.gitlab.com/my-org/db-tools:latest --restart=Never
Enter fullscreen mode Exit fullscreen mode

No tokens are exposed in pipeline logs, and permissions are strictly bounded by Kubernetes RBAC.

7. Key Takeaways

  1. Zero Inbound Attack Surface: The agent connects outbound over WebSockets (wss://), allowing private clusters to remain completely shielded behind firewalls.
  2. Immutable Git Audit Trail: Every deployment, scale event, or configuration change is tracked in Git commits with cryptographic signatures.
  3. Automated Self-Healing: Configuration drift is reconciled automatically to match Git.

Resources & Links

Are you using push or pull-based deployment for your Kubernetes clusters? Share your architecture in the comments below!

Top comments (0)