Daniel Cordeiro

Posted on Jun 2

How Terraform and Helm Split Responsibilities in a Kubernetes CI/CD Pipeline

#kubernetes #devops #terraform #helm

Introduction

Anyone working with Kubernetes for a while will likely face a version of the same question: should Kubernetes resources be managed through Terraform, or through something else?

In addition to provisioning the cluster, Terraform has a mature kubernetes provider that exposes namespaces, Deployments, StatefulSets, and ConfigMaps as first-class resources. Everything can live in the same state file, the same repository, and the same workflow. For teams that already operate Terraform for infrastructure, the case for extending that coverage to application workloads is genuinely strong.

The problem is that infrastructure and application workloads have fundamentally different change rates. A Kubernetes namespace or a monitoring stack might remain untouched for months. A container image changes with every commit. This is the gap Helm was designed to fill: a dedicated tool for packaging, versioning, and deploying application workloads into Kubernetes, with parameterized overrides, release history, and rollback built in.

To make this concrete, this article documents a hands-on project named Personal Blog that explores both scenarios, and how it evolved from the monolithic Stage 2 — where Terraform managed both infrastructure provisioning and application deployment — to the decoupled Stage 3 - a clean separation of concerns where Terraform owns the platform layer and Helm owns the application layer, orchestrated by a GitLab CI/CD pipeline.

The Temptation of "Terraform Everything"

In Stage 2 of the personal blog CI/CD pipeline, a decision was made that seemed reasonable at the time: use Terraform for everything. Not just the cluster — everything. The Kubernetes namespace, MongoDB StatefulSet, backend and frontend Deployments, Services, ConfigMaps for Prometheus and Grafana, the entire monitoring stack. One tool, one state file, one workflow.

Here's what the terraform apply workflow covered in Stage 2:

Provisioning a Kind (Kubernetes in Docker) cluster with the tehcyx/kind provider;
Generating kubeconfig and loading local Docker images into the cluster registry using a Provisioner;
Creating Kubernetes namespace with labels;
Deploying MongoDB as a kubernetes_stateful_set with a PersistentVolumeClaim;
Deploying Spring Boot backend: kubernetes_deployment with 2 replicas, resource limits (500m CPU / 512Mi), NodePort service;
Deploying Angular frontend: kubernetes_deployment with 2 replicas, resource limits (300m CPU / 256Mi), NodePort service;
Deploying Prometheus, Loki, Promtail DaemonSet, and Grafana with pre-configured datasources and dashboards via kubernetes_config_map;

All of this in a single local_kubernetes.tf file — over 1,000 lines.

Look closely at what those 1,000 lines actually contain: Kubernetes Deployments, StatefulSets, Services, ConfigMaps, and DaemonSets — written in HCL syntax instead of YAML. In other words, the file is a collection of implicit Kubernetes manifests embedded inside Terraform. Every resource that would normally be a few lines of YAML becomes a deeply nested HCL block. Each one is hardcoded — image tags, replica counts, resource limits, service ports, environment variables all written inline, duplicated wherever they appear, and entirely specific to one environment.

This is precisely a gap that Helm can fit to close. Without it, you end up with exactly what Stage 2 produced: large, unmaintainable manifests with no reusability across environments, no parameterization strategy, and no mechanism to version or roll back the application as a deployable unit.

It ran. But every time the application changed — a new Docker image, a backend configuration update — the only way to deploy was terraform apply. Which means every code change ran through a tool designed for infrastructure provisioning, not application delivery.

Why Terraform Can Manage Kubernetes Resources at All

Terraform is built around a provider plugin model [1]. A provider is a plugin that translates Terraform resource definitions into API calls for a specific platform. HashiCorp publishes official providers for AWS, Azure, GCP, and Kubernetes. Third parties publish providers for everything else — in this project, the tehcyx/kind provider was used to provision the Kind cluster itself.

The hashicorp/kubernetes provider [2] is what makes managing Kubernetes resources declaratively with Terraform possible. It exposes Kubernetes API objects — Namespaces, Deployments, Services, ConfigMaps, StatefulSets, Secrets, RBAC resources — as first-class Terraform resources [3]. Under the hood, it authenticates against the Kubernetes API server using the configured kubeconfig and issues the equivalent of kubectl apply calls, but managed through Terraform's state model.

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.38.0"
    }
  }
}

provider "kubernetes" {
  config_path = pathexpand(var.kubeconfig_path)
}

From that point on, any Kubernetes object can be declared as a Terraform resource:

resource "kubernetes_deployment" "backend" {
  metadata {
    name      = "personal-blog-backend"
    namespace = kubernetes_namespace.app.metadata[0].name
  }
  spec {
    replicas = 2
    selector {
      match_labels = { app = "personal-blog-backend" }
    }
    template {
      metadata {
        labels = { app = "personal-blog-backend" }
      }
      spec {
        container {
          name  = "backend"
          image = "dancodingbr/personal-blog-backend:latest"
          resources {
            limits   = { cpu = "500m", memory = "512Mi" }
            requests = { cpu = "250m", memory = "256Mi" }
          }
        }
      }
    }
  }
}

This means that Terraform can manage the entire lifecycle of a Kubernetes cluster and the workloads running inside it, from a single configuration and state file.

The Core Problem: Two Different Change Rates

Terraform is designed around the assumption that infrastructure changes infrequently. You create a VPC, a database cluster, a Kubernetes namespace — and those things persist for months or years. Terraform's state model, plan/apply cycle, and provider ecosystem are all optimized for this cadence.

Application deployments have a completely different change rate. A development team might deploy multiple times per day. The image tag changes with every commit. Replica counts get tuned. Environment variables get updated. These are not infrastructure events — they are application lifecycle events.

When you run terraform apply to update an image tag, you're running the full plan/apply cycle — provider initialization, state refresh, dependency graph evaluation — just to change one line in a Deployment spec. It's slow, it carries the risk of unintentional side-effects on other resources in the state file, and it conflates two fundamentally different concerns.

The Solution: Separation of Concerns

Stage 3 of the project refactors this cleanly. Terraform keeps exactly two responsibilities:

Kubernetes namespace creation.
Helm releases [4] for platform-level services: MongoDB and the monitoring stack (Prometheus, Loki, Promtail, Grafana).

# terraform/helm/kubernetes.tf
resource "kubernetes_namespace" "personal_blog_namespace" {
  metadata {
    name   = var.app_namespace
    labels = var.app_labels
  }
}

# terraform/helm/helm.tf
resource "helm_release" "mongodb" {
  name      = "mongodb-release"
  namespace = var.app_namespace
  chart     = "${path.module}/../../charts/mongodb"
  depends_on = [kubernetes_namespace.personal_blog_namespace]
}

resource "helm_release" "monitoring" {
  name      = "monitoring-release"
  namespace = var.app_namespace
  chart     = "${path.module}/../../charts/monitoring"
  depends_on = [kubernetes_namespace.personal_blog_namespace]
}

The frontend and backend are removed from Terraform entirely. They become Helm chart releases managed by the GitLab CI pipeline — deployed with helm upgrade --install and a dynamic image tag override:

helm upgrade --install personal-blog-backend-release \
  ./charts/personal-blog-backend \
  --set image.repository="dancodingbr/personal-blog-backend" \
  --set image.tag="$CI_COMMIT_SHORT_SHA" \
  --namespace personal-blog-app-dev \
  --wait --timeout 90s

In this way, Terraform runs infrequently to provision the platform. Helm runs on every deploy to update the application.

The Result: A Clean Responsibility Matrix

Concern	Tool	Change frequency
Kubernetes namespace	Terraform	Rarely
MongoDB deployment	Terraform + Helm	Rarely
Monitoring stack	Terraform + Helm	Rarely
Backend deployment	Helm (via GitLab CI)	Every commit
Frontend deployment	Helm (via GitLab CI)	Every commit

What Helm Gives You For App Deployments

Kubernetes manifests can become large and repetitive, hard to parameterize across environments, and difficult to version and reuse. A simple application — a Deployment, a Service, a ConfigMap — might span dozens of YAML files, each needing different values for dev, staging, and production. Helm's answer was Charts: reusable, templated application packages with a clean variable model.

The 1,000-line local_kubernetes.tf from Stage 2 is a perfect illustration of what happens when that problem goes unsolved. Every kubernetes_deployment, kubernetes_service, and kubernetes_config_map block in that file is a hardcoded, single-environment, non-reusable Terraform approximation of a Kubernetes manifest. Helm would have expressed the same application as a handful of templated files and a single values.yaml.

The capabilities Helm [5] introduces to solve this are exactly the ones Terraform's Kubernetes provider lacks:

1. Chart packaging and environment parameterization

A Helm chart separates what to deploy (the templates) from how to configure it (the values). The same chart deploys to dev, staging, and production by swapping the values file — no duplication, no environment-specific HCL blocks.

2. Parameterized overrides at deploy time

Helm's values.yaml + --set overrides are designed for the pattern of "here's a base config, override what changes per deploy." In CI/CD, this is how image tags are injected at runtime without modifying source files.

3. Release management

helm list gives you a clean view of what version of each chart is deployed, in which namespace, with which revision. Rolling back is helm rollback <release> <revision>.

4. Immutable, traceable deployments

Using --set image.tag="$CI_COMMIT_SHORT_SHA" means every deployment is tied to a specific Git commit. Combined with helm history, you have a full audit trail.

Key Takeaways

One goal of DevOps is not to minimize the number of tools. It is to give each concern the tool that fits it best. So, when designing a Kubernetes-based deployment pipeline from scratch, the following separation of concerns is suggested:

Terraform for: namespaces, persistent infrastructure services (databases, message queues), RBAC, cluster-level resources.
Helm for: application workloads — anything that has a Deployment, scales independently, and is updated frequently.
CI/CD tool for: sequencing them correctly — run Terraform first (idempotently), then run Helm for the changed application.

Source code: gitlab.com/dancodingbr/personal-blog.