DEV Community: George Lukas

Chapter 4: GitOps with Terraform + ArgoCD — Self-Hosting LLMs as a Platform Product

George Lukas — Fri, 06 Mar 2026 19:25:15 +0000

Each chapter in this series has been a deliberate step toward removing friction from LLM infrastructure management. Chapter 2 introduced Infrastructure as Code by mapping Kubernetes resources directly to Terraform — functional, but painfully verbose. Chapter 3 resolved the abstraction problem by bringing in the Helm provider, collapsing 500 lines of HCL into 50 and allowing Terraform to reason about applications rather than individual resources. Both approaches, however, shared the same fundamental constraint: every change still required a human to run terraform apply. The cluster had no awareness of Git, drift went undetected until someone noticed, and scaling that model to larger teams or more frequent deploys would inevitably make manual execution a bottleneck. Chapter 4 closes that loop by introducing GitOps with ArgoCD — making the cluster itself responsible for continuously reconciling its state against Git, without anyone needing to trigger a command.

The Four Principles of GitOps

GitOps is an operational paradigm built on four interconnected principles:

1. Declarative
Everything the cluster needs to run is expressed as YAML files. There is no procedural "run this command" — only a description of the desired end state.

2. Versioned
All state lives in Git. Every change has an author, a timestamp, and a diff. Rollback becomes as simple as reverting a commit.

3. Pull-based (key differentiator)

Cap 3: Developer → terraform apply → push → Cluster
Cap 4: Developer → git push → Git ← poll ← ArgoCD (no cluster)

4. Continuous reconciliation
ArgoCD runs an infinite loop:

while true:
  git_state = fetch_from_git()
  cluster_state = fetch_from_kubernetes()
  if git_state != cluster_state:
    apply_changes()
  sleep(180)  # 3 minutes

Context and Rationale

With the mechanics of GitOps established, it is worth stepping back to understand why this shift matters beyond a single team running git push. Removing terraform apply from the workflow is only the surface benefit — the deeper payoff is that GitOps creates the operational foundation for treating infrastructure as a product, with real ownership, roadmaps, and SLAs.

Growing Cloud Complexity

Cloud infrastructure promised simplicity. In 2010, AWS offered ~20 intuitive services, provisioned with clicks in the UI. The value proposition was clear: simplify infrastructure.

Today, reality is different:

AWS (2010):
├── ~20 services
├── Provisioning via UI
└── Simplicity as core value

AWS (2025):
├── 175+ services
├── Provisioning via IaC
└── Complexity as new reality

Organizational impact:

According to Max Griffiths (Thoughtworks):

"Rising cloud complexity is putting many organizations and their infrastructure teams right back to where they were 15 years ago — struggling to keep up with demand for new services and instances, and stay on top of an increasingly unmanageable infrastructure footprint."

Consequences:

Time to market increased (instead of decreasing)
Required skills grew exponentially
Self-service became impractical
Full-stack developer scope expanded to the point of becoming a disadvantage

Infrastructure as Product

Infrastructure as Product treats infrastructure not as a centralized service provider, but as a portfolio of internal products that enable product teams to deliver value quickly.

Paradigm shift:

Traditional (project-based):
Developer → Ticket → Ops Team → Wait → Provision → Deploy
Timeline: Days/weeks
Bottleneck: Ops team on critical path

Infrastructure as Product:
Developer → Self-service Platform → Provision → Deploy
Timeline: Minutes/hours
Enabler: Platform removes friction

Three Core Principles

1. Developer as Customer

Infrastructure should be designed around developer experience first. If the platform is hard to use, it won't be adopted — regardless of how technically sound it is.

Success metrics:

Developer satisfaction score
Time to provision (target: <30 min)
Platform adoption rate
Support tickets reduction

Anti-pattern:

# Requires learning a new language just to provision
developer:
  must_learn: [HCL, Pulumi, CloudFormation]
  to_do: "Same work as before"
  result: "Poor developer experience"

Correct pattern:

# Familiar interface, correct abstraction
developer:
  uses: [YAML, familiar APIs]
  self_service: true
  result: "High adoption, low friction"

2. Platform Teams With a Product Mindset

According to Sebastian Straube (Accenture), infrastructure teams shouldn't operate as a shared service that reacts to tickets — they should be restructured into dedicated platform product teams, each owning a slice of the internal platform with a roadmap, SLAs, and real accountability to their users.

Organization example:

Platform Team: Compute & Container
├── Product: Kubernetes-as-a-Service
├── Customers: All development teams
├── Roadmap: 
│   ├── Q1: Auto-scaling GPU nodes
│   ├── Q2: Service mesh integration
│   └── Q3: Cost optimization
├── SLA: 99.9% uptime, <5min provision
└── Metrics: Adoption rate, developer satisfaction

Platform Team: Observability
├── Product: Unified logs/metrics/traces
├── Customers: All teams (dev + ops)
├── Roadmap:
│   ├── Q1: 30-day retention
│   ├── Q2: AIOps integration
│   └── Q3: Cost attribution
├── SLA: <5s query response
└── Metrics: Query volume, alert accuracy

Characteristics:

Long-lived teams (not project-based)
Own product roadmap and backlog
Accountable for adoption and satisfaction
Measure success by customer outcomes

3. Self-Service as a Core Value

The platform must enable true self-service — meaning the platform team is never on the critical path of a deployment.

Anti-pattern:

Developer → Submit PR → Platform review → Approve → Deploy
Problem: Platform team on critical path (bottleneck)

Correct pattern:

Developer → Use platform API → Automated validation → Deploy
Platform: Monitor outcomes

Real-world example:

# Developer workflow
kubectl apply -f app.yaml

# Platform validates automatically:
# ✓ Security policies (OPA)
# ✓ Resource quotas
# ✓ Network policies
# ✓ Deploy without manual approval

GitOps as the Technical Foundation

GitOps is not just "automation" — it is the enabling layer that makes Infrastructure as Product operationally viable. Each of its four properties maps directly to a platform capability:

1. Declarative = Product Catalog

The Git repository becomes a catalog of available products that any team can consume, customize, and deploy independently.

k8s-apps/
├── apps/ollama/          # Product: LLM Inference
├── apps/librechat/       # Product: Chat Interface
└── apps/postgres/        # Product: Database

Developer "consumes" a product via:

git clone k8s-apps
cp -r apps/ollama apps/my-llm
vim apps/my-llm/values.yaml  # Customize
git commit && git push
# ArgoCD auto-deploy

2. Self-Service = Developer Autonomy

ArgoCD removes the platform team from the critical path:

Without ArgoCD:
Developer → Code → Request deploy → Platform team → Manual deploy

With ArgoCD:
Developer → Code → Git push → ArgoCD auto-sync → Deployed

3. API-Driven = Programmatic Access

ArgoCD Application CRDs are the deployment API:

# Developer creates workload via Kubernetes API
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-service
spec:
  source:
    repoURL: https://github.com/company/k8s-apps
    path: apps/my-service
  destination:
    server: https://kubernetes.default.svc
    namespace: my-namespace
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

4. Standardization With Flexibility

Helm charts serve as reusable product templates — the platform defines the structure, teams supply the configuration.

Platform provide templates:
├── web-app/          # Template for web apps
├── ml-service/       # Template for ML workloads
└── data-pipeline/    # Template for ETL

Developer customize:
cp -r templates/web-app apps/my-app
vim apps/my-app/values.yaml  # Configuration only
git push  # Automatic deployment

Architectural Decision

App of Apps with Independent Wrappers

Rather than managing all services as a single unit, this architecture treats each service as an independent package within the App of Apps pattern. GitOps manages deployments, versions, and updates at the individual service level — not the stack level.

Each service (Ollama, LibreChat) has its own Helm chart wrapper that:

References the upstream chart via dependencies in Chart.yaml
Customizes configuration via a local values.yaml
Maintains independent versioning
Allows isolated evolution

Wrapper Structure

apps/
├── librechat/
│   ├── Chart.yaml      # Wrapper chart with dependency
│   └── values.yaml     # Specific configuration
└── ollama/
    ├── Chart.yaml      # Wrapper chart with dependency
    └── values.yaml     # Specific configuration

Chart.yaml defines the dependency:

dependencies:
  - name: ollama
    version: "1.42.0"                              # Fixed version
    repository: "https://otwld.github.io/ollama-helm/"

values.yaml customizes the upstream chart:

ollama:        # Wrapper namespace
  ollama:      # Chart namespace (double hierarchy)
    gpu:
      enabled: true
    models:
      pull:
        - llama3.2:3b

Benefits of This Approach

1. Service Independence

Each app evolves at its own pace
Deploying Ollama does not affect LibreChat
Reduces the risk of cross-service regressions

2. Granular Versioning

apps/ollama/Chart.yaml:      version: "1.42.0"
apps/librechat/Chart.yaml:   version: "1.9.7"

Upstream chart versions pinned individually
Upgrades tested and applied per service
Independent rollback per application

3. Isolated Customization

Values specific per service
No configuration conflicts
Individual testability

4. Per-Service Observability

ArgoCD Application per service
Isolated logs and events
Specific health checking

5. Automated Deployment

Git commit → ArgoCD detects → Helm processes wrapper → Deploy to cluster

ArgoCD manages each wrapper as an Application
Automatic sync per service
Independent self-healing

6. Complete Tracking

git log apps/ollama/values.yaml
# Complete change history for Ollama

git log apps/librechat/values.yaml
# Complete change history for LibreChat

Audit trail per service
Specific PRs per change
Granular rollback via Git

Types of Wrappers

Wrapper with Public Chart (HTTP):

# apps/ollama/Chart.yaml
dependencies:
  - name: ollama
    version: "1.42.0"
    repository: "https://otwld.github.io/ollama-helm/"

Wrapper with OCI Chart:

# apps/librechat/Chart.yaml
dependencies:
  - name: librechat
    version: "1.9.7"
    repository: "oci://ghcr.io/danny-avila/librechat-chart"

Wrapper with Sub-charts:

# apps/custom-app/Chart.yaml
dependencies:
  - name: app
    version: "1.0.0"
    repository: "https://..."
  - name: postgres
    version: "12.0.0"
    repository: "https://charts.bitnami.com/bitnami"
  - name: redis
    version: "17.0.0"
    repository: "https://charts.bitnami.com/bitnami"

Governance and Standardization

No tight coupling:

Each service maintains flexibility
Standards enforced via linting (CI/CD)
Common templates available, not mandatory

Example of suggested standard:

# All services follow label convention
metadata:
  labels:
    app.kubernetes.io/name: {{ .Chart.Name }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/version: {{ .Chart.AppVersion }}
    app.kubernetes.io/managed-by: argocd

But each service can add specific labels without breaking others.

Alternatives Considered and Rejected

❌ Single monorepo without wrappers:

# values.yaml (monolítico)
ollama:
  gpu: ...
librechat:
  config: ...
postgres:
  ...

Problems:

A change in Ollama affects LibreChat's versioning
Atomic all-or-nothing deployment
Rollback impacts all services

❌ Single custom chart:

# custom-chart/
templates/
  ollama.yaml
  librechat.yaml
  postgres.yaml

Problems:

Reinventing the wheel (official charts already exist)
Heavy maintenance burden
Complicates upstream upgrades

✅ Independent wrappers (chosen):

Reuses upstream charts
Independence between services
Easy maintenance
Flexible governance

Trade-offs

Advantages:

Full independence between services
Granular versioning
Individual rollback
Isolated observability
Scalability (adding an app = new directory)

Disadvantages:

Duplication of common configurations (e.g., ingress annotations)
Requires linting to enforce standards
More files to manage

Considerations

Use when:

Medium/large team (5+ people)
Multiple independent services
Frequent deploys per service
Need for granular rollback
Distributed governance (teams own services)

Do not use when:

Monolithic application (single service)
Very small team (1–2 people)
All services always deployed together
Preference for extreme simplicity

Implementing Terraform

This section implements main.tf. It does everything Terraform is responsible for in this architecture — which is intentionally narrow. Rather than managing applications directly, main.tf bootstraps the platform once and then hands control to ArgoCD. Concretely, it does four things in sequence: creates the three namespaces, provisions the credentials secret that LibreChat needs at runtime, installs ArgoCD via Helm, and registers the two ArgoCD Application CRDs that point at the Git repository. After that first terraform apply, Terraform is largely out of the picture — all subsequent application changes flow through Git.

The file is organised into four logical blocks, each separated by a comment header:

main.tf
├── Providers          — Kubernetes + Helm provider config
├── Infrastructure     — Namespaces + credentials secret
├── ArgoCD             — Helm installation + values reference
└── ArgoCD Applications — CRDs that register Ollama and LibreChat

The subsections below walk through each block in order.

Providers

(main.tf lines 1–27 — terraform {} block + provider declarations)

terraform {
  required_version = ">= 1.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "minikube"
}

provider "helm" {
  kubernetes {
    config_path    = "~/.kube/config"
    config_context = "minikube"
  }
}

The provider configuration is identical to Chapter 3: both the kubernetes and helm providers read from ~/.kube/config and target the minikube context. No changes needed here.

Namespaces

(main.tf — INFRASTRUCTURE block)

resource "kubernetes_namespace" "argocd" {
  metadata {
    name = "argocd"
    labels = {
      managed-by = "terraform"
      purpose    = "gitops"
    }
  }
}

resource "kubernetes_namespace" "ollama" {
  metadata {
    name = "ollama"
    labels = {
      managed-by = "terraform"
    }
  }
}

resource "kubernetes_namespace" "librechat" {
  metadata {
    name = "librechat"
    labels = {
      managed-by = "terraform"
    }
  }
}

Design decision: Namespaces are managed by Terraform rather than ArgoCD because they are prerequisites for everything else — ArgoCD itself cannot deploy into a namespace that does not yet exist. They also change rarely enough that the overhead of GitOps reconciliation is not justified. The purpose = "gitops" label distinguishes tooling namespaces from application namespaces at a glance.

Infrastructure Secrets

(main.tf — still inside the INFRASTRUCTURE block, immediately after the namespaces)

resource "kubernetes_secret" "librechat_credentials" {
  metadata {
    name      = "librechat-credentials-env"
    namespace = kubernetes_namespace.librechat.metadata[0].name
  }

  data = {
    JWT_SECRET         = var.jwt_secret
    JWT_REFRESH_SECRET = var.jwt_refresh_secret
    CREDS_KEY          = var.creds_key
    CREDS_IV           = var.creds_iv
    MONGO_URI          = "mongodb://librechat-mongodb:27017/LibreChat"
    MEILI_HOST         = "http://librechat-meilisearch:7700"
    OLLAMA_BASE_URL    = "http://ollama.ollama.svc.cluster.local:11434"
  }

  type = "Opaque"
}

Design decision: Credentials are managed by Terraform precisely because they must not appear in Git. Committing plaintext secrets to a repository — even a private one — violates the principle of least exposure. For production environments, the right path is to replace this with Sealed Secrets, External Secrets Operator, or HashiCorp Vault, all of which allow secrets to live in Git in an encrypted or referenced form. Chapter 5 covers this.

ArgoCD Installation

(main.tf — ARGOCD block)

resource "helm_release" "argocd" {
  name       = "argocd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  namespace  = kubernetes_namespace.argocd.metadata[0].name
  version    = "5.51.6"

  values = [
    file("${path.module}/values/argocd-values.yaml")
  ]

  timeout       = 600
  wait          = true
  wait_for_jobs = true

  depends_on = [
    kubernetes_namespace.argocd
  ]
}

ArgoCD Applications (CRDs)

(main.tf — ARGOCD APPLICATIONS block — the last block in the file)

With ArgoCD installed, the final step is registering the applications it should manage. Each kubernetes_manifest block below creates an ArgoCD Application CRD — a declarative instruction telling ArgoCD which Git path to watch, which cluster namespace to deploy into, and how to handle sync and drift.

Ollama:

resource "kubernetes_manifest" "argocd_app_ollama" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name      = "ollama"
      namespace = kubernetes_namespace.argocd.metadata[0].name
      labels = {
        managed-by = "terraform"
      }
    }
    spec = {
      project = "default"

      source = {
        repoURL        = var.git_repo_url
        targetRevision = var.git_branch
        path           = "apps/ollama"
        helm = {
          valueFiles = ["values.yaml"]
        }
      }

      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = kubernetes_namespace.ollama.metadata[0].name
      }

      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
        syncOptions = [
          "CreateNamespace=false"
        ]
      }
    }
  }

  depends_on = [
    helm_release.argocd,
    kubernetes_namespace.ollama
  ]
}

Breaking Down:

spec.source:

source = {
  repoURL        = "https://github.com/<user>/k8s-apps.git"
  targetRevision = "main"
  path           = "apps/ollama"
  helm = {
    valueFiles = ["values.yaml"]
  }
}

repoURL: Git repository to watch

targetRevision: Branch, tag, or specific SHA

path: Directory within the repo

helm.valueFiles: Array of values files (merged in order)

spec.destination:

destination = {
  server    = "https://kubernetes.default.svc"
  namespace = "ollama"
}

server: API server of the target cluster

kubernetes.default.svc: Local cluster (where ArgoCD is)
External URL: Deploy to a remote cluster

namespace: Destination namespace in the cluster

spec.syncPolicy:

syncPolicy = {
  automated = {
    prune    = true
    selfHeal = true
  }
  syncOptions = [
    "CreateNamespace=false"
  ]
}

automated:

Present: ArgoCD syncs automatically upon detecting changes
Absent: Manual sync only

prune: true:

Resources deleted from Git are deleted from the cluster
false: Orphaned resources remain in the cluster

selfHeal: true:

Manual changes in the cluster are reverted
ArgoCD forces state = Git
false: Manual changes persist (drift)

syncOptions:

CreateNamespace=false: Do not create namespace (already exists via Terraform)
CreateNamespace=true: Create if it doesn't exist
Validate=false: Skip resource validation
PruneLast=true: Delete orphaned resources last

LibreChat:

resource "kubernetes_manifest" "argocd_app_librechat" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name      = "librechat"
      namespace = kubernetes_namespace.argocd.metadata[0].name
      labels = {
        managed-by = "terraform"
      }
    }
    spec = {
      project = "default"

      source = {
        repoURL        = var.git_repo_url
        targetRevision = var.git_branch
        path           = "apps/librechat"
        helm = {
          valueFiles = ["values.yaml"]
        }
      }

      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = kubernetes_namespace.librechat.metadata[0].name
      }

      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
        syncOptions = [
          "CreateNamespace=false"
        ]
      }
    }
  }

  depends_on = [
    helm_release.argocd,
    kubernetes_namespace.librechat,
    kubernetes_secret.librechat_credentials
  ]
}

The LibreChat Application CRD follows the same structure as Ollama. The only meaningful difference is that it carries an additional depends_on reference to kubernetes_secret.librechat_credentials, ensuring the credentials exist in the cluster before ArgoCD attempts its first sync.

Outputs

(main.tf — OUTPUTS block)

output "argocd_url" {
  value = "http://argocd.glukas.space"
}

output "argocd_admin_password" {
  value     = try(data.kubernetes_secret.argocd_initial_admin.data["password"], "")
  sensitive = true
}

output "argocd_password_command" {
  value = "minikube kubectl -- -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d"
}

output "applications_managed" {
  value = {
    ollama = {
      namespace = kubernetes_namespace.ollama.metadata[0].name
      status    = "Managed by ArgoCD"
    }
    librechat = {
      namespace = kubernetes_namespace.librechat.metadata[0].name
      status    = "Managed by ArgoCD"
    }
  }
}

try() function:

try(expression, fallback)

Attempts to execute expression, returns fallback if it fails. Avoids errors when the secret does not yet exist.

With the outputs block in place, main.tf is complete. The full file now captures the entire platform bootstrap in roughly 200 lines: provider config, namespaces, one credentials secret, one Helm release, two ArgoCD Application CRDs, and four outputs. Running terraform apply once produces a live ArgoCD instance watching your Git repository — everything after that point is GitOps.

Implementing Application Charts

The Git repository that ArgoCD watches is separate from the Terraform code. It contains only the Helm wrapper charts — one directory per application — and nothing else. This clean separation means developers working on application configuration never need to touch Terraform, and the platform team can evolve the infrastructure layer independently.

Structure

├── apps
│   ├── librechat
│   │   ├── Chart.yaml
│   │   └── values.yaml
│   └── ollama
│       ├── Chart.yaml
│       └── values.yaml
├── main.tf
├── values
│   └── argocd-values.yaml
└── variables.tf

ArgoCD Configuration (values/argocd-values.yaml)

global:
  domain: argocd.glukas.space

server:
  service:
    type: ClusterIP

  extraArgs:
    - --insecure  # HTTP only — no TLS (development only)

  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - argocd.glukas.space
    paths:
      - /
    pathType: Prefix
    tls: []

  config:
    timeout.reconciliation: 180s  # Polling interval

controller:
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

repoServer:
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 200m
      memory: 256Mi

applicationSet:
  enabled: true

notifications:
  enabled: false

dex:
  enabled: false

Critical parameters:

timeout.reconciliation: 180s

Git polling frequency
Trade-off: Lower value = faster detection, higher load
Recommendation: 180s (3 min) for most cases

extraArgs: [--insecure]

DEVELOPMENT ONLY
Production must use TLS with valid certificates

resources.limits

Controller: Component responsible for reconciliation
RepoServer: Component that reads Git
Values for ~10 applications; scale as needed

Ollama Configuration

# Chart.yaml
apiVersion: v2
name: ollama
description: Ollama deployment managed by ArgoCD
type: application
version: 1.0.0

dependencies:
  - name: ollama
    version: "1.42.0"
    repository: "https://otwld.github.io/ollama-helm/"

Fields:

apiVersion: v2: Helm 3 API version

name: Wrapper chart name

type: application: Deployable chart (vs library)

version: Wrapper version (local control)

dependencies:

name: ollama: Dependency name

Automatically creates namespace .Values.ollama

version: "1.42.0": Fixed version of the upstream chart

CRITICAL: Always pin the version
Avoid: version: "*" or no version

repository: Chart source

Ollama: values.yaml — Analysing its Double Hierarchy

ollama:  # Layer 1: dependency namespace (created automatically by Helm)

  ollama:  # Layer 2: chart's internal namespace
    gpu:
      enabled: true
      type: nvidia
      number: 1

    models:
      pull:
        - llama3.2:3b
        - deepseek-r1:14b

    ingress:
      enabled: true
      className: nginx
      annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: "0"
        nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
        nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
        nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
      hosts:
        - host: ollama.glukas.space
          paths:
            - path: /
              pathType: Prefix
      tls: []

  service:
    type: ClusterIP
    port: 11434

  resources:
    requests:
      memory: "2Gi"
      cpu: "500m"
    limits:
      memory: "4Gi"
      cpu: "2000m"

Why Double Hierarchy?

This is one of the most common silent failures when working with Helm wrapper charts. The configuration appears to apply correctly — ArgoCD reports Synced, no errors surface — but the behaviour you expected (GPU enabled, specific models pulled) simply does not materialise. Understanding the mechanism once prevents hours of debugging later.

Helm automatically creates a value namespace when processing a dependency:

dependencies:
  - name: ollama

As a result, .Values.ollama is automatically created.

Upstream chart has an internal namespace:

helm show values https://otwld.github.io/ollama-helm/ollama

Output:

ollama:  # Chart internal namespace
  gpu:
    enabled: false

Consequence:

Wrapper creates:     .Values.ollama
Chart expects:     .Values.ollama.xxx
Result:  .Values.ollama.ollama.xxx

Technical rules:

Every dependency creates a namespace

dependencies:
  - name: X

Helm always creates .Values.X

Some charts have an internal namespace

If the first line is the chart name, there is an internal namespace.

Combination = Duplication

Layer	Origin	Path
1	Wrapper	`.Values.ollama`
2	Chart	`.Values.ollama`
Final	-	`.Values.ollama.ollama`

Validation:

# Verify internal namespace
helm show values <repo>/<chart>

# Render locally
helm template test apps/ollama/

# Search for specific configuration
helm template test apps/ollama/ | grep -A5 "nvidia.com/gpu"

Common mistake:

# ❌ INCORRECT (only one layer)
ollama:
  gpu:
    enabled: true

Result:

Helm looks for: .Values.ollama.ollama.gpu.enabled
Finds: .Values.ollama.gpu.enabled
Uses default: gpu.enabled: false
Symptom: No error, but GPU not enabled

Solution:

# ✅ CORRECT (double layer)
ollama:
  ollama:
    gpu:
      enabled: true

Specific Configurations

GPU:

ollama:
  ollama:
    gpu:
      enabled: true
      type: nvidia      # Alternativa: amd (ROCm)
      number: 1

Generates:

resources:
  limits:
    nvidia.com/gpu: "1"

nodeSelector:
  nvidia.com/gpu: "true"

tolerations:
  - key: nvidia.com/gpu
    operator: Exists

Models:

models:
  pull:
    - llama3.2:3b
    - deepseek-r1:14b

Chart creates an init container:

initContainers:
  - name: pull-models
    command:
      - /bin/sh
      - -c
      - |
        ollama pull llama3.2:3b
        ollama pull deepseek-r1:14b

Ingress annotations:

annotations:
  nginx.ingress.kubernetes.io/proxy-body-size: "0"
  nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "600"

proxy-body-size: "0": No upload limit (models are large)

proxy-read-timeout: "600": 10min timeout (long inference)

proxy-send-timeout: "600": 10min timeout

Service and Resources (root layer):

Note: service and resources sit in the first ollama layer, not the second. The upstream chart does not expose these fields within its internal namespace, so they must be set at the dependency root level — not nested inside the chart's own namespace.

Full structure:

ollama:              # dependency namespace
  ollama:            # chart's internal namespace
    gpu: ...
    models: ...
    ingress: ...

  service: ...       # root level
  resources: ...     # root level

LibreChat Configuration

Chart.yaml

apiVersion: v2
name: librechat
description: LibreChat deployment managed by ArgoCD
type: application
version: 1.0.0

dependencies:
  - name: librechat
    version: "1.9.7"
    repository: "oci://ghcr.io/danny-avila/librechat-chart"

OCI Repository:

repository: "oci://ghcr.io/danny-avila/librechat-chart"

Syntax: oci://<registry>/<owner>/<chart>

Differences vs. HTTP repository:

Uses the same container registry infrastructure
Faster pull
Better integrated versioning

values.yaml

librechat:  # Layer 1: dependency namespace

  librechat:  # Layer 2: chart's internal namespace (double hierarchy)

    configEnv:
      APP_TITLE: "LibreChat + Ollama (via Terraform)"
      HOST: "0.0.0.0"
      PORT: "3080"
      SEARCH: "true"
      MONGO_URI: "mongodb://librechat-mongodb:27017/LibreChat"
      MEILI_HOST: "http://librechat-meilisearch:7700"
      ALLOW_EMAIL_LOGIN: "true"
      ALLOW_REGISTRATION: "true"
      ALLOW_SOCIAL_LOGIN: "false"
      ALLOW_SOCIAL_REGISTRATION: "false"

    configYamlContent: |
      version: 1.1.5
      cache: true
      endpoints:
        custom:
          - name: "Ollama"
            apiKey: "ollama"
            baseURL: "http://ollama.ollama.svc.cluster.local:11434/v1"
            models:
              default:
                - "llama2:latest"
              fetch: true
            titleConvo: true
            titleModel: "current_model"
            modelDisplayLabel: "Ollama"

  ingress:
    enabled: true
    className: "nginx"
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "25m"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
      nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    hosts:
      - host: librechat.glukas.space
        paths:
          - path: /
            pathType: Prefix
    tls: []

  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "1Gi"
      cpu: "500m"

  persistence:
    enabled: true
    size: 5Gi
    storageClass: "standard"

  replicaCount: 1

  mongodb:
    enabled: true
    image:
      registry: docker.io
      repository: bitnami/mongodb
      tag: "latest"
      pullPolicy: IfNotPresent
    auth:
      enabled: false
    persistence:
      enabled: true
      size: 8Gi
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
      limits:
        memory: "1Gi"
        cpu: "500m"

  meilisearch:
    enabled: true
    auth:
      enabled: false
    environment:
      MEILI_NO_ANALYTICS: "true"
      MEILI_ENV: "development"
    persistence:
      enabled: true
      size: 1Gi
    resources:
      requests:
        memory: "128Mi"
        cpu: "50m"
      limits:
        memory: "512Mi"
        cpu: "250m"

Breaking down:

configEnv:
Environment variables converted to:

env:
  - name: APP_TITLE
    value: "LibreChat + Ollama (via Terraform)"

configYamlContent:
Multi-line YAML (pipe |) written into a ConfigMap and mounted as a file.

Helm processes:

configYamlContent: | ...

Creates:

apiVersion: v1
kind: ConfigMap
metadata:
  name: librechat-config
data:
  librechat.yaml: |
    version: 1.1.5
    cache: true
    ...

Mounts in:

volumeMounts:
  - name: config
    mountPath: /app/librechat.yaml
    subPath: librechat.yaml

Sub-charts (mongodb, meilisearch):

Located in the first librechat layer, not the second.

Structure:

librechat:           # dependency namespace
  librechat:         # chart's internal namespace
    configEnv: ...
    configYamlContent: ...

  mongodb: ...       # sub-chart (root level)
  meilisearch: ...   # sub-chart (root level)
  ingress: ...       # root level
  resources: ...     # root level

.gitignore

# Helm
charts/
Chart.lock

# Secrets
*-secrets.yaml
*.secret.yaml

# Backups
*.bak
*.tmp

# IDE
.vscode/
.idea/
*.swp

Prevents accidental commits of:

Downloaded charts (regeneratable)
Plaintext secrets
Temporary files

Deployment

Prerequisites

# Cluster Minikube
minikube start \
  --driver docker \
  --container-runtime docker \
  --gpus all \
  --memory 8192 \
  --cpus 4

minikube addons enable ingress

# Local DNS
echo "$(minikube ip) ollama.glukas.space" | sudo tee -a /etc/hosts
echo "$(minikube ip) librechat.glukas.space" | sudo tee -a /etc/hosts
echo "$(minikube ip) argocd.glukas.space" | sudo tee -a /etc/hosts

# Git repository created and populated
git clone https://github.com/usuario/k8s-apps.git
cd k8s-apps
# Copy Chart.yaml and values.yaml to apps/ollama/ and apps/librechat/
git add .
git commit -m "Initial commit"
git push origin main

Terraform: Configuration

# terraform.tfvars
cat > terraform.tfvars <<EOF
git_repo_url = "https://github.com/usuario/k8s-apps.git"
git_branch   = "main"

jwt_secret         = "$(openssl rand -hex 32)"
jwt_refresh_secret = "$(openssl rand -hex 32)"
creds_key          = "$(openssl rand -hex 32)"
creds_iv           = "$(openssl rand -hex 16)"
EOF

# .gitignore
echo "terraform.tfvars" >> .gitignore

Terraform: Init

cd terraform/
terraform init

Output:

Initializing provider plugins...
- Installing hashicorp/kubernetes v2.23.0...
- Installing hashicorp/helm v2.11.0...

Terraform has been successfully initialized!

Terraform: Plan

terraform plan -out=tfplan

Planned resources:

Plan: 7 to add, 0 to change, 0 to destroy.

Resources:
  + kubernetes_namespace.argocd
  + kubernetes_namespace.ollama
  + kubernetes_namespace.librechat
  + kubernetes_secret.librechat_credentials
  + helm_release.argocd
  + kubernetes_manifest.argocd_app_ollama
  + kubernetes_manifest.argocd_app_librechat

Terraform: Apply

terraform apply tfplan

Timeline:

[00:00-00:02] Namespaces created
[00:02-00:03] Secret created
[00:03-01:06] ArgoCD installed (Helm chart deployment)
[01:06-01:07] ArgoCD Applications registered (CRDs)

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Note: Terraform only creates the platform. Apps will be deployed by ArgoCD.

ArgoCD: Initial Access

# Get password
ARGOCD_PASSWORD=$(minikube kubectl -- -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d)
echo "URL: http://argocd.glukas.space"
echo "User: admin"
echo "Pass: $ARGOCD_PASSWORD"

Access the UI:

Initial state:

Applications:
  ollama      Status: Syncing...
  librechat   Status: OutOfSync

ArgoCD: Sync Process

ArgoCD runs automatically:

1. Clone the repository:

git clone https://github.com/usuario/k8s-apps.git
git checkout main

2. Change detection:

Current SHA: abc123def456...
Last synced: (none - first sync)
Action: Sync required

3. Helm processing:

# For apps/ollama/
helm dependency build apps/ollama/
helm template ollama apps/ollama/ --values apps/ollama/values.yaml

# Generates YAML manifests

4. Application to the cluster:

kubectl apply -f <manifestos gerados>

5. Health checking:

Waiting:
  - Pods: Ready
  - Deployments: Available
  - StatefulSets: Ready

Observable timeline:

# Terminal 2: Monitor Ollama
watch kubectl get pods -n ollama

# Output evolves as ArgoCD syncs:
NAME                      READY   STATUS
ollama-xxx-yyy           0/1     Pending
ollama-xxx-yyy           0/1     ContainerCreating
ollama-xxx-yyy           0/1     Running  # Init container: pulling models
ollama-xxx-yyy           1/1     Running  # Ready (~2-3 min)

# Terminal 3: Monitor LibreChat
watch kubectl get pods -n librechat

# Output evolves:
NAME                                    READY   STATUS
librechat-mongodb-0                     0/1     Pending
librechat-meilisearch-xxx               0/1     ContainerCreating
librechat-xxx-yyy                       0/1     Pending

librechat-mongodb-0                     1/1     Running  # ~30s
librechat-meilisearch-xxx               1/1     Running  # ~25s
librechat-xxx-yyy                       1/1     Running  # ~1 min

After 3–5 minutes, ArgoCD UI shows:

Verification

# Ollama
curl http://ollama.glukas.space/api/tags
{
  "models": [
    {"name": "llama3.2:3b", ...},
    {"name": "deepseek-r1:14b", ...}
  ]
}

Operations

These five workflows cover the full operational lifecycle under GitOps: making a configuration change, upgrading a chart version, rolling back a broken deploy, understanding self-healing behaviour, and managing multiple environments. In every case the pattern is the same — edit files, push to Git, let ArgoCD do the rest. No kubectl apply, no terraform apply, no manual intervention required.

Workflow 1: Modify Configuration (Add a Model)

Objective: Add the mistral:latest model to Ollama.

Process:

# 1. Clone and branch
git clone https://github.com/usuario/k8s-apps.git
cd k8s-apps
git checkout -b add-mistral

# 2. Edit
vim apps/ollama/values.yaml

# Modify:
models:
  pull:
    - llama3.2:3b
    - deepseek-r1:14b
    - mistral:latest  # Added

# 3. Commit
git add apps/ollama/values.yaml
git commit -m "feat(ollama): Add mistral model"

# 4. Push
git push origin add-mistral

Pull Request:

Create PR on GitHub/GitLab
Visible diff:

 models:
   pull:
     - llama3.2:3b
     - deepseek-r1:14b
+    - mistral:latest

Review and approval
Merge to main

Automatic ArgoCD:

Timeline after merge:

[T+0 min]   Merge to main
[T+0-3 min] ArgoCD polling (waiting for next cycle)
[T+3 min]   ArgoCD detects new SHA
[T+3 min]   Calculates diff: + models.pull: mistral:latest
[T+3 min]   Helm upgrade ollama...
[T+4 min]   Rolling update initiated
[T+4-7 min] Init container: ollama pull mistral:latest
[T+7 min]   New pod Ready
[T+7 min]   Old pod Terminated
[T+7 min]   ArgoCD status: Synced ✓

Total time: ~7 minutes from merge to deploy.

Note: At no point did the developer execute any command against the cluster directly — the entire deployment was driven by a Git push.

Workflow 2: Chart Version Upgrade

Objective: Upgrade LibreChat from 1.9.7 to 1.10.0.

git checkout -b upgrade-librechat

vim apps/librechat/Chart.yaml

# Modify:
dependencies:
  - name: librechat
    version: "1.10.0"  # Era 1.9.7

git commit -am "chore(librechat): Upgrade to v1.10.0"
git push origin upgrade-librechat

CI/CD (optional):

# .github/workflows/helm-lint.yml
name: Helm Lint
on: pull_request

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: azure/setup-helm@v3

      - name: Lint
        run: |
          helm lint apps/*/

      - name: Template Test
        run: |
          helm template test apps/*/ > /dev/null

Pipeline runs:

Helm lint (syntax validation)
Template rendering (detects errors)

After approval and merge, ArgoCD deploys automatically.

Workflow 3: Rollback

Scenario: New deploy caused a problem in production.

Option 1: Git Revert

# View commits
git log --oneline apps/librechat/
# def456 chore(librechat): Upgrade to v1.10.0
# abc123 feat(ollama): Add mistral

# Revert
git revert def456
git push origin main

ArgoCD detects and applies the revert automatically.

Timeline: ~3–5 minutes.

Option 2: ArgoCD UI

1. Open http://argocd.glukas.space
2. Select the "librechat" application
3. Open the "History" tab
4. Sync list:
   Sync 5: def456 (current) ❌
   Sync 4: abc123 ✅
5. Click on Sync 4
6. Click "Rollback"
7. Confirm

Timeline: ~30 segundos.

Important: Rollback via UI is temporary. The next poll will re-sync with Git. For permanence, perform a git revert.

Option 3: ArgoCD CLI

# Install CLI
brew install argocd  # ou método apropriado

# Login
argocd login argocd.glukas.space --username admin

# View history
argocd app history librechat

# Rollback
argocd app rollback librechat <REVISION>

Workflow 4: Self-Healing

Scenario: Manual change in the cluster.

# Someone runs:
kubectl scale deployment ollama --replicas=3 -n ollama

ArgoCD response:

[T+0s]     kubectl scale executed
[T+0s]     Deployment: replicas=3
[T+0-180s] ArgoCD polling interval
[T+180s]   ArgoCD detects drift:
           Git: replicas=1
           Cluster: replicas=3
[T+181s]   Self-heal triggered
           kubectl apply -f deployment.yaml (from Git)
[T+182s]   Kubernetes: replicas=1
           3 extra pods terminated
[T+183s]   ArgoCD status: Synced ✓
           Event: "Self-healed: ollama deployment"

Manual change was automatically reverted.

Responsible configuration:

syncPolicy = {
  automated = {
    selfHeal = true  # This parameter enables automatic revert
  }
}

Disable self-heal:

syncPolicy = {
  automated = {
    prune = true
    selfHeal = false  # Allows manual changes to persist
  }
}

Workflow 5: Multi-Environment

Structure:

k8s-apps/
├── apps/
│   └── ollama/
│       ├── Chart.yaml
│       ├── values-dev.yaml
│       ├── values-staging.yaml
│       └── values-prod.yaml

Differentiated values:

# values-dev.yaml
ollama:
  ollama:
    models:
      pull:
        - llama3.2:3b  # Lightweight model only
    resources:
      limits:
        memory: "2Gi"

# values-prod.yaml
ollama:
  ollama:
    models:
      pull:
        - llama3.2:3b
        - deepseek-r1:14b
        - mistral:latest
    resources:
      limits:
        memory: "8Gi"

ArgoCD Applications (Terraform):

# Dev
resource "kubernetes_manifest" "argocd_app_ollama_dev" {
  manifest = {
    spec = {
      source = {
        repoURL        = var.git_repo_url
        targetRevision = "develop"  # develop branch
        path           = "apps/ollama"
        helm = {
          valueFiles = ["values-dev.yaml"]
        }
      }
      destination = {
        namespace = "ollama-dev"
      }
    }
  }
}

# Prod
resource "kubernetes_manifest" "argocd_app_ollama_prod" {
  manifest = {
    spec = {
      source = {
        repoURL        = var.git_repo_url
        targetRevision = "main"  # main branch
        path           = "apps/ollama"
        helm = {
          valueFiles = ["values-prod.yaml"]
        }
      }
      destination = {
        namespace = "ollama-prod"
      }
    }
  }
}

Promotion flow:

Feature branch → develop (PR) → auto-deploy to Dev
              → staging (PR)  → auto-deploy to Staging
              → main (PR + approvals) → auto-deploy to Prod

Git branches map to environments.

Troubleshooting

This section covers the most common failure modes when running ArgoCD in practice — what they look like, why they happen, and how to fix them.

Problem 1: Application OutOfSync

An OutOfSync status means ArgoCD has detected a difference between what's in Git and what's running in the cluster, but hasn't been able to resolve it. This is usually the first sign that something went wrong during a sync — not necessarily a cluster problem, but worth investigating immediately.

Symptom:

kubectl get application -n argocd
NAME      SYNC STATUS   HEALTH STATUS
ollama    OutOfSync     Unknown

Diagnosis:

# Describe Application
kubectl describe application ollama -n argocd

# View events
kubectl get events -n argocd --sort-by='.lastTimestamp'

# repo-server logs
kubectl logs -n argocd deployment/argocd-repo-server

# application-controller logs
kubectl logs -n argocd statefulset/argocd-application-controller

Common causes:

YAML syntax error

Error: YAML parse error line 15: mapping values are not allowed here

Solution: Fix syntax in values.yaml

Chart version not found

Error: chart "ollama" version "1.42.0" not found

Solution: Check available versions:

helm search repo ollama --versions

Repository unreachable

Error: failed to fetch https://github.com/usuario/k8s-apps.git: authentication required

Solution: Configure credentials in ArgoCD

Local validation:

# Test template rendering
cd k8s-apps/
helm dependency build apps/ollama/
helm template test apps/ollama/

# If there's an error, it will appear here

Problem 2: Pods CrashLoopBackOff

A CrashLoopBackOff means the pod is starting, failing, and being restarted repeatedly. ArgoCD may show the application as Synced — meaning the deployment was applied correctly — but Degraded on health, because the pod never reaches a running state. The problem is almost always in the container itself, not in ArgoCD.

Symptom:

kubectl get pods -n ollama
NAME            READY   STATUS             RESTARTS
ollama-xxx      0/1     CrashLoopBackOff   5

Diagnosis:

# Current pod logs
kubectl logs -n ollama ollama-xxx

# Previous container logs (if it has restarted)
kubectl logs -n ollama ollama-xxx --previous

# Pod events and conditions
kubectl describe pod -n ollama ollama-xxx

Common causes:

GPU not available

Error: failed to initialize NVML: could not load NVML library

Solution:

# Temporarily disable GPU
ollama:
  ollama:
    gpu:
      enabled: false

Insufficient memory

Error: OOMKilled

Solution:

resources:
  limits:
    memory: "8Gi"  # Increase

Model does not exist

Error: pulling model: model 'llama4' not found

Solution: Check model name on values.yaml

Problem 3: Double Hierarchy Not Applied

This is one of the trickier failure modes because ArgoCD reports everything as healthy — the sync succeeded, no errors are visible, but the configuration simply isn't taking effect. It typically happens when the Helm values file is missing one level of nesting, causing the GPU settings to silently fall back to defaults.

Symptom:

ArgoCD shows Synced
GPU not enabled
No visible errors

Diagnosis:

# Render full template
helm template test apps/ollama/

# Search for GPU configuration
helm template test apps/ollama/ | grep -A10 "nvidia.com/gpu"

# If not found, structure is wrong

Cause:

# ❌ Incorrect structure (one layer)
ollama:
  gpu:
    enabled: true

Solution:

# ✅ Correct structure (double layer)
ollama:
  ollama:
    gpu:
      enabled: true

Validation:

# After correction, check diff in ArgoCD UI
# Should show change in spec.template.spec.containers[].resources

Problem 4: Slow Sync

Unlike the previous problems, slow sync isn't a failure — it's expected behavior that becomes surprising when you first encounter it. ArgoCD doesn't watch Git in real time; it polls on a fixed interval, so there will always be a delay between a git push and a deployment.

Symptom:
ArgoCD takes >5 minutes to detect changes.

Cause:
Default polling interval is 3 minutes.

Solution 1: Adjust polling

# values/argocd-values.yaml
server:
  config:
    timeout.reconciliation: 60s  # 1 minute

Trade-off: More load on the cluster and Git repo.

Solution 2: Webhook

Configure a webhook in Git to notify ArgoCD:

# GitHub webhook URL
POST https://argocd.glukas.space/api/webhook

ArgoCD syncs immediately upon receiving a push.

Solution 3: Manual sync

# Via CLI
argocd app sync ollama

# Via UI
Click "Sync" on the application

Ch. 3 vs Ch. 4: When to Use Each

Both approaches are valid, and the right choice depends on team size, deploy frequency, and how much operational overhead you want to absorb upfront. The table below maps the key trade-offs to help you decide:

	Cap 3 (Terraform + Helm)	Cap 4 (Terraform + ArgoCD)
Deploy trigger	Manual: `terraform apply`	Automatic: Git push
Latency	Immediate	3 min (polling)
Reconciliation	Manual: `terraform plan`	Continuous: 3-min loop
Drift detection	Manual	Automatic
Self-healing	Does not exist	Configurable (selfHeal)
Rollback	`git revert` + `terraform apply`	ArgoCD UI (1 click) or `git revert`
Audit trail	Git + Terraform logs	Git + ArgoCD events
Multi-env	Duplicate code or workspaces	Branches + valueFiles
Permissions required	kubectl + Terraform	Git only
Disaster recovery	Re-run Terraform	Automatic ArgoCD re-sync
State management	Terraform state (central)	Git (distributed)
Initial complexity	Medium	High
Scalability (apps)	~20 apps	Unlimited
Ideal team size	1–10	10+

Chapter 3's approach is simpler to set up and sufficient for small teams with controlled deploy cadences — if a weekly terraform apply is acceptable, the added complexity of ArgoCD is not justified. Chapter 4 becomes the right choice once teams grow, deploy frequency increases, or compliance requirements demand an immutable audit trail and automated drift correction. The two are not mutually exclusive: many organisations start with Chapter 3 and migrate to Chapter 4 as their operational maturity grows.

Conclusion

Chapters 1 through 4 trace a deliberate progression — from manual kubectl commands to a fully automated, self-healing platform. Each chapter addressed a specific limitation of the one before it: verbosity, the need for manual execution, the absence of continuous reconciliation. The cumulative result is an architecture where Git is the single source of truth, and the cluster enforces that truth on its own.

The four GitOps principles are not just theoretical framing — each one translates directly into an operational guarantee. Declarative configuration means the desired state is always readable and auditable without touching the cluster. Version control means every change has an author, a rationale, and a rollback path. Pull-based deployment means no external system ever needs credentials to reach the cluster — the cluster reaches out to Git. Continuous reconciliation means drift is detected and corrected automatically, without anyone noticing or reacting.

The architecture also enforces a clean separation of concerns that makes each layer independently replaceable:

Terraform → Platform bootstrap (namespaces, secrets, ArgoCD)
Git       → Application desired state
ArgoCD    → Reconciliation engine
Helm      → Packaging and templating

Changes to one layer do not cascade into the others. You could swap Helm for raw manifests, or replace Terraform with a different provisioner, without touching ArgoCD or the Git repository structure.

This foundation is deliberately extensible. The next steps — security, observability, multi-tenancy — build on top of it without requiring the core architecture to change.

Maturity Journey

Each chapter in this series represents a deliberate step up the maturity ladder — not just in tooling, but in ownership model, speed, and scale:

Stage 1: Manual Deployment (Ch. 1)

Maturity:  Ad-hoc
Ownership: Individuals
Speed:     Slow (days/weeks)
Scale:     Doesn't scale

Stage 2: Infrastructure as Code (Ch. 2–3)

Maturity:  Repeatable
Ownership: Ops team
Speed:     Medium (hours/days)
Scale:     Limited (manual execution)

Stage 3: GitOps Foundation (Ch. 4) ← We are here

Maturity:  Automated
Ownership: Shared (platform + dev)
Speed:     Fast (minutes/hours)
Scale:     Good (self-service enabled)

Stage 4: Infrastructure as Product (Next Steps)

Maturity:  Product-driven
Ownership: Platform teams (product owners)
Speed:     Very fast (minutes)
Scale:     Excellent (true self-service)
Metrics:   DORA, satisfaction, adoption

What Comes Next

Stage 3 is a foundation, not a destination. The architecture built in this chapter is intentionally minimal — one team, two applications, one cluster — and that is the right place to start. But the same GitOps primitives that make this setup work at small scale are exactly what allow it to grow.

The diagram below shows the current state: a single developer workflow, a flat namespace structure, and ArgoCD managing two specific workloads with no shared services, no multi-tenancy, and no separation between platform concerns and application concerns.

The target looks substantially different. The cluster is split into two distinct layers: a Platform Layer of shared services — security, observability, secrets management — owned by a dedicated platform team with SLAs and roadmaps; and a Workload Layer where individual product teams deploy independently via git push, without ever touching the platform layer beneath them.

The gap between the two diagrams is not a rewrite — it is an incremental build. Every component in the Platform Layer gets added as an ArgoCD-managed application in its own namespace, following the exact same wrapper-chart pattern introduced in this chapter. The core architecture does not change; it simply gains more managed services over time.

The next chapters will build out this platform layer starting with the highest-impact additions: security, observability, and secrets management.

Initiative	Domain	Phase	Complexity	Impact	Dependencies	Time	Priority
Pomerium	SECURITY	Foundation	Intermediate	High	ArgoCD	3-5d	P0
Sealed Secrets	SECURITY	Foundation	Basic	High	None	1d	P0
Authentik	SECURITY	Foundation	Intermediate	High	PostgreSQL	3-5d	P0
Prometheus + Grafana	OBSERVABILITY	Foundation	Intermediate	High	None	3-5d	P0
MCP Servers	INTEGRATION	Foundation	Intermediate	High	None	2-3d	P0
RAG (Qdrant)	AI/LLM	Foundation	Advanced	High	None	1w	P1
LangSmith/Langfuse	AI/LLM	Scale	Advanced	High	Prometheus	5-7d	P1
Autoscaling	INFRA	Scale	Intermediate	High	Prometheus	2-3d	P1
Loki	OBSERVABILITY	Scale	Basic	Medium	Grafana	1-2d	P1
SearXNG	INTEGRATION	Scale	Basic	Medium	None	1d	P1
Web Scraper	INTEGRATION	Scale	Intermediate	Medium	None	2d	P1
Tilt	DEVEX	Scale	Basic	Medium	None	1d	P1
Jaeger	OBSERVABILITY	Production Excellence	Advanced	Medium	Prometheus	3-5d	P2
Model Registry	AI/LLM	Production Excellence	Intermediate	Medium	None	3-5d	P2
Multi-region	NETWORK	Production Excellence	Expert	Medium	ArgoCD	2w+	P3
Fine-tuning	AI/LLM	Production Excellence	Expert	Low	Registry	2w	P3

Platform Products (Shared Services):

Pomerium + Authentik:
  product: "Authentication & Authorization Platform"
  customers: "All applications"
  value: "SSO, MFA, zero-trust"
  sla: "99.9% uptime, <200ms auth latency"
  roadmap: ["RBAC granular", "SAML support", "API keys"]

Prometheus + Grafana + Loki:
  product: "Observability Platform"
  customers: "All teams (dev + ops)"
  value: "Unified metrics/logs/traces"
  sla: "30d retention, <5s query time"
  roadmap: ["AIOps", "Cost attribution", "SLO management"]

Sealed Secrets:
  product: "Secrets Management Platform"
  customers: "All teams"
  value: "Git-native secrets, rotation, audit"
  sla: "Zero exposure, <1min sync"
  roadmap: ["Vault integration", "RBAC", "Expiration"]

Workload-Specific Products:

RAG (Qdrant):
  product: "Vector Search Service"
  customers: "AI/ML teams"
  value: "Semantic search, embeddings"
  sla: "<100ms p95 search latency"
  roadmap: ["Multi-model", "Hybrid search"]

MCP Servers:
  product: "Tool Integration Platform"
  customers: "LLM applications"
  value: "Connect LLMs to tools"
  sla: "<50ms tool invocation"
  roadmap: ["Custom tools", "Async execution"]

Developer Experience Products:

Tilt:
  domain: "[DEVEX]"
  phase: "Scale"
  complexity: "Basic"
  impact: "Medium"
  dependencies: ["None"]
  time: "1 day"
  priority: "P1"
  product: "Local Development Platform"
  customers: "All developers"
  value: "Hot-reload, real K8s environment, fast iteration"
  sla: "<5s code sync, <10s service restart"
  roadmap: ["Remote development", "Debugging tools", "Resource snapshots"]

Infrastructure Products:

Multi-region:
  domain: "[NETWORK]"
  phase: "Production Excellence"
  complexity: "Expert"
  impact: "Medium"
  dependencies: ["ArgoCD"]
  time: "2+ weeks"
  priority: "P3"
  product: "Global Load Balancing & Geo-distribution"
  customers: "All production workloads"
  value: "Low latency worldwide, compliance (data residency)"
  sla: "99.99% global availability, <100ms cross-region failover"
  roadmap: ["Active-active", "Traffic shaping", "Cost optimization", "DR automation"]

Production Recommended Extensions

TLS/HTTPS:

# argocd-values.yaml
server:
  ingress:
    tls:
      - secretName: argocd-tls
        hosts:
          - argocd.empresa.com

SSO/OIDC:

server:
  config:
    url: https://argocd.empresa.com
    oidc.config: |
      name: Okta
      issuer: https://empresa.okta.com
      clientID: $oidc.okta.clientId
      clientSecret: $oidc.okta.clientSecret

RBAC:

server:
  rbacConfig:
    policy.csv: |
      p, role:developers, applications, get, */*, allow
      p, role:developers, applications, sync, */*, allow
      g, developers-group, role:developers

Notifications:

notifications:
  enabled: true
  notifiers:
    service.slack: |
      token: $slack-token
  templates:
    template.app-deployed: |
      message: Application {{.app.metadata.name}} deployed
  triggers:
    trigger.on-deployed: |
      - when: app.status.operationState.phase in ['Succeeded']
        send: [app-deployed]

Application Sets:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-apps
spec:
  generators:
    - git:
        repoURL: https://github.com/empresa/k8s-apps.git
        revision: HEAD
        directories:
          - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      source:
        repoURL: https://github.com/empresa/k8s-apps.git
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'

Monitoring:

# ServiceMonitor para Prometheus
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-metrics
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics
  endpoints:
    - port: metrics

Technical Resources

References:

Straube, S. (2025). "Infrastructure as a Product: The Key to Agile and Scalable IT". Medium/Elevate Tech.
Griffiths, M. (2021). "Infrastructure as Product: Accelerating time to market through platform engineering". Thoughtworks Insights.
Strope, L. (2026). "Why Infrastructure Is Becoming Product And How to Capitalize". Akava.

Documentation:

Reference Repositories:

Auxiliary Tools:

argocd CLI
kubectl-argo-rollouts (progressive delivery)
argocd-notifications (alerts)
argocd-image-updater (auto-update images)

End of Chapter 4

Chapter 3: A Better Abstraction — Managing LLM Apps with Terraform + Helm

George Lukas — Fri, 20 Feb 2026 19:09:36 +0000

In Chapter 2, we reached an uncomfortable conclusion: Terraform can manage Kubernetes, but that doesn't mean it should manage everything in Kubernetes.

We observed that:

Terraform → Versioning, auditing, reproducibility
Helm → Simplicity, lifecycle management
Terraform + K8s Provider directly → Verbose, giant state, no rollbacks

The question that lingers: "Is there a way to have the best of both?"

Changing Abstraction Level

The problem in Chapter 2 wasn't Terraform itself, it was the level of abstraction we chose.

Wrong thinking:

Terraform → Manage Deployments, Services, Ingress, etc
             (individual Kubernetes resources)

Right thinking:

Terraform → Manage Helm Releases
             (complete applications as units)

It's a subtle but profound change. Instead of Terraform replacing Helm, Terraform orchestrates Helm.

Layered Architecture of Responsibilities

Let's visualize how responsibilities are divided:

┌──────────────────────────────────────────┐
│         You (Developer)                  │
│    Define desired state in code          │
└────────────────┬─────────────────────────┘
                 │
                 ↓
┌──────────────────────────────────────────┐
│         Terraform (Orchestrator)         │
│  • Manages namespaces                    │
│  • Manages infrastructure secrets        │
│  • Manages RBAC                          │
│  • Manages Helm Releases (pointers)      │
└────────────────┬─────────────────────────┘
                 │
                 ↓
┌──────────────────────────────────────────┐
│         Helm (Package Manager)           │
│  • Renders templates                     │
│  • Applies resources to cluster          │
│  • Maintains release history             │
│  • Manages rollbacks                     │
└────────────────┬─────────────────────────┘
                 │
                 ↓
┌──────────────────────────────────────────┐
│         Kubernetes (Runtime)             │
│  • Runs containers                       │
│  • Manages storage                       │
│  • Routes traffic                        │
│  • Self-healing                          │
└──────────────────────────────────────────┘

Each layer does what it does best. No unnecessary overlap.

Project Structure:

cap3-helm-provider/
├── main.tf                      # Main configuration
├── variables.tf                 # Input variables
├── terraform.tfvars             # Values (don't commit!)
├── outputs.tf                   # Useful outputs
├── .gitignore                   # Secret protection
│
├── values/                      # Helm chart values
│   ├── ollama-values.yaml
│   └── librechat-values.yaml
│
└── README.md                    # Documentation

Separation of code and configuration
- Logic (main.tf) separated from values (values/)
- Easy to version and review changes
Reusable
- Same structure for dev, staging, prod
- Only change terraform.tfvars
Secure
- .gitignore protects secrets
- Values can have public and private versions

Part 1: Provider Declaration

# main.tf
terraform {
  required_version = ">= 1.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11"
    }
  }
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "minikube"
}

provider "helm" {
  kubernetes {
    config_path    = "~/.kube/config"
    config_context = "minikube"
  }
}

Helm Provider

Now we have two providers:

kubernetes: For base infrastructure resources
helm: For managing application releases

Important: The Helm provider doesn't replace the Kubernetes provider. They work together:

Kubernetes provider → Creates namespaces, secrets, RBAC
Helm provider → Deploys applications in those namespaces

Semantic versioning (~> 2.23):

~> 2.23 means:
- 2.23.0, 2.23.1, 2.24.0 (accepts)
- 3.0.0 (rejects - breaking change)

Ensures security updates without breaking compatibility.

Part 2: Base Infrastructure (Terraform Territory)

# Namespaces managed by Terraform
resource "kubernetes_namespace" "ollama" {
  metadata {
    name = "ollama"
    labels = {
      managed-by = "terraform"
    }
  }
}

resource "kubernetes_namespace" "librechat" {
  metadata {
    name = "librechat"
    labels = {
      managed-by = "terraform"
    }
  }
}

Why does Terraform manage namespaces?

Namespaces are infrastructure, not applications. They:

Rarely change
Are prerequisites for everything
Define security boundaries
Need to exist before applications

Label managed-by = "terraform":

# Useful for filtering
kubectl get ns -l managed-by=terraform

# Output:
NAME        STATUS   AGE
ollama      Active   10m
librechat   Active   10m

Makes it clear these resources shouldn't be edited manually.

Part 3: Infrastructure Secrets

# Secret managed by Terraform (infra-level)
resource "kubernetes_secret" "librechat_credentials" {
  metadata {
    name      = "librechat-credentials-env"
    namespace = kubernetes_namespace.librechat.metadata[0].name
  }

  data = {
    JWT_SECRET         = var.jwt_secret
    JWT_REFRESH_SECRET = var.jwt_refresh_secret
    CREDS_KEY          = var.creds_key
    CREDS_IV           = var.creds_iv
    MONGO_URI          = "mongodb://librechat-mongodb:27017/LibreChat"
    MEILI_HOST         = "http://librechat-meilisearch:7700"
    OLLAMA_BASE_URL    = "http://ollama.ollama.svc.cluster.local:11434"
  }

  type = "Opaque"
}

Design decision: Why does Terraform manage this secret?

This secret contains infrastructure credentials that:

Need to exist before application deployment
Don't change frequently
Are shared between environments (same structure, different values)
Should be versioned (structure) but not values (.tfvars)

Important dynamic reference:

namespace = kubernetes_namespace.librechat.metadata[0].name

Terraform ensures the namespace is created first, then the secret. Automatic dependency management!

Variables file (variables.tf):

variable "jwt_secret" {
  description = "JWT secret for LibreChat"
  type        = string
  sensitive   = true
}

variable "jwt_refresh_secret" {
  description = "JWT refresh secret"
  type        = string
  sensitive   = true
}

variable "creds_key" {
  description = "Credentials encryption key"
  type        = string
  sensitive   = true
}

variable "creds_iv" {
  description = "Credentials initialization vector"
  type        = string
  sensitive   = true
}

Values file (terraform.tfvars - DON'T COMMIT!):

jwt_secret         = "abc123def456..."  # generated with openssl rand -hex 32
jwt_refresh_secret = "ghi789jkl012..."
creds_key          = "mno345pqr678..."
creds_iv           = "stu901vwx234..."

.gitignore:

# Terraform
*.tfstate
*.tfstate.*
.terraform/
terraform.tfvars  # ← CRITICAL!

# Sensitive files
values/*-secrets.yaml

Part 4: Helm Releases

This is where the Chapter 3 approach shines.

Ollama deployment:

# Helm Release - Ollama
resource "helm_release" "ollama" {
  name       = "ollama"
  repository = "https://otwld.github.io/ollama-helm/"
  chart      = "ollama"
  namespace  = kubernetes_namespace.ollama.metadata[0].name

  values = [
    file("${path.module}/values/ollama-values.yaml")
  ]

  # Version control
  version = "1.41.0"  # Pin version for reproducibility

  # Deployment settings
  create_namespace = false  # Already created by Terraform
  wait             = true   # Wait for ready
  timeout          = 600    # 10 minutes max

  # Dependency tracking
  depends_on = [
    kubernetes_namespace.ollama
  ]
}

Breaking it down:

1. Chart source:

repository = "https://otwld.github.io/ollama-helm/"
chart      = "ollama"

2. Values file:

values = [
  file("${path.module}/values/ollama-values.yaml")
]

The file() function reads YAML from Terraform module Path.

3. Version pinning:

version = "1.41.0"

Critical for reproducibility! Without this, helm_release would use "latest", which changes over time.

4. Deployment controls:

wait    = true   # Don't return until ready
timeout = 600    # 10 min max

Terraform waits for Pods to be healthy before considering deployment successful.

5. Dependencies:

depends_on = [kubernetes_namespace.ollama]

Terraform creates namespace → then creates release

The values file (values/ollama-values.yaml):

ollama:
  gpu:
    enabled: true
    type: nvidia
    number: 1

  models:
    - llama2
    - codellama

  resources:
    requests:
      cpu: 2
      memory: 8Gi

service:
  type: ClusterIP
  port: 11434

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: ollama.glukas.space
      paths:
        - path: /
          pathType: Prefix

LibreChat deployment:

# Helm Release - LibreChat
resource "helm_release" "librechat" {
  name       = "librechat"
  repository = "oci://ghcr.io/danny-avila/librechat-chart"
  chart      = "librechat"
  namespace  = kubernetes_namespace.librechat.metadata[0].name

  values = [
    file("${path.module}/values/librechat-values.yaml")
  ]

  version = "1.5.0"

  create_namespace = false
  wait             = true
  timeout          = 900  # 15 min (MongoDB initialization)

  depends_on = [
    kubernetes_namespace.librechat,
    kubernetes_secret.librechat_credentials
  ]
}

Notice:

Different repository (OCI registry)
Longer timeout (MongoDB takes time)
Depends on secret (must exist first)

LibreChat depends on Ollama. Terraform ensures order:

Namespace
Secret
Ollama
LibreChat

The values file (values/librechat-values.yaml):

config:
  APP_TITLE: "LibreChat + Ollama (via Terraform)"
  HOST: "0.0.0.0"
  PORT: "3080"
  SEARCH: "true"
  MONGO_URI: "mongodb://librechat-mongodb:27017/LibreChat"
  MEILI_HOST: "http://librechat-meilisearch:7700"

librechat:
  configEnv:
    ALLOW_REGISTRATION: "true"
  configYamlContent: |
    version: 1.1.5
    cache: true

    endpoints:
      custom:
        - name: "Ollama"
          apiKey: "ollama"
          baseURL: "http://ollama.ollama.svc.cluster.local:11434/v1"
          models:
            default:
              - "llama2:latest"
            fetch: true
          titleConvo: true
          titleModel: "llama2:latest"
          summarize: false
          summaryModel: "llama2:latest"
          forcePrompt: false
          modelDisplayLabel: "Ollama"
          addParams:
            temperature: 0.7
            max_tokens: 2000

extraEnvVarsSecret: "librechat-credentials-env"

ingress:
  enabled: true
  className: "nginx"
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "25m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: librechat.glukas.space
      paths:
        - path: /
          pathType: Prefix

mongodb:
  enabled: true
  auth:
    enabled: false
  image:
    repository: bitnami/mongodb
    tag: latest
  persistence:
    enabled: true
    size: 8Gi
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "1Gi"
      cpu: "500m"

meilisearch:
  enabled: true
  auth:
    enabled: false
  environment:
    MEILI_NO_ANALYTICS: "true"
    MEILI_ENV: "development"
  persistence:
    enabled: true
    size: 1Gi
  resources:
    requests:
      memory: "128Mi"
      cpu: "50m"
    limits:
      memory: "512Mi"
      cpu: "250m"

resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "1Gi"
    cpu: "500m"

persistence:
  enabled: true
  size: 5Gi
  storageClass: "standard"

replicaCount: 1

Deployment Workflow

Now let's see this in action.

Initial Deployment

# Generate secrets
export TF_VAR_jwt_secret=$(openssl rand -hex 32)
export TF_VAR_jwt_refresh_secret=$(openssl rand -hex 32)
export TF_VAR_creds_key=$(openssl rand -hex 32)
export TF_VAR_creds_iv=$(openssl rand -hex 16)

# Or create terraform.tfvars
cat > terraform.tfvars <<EOF
jwt_secret         = "$(openssl rand -hex 32)"
jwt_refresh_secret = "$(openssl rand -hex 32)"
creds_key          = "$(openssl rand -hex 32)"
creds_iv           = "$(openssl rand -hex 16)"
EOF

# 1. Initialize
terraform init
# Downloads both kubernetes and helm providers

# 2. Validate
terraform validate
# Checks HCL syntax

# 3. Plan
terraform plan

Plan output:

Terraform will perform the following actions:

  # kubernetes_namespace.ollama will be created
  + resource "kubernetes_namespace" "ollama" {
      + id = (known after apply)

      + metadata {
          + generation = (known after apply)
          + name       = "ollama"
          + labels     = {
              + "managed-by" = "terraform"
            }
        }
    }

  # kubernetes_secret.librechat_credentials will be created
  + resource "kubernetes_secret" "librechat_credentials" {
      + data = (sensitive value)
      + id   = (known after apply)
      + type = "Opaque"

      + metadata {
          + name      = "librechat-credentials-env"
          + namespace = (known after apply)
        }
    }

  # helm_release.ollama will be created
  + resource "helm_release" "ollama" {
      + id          = (known after apply)
      + name        = "ollama"
      + namespace   = (known after apply)
      + repository  = "https://otwld.github.io/ollama-helm/"
      + version     = "1.41.0"
      + status      = (known after apply)
      + values      = [
          + <<-EOT
              ollama:
                gpu:
                  enabled: true
                  ...
            EOT,
        ]
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Notice: Only 5 resources in the plan!

2 namespaces
1 secret
2 helm_releases

Compare to Chapter 2: would be 50+ individual K8s resources.

# 4. Apply
terraform apply

# Output:
kubernetes_namespace.ollama: Creating...
kubernetes_namespace.librechat: Creating...
kubernetes_namespace.ollama: Creation complete after 1s
kubernetes_namespace.librechat: Creation complete after 1s
kubernetes_secret.librechat_credentials: Creating...
kubernetes_secret.librechat_credentials: Creation complete after 1s
helm_release.ollama: Creating...
helm_release.ollama: Still creating... [10s elapsed]
helm_release.ollama: Still creating... [20s elapsed]
helm_release.ollama: Creation complete after 45s [id=ollama]
helm_release.librechat: Creating...
helm_release.librechat: Still creating... [10s elapsed]
helm_release.librechat: Still creating... [20s elapsed]
...
helm_release.librechat: Creation complete after 2m15s [id=librechat]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

What happened behind the scenes:

Terraform created namespaces
Terraform created secret
Terraform told Helm: "Install ollama chart with these values"
Helm rendered templates and created: Deployment, Service, PVC, Ingress, etc
Terraform told Helm: "Install librechat chart with these values"
Helm rendered templates and created: Deployment, Service, MongoDB StatefulSet, MeiliSearch Deployment, Ingress, etc

Terraform state only tracks the 5 high-level resources.
Helm manages all the detailed Kubernetes resources.

Verifying Deployment

# Check Terraform state
terraform state list
# kubernetes_namespace.librechat
# kubernetes_namespace.ollama
# kubernetes_secret.librechat_credentials
# helm_release.librechat
# helm_release.ollama

# Check Helm releases
helm list -A
# NAME        NAMESPACE   REVISION  STATUS    CHART             APP VERSION
# ollama      ollama      1         deployed  ollama-1.41.0     0.1.20
# librechat   librechat   1         deployed  librechat-1.5.0   0.7.0

# Check actual Kubernetes resources
kubectl get all -n ollama
# NAME                         READY   STATUS    RESTARTS   AGE
# pod/ollama-7d8f9c5b6d-xk2p4  1/1     Running   0          2m
#
# NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
# service/ollama   ClusterIP   10.96.245.12   <none>        11434/TCP   2m
#
# NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
# deployment.apps/ollama   1/1     1            1           2m

kubectl get all -n librechat
# (Shows MongoDB, MeiliSearch, LibreChat deployments/services)

# Testing Ollama
curl http://ollama.glukas.space/api/tags

Output:

{
  "models": [
    {
      "name": "llama2:latest",
      "modified_at": "2025-02-07T13:30:00.000Z",
      "size": 3826793677,
      "digest": "sha256:abc123...",
      "details": {
        "format": "gguf",
        "family": "llama",
        "families": ["llama"],
        "parameter_size": "7B",
        "quantization_level": "Q4_0"
      }
    }
  ]
}

Ollama Working!

# Testing LibreChat
curl -I http://librechat.glukas.space

Output:

HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; charset=utf-8
...

LibreChat Working!

Open in browser:

http://librechat.glukas.space
Sign in
Select model Ollama
Start chatting!

All Good!

Operations

Now let's see how daily operations are different.

Upgrading Chart Version

Scenario: New Ollama chart version available.

# 1. Update version
vim main.tf

resource "helm_release" "ollama" {
  # ...
  version = "1.42.0"  # was 1.41.0
}

# 2. Plan
terraform plan

Output:

  ~ resource "helm_release" "ollama" {
        id                         = "ollama"
        name                       = "ollama"
      ~ version                    = "1.41.0" -> "1.42.0"
        # (15 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Only 1 change: chart version!

# 3. Apply
terraform apply

Helm does rolling update, zero downtime!

Changing Configuration

Scenario: Add CodeLlama model.

# 1. Edit values
vim values/ollama-values.yaml

ollama:
  models:
    - llama2
    - codellama  # ← NEW

# 2. Plan
terraform plan

Output:

  ~ resource "helm_release" "ollama" {
      ~ values   = [
          ~ <<-EOT
                ollama:
                  models:
                    - llama2
              +     - codellama
            EOT,
        ]
    }

Terraform detects diff in YAML!

# 3. Apply
terraform apply

Helm updates Deployment → Pod restarts → Downloads CodeLlama → Ready!

Rollback

Scenario: New version broke.

Option 1: Via Terraform

# Revert commit in Git
git revert HEAD

# Apply previous version
terraform apply

Option 2: Via Helm (faster)

# View history
helm history ollama -n ollama
# REVISION  UPDATED                   STATUS      CHART           DESCRIPTION
# 1         Thu Feb 07 10:29:30 2025  superseded  ollama-1.41.0   Install complete
# 2         Thu Feb 07 11:15:20 2025  deployed    ollama-1.42.0   Upgrade complete

# Rollback
helm rollback ollama -n ollama
# Rollback was a success! Happy Helming!

# Terraform detects on next plan
terraform plan
# (will show drift, but it's not a problem)

Best practice: Always use Terraform, but Helm is available for emergencies.

Comparison: Chapter 2 vs Chapter 3

Let's put side by side to visualize the gain.

Code Required

Metric	Ch 2 (TF + K8s)	Ch 3 (TF + Helm)	Reduction
Lines of HCL	~500	~100	80% ↓
Lines of YAML	0	~100	-
Total code	500	200	60% ↓
Files	1 monolith	5 organized	-

State Management

Aspect	Ch 2	Ch 3
Resources in state	50+	5
State size	2.3 MB	15 KB
Plan time	2 minutes	10 seconds
Detectable drift	Partially	Yes (via Helm)

Operations

Operation	Ch 2	Ch 3
Initial deploy	`terraform apply` (5 min)	`terraform apply` (2 min)
Version upgrade	Edit multiple blocks	Change 1 line
Rollback	`git revert` + apply	`helm rollback` (instant)
View status	`terraform state list`	`helm list`
Debug	`kubectl` + state inspection	`helm status`

Maintainability

Factor	Ch 2	Ch 3
Learning curve	High (HCL + K8s)	Medium (HCL + familiar YAML)
New dev onboarding	Difficult	Reasonable
Code review	Complex (many changes)	Simple (clear diff)
Reusability	Low	High (public charts)

The Advantages Scale

Now imagine you don't have 2 applications, but 20:

Chapter 2:

20 applications × 250 lines = 5,000 lines of HCL
20 applications × 50 resources = 1,000 resources in state
terraform plan = 10+ minutes
State file = 50+ MB

Chapter 3:

20 applications × 15 lines = 300 lines of HCL
20 applications × 100 lines YAML = 2,000 lines (familiar)
20 releases in state
terraform plan = 30 seconds
State file = 300 KB

The difference becomes even more dramatic at scale.

Chapter 3 solves 90% of Chapter 2's problems, and in many scenarios that's sufficient.

Terraform + Helm is the sweet spot for managing Kubernetes applications in a reproducible and versioned way.

Recapping what we achieved:

Total versioning — Everything in Git
Reproducibility — terraform apply = identical environment
Separation of responsibilities — Terraform (infra) + Helm (apps)
Manageable state — Few tracked resources
Possible rollbacks — Via Helm or Git
Less code — 94% reduction vs Ch 2
Ecosystem — Thousands of public charts
Maintainable — Familiar YAML, minimal HCL

But there are still limitations:

1. Deployment is Manual

# Always needs someone executing
terraform apply

There's no real continuous deployment. Git isn't the single source of truth, it's an input that requires manual action.

2. No Continuous Reconciliation

# If someone does this:
kubectl edit deployment ollama -n ollama

# Terraform only detects on next plan
# Until then, there's divergence

There's no automatic process ensuring cluster = code.

3. Limited Auditing

Who deployed version 1.42.0?
git log  # Shows commit
# But who executed terraform apply?
# There's no central record

Terraform state has some information, but it's not a complete audit log.

4. Approvals and Gates

How to ensure production deployment:
- Passed automated tests?
- Was approved by PO/PM?
- Has automatic rollback if it fails?

Terraform doesn't have this built-in. You need to build custom pipelines.

5. Complex Multi-Tenancy

How to allow:
- Team A to manage their apps in namespace team-a
- Team B to manage their apps in namespace team-b
- But both use the same cluster?
- Without giving access to the entire Terraform state?

Possible, but requires complex architecture.

What's Missing: GitOps

GitOps principles:

Declarative: Desired state described declaratively (we have this!)
Versioned: Everything in Git (we have this!)
Pull-based: Cluster pulls changes from Git automatically (we don't have this)
Continuous reconciliation: Agent ensures cluster = Git always (we don't have this)

What would change with GitOps:

Without GitOps (Ch 3):

Developer → commits → Git
Developer → terraform apply → Cluster
(Push-based, manual)

With GitOps (Ch 4):

Developer → commits → Git
ArgoCD (agent in cluster) → polls Git → applies changes
(Pull-based, automatic)

Additional benefits:

Zero-touch deployment: Commit = automatic deploy
Auto-healing: Cluster self-corrects if it diverges
Complete audit: Each deploy is a commit
Approvals: PR process = deployment approval
Multi-tenancy: Each team has their repo/branch

Next Chapter:

In Chapter 4, we'll discover how Kubernetes infrastructure is managed at scale:

ArgoCD: Real continuous deployment
Application Sets: Deploy to multiple clusters
Granular RBAC: Secure multi-tenancy
Sync waves: Dependency orchestration
Auto-sync: Git → Cluster automatic
Automatic rollbacks: Integrated health checks

Chapter 2: Infrastructure as Code Introduction — Terraform + Kubernetes Provider

George Lukas — Thu, 12 Feb 2026 23:44:16 +0000

Before diving into Terraform, we need to understand the mental shift that Infrastructure as Code (IaC) represents.

From Imperative to Declarative

Imperative Approach (what we did in Chapter 1):

# Step 1: Do this
kubectl create namespace ollama

# Step 2: Now do that  
kubectl create secret generic credentials...

# Step 3: Then do this other thing
helm install ollama...

# Like a chef giving instructions: "First heat the oven,
# then mix the ingredients, then bake for 30 minutes"

Declarative Approach (Infrastructure as Code):

# Describe the desired end state
resource "kubernetes_namespace" "ollama" {
  metadata {
    name = "ollama"
  }
}

resource "helm_release" "ollama" {
  name      = "ollama"
  namespace = kubernetes_namespace.ollama.metadata[0].name
  # ...
}

# Like a shopping list: "I need flour, eggs, sugar"
# The system figures out HOW to get it

The difference is subtle but profound:

Imperative: You say how to do it
Declarative: You say what you want

The Three Pillars of IaC

1. Versioning

git log infrastructure/
# Complete change history
# Who changed what, when, and why

2. Reproducibility

git clone repo
terraform apply
# Identical infrastructure anywhere

3. Auditing

git blame main.tf
# Each line traced to its author
# Pull requests = infrastructure review

Our First Step: Terraform + Kubernetes Provider

Let's start with the most direct approach, using Terraform to manage Kubernetes resources directly.

Project Structure

kubernetes-terraform/
├── main.tf           # Main configuration
├── variables.tf      # Input variables
├── outputs.tf        # Output values
├── terraform.tfvars  # Variable values (don't commit!)
└── .gitignore        # Ignore secrets and state

Initial Configuration: Provider

# main.tf
terraform {
  required_version = ">= 1.0"

  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.23"
    }
  }
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "minikube"
}

What's happening here?

terraform: We declare requirements
- Minimum Terraform version
- Providers we'll use and their versions
provider "kubernetes": We configure connection
- config_path: Where the kubeconfig is (credentials)
- config_context: Which cluster to use (can have multiple)

Terraform will read your ~/.kube/config (same file that kubectl uses) and authenticate to the cluster.

Creating Namespaces: The Simplest

resource "kubernetes_namespace" "ollama" {
  metadata {
    name = "ollama"
    labels = {
      managed-by = "terraform"
      app        = "ollama"
      env        = "development"
    }
  }
}

resource "kubernetes_namespace" "librechat" {
  metadata {
    name = "librechat"
    labels = {
      managed-by = "terraform"
      app        = "librechat"
      env        = "development"
    }
  }
}

Important concepts:

Resource: Terraform's basic unit

resource "TYPE" "LOCAL_NAME" {
  # configuration
}

TYPE: Which resource to create (kubernetes_namespace)
LOCAL_NAME: How to reference it in Terraform code
The real resource will have metadata.name (in K8s)

Labels: Organizational metadata

managed-by = "terraform": Indicates who manages this resource
Useful for filtering: kubectl get ns -l managed-by=terraform

Managing Secrets: Sensitive Variables

# variables.tf
variable "jwt_secret" {
  description = "JWT secret for LibreChat"
  type        = string
  sensitive   = true
}

variable "jwt_refresh_secret" {
  description = "JWT refresh secret for LibreChat"
  type        = string
  sensitive   = true
}

variable "creds_key" {
  description = "Credentials encryption key"
  type        = string
  sensitive   = true
}

variable "creds_iv" {
  description = "Credentials initialization vector"
  type        = string
  sensitive   = true
}

# main.tf
resource "kubernetes_secret" "librechat_credentials" {
  metadata {
    name      = "librechat-credentials-env"
    namespace = kubernetes_namespace.librechat.metadata[0].name
  }

  data = {
    JWT_SECRET         = var.jwt_secret
    JWT_REFRESH_SECRET = var.jwt_refresh_secret
    CREDS_KEY          = var.creds_key
    CREDS_IV           = var.creds_iv
    MONGO_URI          = "mongodb://librechat-mongodb:27017/LibreChat"
    MEILI_HOST         = "http://librechat-meilisearch:7700"
    OLLAMA_BASE_URL    = "http://ollama.ollama.svc.cluster.local:11434"
  }

  type = "Opaque"
}

Security patterns:

Sensitive variables:

   sensitive = true
   # Terraform won't show values in logs

Separate file (terraform.tfvars):

   jwt_secret         = "abc123..."
   jwt_refresh_secret = "def456..."
   creds_key          = "ghi789..."
   creds_iv           = "jkl012..."

CRITICAL: Add to .gitignore!

Dynamic references:

   namespace = kubernetes_namespace.librechat.metadata[0].name
   # Terraform creates the namespace FIRST, then uses the name
   # Automatic dependency tracking!

ConfigMaps: Versioned Configuration

resource "kubernetes_config_map" "librechat_config" {
  metadata {
    name      = "librechat-config"
    namespace = kubernetes_namespace.librechat.metadata[0].name
  }

  data = {
    "librechat.yaml" = <<-EOT
      version: 1.1.5
      cache: true

      endpoints:
        custom:
          - name: "Ollama"
            apiKey: "ollama"
            baseURL: "http://ollama.ollama.svc.cluster.local:11434/v1"
            models:
              default:
                - "llama2:latest"
              fetch: true
            titleConvo: true
            titleModel: "llama2:latest"
            summarize: false
            forcePrompt: false
            modelDisplayLabel: "Ollama"
            addParams:
              temperature: 0.7
              max_tokens: 2000
    EOT
  }
}

Heredoc syntax (<<-EOT ... EOT):

Allows multi-line strings
Automatic indentation
Perfect for YAML inside HCL

Advantage over manual:

# Before:
kubectl create configmap librechat-config --from-file=config.yaml

# Now:
git diff librechat_config.tf
# See exactly what changed in the configuration

The Terraform Workflow

# 1. Initialize (first time)
terraform init
# Downloads providers, prepares backend

# 2. Validate syntax
terraform validate
# Checks if HCL is correct

# 3. Format code
terraform fmt
# Standardizes formatting

# 4. Plan changes
terraform plan
# Preview what will happen

# 5. Apply changes
terraform apply
# Creates/updates resources

# 6. View current state
terraform state list
# Lists all managed resources

What terraform plan shows:

Terraform will perform the following actions:

  # kubernetes_namespace.ollama will be created
  + resource "kubernetes_namespace" "ollama" {
      + id = (known after apply)
      + metadata {
          + generation       = (known after apply)
          + name             = "ollama"
          + labels           = {
              + "app"        = "ollama"
              + "env"        = "development"
              + "managed-by" = "terraform"
            }
        }
    }

  # kubernetes_secret.librechat_credentials will be created
  + resource "kubernetes_secret" "librechat_credentials" {
      + data = (sensitive value)
      + id   = (known after apply)
      + type = "Opaque"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Key points:

+ = Will be created
~ = Will be updated
- = Will be destroyed
(sensitive value) = Hidden for security
(known after apply) = Terraform doesn't know yet (will be generated)

Deploying a Complete Application

Here's where things get... verbose.

resource "kubernetes_deployment" "ollama" {
  metadata {
    name      = "ollama"
    namespace = kubernetes_namespace.ollama.metadata[0].name
    labels = {
      app = "ollama"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        app = "ollama"
      }
    }

    template {
      metadata {
        labels = {
          app = "ollama"
        }
      }

      spec {
        container {
          name  = "ollama"
          image = "ollama/ollama:latest"

          port {
            container_port = 11434
          }

          volume_mount {
            name       = "ollama-data"
            mount_path = "/root/.ollama"
          }

          resources {
            limits = {
              "nvidia.com/gpu" = "1"
            }
            requests = {
              cpu    = "1"
              memory = "4Gi"
            }
          }

          env {
            name  = "OLLAMA_HOST"
            value = "0.0.0.0"
          }
        }

        volume {
          name = "ollama-data"
          persistent_volume_claim {
            claim_name = kubernetes_persistent_volume_claim.ollama_data.metadata[0].name
          }
        }
      }
    }
  }
}

resource "kubernetes_persistent_volume_claim" "ollama_data" {
  metadata {
    name      = "ollama-data"
    namespace = kubernetes_namespace.ollama.metadata[0].name
  }

  spec {
    access_modes = ["ReadWriteOnce"]
    resources {
      requests = {
        storage = "10Gi"
      }
    }
  }
}

resource "kubernetes_service" "ollama" {
  metadata {
    name      = "ollama"
    namespace = kubernetes_namespace.ollama.metadata[0].name
  }

  spec {
    selector = {
      app = "ollama"
    }

    port {
      port        = 11434
      target_port = 11434
    }

    type = "ClusterIP"
  }
}

resource "kubernetes_ingress_v1" "ollama" {
  metadata {
    name      = "ollama-ingress"
    namespace = kubernetes_namespace.ollama.metadata[0].name
    annotations = {
      "nginx.ingress.kubernetes.io/rewrite-target" = "/"
    }
  }

  spec {
    ingress_class_name = "nginx"

    rule {
      host = "ollama.glukas.space"

      http {
        path {
          path      = "/"
          path_type = "Prefix"

          backend {
            service {
              name = kubernetes_service.ollama.metadata[0].name
              port {
                number = 11434
              }
            }
          }
        }
      }
    }
  }
}

That's 160+ lines for ONE application!

Compare:

Helm (Chapter 1): 1 command + ~10 lines of values.yaml
kubectl manual: ~200 lines of YAML
Terraform + K8s Provider: ~160 lines of HCL

Terraform didn't win much in simplicity here.

The Problems Emerge:

Now comes the crucial part—the problems that only appear when you try to use this for real.

Problem 1: State Explosion

terraform apply
# Creates resources...

terraform state list

Output:

kubernetes_namespace.ollama
kubernetes_persistent_volume_claim.ollama_data
kubernetes_deployment.ollama
kubernetes_replica_set.ollama-5d9c8f7b6d        # Created automatically
kubernetes_pod.ollama-5d9c8f7b6d-xk2j9          # Created by ReplicaSet
kubernetes_service.ollama
kubernetes_endpoints.ollama                     # Created by Service
kubernetes_endpoint_slice.ollama-xxxxx          # Created by Service
kubernetes_ingress_v1.ollama

The problem:

Kubernetes creates resources automatically:

Deployment → creates ReplicaSet
ReplicaSet → creates Pods
Service → creates Endpoints and EndpointSlices

Terraform tracks all of them in state, even though you didn't declare them explicitly.

Consequences:

Giant state file

   ls -lh terraform.tfstate
   # 2.3MB for just 2 applications

Slow plans

   terraform plan
   # Needs to query state of hundreds of resources
   # Takes 30+ seconds

Fragility
- If a Pod dies and gets recreated, state becomes inconsistent
- terraform refresh tries to sync, but it's expensive

Problem 2: Inevitable Drift

# Deployed with 1 replica
terraform apply

# User scales manually (common in production)
kubectl scale deployment ollama --replicas=3

# Terraform doesn't detect it!
terraform plan
# Output: No changes. Infrastructure is up-to-date.

# But kubectl shows 3 Pods running
kubectl get pods -n ollama
# ollama-xxx-aaa
# ollama-xxx-bbb
# ollama-xxx-ccc

Why does this happen?

Terraform compares:

State file (snapshot of what was created)
Code (desired state)

Changes made directly with kubectl aren't reflected in the state file immediately. Only when you run terraform refresh or terraform apply.

In dynamic environments:

HPAs (Horizontal Pod Autoscalers) change replicas
Teams do hotfixes via kubectl
CI/CD pipelines update deployments

Terraform is constantly outdated.

Problem 3: No Natural Rollback

With Helm:

# Current version works
helm list
# ollama  1  deployed

# Deploy new version
helm upgrade ollama ollama-helm/ollama -f new-values.yaml
# ollama  2  deployed

# New version broke!
helm rollback ollama
# ollama  3  deployed (back to rev 1 state)

Helm maintains release history. Rollback is instantaneous.

With Terraform:

# Initial deploy
terraform apply

# Change in code
vim main.tf
terraform apply

# It broke! How to go back?
# Option 1: Git revert
git revert HEAD
terraform apply
# Can take minutes to recreate resources

# Option 2: State manipulation (dangerous)
terraform state rm ...
terraform import ...
# Risky and manual

There's no native concept of "release" or "revision".

Problem 4: Complex Lifecycle

Kubernetes has resources that manage other resources:

Deployment manages ReplicaSets
ReplicaSet manages Pods
Service manages Endpoints

Terraform wasn't designed for this. It expects to manage resources directly, not via controllers.

Practical example:

resource "kubernetes_deployment" "ollama" {
  spec {
    replicas = 2
  }
}

# Terraform creates:
# 1. Deployment
# 2. ReplicaSet (created by Deployment controller)
# 3. Pods (created by ReplicaSet controller)

# If you delete the Deployment:
terraform destroy -target kubernetes_deployment.ollama

# Pods die BEFORE the Deployment is deleted
# Destruction order is hard to control

Problem 5: In-Place Updates vs Replacements

# Simple change: update image
resource "kubernetes_deployment" "ollama" {
  spec {
    template {
      spec {
        container {
          image = "ollama/ollama:v0.1.21"  # was v0.1.20
        }
      }
    }
  }
}

terraform plan

Expected output:

~ kubernetes_deployment.ollama will be updated in-place

Actual output:

-/+ kubernetes_deployment.ollama must be replaced

Terraform sometimes decides resources need to be recreated instead of updated, causing unnecessary downtime.

When to Use Each Approach

Comparison Table

Aspect	kubectl Manual	Terraform + K8s	Helm	Verdict
Initial setup	Instant	Configuration	1 command	Helm wins
Versioning	None	Git	Via Git	Terraform/Helm
Reproducibility	Low	High	High	Terraform/Helm
Rollback	Manual	Via Git	Native	Helm wins
State management	N/A	Complex	Simple	Helm wins
Drift detection	None	Limited	Good	Helm wins
Apply time	Fast	Slow	Fast	kubectl/Helm
Learning curve	Medium	High	Medium	kubectl/Helm

When Terraform + K8s Provider Makes Sense

Use for:

Base infrastructure resources:

   # Namespaces
   resource "kubernetes_namespace" "team_apps" { }

   # RBAC
   resource "kubernetes_role" "developer" { }
   resource "kubernetes_role_binding" "dev_binding" { }

   # Storage Classes
   resource "kubernetes_storage_class" "fast_ssd" { }

Resources that rarely change:
- Network Policies
- Resource Quotas
- Limit Ranges
- Priority Classes
Integration with other providers:

   # Create AWS infra and configure K8s in one go
   resource "aws_eks_cluster" "main" { }

   resource "kubernetes_namespace" "app" {
     depends_on = [aws_eks_cluster.main]
   }

Avoid for:

Complete applications (use Helm)
Frequently changing resources (use GitOps)
Multiple interdependent components (use Helm charts)
When an official chart already exists (don't reinvent)

Conclusion of Chapter 2: The Middle Ground Exists

Terraform + Kubernetes Provider isn't a bad approach—it's an incomplete approach.

What we learned:

Terraform is excellent for:

Versioning and auditing
Guaranteed reproducibility
Multi-cloud integration
Base resource management

Terraform is problematic for:

Complex deployments (too verbose)
Resources managed by controllers (state explosion)
Frequently changing applications (drift)
Rollbacks and lifecycle management

The natural question emerges:

"Is there a way to combine the best of both worlds?

Terraform's versioning + Helm's simplicity?"

Answer: YES!

And that's exactly what we'll explore in Chapter 3.

Next Chapter: A Better Abstraction

In Chapter 3, we'll discover that Terraform can manage Helm releases. Instead of describing each Kubernetes resource manually, we'll treat Helm charts as "deployable units" and use Terraform only to orchestrate.

Continue to:

Chapter 3: A Better Abstraction — Managing LLM Apps with Terraform + Helm →

Additional Resources

To dive deeper into Terraform:

Terraform: Up & Running — The definitive book
Terraform Best Practices — Community patterns
HashiCorp Learn — Official tutorials

To better understand Kubernetes:

Kubernetes Patterns — Design patterns
Controllers and Operators — How K8s works internally

Chapter 1: Kubernetes — Operational Fundamentals

George Lukas — Fri, 06 Feb 2026 22:51:25 +0000

You've probably had this thought: "What if I could run my own AI, without depending on external APIs, without token limits, without privacy concerns?"

You're not alone. We're living in a historic moment where running cutting-edge language models is no longer the exclusive privilege of tech giants. With a reasonable NVIDIA GPU and the right tools, you can have your own AI infrastructure, complete, private, under your total control.

Why Self-Hosting LLMs Matters

Real Privacy, Not Just Expectations

Your confidential data, strategic conversations, proprietary code—everything stays on your infrastructure. No logs on third-party servers, no training models with your information, no terms of service that change overnight.

Predictable Cost vs. API Uncertainty

Public API:
$20/month base + $0.002/1k tokens = ???
(how much did that project that exploded cost again?)

Self-hosted:
GPU cloud: $0.50/hour = $360/month fixed
(or own hardware: one-time cost)

Unlimited Experimentation

Want to test 47 prompt variations? Do fine-tuning with proprietary datasets? Run exhaustive benchmarks? With your infrastructure, you're not burning credits or hitting rate limits—you're using resources that are already yours.

Customized and Specialized Models

The real magic happens when you go beyond generic models. Fine-tuning for your specific domain, custom embeddings, specialized agents—all of this requires total control over the stack.

Your journey begins...

You download Ollama, run docker run, pull the llama3:latest model, and boom—you're chatting with an AI running on your hardware. Magical.

Then the questions start:

"How do I put a nice web interface on this?"
"What if I want multiple models with different contexts?"
"How do I backup conversations?"
"What if I want to add semantic search?"
"How do I make this accessible to my team without exposing it to the world?"
"What happens if my machine reboots?"

Suddenly, you're no longer playing with Docker—you're building distributed infrastructure.

The Gap Between "Works on My Machine" and "Production System"

This is the moment where many get stuck with fragile solutions:

# The solution that works... until it doesn't:
$ docker run -d ollama
$ docker run -d mongodb
$ docker run -d librechat
# (Ctrl+C to exit)
# (Machine reboots)
# (Everything dies)
# (You don't remember the exact commands)
# (Configurations are lost)

You need:

Orchestration of multiple interdependent services
Guaranteed data persistence
Automatic service discovery
Intelligent traffic routing
Secrets and configuration management
High availability and self-recovery
Scalability when growing

In other words: you need Kubernetes.

From Self-Hosting to Enterprise Infrastructure

This guide was born exactly from this need. It started with the simple desire to run Ollama + LibreChat locally, and evolved into a complete exploration of how to build modern infrastructure that:

Survives reboots
Can be recreated from scratch in minutes
Has versioning and change auditing
Scales from a laptop to dozens of servers
Follows the same standards that large companies use

What you'll build:

A complete self-hosted AI stack with:

Ollama — Your AI models running with GPU acceleration
LibreChat — Elegant ChatGPT-style interface
MongoDB — Persistence of conversations and contexts
MeiliSearch — Semantic search
NGINX Ingress — Secure and professional access

What you'll learn:

Much more than just "making it work". You'll understand the complete evolution of modern infrastructure:

Chapter 1: Kubernetes — Operational Fundamentals: Configure everything by hand to understand concepts
Chapter 2: Terraform + Kubernetes Provider — Infrastructure as Code: Automate with Infrastructure as Code
Chapter 3: Terraform + Helm — A Better Abstraction: Correct abstractions emerge
Chapter 4: GitOps with Terraform + ArgoCD — A LLM Self-Hosting Product Infrastructure: Enterprise patterns that scale

Motivation

Self-hosting isn't just about saving money or having privacy. It's about technological autonomy.

When you master the complete stack—from the AI model to the infrastructure that supports it—you're no longer dependent on:

Companies that can suddenly shut down APIs
Unilateral price changes
Arbitrary usage restrictions
Latencies from distant datacenters

You have real control over one of the most transformative technologies of our era.

And the best part? The skills you'll develop managing infrastructure for LLMs are exactly the same ones needed to manage any modern distributed system. You're learning Kubernetes through a real use case.

Part I: Fundamentals — Understanding the Ecosystem

What Are We Building?

Throughout this journey, we'll deploy a complete and real stack:

Ollama — AI model backend with NVIDIA GPU acceleration

A stateful application that needs specialized resources (GPU) and careful configuration.

LibreChat — Modern web interface for interacting with LLMs

Frontend application that communicates with multiple backends.

MongoDB — NoSQL database

Persistent storage with backup and recovery requirements.

MeiliSearch — Search engine

Indexes that need to be maintained and synchronized.

NGINX Ingress — Intelligent traffic routing

Gateway for all HTTP/HTTPS traffic.

Technology Stack:

# Foundation
Kubernetes: 1.28+
  └─ Minikube: Local cluster for development
  └─ Docker: Container runtime
  └─ NVIDIA Container Toolkit: Bridge to GPUs

# Management
kubectl: Official Kubernetes CLI
Helm: Package manager (think npm/apt for K8s)
Terraform: Infrastructure as Code (comes in chapters 2-4)
ArgoCD: GitOps operator (chapter 4)

# Applications
Ollama: ollama/ollama (with GPU)
LibreChat: ghcr.io/danny-avila/librechat
MongoDB: bitnami/mongodb
MeiliSearch: getmeili/meilisearch

Each tool here has a specific purpose, and throughout the journey you'll understand exactly when and why to use each one.

Chapter 1: Kubernetes "By Hand" — Learning Through Direct Experience

It may seem counterproductive to start doing everything manually when the final goal is total automation. But there's method to this apparent madness:

Solid conceptual foundation — You need to understand Pods, Services, Ingress, internal DNS
Effective debugging — When something breaks in production (and it will), you need to know what to look for
Appreciation of abstractions — You only value Helm/Terraform after writing YAML manually
Knowledge of conventions — Each tool has its idiosyncrasies

Preparing the Environment: Your Local Cluster

First, let's create a safe playground—a Kubernetes cluster running locally on your machine.

Minikube Installation

# Arch Linux (adjust for your distro)
pacman -S minikube nvidia-container-toolkit libnvidia-container helm ollama terraform

# Start cluster with GPU Support
minikube start \
  --driver docker \
  --container-runtime docker \
  --gpus all \
  --memory 8192 \
  --cpus 4

What's happening here?

When you execute minikube start, a sequence of events is triggered:

Node creation: Minikube creates a Docker container that acts as your cluster's "server"
Kubernetes installation: Inside this container, K8s core components are installed:
- kube-apiserver — The brain, receives all requests
- etcd — Distributed database that stores state
- kube-scheduler — Decides which node each Pod will run on
- kube-controller-manager — Ensures current state = desired state
- kubelet — Agent running on each node, manages containers
Container runtime: Docker is configured as the runtime (who actually runs the containers)
GPU exposure: --gpus all maps the host's GPUs into the cluster

The result? A real Kubernetes cluster, running completely on your machine, isolated and safe for experimentation.

Enabling the Ingress Controller

# Activate NGINX Ingress addon
minikube addons enable ingress

# Verify
minikube kubectl -- get pods -n ingress-nginx

What is Ingress?

Think of Ingress as the "smart front door" of your cluster. Instead of each service having its own external IP, Ingress acts as a reverse proxy that:

Routes traffic based on hostnames and paths
Handles SSL/TLS termination
Provides load balancing
Implements rate limiting and authentication

Without Ingress:

User → NodePort :30080 → Pod1
User → NodePort :30081 → Pod2
User → NodePort :30082 → Pod3

You would need to remember each port for application.

With Ingress:

User → http://app1.company.com → Ingress → Pod1
User → http://app2.company.com → Ingress → Pod2
User → http://app3.company.com → Ingress → Pod3

You have intuitive hostnames.

Part II: Deploying the Stack

Now we'll deploy our applications. We'll start with Ollama (the AI backend) and then add LibreChat (the interface).

1. Installing Ollama with GPU Support

Creating the namespace:

minikube kubectl -- create ns ollama

Namespaces are logical separations within the cluster. Think of them as "folders" for organizing resources.

cluster/
├── default/          # Defalt Namespace, avoid using
├── kube-system/      # K8s core components
├── kube-public/      # Public resources
├── ollama/           # Ollama 
└── librechat/        # Librechat

This keeps Ollama isolated from other applications.

Namespace also provides:

Organization: Logically separates applications
Permissions: RBAC can be applied by namespace
Quotas: Limit resources by application
Network isolation: Policies can restric communication

Installing via Helm:

Helm is often called "Kubernetes package manager", more than that, it's also a template engine with lifecycle management.

# Add repo
helm repo add ollama-helm https://otwld.github.io/ollama-helm/
helm repo update

# Create configuration file
cat > ollama-values.yaml <<EOF
ollama:
  gpu:
    enabled: true
    type: nvidia
    number: 1
  models:
    - llama2

ingress:
  enabled: true
  hosts:
    - host: ollama.glukas.space
      paths:
        - path: /
          pathType: Prefix
EOF

# Install
helm install ollama ollama-helm/ollama \
  -f ollama-values.yaml \
  --namespace ollama

What's Helm doing?

When you run helm install, Helm:

Template rendering:

   values.yaml + chart templates → YAMLs do Kubernetes

Helm take values and inject them on chart templates, generating Deployments, Services, ConfigMaps, etc.

Dependency resolution:
If chart have dependencies(sub-charts), Helm resolve it, then installs.
Pre-install hooks:
Some charts execute jobs before installation (such migrations, etc)
Resource creation:
Rendered YAMLs are then applied on cluster via kubectl apply
Release tracking:
Helm then save the metadata on cluster, allowing upgrades and rollbacks

Verifying the installation:

By running minikube kubectl get all -n ollama:

NAME                               READY             STATUS              RESTARTS      AGE
pod/ollama-5d9c8f7b6-xk2j9         1/1               Running             0             2m

NAME                               TYPE              CLUSTER-IP         EXTERNAL-IP   PORT(S)
service/ollama                     ClusterIP         10.96.234.123      <none>        11434/TCP

NAME                               READY             UP-TO-DATE         AVAILABLE     AGE
deployment.apps/ollama             1/1               1                  1             2m

NAME                               DESIRED           CURRENT            READY         AGE
replicaset.apps/ollama-5d9c8f7b6   1                 1                  1             2m

Each line represents a Kubernetes resource created by Helm.

Local DNS Configuration

This is the IP you'll use to access services exposed through Ingress.

# Cluster IP
minikube ip
# Output example: 192.168.49.2

# Add to /etc/hosts
echo "192.168.49.2 ollama.glukas.space" | sudo tee -a /etc/hosts

How it works?

Your browser tries resolves ollama.glukas.space
OS first query /etc/hosts
Finds 192.168.49.2 (Minikube IP)
Request goes to k8s cluster
Ingress Controller see the header Host: ollama.glukas.space
Routes to Ollama service
Service distributes to Pods

Pulling an AI model:

# Execute a command inside the Ollama pod
minikube kubectl exec -n ollama deploy/ollama -- \
  ollama pull llama3

# This downloads the Llama 3 model
# The first time takes a few minutes (several GB)

Testing the model:

minikube kubectl exec -n ollama deploy/ollama -- \
  ollama run llama3 "Explain Kubernetes in one sentence"

# Response:
# Kubernetes is a container orchestration platform that automates
# deployment, scaling, and management of containerized applications
# across a cluster of machines.

If you got a response, congratulations! You have a local LLM running with GPU acceleration.

Testing access:

curl http://ollama.glukas.space/api/tags

# Expected response: JSON with available models
{
  "models": [
    {
      "name": "llama3:latest",
      "modified_at": "2024-01-15T10:30:00Z",
      ...
    }
  ]
}

If the JSON appeared, Ollama is successfully accessible via Ingress!

LibreChat Deploy:

Now comes the fun part: a proper web interface to interact with your AI.

Understanding LibreChat's Architecture

LibreChat is more than just a frontend. It's a complete application that needs:

Database (MongoDB) — To store conversations, users, settings
Search (MeiliSearch) — For semantic search in conversations
Backend — Node.js API that orchestrates everything
Frontend — React interface

The official Helm chart manages all of this.

Creating the namespace:

minikube kubectl -- create bs librechat

Creating Secrets:

# Generate secrets
JWT_SECRET=$(openssl rand -hex 32)
JWT_REFRESH_SECRET=$(openssl rand -hex 32)
CREDS_KEY=$(openssl rand -hex 32)
CREDS_IV=$(openssl rand -hex 16)

# Kubernetes secret creation
kubectl create secret generic librechat-credentials-env \
  --from-literal=JWT_SECRET="${JWT_SECRET}" \
  --from-literal=JWT_REFRESH_SECRET="${JWT_REFRESH_SECRET}" \
  --from-literal=CREDS_KEY="${CREDS_KEY}" \
  --from-literal=CREDS_IV="${CREDS_IV}" \
  --from-literal=MONGO_URI="mongodb://librechat-mongodb:27017/LibreChat" \
  --from-literal=MEILI_HOST="http://librechat-meilisearch:7700" \
  --from-literal=OLLAMA_BASE_URL="http://ollama.ollama.svc.cluster.local:11434" \
  --namespace librechat

Secrets vs ConfigMaps: When to Use Each?

	ConfigMap	Secret
Purpose	Public configuration	Sensitive data
Encoding	Plain text	Base64 (not encryption!)
Typical Use	Feature flags, public URLs	Passwords, tokens, keys
Auditing	Can be committed to Git	NEVER commit
Example	`API_ENDPOINT=https://api.com`	`API_KEY=sk_live_xxxxx`

Important: Base64 is not encryption! It's just encoding. In production, use solutions such as:

Sealed Secrets (Bitnami)
External Secrets Operator + HashiCorp Vault
SOPS (Mozilla)
Cloud provider KMS (AWS Secrets Manager, GCP Secret Manager)

Configuration file: librechat-values.yaml

cat > librechat-values.yaml <<'EOF'
# Env Config (Level 1)
config:
  APP_TITLE: "LibreChat + Ollama"
  HOST: "0.0.0.0"
  PORT: "3080"
  SEARCH: "true"
  MONGO_URI: "mongodb://librechat-mongodb:27017/LibreChat"
  MEILI_HOST: "http://librechat-meilisearch:7700"

# Application Conig (Level 2)
librechat:
  configEnv:
    ALLOW_REGISTRATION: "true"

  # YAML Injection (Level 3)
  configYamlContent: |
    version: 1.1.5
    cache: true

    endpoints:
      custom:
        - name: "Ollama"
          apiKey: "ollama"
          # Kubernetes internal DNS: <service>.<namespace>.svc.cluster.local
          baseURL: "http://ollama.ollama.svc.cluster.local:11434/v1"
          models:
            default:
              - "llama2:latest"
            fetch: true
          titleConvo: true
          titleModel: "llama2:latest"
          summarize: false
          summaryModel: "llama2:latest"
          forcePrompt: false
          modelDisplayLabel: "Ollama"
          addParams:
            temperature: 0.7
            max_tokens: 2000

# Reference to Secret previously created
extraEnvVarsSecret: "librechat-credentials-env"

# External access configuration
ingress:
  enabled: true
  className: "nginx"
  hosts:
    - host: librechat.glukas.space
      paths:
        - path: /
          pathType: Prefix

# Sub-charts (dependencies)
mongodb:
  enabled: true
  auth:
    enabled: false  # NEVER in production
  image:
    repository: bitnami/mongodb
    tag: latest
  persistence:
    enabled: true
    size: 8Gi

meilisearch:
  enabled: true
  auth:
    enabled: false  # NEVER in production
  environment:
    MEILI_NO_ANALYTICS: "true"
    MEILI_ENV: "development"
  persistence:
    enabled: true
    size: 1Gi

# Main application storage
persistence:
  enabled: true
  size: 5Gi
  storageClass: "standard"
EOF

Configuration breaking down:

Level 1 — Environment Variables Simples

config:
  APP_TITLE: "LibreChat + Ollama"

Environment variables injected on container.

Level 2 — Application Configuration

librechat:
  configEnv:
    ALLOW_REGISTRATION: "true"

LibreChat specific configuration(it will also be env vars).

Level 3 — Nested YAML

configYamlContent: |
  version: 1.1.5
  endpoints:
    custom: ...

A YAML file that will be created inside thecontainer. O pipe | preserves line breaks.

Internal Kubernetes DNS:

LibreChat inside the cluster talks to Ollama via internal DNS
No need to go through external Ingress

Kubernetes FQDN (Fully Qualified Domain Name) Anatomy:

baseURL: "http://ollama.ollama.svc.cluster.local:11434/v1"

http://ollama.ollama.svc.cluster.local:11434/v1
       └─┬─┘  └─┬─┘  └┬┘ └────┬─────┘  └─┬─┘└┬┘
      Service   NS   Type   Domain     Port Path

ollama — Service Name
ollama — Namespace where the Service is
svc — Resource Type (service)
cluster.local — cluster Domain
11434 — Port
/v1 — API Path

Shorthand that also works (inside the same namespace):

http://ollama:11434              # Same namespace
http://ollama.ollama:11434       # Explicit Namespace
http://ollama.ollama.svc:11434   # With Type

Why using full FQDN?

LibreChat is inside librechat namespace and needs to access Ollama on ollama namespace. Cross-namespace communication requires <service>.<namespace>.

Deploying

# Install LibreChat and its dependencies
helm install librechat oci://ghcr.io/danny-avila/librechat-chart/librechat \
  -f librechat-values.yaml \
  --namespace librechat

# Watch the deploy
watch minikube kubectl -- get pods -n librechat -w

Watch to Expect:

NAME                                     READY   STATUS              RESTARTS   AGE
librechat-5d8f4b6c9d-xk7p2               0/1     ContainerCreating   0          10s
librechat-mongodb-0                      0/1     Pending             0          10s
librechat-meilisearch-7f9d8c5b6-j4k8m    0/1     ContainerCreating   0          10s

Wait Status to be Running and READY to be 1/1.

Testing the System

# Adding to /etc/hosts
echo "192.168.49.2 librechat.glukas.space" | sudo tee -a /etc/hosts

# Access in browser
# http://librechat.glukas.space

If everything worked, you'll see the LibreChat interface. Create an account, configure the Ollama model, and chat with the AI running completely locally.

Let's ask some difficult questions:

1. Reproducibility: If I delete the cluster now, can you recreate everything exactly the same?

Technically yes, but you'll need to:

Remember all the commands in the right order
Recreate all secrets with the same values
Hope the charts haven't changed
Hope you didn't forget to document some configuration

2. Versioning: How do you rollback if something breaks?

You don't. Helm has releases, but the configurations are on your machine. If you changed a value three weeks ago, good luck remembering what it was before.

3. Auditing: Who changed what and when?

Nobody knows. There's no record.

4. Consistency: How do you ensure dev, staging, and production are the same?

Copy and paste values between environments. And pray.

5. Drift Detection: Is the cluster as you think it is?

Maybe. Someone might have run kubectl edit directly. You'll never know.

Fundamental Problems of the Manual Approach

1. The Reproducibility Problem

# What you DID:
$ kubectl create namespace ollama
$ helm install ollama ...
$ kubectl create secret ...
$ helm install librechat ...

# What you HAVE documented:
$ # (cricket sounds)

You made it work, but this knowledge is in your head and bash history. If the cluster dies tomorrow, you'll need to rebuild everything from memory or by mining logs.

2. The Distributed State Problem

Your system now has state in several places:

Helm releases — Metadata in the cluster
Secrets — Created via kubectl
Configurations — YAML files on your machine
DNS — Entries in /etc/hosts
Commands — Bash history

There's no single source of truth.

3. The Scale Problem

This worked for 2 applications. Now imagine:

10 applications
3 environments (dev, staging, prod)
5 people on the team

Will you create 30 manual configurations? And when someone on the team updates one, how do the others know?

4. The Drift Problem

# On day 1:
$ helm install app v1.0

# On day 30, someone does:
$ kubectl edit deployment app

# Now there's divergence between:
# - What Helm thinks it deployed
# - What's actually running
# - What you think is running

Drift is when the current state diverges from the desired state. In manual infrastructure, drift is invisible until it breaks.

5. The Collaboration Problem

Developer A: "I changed the Ollama configuration"
Developer B: "Which one? Where?"
Developer A: "In my values.yaml"
Developer B: "Did you commit it?"
Developer A: "Was I supposed to commit it?"

Without centralized versioning, collaboration is chaotic.

Lessons from Chapter 1:

Despite the problems mentioned, this manual implementation was essential:

You now understand Kubernetes from the inside

Pods aren't abstractions, they're running processes
Services aren't magic, they're iptables rules
Ingress isn't complex, it's HTTP proxy with rules

You gained technical vocabulary

Can talk about namespaces, secrets, internal DNS
Understand what Helm does vs what Kubernetes does
Know the difference between StatefulSet and Deployment

You felt the pain that motivates automation

Each manual command is an opportunity for error
Each unversioned configuration is technical debt
Each manual change is impossible to audit

You built intuition about the ecosystem

Know when to use Helm chart vs pure YAML
Understand design trade-offs (simplicity vs flexibility)
Can debug networking and DNS problems

The Next Step: Infrastructure as Code

Now that you deeply understand what is running and why, you're ready to learn how to automate.

In Chapter 2, we'll take everything we did here and transform it into versioned, reproducible, and auditable code. We'll introduce Terraform and explore the first approach to managing Kubernetes with Infrastructure as Code.

Next Chapters

Chapter 2: Terraform + Kubernetes Provider — Infrastructure as Code

How to use Terraform to manage Kubernetes resources directly. We'll explore why this seems like a good idea, where it works well, and where it starts to break. We'll learn about state management, change planning, and the limits of the approach.

Chapter 3: Terraform + Helm — A Better Abstraction

The correct abstraction emerges. We'll use Terraform to manage Helm releases, combining the best of both worlds: Terraform's versioning and auditing with Helm's packaging expertise.

Chapter 4: GitOps with Terraform + ArgoCD — A LLM Self-Hosting Product Infrastructure

Robust architecture that completely separates infrastructure management (Terraform) from application management (ArgoCD), implementing real continuous delivery with GitOps, pull-based reconciliation, and multi-tenancy.

Continue to:

Chapter 2: Terraform + Kubernetes Provider — Infrastructure as Code →

Additional Resources

Official Documentation:

Important Concepts:

The Twelve-Factor App — Methodology for cloud-native apps
Kubernetes Patterns — Design patterns for K8s
DNS for Services and Pods — Understanding internal DNS

Community:

Kubernetes Slack
r/kubernetes
CNCF Landscape — Complete map of the cloud-native ecosystem