DEV Community

augustine Egbuna
augustine Egbuna

Posted on • Originally published at fivenineslab.com

Your CI Pipeline Validates Three Languages While Your Codebase Uses Eleven

#devops #kubernetes #observability #docker

Last Tuesday, a Terraform module made it through CI, passed code review, merged to main, and immediately broke three environments because someone had used deprecated syntax from version 0.12. The CI system never ran terraform validate. Why? Because the platform doesn't have a Terraform runner preconfigured, and nobody had 90 minutes to figure out how to add one.

Meanwhile, your team runs eleven different language stacks in production. Your CI platform supports three of them out of the box.

This is the polyglot infrastructure tax: the gap between "supported languages" and "languages we actually use." Every modern infrastructure team hits it. You're running Rust for performance-critical data pipelines, Go for Kubernetes operators, Python for ML serving, TypeScript for internal tools, and Terraform/Pulumi/CDK for infrastructure. Your CI platform pretends six of those don't exist.

The Problem: Language Coverage Is a False Metric

When you evaluate CI platforms, they tout "first-class support for Node, Python, Go!" What they mean: they have three Docker images with those runtimes preinstalled. Everything else requires custom runner configuration, which means:

  1. Someone has to write and maintain a custom Docker image
  2. That image lives in a private registry nobody remembers to update
  3. Security scanning fails because the base image is 18 months old
  4. The next engineer doesn't know the image exists and rebuilds it from scratch

Here's what actually happens in a polyglot shop. Your .gitlab-ci.yml or GitHub Actions workflow looks like this:

# Validates the 3 "blessed" languages
lint-python:
  image: python:3.11
  script:
    - pip install ruff
    - ruff check .

lint-go:
  image: golang:1.21
  script:
    - go vet ./...
    - golangci-lint run

# Everything else? Custom images nobody maintains
lint-terraform:
  image: registry.internal/custom-terraform:latest  # Built 14 months ago
  script:
    - terraform fmt -check
    - terraform validate
  allow_failure: true  # Because it's probably broken
Enter fullscreen mode Exit fullscreen mode

That allow_failure: true is the tell. You've given up. The Terraform validation is aspirational. It breaks every third run because the image is stale, so you mark it as non-blocking. Congratulations: your Terraform code is no longer validated.

Why This Happens: The Docker Image Maintenance Trap

CI platforms are built around the assumption that language runtimes are stable artifacts. Install Node 18, cache it, reuse it for six months. Works great for the big three.

But real infrastructure stacks evolve faster:

  • Terraform releases break backwards compatibility every minor version
  • Rust updates every six weeks
  • Your Kubernetes manifests use kustomize, which you need to pin to match cluster versions
  • Someone added a Scala service and now you need SBT
  • Your data team writes pipelines in Julia

Each of those needs a custom image. You build it once, it works, then six months later:

  • The base image has 14 CVEs
  • The language version is EOL
  • The pipeline fails with a cryptic error because a dependency URL moved

Nobody has time to maintain eleven custom images. So validation coverage degrades. Tests start failing intermittently — "works on my machine" becomes "works in three of our eleven language stacks."

What You Actually Need: Universal Execution, Not Universal Images

The fix isn't better Docker images. It's CI systems that treat language runtimes as just another dependency to download on-demand, the way nix or mise (formerly rtx) handle polyglot environments.

Here's what that looks like in practice. Instead of maintaining a custom Terraform image, your CI config becomes:

validate-terraform:
  script:
    - curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
    - apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
    - apt-get update && apt-get install -y terraform=1.6.0
    - terraform fmt -check
    - terraform validate
Enter fullscreen mode Exit fullscreen mode

Terrible, right? You're installing the runtime every time. But now look at the maintenance cost: zero. No custom image to update. No registry to manage. Terraform 1.7 comes out? Change one line. It's deterministic, auditable, and doesn't require Docker expertise.

Better yet, use a tool that handles this: asdf, mise, or Nix. Here's a .tool-versions file that pins every runtime in your stack:

terraform 1.6.0
golang 1.21.5
python 3.11.7
nodejs 20.10.0
rust 1.75.0
kotlin 1.9.21
elixir 1.15.7
Enter fullscreen mode Exit fullscreen mode

Your CI runner just needs mise installed. Every job runs:

mise install  # Installs whatever .tool-versions specifies
mise exec -- terraform validate
mise exec -- go test ./...
mise exec -- pytest
Enter fullscreen mode Exit fullscreen mode

One tool. Eleven languages. No custom Docker images.

The Observability Blind Spot

Here's what you can't see with three-language CI: where your time is actually going. When Terraform validation is a custom job that fails silently, you have no metrics on:

  • How often it fails (so you stop trusting it)
  • How long it takes (so you can't optimize it)
  • Which repos are skipping it entirely

Production CI systems need per-language tracing. Not just "job succeeded/failed" — you need:

  • Dependency install time (is pip install taking 4 minutes because you're not caching?)
  • Lint/test/build breakdown (is terraform init 80% of your validation time?)
  • Flaky test detection (is your Rust test suite failing 10% of the time because of a known timer race?)

Most CI platforms give you a wall-clock job duration and a red/green status. That's it. You're flying blind.

Here's the metric that matters: per-language pass rate over time. If your Kotlin linting has an 85% pass rate while Go is at 99%, you have a signal. Either the Kotlin checks are misconfigured, or the team needs tooling help. You can't fix what you can't measure.

The Real Cost: Trust Erosion

When your CI system only validates half your stack, engineers stop trusting it. They merge code with a shrug — "CI passed, but it doesn't check Terraform anyway." That shrug is your actual SLA. It's the gap between what CI validates and what breaks production.

You end up with a split workflow:

  • Python/Go/Node: CI validates everything, blocks on failure, team trusts it
  • Everything else: CI is advisory, manual review required, maybe someone runs the tests locally

That second tier is where the outages come from. A bad Terraform apply. A misconfigured Kubernetes manifest. A Rust binary that segfaults because nobody ran cargo clippy before merge.

The fix isn't better code review. It's CI coverage that matches your actual stack. If you write it in production, CI should validate it.

Making This Work Today

You don't need to rip out your existing CI system. Start with an inventory:

  1. List every language your team deploys to production
  2. Check which ones have CI validation (not just "we run tests", but linting, type checking, security scanning)
  3. For the gaps, add a single job with runtime-on-demand installation

Start with Terraform — it's the highest-risk gap in most infrastructure teams. A working validation job looks like:

terraform-validate:
  image: ubuntu:22.04
  before_script:
    - apt-get update && apt-get install -y curl unzip
    - curl -fsSL https://releases.hashicorp.com/terraform/1.6.6/terraform_1.6.6_linux_amd64.zip -o terraform.zip
    - unzip terraform.zip && mv terraform /usr/local/bin/
    - terraform version
  script:
    - terraform fmt -check -recursive
    - terraform init -backend=false
    - terraform validate
Enter fullscreen mode Exit fullscreen mode

No custom image. No registry. Runs in 45 seconds including the Terraform download. Add caching and it's under 10 seconds.

Do this for each language in your stack. Yes, it's tedious. But it's one afternoon of work versus the ongoing cost of maintaining custom images or the even higher cost of unvalidated code reaching production.

The goal isn't perfect CI. It's CI that covers the code you actually ship. If your platform makes that hard, the platform is the problem — not your polyglot stack.

This post is an excerpt from Practical AI Infrastructure Engineering — a production handbook covering Docker, GPU infrastructure, vector databases, and LLM APIs. Full book with 4 hands-on capstone projects available at https://activ8ted.gumroad.com/l/ssmfkx

Originally published at fivenineslab.com

Top comments (0)