Solved: Do you separate template browsing from deployment in your internal IaC tooling?

🚀 Executive Summary

TL;DR: Many organizations struggle with blurred lines between IaC template discovery and deployment, leading to cognitive overload, error-prone provisioning, and slow self-service adoption. This can be solved by separating these processes through dedicated template catalogs, enhanced CLI tools, or GitOps workflows, which improve discoverability, reduce errors, and enforce standardization. By making templates easily accessible and deployments predictable, teams can innovate faster and more securely.

🎯 Key Takeaways

• Coupling IaC template browsing and deployment often results in ‘template sprawl,’ cognitive overload, error-prone deployments, and slow self-service adoption due to lack of clear categorization and documentation.
• Three primary solutions for separation are: a Dedicated Template Catalog (user-friendly UI), a CLI-Driven approach with enhanced discovery (scripting-friendly), and GitOps with Self-Service via Pull Requests (Git-centric, auditable).
• GitOps provides a single source of truth, strong auditability, and built-in review processes through PRs, making it ideal for organizations prioritizing version control and automated reconciliation, despite a higher initial setup complexity.

Navigating and deploying Infrastructure as Code (IaC) templates can be a maze for internal teams. This post explores how to effectively separate template discovery from the deployment process, reducing friction and boosting developer productivity.

Symptoms: When Template Browsing and Deployment Collide

In many organizations, the line between discovering available IaC templates and actually deploying them is blurred, leading to a myriad of operational challenges. This coupling often manifests in several ways, creating friction for development teams and increasing the risk of misconfigurations.

• ### Cognitive Overload and “Template Sprawl”

Developers are often directed to a large Git repository containing hundreds or thousands of IaC files (Terraform modules, CloudFormation templates, Helm charts, etc.). Without clear categorization, documentation, or an intuitive search mechanism, finding the right template for a specific need becomes a daunting task. This “template sprawl” leads to wasted time, duplicated efforts, and a reluctance to use existing, well-vetted IaC.

• ### Error-Prone Deployments

When template parameters are poorly documented or require manual extraction from code, users are prone to making mistakes. Misconfigured resource types, incorrect environment variables, or overlooked security settings can lead to failed deployments, security vulnerabilities, or suboptimal infrastructure.

• ### Slow Provisioning and Low Self-Service Adoption

The complexity of discovering, understanding, and then correctly executing a deployment often forces developers to rely on a central DevOps or Platform team. This creates bottlenecks, delays project timelines, and undermines the goal of empowering teams with self-service infrastructure provisioning.

• ### Lack of Standardization and Guardrails

Without a clear separation or an opinionated pathway, teams might resort to copying and modifying templates haphazardly. This can lead to deviation from organizational standards, compliance drift, and a lack of consistent security controls across the infrastructure landscape.

Solution 1: Dedicated Template Catalog & Self-Service Portal

A dedicated template catalog or self-service portal provides a user-friendly interface for browsing, understanding, and initiating deployments of approved IaC templates. This completely decouples discovery from the underlying deployment mechanisms, offering a guided experience.

How it Works

Users interact with a web-based interface that presents a curated list of IaC templates. Each template entry typically includes:

• A human-readable name and description.
• Categorization (e.g., “Web App,” “Database,” “Networking”).
• Input parameters with descriptions, default values, and validation rules.
• Estimated cost (if integrated with cost management tools).
• Links to detailed documentation or the source code.

Upon selecting a template, the user provides the necessary parameters via a form. The portal then orchestrates the deployment using an underlying CI/CD pipeline or directly through IaC tools, abstracting away the complexities of commands and configuration files.

Examples & Configuration

Tools like Spotify’s Backstage, HashiCorp’s Terraform Cloud/Enterprise Workspaces, or custom-built internal portals can serve this purpose.

Example: Terraform Cloud Workspace (Simplified Concept)

Imagine a template for an “AWS S3 Static Website.” In Terraform Cloud, you’d set up a workspace configured to use this module. The UI would then prompt for specific variables.

# main.tf for S3 static website module (in your template repo)
resource "aws_s3_bucket" "website_bucket" {
  bucket = var.bucket_name
  acl    = "public-read"

  website {
    index_document = "index.html"
    error_document = "error.html"
  }

  tags = {
    Environment = var.environment
    Project     = var.project
  }
}

resource "aws_s3_bucket_policy" "website_policy" {
  bucket = aws_s3_bucket.website_bucket.id

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect    = "Allow",
        Principal = "*",
        Action    = "s3:GetObject",
        Resource  = "${aws_s3_bucket.website_bucket.arn}/*"
      }
    ]
  })
}

variable "bucket_name" {
  description = "The unique name for the S3 bucket."
  type        = string
}

variable "environment" {
  description = "The deployment environment (e.g., dev, staging, prod)."
  type        = string
}

variable "project" {
  description = "The project name associated with this resource."
  type        = string
}

In a portal, these variables (bucket\_name, environment, project) would be presented as form fields, with their descriptions guiding the user. The portal would then trigger a Terraform run in the configured workspace using the user-provided values.

Benefits

• Enhanced Discoverability: Centralized, searchable catalog makes finding templates easy.
• Improved User Experience: Intuitive forms, clear descriptions, and pre-validation reduce errors.
• Standardization & Governance: Only approved templates are available, promoting best practices and compliance.
• Reduced Cognitive Load: Users don’t need to know IaC syntax or CLI commands.

Drawbacks

• Development & Maintenance Overhead: Building and maintaining a custom portal can be resource-intensive.
• Potential for Drift: If the portal’s template definitions aren’t kept in sync with the actual IaC code, it can lead to confusion.
• Limited Flexibility: May not support highly custom or experimental deployments easily.

Solution 2: CLI-Driven with Enhanced Template Discovery & Validation

This approach emphasizes a robust command-line interface (CLI) that provides specific commands for template discovery, detailed parameter explanation, and pre-deployment validation, separate from the actual deployment command. It’s ideal for teams comfortable with CLI tools and scripting.

How it Works

Instead of a GUI, users interact with a custom CLI tool or a set of well-defined scripts. This CLI provides commands such as list-templates, describe-template, and validate-template before deploy-template. The discovery phase focuses on providing rich metadata directly at the command line, often pulling from structured documentation or code comments.

Examples & Configuration

This often involves a wrapper script around your core IaC tools (e.g., Terraform, CloudFormation CLI, Helm). Let’s consider a custom iac-helper CLI:

Example: Custom iac-helper CLI

Imagine a directory structure for your IaC templates:

iac/
├── modules/
│   ├── s3-website/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── README.md
│   └── rds-instance/
│       ├── main.tf
│       ├── variables.tf
│       └── README.md
├── environments/
│   ├── dev/
│   ├── prod/
│   └── staging/
└── templates/
    ├── s3-static-site.yaml # CloudFormation template
    ├── ec2-webserver.json  # CloudFormation template
    └── eks-cluster.tf      # Terraform root module for EKS

Your iac-helper could use jq to parse variables.tf or template.json to extract parameter info. A README.md within each template directory would serve as detailed documentation.

# bash script for iac-helper (simplified)
#!/bin/bash

IAC_ROOT_DIR="iac"

list_templates() {
  echo "Available IaC Templates:"
  find "$IAC_ROOT_DIR/modules" -maxdepth 1 -type d -and -not -name "modules" -print0 | xargs -0 -n 1 basename | while read -r template; do
    echo "  - $template"
  done
  find "$IAC_ROOT_DIR/templates" -maxdepth 1 -type f \( -name "*.tf" -o -name "*.yaml" -o -name "*.json" \) -print0 | xargs -0 -n 1 basename | sed 's/\.tf\|\.yaml\|\.json//g' | while read -r template; do
    echo "  - $template (Root Module)"
  done
}

describe_template() {
  TEMPLATE_NAME=$1
  if [ -d "$IAC_ROOT_DIR/modules/$TEMPLATE_NAME" ]; then
    echo "Description for module: $TEMPLATE_NAME"
    cat "$IAC_ROOT_DIR/modules/$TEMPLATE_NAME/README.md" 2>/dev/null || echo "No README.md found."
    echo ""
    echo "Variables:"
    terraform-docs md "$IAC_ROOT_DIR/modules/$TEMPLATE_NAME" | sed -n '/^| Name | Description | Type | Default | Required |/,/^$/p'
  elif [ -f "$IAC_ROOT_DIR/templates/$TEMPLATE_NAME.tf" ]; then
    echo "Description for root module: $TEMPLATE_NAME.tf"
    # For Terraform root modules, you might describe input variables
    terraform-docs md "$IAC_ROOT_DIR/templates/$TEMPLATE_NAME.tf" | sed -n '/^| Name | Description | Type | Default | Required |/,/^$/p'
  elif [ -f "$IAC_ROOT_DIR/templates/$TEMPLATE_NAME.yaml" ]; then
    echo "Description for CloudFormation template: $TEMPLATE_NAME.yaml"
    # For CloudFormation, parse parameters from YAML
    yq e '.Parameters | to_entries[] | .key + ": " + .value.Description' "$IAC_ROOT_DIR/templates/$TEMPLATE_NAME.yaml"
  else
    echo "Template '$TEMPLATE_NAME' not found."
  fi
}

deploy_template() {
  TEMPLATE_NAME=$1
  # ... logic to run terraform apply or aws cloudformation create-stack
  echo "Deploying $TEMPLATE_NAME..."
  # Example: terraform apply -var="bucket_name=$2" -var="environment=$3" "$IAC_ROOT_DIR/modules/$TEMPLATE_NAME"
}

case "$1" in
  list)
    list_templates
    ;;
  describe)
    if [ -z "$2" ]; then
      echo "Usage: iac-helper describe "
      exit 1
    fi
    describe_template "$2"
    ;;
  deploy)
    # This would be much more complex, requiring dynamic var parsing, envs etc.
    echo "Deploy command not fully implemented in this example."
    echo "Usage: iac-helper deploy  [params...]"
    ;;
  *)
    echo "Usage: iac-helper [list|describe|deploy] ..."
    ;;
esac

Users would then run:

iac-helper list
iac-helper describe s3-website
iac-helper deploy s3-website bucket_name=my-unique-site-name environment=dev

Benefits

• Low Overhead: Less development effort compared to a full UI portal.
• Scripting Friendly: Easily integrates into CI/CD pipelines or local automation scripts.
• Precision & Control: Advanced users can specify exactly what they need.
• Version Control Native: Documentation and helper scripts live alongside the IaC.

Drawbacks

• Steeper Learning Curve: Requires users to be comfortable with CLI tools and syntax.
• Documentation Dependent: Effectiveness heavily relies on comprehensive, up-to-date in-code documentation.
• Less Intuitive: Can be less visual and discoverable for non-technical users.

Solution 3: GitOps with Self-Service via Pull Requests

GitOps fundamentally separates the desired state of infrastructure (defined in Git) from the operational process of achieving that state. Template browsing becomes reviewing available IaC modules and applying them by submitting Pull Requests (PRs) to a configuration repository, which then triggers automated deployment by a GitOps operator.

How it Works

In this model, your core IaC templates (e.g., Terraform modules, Helm charts) reside in one or more “template repositories.” Teams needing to provision infrastructure will browse these repositories to identify the appropriate templates. To deploy, they won’t run terraform apply directly. Instead, they create a PR against a separate “configuration repository” (often one per environment, e.g., infra-prod, infra-dev). This PR defines the desired state, typically by referencing the core templates and providing specific values. A GitOps operator (like Argo CD or Flux CD) monitors this configuration repository. When a PR is merged, the operator detects the change and applies the IaC to achieve the declared state.

Examples & Configuration

Example: Deploying an EKS Cluster with Terraform and Argo CD

Let’s assume you have a Terraform module for an EKS cluster in modules/eks-cluster/.

1. Template Repository (iac-templates):

# modules/eks-cluster/main.tf (simplified)
resource "aws_eks_cluster" "main" {
  name     = var.cluster_name
  role_arn = var.cluster_role_arn
  vpc_config {
    subnet_ids = var.subnet_ids
  }
  # ... other EKS configurations
}

variable "cluster_name" {
  description = "Name of the EKS cluster."
  type        = string
}
variable "cluster_role_arn" {
  description = "ARN of the IAM role for the EKS cluster."
  type        = string
}
variable "subnet_ids" {
  description = "List of subnet IDs for the EKS cluster."
  type        = list(string)
}

2. Configuration Repository (infra-dev):

A developer wants to provision a new EKS cluster for a new project. They would browse the iac-templates repo, understand the eks-cluster module, then create a new directory and main.tf in their infra-dev repository:

# infra-dev/new-project/eks/main.tf
module "my_new_eks_cluster" {
  source = "git::ssh://[e-mail]/my-org/iac-templates.git//modules/eks-cluster?ref=v1.0.0"

  cluster_name     = "new-project-dev-eks"
  cluster_role_arn = "arn:aws:iam::123456789012:role/eks-cluster-role"
  subnet_ids       = ["subnet-0abcdef1234567890", "subnet-0fedcba9876543210"]
}

3. GitOps Workflow:

• The developer creates a PR against infra-dev with these changes.
• The PR undergoes code review (colleagues, platform team) and potentially automated checks (e.g., terraform validate, terraform plan output posted as a comment).
• Upon merging the PR, Argo CD (configured to watch infra-dev) detects the change.
• Argo CD then executes terraform apply with the specified module and variables, provisioning the EKS cluster.

Benefits

• Single Source of Truth: Git becomes the definitive desired state for all infrastructure.
• Auditability & Rollback: Every change is a commit, providing full history and easy rollback.
• Built-in Review Process: PRs naturally facilitate peer review and approval workflows.
• Security & Compliance: Policies and guardrails can be enforced via PR checks and immutable infrastructure deployments.
• Idempotency: GitOps operators continuously reconcile the actual state with the desired state.

Drawbacks

• Higher Initial Setup: Requires setting up GitOps operators (Argo CD, Flux CD) and structuring repositories.
• Steeper Learning Curve: Teams new to GitOps might find the mental model challenging initially.
• Indirect Feedback: Deployment feedback is often observed through the GitOps operator’s UI or logs, rather than direct CLI output.
• PR-centric Flow: For very rapid, iterative changes, the PR cycle might feel slower than direct CLI interaction.

Comparison of Solutions

Feature	Dedicated Template Catalog	CLI-Driven with Enhanced Discovery	GitOps with Self-Service via PRs
User Experience	Visual, intuitive forms, guided process. Low technical barrier.	Text-based, command-line focused. Requires CLI proficiency.	Git-centric (PRs, commits). Requires Git & IaC familiarity.
Discovery Mechanism	Curated web portal with search, categories, descriptions.	CLI commands (list, describe) pulling from structured docs/code.	Browse Git repositories for IaC modules/examples.
Deployment Trigger	“Submit” button in UI, triggers backend pipeline.	Explicit deploy command (often wrapped CI/CD).	Merge of a Pull Request in a configuration repository.
Governance & Controls	Built-in approval workflows, restricted template access.	Relies on CI/CD pipelines for validation & approval.	Mandatory PR review, automated checks, operator enforcement.
Setup Complexity	High (portal development/integration).	Medium (wrapper scripts, structured documentation).	High (GitOps operator, repo structure, CI/CD).
Maintenance Burden	Keeping portal UI/APIs in sync with IaC.	Keeping scripts & documentation updated.	Maintaining GitOps operators, ensuring state reconciliation.
Best For	Large organizations, non-technical users, high standardization needs.	Teams comfortable with CLI, scripting, rapid iteration.	Any organization committed to strong version control, auditability, and automated reconciliation.

Conclusion

Separating template browsing from deployment is a crucial step toward mature IaC adoption. Each solution offers distinct advantages, catering to different organizational structures, technical proficiencies, and governance requirements. Whether you opt for the guided experience of a self-service portal, the flexibility of an enhanced CLI, or the robust, auditable flow of GitOps, the key is to reduce the friction engineers face when trying to provision infrastructure. By making templates easily discoverable and deployments predictable, you empower your teams to innovate faster and more securely.

---

👉 Read the original article on TechResolve.blog