DEV Community: Michael Ethridge

Workstation Setup for Terraform

Michael Ethridge — Fri, 21 Aug 2026 18:15:36 +0000

Introduction

In my last post, I covered how I set up
VSCode for Terraform. The second most
popular question is how I set up my workstation for Terraform development.

As with most things in the tech world, there are many ways to do things. The
following is the way that works for me. I am always looking for ways to improve
my workflow, so if you have suggestions, please let me know.

Additionally, I am a Mac user, so some of the tools I use are harder to use on
Windows. However, the tools can be used in WSL or DevContainers if you use
either of those tools.

On a Mac, I use Homebrew to install my tools. Also, Since we
have to store all our code in a version control system, I installed the latest
version of git with Homebrew.

# Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install Git
brew install git

Installing Terraform

I have to use multiple versions of Terraform for different projects, so I use
TFSwitch to switch the version of Terraform
installed based on the directory I am in and the version constraint in my
Terraform configuration.

brew install warrensbox/tap/tfswitch

I have configured tfswitch to link the correct version of Terraform to
~/bin/terraform as the ~/bin directory is in my $PATH. This is done by
creating a ~/.tfswitch.toml file with the following contents:

bin="~/bin/terraform"

I use the following zsh function to load tfswitch in a directory with
Terraform configuration files.

load-tfswitch() {
  local tfswitchrc_path="${HOME}/.tfswitch.toml"

  # if [[ -f "$tfswitchrc_path" ]] && [[ -f "terraform.tf" ]]; then
  if [[ -f "$tfswitchrc_path" ]]; then
    if [[ $(ls -l *.tf | wc -l ) -gt 0 ]] 2> /dev/null; then
      tfswitch
    fi
  fi
}

add-zsh-hook chpwd load-tfswitch
load-tfswitch

Now, if you don't need to support multiple versions of Terraform, you can
install it with Homebrew from HashiCorp's tap.

  brew install hashicorp/tap/terraform

Otherwise, you can download it directly from the
Terraform Install page.

Git Tools

Storing your Terraform configuration in a version control system is a must. I
don't do development daily, so I forget to format my code and check for secrets
or other vulnerabilities. One of the worst is that I forgot to update the
README.md with any variable or output changes. The most annoying thing is that I
forgot to create a new branch for my changes, as most of the git repos I work in
have the default branch protected, so I can't push directly to it.

To help me with all of these problems, I use
Git Hooks,
specifically the pre-commit hook to check for all of these things before I
commit my changes. If you are unfamiliar with Git Hooks, they are scripts that
run automatically when specific actions occur in a Git repository. You can use
these scripts to enforce coding standards, check for vulnerabilities, or even
run tests before you commit your changes.

Traditionally, you would have to write these scripts yourself, but there is a
tool called pre-commit that makes it easy to install
and manage these hooks.

brew install pre-commit

Once you have pre-commit installed, you can create a .pre-commit-config.yaml
file in the root of your repository with the following contents:

# yaml-language-server: $schema=https://json.schemastore.org/pre-commit-config.json

repos:
  - repo: local
    hooks:
      - id: trufflehog
        name: TruffleHog
        description: Detect secrets in your data.
        entry: trufflehog git file://. --since-commit HEAD --fail
        language: golang
        pass_filenames: false
        stages: ["pre-commit", "pre-push"]
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0
    hooks:
      - id: check-merge-conflict
      - id: end-of-file-fixer
      - id: no-commit-to-branch
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.97.0
    hooks:
      - id: terraform_fmt
      - id: terraform_validate
        args:
          - --hook-config=--retry-once-with-cleanup=true
      - id: terraform_docs
        args:
          - --args=--lockfile=false
      - id: terraform_tflint
        args:
          - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
      - id: terraform_checkov
      - id: infracost_breakdown
        args:
          - --args=--path=.
          - --args=--terraform-var-file="terraform.tfvars"
        # verbose: true

Every time you commit your changes, the pre-commit tool runs and checks each
hook you have defined. If any hook fails, the commit will abort, and you must
fix the issues before committing your changes.

We haven't covered all of the hooks in the .pre-commit-config.yaml file, but
I'll list the ones I use here:

trufflehog: Scans your git repo for committed secrets 😱.
check-merge-conflict: Checks for files that contain merge conflict strings.
end-of-file-fixer: Ensures that files end with a newline.
no-commit-to-branch: Prevents commits directly to a branch (default branch in our case).
terraform_fmt: Formats your Terraform code.
terraform_validate: Validates your Terraform code.
terraform_docs: Dynamically updates your README.md with information on your module's inputs, outputs, and requirements.
terraform_tflint: A Terraform linter that checks for best practices and errors in your Terraform code.
terraform_checkov: A tool that checks your Terraform code for security vulnerabilities.
infracost_breakdown: Gives you a cost estimate for the cloud resources your module would deploy.

Tools needed for the Pre-commit hooks that I use

You will need to install a few of these tools to use all of the pre-commit
hooks that I have listed above.

Trufflehog scans your git repo
for secrets 😱. Doing this as a pre-commit hook lets you catch secrets before
committing them. This way, they don't end up in your git history, keeping your
security team happy.

brew install trufflesecurity/trufflehog/trufflehog

Terraform-docs dynamically
updates your README.md with information on your module's inputs, outputs, and
requirements.

brew install terraform-docs

In your README.md file, you can add the following comments to have
terraform-docs update the file.

<!-- BEGIN_TF_DOCS -->
<!-- END_TF_DOCS -->

Infracost provides a cost estimate for
the cloud resources on which your configuration will be deployed.

brew install infracost

Jq is a lightweight and flexible command-line
JSON processor. required for terraform_validate with
--retry-once-with-cleanup flag, and for infracost_breakdown hook.

brew install jq

TFLint is a Terraform linter that
checks for best practices and errors in your Terraform code.

brew install tflint

Checkov is a static code analysis
tool for infrastructure as code (IaC).

brew install checkov

Convenience Tools

With Terraform installed via tfswitch and the pre-commit hooks setup, I have
a solid foundation for my Terraform development workflow. But I use a few more
tools to make my life easier.

First, I use 1Password to store my secrets, API keys,
and passwords. I also use the
1Password CLI to access my
secrets via environment variables and shell scripts.

brew install 1password 1password-cli

With 1Password and the 1Password CLI installed, I can access my secrets via the
op command.

export TFE_TOKEN=$(op read --cache "op://Vault/Item/Key")

This allows me to easily set environment variables for my secrets in my shell.
However, I need to remember to unset these variables when I'm done with them. To
help with this, I use Direnv to set and unset variables
based on my directory.

brew install direnv

I use the following .envrc file in the root of my Terraform configuration to
set my secrets.

# Exports
export TFE_TOKEN=$(op read --cache "op://Vault/Item/Key")

# Terraform Variable exports
export TF_VAR_okta_token=$(op read --cache "op://Vault/Item/Section/Key")
export TF_VAR_okta_client_id=$(op read --cache "op://Vault/Item/Section/Key")
export TF_VAR_okta_client_secret=$(op read --cache "op://Vault/Item/Section/Key")

The above example exports the TFE_TOKEN and a few Terraform variables. When I
change into or out of the directory with the .envrc file, direnv will set
and unset these variables for me.

With so many tools in use, remembering all the commands to run can be a pain.
Yes, I could create aliases for them, but I like to keep my shell clean. So, I
use Task to create a Taskfile.yml with all the
commands I use.

# yaml-language-server: $schema=https://taskfile.dev/schema.json
# https://taskfile.dev

version: "3"

dotenv: [".envrc"]

vars:
  CURRENT_DATE:
    sh: date +"%Y-%m-%dT%H:%M:%S%Z"

tasks:
  default:
    cmds:
      - task: pre

  hog:
    cmds:
      - trufflehog git file://. --since-commit HEAD --only-verified --fail

  pre:
    cmds:
      - pre-commit autoupdate
      - pre-commit run -a

  push:
    cmds:
      - git add .
      - git commit -m "{{.CURRENT_DATE}}"
      - git push
    silent: true

  tag:
    cmds:
      - git push
      - git tag -s {{.CLI_ARGS}} -m "{{.CLI_ARGS}}"
      - git push --tags

Now I can run task hog to scan my git repo for secrets, task pre to run all
of my pre-commit hooks, task push to commit my changes and push them to the
remote, and task tag -- v1.0.0 to tag my release.

Conclusion

This is how I set up my workstation for Terraform development. I hope you found
it helpful. If you have any suggestions or tools that you use, please let me
know. I am always looking for ways to improve my workflow.

VS Code Setup for Terraform

Michael Ethridge — Fri, 21 Aug 2026 18:15:35 +0000

Introduction

One of the first questions I get when teaching Terraform is, "What editor should
I use?" My answer is always, "Use what you are comfortable with. If you don't
have a preference, I recommend Visual Studio Code." Over the years, I have used
everything from vi to Notepad++ to Sublime Text to Atom to
Visual Studio Code. Visual Studio Code is easy to pick up for new users and
easy to customize to your liking.

I am not a developer by trade; I'm more of a hack-it-together person. I copy a
lot of code from the internet or use GenAI and modify it to fit my needs. So, I
don't work in an IDE every day. I need something easy to use and easy to
customize.

TL;DR

For those of you like me who want the quick and dirty, here are the steps to set
up your VS Code for Terraform development:

Install Visual Studio Code - Download
Create a profile for Terraform - Profiles
Clone my Terraform template repository - Terraform Template
Open the repository in VS Code - This will prompt you to install the recommended extensions and import the settings.
Go forth and Terraform!

Visual Studio Code Profiles

One of the reasons I like Visual Studio Code is that it supports profiles. I can
create a profile for each type of work I do. I have profiles for Terraform,
Python, Ansible, and Markdown. Each profile has its settings, extensions, and
keybindings. By using profiles, I can tailor the editor to the work I am doing.

Creating a Profile

To create a profile, click on the gear icon in the lower-left corner of the VS
Code window. Click on the Profiles option in the' Settings' window.

The Profiles tab allows you to view your current profiles and create new ones.
To create a new profile, click the New Profile button.

This creates a new Untitled profile.

You can now name your profile by editing the name field. You also have the
option to copy the settings from an existing profile. This is useful for
creating a new profile similar to an existing one or importing your default
settings.

VS Code Extensions for Terraform

I will admit that I am a bit of an extension junkie. I have a lot of extensions,
but some don't play well together, so I use profiles to help limit the number of
extensions I have enabled at any one time.

I use four extensions for Terraform development in Visual Studio Code:

You're probably looking at the list and wondering, "Why do you need a case
preserver?" How many times have you decided to change the name of a variable and
then had to go back and change the case of the variable in multiple places? This
extension will preserve the case of the variable when you change it. It's a
small thing, but it saves me a lot of time.

VS Code Settings for Terraform

I like to use a few settings when working with Terraform. Most of these can be
set in the VS Code UI, but they can also be shared in a JSON file within the
.vscode. directory of your working directory.

Here are my Terraform specific VS Code settings:

{
  "[terraform]": {
    "editor.defaultFormatter": "hashicorp.terraform"
  },
  "[tfvars]": {
    "editor.defaultFormatter": "hashicorp.terraform"
  },
  "editor.bracketPairColorization.enabled": true,
  "editor.formatOnPaste": true,
  "editor.formatOnSave": true,
  "editor.formatOnType": true,
  "editor.guides.bracketPairs": "active",
  "editor.inlineSuggest.enabled": true,
  "editor.linkedEditing": true,
  "editor.multiCursorModifier": "alt",
  "editor.renderControlCharacters": true,
  "editor.renderWhitespace": "all",
  "editor.rulers": [
    {
      "color": "#A5FF90",
      "column": 80
    },
    {
      "color": "#FF628C",
      "column": 100
    }
  ],
  "editor.stickyScroll.enabled": true,
  "editor.suggestSelection": "first",
  "editor.tabCompletion": "on",
  "editor.tabSize": 2,
  "files.associations": {
    "*.sh.tmpl": "shellscript"
  },
  "files.trimTrailingWhitespace": true,
  "terraform.languageServer.enable": true
}

Not all of these settings are needed, but they are the ones I like to use. You
can adjust them to fit your needs.

editor.defaultFormatter - This sets the default formatter for the editor. I like to use the HashiCorp Terraform formatter for all of my Terraform, tfvars, and Sentinel policy files.
editor.bracketPairColorization.enabled - This will colorize the brackets in your code to make it easier to see where they start and end.
editor.formatOnPaste, editor.formatOnSave, editor.formatOnType - These settings will format your code when you paste it, save the file, or type in the editor.
editor.guides.bracketPairs - This will highlight the bracket pair you work within.
editor.inlineSuggest.enabled - This will enable inline suggestions in the editor.
editor.linkedEditing - This will link the editing of a variable name to all instances of that variable in the file.
editor.multiCursorModifier - This sets the keybinding for creating multiple cursors.
editor.renderControlCharacters - This will render control characters in the editor.
editor.renderWhitespace - This will render whitespace in the editor.
editor.rulers - This will add rulers to the editor in specified columns. I don't like to have lines that are too long, so I set my rulers at 80 and 100 columns.
editor.stickyScroll.enabled - This will enable sticky scrolling so that the block header stays at the editor's top as you scroll through the code section.
editor.suggestSelection - This sets the default suggestion selection to the first suggestion.
editor.tabCompletion - This will enable tab completion in the editor.
editor.tabSize - This sets the tab size to 2 spaces.
files.associations - This will associate the specified file extensions with the specified language. I also work with a lot of shell script templates that get processed by Terraform, so I associate the .sh.tmpl files with the shellscript language.
files.trimTrailingWhitespace - This will trim trailing whitespace from the end of lines.
terraform.languageServer.enable will enable the Terraform language server.

Conclusion

This helps you get started with Terraform development in Visual Studio.

Updates

2025-01-25

I removed settings for an older Terraform extension I used before the official
HashiCorp extension was released. Also, I removed file associations that used
the Terraform extension to format Sentinel policy files and generic HCL files. I
now use the HashiCorp Sentinel extension for Sentinel policy files and the
HashiCorp HCL extension for generic HCL files.

Running Vault on OpenShift with HCP Vault Auto-Unseal: Lessons Learned

Michael Ethridge — Fri, 21 Aug 2026 18:15:34 +0000

Introduction

I recently set up a three-node Vault Enterprise HA cluster on OpenShift, using
HCP Vault as the auto-unseal provider via the transit secrets engine. On paper
this is a straightforward combination of well-documented features. In practice,
it was a series of traps — some subtle, some spectacular — that took multiple
sessions to fully work through.

This post covers the four main challenge areas I hit: getting IPC_LOCK right
on OpenShift, wiring up the auto-unseal token flow securely, managing Raft
quorum safely during rolling updates, and working around a reconciliation bug in
Vault Secrets Operator. I'll focus on what caught me off guard and what the
correct solution looks like.

The deployment is GitOps-managed via ArgoCD using a three-source Helm pattern:
the upstream Vault chart, a values file from the repo, and raw manifests for
cluster-level resources (SCCs, Routes, ConfigMaps) that Helm can't cleanly
own.

Challenge 1: IPC_LOCK on OpenShift

The Problem

Vault requires the IPC_LOCK Linux capability so it can call mlockall() to
prevent secrets from being swapped to disk. This is non-negotiable — if the
capability is missing, Vault exits at startup. On OpenShift this runs into the
default restricted-v2 Security Context Constraint (SCC), which doesn't
include IPC_LOCK in its allowed capabilities.

I discovered this the hard way when all three pods went into CrashLoopBackOff
immediately after deploying. What made it more confusing was that the error
message differed depending on the image:

Upstream image (docker.io/hashicorp/vault-enterprise): Vault starts, calls mlockall(), gets EPERM, and exits with:

  Vault requires the IPC_LOCK capability

Red Hat partner image (registry.connect.redhat.com/hashicorp/vault-enterprise): The binary ships with cap_ipc_lock+ep set via setcap. When the kernel drops that from the bounding set at exec time (because the SCC doesn't allow it), the exec() itself fails before Vault even starts:

  exec: /usr/bin/vault: operation not permitted

Both root causes are the same — no IPC_LOCK in the SCC — but the symptoms
look completely different, which costs time when debugging.

The Solution: Custom SCC

The fix is a custom SCC that adds IPC_LOCK as both an allowed and default
capability:

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: vault-ipc-lock
  annotations:
    kubernetes.io/description: ">"
      Extends restricted-v2 to allow IPC_LOCK for Vault mlock support.
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities:
  - IPC_LOCK
defaultAddCapabilities:
  - IPC_LOCK
fsGroup:
  type: MustRunAs
readOnlyRootFilesystem: false
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
seccompProfiles:
  - runtime/default
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - ephemeral
  - persistentVolumeClaim
  - projected
  - secret

Custom SCCs don't automatically get a ClusterRole the way built-in ones do,
so you also need to create one manually and bind it to the Vault service account:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:openshift:scc:vault-ipc-lock
rules:
  - apiGroups: ["security.openshift.io"]
    resources: ["securitycontextconstraints"]
    resourceNames: ["vault-ipc-lock"]
    verbs: ["use"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vault-ipc-lock
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:vault-ipc-lock
subjects:
  - kind: ServiceAccount
    name: vault
    namespace: vault

The SCC Priority Gotcha

Here's the non-obvious part. When multiple SCCs have the same priority,
OpenShift's SCC resolver picks one based on a deterministic but non-obvious
ordering algorithm. In my cluster, a pre-existing odf-blackbox-scc had the
same default priority of 10. Without an explicit hint in the pod spec, the
init container landed on odf-blackbox-scc and failed — despite the
vault service account being bound to my new SCC.

The fix is to explicitly declare IPC_LOCK in the container's securityContext:

# In the Vault Helm values.yaml
server:
  extraContainers: []
  statefulSet:
    securityContext:
      container:
        capabilities:
          add:
            - IPC_LOCK

This hint steers the SCC resolver to select vault-ipc-lock over any
same-priority SCC that doesn't advertise IPC_LOCK support. The hint is
required even when defaultAddCapabilities is set — without it, the resolver
may never reach your SCC.

Image Choice

I also switched from the Red Hat partner image to the upstream
docker.io/hashicorp/vault-enterprise:2.0.1-ent. Without the docker.io/
prefix, RHCOS short-name rewriting rewrites unqualified image tags to
registry.connect.redhat.com/... at pull time. That's fine in theory, but in
practice the partner mirror had divergent cached bytes across nodes (different
sha256 digests for the same tag) and was using a floating 2.0-ent tag
rather than a pinned version. Once the SCC was in place, the upstream image
with an explicit registry and pinned tag worked cleanly.

Challenge 2: Auto-Unseal Token Flow

Why HCP Vault Dedicated?

Before getting into the mechanics, it's worth explaining the choice of HCP
Vault as the transit unseal provider rather than a self-hosted instance.

The transit auto-unseal mechanism doesn't require HCP Vault — any Vault cluster
(including Community Edition) can host a transit secrets engine and serve as the
unseal provider. The problem is the bootstrap paradox: if your transit unseal
cluster is also self-hosted, you need a way to unseal it before it can unseal
anything else. You've just moved the problem one level up.

HCP Vault Dedicated sidesteps this entirely. HashiCorp operates the cluster,
handles HA failover, applies upgrades, and manages the underlying infrastructure.
Critically, it uses cloud KMS for its own unsealing, so there's no chicken-and-egg
problem — it's always available when your OpenShift pods start up.

For a use case like this — a single transit key used only at pod startup — the
Development tier
is sufficient. It's a single-node cluster (no SLA, fine for a non-critical
dependency like this one), capped at 25 clients, and costs $0.03/hour —
roughly $21.60/month. That's a low price for eliminating the unseal
bootstrapping problem entirely and offloading upgrades and operations to
HashiCorp.

The Architecture

Vault's transit auto-unseal requires a token with encrypt/decrypt access to an
HCP Vault transit key. On every pod startup, Vault needs that token to be
available before it can unseal. The token can't live in a static Kubernetes
Secret (that would be a long-lived credential), so it needs to be minted fresh
at startup using the pod's service account JWT.

The flow is:

OpenShift projects each pod's service account token to /var/run/secrets/kubernetes.io/serviceaccount/token
An init container uses that JWT to authenticate to HCP Vault's JWT auth method (POST /v1/auth/jwt/login)
On successful auth, it mints a short-lived child token (24h TTL, no_parent so it's an orphan, autounseal policy)
It renders the transit seal stanza with that token to a shared emptyDir
The main Vault container reads that stanza via vault server -config=...

HCP Vault's JWT auth method is configured with:

bound_audiences: "https://kubernetes.default.svc" — matches the audience in the SA token
bound_subject: "system:serviceaccount:vault:vault" — locks it to exactly this service account
jwks_url pointing to OpenShift's OIDC JWKS endpoint
The cluster CA cert for TLS verification

This gives each pod a fresh, per-pod-bound token that's useless to any other
workload. It's about as tight as you can make a startup credential.

The Evolution: curl to Vault Agent

My first implementation used a curl+bash init container — about 50 lines of
shell inline in values.yaml. It worked but had several problems:

JSON parsing with jq and grep with fragile exit code handling
The rendered seal.hcl had the token in plaintext on disk (acceptable but not great)
Silent failure modes: if the token-create call failed, the shell would continue and render an empty seal.hcl
Hard to maintain and test

I replaced it with a declarative Vault Agent running in one-shot mode
(exit_after_auth = true). The agent config lives in a ConfigMap:

# vault-agent-config.yaml
vault {
  address = "https://vault.hashicorp.cloud:8200"

  tls {
    ca_cert = "/vault/userconfig/hcp-ca/ca.crt"
  }
}

auto_auth {
  method "jwt" {
    namespace = "admin/my-org"
    mount_path = "auth/jwt"
    config {
      path       = "/var/run/secrets/kubernetes.io/serviceaccount/token"
      role       = "autounseal-token"
    }
  }
  sink "file" {
    config {
      path = "/tmp/vault-agent-token"
    }
  }
}

template {
  contents = <<EOT
{{- with secret "auth/token/create" "policies=autounseal" "period=24h" "no_parent=true" "orphan=true" -}}
seal "transit" {
  address     = "https://vault.hashicorp.cloud:8200"
  namespace   = "admin/my-org"
  token       = "{{ .Auth.ClientToken }}"
  key_name    = "autounseal"
  mount_path  = "transit"
  tls_ca_cert = "/vault/userconfig/hcp-ca/ca.crt"
}
{{- end -}}
EOT
  destination = "/vault/userconfig/vault-config/seal.hcl"
}

The agent logs clearly report whether JWT auth succeeded and whether the
template rendered — a significant improvement over silent shell failures. The
exit_after_auth = true flag makes it act as a one-shot runner rather than a
persistent sidecar.

The `command:` Override Gotcha

One subtle issue: if you override command: in the init container spec, you
bypass the image's docker-entrypoint.sh. That entrypoint runs a
setcap -r /usr/bin/vault cap-clearing step before execing the binary —
important because the partner image ships with cap_ipc_lock+ep baked into the
binary's extended attributes.

The fix is to only set args: (e.g., ["agent", "-config=/path/to/agent.hcl"])
and leave command: unset. The entrypoint handles the rest.

Challenge 3: Raft Quorum During Rolling Updates

The OnDelete Trap

The Vault Helm chart sets updateStrategy: OnDelete on the StatefulSet by
default. This means ArgoCD applying a new chart revision stages the change in a
new controllerRevision but does not restart any pods. You have to manually
delete each pod to trigger the rollout.

The danger is that multiple staged-but-unrolled revisions can accumulate. If
you pushed three separate changes without rolling, deleting a pod applies all
three at once. Always check oc get statefulset vault -o json | jq '.status | {currentRevision, updateRevision}' before starting a rollout.

The Voter vs. Follower Distinction

When a Vault pod restarts and rejoins the Raft cluster, it goes through a
stabilization window (default 10 seconds) before Raft's autopilot promotes it
to a full voter. During this window, the pod is a non-voter — it's replicating
and appears healthy, but it doesn't count toward quorum.

I learned this the hard way. A 3-node cluster needs 2 voters for quorum. I
bounced pod 0, saw it come up 1/1 Ready with
replication.standby: stream started, assumed it was healthy, and immediately
deleted pod 1. Pod 1 went down before pod 0 achieved voter status, dropping the
active voter count to 1 — below quorum — causing a brief write outage until
things stabilized.

1/1 Ready does not mean voter. Always verify:

# Check who the current leader is (unauthenticated)
oc -n vault exec vault-0 -- \
  wget -qO- http://127.0.0.1:8200/v1/sys/leader | jq .

# Verify the just-rolled pod is a voter before touching the next
VAULT_TOKEN=<root-token> vault operator raft list-peers

Look for Voter: true on the pod you just rolled before deleting the next one.

The Correct Rollout Order

Identify the current leader at rollout time (don't assume it's pod 0 — it could have changed since the last restart)
Delete a follower and wait for 1/1 Ready and Voter: true in raft list-peers
Repeat for the remaining follower
Delete the leader last — it triggers re-election to one of the now-healthy standbys before the leader pod restarts

This sequence ensures quorum is never threatened.

Challenge 4: VSO Reconciliation Bugs

I'm using Vault Secrets Operator (VSO) 1.4.0 to sync Vault secrets into
Kubernetes Secrets. It hit two distinct bugs during this deployment.

Bug 1: Static Database Credentials Not Refreshing

Vault was rotating the Terraform Enterprise database password weekly via the
database secrets engine's static role. VSO was supposed to detect the rotation
and update the Kubernetes Secret — but it wasn't. TFE pods started failing with
SQLSTATE 28P01 (authentication failure) after each rotation.

VSO's logs showed the reason:

Vault secret does not support periodic renewal/refresh via reconciliation
horizon: 0

VSO 1.4.0 doesn't treat static-creds responses as renewable by default. The
fix is one field on the VaultDynamicSecret:

spec:
  allowStaticCreds: true
  refreshAfter: 24h

allowStaticCreds: true tells VSO to use the static-creds rotation tracker and
honor the rotation_period configured in Vault. refreshAfter is a safety net
in case the rotation tracker misses an event.

Bug 2: VSO Stops Reconciling After a Transient Vault HA Event

After Vault HA events (leader re-election, pod restarts), VSO occasionally
emits 500 errors like:

local node not active but active cluster node not found

This is transient and expected — it resolves once the new leader is fully
established. The problem is that VSO 1.4.0 records the failure and then stops
requeuing the affected VaultStaticSecret. Days later, the secret is stale and
VSO's controller shows zero log entries for it since the failure event.

Restarting the VSO controller pod alone is not sufficient — it re-initializes
but doesn't automatically re-pick stuck resources.

The fix is to touch the VSS spec to force a requeue:

# Temporarily change refreshAfter to trigger the watcher
oc -n <namespace> patch vaultstaticsecret <name> \
  --type=merge -p '{"spec":{"refreshAfter":"6m"}}'

# Verify reconciliation fires, then revert to original value
oc -n <namespace> patch vaultstaticsecret <name> \
  --type=merge -p '{"spec":{"refreshAfter":"5m"}}'

Note: the vso.secrets.hashicorp.com/refresh annotation documented for
triggering immediate reconciliation works for VaultDynamicSecret but not
VaultStaticSecret. Only the spec-patch method reliably works for the latter.

Update (2026): After some follow-up questions about whether this was still a
problem, I dug in further — and the picture is more nuanced than "VSO gives up
too early."

It's a low-probability race, not a timeout. The stuck state only happens
when a reconcile is in flight during the few-second leader-election window and
lands on a node still returning the 500. With a multi-minute refreshAfter,
that overlap is rare — which is exactly why it's hard to reproduce on demand. I
only caught it originally because it happened opportunistically across weeks of
real rollouts.

Upgrading does not fix it. I assumed a later VSO release must have addressed
this, but the release after 1.4.0 is dependency/build-only with no reconciler
changes, and 1.4.0 already ships the VaultStaticSecret client-callback handler
(GH-867). The
code path that sticks is present in both.

I tried hard to reproduce it and couldn't. I stood up a throwaway
VaultStaticSecret with refreshAfter: 5s (so a reconcile is almost always in
flight) and forced six real leader elections back-to-back. It self-healed every
time — the 500 never even surfaced. Reliably reproducing it seems to need many
more election/reconcile overlaps, not a handful.

The open question: does GH-867's client-callback handler cover the "local node
not active" case during an election, or only clean reconnect events? What I
saw suggests it doesn't fully cover the election race. The spec-patch recovery
above still applies.

Conclusion

Running Vault on OpenShift is entirely doable and production-viable, but the
combination of OpenShift's SCC model, Vault's Raft consensus requirements, and
VSO's edge cases means there are more failure modes than the documentation
covers. The main takeaways:

IPC_LOCK needs an explicit SCC — and the securityContext.capabilities.add hint is required to steer SCC selection when multiple SCCs have equal priority
Qualify your image references — docker.io/hashicorp/vault-enterprise:2.0.1-ent is unambiguous; bare hashicorp/vault-enterprise is not
Vault Agent beats curl+bash — one-shot mode gives you declarative auth, structured failure logging, and no shell fragility
1/1 Ready is not Voter: true — wait for raft list-peers before rolling the next pod
VSO 1.4.0 has a stuck-VSS bug — know the spec-patch workaround before you need it in production

Up Next: Terraform Enterprise on OpenShift

The same lab cluster runs Terraform Enterprise in active-active mode alongside
Vault, and it has its own set of OpenShift-specific challenges worth a dedicated
post. The short preview:

PostgreSQL via CloudNativePG — TFE needs a PostgreSQL cluster with a specific schema owner and connection string format. Getting the VSO-managed credentials to rotate cleanly with TFE's connection pool is where the allowStaticCreds fix from Challenge 4 above actually first surfaced.
NooBaa S3 for object storage — TFE uses S3-compatible storage for run logs and state. OpenShift ships with NooBaa (via ODF) as an in-cluster S3 provider, which avoids a cloud dependency but requires matching TFE's expected bucket configuration and path-style addressing.
Redis for active-active coordination — TFE's active-active mode uses Redis as its coordination layer. Running Redis on OpenShift under restricted-v2 has its own SCC story.
The same VSO stuck-reconciliation bug — hits TFE's Redis and S3 secrets independently after any Vault HA event, so the spec-patch workaround becomes a standard part of the Vault rollout runbook.

If you're planning a full HashiCorp stack on OpenShift, the Vault work above is
the prerequisite — get Vault stable and auto-unsealing before bringing TFE up,
since TFE uses Vault as its secrets backend from day one.

Running Terraform Enterprise Active-Active on OpenShift: Lessons Learned

Michael Ethridge — Fri, 21 Aug 2026 18:15:33 +0000

Introduction

In my last post, I covered the four
challenges I hit running Vault Enterprise on OpenShift. That same lab cluster
also runs Terraform Enterprise (TFE) in active-active mode, using that Vault
cluster as its secrets backend. TFE brought its own set of OpenShift-specific
problems — some in TFE itself, some in the supporting PostgreSQL and object
storage layers, and one that turned out to be the same VSO bug from the Vault
post showing up again in a different place.

The deployment follows the same GitOps pattern as Vault: ArgoCD, a three-source
Helm setup (upstream TFE chart, a values file from the repo, and raw manifests
for VSO custom resources, the Route, and the ImageStream/BuildConfig that
supporting pieces need). TFE runs with 2 replicas against a CloudNativePG-based
PostgreSQL cluster, a Redis deployment for active-active coordination, and a
NooBaa S3 bucket (via OpenShift Data Foundation) for object storage. Every
secret TFE needs — license, encryption password, database credentials, Redis
password, S3 keys, registry pull secret, TLS certificate — comes from the local
Vault cluster through Vault Secrets Operator (VSO).

This post covers five things that caught me off guard: TFE's hard refusal to
run mixed versions during a rolling update, why one Postgres cluster ended up
with two different Vault-managed role designs, a missing trust anchor for the
in-cluster S3 endpoint, the VSO stuck-reconciliation bug recurring across two
more secrets, and what it takes to get TFE's job agent running under
OpenShift's restricted security model.

Challenge 1: Active-Active Means Lockstep Versions

The Problem

TFE's active-active mode replicates its Rails processes across multiple pods
that all have to agree on the running version. I found this out during what
should have been a routine chart and image bump. The new pod came up as
CrashLoopBackOff while the two existing pods stayed healthy:

startup check failed: check=upgrade ... TFE version (2.0.4) is not
inter-compatible with system version (2.0.2), to upgrade to this version
active TFE versions must be stopped

The root cause is the Helm chart's default RollingUpdate strategy with
maxSurge: 25% — it starts the new-version pod alongside the two
still-running old-version pods to keep capacity up during the rollout. TFE's
own startup check treats that mixed-version state as invalid and refuses to
come up, so the new pod crash-loops forever while the old pods sit there
looking healthy.

The Solution

The fix is to pin the Deployment to Recreate:

# values.yaml
strategy:
  type: Recreate

Recreate tears down all pods before bringing any of them back up on the new
image, so there's never a moment where two versions coexist. The trade-off is
real and worth stating plainly: every deploy that touches the pod template —
image bump or not — causes a brief full TFE outage while pods restart
(around 3 minutes, observed). For an active-active system, that's a step back
from zero-downtime rollouts, but it's the only option that matches TFE's
lockstep version requirement.

If you ever hit the crash-loop mid-upgrade (say, the strategy setting got
reverted by mistake), the recovery is a coordinated restart:

oc scale deployment terraform-enterprise -n tfe --replicas=0
oc scale deployment terraform-enterprise -n tfe --replicas=2

One subtlety if you're running ArgoCD with selfHeal: true: it will fight the
scale-to-0 and start restoring replicas almost immediately. That's fine here —
the new ReplicaSet on the new image brings both pods up together, which is
exactly what you want. Both pods reached 1/1 Running within a few minutes.

Challenge 2: One Postgres Cluster, Two Vault Role Designs

The Problem

Vault's database secrets engine has an obvious, idiomatic pattern: a dynamic
role that creates a brand-new Postgres user on every lease, with a short TTL.
That's what I set up first:

vault write tfe-psql/roles/tfe-psql-role \
  db_name=tfe \
  creation_statements="
    CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
    GRANT ALL PRIVILEGES ON DATABASE tfe TO \"{{name}}\";
  " \
  default_ttl="1h" \
  max_ttl="24h"

It works — Vault happily mints a fresh, uniquely-named Postgres role every
hour. The problem is that TFE doesn't want that. It expects to connect as one
stable database user across restarts, and its migrations need that user to
own the database and the public schema — not just have grants on them.
Dynamic per-lease roles with randomized names don't fit that model at all.

The Solution

The fix was to layer Vault's static role feature on top of the same
database connection, rather than replacing the dynamic role. A static role
manages the password for an existing, stable-named Postgres user on a fixed
rotation schedule instead of creating new ones:

# Allow both role types on the same connection
vault write tfe-psql/config/tfe-psql \
  allowed_roles="tfe-psql-role,tfe-static"

# Register the static role
vault write tfe-psql/static-roles/tfe-static \
  db_name=tfe-psql \
  username=tfe_app \
  rotation_period=604800

But the tfe_app user has to exist and own the right objects before Vault can
manage its password. That's a one-time manual bootstrap: create the role, then
hand over ownership as the Postgres superuser:

# Create the role using the cluster's admin user
psql "host=tfe-psql-primary.tfe.svc.cluster.local user=tfe dbname=tfe sslmode=require" \
  -c "CREATE ROLE tfe_app WITH LOGIN PASSWORD '${INIT_PW}';"

# As the postgres superuser, transfer ownership
psql -U postgres -d tfe -c "
  ALTER DATABASE tfe OWNER TO tfe_app;
  GRANT ALL ON SCHEMA public TO tfe_app;
  ALTER SCHEMA public OWNER TO tfe_app;
"

After that, Vault rotates tfe_app's password every 7 days without ever
changing the username, which is exactly what TFE needs. The dynamic role
stays registered too — it's a fine pattern for ad hoc admin access, just not
for the application's own connection.

Challenge 3: NooBaa S3 and a Missing Trust Anchor

The Problem

TFE uses S3-compatible object storage for run logs and state, and this
cluster provides that in-cluster via NooBaa (part of OpenShift Data
Foundation) rather than a cloud S3 bucket. Pointing TFE at NooBaa's internal
endpoint seemed straightforward — until TFE crashed on startup with:

x509: certificate signed by unknown authority

The NooBaa S3 endpoint's certificate is signed by OpenShift's own internal
service-CA, and TFE's container doesn't trust that CA by default. Nothing
about the bucket configuration was wrong; TFE just couldn't verify the TLS
certificate on the connection.

The Solution

Every OpenShift namespace gets an auto-populated ConfigMap
(openshift-service-ca.crt) containing exactly that CA certificate. The fix
is to mount it into the path TFE's chart already exposes for a custom CA
bundle:

# values.yaml
tls:
  caCertBaseDir: /etc/ssl/certs
  caCertFileName: custom_ca_certs.pem

extraVolumes:
  - name: openshift-service-ca
    configMap:
      name: openshift-service-ca.crt
      items:
        - key: service-ca.crt
          path: custom_ca_certs.pem

extraVolumeMounts:
  - name: openshift-service-ca
    mountPath: /etc/ssl/certs/custom_ca_certs.pem
    subPath: custom_ca_certs.pem
    readOnly: true

Once that ConfigMap is mounted and referenced by caCertBaseDir/
caCertFileName, TFE trusts the NooBaa endpoint's certificate and connects
cleanly.

Worth noting on the bucket side: the ObjectBucketClaim that provisions the
bucket also produces a Secret with the S3 access keys, but TFE never reads
that Secret directly. Those keys get copied into Vault, and VSO syncs them
back out into the Secret TFE actually consumes — keeping VSO/Vault as the
single source of truth for every credential, not just the ones that
originate there.

Challenge 4: The VSO Stuck-Reconciliation Bug, Again

If you read the Vault post, you'll remember VSO 1.4.0's bug where a
VaultStaticSecret can stop reconciling after a transient Vault HA error —
and doesn't recover on its own even if you restart the VSO controller. That
bug wasn't a one-off. It hit twice more here, on two of TFE's secrets, each
time triggered by an unrelated Vault rollout causing a brief Raft leader
election:

local node not active but active cluster node not found

Once on the VaultStaticSecret backing TFE's S3 credentials, and once on the
one backing its Redis password — both during Vault maintenance that had
nothing to do with TFE. Same symptom both times: SecretSynced=False, a
lastTransitionTime stuck days in the past, and zero new controller log
lines for the resource.

The fix is the same spec-patch workaround from the Vault post — touch the
resource to force VSO to requeue it, then revert:

oc -n tfe patch vaultstaticsecret tfe-s3 \
  --type=merge -p '{"spec":{"refreshAfter":"6m"}}'

# confirm SecretSynced=True and a fresh lastTransitionTime, then revert
oc -n tfe patch vaultstaticsecret tfe-s3 \
  --type=merge -p '{"spec":{"refreshAfter":"5m"}}'

There's a related but distinct issue on TFE's database credentials, which
come from a VaultDynamicSecret rather than a VaultStaticSecret. VSO 1.4.0
doesn't treat a static-role database response as renewable unless told
otherwise, so without one extra field it syncs once and never refetches —
meaning the next scheduled password rotation leaves TFE with a stale
password and failing Postgres logins:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: tfe-database
  namespace: tfe
spec:
  vaultAuthRef: tfe-auth
  mount: tfe-psql
  path: static-creds/tfe-static
  allowStaticCreds: true
  refreshAfter: 24h
  destination:
    name: tfe-database
    create: true
  rolloutRestartTargets:
    - kind: Deployment
      name: terraform-enterprise

Two different VSO resource types, two different failure modes, two different
fixes — but both are now a standard part of the runbook rather than a
surprise, since between the Vault deployment and this one, the same bug
family has now hit three separate secrets.

Update (2026): Digging in later reframed this. The stuck-reconciliation
part is a low-probability race — it needs a VSO reconcile in flight during
the brief leader-election window — not a frequent recurrence, and it isn't
fixed by upgrading VSO (the release after 1.4.0 is dependency/build-only, and
1.4.0 already ships the relevant VaultStaticSecret client-callback handler).
I couldn't reproduce it in six back-to-back forced elections. Full write-up in
the Vault post's Bug 2 update.
The spec-patch recovery above still applies.

Challenge 5: A Custom Job-Agent Image for OpenShift

The Problem

TFE runs each Terraform run in a short-lived "agent" container. The
stock hashicorp/tfc-agent image fails admission under OpenShift's
restricted security model — it expects to write to a working directory owned
by a fixed UID, which doesn't hold up once OpenShift assigns the container an
arbitrary UID from the namespace's allowed range.

The Solution

HashiCorp's own OpenShift deployment guide covers this: build a thin wrapper
image that opens up the permissions the arbitrary UID needs. I wired it up as
an OpenShift BuildConfig that layers a couple of chmod calls onto the
upstream image:

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: tfc-agent
  namespace: tfe-agents
spec:
  output:
    to:
      kind: ImageStreamTag
      name: tfc-agent:latest
  source:
    type: Dockerfile
    dockerfile: |
      FROM docker.io/hashicorp/tfc-agent:latest
      USER root
      RUN mkdir /.tfc-agent && \
          chmod og+rw /.tfc-agent && \
          chmod o+rx /home/tfc-agent
      USER tfc-agent
  strategy:
    type: Docker
    dockerStrategy:
      from:
        kind: DockerImage
        name: docker.io/hashicorp/tfc-agent:latest

TFE points at the built image via TFE_RUN_PIPELINE_IMAGE, referencing the
in-cluster image registry rather than Docker Hub directly. Re-running
oc start-build tfc-agent picks up new upstream releases of tfc-agent
without needing to restart TFE itself — the new image is used the next time
an agent pod spawns.

A Few Smaller Landmines

Two more things that didn't warrant a full section but are worth knowing
before you deploy TFE on OpenShift yourself:

The chart hard-codes its own resource names. The Deployment and Service are both named the literal string terraform-enterprise, regardless of your Helm release name — nameOverride/fullnameOverride exist in the chart's helpers but the main templates never use them. Anything that needs to reference TFE by name (VSO's rolloutRestartTargets, a Route's to.name) has to use terraform-enterprise, not your release name. The Service's port is also named https-port, not the more conventional https — a Route with the wrong targetPort gives a silent 503 rather than an obvious error.
The chart defaults to a LoadBalancer Service. On a bare-metal cluster with no LoadBalancer provider, that sits at EXTERNAL-IP: <pending> forever and keeps ArgoCD's health check stuck on Progressing. Overriding to service.type: ClusterIP and fronting it with an OpenShift Route clears this up immediately.

Conclusion

Running TFE active-active on OpenShift is straightforward once you've hit
each of these once, but the documentation doesn't cover any of them:

Active-active means lockstep versions — RollingUpdate will crash-loop a mixed-version rollout every time; strategy: Recreate is the only correct answer, at the cost of a brief outage on every deploy
Dynamic Vault roles don't fit every app — TFE's stable-username, owns-the-schema requirement needed a static role layered onto the same database connection, with a one-time manual ownership handoff
In-cluster S3 needs in-cluster trust — mount OpenShift's own openshift-service-ca.crt ConfigMap if you're pointing TFE at NooBaa instead of a cloud S3 endpoint
The VSO stuck-reconciliation bug isn't Vault-specific — it hit TFE's S3 and Redis secrets independently, both times triggered by unrelated Vault HA events; the spec-patch workaround is now a standing runbook step
OpenShift's restricted SCC reaches into the job agent, too — not just the TFE pods themselves; budget time for the custom tfc-agent image build

Between this post and the Vault one, that's the full HashiCorp-on-OpenShift
stack this lab runs — Vault as the secrets backend, VSO as the sync layer,
and TFE consuming both. Every failure mode above came from the same root
cause in different clothes: OpenShift's stricter defaults (SCCs, arbitrary
UIDs, in-cluster CAs) and TFE's own strict operational assumptions (lockstep
versions, stable database identity) both push back against defaults that
work fine on vanilla Kubernetes.

How I Structure a Terraform Module

Michael Ethridge — Fri, 21 Aug 2026 18:15:32 +0000

Introduction

Open a Terraform repository you have never seen before. There is a good chance
it looks like this: one main.tf with eight hundred lines in it, a
variables.tf where required and optional inputs are shuffled together in
alphabetical order, a provider block with no version constraint at all, and a
README.md that documented the inputs accurately about four months ago.

None of that is anybody's fault. Terraform doesn't care how you arrange your
files. Every file ending in .tf in a directory gets concatenated before
anything is evaluated, so the language gives you no reason to prefer one layout
over another. The tutorials all put everything in main.tf because they're
teaching one resource at a time, and then that becomes the shape of the repo
forever.

I'm not a developer by trade. I don't have the muscle memory that lets someone
navigate an unfamiliar codebase by feel. What I have instead is a layout I use
every single time, so that six months from now, when I come back to a module I
half remember, I know where things are before I open anything.

This post is that layout, and the reasoning behind each piece of it. Every
convention here is a response to a specific way I have watched a module go bad.
It's all based on the
HashiCorp style guide,
with opinions layered on where the style guide leaves room.

Everything below is scaffolded in
terraform-module-template,
which I use for modules and root modules alike.

TL;DR

data.tf, locals.tf, providers.tf, terraform.tf, backend.tf, variables.tf, and outputs.tf always exist, even when empty.
Resources live in main.tf until there are roughly twenty of them, then split by service or function.
variables.tf is two sections, required before optional, alphabetized inside each. outputs.tf is one alphabetized list.
terraform.tf holds required_version and nothing else. required_providers belongs in providers.tf.
Pin everything pinnable with ~> Maj.Min.
Generate the README with terraform-docs, do not write it.
Run fmt, validate, docs, tflint, and trivy in pre-commit, so you find problems before a plan does.

Problem 1: Everything Ends Up in main.tf

A module starts with three resources, so of course they go in main.tf. Then it
grows. Nobody ever decides to put eight hundred lines in one file. It happens
because the alternative requires a decision, and there's never a good moment to
make it.

The Fix: Files by Role First, Size Second

Some file names are reserved for a role, and those files always exist:

backend.tf      state backend configuration
data.tf         data sources
locals.tf       local values
main.tf         resources
outputs.tf      module outputs
providers.tf    required_providers, plus provider configuration
terraform.tf    required_version
variables.tf    input variables

Resources stay in main.tf until there are around twenty of them. Past that,
main.tf splits by service or function: network.tf, iam.tf, database.tf.
The number isn't sacred. The point is that the split is triggered by size, and
only resources ever get split. Data sources, locals, and provider configuration
stay in their own files no matter how few or how many there are.

That asymmetry is deliberate. Resource count varies enormously between modules,
so where resources live has to scale. The other categories are small and
bounded, and their value comes from being findable at a fixed address rather
than from being organized well.

Why Empty Files Stay

In a fresh module, data.tf and locals.tf are empty except for a header
comment:

###############################################################################
# data.tf
#
# Contains any data sources
###############################################################################

This is the convention people push back on most, and I'll keep defending it. An
absent file carries no information. You can't tell whether this module has no
data sources or whether someone put them in main.tf without looking. A
present, empty file answers both questions at once: there are no data sources
yet, and here is where the next one goes.

Most structural decay in a repo doesn't come from someone disagreeing with the
layout. It comes from someone adding a data source at 5pm, not seeing an obvious
home for it, and dropping it at the bottom of main.tf. An empty data.tf
removes the decision, which is the only reliable way to make a convention
survive contact with a deadline.

Problem 2: You Can't Tell Required From Optional

Alphabetical ordering in variables.tf sounds obviously correct, and it's the
default advice. It also means that the first thing you want to know about a
module - what do I have to give it? - is the one thing the file won't tell you.
You have to read every block and check each one for a default.

The Fix: Two Sections, Alphabetized Inside Each

variables.tf is split into required variables, meaning no default, followed
by optional variables, meaning there is one. Each section is alphabetized
independently:

###############################################################################
# Required Variables (no default values)
###############################################################################
variable "image_id" {
  type        = string
  description = "The id of the machine image (AMI) to use for the server."

  validation {
    condition     = length(var.image_id) > 4 && substr(var.image_id, 0, 4) == "ami-"
    error_message = "The image_id value must be a valid AMI id, starting with \"ami-\"."
  }
}

###############################################################################
# Optional Variables (has a default value)
###############################################################################
variable "instance_type" {
  type        = string
  description = "Instance Type"
  default     = "t3.micro"
}

Now the top of the file is the module's contract. Everything above the second
banner is what a caller must supply. Everything below it is a knob they may
choose to turn. Sorting alphabetically inside each section keeps lookup fast
without giving up that distinction.

outputs.tf gets plain alphabetical ordering, because outputs have no
equivalent of the required/optional split. There is only one axis to sort on, so
sort on it.

Make the Generated Docs Agree

A convention that only lives in the source file will drift the moment something
else renders the same information in a different order. terraform-docs
produces the inputs table in the README, and by default it sorts by name, which
would immediately contradict variables.tf.

So .terraform-docs.yml is configured to sort the same way:

sort:
  enabled: true
  by: required

settings:
  anchor: false

Required inputs come first in the generated table, exactly as they do in the
file that generated it. It's a small thing, but it is the difference between a
convention that is asserted once in a style guide and one that is enforced in
two places and therefore holds.

Problem 3: One File for Every Version Constraint

The common layout is a single versions.tf holding both required_version and
required_providers, or a terraform block sitting at the top of main.tf
with everything in it. Both work. Both bury two unrelated decisions in the same
place.

The Fix: Three Files, Three Concerns

The version of Terraform itself, alone in its own file:

# terraform.tf
terraform {
  required_version = "~> 1.0"
}

The providers, their versions, and their configuration, together in another:

# providers.tf
terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.28"
    }
  }
}

provider "google" {
  # Configuration options
}

And where state lives, in a third:

# backend.tf
terraform {
  cloud {
    organization = "example_corp"
    hostname     = "app.terraform.io"

    workspaces {
      tags = ["app"]
    }
  }
}

Yes, that's three separate terraform blocks across three files. Terraform
merges them, so this costs nothing at evaluation time and buys a clean
separation:

terraform.tf is about the tool. It changes when I decide to adopt a new Terraform version, which is a deliberate, infrequent, repo-wide decision.
providers.tf is about dependencies. It changes when a provider ships something I need, which happens on somebody else's schedule and much more often. It also holds provider configuration, which is what I'm editing most of the time.
backend.tf is about where state lives. It's the one file most likely to differ between environments or to be supplied at init time, and the one most likely to be the reason a terraform init behaves differently on someone else's machine.

Keeping them apart means a diff tells you which kind of change you are looking
at before you read a single line of it.

Problem 4: Pins That Are Too Loose or Too Tight

There are two failure modes here and they look nothing alike.

Too loose, usually meaning no constraint at all, gives you a plan that worked
yesterday and doesn't work today, because a provider released a major version
overnight and terraform init on a clean checkout happily took it.

Too tight, meaning an exact pin like version = "6.28.0", gives you a module
that never picks up a bug fix unless someone edits it, and a fleet of repos
pinned to a scatter of slightly different patch releases.

The Fix: Pessimistic Constraints on Everything Pinnable

Everything that can be pinned gets pinned with the pessimistic operator at the
major-minor level: Terraform core, every provider, every module call.

version = "~> 6.28"    # >= 6.28.0, < 7.0.0

Minor and patch releases flow in. A major version, which is where the breaking
changes live by convention, does not. That's the tradeoff I want almost
everywhere: I get fixes without being asked, and I get to schedule the upgrades
that cost me something.

One edge case. For a provider still on 0.x, ~> 0.104 expands to
>= 0.104.0, < 1.0.0, and pre-1.0 providers routinely make breaking changes on
a minor bump. If a dependency matters and is still pre-1.0, tighten it to
~> 0.104.0 and accept the manual bumps.

Commit the Lock File

~> 6.28 is a statement about what you will tolerate. .terraform.lock.hcl is
a record of what you actually got, and it belongs in version control.

The lock file holds on to a provider entry as long as state still references
that provider, even after you've deleted every last resource for it from your
configuration. The instinct is to open the lock file and delete the stanza.
Don't. Apply first so the resources leave state, and the next terraform init
will prune the entry on its own. Hand-editing a lock file to fix a problem that
an apply would have fixed tends to produce a second, stranger problem.

Problem 5: The README Drifts

Every Terraform README starts with an accurate inputs table. Then someone adds a
variable in a hurry. The code is right and the documentation is wrong, and now
the README is worse than no README, because people trust it.

The Fix: Generate It

terraform-docs reads the actual variable and output blocks and writes the
tables into the README between marker comments. It runs as a pre-commit hook, so
it is not something anyone has to remember:

- id: terraform_docs
  args:
    - --args=--lockfile=false

The one thing that catches everybody the first time: the hook rewrites
README.md during the commit, which means the commit fails, because the file it
just changed was not part of what you staged. This is correct behavior and it
looks like a broken hook. Re-stage and commit again and it passes. It only
happens when the generated content actually changed, so after the first time it
mostly disappears.

Problem 6: You Find the Error at Apply Time

The expensive version of every mistake in this post is the one you discover
after terraform apply has already created four of seven resources.

The Fix: A Pre-Commit Chain, in Order

- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.108.1
  hooks:
    - id: terraform_fmt
    - id: terraform_validate
      args:
        - --hook-config=--retry-once-with-cleanup=true
    - id: terraform_docs
      args:
        - --args=--lockfile=false
    - id: terraform_tflint
      args:
        - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
    - id: terraform_trivy

The order is cheapest and most mechanical first, so that later hooks are reading
code that is already well formed:

terraform_fmt normalizes whitespace and alignment. Running it first means no other hook, and no future diff, is ever about formatting.
terraform_validate catches syntax and internal consistency errors. --retry-once-with-cleanup=true handles the common case where validation fails only because of a stale .terraform directory: it clears it and tries once more instead of reporting a problem you don't have.
terraform_docs regenerates the README, now that the code it is documenting is known to be valid.
terraform_tflint catches the things validate structurally cannot - deprecated syntax, unused declarations, provider-specific mistakes such as an instance type that doesn't exist.
terraform_trivy scans for security misconfigurations. It goes last because it is the slowest and the most likely to need a human judgment call about whether a finding applies.

Alongside those, the standard hygiene hooks run on everything, and one of them
earns a special mention:

- id: no-commit-to-branch

That blocks commits directly to main. It's saved me from myself more than any
of the Terraform-specific hooks.

The Directories You Don't Need Yet

The template ships five directories that are empty apart from a README.md:
examples/, modules/, tests/, files/, and templates/.

The placeholder README exists because git won't track an empty directory, so
without a file in it the directory would not survive a clone. But the reason to
scaffold them at all is the same reason the empty data.tf earns its place.
When it's time to write the first test, there is no decision to make about where
tests go. When someone needs a usage example, examples/ is already there with
a note explaining what belongs in it.

Scaffolding an empty directory costs nothing today. Adding one to a repo that
has already grown around its absence means moving files and updating every path
that pointed at them.

Conclusion

None of this is clever, and that's the point. It's a set of decisions made once
so that they do not have to be made again while tired:

Reserved file names, always present - data.tf, locals.tf, providers.tf, terraform.tf, backend.tf, variables.tf, outputs.tf. Empty is fine, and an empty file still tells you something.
Only resources scale - main.tf splits by service past roughly twenty resources. Nothing else splits, ever.
Required before optional in variables.tf, alphabetized within each section, with terraform-docs configured to sort the same way so the generated table cannot contradict the source.
Three files for three kinds of version decision - the tool, its dependencies, and where state lives.
~> Maj.Min on everything pinnable, with the pre-1.0 caveat, and the lock file committed as the record of what you actually resolved.
Generate the README, never hand-maintain it.
fmt, validate, docs, tflint, trivy in that order in pre-commit, because the cheapest place to find any of these problems is before the commit, and the most expensive is halfway through an apply.

The whole thing is in
terraform-module-template
if you would rather clone it than assemble it.

If you disagree with a specific convention here, that's fine and probably
healthy. What matters far more than which layout you pick is that you pick one
and it's the same in every repo. Decide once, and you stop spending attention on
the question.

Up Next: Terraform in Practice

This kicks off a series on the practices that sit around the Terraform code
rather than in it. Next up: getting static credentials out of your
infrastructure entirely - a Vault SSH certificate authority instead of
distributing key pairs, OIDC instead of long-lived AWS access keys, and dynamic
database credentials instead of a shared application user. Three substitutions,
and the gotcha that bites on each one.

DEV Community: Michael Ethridge

Workstation Setup for Terraform

Introduction

Installing Terraform

Git Tools

Tools needed for the Pre-commit hooks that I use

Convenience Tools

Conclusion

VS Code Setup for Terraform

Introduction

TL;DR

Visual Studio Code Profiles

Creating a Profile

VS Code Extensions for Terraform

VS Code Settings for Terraform

Conclusion

Updates

2025-01-25

Running Vault on OpenShift with HCP Vault Auto-Unseal: Lessons Learned

Introduction

Challenge 1: IPC_LOCK on OpenShift

The Problem

The Solution: Custom SCC

The SCC Priority Gotcha

Image Choice

Challenge 2: Auto-Unseal Token Flow

Why HCP Vault Dedicated?

The Architecture

The Evolution: curl to Vault Agent

The command: Override Gotcha

Challenge 3: Raft Quorum During Rolling Updates

The OnDelete Trap

The Voter vs. Follower Distinction

The Correct Rollout Order

Challenge 4: VSO Reconciliation Bugs

Bug 1: Static Database Credentials Not Refreshing

Bug 2: VSO Stops Reconciling After a Transient Vault HA Event

Conclusion

Up Next: Terraform Enterprise on OpenShift

Running Terraform Enterprise Active-Active on OpenShift: Lessons Learned

Introduction

Challenge 1: Active-Active Means Lockstep Versions

The Problem

The Solution

Challenge 2: One Postgres Cluster, Two Vault Role Designs

The Problem

The Solution

Challenge 3: NooBaa S3 and a Missing Trust Anchor

The Problem

The Solution

Challenge 4: The VSO Stuck-Reconciliation Bug, Again

Challenge 5: A Custom Job-Agent Image for OpenShift

The Problem

The Solution

A Few Smaller Landmines

Conclusion

How I Structure a Terraform Module

Introduction

TL;DR

Problem 1: Everything Ends Up in main.tf

The Fix: Files by Role First, Size Second

Why Empty Files Stay

Problem 2: You Can't Tell Required From Optional

The Fix: Two Sections, Alphabetized Inside Each

Make the Generated Docs Agree

Problem 3: One File for Every Version Constraint

The Fix: Three Files, Three Concerns

Problem 4: Pins That Are Too Loose or Too Tight

The Fix: Pessimistic Constraints on Everything Pinnable

Commit the Lock File

Problem 5: The README Drifts

The Fix: Generate It

Problem 6: You Find the Error at Apply Time

The Fix: A Pre-Commit Chain, in Order

The Directories You Don't Need Yet

Conclusion

Up Next: Terraform in Practice

The `command:` Override Gotcha