Running Terraform Enterprise Active-Active on OpenShift: Lessons Learned

Introduction

In my last post, I covered the four
challenges I hit running Vault Enterprise on OpenShift. That same lab cluster
also runs Terraform Enterprise (TFE) in active-active mode, using that Vault
cluster as its secrets backend. TFE brought its own set of OpenShift-specific
problems — some in TFE itself, some in the supporting PostgreSQL and object
storage layers, and one that turned out to be the same VSO bug from the Vault
post showing up again in a different place.

The deployment follows the same GitOps pattern as Vault: ArgoCD, a three-source
Helm setup (upstream TFE chart, a values file from the repo, and raw manifests
for VSO custom resources, the Route, and the ImageStream/BuildConfig that
supporting pieces need). TFE runs with 2 replicas against a CloudNativePG-based
PostgreSQL cluster, a Redis deployment for active-active coordination, and a
NooBaa S3 bucket (via OpenShift Data Foundation) for object storage. Every
secret TFE needs — license, encryption password, database credentials, Redis
password, S3 keys, registry pull secret, TLS certificate — comes from the local
Vault cluster through Vault Secrets Operator (VSO).

This post covers five things that caught me off guard: TFE's hard refusal to
run mixed versions during a rolling update, why one Postgres cluster ended up
with two different Vault-managed role designs, a missing trust anchor for the
in-cluster S3 endpoint, the VSO stuck-reconciliation bug recurring across two
more secrets, and what it takes to get TFE's job agent running under
OpenShift's restricted security model.

---

Challenge 1: Active-Active Means Lockstep Versions

The Problem

TFE's active-active mode replicates its Rails processes across multiple pods
that all have to agree on the running version. I found this out during what
should have been a routine chart and image bump. The new pod came up as
CrashLoopBackOff while the two existing pods stayed healthy:

startup check failed: check=upgrade ... TFE version (2.0.4) is not
inter-compatible with system version (2.0.2), to upgrade to this version
active TFE versions must be stopped

The root cause is the Helm chart's default RollingUpdate strategy with
maxSurge: 25% — it starts the new-version pod alongside the two
still-running old-version pods to keep capacity up during the rollout. TFE's
own startup check treats that mixed-version state as invalid and refuses to
come up, so the new pod crash-loops forever while the old pods sit there
looking healthy.

The Solution

The fix is to pin the Deployment to Recreate:

# values.yaml
strategy:
  type: Recreate

Recreate tears down all pods before bringing any of them back up on the new
image, so there's never a moment where two versions coexist. The trade-off is
real and worth stating plainly: every deploy that touches the pod template —
image bump or not — causes a brief full TFE outage while pods restart
(around 3 minutes, observed). For an active-active system, that's a step back
from zero-downtime rollouts, but it's the only option that matches TFE's
lockstep version requirement.

If you ever hit the crash-loop mid-upgrade (say, the strategy setting got
reverted by mistake), the recovery is a coordinated restart:

oc scale deployment terraform-enterprise -n tfe --replicas=0
oc scale deployment terraform-enterprise -n tfe --replicas=2

One subtlety if you're running ArgoCD with selfHeal: true: it will fight the
scale-to-0 and start restoring replicas almost immediately. That's fine here —
the new ReplicaSet on the new image brings both pods up together, which is
exactly what you want. Both pods reached 1/1 Running within a few minutes.

---

Challenge 2: One Postgres Cluster, Two Vault Role Designs

The Problem

Vault's database secrets engine has an obvious, idiomatic pattern: a dynamic
role that creates a brand-new Postgres user on every lease, with a short TTL.
That's what I set up first:

vault write tfe-psql/roles/tfe-psql-role \
  db_name=tfe \
  creation_statements="
    CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
    GRANT ALL PRIVILEGES ON DATABASE tfe TO \"{{name}}\";
  " \
  default_ttl="1h" \
  max_ttl="24h"

It works — Vault happily mints a fresh, uniquely-named Postgres role every
hour. The problem is that TFE doesn't want that. It expects to connect as one
stable database user across restarts, and its migrations need that user to
own the database and the public schema — not just have grants on them.
Dynamic per-lease roles with randomized names don't fit that model at all.

The Solution

The fix was to layer Vault's static role feature on top of the same
database connection, rather than replacing the dynamic role. A static role
manages the password for an existing, stable-named Postgres user on a fixed
rotation schedule instead of creating new ones:

# Allow both role types on the same connection
vault write tfe-psql/config/tfe-psql \
  allowed_roles="tfe-psql-role,tfe-static"

# Register the static role
vault write tfe-psql/static-roles/tfe-static \
  db_name=tfe-psql \
  username=tfe_app \
  rotation_period=604800

But the tfe_app user has to exist and own the right objects before Vault can
manage its password. That's a one-time manual bootstrap: create the role, then
hand over ownership as the Postgres superuser:

# Create the role using the cluster's admin user
psql "host=tfe-psql-primary.tfe.svc.cluster.local user=tfe dbname=tfe sslmode=require" \
  -c "CREATE ROLE tfe_app WITH LOGIN PASSWORD '${INIT_PW}';"

# As the postgres superuser, transfer ownership
psql -U postgres -d tfe -c "
  ALTER DATABASE tfe OWNER TO tfe_app;
  GRANT ALL ON SCHEMA public TO tfe_app;
  ALTER SCHEMA public OWNER TO tfe_app;
"

After that, Vault rotates tfe_app's password every 7 days without ever
changing the username, which is exactly what TFE needs. The dynamic role
stays registered too — it's a fine pattern for ad hoc admin access, just not
for the application's own connection.

---

Challenge 3: NooBaa S3 and a Missing Trust Anchor

The Problem

TFE uses S3-compatible object storage for run logs and state, and this
cluster provides that in-cluster via NooBaa (part of OpenShift Data
Foundation) rather than a cloud S3 bucket. Pointing TFE at NooBaa's internal
endpoint seemed straightforward — until TFE crashed on startup with:

x509: certificate signed by unknown authority

The NooBaa S3 endpoint's certificate is signed by OpenShift's own internal
service-CA, and TFE's container doesn't trust that CA by default. Nothing
about the bucket configuration was wrong; TFE just couldn't verify the TLS
certificate on the connection.

The Solution

Every OpenShift namespace gets an auto-populated ConfigMap
(openshift-service-ca.crt) containing exactly that CA certificate. The fix
is to mount it into the path TFE's chart already exposes for a custom CA
bundle:

# values.yaml
tls:
  caCertBaseDir: /etc/ssl/certs
  caCertFileName: custom_ca_certs.pem

extraVolumes:
  - name: openshift-service-ca
    configMap:
      name: openshift-service-ca.crt
      items:
        - key: service-ca.crt
          path: custom_ca_certs.pem

extraVolumeMounts:
  - name: openshift-service-ca
    mountPath: /etc/ssl/certs/custom_ca_certs.pem
    subPath: custom_ca_certs.pem
    readOnly: true

Once that ConfigMap is mounted and referenced by caCertBaseDir/
caCertFileName, TFE trusts the NooBaa endpoint's certificate and connects
cleanly.

Worth noting on the bucket side: the ObjectBucketClaim that provisions the
bucket also produces a Secret with the S3 access keys, but TFE never reads
that Secret directly. Those keys get copied into Vault, and VSO syncs them
back out into the Secret TFE actually consumes — keeping VSO/Vault as the
single source of truth for every credential, not just the ones that
originate there.

---

Challenge 4: The VSO Stuck-Reconciliation Bug, Again

If you read the Vault post, you'll remember VSO 1.4.0's bug where a
VaultStaticSecret can stop reconciling after a transient Vault HA error —
and doesn't recover on its own even if you restart the VSO controller. That
bug wasn't a one-off. It hit twice more here, on two of TFE's secrets, each
time triggered by an unrelated Vault rollout causing a brief Raft leader
election:

local node not active but active cluster node not found

Once on the VaultStaticSecret backing TFE's S3 credentials, and once on the
one backing its Redis password — both during Vault maintenance that had
nothing to do with TFE. Same symptom both times: SecretSynced=False, a
lastTransitionTime stuck days in the past, and zero new controller log
lines for the resource.

The fix is the same spec-patch workaround from the Vault post — touch the
resource to force VSO to requeue it, then revert:

oc -n tfe patch vaultstaticsecret tfe-s3 \
  --type=merge -p '{"spec":{"refreshAfter":"6m"}}'

# confirm SecretSynced=True and a fresh lastTransitionTime, then revert
oc -n tfe patch vaultstaticsecret tfe-s3 \
  --type=merge -p '{"spec":{"refreshAfter":"5m"}}'

There's a related but distinct issue on TFE's database credentials, which
come from a VaultDynamicSecret rather than a VaultStaticSecret. VSO 1.4.0
doesn't treat a static-role database response as renewable unless told
otherwise, so without one extra field it syncs once and never refetches —
meaning the next scheduled password rotation leaves TFE with a stale
password and failing Postgres logins:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: tfe-database
  namespace: tfe
spec:
  vaultAuthRef: tfe-auth
  mount: tfe-psql
  path: static-creds/tfe-static
  allowStaticCreds: true
  refreshAfter: 24h
  destination:
    name: tfe-database
    create: true
  rolloutRestartTargets:
    - kind: Deployment
      name: terraform-enterprise

Two different VSO resource types, two different failure modes, two different
fixes — but both are now a standard part of the runbook rather than a
surprise, since between the Vault deployment and this one, the same bug
family has now hit three separate secrets.

Update (2026): Digging in later reframed this. The stuck-reconciliation
part is a low-probability race — it needs a VSO reconcile in flight during
the brief leader-election window — not a frequent recurrence, and it isn't
fixed by upgrading VSO (the release after 1.4.0 is dependency/build-only, and
1.4.0 already ships the relevant VaultStaticSecret client-callback handler).
I couldn't reproduce it in six back-to-back forced elections. Full write-up in
the Vault post's Bug 2 update.
The spec-patch recovery above still applies.

---

Challenge 5: A Custom Job-Agent Image for OpenShift

The Problem

TFE runs each Terraform run in a short-lived "agent" container. The
stock hashicorp/tfc-agent image fails admission under OpenShift's
restricted security model — it expects to write to a working directory owned
by a fixed UID, which doesn't hold up once OpenShift assigns the container an
arbitrary UID from the namespace's allowed range.

The Solution

HashiCorp's own OpenShift deployment guide covers this: build a thin wrapper
image that opens up the permissions the arbitrary UID needs. I wired it up as
an OpenShift BuildConfig that layers a couple of chmod calls onto the
upstream image:

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: tfc-agent
  namespace: tfe-agents
spec:
  output:
    to:
      kind: ImageStreamTag
      name: tfc-agent:latest
  source:
    type: Dockerfile
    dockerfile: |
      FROM docker.io/hashicorp/tfc-agent:latest
      USER root
      RUN mkdir /.tfc-agent && \
          chmod og+rw /.tfc-agent && \
          chmod o+rx /home/tfc-agent
      USER tfc-agent
  strategy:
    type: Docker
    dockerStrategy:
      from:
        kind: DockerImage
        name: docker.io/hashicorp/tfc-agent:latest

TFE points at the built image via TFE_RUN_PIPELINE_IMAGE, referencing the
in-cluster image registry rather than Docker Hub directly. Re-running
oc start-build tfc-agent picks up new upstream releases of tfc-agent
without needing to restart TFE itself — the new image is used the next time
an agent pod spawns.

---

A Few Smaller Landmines

Two more things that didn't warrant a full section but are worth knowing
before you deploy TFE on OpenShift yourself:

• The chart hard-codes its own resource names. The Deployment and Service are both named the literal string terraform-enterprise, regardless of your Helm release name — nameOverride/fullnameOverride exist in the chart's helpers but the main templates never use them. Anything that needs to reference TFE by name (VSO's rolloutRestartTargets, a Route's to.name) has to use terraform-enterprise, not your release name. The Service's port is also named https-port, not the more conventional https — a Route with the wrong targetPort gives a silent 503 rather than an obvious error.
• The chart defaults to a LoadBalancer Service. On a bare-metal cluster with no LoadBalancer provider, that sits at EXTERNAL-IP: <pending> forever and keeps ArgoCD's health check stuck on Progressing. Overriding to service.type: ClusterIP and fronting it with an OpenShift Route clears this up immediately.

---

Conclusion

Running TFE active-active on OpenShift is straightforward once you've hit
each of these once, but the documentation doesn't cover any of them:

• Active-active means lockstep versions — RollingUpdate will crash-loop a mixed-version rollout every time; strategy: Recreate is the only correct answer, at the cost of a brief outage on every deploy
• Dynamic Vault roles don't fit every app — TFE's stable-username, owns-the-schema requirement needed a static role layered onto the same database connection, with a one-time manual ownership handoff
• In-cluster S3 needs in-cluster trust — mount OpenShift's own openshift-service-ca.crt ConfigMap if you're pointing TFE at NooBaa instead of a cloud S3 endpoint
• The VSO stuck-reconciliation bug isn't Vault-specific — it hit TFE's S3 and Redis secrets independently, both times triggered by unrelated Vault HA events; the spec-patch workaround is now a standing runbook step
• OpenShift's restricted SCC reaches into the job agent, too — not just the TFE pods themselves; budget time for the custom tfc-agent image build

Between this post and the Vault one, that's the full HashiCorp-on-OpenShift
stack this lab runs — Vault as the secrets backend, VSO as the sync layer,
and TFE consuming both. Every failure mode above came from the same root
cause in different clothes: OpenShift's stricter defaults (SCCs, arbitrary
UIDs, in-cluster CAs) and TFE's own strict operational assumptions (lockstep
versions, stable database identity) both push back against defaults that
work fine on vanilla Kubernetes.