DEV Community: drewmullen

Vault as an OIDC Identity Provider: Exposing Private Discovery Endpoints to the World

drewmullen — Fri, 17 Jul 2026 17:55:25 +0000

Note: this blog was co-authored with the excellent David Alexander

Introduction

HashiCorp Vault can be used to centralize third-party service access via Secret Engines. To do that Vault needs a way to authenticate to the service which can involve storing static keys often known as a foothold credential. Although many of these static keys can be rotated on a schedule by Vault, some security teams do not allow any static keys, even as footholds.

Vault has a built-in OIDC identity provider. Any service that supports Workload Identity Federation (WIF) — AWS, GCP, Azure, GitHub Actions, and others — can be configured to trust JWTs that Vault mints, eliminating the need for long-lived static credentials entirely.

The pattern is always the same: Vault issues a short-lived JWT, the relying service validates it by fetching Vault's OIDC discovery document and JWKS, and if everything checks out, it grants access. No secrets stored. No rotation scripts. No credential leakage.

But there's a catch: This workflow requires the service to have a network route to Vault's OIDC endpoints... and... most users host Vault in private network space so Vault's OIDC endpoints are not publicly reachable for validation.

There are two fundamentally different ways to solve this:

Expose Vault's endpoints — put something in front of Vault that makes its existing OIDC URLs reachable
Host the documents yourself — copy the discovery document and JWKS out of Vault and serve them from a public location you control

This post explores both approaches, their trade-offs, and when to choose each. As a bonus we include example code for setting up the 2nd approach. The concepts apply regardless of which service is consuming the tokens (JWTs).

Background:

Appendix: How WIF Token Validation Works for a detailed step-by-step breakdown.
Appendix: Vault's OIDC URL Structure for more detail on namespace path construction.

Vault Secrets Engines That Supports WIF

Although this post highlights use with AWS, Vault's WIF supports many secrets engines or auth method. Any that present a JWT to an external service can participate in this pattern. Examples include:

AWS secrets engine — assumes an IAM role via sts:AssumeRoleWithWebIdentity
GCP secrets engine — impersonates a service account via Workload Identity Federation
Azure secrets engine
Kubernetes auth — note this is the inverse direction — Kubernetes validates Vault's token

Approach 1: Expose Vault's Endpoints

The simplest conceptual model is to make Vault's own OIDC endpoints publicly reachable. Vault already serves the discovery document and JWKS — you just need to put a publicly accessible layer in front of them.

Example implementations exist in the wild, built to solve exactly this problem.

Trade-offs


✅ Always live	The relying service always sees current keys
✅ No sync process	Nothing to schedule or maintain
❌ Requires persistent infrastructure	Something must always be running and reachable
❌ New ingress surface	Even read-only, it's another path into your network

Approach 2: Host the Documents off Vault

Instead of routing traffic to Vault, you copy the discovery document and JWKS out of Vault and serve them from a static, publicly accessible location. Vault's OIDC endpoints never need to be reachable from the outside.

An example implementation can be found here

How it works

Configure Vault's issuer to point at your static hosting URL
Periodically fetch the discovery document and JWKS from Vault (from inside your network, where Vault is reachable) and publish them to the public location
The relying service fetches from your static host, not from Vault directly

The sync schedule requirement

Vault manages signing keys per OIDC key configuration. The default key (identity/oidc/key/default) has two relevant settings:

rotation_period    24h   # how often Vault generates a new signing key
verification_ttl   24h   # how long an old key remains in the JWKS after rotation

Vault keeps both the current and the next signing key in the JWKS endpoint simultaneously. When a rotation occurs, the outgoing key stays in the JWKS for the full verification_ttl window so that tokens signed with it can still be validated. This means the JWKS endpoint always contains at least two keys during any rotation window.

For the static hosting approach, this overlap is what makes the sync manageable: you don't need to sync at the exact moment of rotation. You need to sync at least once per rotation_period, before the previous key ages out of the JWKS. With the default 24h/24h settings, syncing once a day is sufficient — but syncing more frequently (e.g. every few hours) gives you a safety margin if a sync job fails.

This introduces an operational dependency: a scheduled job that runs the sync. The sync itself is simple (two unauthenticated HTTP GETs and a file upload), but it must be reliable and its schedule must be aligned with the key's verification_ttl.

Hosting

The static host just needs to serve two JSON files over HTTPS at predictable paths. Any of the following work:

Object storage (S3, GCS, Azure Blob) - zero infrastructure, globally available, integrates naturally with the cloud you're already using

CDN or static site host (Cloudflare Pages, GitHub Pages, Netlify) - useful if you want a custom domain, edge caching, or already have a CDN in front of your infrastructure

Any HTTPS file server - for self-hosted environments — even a simple nginx serving static files works

See Appendix: Vault's OIDC File Contents for more detail security concerns related to file contents.

Trade-offs


✅ No persistent infrastructure	Static files need no running process
✅ Vault stays fully private	No inbound network exposure whatsoever
✅ Works across clouds	The static host can be in a different cloud than the relying service
❌ Requires a sync process	Keys must be refreshed on a schedule
❌ Stale keys break auth	A missed sync during a rotation window causes failures
❌ Two-phase bootstrap	The static host must be populated before the relying service's trust config can be created

Multi-Namespace Considerations

Vault namespaces are independent OIDC providers. Each has its own keys, its own issuer URL, and its own discovery document. If you're using multiple namespaces — for example, one per team or environment — both approaches need to account for this.

Each namespace's discovery doc lives at v1/<namespace>/identity/oidc/plugins/.well-known/
With Approach 1, the proxy must correctly route per-namespace paths to Vault
With Approach 2, the static host must store and serve per-namespace documents at matching paths
A single host (one bucket, one proxy) can serve all namespaces — the namespace is encoded in the path, not the hostname

Conclusion

Eliminating static credentials from your infrastructure is a worthwhile goal, and Vault's built-in OIDC identity provider makes it achievable across AWS, GCP, Azure, and beyond. The challenge is almost never the WIF configuration itself — it's getting the discovery documents to a place the relying service can reach.

Both approaches described here solve the same network topology problem. Exposing Vault's endpoints is operationally simpler but introduces a persistent ingress surface and ties credential issuance to Vault's availability. Hosting the documents yourself keeps Vault fully private and decouples auth from Vault uptime, at the cost of a sync process you have to operate and monitor.

Neither approach is universally better. The right choice depends on your network constraints, your tolerance for operational overhead, and how strictly your security policy treats inbound access to Vault. What matters is that you make the choice deliberately — and that whichever path you take, the JWKS your relying service sees always reflects what Vault is actually signing with.

Appendix

How WIF Token Validation Works

At a high level:

Vault mints a JWT with an iss (issuer) claim pointing to its configured issuer URL
The application or platform presents that JWT to the relying service
The relying service reads the iss claim and fetches <issuer>/.well-known/openid-configuration
From the discovery document, it reads the jwks_uri field and fetches the public keys
It verifies the JWT signature against those keys, and checks the aud (audience) and other claims
If everything matches its trust configuration, it grants access

The critical constraint: steps 3 and 4 happen from the relying service's infrastructure. Whatever URL is in the iss claim must be reachable from there. For cloud services like AWS STS, GCP STS, or Azure AD, that means the public internet.

Vault's OIDC URL Structure

Vault serves its OIDC discovery documents at a predictable path:

<issuer>/.well-known/openid-configuration
<issuer>/.well-known/keys

The issuer itself is constructed from a base URL you configure in identity/oidc/config, with Vault appending the namespace path automatically:

https://<base>/<vault-path>/identity/oidc/plugins

Each Vault namespace has its own independent OIDC provider, its own keys, and its own issuer URL. This matters for both approaches described above.

Configuring New or Existing Keys

These values are configurable per key. Teams with stricter security requirements can shorten the rotation period; teams that need a longer sync window can extend verification_ttl. Both can be adjusted without recreating the key:

vault write identity/oidc/key/default \
  rotation_period=6h \
  verification_ttl=12h

File Content Security Concerns

The discovery document and JWKS contain only public information — metadata about the issuer and the public half of the signing key pairs. The private keys never leave Vault. Hosting these files publicly is not only safe, it is required by the OIDC specification: the relying service must be able to fetch them without authentication.

The practical implication is that anyone who can reach your static host can read your issuer URL and key IDs. This is low-risk — knowing a public key does not allow an attacker to forge tokens, only to verify them. It does reveal that you are using Vault as an identity provider, which may be relevant to your threat model, but is generally considered acceptable for the same reason that public TLS certificates are acceptable.

aws s3 cp s3://vault-oidc-discovery-123456789012/v1/admin/identity/oidc/plugins/.well-known/openid-configuration - | python3 -m json.tool
{
    "issuer": "https://vault-oidc-discovery-123456789012.s3.amazonaws.com/v1/admin/identity/oidc/plugins",
    "jwks_uri": "https://vault-oidc-discovery-123456789012.s3.amazonaws.com/v1/admin/identity/oidc/plugins/.well-known/keys",
    "response_types_supported": [
        "id_token"
    ],
    "subject_types_supported": [
        "public"
    ],
    "id_token_signing_alg_values_supported": [
        "RS256",
        "RS384",
        "RS512",
        "ES256",
        "ES384",
        "ES512",
        "EdDSA"
    ]
}

Adopting Terraform Ephemeral Resources

drewmullen — Thu, 09 Jul 2026 12:50:48 +0000

In version 1.11, HashiCorp introduced Terraform Ephemeral resources and write-only attributes to allow for root configs that do not store secrets in the Terraform statefile. But many users ask about how they can adopt ephemerals. This blog attempts to lay out the ways secrets can be stored in state and how you should update your configurations to remove those secrets.

Note: For a primer on ephemerals (see this blog post).

Scenarios to consider:

Data sources that fetch a static secret
Resources that receive a secret
Resources that generate a dynamic secret
Resources that fetch generated secrets to store in another 3rd party system

Scenario 1: Data sources with static secrets

Ephemeral resources can often be a drop-in replacement for data sources pulling static values:

data "vault_kv_secret_v2" "static_kv" {
  mount = "kvv2"
  name  = "my_password"
}

ephemeral "vault_kv_secret_v2" "static_kv" {
  mount = "kvv2"
  name = "my_password"
}

However, using these values has 1 specific difference. The attributes on a ephemeral resource are considered ephemeral and can only be used as ephemeral arguments. That means 2 places:

Provider blocks

Provider blocks are considered ephemeral, so ephemeral resources may populate arguments:

provider "example" {
  password = tostring(ephemeral.vault_kv_secret_v2.static_kv.data.password)
}

Write-only arguments

Write-only arguments are special arguments that require the ephemeral taint for values:

resource "aws_db_instance" "example" {
  ...
  password_wo = tostring(ephemeral.vault_kv_secret_v2.static_kv.data.password)
}

If the resource you wish to pass a value to does not have an available write-only attribute, open an issue with that provider. You can reference this blog post

Scenario 2: Resources that receive a static secret

Without duplicating to the section above, write-only arguments are a way to get secrets out of state. Above has guidance if the secret value comes from a data source, but what if its from a variable? In that case, we just need to add the ephemeral taint to the variable declaration:

variable "my_password" {
  type       = string
  ephemeral  = true
  senstitive = true
}

resource "aws_db_instance" "example" {
  ...
  password_wo = var.my_password
}

Scenario 3: Resources that generate a dynamic secret

Now is when things get slightly trickier. There are some resources that generate dynamic secret content:

# generates a random password
resource "random_password" "pw" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

These can also be replaced with an ephemeral resource but theres 1 difference. Ephemeral resources have no state and run every plan & apply. This means that every time plan or apply is run, the random_password will be regenerated.

To account for this, users should use the wo_version arguments:

variable "password_keeper" {
  type        = number
  description = "Bump this number to update the password associated with `ephemeral.random_password.pw`."
  default     = 1
}

ephemeral "random_password" "pw" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "aws_db_instance" "example" {
  ...
  password_wo         = ephemeral.random_password.pw.result
  password_wo_version = var.password_keeper
}

If you would like to know about how to implement this pattern in modules, see this blog post.

Scenario 4: Resources or data sources that fetch generated secrets to store in another 3rd party system

There are many situations when a user would like to use Terraform to generate and hand off a secret between 2 3rd party systems. A typical workflow is asking Terraform to generate a API key that will be given to another 3rd party system to give it access. This is currently a feature gap for ephemerals.

Take the example of giving a HCP Terraform API token to give to another tool (GitHub Pipeline, auditing tool, etc):

resource "tfe_team_token" "test" {
  team_id     = tfe_team.test.id
  description = "my team token"
}

Although you may think an ephemeral resource for this could fix the issue, it sneakily does not. Heres why:

ephemeral "tfe_team_token" "example" {
  team_id = tfe_team.example.id
}

The above code would execute every plan & apply and delete the token at the end of the run. This means that for normal terraform run workflow, 2 team tokens would be generated and deleted. Even if you gave the token to a 3rd party system (AWS secrets manager), the value would be stored but the key deleted.

As stated above, these ephemerals can be used to authenticate providers. This is commonly done with the value dynamic credentials backends:

ephemeral "vault_terraform_token" "tf_token" {
  role_name = "tfe_admin"
}

provider "tfe" {
  token = ephemeral.vault_terraform_token.tf_token.token
}

Closing

This blog walked you through the scenarios and solutions you will encounter when trying to remove secrets from your Terraform statefile.

If you would like a way to detect when your workspaces are using resources that store secrets in state, check out this Sentinel Ephemerality Policy.

Fully Migrate Secrets Out Of Terraform Module State Without Breaking Existing Users

drewmullen — Wed, 29 Apr 2026 19:18:08 +0000

A few weeks ago I published a similar blog that shows how you can update your modules to optionally utilize ephemeral secrets, removing secrets from state for all new deployments. However, to maintain totally programmatic, older deployments still retained the secret.

This blog explores a method to fully remove secrets from state, even on old deployments. However, it requires a manual step on behalf of users.

Setup

V1 of your module had a resource which introduced a secret into state:

resource "tls_private_key" "legacy" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "vault_kv_secret_v2" "legacy" {
  mount     = "kvv2"
  name      = "mytls"
  data_json = jsonencode({ 
    private_key = tls_private_key.legacy.private_key_pem
  })
}

tls_private_key.legacy.private_key_pem contains a secret value that is stored in state.

Update

By updating your module (V2) to either build new with a ephemeral from the start or sourcing the legacy private_key_pem from an ephemeral variable, we can remove the secret from state in both circumstances.

The kick is getting the secret from state to a tf variable

variable "secret_version" {
  description = "Increment to trigger a re-write of the Vault secret. Only relevant when use_ephemeral_key = true."
  type        = number
  default     = 1
}

variable "private_key_data" {
  description = "The private key data, sourced from a legacy private_key resource depending on use_ephemeral_key."
  type        = string
  ephemeral   = true
  sensitive   = true
  default     = null
}

removed {
  lifecycle {
    destroy = false
  }

  from = tls_private_key.legacy
}

ephemeral "tls_private_key" "ephemeral" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "vault_kv_secret_v2" "legacy" {
  mount        = "kvv2"
  name         = "mytls"
  data_json_wo = var.private_key_data != null ? jsonencode(
    { private_key = var.private_key_data }) : jsonencode(
    { private_key = ephemeral.tls_private_key.ephemeral.private_key_pem })
  data_json_wo_version = var.secret_version
}

The code above offers 2 branches:

Legacy
New build

Legacy

For v1 users, if they extract the tls_private_key.legacy.private_key_pem value to var.private_key_data, the next run will remove the old resource from state and update the resource to use the write-only value!

New builds

For net new builds, do not set var.private_key_data, the new ephemeral resource will write to data_json_wo.

Final thoughts

Its possible you'll want to condition your secret producing resource (what I have as ephemeral.tls_private_key) you can add a conditional but keep these notes in mind.

Writing Terraform Resources with Write-Only Parameters

drewmullen — Thu, 22 Jan 2026 22:10:57 +0000

When building Terraform providers that handle sensitive data like passwords, API tokens, or secret keys, you'll eventually encounter the need for write-only parameters. These are values that should be sent to an API but never stored in Terraform state files.

In this post, I'll walk through implementing write-only parameters in a Terraform provider, the challenges involved, and design options to consider.

What Are Write-Only Parameters?

Write-only parameters (introduced in Terraform 1.11) are resource arguments that:

Are sent to the API during creation and updates
Never appear in the Terraform state file

This feature is crucial for security. Without write-only parameters, sensitive values like passwords would be stored in plaintext in your state files.

Here's an example from my fork of the Event Drive Ansible (EDA) provider:

resource "aap_eda_credential" "example" {
  name               = "my-api-credential"
  credential_type_id = 1

  # Write-only: sent to API but NEVER stored in state
  inputs_wo = jsonencode({
    username  = "service-account"
    api_token = var.api_token
  })
}

Design Considerations

The `_version` Argument

Since Terraform's statefile does not manage the value of the secret, Terraform requires way to know if the secret must be updated. The standard solution to detecting changes in write-only parameters is to add a companion _version field that users must manually increment:

resource "aap_eda_credential" "manual" {
  name      = "my-credential"
  inputs_wo = jsonencode({
    password = var.new_password
  })

  # User must increment this to trigger an update
  inputs_wo_version = 1
}

How updates work:

User updates the secret in inputs_wo
User manually increments inputs_wo_version
Terraform detects the version change and triggers an update
The new secret is sent to the API

This works, but it's not user-friendly. Users have to remember to:

Increment the version every time they change secrets
Track version numbers manually

Automatic Change Detection

Ideally, users shouldn't need to manage version numbers. They should just update the secret value and Terraform should detect the change automatically. But here's the problem:

If inputs_wo is write-only and not in state, how do we detect when it changes?

Lucikly, there is a solution: store a deterministic hash in private state.

Using Private State and Hashing

Terraform's plugin framework provides private state - a place to store provider-managed data that's:

Stored in the state file (so it persists)
Not visible to users (doesn't appear in terraform show)
Perfect for storing things like hashes

Here's the implementation approach:

// Create operation
func (r *EDACredentialResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
    var data EDACredentialResourceModel

    req.Plan.Get(ctx, &data)

    // Calculate SHA-256 hash of the inputs
    inputsHash := calculateInputsHash(data.InputsWO.ValueString())

    // Store hash in private state (JSON-wrapped for validity)
    hashJSON := fmt.Sprintf(`{"hash":"%s"}`, inputsHash)
    resp.Private.SetKey(ctx, "inputs_wo_hash", []byte(hashJSON))

    // Set auto-managed version
    data.InputsWOVersion = tftypes.Int64Value(1)

    // ... send to API and save state ...
}

func calculateInputsHash(inputs string) string {
    h := sha256.New()
    h.Write([]byte(inputs))
    return hex.EncodeToString(h.Sum(nil))
}

In the Update operation, we compare hashes:

func (r *EDACredentialResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
    var data, state EDACredentialResourceModel

    req.Plan.Get(ctx, &data)
    req.State.Get(ctx, &state)

    // Get stored hash from private state
    oldHashBytes, _ := req.Private.GetKey(ctx, "inputs_wo_hash")

    var hashWrapper struct {
        Hash string `json:"hash"`
    }
    json.Unmarshal(oldHashBytes, &hashWrapper)
    oldHash := hashWrapper.Hash

    // Calculate new hash
    newHash := calculateInputsHash(data.InputsWO.ValueString())

    if newHash != oldHash {
        // Inputs changed! Auto-increment version
        data.InputsWOVersion = tftypes.Int64Value(state.InputsWOVersion.ValueInt64() + 1)

        // Update stored hash
        hashJSON := fmt.Sprintf(`{"hash":"%s"}`, newHash)
        resp.Private.SetKey(ctx, "inputs_wo_hash", []byte(hashJSON))
    } else {
        // No change, keep version as-is
        data.InputsWOVersion = state.InputsWOVersion
    }

    // ... send to API and save state ...
}

Now users can just update the secret and Terraform automatically detects it:

resource "aap_eda_credential" "auto" {
  name      = "my-credential"
  inputs_wo = jsonencode({
    password = var.new_password  # Just change this!
  })
  # inputs_wo_version auto-increments (computed field)
}

Best of both worlds

While automatic detection is convenient, some users may not want a hash of their secrets stored in state - even if it's SHA-256 and in private state. The hash is still derived from the secret value.

Solution: Support both modes

func (r *EDACredentialResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
    var data EDACredentialResourceModel
    req.Plan.Get(ctx, &data)

    var versionToSet tftypes.Int64

    if data.InputsWOVersion.IsNull() || data.InputsWOVersion.IsUnknown() {
        // Auto-managed mode: store hash
        inputsHash := calculateInputsHash(data.InputsWO.ValueString())
        hashJSON := fmt.Sprintf(`{"hash":"%s"}`, inputsHash)
        resp.Private.SetKey(ctx, "inputs_wo_hash", []byte(hashJSON))
        versionToSet = tftypes.Int64Value(1)
    } else {
        // Manual mode: user provided version, don't store hash
        versionToSet = data.InputsWOVersion
    }

    // ... rest of create logic ...
    data.InputsWOVersion = versionToSet
}

Schema definition:

"inputs_wo_version": schema.Int64Attribute{
    Optional: true,
    Computed: true,
    PlanModifiers: []planmodifier.Int64{
        int64planmodifier.UseStateForUnknown(),
    },
    Description: "Version number for managing credential updates. " +
        "If not set, the provider will automatically detect changes to inputs_wo " +
        "using a SHA-256 hash stored in private state. If set manually, you " +
        "control when the credential is updated by incrementing this value yourself.",
}

Users can choose their preferred mode:

# Auto-managed (default)
resource "aap_eda_credential" "auto" {
  name      = "auto-credential"
  inputs_wo = jsonencode({ password = var.pwd })
  # version auto-increments
}

# Manual control
resource "aap_eda_credential" "manual" {
  name              = "manual-credential"
  inputs_wo         = jsonencode({ password = var.pwd })
  inputs_wo_version = 1  # I control this
}

Handling Mode Switching

Unfortunately, a weakness of this design is that its not possible to "switch modes". Why?

Auto → Manual: If a user suddenly sets inputs_wo_version, do we remove the hash? What if they're just trying to force an update?
Manual → Auto: We'd need to create a hash, but we can't compare it to the old inputs (they're write-only and not in state), so we can't tell if inputs changed during the transition

Even if we could find a way to handle this, the logic in your resource becomes complex and error-prone.

Better approach: Prevent mode switching

func (r *EDACredentialResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
    // Determine current mode from private state
    oldHashBytes, _ := req.Private.GetKey(ctx, "inputs_wo_hash")
    wasAutoManaged := oldHashBytes != nil

    // Determine desired mode from config
    var configModel EDACredentialResourceModel
    req.Config.Get(ctx, &configModel)
    isNowManual := !configModel.InputsWOVersion.IsNull() && !configModel.InputsWOVersion.IsUnknown()

    var state EDACredentialResourceModel
    req.State.Get(ctx, &state)
    wasManual := !wasAutoManaged && !state.InputsWOVersion.IsNull()

    // Prevent mode switching
    if wasAutoManaged && isNowManual {
        resp.Diagnostics.AddError(
            "Cannot switch from auto-managed to manual version management",
            "The inputs_wo_version field was previously auto-managed. Once auto-managed, "+
            "it cannot be switched to manual mode. If you need to manually control the version, "+
            "you must recreate the resource with inputs_wo_version set from the start.",
        )
        return
    }

    if wasManual && !isNowManual {
        resp.Diagnostics.AddError(
            "Cannot switch from manual to auto-managed version management",
            "The inputs_wo_version field was previously manually managed. Once manually managed, "+
            "it cannot be switched to auto mode. If you need auto-managed version control, "+
            "you must recreate the resource without setting inputs_wo_version.",
        )
        return
    }

    // ... rest of update logic for the chosen mode ...
}

This gives users a clear error message:

Error: Cannot switch from auto-managed to manual version management

The inputs_wo_version field was previously auto-managed. Once auto-managed,
it cannot be switched to manual mode. If you need to manually control the
version, you must recreate the resource with inputs_wo_version set from the start.

Key Takeaways

When implementing write-only parameters in Terraform providers:

Use private state for hashes - It's stored but not visible to users
Support both auto and manual modes - Give users choice for their security preferences

The complete implementation provides a great user experience while respecting privacy concerns:

# Most users: auto-managed, no version to track
resource "aap_eda_credential" "api" {
  name      = "api-credential"
  inputs_wo = jsonencode({ token = var.token })
}

# Privacy-conscious users: manual control, no hash stored
resource "aap_eda_credential" "secure" {
  name              = "secure-credential"
  inputs_wo         = jsonencode({ token = var.token })
  inputs_wo_version = 1
}

Both approaches are valid. Both are clear. And switching between them requires a resource recreation, which is easy to communicate and reason about.

Have you implemented write-only parameters in your Terraform provider? What challenges did you face? Let me know in the comments!

Making Terraform Workspaces in the same project

drewmullen — Fri, 27 Jun 2025 14:10:43 +0000

I am working on a artifact for helping TFE users provide some self service (to be unveiled soon!). One of the things I needed to do was provide a repeatable way to have a tfe workspace create workspaces in the same project. At first I thought I'd need a new data source for this since the tfe data sources require organization and at the very least.

Turns out I can do it today with some magic runtime env vars:

variable "TFC_WORKSPACE_SLUG" {}
variable "TFC_PROJECT_ID" {}
locals {
  org_name = split("/",var.TFC_WORKSPACE_SLUG)[0]
}

resource "tfe_workspace" "ws" {
  name         = "new-ws"
  organization = local.org_name
  project_id   = var.TFC_PROJECT_ID
}

Executing that code in a TFE workspace will let you create workspaces without knowing the organization name or project! Niche but pretty nifty!

Setting a multi-line variable as an ENV in Terraform Cloud

drewmullen — Wed, 12 Feb 2025 15:55:56 +0000

In this post I will describe the exploration I did after running into a problem trying to setup temporary auth to my kubernetes cluster with Terraform. I will first describe the problem(s) and several ways to solve them.

Preamble

I auth to my homelab k8s cluster using PEM keys. Being a long time Terraform user I almost always auth to a provider using environment variables. This allows me to decouple auth from the HCL, prevent secrets from getting into the statefile, and often allows completely avoiding defining a particular provider block, relying on the "empty default configuration".

Terraform Cloud (TFC) (recently rebranded to HCP TF) allows setting variables outside of the code and in the web ui instead (WS Vars). You can set WS Vars as either type terraform or type env. Type terraform behaves exactly like setting -var myname=drew. Only problem is that type env vars cannot be multi-line which breaks PEM validity.

Solutions

Ephemeral Variables!

Starting Terraform v1.10 HashiCorp introduced ephemeral resource types as well as ephemeral variables. This allows me to set the PEM in a WS Var type terraform but not expose to the statefile. It does require me to hardcode the provider version and auth.

variable "kube_client_key" {
  description = "Private key data for authenticating to k8s."
  ephemeral   = true
}

provider "kubernetes" {
  client_certificate = var.kube_client_certificate
  client_key         = var.kube_client_key
}

Base64 & TF_VAR_

You can pass the multi-line content base64 encoded then leverage terraform's base64decode() function in a TF_VAR_. This allows using an env variable but still requires hardcoding the auth. This solution doesn't make much sense for me but it is an option

provider "kubernetes" {
  client_key = base64decode(var.kubekey)
}

Dynamic K8s Access

You can setup a trust between TFC and your k8s cluster. This is my end game but on the back burner while I figure out how I plan to organize somethings

Doc Link: https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials/kubernetes-configuration

E1it3 Hax

This last option is totally unexpected but actually quite interesting. Turns out that part of TFC's implementation is that WS Vars that are type terraform are also written to agents as environment variables. This means that you can create a WS Var in TFC that matches the naming convention of the expected provider env var and it will still work. This does not require you to create Terraform HCL for the vars but it does surface a warning. You can suppress the warning by providing a empty variable with the same name and NOT defining it in the provider block.

To suppress the warning just insert a variable with the corresponding name and a default value:

variable "KUBE_CLIENT_KEY_DATA" {
  default = ""
}

Salutation

I hope this is helpful to you. Choose the solution that fits your use-case best!

Deploy Cloudflare Tunnels with Terraform & Kubernetes

drewmullen — Tue, 21 Jan 2025 18:32:07 +0000

This blog will give you a brief overview for exposing services from a private network to the public network using zero trust tool, CloudFlare Zero Trust Tunnels. They have a generous free tier that you can leverage. Please note that this post does not encompass a fully prescriptive set of security best practices, merely it gets the setup working. Getting this right took me longer than I expected. Hopefully its helpful to you.

Due to to my homelab setup, namespaces are created by ArgoCD which kicks off ahead of terraform apply. You may need to make a namespace cloudflared.

Access

Based on the guide from cloudflares docs you'll need to create a API token:

Note: When creating your api token know that "Cloudflare Tunnel" is not the same thing as Zero Trust in their permissions. This was confusing to me because tunnels are now named "zero trust tunnels". For the sake of permissions make sure you provider "Cloudflare Tunnel: Edit".

Terraform

The following code deploys a single tunnel and sets up DNS records for all sites indicated in "services". TLS verify is disabled below so update it if you want.

variable "cloudflare_zone_id" {
  description = "Zone ID for your domain"
  type        = string
  sensitive   = true
}

variable "cloudflare_account_id" {
  description = "Account ID for your Cloudflare account"
  type        = string
  sensitive   = true
}

variable "cloudflare_email" {
  description = "Email address for your Cloudflare account"
  type        = string
  sensitive   = true
}

variable "services" {
  description = "Values for the services to be exposed via Cloudflare Tunnel"
  type = map(object({
    hostname = string # public domain name. ex: "service.example.com"
    service  = string # private service endpoint. ex: service.apps.internaldomain.com
  }))
}

locals {
  cf_tunnel_secret = jsonencode({
    "AccountTag" : "${var.cloudflare_account_id}",
    "TunnelSecret" : "${base64sha256(random_password.tunnel_secret.result)}",
    "TunnelID" : "${cloudflare_zero_trust_tunnel_cloudflared.homelab.id}"
  })

  ingress_rules = [
    for service in var.services : {
      hostname = service.hostname
      origin_request = {
        no_tls_verify = true
      }
      service = service.service
    }
  ]
}

resource "random_password" "tunnel_secret" {
  length = 64
}

resource "cloudflare_zero_trust_tunnel_cloudflared" "homelab" {
  name          = "homelab-tunnel"
  config_src    = "cloudflare"
  account_id    = var.cloudflare_account_id
  tunnel_secret = base64sha256(random_password.tunnel_secret.result)
}

resource "cloudflare_zero_trust_tunnel_cloudflared_config" "homelab" {
  account_id = var.cloudflare_account_id
  tunnel_id  = cloudflare_zero_trust_tunnel_cloudflared.homelab.id
  config = {
    ingress = concat(local.ingress_rules, [
      {
        origin_request = {
          connect_timeout        = 0
          keep_alive_connections = 0
          keep_alive_timeout     = 0
          tcp_keep_alive         = 0
          tls_timeout            = 0
        }
        service = "http_status:503"
      },
    ])
    warp_routing = {
      enabled = false
    }
  }
}

resource "kubernetes_secret" "cloudflare_credentials" {
  metadata {
    name      = "tunnel-credentials"
    namespace = "cloudflared"
  }

  data = {
    "credentials.json" = local.cf_tunnel_secret
  }
}

resource "cloudflare_dns_record" "vault_homelab_tunnel" {
  for_each = var.services

  zone_id = var.cloudflare_zone_id
  comment = "${each.key} tunnel record"
  content = join(".", [cloudflare_zero_trust_tunnel_cloudflared.homelab.id, "cfargotunnel.com"])
  name    = each.value.hostname
  proxied = true
  ttl     = 1
  type    = "CNAME"
}

Kubernetes manifest:

In my setup, k8s resources are mostly deployed via ArgoCD. Below, this manifest deploys cloudflared pod and connects it to the tunnel homelab-tunnel from your account. It authenticates via the public cloudflare API using credentials from a kubernetes secret tunnel-credentials which is created by Terraform.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: cloudflared
spec:
  selector:
    matchLabels:
      app: cloudflared
  replicas: 1 
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
      - name: cloudflared
        image: cloudflare/cloudflared:latest
        args:
        - tunnel
        - --credentials-file
        - /etc/cloudflared/creds/credentials.json
        - --protocol # https://github.com/cloudflare/cloudflared/issues/1176#issuecomment-2404546711
        - http2      # didnt seem to need the sysctl changes
        - --metrics
        - 0.0.0.0:2000
        - run
        - homelab-tunnel
        livenessProbe:
          httpGet:
            path: /ready
            port: 2000
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
        volumeMounts:
        - name: creds
          mountPath: /etc/cloudflared/creds
          readOnly: true
      volumes:
      - name: creds
        secret:
          secretName: tunnel-credentials

Disclaimer: The following code stores the secret to joining your tunnel in your terraform statefile in 2 locations random_password.tunnel_secret and cloudflare_zero_trust_tunnel_cloudflared.homelab. With 1.10 we have ephemeral resources which will remove that but the random provider doesnt have it yet. Hopefully it will be removed from tunnel resource once available.

Terraform state with native s3 locking

drewmullen — Mon, 02 Dec 2024 20:21:58 +0000

Terraform has many popular mechanisms for storing its state file. While I have grown quite fond of HCP Terraform, there are occasional scenarios where I want to use S3 backend to hold my Terraform state files.

Best practice when sharing a state file among multiple parties is to have a locking mechanism, to avoid concurrent writes from corrupting the Terraform state. This prevents 2 parties from clobbering each other if they happen to be working on the same infrastructure at the same time.

Starting in Terraform v1.10 the S3 backend features S3 native state locking. Prior to this feature state file lock setups required access to a DynamoDB table - which can be completely foregone now!

How?

Lets start by creating a S3 bucket that has the required settings:

Note: It is common to create the S3 state file bucket outside of Terraform. However, you can create the bucket with Terraform, its just generally not advised. Example shell commands below.

$ BUCKET_NAME="<your bucket name>"
$ REGION="us-east-1"

$ aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --region $REGION

$ aws s3api put-bucket-versioning \
    --bucket $BUCKET_NAME \
    --versioning-configuration Status=Enabled

Using the bucket

Create a new backend.tf file (or add this block to an existing .tf file). Make sure you fill in the bucket name!

Below we use the parameter use_lockfile which is an experimental feature that enforces using the s3 buckets lock file

terraform {
  backend "s3" {
    bucket       = <bucket name>
    key          = "backend/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
  }
}

Wrapping up

For a deep dive on this feature I recommend you read through the initial PR by @bschaatsbergen who wrote this great feature

Optional: Create bucket using Terraform

resource "aws_s3_bucket" "backend" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_versioning" "backend" {
  bucket = aws_s3_bucket.backend.id
  versioning_configuration {
    status = "Enabled"
  }
}

Edited 12/19 to remove ObjectLock which is not required. Kudos to @philippe_robitaille_f3908

AWS Creds in the CLI via SSO

drewmullen — Thu, 16 Nov 2023 15:59:07 +0000

Problem

Do you often need credentials in your AWS CLI?

Are you overly familiar with this screen (AWS SSO start page)?

If you do not have AWS SSO setup, check out the AWS documentation then come back to this post!

A better credential experience

Yesterday I learned from my colleague @danquack a fun feature built into the AWS CLI. If you already have AWS SSO configured for your Org and are using the GUI to get credentials, follow this post and you can expect an improved AWS CLI credential management experience.

My new, simpler process to get temporary creds from SSO:

Specify which profile I want and login:

$ export AWS_PROFILE=<>
$ aws sso login

Confirm the authorization request in my browser:

Boom! Now my CLI has usable temporary credentials!

Setup

This setup is honestly extremely simple. AWS provides a guided CLI wizard and ill show the examples below.

$ aws configure sso
SSO session name (Recommended):
WARNING: Configuring using legacy format (e.g. without an SSO session).
Consider re-running "configure sso" command and providing a session name.
SSO start URL [https://example.awsapps.com/start#/]:
SSO region [us-east-1]:
There are 2 AWS accounts available to you.
> DeveloperAccount, developer-account-admin@example.com (123456789011) 
  ProductionAccount, production-account-admin@example.com (123456789022)
Using the account ID 123456789011
The only role available to you is: AdministratorAccess
Using the role name "AdministratorAccess"
CLI default client Region [us-east-1]: us-east-2
CLI default output format [None]:
CLI profile name [AdministratorAccess-<>]: providerdev

There are 2 questions regarding region. The first is the region SSO is setup in. The second is the default region you want your CLI setup to use.
Setting a profile name providerdev is now the name ill set for export AWS_PROFILE=providerdev

Once that is complete you can see the configuration:

$ cat -p ~/.aws/config
[profile providerdev]
sso_start_url = https://example.awsapps.com/start#/
sso_region = us-east-1
sso_account_id = 123456789011
sso_role_name = AdministratorAccess
region = us-east-2

Summary

AWS SSO is a great service for providing temporary credentials to known identities in your organization. The new command aws sso login will help you and your engineers get credentials fast, easy, and securely!

Implement Rollback with Terraform

drewmullen — Thu, 12 Jan 2023 14:06:56 +0000

There's no magic here, friends.

$ terraform apply # fails
$ git checkout HEAD~ # checkout the prior commit
$ terraform apply

You can checkout a specific commit sha with:
git checkout <commit sha>

I don't recommend using this with any stateful resource types... RDS, EKS, etc.

Terraform: Variable validation with samples

drewmullen — Fri, 04 Mar 2022 18:57:38 +0000

Terraform allows you to validate variable input in using validation blocks using custom condition and yielding a custom error_message. Below are some examples:

Note: Please share your common validation rules you've written and I'll update here

Update: Please check out this awesome project by @bschaatsbergen that provides built in functions for assertions: https://github.com/bschaatsbergen/terraform-provider-assert

Strings

String may not contain character

Scenario: String may not contain a /.

variable "string_may_not_contain" {
  type = string
  default = "test"

  validation {
    error_message = "Value cannot contain a \"/\"."
    condition = !can(regex("/", var.string_may_not_contain))
  }
}

String with valid options

Scenario: Here we have a string and we only allow to values "approved" or "disapproved". I show 2 examples of the same check using different methods:

variable "string_only_valid_options" {
  type = string
  default = "approved"

  # using regex
  validation {
    condition     = can(regex("^(approved|disapproved)$", var.string_only_valid_options))
    error_message = "Invalid input, options: \"approved\", \"disapproved\"."
  }

  # using contains()
  validation {
    condition     = contains(["approved", "disapproved"], var.string_only_valid_options)
    error_message = "Invalid input, options: \"approved\", \"disapproved\"."
  }
}

Valid AWS Region Name

Scenario: string must be like AWS region

variable "string_like_aws_region" {
  type = string
  default = "us-east-1"

  validation {
    condition     = can(regex("[a-z][a-z]-[a-z]+-[1-9]", var.string_like_aws_region))
    error_message = "Must be valid AWS Region names."
  }

Valid IAM Role Name

Scenario: Your string must be a valid IAM role name

variable "string_valid_iam_role_name" {
    type = string
    default = "MyCoolRole"
    # arn example: "arn:aws:iam::123456789012:role/MyCoolRole"

    validation {
      condition     = can(regex("^[a-zA-Z][a-zA-Z\\-\\_0-9]{1,64}$", var.string_valid_iam_role_name))
      error_message = "IAM role name must start with letter, only contain letters, numbers, dashes, or underscores and must be between 1 and 64 characters."
    }
}

Valid IPv4 CIDR

Scenario: Your string input needs to look like a IPv4 CIDR. Thank you @entscheidungsproblem for reporting and providing a fix for /32.

variable "string_like_valid_ipv4_cidr" {
  type    = string
  default = "10.0.0.0/16"

  validation {
    condition     = can(cidrhost(var.string_like_valid_ipv4_cidr, 0))
    error_message = "Must be valid IPv4 CIDR."
  }
}

Semantic Version

variable "semv1" {
  default = "10.57.123"

  validation {
    error_message = "Must be valid semantic version."
    condition     = can(regex("^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\\+([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?$", var.semv1))
  }
}

Maps

Map with optional conflicting keys

Scenario: You have a map variable and 2 keys conflict, in this case, you can only set either cidr or netmask.

variable "only_one_optional_key" {
    type = object({
        name = optional(string)
        cidrs = optional(list(string))
        netmask = optional(number)
    })

    default = {
        cidr = "10.0.0.0/16"
        name = "test"
    }

    validation {
        error_message = "Can only specify either \"cidrs\", or \"netmask\"."
        condition = length(setintersection(keys(var.only_one_optional_key), ["cidrs", "netmask"])) == 1
    }
}

Numbers

Number within a range

Scenario: number must be between 1-16.
Thanks: @tlindsay42

variable "num_in_range" {
  type        = number
  default     = 1

  validation {
    condition     = var.num_in_range >= 1 && var.num_in_range <= 16 && floor(var.num_in_range) == var.num_in_range
    error_message = "Accepted values: 1-16."
  }
}

If you liked this post, please like. If you think it would be helpful in the future as a reference, please bookmark!

Terraform: Common aws_ami Searches

drewmullen — Wed, 02 Feb 2022 20:03:11 +0000

Searching for AMI's is not fun. Here are some you might be searching for:

Amazon Linux 2 (latest):

data "aws_ami" "amazon_linux_2" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-ebs"]
  }
}

Graviton 2

RHEL 7 (latest)

Thanks: @wsarwariamzn

data "aws_ami" "rhel_7" {
  most_recent = true
  owners      = ["309956199498"]

  filter {
    name = "name"
    values = ["RHEL-7.9_HVM_GA*-x86_64-0-Hourly2-GP2"]
  }
}

Please send me more and I'll include them in this post!

DEV Community: drewmullen

Vault as an OIDC Identity Provider: Exposing Private Discovery Endpoints to the World

Introduction

Background:

Vault Secrets Engines That Supports WIF

Approach 1: Expose Vault's Endpoints

Trade-offs

Approach 2: Host the Documents off Vault

How it works

The sync schedule requirement

Hosting

Trade-offs

Multi-Namespace Considerations

Conclusion

Appendix

How WIF Token Validation Works

Vault's OIDC URL Structure

Configuring New or Existing Keys

File Content Security Concerns

Adopting Terraform Ephemeral Resources

Scenarios to consider:

Scenario 1: Data sources with static secrets

Provider blocks

Write-only arguments

Scenario 2: Resources that receive a static secret

Scenario 3: Resources that generate a dynamic secret

Scenario 4: Resources or data sources that fetch generated secrets to store in another 3rd party system

Closing

Fully Migrate Secrets Out Of Terraform Module State Without Breaking Existing Users

Setup

Update

Legacy

New builds

Final thoughts

Writing Terraform Resources with Write-Only Parameters

What Are Write-Only Parameters?

Design Considerations

The _version Argument

Automatic Change Detection

Using Private State and Hashing

Best of both worlds

Handling Mode Switching

Key Takeaways

Making Terraform Workspaces in the same project

Setting a multi-line variable as an ENV in Terraform Cloud

Preamble

Solutions

Ephemeral Variables!

Base64 & TF_VAR_

Dynamic K8s Access

E1it3 Hax

Salutation

Deploy Cloudflare Tunnels with Terraform & Kubernetes

Access

Terraform

Kubernetes manifest:

Terraform state with native s3 locking

How?

Using the bucket

Wrapping up

Optional: Create bucket using Terraform

AWS Creds in the CLI via SSO

Problem

A better credential experience

Specify which profile I want and login:

Confirm the authorization request in my browser:

Setup

Summary

Implement Rollback with Terraform

Terraform: Variable validation with samples

Strings

String may not contain character

String with valid options

Valid AWS Region Name

Valid IAM Role Name

Valid IPv4 CIDR

Semantic Version

Maps

Map with optional conflicting keys

Numbers

The `_version` Argument