Vaulting Beyond Defaults: The Evolution of Secrets Management on Red Hat OpenShift

Why using Vault is better to manage your secrets?

In modern containerized infrastructures, managing sensitive data like database passwords, API tokens, and private keys is one of the most critical aspects of day-to-day operations. When moving to Kubernetes and Red Hat OpenShift, understanding the default mechanisms — and where they fall short for enterprise security — is the first step toward building a truly secure by design architecture.

💡 Note from the Field: Before we jump into the post, all code, deployment manifests, and architectural implementation logic featured throughout this guide were designed, orchestrated, and automated by Bob (IBM Bob SDLC).

🤓

---

The Baseline: Secrets Management in Kubernetes

In vanilla Kubernetes, the native Secret object provides a way to separate sensitive data from application code or standard configurations (ConfigMaps). However, the word "Secret" can be misleading for newcomers:

1. Base64 Encoding is Not Encryption: By default, Kubernetes secrets are stored as flat, base64-encoded strings within the cluster data store. Base64 is a data transformation format designed to handle binary data over text-based networks; it provides zero obfuscation or security. Anyone with access to the YAML declaration or read permissions to the API can immediately reverse the encoding (echo "bXktcGFzc3dvcmQ=" | base64 -d).
2. The etcd Vulnerability: In a standard Kubernetes control plane, unless specifically configured otherwise, these base64 strings are stored in plain text inside the etcd database. If an attacker gains access to the underlying master host storage or backups of etcd, the entire cluster's credentials are compromised.
3. Coarse-Grained Access Control: Kubernetes Roles and RoleBindings restrict access at the resource level, but managing fine-grained programmatic access to specific keys within a single namespace becomes complex and error-prone.

The Upgraded Standard: How OpenShift Secures Native Secrets

Red Hat OpenShift takes the fundamental concepts of Kubernetes and hardens them to meet enterprise compliance and security baselines out-of-the-box.

• Mandatory Multi-Tenancy via Projects: OpenShift extends Kubernetes namespaces into Projects. Projects enforce strict multi-tenancy from the start, isolating network traffic, service accounts, and resource limits (ResourceQuotas) so that an application team cannot accidentally view or cross-contaminate another team's secrets.

• Automated etcd Encryption at Rest: Unlike vanilla Kubernetes where encryption must be manually configured via an encryption configuration file, OpenShift enables cluster-wide etcd encryption at rest smoothly through its cluster operators. When enabled, secret resources are automatically encrypted using robust cryptographic algorithms (such as AES-256-GCM) before hit-to-disk inside etcd.

• Strict Security Context Constraints (SCCs): OpenShift mitigates runtime risks by ensuring pods run under rigorous security rules. Applications cannot blindly run as root or mount forbidden host directories, preventing compromised containers from harvesting cluster tokens or local secrets files directly from the underlying node storage.

While OpenShift native secrets are significantly more robust than vanilla Kubernetes, enterprise security teams often demand zero-trust paradigms, live credential rotation without downtime, centralized audit trails, and elimination of sensitive data from Git repositories entirely. This is where external secrets managers like HashiCorp Vault come into play.

---

Chapter 1: The Classical Approach — OpenShift Native Secrets (Approach A)

To truly appreciate the advantages of a dynamic secrets manager like HashiCorp Vault, we must first analyze the reference architecture, implementation, and operational lifecycle of the classical, platform-native approach.

High-Level Architecture

In this architecture, the cluster relies entirely on OpenShift’s built-in control plane. Secrets are submitted via the OpenShift API, committed encrypted into etcd, and then bound to the application pod via environment variables or read-only volume mounts.

Step-by-Step Reference Implementation

Declaring the Hardened Project and RBAC Boundaries

First, we establish a clean, production-ready namespace coupled with a dedicated ServiceAccount and explicit Role bindings utilizing the principle of least privilege.

# 00-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: demo
  labels:
    environment: production
    secrets-approach: native
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: demo-quota
  namespace: demo
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "20"

# 01-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-sa
  namespace: demo
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: demo
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: demo-sa-secret-reader
  namespace: demo
subjects:
  - kind: ServiceAccount
    name: demo-sa
    namespace: demo
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Defining the Platform Secrets

Next, we define our credentials using native Secret configurations. Notice that values are raw base64 placeholders.

# 02-secrets.yaml (Excerpt)
apiVersion: v1
kind: Secret
metadata:
  name: demo-db-secret
  namespace: demo
  annotations:
    secrets.management/rotation-interval: "90d"
    secrets.management/last-rotated: "2026-01-01"
type: Opaque
data:
  DB_USER: ZGVtb191c2Vy                         # demo_user
  DB_PASSWORD: CHANGEME_BASE64_ENCODED_PASSWORD==
  DB_HOST: cG9zdGdyZXMuZGVtby5zdmMuY2x1c3Rlci5sb2NhbA==
  DB_PORT: NTQzMg==                             # 5432
  DB_NAME: ZGVtb2Ri                             # demodb

Injecting Secrets into the Application Deployment

The application deployment can consume these secrets in two distinct formats: as environment variables mapped straight into the container’s environment via envFrom, or mounted explicitly as read-only files through standard volumes.

# 04-deployment.yaml (Configuration Excerpt)
spec:
  template:
    spec:
      serviceAccountName: demo-sa
      containers:
        - name: demo-app
          image: image-registry.openshift-image-registry.svc:5000/demo/demo-app:latest
          envFrom:
            - configMapRef:
                name: demo-config
            - secretRef:
                name: demo-db-secret
            - secretRef:
                name: demo-admin-secret
          volumeMounts:
            - name: ldap-secret-vol
              mountPath: /secrets/ldap
              readOnly: true
      volumes:
        - name: ldap-secret-vol
          secret:
            secretName: demo-ldap-secret

How the Application Consumes Native Secrets?

From an application development perspective, using environment-injected native secrets is incredibly simple. There are no specialized APIs, external packages, or external client initializations required. The application merely reads from its operating system context.

# app.py (Excerpt)
import os
from flask import Flask, jsonify

app = Flask(__name__)

class Config:
    SECRET_SOURCE = "environment variables (OpenShift native Secret → envFrom)"

    # The platform injected the value into the process environment
    # before the application process started execution.
    DB_USER     = os.environ.get("DB_USER",     "not-set")
    DB_PASSWORD = os.environ.get("DB_PASSWORD", "not-set")
    DB_HOST     = os.environ.get("DB_HOST",     "not-set")
    DB_PORT     = os.environ.get("DB_PORT",     "5432")
    DB_NAME     = os.environ.get("DB_NAME",     "not-set")

@app.route("/config")
def config():
    return jsonify({
        "approach": "A — OpenShift Native Secrets",
        "secret_source": Config.SECRET_SOURCE,
        "loaded_at": "process start (env vars set by Kubelet before exec)",
        "rotation_note": "Pod RESTART required to pick up a new secret value"
    })

The Operational Reality of Native Secrets

While this appraoch is easy to grasp, has no external system requirements, and operates out-of-the-box in air-gapped data centers, it creates massive friction under enterprise conditions:

The Rotation Problem

When security regulations require quarterly or emergency credential changes, the operations team updates the database engine’s backend password, updates the target Kubernetes Secret object, and must perform a mandatory rolling restart of the application deployments. Because environment variables are bound to the process execution stack at launch time, the containers cannot dynamically fetch the changes, risking service disruption during the rollout window.

The GitOps Paradox

In contemporary workflows, configurations are version-controlled using GitOps engines (like ArgoCD). However, storing raw OpenShift native secret manifests inside Git exposes plaintext credentials to unauthorized auditors or pipeline engineers. Teams must rely on secondary toolsets like Bitnami SealedSecrets. While kubeseal securely encrypts the manifests using asymmetric key pairs before pushing them to public repositories, it introduces an administrative burden—managing cryptographic keys, handling cluster reinstallation decryption failures, and onboarding third-party configurations.

In the next chapter, we will look at A*pproach B: HashiCorp Vault + Vault Agent Injector*, exploring how a specialized identity-driven architecture resolves these operational limitations entirely without rewriting a single line of application code.

---

Chapter 2: The Modern Alternative — Identity-Driven Secrets with HashiCorp Vault (Approach B)

To overcome the operational limitations of static, platform-bound credentials, modern enterprise architectures transition to an identity-driven security model using HashiCorp Vault.

In this setup, secrets no longer live inside OpenShift’s native etcd storage database, nor are they ever checked into a Git repository. Instead, OpenShift becomes a platform consumer, leveraging its internal cryptographic identities (ServiceAccounts) to request short-lived tokens from a centralized, highly auditable Vault cluster.

Deploying Vault on Red Hat OpenShift

Deploying Vault on OpenShift requires specialized configurations to accommodate strict platform rules, such as Security Context Constraints (SCCs), multi-replica high availability, and native routing. The standard deployment standard utilizes the official Helm chart customized with OpenShift-native parameters.

Vault Cluster Configuration

Below is the reference configuration blueprint using Raft integrated storage (eliminating the operational footprint of external storage backends like Consul):

# vault-values.yaml
global:
  enabled: true
  tlsDisable: false    # Forced TLS for all transit lines
  openshift: true      # Automatically tunes Helm chart to handle OpenShift compliance and SCCs

server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable     = 0
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path    = "/vault/data"
          node_id = "HOSTNAME"

          # Auto-join configuration for Raft quorum consensus
          retry_join { leader_api_addr = "http://vault-0.vault-internal:8200" }
          retry_join { leader_api_addr = "http://vault-1.vault-internal:8200" }
          retry_join { leader_api_addr = "http://vault-2.vault-internal:8200" }
        }

Bootstrapping Identity-Based Authentication

Once the Vault instance is running on the cluster, we leverage Vault’s native Kubernetes Authentication Method. This allows applications to use their standard OpenShift ServiceAccount token as a secure, temporary proof of identity.

The configuration workflow establishes a secure cryptographic trust loop between Vault and the OpenShift Control Plane API:

Scripted Authentication Setup

The following reference script automates the creation of the auth backend, defines the system boundaries, and maps permissions directly to the target namespace service account:

#!/usr/bin/env bash
# 01-vault-k8s-auth.sh (Excerpt)
set -euo pipefail

echo "=== Configuring Vault Kubernetes Auth for Demo App ==="

# 1. Enable the Kubernetes auth engine
vault auth enable kubernetes || echo "Already enabled"

# 2. Point Vault to the host OpenShift API server for token verification
vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# 3. Create a strict role matching the application's runtime platform identity
vault write auth/kubernetes/role/demo-role \
    bound_service_account_names="demo-sa" \
    bound_service_account_namespaces="demo" \
    policies="demo-policy" \
    ttl="1h" \
    max_ttl="24h"

Defining Fine-Grained ACL Rules

To adhere to the principle of least privilege, applications must never be handed a broad root token. Vault uses HashiCorp Configuration Language (HCL) policies to enforce strict path isolation. The demo-app is restricted entirely to its own data engine space:

# 00-vault-policy.hcl
# Allow the demo application read-only access to its dedicated database & infrastructure credentials
path "secret/data/demo" {
  capabilities = ["read"]
}

# Allow reading metadata details (e.g. tracking versions for rotation) without exposing values
path "secret/metadata/demo" {
  capabilities = ["list", "read"]
}

# Allow token renewal and self-inspection capabilities (crucial for automated sidecars)
path "auth/token/renew-self" {
  capabilities = ["update"]
}
path "auth/token/lookup-self" {
  capabilities = ["read"]
}

Injecting Secrets securely via the Key-Value Engine

With policies and authentication boundaries active, parameters can be loaded straight into the Vault Secure KV Store. Because this operates entirely through the API or CLI, configurations are maintained out-of-band — safely isolating them from human eyes and source repositories.

#!/usr/bin/env bash
# 02-vault-secrets.sh (Excerpt)
source .env # Local isolated parameters, ignored by Git version control

echo "=== Committing App Secrets to Vault Secure KV Store ==="

# Write all credentials atomically into Version 2 KV backend
vault kv put secret/demo \
    db_user="${APP_DB_USER}" \
    db_password="${APP_DB_PASSWORD}" \
    db_host="${APP_DB_HOST}" \
    db_port="${APP_DB_PORT}" \
    db_name="${APP_DB_NAME}" \
    ldap_url="${APP_LDAP_URL}"

echo "✅ Master credentials stored. These are completely absent from OpenShift's etcd."

In the next sections, we will explore how the Vault Agent Injector seamlessly introduces these paths directly to the pods as ephemeral files, allowing applications to stay zero-trust compliant without writing complex API clients or modification blocks.

---

Chapter 3: Implementation — Decoupled Vault Injection (Approach B)

Now that HashiCorp Vault is deployed and trusting OpenShift’s identities, we need to bridge the secrets into our application. The gold standard for this without rewriting application code is the Vault Agent Injector.

The Vault Agent Injector is a Mutating Admission Webhook. It monitors incoming pod specifications for specific annotations. When it spots them, it dynamically alters the Pod deployment manifest by embedding two specific platform sidecars into your application stack:

• vault-agent-init (Init Container): Runs before your application container executes. It hits Vault using the pod's ServiceAccount identity, downloads the requested secrets, formats them, and writes them to a shared, high-speed ephemeral memory storage (tmpfs) volume mounted at /vault/secrets/.
• vault-agent (Runtime Sidecar Container): Runs alongside your application indefinitely. It monitors Vault for updates and automatically rewrites the temporary workspace file on the fly if a secret value is rotated.

High-Level Architecture Flow

This setup decouples the application from Vault completely. The application code only needs to interact with a standard local filesystem file, remaining entirely unaware of Vault’s underlying REST APIs or authentication loops.

Manifest Configuration: Injecting Secrets via Annotations

To apply this pattern, your OpenShift deployment manifest doesn’t configure any native Kubernetes Secret values. Instead, you define annotations that guide the webhook engine.

# 04-deployment-with-vault.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: demo
  labels:
    app.kubernetes.io/name: demo-app
    secrets-approach: vault-injector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
      annotations:
        # 1. Activate the injector webhook engine
        vault.hashicorp.com/agent-inject: "true"

        # 2. Specify the authentication role mapped inside Vault
        vault.hashicorp.com/role: "demo-role"

        # 3. Request secret paths and define the output filenames
        vault.hashicorp.com/agent-inject-secret-db.env: "secret/data/demo"
        vault.hashicorp.com/agent-inject-secret-ldap.env: "secret/data/demo"

        # 4. Define custom Go-Template structures to format the data cleanly for consumption
        vault.hashicorp.com/agent-inject-template-db.env: |
          {{- with secret "secret/data/demo" -}}
          export DB_USER="{{ .Data.data.db_user }}"
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          export DB_HOST="{{ .Data.data.db_host }}"
          export DB_PORT="{{ .Data.data.db_port }}"
          export DB_NAME="{{ .Data.data.db_name }}"
          {{- end -}}

        vault.hashicorp.com/agent-inject-template-ldap.env: |
          {{- with secret "secret/data/demo" -}}
          export LDAP_URL="{{ .Data.data.ldap_url }}"
          {{- end -}}
    spec:
      serviceAccountName: demo-sa
      containers:
        - name: demo-app
          image: image-registry.openshift-image-registry.svc:5000/demo/demo-app:latest
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: demo-config
          # Enforce runtime constraints
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1001
            capabilities:
              drop: ["ALL"]

Code Implementation: Reading secrets dynamically from Python

Because the sidecar drops standard configuration shell files into the ephemeral memory directory, reading them becomes a direct file parsing operation.

Below is how the application logic manages both initialization and dynamic monitoring loops to ingest updates without process restarts:

# app.py (Excerpt for Vault Approach B)
import os
import re
from pathlib import Path
from flask import Flask, jsonify

app = Flask(__name__)

# Defaults to the injector standard directory path
VAULT_SECRETS_PATH = os.environ.get("VAULT_SECRETS_PATH", "/vault/secrets")

class VaultConfig:
    SECRET_SOURCE = "Vault Agent Injector (Shared tmpfs Volume)"

    @staticmethod
    def parse_env_file(filename: str):
        """Parses a Go-templated export file into a dictionary format."""
        secrets = {}
        filepath = Path(VAULT_SECRETS_PATH) / filename

        if not filepath.exists():
            return secrets

        # Match pattern: export KEY="VALUE"
        pattern = re.compile(r'^\s*export\s+([A-Z0-9_]+)\s*=\s*"(.*)"\s*$')
        with open(filepath, "r") as f:
            for line in f:
                match = pattern.match(line)
                if match:
                    key, value = match.groups()
                    secrets[key] = value
        return secrets

    # Dynamic property mapping definitions
    @property
    def DB_PASSWORD(self):
        # Always read fresh from the filesystem to fetch hot rotations on the fly
        secrets = self.parse_env_file("db.env")
        return secrets.get("DB_PASSWORD", "not-found")

    @property
    def DB_USER(self):
        secrets = self.parse_env_file("db.env")
        return secrets.get("DB_USER", "not-found")

    @property
    def DB_HOST(self):
        secrets = self.parse_env_file("db.env")
        return secrets.get("DB_HOST", "localhost")

config_manager = VaultConfig()

@app.route("/config")
def config():
    return jsonify({
        "approach": "B — Vault Token Integration",
        "secret_source": VaultConfig.SECRET_SOURCE,
        "loaded_at": "On-demand filesystem reads (Hot-swappable dynamically)",
        "rotation_note": "Zero downtime or rollouts required during backend password shifts."
    })

@app.route("/db-status")
def db_status():
    return jsonify({
        "db_host": config_manager.DB_HOST,
        "db_user": config_manager.DB_USER,
        # Display masked password to prove it loaded successfully
        "db_password_used": f"{config_manager.DB_PASSWORD[:2]}******" if len(config_manager.DB_PASSWORD) > 2 else "short"
    })

The Lifecycle Revolution: Direct Structural Comparisons

| Security Lifecycle           | Approach A: Native OpenShift Secrets                         | Approach B: Centralized HashiCorp Vault                      |
| ---------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
| **Storage Engine Location**  | Standard plain `etcd` Key-Value database.                    | Highly decoupled, encrypted Raft storage layer.              |
| **GitOps/CI-CD Handling**    | Requires third-party tools like `SealedSecrets` or encryption wrappers. | Manifests contain nothing but metadata annotations. Secure parameters are injected entirely out-of-band. |
| **Credential Rotation Loop** | Requires updating objects and performing a rolling deployment update. | The runtime sidecar modifies the local shared volume. Your application picks it up instantly without restarts. |
| **Audit Capabilities**       | Shared cluster-wide. Difficult to isolate individual container tracking parameters cleanly. | Fine-grained, immutable tracking policies. Records every read, renewal, and authentication event instantly. |

By implementing identity-driven secrets through HashiCorp Vault and utilizing the automated sidecar injector on Red Hat OpenShift, engineering teams achieve true zero-trust execution environments. You eliminate the leakage vectors inherent to version control repositories, decouple configurations from infrastructure databases, and enable seamless credential rotation — all without polluting your software assets with complex third-party SDK dependencies.

---

Conclusion: Elevating OpenShift Native Defense with Zero-Trust Automation

Red Hat OpenShift’s built-in secrets management sets an impressive enterprise baseline. By automating etcd encryption at rest out of the box and enforcing rigid boundaries through multi-tenant Projects and Security Context Constraints, it fixes the glaring security gaps found in vanilla Kubernetes.

However, enterprise governance demands more than just hardened storage; it requires dynamic agility. Transitioning to HashiCorp Vault elevates your security posture from a robust static platform to a proactive, identity-driven ecosystem. Even though OpenShift natively keeps data highly secure on disk, Vault solves the operational overhead of the application lifecycle. By eliminating the GitOps paradox of managing encrypted manifests in repositories, introducing granular, centralized audit logs for strict compliance tracking, and streaming live credential rotations to application containers without triggering disruptive rolling updates, Vault turns a complex security checklist into seamless, zero-trust automation.

| Feature / Security Lifecycle  | ☸️ Standard Kubernetes                                        | 🔴 Standard OpenShift                                         | 🔒 OpenShift + HashiCorp Vault                                |
| ----------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| **Storage Engine & Security** | Stored in `etcd`. **Not encrypted at rest by default** (requires manual KMS provider configuration). base64 encoding only. | Stored in `etcd`. **Encrypted at rest by default** out of the box using AES-256-GCM cryptography. | Stored in highly decoupled, dedicated **Vault Raft storage**. Secrets never land in the cluster’s `etcd` layer. |
| **GitOps & Version Control**  | Plain YAML manifests contain secrets in clear text (base64). Risks repo leakage without 3rd party tools (e.g., `SealedSecrets`). | Same GitOps limitation as Kubernetes; manifests require client-side encryption before committing to git. | **Zero-secret manifests.** Deployment manifests contain only metadata annotations; actual values are handled out-of-band. |
| **Runtime Pod Delivery**      | Injected directly into the container as environment variables or mounted files via standard Kubelet API. | Injected similarly via environment variables or volume mounts, strictly governed by platform **Security Context Constraints (SCC)**. | Injected transparently into a shared, high-speed ephemeral memory storage (`tmpfs` volume) via a **Mutating Webhook Sidecar**. |
| **Credential Rotation Loop**  | Manual or custom scripting. Requires a rolling update deployment to restart pods and pick up new values. | Native rotation still requires object modification and a rolling deployment restart (`oc rollout restart`) to flush stale values. | **Dynamic & Hot-Swappable.** The `vault-agent` sidecar live-streams new values into the pod filesystem without restarting the container. |
| **Audit Capabilities**        | Basic cluster auditing. Difficult to trace exact API read requests to specific container parameters. | Strong platform-wide auditing via standard OpenShift cluster logging and cluster-level audit trails. | **Granular, cryptographic audit logs.** Tracks every authentication, lookup, token renewal, and precise key access. |
| **Least Privilege Isolation** | Scoped broadly by Namespace RBAC boundaries.                 | Scoped tightly by multi-tenant Projects and hardened Security Context Constraints (SCCs). | Scoped mathematically by fine-grained **Vault HCL Policies** mapped directly to individual `ServiceAccount` identities. |

Thanks for reading 🔏

Links

• Hasicorp Vault: https://www.hashicorp.com/en/products/vault
• Hashicorp Vault Github: https://github.com/hashicorp/vault
• Sample code repository of this post: https://github.com/aairom/openshift-vault-post