Demonstration of secrets rotation using a sample application written by Bob!
Introduction — What is Hashicorp Vault?
In the modern era of cloud-native applications, the “castle and moat” security model is dead. We no longer have a single perimeter to defend; instead, we have thousands of microservices, containers, and cloud resources that all need to talk to each other. To do that securely, they need credentials — API keys, database passwords, and TLS certificates.
HashiCorp Vault is the industry-standard solution for managing these “secrets.” At its core, Vault is an identity-based secret and encryption management system. Rather than hardcoding passwords into a configuration file (a major security risk) or storing them in plain text, developers use Vault to centrally store, access, and protect sensitive data.
Vault doesn’t just act as a digital safe, though. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log. But its most powerful feature — and the one that separates the pros from the amateurs — is its ability to handle Secret Rotation.
Why You Should Implement Secret Rotation?
Image from”https://developer.hashicorp.com/vault/docs/internals/rotation”
If Vault is the safe, Secret Rotation is the mechanism that automatically changes the combination at regular intervals. Here is why this isn’t just a “nice-to-have” feature, but a critical security requirement.
Drastically Reducing the “Blast Radius”
In a traditional setup, if a database password is leaked, it remains valid until a human manually changes it — which could be months. With rotation.
- The Scenario: An attacker gains access to a legacy log file containing a database credential.
- The Vault Defense: Because Bob set up a 24-hour rotation schedule (as shown in my demo), that credential is already useless by the time the attacker tries to use it. The “window of opportunity” for a breach is slammed shut automatically.
Achieving “Zero Trust” with Dynamic Secrets
Rotation allows you to move toward Dynamic Secrets. Instead of a static username like admin_user, Vault can generate a unique, short-lived user for every single request.
- The Benefit: If one microservice is compromised, the attacker only has access to a single, temporary credential that cannot be used to move laterally through your entire network.
Compliance Without the Headache
Regulators (SOC2, PCI-DSS, HIPAA) often mandate that secrets be rotated every 30, 60, or 90 days.
- Manual Way: A frantic week of manual updates and potential downtime every quarter.
- The “automatic vault” Way: Vault handles the handshake with the database or cloud provider, updates the secret, and logs the event. Compliance becomes a background process rather than a fire drill.
Eliminating the “Human Factor”
Humans are the weakest link in security. We forget to update documentation, we reuse passwords, and we leave credentials in Slack messages.
- Automation is Key: By using the Rotation Activity Log featured in our demo, you get an immutable record of every change. There’s no guessing if the rotation happened — the log proves it.
Rotation in Action: The Demo Breakdown
Based on the Vault Secret Rotation Demo (built with help from Bob), the application showcases a production-ready workflow for credential management. Here is how the internal architecture supports the security goals discussed above.
The “Heartbeat” (APScheduler & HVAC)
The application doesn’t just wait for a failure; it proactively manages the secret lifecycle.
- Technical Implementation: As seen in
app.py, the backend uses theAPSchedulerlibrary to trigger therotate_database_credentialsfunction.
"""
Vault Secret Rotation Demo Application
This Flask application demonstrates automatic secret rotation using HashiCorp Vault.
It rotates database credentials daily and displays the current secret information.
"""
import os
import logging
from datetime import datetime, timedelta
from flask import Flask, render_template, jsonify
import hvac
from apscheduler.schedulers.background import BackgroundScheduler
import psycopg2
from psycopg2 import OperationalError
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
app = Flask(__name__)
# Vault configuration
VAULT_ADDR = os.getenv('VAULT_ADDR', 'http://vault:8200')
VAULT_TOKEN = os.getenv('VAULT_TOKEN', 'root')
VAULT_NAMESPACE = os.getenv('VAULT_NAMESPACE', '')
DB_SECRET_PATH = os.getenv('DB_SECRET_PATH', 'database/creds/app-role')
# Application state
current_credentials = {
'username': None,
'password': None,
'last_rotation': None,
'next_rotation': None,
'rotation_count': 0
}
vault_client = None
def initialize_vault_client():
"""Initialize the Vault client with authentication."""
global vault_client
try:
vault_client = hvac.Client(
url=VAULT_ADDR,
token=VAULT_TOKEN,
namespace=VAULT_NAMESPACE
)
if vault_client.is_authenticated():
logger.info("Successfully authenticated with Vault")
return True
else:
logger.error("Failed to authenticate with Vault")
return False
except Exception as e:
logger.error(f"Error initializing Vault client: {e}")
return False
def rotate_database_credentials():
"""
Rotate database credentials by requesting new ones from Vault.
This simulates the secret rotation process.
"""
global current_credentials
try:
logger.info("Starting credential rotation...")
# Read new credentials from Vault
# In a real scenario, this would generate new dynamic credentials
response = vault_client.secrets.kv.v2.read_secret_version(
path='app/database',
mount_point='secret'
)
if response and 'data' in response and 'data' in response['data']:
data = response['data']['data']
# Update current credentials
old_username = current_credentials['username']
current_credentials['username'] = data.get('username')
current_credentials['password'] = data.get('password')
current_credentials['last_rotation'] = datetime.utcnow().isoformat()
current_credentials['next_rotation'] = (
datetime.utcnow() + timedelta(days=1)
).isoformat()
current_credentials['rotation_count'] += 1
logger.info(
f"Credentials rotated successfully. "
f"Old user: {old_username}, New user: {current_credentials['username']}"
)
# In a production environment, you would:
# 1. Update application database connections
# 2. Revoke old credentials
# 3. Test new credentials
return True
else:
logger.error("No data received from Vault")
return False
except Exception as e:
logger.error(f"Error rotating credentials: {e}")
return False
def test_database_connection():
"""
Test database connection with current credentials.
This is a simulation - adjust for your actual database.
"""
try:
# This is a mock test - in production, use actual DB connection
if current_credentials['username'] and current_credentials['password']:
logger.info("Database connection test: SUCCESS (simulated)")
return True
else:
logger.warning("Database connection test: FAILED - No credentials")
return False
except Exception as e:
logger.error(f"Database connection test failed: {e}")
return False
def initialize_secrets():
"""Initialize secrets on application startup."""
logger.info("Initializing secrets...")
# Create initial secret in Vault if it doesn't exist
try:
vault_client.secrets.kv.v2.create_or_update_secret(
path='app/database',
secret={
'username': f'app_user_{datetime.utcnow().strftime("%Y%m%d_%H%M%S")}',
'password': f'secure_pass_{os.urandom(8).hex()}',
'host': 'postgres',
'port': '5432',
'database': 'appdb'
},
mount_point='secret'
)
logger.info("Initial secret created in Vault")
except Exception as e:
logger.warning(f"Could not create initial secret: {e}")
# Perform initial rotation
rotate_database_credentials()
@app.route('/')
def index():
"""Main page showing current secret status."""
return render_template('index.html')
@app.route('/api/status')
def get_status():
"""API endpoint to get current secret rotation status."""
return jsonify({
'vault_connected': vault_client.is_authenticated() if vault_client else False,
'current_user': current_credentials['username'],
'last_rotation': current_credentials['last_rotation'],
'next_rotation': current_credentials['next_rotation'],
'rotation_count': current_credentials['rotation_count'],
'db_connection_ok': test_database_connection()
})
@app.route('/api/rotate', methods=['POST'])
def manual_rotate():
"""API endpoint to manually trigger credential rotation."""
success = rotate_database_credentials()
return jsonify({
'success': success,
'message': 'Rotation completed' if success else 'Rotation failed',
'current_user': current_credentials['username'],
'rotation_count': current_credentials['rotation_count']
})
@app.route('/health')
def health():
"""Health check endpoint for Kubernetes."""
vault_ok = vault_client.is_authenticated() if vault_client else False
db_ok = test_database_connection()
if vault_ok and db_ok:
return jsonify({'status': 'healthy'}), 200
else:
return jsonify({
'status': 'unhealthy',
'vault': vault_ok,
'database': db_ok
}), 503
@app.route('/ready')
def ready():
"""Readiness check endpoint for Kubernetes."""
if current_credentials['username'] is not None:
return jsonify({'status': 'ready'}), 200
else:
return jsonify({'status': 'not ready'}), 503
def setup_scheduler():
"""Setup the background scheduler for automatic rotation."""
scheduler = BackgroundScheduler()
# Schedule rotation every 24 hours
scheduler.add_job(
func=rotate_database_credentials,
trigger='interval',
hours=24,
id='rotate_credentials',
name='Rotate database credentials',
replace_existing=True
)
# For demo purposes, also schedule every 5 minutes
scheduler.add_job(
func=rotate_database_credentials,
trigger='interval',
minutes=5,
id='rotate_credentials_demo',
name='Rotate credentials (demo - every 5 min)',
replace_existing=True
)
scheduler.start()
logger.info("Scheduler started - credentials will rotate every 24 hours (and every 5 minutes for demo)")
# Initialize application on module load (for gunicorn)
logger.info("Starting Vault Secret Rotation Demo Application")
# Initialize Vault client
if initialize_vault_client():
# Initialize secrets
initialize_secrets()
# Setup automatic rotation scheduler
setup_scheduler()
logger.info("Application initialized successfully")
else:
logger.error("Failed to initialize Vault client")
if __name__ == '__main__':
# Start Flask application directly (for development)
app.run(host='0.0.0.0', port=5000, debug=False)
# Made with Bob
- The Cadence: While production secrets might rotate every 24 hours, our demo is configured to rotate every 5 minutes to showcase the automation in real-time.
Orchestration (The Kubernetes Layer)
The demo is built to run in a Kubernetes (Minikube) environment, ensuring the application and Vault are isolated but communicative.
- Vault Pod: Runs the Vault Server (Port
8200) using a KV v2 Engine. - Application Pod: A Flask-based container that uses the hvac Python SDK to communicate with Vault over the internal cluster network.
Real-Time Observability
The frontend (index.html) provides a dashboard that visualizes the "invisible" work Vault is doing:
- Connection Status: Confirms the Flask app can still reach the database using the latest rotated secret.
- Current Credentials: Displays the current active username (e.g.,
app_user_[timestamp]), proving that the credentials are no longer static. - Rotation Activity Log: An event feed that captures every manual and scheduled rotation, providing a clear audit trail of who (or what) changed a secret and when.
Forcing the update!
Vault Admin Interface
The vault admin is accessible through the interface !
Infrastructure-as-Code
The entire setup is portable and reproducible. Using Podman and Kubernetes manifests, the environment can be spun up or shut down (via cleanup.sh) in seconds, ensuring that security configurations are treated with the same rigor as application code.
⚠️ A Note on this Demonstration
While this project provides a comprehensive look at the automation workflow and Vault integration, it is important to note that the database functionality in this specific demo is a simulated mockup. In a production environment, Vault would interface directly with a live database — such as PostgreSQL, MySQL, or MongoDB — using its dedicated Database Secrets Engine.
This architectural choice allows the demo to focus entirely on the “handshake” between the application and Vault, proving the rotation logic works seamlessly before you move to a complex, real-world data layer.
Conclusion: From Vulnerable to Verifiable
Implementing secret rotation is the difference between reactive and proactive security. By using HashiCorp Vault and the patterns demonstrated in Bob’s demo, you can ensure that your most sensitive credentials are never static, never stale, and always protected.
In this demonstration, we’ve moved beyond theory to a functioning proof-of-concept where:
- Automation Replaces Manual Labor: Bob implemented APScheduler to handle the heavy lifting, ensuring that the lifecycle of a secret is governed by code, not human memory.
- Observability provides Confidence: Through a real-time dashboard and server-sent events, the demonstration transformed a “black box” security process into a transparent, auditable stream of activity.
- Infrastructure is Code: By leveraging Podman and Kubernetes, Bob created a portable environment where security policies are versioned and deployed alongside the application itself.
By adopting these patterns, you aren’t just “changing passwords” — you are building a self-healing security architecture that assumes credentials will be targeted and ensures that, even if they are, their utility to an attacker is practically zero.
>>> Thanks for reading <<<
Links and resources
- Hashicorp Vault: https://www.hashicorp.com/fr/products/vault
- Vault developer documenation: https://developer.hashicorp.com/vault
- Key Rotation: https://developer.hashicorp.com/vault/docs/internals/rotation
- Vault Github: https://github.com/hashicorp/vault
- Blog post code: https://github.com/aairom/vault-security-rotation/tree/main
- IBM Project Bob: https://www.ibm.com/products/bob (stay tuned, Bob’s GA is coming 😉
Top comments (0)