Bulletproofing Bob: An End-to-End Deep Dive into a cloud-native application with Terraform, Ansible, Vault, and OpenTelemetry!
Introduction — Mastering the DevSecOps Lifecycle with “Bob”
The application showcases a complete end-to-end, cloud-native ecosystem where every stage of the application lifecycle — from infrastructure provisioning to real-time monitoring — is automated, secure, and observable, implemented and document by Bob, IBM’s SDLC.
Member Management Application
The core
The Member Management Application (affectionately dubbed “Bob”). Showcasing that in modern software, “it works on my machine” isn’t enough. We need a system that is:
- Self-Healing & Scalable (Kubernetes/Docker).
- Provisioned by Code (Terraform).
- Configured Automatically (Ansible).
- Fortified by Design (HashiCorp Vault).
- Last but not least, observable by design (OpenTelemetry).
- Press enter or click to view image in full size
from flask import Flask, render_template, request, jsonify, redirect, url_for
import sqlite3
import os
from datetime import datetime
import hashlib
import hvac
import re
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlite3 import SQLite3Instrumentor
# Initialize OpenTelemetry
resource = Resource.create({"service.name": "member-management-app"})
# Trace provider
trace_provider = TracerProvider(resource=resource)
otlp_trace_exporter = OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
insecure=True
)
trace_provider.add_span_processor(BatchSpanProcessor(otlp_trace_exporter))
trace.set_tracer_provider(trace_provider)
# Metrics provider
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
insecure=True
)
)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
# Initialize Flask app
app = Flask(__name__)
app.config['SECRET_KEY'] = os.getenv('FLASK_SECRET_KEY', 'dev-secret-key')
# Instrument Flask and SQLite
FlaskInstrumentor().instrument_app(app)
SQLite3Instrumentor().instrument()
# Get tracer and meter
tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)
# Create metrics
member_counter = meter.create_counter(
"member.operations",
description="Number of member operations",
unit="1"
)
# Database configuration
DATABASE = os.getenv('DATABASE_PATH', 'members.db')
# Vault configuration
VAULT_ADDR = os.getenv('VAULT_ADDR', 'http://localhost:8200')
VAULT_TOKEN = os.getenv('VAULT_TOKEN', 'dev-token')
VAULT_MOUNT_POINT = os.getenv('VAULT_MOUNT_POINT', 'secret')
def get_vault_client():
"""Initialize and return Vault client"""
try:
client = hvac.Client(url=VAULT_ADDR, token=VAULT_TOKEN)
if not client.is_authenticated():
print("Warning: Vault authentication failed")
return None
return client
except Exception as e:
print(f"Warning: Could not connect to Vault: {e}")
return None
def store_secret_in_vault(username, secret_type, secret_value):
"""Store a secret in Vault"""
with tracer.start_as_current_span("store_secret_in_vault"):
client = get_vault_client()
if client:
try:
path = f"{VAULT_MOUNT_POINT}/data/members/{username}/{secret_type}"
client.secrets.kv.v2.create_or_update_secret(
path=f"members/{username}/{secret_type}",
secret=dict(value=secret_value),
mount_point=VAULT_MOUNT_POINT
)
return True
except Exception as e:
print(f"Error storing secret in Vault: {e}")
return False
return False
def get_secret_from_vault(username, secret_type):
"""Retrieve a secret from Vault"""
with tracer.start_as_current_span("get_secret_from_vault"):
client = get_vault_client()
if client:
try:
path = f"members/{username}/{secret_type}"
secret = client.secrets.kv.v2.read_secret_version(
path=path,
mount_point=VAULT_MOUNT_POINT
)
return secret['data']['data']['value']
except Exception as e:
print(f"Error retrieving secret from Vault: {e}")
return None
return None
def hash_password(password):
"""Hash password using SHA-256"""
return hashlib.sha256(password.encode()).hexdigest()
def get_db_connection():
"""Create database connection"""
conn = sqlite3.connect(DATABASE)
conn.row_factory = sqlite3.Row
return conn
def init_db():
"""Initialize database with schema and sample data"""
with tracer.start_as_current_span("init_db"):
conn = get_db_connection()
cursor = conn.cursor()
# Create table
cursor.execute('''
CREATE TABLE IF NOT EXISTS members (
id INTEGER PRIMARY KEY AUTOINCREMENT,
last_name TEXT NOT NULL,
first_name TEXT NOT NULL,
date_of_birth TEXT NOT NULL,
username TEXT UNIQUE NOT NULL,
password_hash TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
''')
# Check if we need to add sample data
cursor.execute('SELECT COUNT(*) FROM members')
count = cursor.fetchone()[0]
if count == 0:
# Insert 5 sample records
sample_members = [
('Smith', 'John', '1985-03-15', 'jsmith', 'password123', 'secret_data_1'),
('Johnson', 'Emily', '1990-07-22', 'ejohnson', 'password456', 'secret_data_2'),
('Williams', 'Michael', '1988-11-30', 'mwilliams', 'password789', 'secret_data_3'),
('Brown', 'Sarah', '1992-05-18', 'sbrown', 'passwordabc', 'secret_data_4'),
('Davis', 'Robert', '1987-09-25', 'rdavis', 'passwordxyz', 'secret_data_5')
]
for last_name, first_name, dob, username, password, secret in sample_members:
password_hash = hash_password(password)
cursor.execute('''
INSERT INTO members (last_name, first_name, date_of_birth, username, password_hash)
VALUES (?, ?, ?, ?, ?)
''', (last_name, first_name, dob, username, password_hash))
# Store secret in Vault
store_secret_in_vault(username, 'secret', secret)
conn.commit()
conn.close()
@app.route('/')
def index():
"""Home page - list all members with optional search"""
with tracer.start_as_current_span("index"):
search_query = request.args.get('search', '').strip()
conn = get_db_connection()
if search_query:
# Convert wildcard pattern to SQL LIKE pattern
# Replace * with % for SQL LIKE
sql_pattern = search_query.replace('*', '%')
# Search in last_name, first_name, and username
members = conn.execute('''
SELECT * FROM members
WHERE last_name LIKE ?
OR first_name LIKE ?
OR username LIKE ?
ORDER BY last_name, first_name
''', (sql_pattern, sql_pattern, sql_pattern)).fetchall()
member_counter.add(1, {"operation": "search"})
else:
members = conn.execute('SELECT * FROM members ORDER BY last_name, first_name').fetchall()
member_counter.add(1, {"operation": "list"})
conn.close()
return render_template('index.html', members=members, search_query=search_query)
@app.route('/member/<int:member_id>')
def view_member(member_id):
"""View member details"""
with tracer.start_as_current_span("view_member"):
conn = get_db_connection()
member = conn.execute('SELECT * FROM members WHERE id = ?', (member_id,)).fetchone()
conn.close()
if member is None:
return "Member not found", 404
# Retrieve secret from Vault
secret = get_secret_from_vault(member['username'], 'secret')
member_counter.add(1, {"operation": "view"})
return render_template('view_member.html', member=member, secret=secret)
@app.route('/member/new', methods=['GET', 'POST'])
def create_member():
"""Create new member"""
with tracer.start_as_current_span("create_member"):
if request.method == 'POST':
last_name = request.form['last_name']
first_name = request.form['first_name']
date_of_birth = request.form['date_of_birth']
username = request.form['username']
password = request.form['password']
secret = request.form.get('secret', '')
password_hash = hash_password(password)
conn = get_db_connection()
try:
cursor = conn.cursor()
cursor.execute('''
INSERT INTO members (last_name, first_name, date_of_birth, username, password_hash)
VALUES (?, ?, ?, ?, ?)
''', (last_name, first_name, date_of_birth, username, password_hash))
conn.commit()
# Store secret in Vault
if secret:
store_secret_in_vault(username, 'secret', secret)
member_counter.add(1, {"operation": "create"})
return redirect(url_for('index'))
except sqlite3.IntegrityError:
return "Username already exists", 400
finally:
conn.close()
return render_template('create_member.html')
@app.route('/member/<int:member_id>/edit', methods=['GET', 'POST'])
def edit_member(member_id):
"""Edit member"""
with tracer.start_as_current_span("edit_member"):
conn = get_db_connection()
member = conn.execute('SELECT * FROM members WHERE id = ?', (member_id,)).fetchone()
if member is None:
conn.close()
return "Member not found", 404
if request.method == 'POST':
last_name = request.form['last_name']
first_name = request.form['first_name']
date_of_birth = request.form['date_of_birth']
password = request.form.get('password')
secret = request.form.get('secret')
if password:
password_hash = hash_password(password)
conn.execute('''
UPDATE members
SET last_name = ?, first_name = ?, date_of_birth = ?, password_hash = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?
''', (last_name, first_name, date_of_birth, password_hash, member_id))
else:
conn.execute('''
UPDATE members
SET last_name = ?, first_name = ?, date_of_birth = ?, updated_at = CURRENT_TIMESTAMP
WHERE id = ?
''', (last_name, first_name, date_of_birth, member_id))
conn.commit()
# Update secret in Vault if provided
if secret:
store_secret_in_vault(member['username'], 'secret', secret)
conn.close()
member_counter.add(1, {"operation": "update"})
return redirect(url_for('view_member', member_id=member_id))
# Retrieve secret from Vault
secret = get_secret_from_vault(member['username'], 'secret')
conn.close()
return render_template('edit_member.html', member=member, secret=secret)
@app.route('/member/<int:member_id>/delete', methods=['POST'])
def delete_member(member_id):
"""Delete member"""
with tracer.start_as_current_span("delete_member"):
conn = get_db_connection()
member = conn.execute('SELECT username FROM members WHERE id = ?', (member_id,)).fetchone()
if member:
conn.execute('DELETE FROM members WHERE id = ?', (member_id,))
conn.commit()
member_counter.add(1, {"operation": "delete"})
conn.close()
return redirect(url_for('index'))
@app.route('/admin')
def admin():
"""Admin dashboard with links to monitoring tools"""
with tracer.start_as_current_span("admin"):
return render_template('admin.html')
@app.route('/health')
def health():
"""Health check endpoint"""
return jsonify({"status": "healthy", "timestamp": datetime.utcnow().isoformat()})
@app.route('/metrics')
def metrics_endpoint():
"""Metrics endpoint for Prometheus"""
return "Metrics are exported via OTLP", 200
if __name__ == '__main__':
init_db()
port = int(os.getenv('PORT', 8080))
app.run(host='0.0.0.0', port=port, debug=False)
# Made with Bob
Infrastructure as Code: The Foundation
Demonstrating a deployment starts with Terraform to provision the Kubernetes resources (Namespaces, PVCs, and Services).
- Benefit: Consistency across environments — whether it’s Minikube for local dev or a cloud provider for production.
Terraform serves as the architect, defining the entire environment before a single line of application code runs.
- Provider Management: It initializes the Kubernetes provider to interact with clusters like Minikube.
- Resource Orchestration: Terraform handles the creation of Namespaces, ConfigMaps, and Secrets to ensure the environment is ready for deployment.
- Deployment Automation: It defines the specifications for the application pods, including replica counts, resource limits (512Mi memory/500m CPU), and container images.
- Service Exposure: It automatically provisions NodePort services for the App (30080), Prometheus (30090), and Grafana (30030).
terraform {
required_version = ">= 1.0"
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.23"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.11"
}
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
config_context = "minikube"
}
provider "helm" {
kubernetes {
config_path = "~/.kube/config"
config_context = "minikube"
}
}
# Create namespace
resource "kubernetes_namespace" "member_management" {
metadata {
name = "member-management"
labels = {
name = "member-management"
app = "member-management"
}
}
}
# Create ConfigMaps
resource "kubernetes_config_map" "app_config" {
metadata {
name = "app-config"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
data = {
DATABASE_PATH = "/data/members.db"
PORT = "8080"
VAULT_ADDR = "http://vault:8200"
VAULT_MOUNT_POINT = "secret"
OTEL_EXPORTER_OTLP_ENDPOINT = "http://otel-collector:4317"
}
}
resource "kubernetes_config_map" "otel_collector_config" {
metadata {
name = "otel-collector-config"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
data = {
"otel-collector-config.yaml" = file("${path.module}/../otel-collector-config.yaml")
}
}
resource "kubernetes_config_map" "prometheus_config" {
metadata {
name = "prometheus-config"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
data = {
"prometheus.yml" = file("${path.module}/../prometheus.yml")
}
}
resource "kubernetes_config_map" "grafana_datasources" {
metadata {
name = "grafana-datasources"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
data = {
"datasources.yml" = file("${path.module}/../grafana-datasources.yml")
}
}
# Create Secrets
resource "kubernetes_secret" "app_secrets" {
metadata {
name = "app-secrets"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
data = {
VAULT_TOKEN = base64encode("dev-token")
FLASK_SECRET_KEY = base64encode("production-secret-key-change-me")
}
type = "Opaque"
}
# Create PersistentVolumeClaims
resource "kubernetes_persistent_volume_claim" "app_data" {
metadata {
name = "app-data-pvc"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "1Gi"
}
}
storage_class_name = "standard"
}
}
resource "kubernetes_persistent_volume_claim" "prometheus_data" {
metadata {
name = "prometheus-data-pvc"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "5Gi"
}
}
storage_class_name = "standard"
}
}
resource "kubernetes_persistent_volume_claim" "grafana_data" {
metadata {
name = "grafana-data-pvc"
namespace = kubernetes_namespace.member_management.metadata[0].name
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "2Gi"
}
}
storage_class_name = "standard"
}
}
Security First: Locking Down the Vault
Security isn’t an afterthought; it’s baked into the core architecture via HashiCorp Vault. Detail the Security Architecture:
- Password Hashing: Using SHA−256 for member passwords.
- Secret Management: Integration with HashiCorp Vault (Port 8200) to store and retrieve sensitive member data.
- K8s Secrets: Utilizing Base64 encoding and encryption for cluster-level security.
- Centralized Secrets: Instead of hardcoding credentials, the application retrieves sensitive data from Vault’s KV v2 engine.
- Identity & Access: The system uses Token Authentication (dev-token) to verify the application's identity before granting access to secrets.
- Encryption at Rest: Vault ensures that sensitive member data is protected, while the application itself hashes passwords using SHA-256 for a multi-layered defense.
- Integration: The Flask app communicates with the Vault API on port 8200 to store and retrieve member-specific secrets during the creation and retrieval flows.
# Vault configuration
VAULT_ADDR = os.getenv('VAULT_ADDR', 'http://localhost:8200')
VAULT_TOKEN = os.getenv('VAULT_TOKEN', 'dev-token')
VAULT_MOUNT_POINT = os.getenv('VAULT_MOUNT_POINT', 'secret')
def get_vault_client():
"""Initialize and return Vault client"""
try:
client = hvac.Client(url=VAULT_ADDR, token=VAULT_TOKEN)
if not client.is_authenticated():
print("Warning: Vault authentication failed")
return None
return client
except Exception as e:
print(f"Warning: Could not connect to Vault: {e}")
return None
def store_secret_in_vault(username, secret_type, secret_value):
"""Store a secret in Vault"""
with tracer.start_as_current_span("store_secret_in_vault"):
client = get_vault_client()
if client:
try:
path = f"{VAULT_MOUNT_POINT}/data/members/{username}/{secret_type}"
client.secrets.kv.v2.create_or_update_secret(
path=f"members/{username}/{secret_type}",
secret=dict(value=secret_value),
mount_point=VAULT_MOUNT_POINT
)
return True
except Exception as e:
print(f"Error storing secret in Vault: {e}")
return False
return False
def get_secret_from_vault(username, secret_type):
"""Retrieve a secret from Vault"""
with tracer.start_as_current_span("get_secret_from_vault"):
client = get_vault_client()
if client:
try:
path = f"members/{username}/{secret_type}"
secret = client.secrets.kv.v2.read_secret_version(
path=path,
mount_point=VAULT_MOUNT_POINT
)
return secret['data']['data']['value']
except Exception as e:
print(f"Error retrieving secret from Vault: {e}")
return None
return None
def hash_password(password):
"""Hash password using SHA-256"""
return hashlib.sha256(password.encode()).hexdigest()
Post-Deployment Magic with Ansible
Lifecycle management is essential. While Terraform builds the “house,” Ansible handles the “furniture.”
- Inventory Management: Ansible maintains a structured inventory of the target hosts, grouping them into application and monitoring tiers.
all:
hosts:
localhost:
ansible_connection: local
ansible_python_interpreter: /usr/bin/python3
vars:
project_root: "{{ playbook_dir }}/.."
namespace: member-management
docker_image: member-management-app
docker_tag: latest
# Made with Bob
- Automated Deployment: The
deploy.ymlplaybook automates the pull of Docker images and the startup of the entire container stack.
---
- name: Deploy Member Management Application
hosts: localhost
gather_facts: yes
tasks:
- name: Check if Minikube is running
command: minikube status
register: minikube_status
ignore_errors: yes
changed_when: false
- name: Start Minikube if not running
command: minikube start
when: minikube_status.rc != 0
- name: Build Docker image
command: docker build -t {{ docker_image }}:{{ docker_tag }} {{ project_root }}
args:
chdir: "{{ project_root }}"
- name: Load Docker image into Minikube
command: minikube image load {{ docker_image }}:{{ docker_tag }}
- name: Create namespace
kubernetes.core.k8s:
state: present
definition:
apiVersion: v1
kind: Namespace
metadata:
name: "{{ namespace }}"
labels:
name: "{{ namespace }}"
app: "{{ namespace }}"
- name: Apply Kubernetes manifests
kubernetes.core.k8s:
state: present
src: "{{ item }}"
namespace: "{{ namespace }}"
loop:
- "{{ project_root }}/k8s/configmap.yaml"
- "{{ project_root }}/k8s/secrets.yaml"
- "{{ project_root }}/k8s/pvc.yaml"
- "{{ project_root }}/k8s/vault-deployment.yaml"
- "{{ project_root }}/k8s/otel-collector-deployment.yaml"
- "{{ project_root }}/k8s/prometheus-deployment.yaml"
- "{{ project_root }}/k8s/grafana-deployment.yaml"
- "{{ project_root }}/k8s/app-deployment.yaml"
- name: Wait for Vault to be ready
kubernetes.core.k8s_info:
kind: Pod
namespace: "{{ namespace }}"
label_selectors:
- app=vault
wait: yes
wait_condition:
type: Ready
status: "True"
wait_timeout: 300
- name: Wait for application to be ready
kubernetes.core.k8s_info:
kind: Pod
namespace: "{{ namespace }}"
label_selectors:
- app=member-management-app
wait: yes
wait_condition:
type: Ready
status: "True"
wait_timeout: 300
- name: Get Minikube IP
command: minikube ip
register: minikube_ip
changed_when: false
- name: Display access URLs
debug:
msg:
- "Application URL: http://{{ minikube_ip.stdout }}:30080"
- "Prometheus URL: http://{{ minikube_ip.stdout }}:30090"
- "Grafana URL: http://{{ minikube_ip.stdout }}:30030"
- "Grafana credentials: admin/admin"
# Made with Bob
- Seamless Updates: The
update.ymlplaybook allows for rolling updates, ensuring that the application can be patched or upgraded with minimal downtime. - Orchestration: It ensures dependencies — like ensuring Vault is unsealed and OTel is running — are handled in the correct sequence.
Observability: Seeing Inside the Black Box
This is where Bob shines; the Observability Stack:
- OpenTelemetry (OTel): The application sends traces and metrics to an OTel Collector.
OTel Collector: A central collector (Port 4317/4318) receives telemetry data, acting as a vendor-neutral gateway.
- Prometheus: the collector exports data to Prometheus, where it is stored as time-series data for analysis.
- Grafana: serves as the “single pane of glass,” providing pre-configured dashboards that track member operations, system health, and request latencies.
- Instrumentation: the Flask application and SQLite database are instrumented to produce traces and metrics automatically.
# Trace provider
trace_provider = TracerProvider(resource=resource)
otlp_trace_exporter = OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
insecure=True
)
trace_provider.add_span_processor(BatchSpanProcessor(otlp_trace_exporter))
trace.set_tracer_provider(trace_provider)
# Metrics provider
metric_reader = PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
insecure=True
)
)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
Conclusion
In summary, “Bob” represents a comprehensive shift from manual deployment to a fully automated, production-ready ecosystem. Starting from the ground up, the application provides a functional and intuitive UI for complete database record management, allowing users to handle member data with ease. This application layer is seamlessly integrated with HashiCorp Vault, ensuring that every piece of sensitive information is handled with industry-standard security through centralized secret management and robust encryption. By combining a reliable Python-based backend with a modern frontend, Bob proves that internal business tools can be both powerful and user-friendly right out of the box.
The true power of Bob, however, lies in its sophisticated infrastructure and lifecycle automation. Through the use of Terraform, the entire Kubernetes environment — including networking, storage, and pod configurations — is defined as code, ensuring consistent deployments across any cluster. Ansible further enhances this by orchestrating the operational lifecycle, from initial image builds to rolling updates. This is all capped off by an “out-of-the-box” observability stack where OpenTelemetry, Prometheus, and Grafana work in unison to provide real-time insights into the system’s health. Ultimately, Bob demonstrates that with the right integration of IaC, security, and monitoring, a complex cloud-native application can be deployed and managed with a single command.
>>> Thanks for reading <<<
Links
- Code repository for this post: https://github.com/aairom/app-automation-demo
- IBM Project Bob (GA in upcoming days 🎇): https://www.ibm.com/products/bob
Top comments (0)