When you're building applications for Kubernetes, one of the first challenges you'll face is: "Where do I put my configuration data?" Should database connection strings live in your Docker image? What about API keys? How do you handle different configurations for development, staging, and production environments?
This is where Kubernetes ConfigMaps and Secrets come to the rescue. In this post, I would like to explore, with you, these essential Kubernetes resources that separate your application code from its configuration, making your deployments more secure, flexible, and maintainable.
The Problem: Hardcoded Configuration
Before we dive into the solution, let's understand the problem. Consider this simple Node.js application:
// app.js - DON'T DO THIS
const express = require('express');
const mysql = require('mysql2');
const app = express();
// Hardcoded configuration - BAD PRACTICE
const config = {
port: 3000,
database: {
host: 'mysql-prod.company.com',
user: 'admin',
password: 'super-secret-password',
database: 'production_db'
},
apiKey: 'sk-1234567890abcdef'
};
const connection = mysql.createConnection(config.database);
app.listen(config.port);
This approach has several critical problems:
- Security Risk: Sensitive data like passwords and API keys are embedded in your code
- Environment Rigidity: The same image can't work across different environments
- Secret Exposure: Passwords end up in your Git repository and Docker images
- Deployment Complexity: Changing configuration requires rebuilding and redeploying your entire application
The Kubernetes Solution: Externalizing Configuration
Kubernetes provides two primary resources for managing configuration:
- ConfigMaps: For non-sensitive configuration data
- Secrets: For sensitive information like passwords, tokens, and keys
Both resources follow the same core principle: separate configuration from application code.
ConfigMaps: Your Configuration Data Store
What is a ConfigMap?
A ConfigMap is a Kubernetes API object that stores configuration data as key-value pairs. Think of it as a dictionary or hash map that your applications can reference at runtime.
Creating ConfigMaps
There are several ways to create ConfigMaps. Let's explore each method:
Method 1: Using kubectl with Literal Values
kubectl create configmap app-config \
--from-literal=PORT=3000 \
--from-literal=DATABASE_HOST=mysql.default.svc.cluster.local \
--from-literal=DATABASE_NAME=myapp \
--from-literal=LOG_LEVEL=info
Method 2: From a Configuration File
First, create a configuration file:
# app.properties
PORT=3000
DATABASE_HOST=mysql.default.svc.cluster.local
DATABASE_NAME=myapp
LOG_LEVEL=info
DEBUG_MODE=false
Then create the ConfigMap:
kubectl create configmap app-config --from-file=app.properties
Method 3: Using YAML Manifests (Recommended)
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: default
data:
PORT: "3000"
DATABASE_HOST: "mysql.default.svc.cluster.local"
DATABASE_NAME: "myapp"
LOG_LEVEL: "info"
DEBUG_MODE: "false"
# You can also include entire configuration files
app.properties: |
PORT=3000
DATABASE_HOST=mysql.default.svc.cluster.local
DATABASE_NAME=myapp
LOG_LEVEL=info
DEBUG_MODE=false
Apply the ConfigMap:
kubectl apply -f configmap.yaml
Consuming ConfigMaps in Your Applications
There are three primary ways to use ConfigMaps in your pods:
1. Environment Variables
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:latest
env:
# Individual environment variables
- name: PORT
valueFrom:
configMapKeyRef:
name: app-config
key: PORT
- name: DATABASE_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: DATABASE_HOST
# Or load all ConfigMap keys as environment variables
envFrom:
- configMapRef:
name: app-config
2. Volume Mounts
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:latest
volumeMounts:
- name: config-volume
mountPath: /etc/config
readOnly: true
volumes:
- name: config-volume
configMap:
name: app-config
With this approach, your configuration files will be available at /etc/config/ inside the container.
3. Command Line Arguments
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
image: myapp:latest
command: ["./myapp"]
args:
- "--port=$(PORT)"
- "--database-host=$(DATABASE_HOST)"
env:
- name: PORT
valueFrom:
configMapKeyRef:
name: app-config
key: PORT
- name: DATABASE_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: DATABASE_HOST
Secrets: Protecting Sensitive Data
What is a Secret?
A Secret is similar to a ConfigMap, but specifically designed for sensitive data. Kubernetes stores Secrets in base64 encoding and provides additional security features like:
- Secrets are stored in etcd in encrypted form (when encryption at rest is enabled)
- Secrets are only sent to nodes that have pods requiring them
- Secrets are stored in memory (tmpfs) and never written to disk
- Access can be controlled with RBAC policies
Types of Secrets
Kubernetes supports several Secret types:
- Opaque: Generic secrets for arbitrary user data (default)
- kubernetes.io/dockerconfigjson: Docker registry credentials
- kubernetes.io/tls: TLS certificates and keys
- kubernetes.io/service-account-token: Service account tokens
Creating Secrets
Method 1: Using kubectl with Literal Values
kubectl create secret generic app-secrets \
--from-literal=DATABASE_PASSWORD=super-secret-password \
--from-literal=API_KEY=sk-1234567890abcdef \
--from-literal=JWT_SECRET=my-jwt-secret-key
Method 2: From Files
# Create files with sensitive data
echo -n 'super-secret-password' > db-password.txt
echo -n 'sk-1234567890abcdef' > api-key.txt
kubectl create secret generic app-secrets \
--from-file=DATABASE_PASSWORD=db-password.txt \
--from-file=API_KEY=api-key.txt
# Clean up the files
rm db-password.txt api-key.txt
Method 3: Using YAML Manifests
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
namespace: default
type: Opaque
data:
# Values must be base64 encoded
DATABASE_PASSWORD: c3VwZXItc2VjcmV0LXBhc3N3b3Jk # super-secret-password
API_KEY: c2stMTIzNDU2Nzg5MGFiY2RlZg== # sk-1234567890abcdef
JWT_SECRET: bXktand0LXNlY3JldC1rZXk= # my-jwt-secret-key
Pro Tip: Use the stringData field to avoid manual base64 encoding:
apiVersion: v1
kind: Secret
metadata:
name: app-secrets
type: Opaque
stringData:
DATABASE_PASSWORD: super-secret-password
API_KEY: sk-1234567890abcdef
JWT_SECRET: my-jwt-secret-key
Consuming Secrets
Secrets are consumed in the same ways as ConfigMaps:
Environment Variables
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
image: myapp:latest
env:
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: app-secrets
key: DATABASE_PASSWORD
- name: API_KEY
valueFrom:
secretKeyRef:
name: app-secrets
key: API_KEY
envFrom:
- secretRef:
name: app-secrets
Volume Mounts
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
image: myapp:latest
volumeMounts:
- name: secret-volume
mountPath: /etc/secrets
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: app-secrets
defaultMode: 0400 # Read-only for owner only
In the next post we’ll see how to use secrets and ConfigMap in a complete application setup.
Real-World Example: Complete Application Setup
Let's put it all together with a realistic example. We'll deploy a Node.js application that connects to a MySQL database.
Step 1: Create the ConfigMap
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: webapp-config
data:
PORT: "3000"
NODE_ENV: "production"
DATABASE_HOST: "mysql.default.svc.cluster.local"
DATABASE_PORT: "3306"
DATABASE_NAME: "webapp"
LOG_LEVEL: "info"
CACHE_TTL: "3600"
RATE_LIMIT: "100"
Step 2: Create the Secret
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: webapp-secrets
type: Opaque
stringData:
DATABASE_USER: admin
DATABASE_PASSWORD: my-secure-password-123
JWT_SECRET: super-secret-jwt-key-that-should-be-random
API_KEY: sk-live-1234567890abcdef
Step 3: Create the Deployment
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
labels:
app: webapp
spec:
replicas: 3
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
containers:
- name: webapp
image: mycompany/webapp:v1.2.0
ports:
- containerPort: 3000
# Load non-sensitive config as environment variables
envFrom:
- configMapRef:
name: webapp-config
# Load sensitive data as environment variables
- secretRef:
name: webapp-secrets
# Health checks
livenessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 3000
initialDelaySeconds: 5
periodSeconds: 5
# Resource limits
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "200m"
Step 4: Updated Application Code
Now your application code becomes much cleaner:
// app.js - Clean configuration management
const express = require('express');
const mysql = require('mysql2/promise');
const app = express();
// Configuration from environment variables
const config = {
port: process.env.PORT || 3000,
nodeEnv: process.env.NODE_ENV || 'development',
database: {
host: process.env.DATABASE_HOST || 'localhost',
port: process.env.DATABASE_PORT || 3306,
user: process.env.DATABASE_USER || 'root',
password: process.env.DATABASE_PASSWORD || '',
database: process.env.DATABASE_NAME || 'myapp'
},
jwtSecret: process.env.JWT_SECRET,
apiKey: process.env.API_KEY,
logLevel: process.env.LOG_LEVEL || 'info',
cacheTtl: parseInt(process.env.CACHE_TTL) || 3600,
rateLimit: parseInt(process.env.RATE_LIMIT) || 100
};
// Validate required configuration
if (!config.jwtSecret) {
console.error('JWT_SECRET is required');
process.exit(1);
}
if (!config.apiKey) {
console.error('API_KEY is required');
process.exit(1);
}
// Initialize database connection
let connection;
async function initDatabase() {
try {
connection = await mysql.createConnection(config.database);
console.log('Database connected successfully');
} catch (error) {
console.error('Database connection failed:', error);
process.exit(1);
}
}
// Health check endpoints
app.get('/health', (req, res) => {
res.status(200).json({ status: 'healthy' });
});
app.get('/ready', async (req, res) => {
try {
await connection.ping();
res.status(200).json({ status: 'ready' });
} catch (error) {
res.status(503).json({ status: 'not ready', error: error.message });
}
});
// Start the application
async function start() {
await initDatabase();
app.listen(config.port, () => {
console.log(`Server running on port ${config.port}`);
console.log(`Environment: ${config.nodeEnv}`);
console.log(`Log Level: ${config.logLevel}`);
});
}
start().catch(error => {
console.error('Failed to start application:', error);
process.exit(1);
});
Step 5: Deploy Everything
kubectl apply -f configmap.yaml
kubectl apply -f secret.yaml
kubectl apply -f deployment.yaml
Advanced Patterns and Best Practices
1. Environment-Specific Configurations
Use different ConfigMaps and Secrets for different environments:
# Development
kubectl create configmap webapp-config --from-env-file=dev.env
kubectl create secret generic webapp-secrets --from-env-file=dev-secrets.env
# Production
kubectl create configmap webapp-config --from-env-file=prod.env
kubectl create secret generic webapp-secrets --from-env-file=prod-secrets.env
2. Configuration Validation
Add configuration validation to your application:
const joi = require('joi');
const configSchema = joi.object({
port: joi.number().port().required(),
nodeEnv: joi.string().valid('development', 'production', 'test').required(),
database: joi.object({
host: joi.string().hostname().required(),
port: joi.number().port().required(),
user: joi.string().required(),
password: joi.string().min(8).required(),
database: joi.string().required()
}).required(),
jwtSecret: joi.string().min(32).required(),
apiKey: joi.string().required(),
logLevel: joi.string().valid('error', 'warn', 'info', 'debug').required(),
cacheTtl: joi.number().min(0).required(),
rateLimit: joi.number().min(1).required()
});
const { error, value } = configSchema.validate(config);
if (error) {
console.error('Configuration validation failed:', error.details);
process.exit(1);
}
3. Hot Reloading Configuration
Monitor ConfigMap changes and reload configuration without restarting:
const fs = require('fs');
const path = require('path');
// If using volume mounts
function watchConfigFile() {
const configPath = '/etc/config/app.properties';
if (fs.existsSync(configPath)) {
fs.watchFile(configPath, (curr, prev) => {
console.log('Configuration file changed, reloading...');
loadConfigFromFile(configPath);
});
}
}
function loadConfigFromFile(filePath) {
try {
const content = fs.readFileSync(filePath, 'utf8');
const newConfig = parseConfig(content);
// Update non-critical configuration
config.logLevel = newConfig.LOG_LEVEL;
config.cacheTtl = parseInt(newConfig.CACHE_TTL);
config.rateLimit = parseInt(newConfig.RATE_LIMIT);
console.log('Configuration reloaded successfully');
} catch (error) {
console.error('Failed to reload configuration:', error);
}
}
4. Secret Rotation
Implement graceful secret rotation:
const crypto = require('crypto');
class SecretManager {
constructor() {
this.currentSecrets = this.loadSecrets();
this.watchSecrets();
}
loadSecrets() {
return {
jwtSecret: process.env.JWT_SECRET,
apiKey: process.env.API_KEY,
databasePassword: process.env.DATABASE_PASSWORD
};
}
watchSecrets() {
// Watch for secret file changes if using volume mounts
const secretsPath = '/etc/secrets';
if (fs.existsSync(secretsPath)) {
fs.watch(secretsPath, (eventType, filename) => {
if (eventType === 'change') {
console.log(`Secret ${filename} changed, preparing rotation...`);
this.rotateSecret(filename);
}
});
}
}
async rotateSecret(secretName) {
try {
const newSecret = fs.readFileSync(`/etc/secrets/${secretName}`, 'utf8');
// Graceful rotation logic here
// For example, accept both old and new JWT secrets temporarily
if (secretName === 'JWT_SECRET') {
this.gracefulJwtRotation(newSecret);
}
} catch (error) {
console.error(`Failed to rotate secret ${secretName}:`, error);
}
}
gracefulJwtRotation(newSecret) {
// Keep both secrets active for a transition period
this.acceptedJwtSecrets = [this.currentSecrets.jwtSecret, newSecret];
// After transition period, use only the new secret
setTimeout(() => {
this.currentSecrets.jwtSecret = newSecret;
this.acceptedJwtSecrets = [newSecret];
console.log('JWT secret rotation completed');
}, 300000); // 5 minutes
}
}
Security Best Practices
1. Principle of Least Privilege
Only give pods access to the ConfigMaps and Secrets they actually need:
apiVersion: v1
kind: ServiceAccount
metadata:
name: webapp-service-account
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: webapp-config-reader
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["webapp-config"]
verbs: ["get"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["webapp-secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: webapp-config-binding
subjects:
- kind: ServiceAccount
name: webapp-service-account
roleRef:
kind: Role
name: webapp-config-reader
apiGroup: rbac.authorization.k8s.io
2. Avoid Logging Sensitive Data
Never log environment variables or configuration that might contain secrets:
// DON'T DO THIS
console.log('Starting with config:', JSON.stringify(process.env));
// DO THIS INSTEAD
const safeConfig = {
port: config.port,
nodeEnv: config.nodeEnv,
logLevel: config.logLevel,
// Mask sensitive data
databaseHost: config.database.host,
databaseUser: config.database.user.replace(/./g, '*'),
// Don't log secrets at all
};
console.log('Starting with config:', JSON.stringify(safeConfig));
3. Use Dedicated Service Accounts
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
spec:
template:
spec:
serviceAccountName: webapp-service-account # Dedicated service account
automountServiceAccountToken: false # Don't mount unless needed
containers:
- name: webapp
image: mycompany/webapp:v1.2.0
# ... rest of configuration
Troubleshooting Common Issues
Problem 1: ConfigMap Not Updating in Pods
Symptoms: You updated a ConfigMap, but your pods still see the old values.
Causes and Solutions:
-
Environment Variables: Environment variables are set at pod startup and don't update automatically.
# Force pod recreation kubectl rollout restart deployment/webapp -
Volume Mounts: May take up to 60 seconds to propagate.
# Check if files are updated kubectl exec -it webapp-xxx -- cat /etc/config/app.properties
Problem 2: Secret Values Not Decoding Properly
Symptoms: Your application receives base64-encoded values instead of plain text.
Solution: Kubernetes automatically decodes Secret values. If you're getting encoded values, check if you're reading from the wrong source:
// Wrong - reading from a manually created file
const secret = fs.readFileSync('/tmp/my-secret', 'utf8'); // Still encoded
// Right - from environment variable or volume mount
const secret = process.env.MY_SECRET; // Automatically decoded
Problem 3: Permission Denied Errors
Symptoms: Pods can't read ConfigMaps or Secrets.
Solution: Check RBAC permissions and service account configuration:
# Check service account
kubectl get pods webapp-xxx -o yaml | grep serviceAccount
# Check RBAC permissions
kubectl auth can-i get configmap --as=system:serviceaccount:default:webapp-service-account
# Debug with a temporary pod
kubectl run debug --rm -it --image=busybox --restart=Never -- sh
Performance Considerations
1. ConfigMap and Secret Size Limits
- ConfigMaps and Secrets have a 1MB size limit
- For larger configuration files, consider using init containers or external configuration services
2. Update Propagation
- Environment variables: Require pod restart
- Volume mounts: Propagate within ~60 seconds
- Choose the right method based on your update frequency needs
3. Memory Usage
- Secrets are stored in memory (tmpfs) and count against container memory limits
- Consider this when setting memory limits for pods with large secrets
Monitoring and Observability
Track ConfigMap and Secret usage:
# Add labels for better organization
apiVersion: v1
kind: ConfigMap
metadata:
name: webapp-config
labels:
app: webapp
version: v1.2.0
environment: production
config-type: application
data:
# ... configuration data
Monitor configuration changes:
# Watch for ConfigMap changes
kubectl get events --field-selector involvedObject.kind=ConfigMap -w
# Check ConfigMap history
kubectl describe configmap webapp-config
Conclusion
ConfigMaps and Secrets are fundamental building blocks for managing configuration in Kubernetes. They enable you to:
- Separate concerns: Keep configuration separate from application code
- Enhance security: Protect sensitive data with proper access controls
- Improve flexibility: Use the same container images across different environments
- Simplify operations: Update configuration without rebuilding images
Key takeaways for entry-level engineers:
- Use ConfigMaps for non-sensitive data like database hostnames, port numbers, and feature flags
- Use Secrets for sensitive data like passwords, API keys, and certificates
- Choose the right consumption method based on your update requirements
- Follow security best practices with proper RBAC and service accounts
- Validate configuration in your application code
- Monitor and log safely without exposing sensitive information
As you continue your Kubernetes journey, remember that good configuration management is not just about the technology—it's about creating maintainable, secure, and scalable applications. ConfigMaps and Secrets provide the foundation, but how you use them will determine the success of your deployments.
Start small, experiment with different approaches, and gradually build more sophisticated configuration management patterns as your applications and team grow. The investment in proper configuration management will pay dividends in reduced deployment complexity, improved security, and faster development cycles.
👉 If you enjoyed this post, consider supporting me by buying me a coffee.
👉 If you want to stay updated with similar posts, consider joining my newsletter.
Top comments (0)