DEV Community

Janak Shrestha
Janak Shrestha

Posted on

Day 62: Manage Secrets in Kubernetes

#100daysofdevops #devops #kubernetes

The Nautilus DevOps team is working to deploy some tools in Kubernetes cluster. Some of the tools are licence based so that licence information needs to be stored securely within Kubernetes cluster. Therefore, the team wants to utilize Kubernetes secrets to store those secrets. Below you can find more details about the requirements:

  1. We already have a secret key file media.txt under the /opt/ directory. Create a generic secret named media, it should contain the password/license-number present in media.txt file.
  2. Also create a pod named secret-nautilus.
  3. Configure pod's spec as container name should be secret-container-nautilus, image should be fedora with latest tag (remember to mention the tag with image). Use sleep command for container so that it remains in running state. Consume the created secret and mount it under /opt/demo within the container.
  4. To verify you can exec into the container secret-container-nautilus, to check the secret key under the mounted path /opt/demo. Before hitting the Check button please make sure pod/pods are in running state, also validation can take some time to complete so keep patience.

Understanding Kubernetes Secrets

What Are Secrets?

Secrets are Kubernetes objects that store sensitive information. They are similar to ConfigMaps but are specifically designed for confidential data.

Why Use Secrets?

Reason Description
Security Secrets are not stored in plain text in Pod definitions or container images
Separation Sensitive data is separated from application code
Access Control Secrets can be restricted using RBAC
Auditability Secret access can be logged and monitored
Flexibility Secrets can be updated without rebuilding images

Secret Types

Type Use Case
Opaque Arbitrary user-defined data (default)
kubernetes.io/service-account-token Service account tokens
kubernetes.io/dockercfg Docker registry credentials
kubernetes.io/tls TLS certificates
bootstrap.kubernetes.io/token Bootstrap tokens for node joining

What We Will Build

Task Overview

Component Specification
Secret Name media
Secret Source /opt/media.txt file
Secret Data 5ecur3 (license key)
Pod Name secret-nautilus
Container Name secret-container-nautilus
Image fedora:latest
Command sleep infinity
Mount Path /opt/demo

Architecture Diagram 

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Kubernetes Cluster                                 │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │  Secret: media                                                         │ │
│  │  - Type: Opaque                                                       │ │
│  │  - Data: media.txt → NWVjdXIzCg== (base64 encoded)                   │ │
│  │  - Original: 5ecur3                                                   │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
│                                    │                                         │
│                                    ▼                                         │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │  Pod: secret-nautilus                                                 │ │
│  │                                                                       │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │  Container: secret-container-nautilus                           │ │ │
│  │  │  Image: fedora:latest                                           │ │ │
│  │  │  Command: sleep infinity                                        │ │ │
│  │  │                                                                 │ │ │
│  │  │  Volume Mount: /opt/demo                                        │ │ │
│  │  │  └── media.txt → 5ecur3                                         │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  │                                    │                                   │ │
│  │                                    ▼                                   │ │
│  │  ┌──────────────────────────────────────────────────────────────────┐ │ │
│  │  │  Volume: secret-volume                                          │ │ │
│  │  │  Type: Secret (media)                                           │ │ │
│  │  └──────────────────────────────────────────────────────────────────┘ │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Step-by-Step Implementation

Step 1: Check the Secret File

First, verify the contents of the file that will be used to create the Secret:

cat /opt/media.txt
Enter fullscreen mode Exit fullscreen mode

Output:

5ecur3
Enter fullscreen mode Exit fullscreen mode

Step 2: Create the Secret

Create a generic Secret from the file:

kubectl create secret generic media --from-file=/opt/media.txt
Enter fullscreen mode Exit fullscreen mode

Output:

secret/media created
Enter fullscreen mode Exit fullscreen mode

Step 3: Verify the Secret

Check that the Secret was created successfully:

kubectl get secrets
Enter fullscreen mode Exit fullscreen mode

Output:

NAME    TYPE     DATA   AGE
media   Opaque   1      8s
Enter fullscreen mode Exit fullscreen mode

View detailed information about the Secret:

kubectl describe secret media
Enter fullscreen mode Exit fullscreen mode

Output:

Name:         media
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
media.txt:  7 bytes
Enter fullscreen mode Exit fullscreen mode

Step 4: Create the Pod YAML

Create a file named secret-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: secret-nautilus
spec:
  containers:
  - name: secret-container-nautilus
    image: fedora:latest
    command: ["sleep"]
    args: ["infinity"]
    volumeMounts:
    - name: secret-volume
      mountPath: /opt/demo
  volumes:
  - name: secret-volume
    secret:
      secretName: media
Enter fullscreen mode Exit fullscreen mode

YAML Breakdown

Volume Definition:

volumes:
- name: secret-volume
  secret:
    secretName: media
Enter fullscreen mode Exit fullscreen mode

This defines a volume that sources its content from the media Secret.

Volume Mount:

volumeMounts:
- name: secret-volume
  mountPath: /opt/demo
Enter fullscreen mode Exit fullscreen mode

This mounts the Secret volume at /opt/demo inside the container.

Container Command:

command: ["sleep"]
args: ["infinity"]
Enter fullscreen mode Exit fullscreen mode

This keeps the container running so we can verify the mounted Secret.

Step 5: Create the Pod

Apply the configuration:

kubectl apply -f secret-pod.yaml
Enter fullscreen mode Exit fullscreen mode

Output:

pod/secret-nautilus created
Enter fullscreen mode Exit fullscreen mode

Step 6: Verify the Pod

Check that the pod is running:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Output:

NAME              READY   STATUS    RESTARTS   AGE
secret-nautilus   1/1     Running   0          8s
Enter fullscreen mode Exit fullscreen mode

Step 7: Verify the Mounted Secret

Access the container to verify the Secret:

kubectl exec secret-nautilus -c secret-container-nautilus -- cat /opt/demo/media.txt
Enter fullscreen mode Exit fullscreen mode

Output:

5ecur3
Enter fullscreen mode Exit fullscreen mode

For a detailed view of the mounted Secret:

kubectl exec -it secret-nautilus -c secret-container-nautilus -- /bin/bash
[root@secret-nautilus /]# ls -la /opt/demo/
total 4
drwxrwxrwt 3 root root  100 Sep  1 09:35 .
drwxr-xr-x 1 root root 4096 Sep  1 09:35 ..
drwxr-xr-x 2 root root   60 Sep  1 09:35 ..2026_09_01_09_35_00.900118244
lrwxrwxrwx 1 root root   31 Sep  1 09:35 ..data -> ..2026_09_01_09_35_00.900118244
lrwxrwxrwx 1 root root   16 Sep  1 09:35 media.txt -> ..data/media.txt
[root@secret-nautilus /]# cat /opt/demo/media.txt
5ecur3
[root@secret-nautilus /]# exit
Enter fullscreen mode Exit fullscreen mode

Step 8: Examine the Secret Data

View the encoded Secret data:

kubectl get secret media -o yaml
Enter fullscreen mode Exit fullscreen mode

Output:

apiVersion: v1
data:
  media.txt: NWVjdXIzCg==
kind: Secret
metadata:
  creationTimestamp: "2026-09-01T09:34:04Z"
  name: media
  namespace: default
  resourceVersion: "1297"
  uid: 31b18ac6-bd08-4a2f-934e-60e1effffb95
type: Opaque
Enter fullscreen mode Exit fullscreen mode

Step 9: Decode the Secret

To verify the encoded value:

echo "NWVjdXIzCg==" | base64 -d
Enter fullscreen mode Exit fullscreen mode

Output:

5ecur3
Enter fullscreen mode Exit fullscreen mode

How Secrets Work

Secret Creation Flow 

┌─────────────────────────────────────────────────────────────────────────────┐
│                    Secret Workflow                                          │
│                                                                              │
│  1. Source File: /opt/media.txt                                            │
│     └── Content: 5ecur3                                                    │
│                                                                              │
│  2. Secret Creation: kubectl create secret generic media                   │
│     └── File content is read                                              │
│     └── Base64 encoded: NWVjdXIzCg==                                      │
│     └── Stored in etcd                                                    │
│                                                                              │
│  3. Pod References Secret                                                  │
│     └── volumes:                                                          │
│         - name: secret-volume                                              │
│           secret:                                                          │
│             secretName: media                                              │
│                                                                              │
│  4. Secret Mounted as Volume                                                │
│     └── volumeMounts:                                                      │
│         - name: secret-volume                                              │
│           mountPath: /opt/demo                                             │
│                                                                              │
│  5. Container Access                                                       │
│     └── cat /opt/demo/media.txt                                            │
│     └── Returns: 5ecur3                                                    │
└─────────────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Secret Mount Behavior

When a Secret is mounted as a volume:

  • Each key in the Secret becomes a file
  • The filename is the key name
  • The file content is the decoded value
  • The volume is read-only
  • Files are updated if the Secret is updated

Key Concepts

Secret Creation Methods

Method Command Use Case
From File kubectl create secret generic <name> --from-file=<path> Single file
From Directory kubectl create secret generic <name> --from-file=<directory> Multiple files
From Literal kubectl create secret generic <name> --from-literal=key=value Simple values
From YAML kubectl apply -f secret.yaml Declarative approach

Secret Consumption Methods

Method Description When to Use
Volume Mount Mount Secret as files in a volume Files needed by the application
Environment Variable Inject Secret as environment variable Simple values needed as env vars
Image Pull Secrets Used for pulling images from private registries Container registry authentication

Secret Security Considerations

Concern Mitigation
Storage Secrets are stored in etcd; use encryption at rest
Transmission Secrets are transmitted over TLS
Access Use RBAC to control who can access Secrets
Logging Secrets are not automatically logged
Mutation Secrets can be updated using kubectl edit or kubectl patch

Troubleshooting

Secret Not Found 

# Verify the Secret exists
kubectl get secret media

# Check if the Secret name matches the Pod specification
kubectl describe pod secret-nautilus | grep -A 5 "Volumes:"
Enter fullscreen mode Exit fullscreen mode

Pod Not Starting 

# Check pod status
kubectl get pods

# Check pod details
kubectl describe pod secret-nautilus

# Check events
kubectl get events --field-selector involvedObject.name=secret-nautilus
Enter fullscreen mode Exit fullscreen mode

Secret Not Mounted 

# Check volume mounts in the pod
kubectl describe pod secret-nautilus | grep -A 10 "Mounts:"

# Check volumes in the pod
kubectl describe pod secret-nautilus | grep -A 10 "Volumes:"

# Check the mount path exists
kubectl exec secret-nautilus -c secret-container-nautilus -- ls -la /opt/demo/
Enter fullscreen mode Exit fullscreen mode

Secret Data Incorrect 

# Check the Secret data
kubectl get secret media -o yaml

# Decode the data
kubectl get secret media -o jsonpath='{.data.media.txt}' | base64 -d

# Compare with the source file
cat /opt/media.txt
Enter fullscreen mode Exit fullscreen mode

Best Practices

1. Use Secrets for Sensitive Data

Do not store sensitive data in:

  • Container images
  • Configuration files
  • Environment variables in Pod definitions

2. Encrypt Secrets at Rest

Enable encryption at rest for etcd to protect Secrets.

3. Use RBAC for Access Control

Restrict access to Secrets using Role-Based Access Control.

4. Rotate Secrets Regularly

Implement a rotation policy for sensitive credentials.

5. Use External Secret Management

Consider using tools like HashiCorp Vault for advanced secret management.

6. Avoid Base64 Encoding for Security

Base64 encoding is not encryption; use proper encryption for sensitive data.

Summary

In this challenge, we successfully:

  1. Created a Secret from a file using kubectl create secret generic
  2. Deployed a pod that consumes the Secret as a volume mount
  3. Verified that the Secret data is correctly mounted and accessible within the container
  4. Examined the encoded Secret data and decoded it to confirm the original value

Kubernetes Secrets provide a secure and flexible way to manage sensitive information in containerized applications. By separating secrets from application code, we improve security, maintainability, and operational flexibility.

Useful Commands Reference

Command Purpose
kubectl create secret generic <name> --from-file=<path> Create Secret from file
kubectl get secrets List all Secrets
kubectl describe secret <name> Detailed Secret information
kubectl get secret <name> -o yaml Secret in YAML format
`kubectl get secret -o jsonpath='{.data.}' \ base64 -d`
kubectl apply -f <pod.yaml> Deploy a Pod
kubectl get pods List Pods
kubectl exec <pod> -c <container> -- <command> Execute command in container
kubectl describe pod <pod> Detailed Pod information

Top comments (0)