Aisalkyn Aidarova

Posted on Dec 2

WHAT IS A SECRET?

#cicd #devops #security

A secret is any piece of sensitive information that must be protected from unauthorized access.

Examples:

API keys
Access tokens
Database passwords
Private keys (.pem)
Confluent Cloud credentials
Terraform backend credentials
OAuth tokens
SSH keys

Goal: Keep secrets encrypted at rest, encrypted in transit, and never stored in plaintext (repo, logs, artifacts).

🔥 2. WHY SECRETS MANAGEMENT IS CRITICAL

A senior DevOps engineer must prevent:

Credential leaks
Unauthorized access
Accidental commits to Git
Hardcoding in Terraform, Kubernetes, Docker, or CI/CD

Secrets leaks cause:

Environment compromise
Data breaches
Unauthorized AWS usage costing thousands
Repository takeovers

This is why we use secure secret stores, not files.

🟦 3. SECRET STORAGE OPTIONS DevOps MUST know

There are 4 main secret management solutions you must understand:

Tool	Where Used	Strengths	Weaknesses
GitHub Secrets	GitHub CI/CD	Easy to use, encrypted	Not for runtime apps
AWS Secrets Manager	Apps running on AWS	Automatic rotation, IAM integration	Expensive at scale
AWS SSM Parameter Store	AWS Systems Manager	Cheaper than Secrets Manager	Rotation not native
HashiCorp Vault	Enterprise multi-cloud	Most secure, dynamic secrets	Complex to manage

🟩 4. GITHUB SECRETS — Used for CI/CD Only

✔ Where used:

GitHub Actions CI/CD pipelines

✔ What it stores:

AWS access key + secret key
Docker registry token
Terraform Cloud token
Confluent Cloud credentials
Any deployment API keys

✔ How it works:

GitHub encrypts the secret with libsodium
Only GitHub Actions that run in your repository can access it
Not available to fork PRs

✔ Security rules for senior DevOps:

Never store database passwords here for applications
Never store long-lived AWS keys (prefer OIDC)
Rotate keys every 90 days
Give repositories minimum access
Avoid storing complex JSON — use AWS Parameter Store instead

❌ GitHub Secrets DO NOT replace:

Secrets Manager
Vault
Kubernetes Secrets
Application runtime secrets

GitHub Secrets are ONLY for CI/CD.

🟥 5. AWS SECRETS MANAGER — Production-grade secret storage

✔ Where used:

Production microservices on AWS.

✔ Features:

Automatic rotation (Lambda)
Version history
Multi-account access with IAM
Replication across regions
KMS encryption (built-in)

✔ Typical use cases:

Store RDS master password
Store Confluent API secret
Store Stripe keys
Store OAuth tokens
Store DB credentials for ECS tasks

✔ Access via IAM:

ecsTaskExecutionRole:
  can access secret: arn:aws:secretsmanager:...

✔ Code example (ECS task environment):

{
  "name": "DB_PASSWORD",
  "valueFrom": "arn:aws:secretsmanager:us-east-2:xxx:secret:db_pass"
}

❗ When to choose Secrets Manager:

You need rotation
You need strict auditing
You manage cross-account apps

🟨 6. AWS SSM PARAMETER STORE

(SecureString parameters)

✔ Cheaper alternative to Secrets Manager

Costs: $0 for Standard tier
(Secrets Manager costs $0.40 per secret per month)

✔ Good for Dev / QA / Non-critical secrets

✔ Use cases:

Microservice configs
Non-rotating tokens
S3 bucket names
Feature flags

❗ NOT recommended

for production database passwords (no rotation).

🟪 7. HASHICORP VAULT — The most advanced system

This is enterprise-level secret management.

✔ Why Vault is used:

Supports AWS, GCP, Azure, Kubernetes, On-prem
Dynamic secrets (temporary DB creds)
Encryption as a service (Transit)
PKI certificate generation
Fine-grained access policies
Audit logs
Can run on-prem or as HCP Vault Cloud

✔ Dynamic secrets example:

Vault generates:

A PostgreSQL username/password
Valid for 1 hour
Automatically deleted afterward

Perfect for:

Short-lived CI/CD tasks
High-security environments
Banks, healthcare, fintech

✔ Vault is used by:

Uber
Stripe
Goldman Sachs
Netflix

🟦 8. Kubernetes Secrets (Optional but DevOps MUST know)

✔ Stored inside etcd (encrypted with KMS in prod)

✔ Used for:

API keys
DB passwords
TLS certs

✔ Mounted as:

env variables
files

🟫 9. Terraform & Secrets — Senior Level Knowledge

Terraform NEVER stores secrets in:

Git
tf files
modules

❗❗ Secrets MUST be passed via:

terraform.tfvars (locally only)
CI/CD environment variables
SSM Parameter Store
Secrets Manager

✔ Example bad code (DO NOT DO):

password = "MySecret123"

✔ Good:

password = var.db_password

✔ Best:

password = data.aws_secretsmanager_secret_version.db_password.secret_string

🟩 10. How Secrets Flow in a Real CI/CD Pipeline

Example using GitHub Actions + AWS Secrets Manager:

Step 1:

Secrets stored in Secrets Manager

Step 2:

EC2/ECS Lambda uses IAM role to access secrets

Step 3:

GitHub Actions stores only:

AWS Access Key
Secret Key
Confluent API key

Step 4:

Terraform deploys infrastructure
→ references secrets with ARN

Step 5:

Applications retrieve secrets using:

AWS SDK
IAM role permissions

🟥 11. What NOT TO DO (Senior DevOps Knowledge)

❌ Never store secrets in GitHub repository
❌ Never store secrets in Slack or Teams
❌ Never store secrets in Docker image
❌ Never store secrets in YAML files
❌ Never store secrets in Terraform state
❌ Never store secrets in code comments
❌ Never echo secrets in CI logs
❌ Never send secrets in email

If leaked → rotate immediately.

🟦 12. Interview-Level Explanation (You can say this)

“In my pipelines, GitHub Secrets are used only for CI/CD credentials.
For application runtime secrets, I use AWS Secrets Manager or SSM Parameter Store depending on rotation requirements.
I avoid hardcoding secrets in Terraform by pulling them from the secret stores at runtime.
For enterprise multi-cloud environments, I integrate HashiCorp Vault with AWS IAM and Kubernetes service accounts for secure authentication and dynamic secrets.
All secrets are KMS-encrypted and never exposed in logs.”

This is senior-level.

🔵 SECRETS FLOW — High-Level Diagram

                        ┌──────────────────────────┐
                        │     Developer Machine     │
                        │    (Push Git Changes)     │
                        └─────────────┬────────────┘
                                      │
                                      ▼
                         ┌────────────────────────┐
                         │    GitHub Repository    │
                         └─────────────┬───────────┘
                                      │
                                      ▼
                         ┌──────────────────────────────┐
                         │     GitHub Actions Runner     │
                         │   (CI/CD Workflow Execution)  │
                         └──────────────┬────────────────┘
      SECRETS ENTER HERE FROM GITHUB →  │ 
                                      ▼
   ┌───────────────────────────────────────────────────────────────────────────────┐
   │                           GitHub Secrets Storage                               │
   │    - AWS_ACCESS_KEY_ID                                                         │
   │    - AWS_SECRET_ACCESS_KEY                                                     │
   │    - CONFLUENT_API_KEY                                                         │
   │    - CONFLUENT_API_SECRET                                                      │
   │    - EXISTING_VPC_ID                                                           │
   │    - SUBNETS / SG IDs                                                          │
   └───────────────┬──────────────────────────────────────────────────────────────┘
                   │ ENV VARIABLES PASSED TO TERRAFORM
                   ▼
        ┌───────────────────────────────────────┐
        │            TERRAFORM ENGINE           │
        │   terraform init / plan / apply       │
        └─────────────────────────┬─────────────┘
                                  │
         TERRAFORM USES SECRETS → │ 
                                  ▼
         ┌─────────────────────────────────────────────┐
         │          AWS Terraform Provider              │
         └───────────┬──────────────────────────────────┘
                     │
                     ▼
────────────────────────────────────────────────────────────────────────
│                         AWS Cloud                                     │
│                                                                        │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │                     AWS IAM (Identity)                          │   │
│   │   - Permissions for Terraform                                   │   │
│   │   - Permissions for ECS tasks                                   │   │
│   └────────────────────────────────────────────────────────────────┘   │
│                                                                        │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │              AWS Secrets Manager / Parameter Store              │   │
│   │   - Terraform can CREATE secrets here                           │   │
│   │   - ECS tasks retrieve secrets automatically                    │   │
│   └────────────────────────────────────────────────────────────────┘   │
│                                                                        │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │                        AWS ECS Cluster                         │   │
│   │   - Backend container                                           │   │
│   │   - Producer container                                          │   │
│   │   - Payment / Fraud / Analytics                                 │   │
│   │   - Containers read secrets at runtime                          │   │
│   └────────────────────────────────────────────────────────────────┘   │
│                                                                        │
────────────────────────────────────────────────────────────────────────

🔐 Explanation of Every Secret Component

Everything explained as a senior DevOps must understand.

#1 — GitHub Secrets (CI/CD)

GitHub Secrets are stored encrypted in GitHub.
They are used only during workflow execution.

Used in your project:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
RDS_PASSWORD
CONFLUENT_API_KEY
CONFLUENT_API_SECRET
VPC_ID / SUBNET_IDS / SG_IDS

GitHub Secrets lifetime:

Used only during pipeline execution
Not accessible after the run
Good for CI/CD only, NOT for runtime

#2 — AWS Secrets Manager

This is AWS’s official secret storage system.

What you store here:

Database passwords
API keys
Confluent secrets
JWT secrets
Backend environment variables

Why AWS Secrets Manager is better than GitHub Secrets:

GitHub Secrets = CI/CD
AWS Secrets = Runtime

ECS Tasks → automatically fetch secrets from Secrets Manager and inject into containers.

Benefits:

Automatic rotation
KMS encryption
IAM auth
Direct injection into ECS Task Definitions
No need to expose environment variables in Terraform

#3 — AWS Parameter Store (SSM)

Simpler version of Secrets Manager.

When to use:

When you need configuration (not secrets)
When cost matters (cheaper than Secrets Manager)
When you need infrastructure parameters

Sample:

/backend/SERVICE_URL
/kafka/bootstrap
/env/prod/feature-flag

#4 — HashiCorp Vault (Senior-level DevOps Topic)

Vault is used in enterprise environments for high-grade secret management.

Why Vault?

Dynamic secrets (MySQL, AWS IAM, Kafka credentials)
Zero-trust access
Multi-cloud support
Token-based authentication
Secret leasing (expires automatically)
Audit logs

Vault is often used when:

You have Kubernetes clusters
You need dynamic credentials
You need multi-cloud
You need compliance (PCI, HIPAA)
You need secret encryption policies

#5 — Terraform and Secrets

Terraform itself should never store secrets in .tf files.

Correct ways:

Pass secrets through TF_VAR_* from GitHub
Read secrets from Secrets Manager
Use Sensitive = true variables

Incorrect:

Hardcoding secrets
Committing terraform.tfvars with passwords

#6 — How Secrets flow in YOUR project

Step 1 — GitHub Actions Reads GitHub Secrets

GitHub Secrets → environment variables → Terraform variables

Step 2 — Terraform writes:

VPC, subnets, SGs
RDS
ECS cluster
Task definitions
ALB

Step 3 — Optional: Terraform can push secrets to AWS Secrets Manager

Then ECS Task Definitions read from Secrets Manager at runtime.

#7 — Interview-Level Summary

A Senior DevOps must know:

✔ GitHub Secrets

Used for pipeline-level secrets only.

✔ AWS Secrets Manager

For production runtime secrets.

✔ AWS Parameter Store

For configuration and non-secret values.

✔ HashiCorp Vault

Enterprise-grade, dynamic secrets, KMS integration.

✔ Terraform Secrets Handling

Never hardcode.
Use TF_VAR + Secrets Manager injection.

✔ ECS Secret Injection

ECS can read secrets directly
(no environment variables exposed).

🟦 B — Interview Cheat Sheet

Here are short, crisp answers:

❓What is GitHub Secrets?

Pipeline-only encrypted secret store.
Used to authenticate Terraform, Docker, AWS during CI/CD.

❓Why not store runtime secrets in GitHub?

Because GitHub Secrets only live during CI/CD.
Containers need secrets at runtime → use AWS Secrets Manager.

❓What is AWS Secrets Manager?

Fully managed encrypted secret store with rotation, IAM, audit logging.

❓What is Parameter Store?

Cheaper config store for non-secrets.

❓What is HashiCorp Vault?

Enterprise secret management offering dynamic credentials and zero-trust access.

🔥 2. WHY SECRETS MANAGEMENT IS CRITICAL

🟦 3. SECRET STORAGE OPTIONS DevOps MUST know

🟩 4. GITHUB SECRETS — Used for CI/CD Only

✔ Where used:

✔ What it stores:

✔ How it works:

✔ Security rules for senior DevOps:

❌ GitHub Secrets DO NOT replace:

🟥 5. AWS SECRETS MANAGER — Production-grade secret storage

✔ Where used:

✔ Features:

✔ Typical use cases:

✔ Access via IAM:

✔ Code example (ECS task environment):

❗ When to choose Secrets Manager:

🟨 6. AWS SSM PARAMETER STORE

✔ Cheaper alternative to Secrets Manager

✔ Good for Dev / QA / Non-critical secrets

✔ Use cases:

❗ NOT recommended

🟪 7. HASHICORP VAULT — The most advanced system

✔ Why Vault is used:

✔ Dynamic secrets example:

✔ Vault is used by:

🟦 8. Kubernetes Secrets (Optional but DevOps MUST know)

✔ Stored inside etcd (encrypted with KMS in prod)

✔ Used for:

✔ Mounted as:

🟫 9. Terraform & Secrets — Senior Level Knowledge

❗❗ Secrets MUST be passed via:

✔ Example bad code (DO NOT DO):

✔ Good:

✔ Best:

🟩 10. How Secrets Flow in a Real CI/CD Pipeline

Step 1:

Step 2:

Step 3:

Step 4:

Step 5:

🟥 11. What NOT TO DO (Senior DevOps Knowledge)

🟦 12. Interview-Level Explanation (You can say this)

*🔵 SECRETS FLOW — High-Level Diagram *

🔐 Explanation of Every Secret Component

#1 — GitHub Secrets (CI/CD)

Used in your project:

GitHub Secrets lifetime:

#2 — AWS Secrets Manager

What you store here:

Why AWS Secrets Manager is better than GitHub Secrets:

Benefits:

#3 — AWS Parameter Store (SSM)

When to use:

Sample:

#4 — HashiCorp Vault (Senior-level DevOps Topic)

Why Vault?

Vault is often used when:

#5 — Terraform and Secrets

#6 — How Secrets flow in YOUR project

Step 1 — GitHub Actions Reads GitHub Secrets

Step 2 — Terraform writes:

Step 3 — Optional: Terraform can push secrets to AWS Secrets Manager

#7 — Interview-Level Summary

✔ GitHub Secrets

✔ AWS Secrets Manager

✔ AWS Parameter Store

✔ HashiCorp Vault

✔ Terraform Secrets Handling

✔ ECS Secret Injection

🟦 B — Interview Cheat Sheet

❓What is GitHub Secrets?

❓Why not store runtime secrets in GitHub?

❓What is AWS Secrets Manager?

❓What is Parameter Store?

❓What is HashiCorp Vault?

❓How does Terraform handle secrets?

❓How does ECS access secrets?

🔵 SECRETS FLOW — High-Level Diagram