1. TERRAFORM
Workflow: init → fmt → validate → plan → apply → destroy
Example: remote backend + module + for_each
terraform {
required_version = ">= 1.5"
required_providers {
aws = { source = "hashicorp/aws", version = "~> 5.0" }
}
backend "s3" {
bucket = "my-tf-state"
key = "prod/network.tfstate"
region = "ap-south-1"
dynamodb_table = "tf-lock" # newer versions: use_lockfile = true
encrypt = true
}
}
variable "buckets" { default = ["logs", "assets"] }
resource "aws_s3_bucket" "this" {
for_each = toset(var.buckets)
bucket = "myapp-${each.key}"
lifecycle { prevent_destroy = true }
}
module "vpc" {
source = "git::https://github.com/org/tf-modules.git//vpc?ref=v1.2.0"
cidr = "10.0.0.0/16"
}
output "bucket_names" { value = [for b in aws_s3_bucket.this : b.id] }
Handy snippets
# Rename without destroy
moved {
from = aws_instance.old
to = aws_instance.new
}
# Import existing resource (TF 1.5+)
import {
to = aws_s3_bucket.this["logs"]
id = "myapp-logs"
}
terraform plan -refresh-only # detect drift
terraform apply -replace=aws_instance.web # force recreate (replaces taint)
terraform state list | mv | rm
terraform force-unlock <LOCK_ID>
terraform plan -target=module.vpc # emergency use only
Troubleshooting
| Problem | Fix |
|---|---|
| State lock error | Confirm no run is active, then force-unlock
|
| Drift |
plan -refresh-only, then import or apply |
| "Already exists" | terraform import |
| Cycle error | Break the circular reference (e.g., separate SG rules) |
| Plan wants to destroy and recreate | Check for a rename (use moved) or an immutable-field change |
| Lost state | Restore the previous S3 version |
| Provider upgrade broke code | Pin versions, commit .terraform.lock.hcl
|
Q&A
-
count vs for_each?
countis index-based, so removing item 0 shifts all the others and recreates them.for_eachis key-based and stable. -
Multiple envs? Separate state per env (folders or workspaces), shared modules,
env.tfvars. -
Secrets? Encrypted remote state,
sensitive = true, fetch from a secrets manager, never commit tfstate. - Module best practices? Single responsibility, versioned, documented inputs and outputs, no hardcoding.
-
Protect prod? PR plan review, approval gate,
prevent_destroy, least-privilege CI role.
2. KUBERNETES
Example: Deployment + Service + HPA
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
replicas: 3
strategy:
rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
selector:
matchLabels: { app: web }
template:
metadata:
labels: { app: web }
spec:
containers:
- name: web
image: myrepo/web:1.0.3
ports: [{ containerPort: 8080 }]
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: 500m, memory: 256Mi }
readinessProbe:
httpGet: { path: /ready, port: 8080 }
initialDelaySeconds: 5
livenessProbe:
httpGet: { path: /health, port: 8080 }
initialDelaySeconds: 15
envFrom:
- configMapRef: { name: web-config }
- secretRef: { name: web-secret }
---
apiVersion: v1
kind: Service
metadata: { name: web }
spec:
selector: { app: web } # must match pod labels
ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
Troubleshooting commands
kubectl get pods -o wide
kubectl describe pod <pod> # read Events first
kubectl logs <pod> --previous # for crash loops
kubectl get events --sort-by=.lastTimestamp
kubectl get endpoints web # empty = selector mismatch
kubectl top pod
kubectl rollout undo deployment/web
kubectl auth can-i get pods --as=user1 # RBAC check
kubectl run tmp --rm -it --image=busybox -- sh # test DNS/connectivity
Status → Cause
| Status | Cause and fix |
|---|---|
| Pending | No resources, taints, unbound PVC → describe
|
| CrashLoopBackOff | App crash, bad env, failing probe → logs --previous
|
| ImagePullBackOff | Wrong tag, missing imagePullSecret
|
| OOMKilled | Raise memory limit or fix the leak |
| Running but 503 | Readiness failing or no endpoints |
| Node NotReady | kubelet down, disk/memory pressure, CNI |
| Evicted | Node pressure; set proper requests/limits |
Mini scenario
"Service is not reachable."
kubectl get pods(are they Ready?)kubectl get endpoints(empty? label mismatch)- Check
targetPortvscontainerPort- Test from a debug pod:
curl web:80- Check NetworkPolicy and Ingress rules
Q&A
- Liveness vs readiness vs startup? Restart the container / remove from traffic / protect slow-starting apps.
- Deployment vs StatefulSet vs DaemonSet? Stateless / stable identity and storage / one pod per node.
- ClusterIP vs NodePort vs LoadBalancer? Internal / node port / external cloud LB.
- Requests vs limits? Requests are used for scheduling, limits are the hard cap (memory over limit means OOMKill, CPU over limit means throttling).
-
Zero-downtime deploy? Rolling update, readiness probe,
maxUnavailable: 0, PDB,preStophook. - ConfigMap vs Secret? Secrets are base64 only (not encrypted by default), so enable encryption at rest or use an external secrets operator.
- RBAC? Role/ClusterRole + RoleBinding/ClusterRoleBinding + ServiceAccount.
-
Rollback?
kubectl rollout undoor revert the Git commit (GitOps). - Helm vs Kustomize? Templating and packaging vs overlay patching.
3. DOCKER
Example: multi-stage, non-root
# Build stage
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
# Runtime stage
FROM node:20-alpine
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER node
EXPOSE 8080
HEALTHCHECK CMD wget -qO- http://localhost:8080/health || exit 1
CMD ["node", "dist/server.js"]
docker build -t web:1.0 .
docker run -d -p 8080:8080 --name web web:1.0
docker logs -f web
docker exec -it web sh
docker inspect web | grep -i exitcode
Troubleshooting
-
Container exits immediately. Check
docker logsand the exit code (137= OOM/SIGKILL,1= app error,127= command not found). -
Port not reachable. The app is bound to
127.0.0.1instead of0.0.0.0, or-pis missing. -
Image too big. Multi-stage build, alpine/distroless,
.dockerignore. -
Slow builds. Put
COPY package.jsonand the dependency install beforeCOPY .to use the layer cache.
Q&A
- CMD vs ENTRYPOINT? ENTRYPOINT is the fixed executable, CMD is the default args (overridable).
- COPY vs ADD? Prefer COPY. ADD also extracts tars and fetches URLs.
- Image vs container? Template vs running instance.
- Volumes vs bind mounts? Docker-managed storage vs host path.
-
Reduce attack surface? Minimal base, non-root, scan with Trivy, pin tags (no
latest).
4. CI/CD
GitHub Actions example
name: ci-cd
on:
push: { branches: [main] }
pull_request:
permissions:
id-token: write # OIDC, no stored cloud keys
contents: read
jobs:
build-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci && npm test
- run: docker build -t app:${{ github.sha }} .
- name: Scan
uses: aquasecurity/trivy-action@master
with: { image-ref: "app:${{ github.sha }}", severity: "HIGH,CRITICAL", exit-code: 1 }
deploy:
needs: build-test
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: production # manual approval gate
steps:
- run: echo "kubectl set image deploy/web web=app:${{ github.sha }}"
Jenkinsfile (declarative)
pipeline {
agent any
stages {
stage('Build') { steps { sh 'docker build -t app:${BUILD_NUMBER} .' } }
stage('Test') { steps { sh 'npm test' } }
stage('Deploy') {
when { branch 'main' }
steps { input 'Approve prod?'; sh './deploy.sh' }
}
}
post { failure { echo 'Notify team' } }
}
Terraform in CI (common flow)
PR → fmt/validate/tflint/checkov → plan (comment on PR) → merge → approval → apply
Troubleshooting
| Problem | Fix |
|---|---|
| Flaky pipeline | Isolate flaky tests, check runner resources, registry rate limits |
| Auth failure to cloud | Expired creds, OIDC trust or role mismatch |
| Jenkins agent offline | Network, disk full, Java version, agent service |
| Deploy "green" but app broken | Add smoke tests and health gating, then auto-rollback |
| Slow builds | Cache deps and layers, parallel stages |
| Secret leaked in logs | Use masked secrets, rotate immediately |
Q&A
- Describe your pipeline. Build → test → scan → push image → deploy dev → approval → prod → smoke test → notify.
- Secrets in CI? Vault or secret manager, masked variables, OIDC, never in the repo.
- Deployment strategies? Rolling (default), blue/green (instant switch back), canary (gradual %).
- GitOps? Git is the source of truth, and ArgoCD/Flux pulls and reconciles the cluster state.
- CI vs CD vs Continuous Deployment? Integrate and test / deliver with a manual gate / fully automated to prod.
5. MONITORING AND LOGGING
Golden signals: Latency, Traffic, Errors, Saturation
PromQL examples
# Error rate %
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100
# p95 latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# Pod restarts
increase(kube_pod_container_status_restarts_total[1h]) > 3
Alert rule
groups:
- name: app
rules:
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels: { severity: critical }
annotations:
summary: "5xx rate above 5%"
runbook: "https://wiki/runbooks/high-5xx"
Q&A
- Prometheus architecture? Server scrapes exporters, stores a TSDB, Alertmanager routes alerts, Grafana visualizes.
- Metrics vs logs vs traces? What is happening / why / where in the request path.
- ELK flow? App → Filebeat/Fluent Bit → Logstash → Elasticsearch → Kibana.
- Good alert? Actionable, symptom-based, has a runbook, avoids fatigue.
-
High CPU on server? Check the dashboard,
top, find the process, correlate with deploys or traffic, then scale or fix. - Pushgateway? For short-lived batch jobs only.
6. SECURITY
- Least privilege: only the permissions needed.
- Shift-left: secret scanning, SAST, dependency scan, image scan (Trivy), IaC scan (checkov/tfsec).
- K8s: RBAC, NetworkPolicy, non-root pods, Pod Security Standards, secrets via External Secrets/Vault.
Example NetworkPolicy (default deny)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny }
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
- Leaked credentials? Revoke and rotate, audit logs for misuse, remove from Git history, add secret scanning.
7. BACKUP, DR, PATCHING
- RPO is the max data loss, RTO is the max downtime.
- DR tiers: Backup & Restore → Pilot Light → Warm Standby → Active/Active.
- K8s backup: Velero plus etcd snapshots.
- Patching: Test in non-prod, roll in batches, keep a rollback path, use immutable images (Packer).
- Rule: an untested backup is not a backup, so run restore drills.
8. INCIDENT MANAGEMENT AND RCA
Flow: Detect → Triage → Communicate → Mitigate → Resolve → RCA → Prevent
First response: What changed recently? If a deploy caused it, roll back first and investigate later.
RCA template
Impact: 30 min outage, 20% of users
Timeline: 10:00 deploy → 10:05 alerts → 10:12 rollback → 10:30 stable
Root cause: Missing DB index; query timeouts under load
5 Whys: ...
Fix: Added index
Prevention: Load test in CI, slow-query alert (owner, date)
STAR story template
Situation → Task → Action (what you did) → Result (numbers) → Learned.
9. RAPID-FIRE TROUBLESHOOTING
| Scenario | First steps |
|---|---|
| Pod CrashLoopBackOff |
logs --previous, describe, check config/probes |
| App slow after deploy | Compare metrics, check logs and DB, roll back |
| Terraform apply failed midway | Read the error, state is partial, fix and re-apply |
| Pipeline can't push the image | Registry creds, rate limit, network |
| Intermittent 5xx | Check readiness, resource limits, pod restarts, upstream deps |
| Disk full on Linux |
df -h, du -sh /*, rotate logs, extend volume |
| DNS issue in cluster |
nslookup from a debug pod, check CoreDNS pods and logs |
Linux quick-hit commands
top / htop / free -m / df -h / du -sh *
ss -tulnp # open ports
journalctl -u <svc> -f
curl -v / dig / traceroute
systemctl status <svc>
10. FINAL TIPS
- Answer pattern: concept → short example → how you'd troubleshoot.
- Say your process aloud: "First I'd check X, then Y."
- Have 3 stories ready: an outage, an automation, an optimization.
- Practice hands-on: break a pod, fix it; break Terraform state, recover it.
Want me to run a mock interview (I ask, you answer, I score you), or go deeper on Terraform labs or K8s scenarios?
Top comments (0)