DEV Community

Chayan Samanta
Chayan Samanta

Posted on Edited on

DevOps Prep Short Notes + Examples

1. TERRAFORM

Workflow: init → fmt → validate → plan → apply → destroy

Example: remote backend + module + for_each

terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
  backend "s3" {
    bucket         = "my-tf-state"
    key            = "prod/network.tfstate"
    region         = "ap-south-1"
    dynamodb_table = "tf-lock"      # newer versions: use_lockfile = true
    encrypt        = true
  }
}

variable "buckets" { default = ["logs", "assets"] }

resource "aws_s3_bucket" "this" {
  for_each = toset(var.buckets)
  bucket   = "myapp-${each.key}"

  lifecycle { prevent_destroy = true }
}

module "vpc" {
  source = "git::https://github.com/org/tf-modules.git//vpc?ref=v1.2.0"
  cidr   = "10.0.0.0/16"
}

output "bucket_names" { value = [for b in aws_s3_bucket.this : b.id] }
Enter fullscreen mode Exit fullscreen mode

Handy snippets

# Rename without destroy
moved {
  from = aws_instance.old
  to   = aws_instance.new
}

# Import existing resource (TF 1.5+)
import {
  to = aws_s3_bucket.this["logs"]
  id = "myapp-logs"
}
Enter fullscreen mode Exit fullscreen mode
terraform plan -refresh-only          # detect drift
terraform apply -replace=aws_instance.web   # force recreate (replaces taint)
terraform state list | mv | rm
terraform force-unlock <LOCK_ID>
terraform plan -target=module.vpc     # emergency use only
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

Problem Fix
State lock error Confirm no run is active, then force-unlock
Drift plan -refresh-only, then import or apply
"Already exists" terraform import
Cycle error Break the circular reference (e.g., separate SG rules)
Plan wants to destroy and recreate Check for a rename (use moved) or an immutable-field change
Lost state Restore the previous S3 version
Provider upgrade broke code Pin versions, commit .terraform.lock.hcl

Q&A

  • count vs for_each? count is index-based, so removing item 0 shifts all the others and recreates them. for_each is key-based and stable.
  • Multiple envs? Separate state per env (folders or workspaces), shared modules, env.tfvars.
  • Secrets? Encrypted remote state, sensitive = true, fetch from a secrets manager, never commit tfstate.
  • Module best practices? Single responsibility, versioned, documented inputs and outputs, no hardcoding.
  • Protect prod? PR plan review, approval gate, prevent_destroy, least-privilege CI role.

2. KUBERNETES

Example: Deployment + Service + HPA

apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 3
  strategy:
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
      - name: web
        image: myrepo/web:1.0.3
        ports: [{ containerPort: 8080 }]
        resources:
          requests: { cpu: 100m, memory: 128Mi }
          limits:   { cpu: 500m, memory: 256Mi }
        readinessProbe:
          httpGet: { path: /ready, port: 8080 }
          initialDelaySeconds: 5
        livenessProbe:
          httpGet: { path: /health, port: 8080 }
          initialDelaySeconds: 15
        envFrom:
        - configMapRef: { name: web-config }
        - secretRef:    { name: web-secret }
---
apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  selector: { app: web }       # must match pod labels
  ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
Enter fullscreen mode Exit fullscreen mode

Troubleshooting commands

kubectl get pods -o wide
kubectl describe pod <pod>              # read Events first
kubectl logs <pod> --previous           # for crash loops
kubectl get events --sort-by=.lastTimestamp
kubectl get endpoints web               # empty = selector mismatch
kubectl top pod
kubectl rollout undo deployment/web
kubectl auth can-i get pods --as=user1  # RBAC check
kubectl run tmp --rm -it --image=busybox -- sh   # test DNS/connectivity
Enter fullscreen mode Exit fullscreen mode

Status → Cause

Status Cause and fix
Pending No resources, taints, unbound PVC → describe
CrashLoopBackOff App crash, bad env, failing probe → logs --previous
ImagePullBackOff Wrong tag, missing imagePullSecret
OOMKilled Raise memory limit or fix the leak
Running but 503 Readiness failing or no endpoints
Node NotReady kubelet down, disk/memory pressure, CNI
Evicted Node pressure; set proper requests/limits

Mini scenario

"Service is not reachable."

  1. kubectl get pods (are they Ready?)
  2. kubectl get endpoints (empty? label mismatch)
  3. Check targetPort vs containerPort
  4. Test from a debug pod: curl web:80
  5. Check NetworkPolicy and Ingress rules

Q&A

  • Liveness vs readiness vs startup? Restart the container / remove from traffic / protect slow-starting apps.
  • Deployment vs StatefulSet vs DaemonSet? Stateless / stable identity and storage / one pod per node.
  • ClusterIP vs NodePort vs LoadBalancer? Internal / node port / external cloud LB.
  • Requests vs limits? Requests are used for scheduling, limits are the hard cap (memory over limit means OOMKill, CPU over limit means throttling).
  • Zero-downtime deploy? Rolling update, readiness probe, maxUnavailable: 0, PDB, preStop hook.
  • ConfigMap vs Secret? Secrets are base64 only (not encrypted by default), so enable encryption at rest or use an external secrets operator.
  • RBAC? Role/ClusterRole + RoleBinding/ClusterRoleBinding + ServiceAccount.
  • Rollback? kubectl rollout undo or revert the Git commit (GitOps).
  • Helm vs Kustomize? Templating and packaging vs overlay patching.

3. DOCKER

Example: multi-stage, non-root

# Build stage
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Runtime stage
FROM node:20-alpine
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER node
EXPOSE 8080
HEALTHCHECK CMD wget -qO- http://localhost:8080/health || exit 1
CMD ["node", "dist/server.js"]
Enter fullscreen mode Exit fullscreen mode
docker build -t web:1.0 .
docker run -d -p 8080:8080 --name web web:1.0
docker logs -f web
docker exec -it web sh
docker inspect web | grep -i exitcode
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

  • Container exits immediately. Check docker logs and the exit code (137 = OOM/SIGKILL, 1 = app error, 127 = command not found).
  • Port not reachable. The app is bound to 127.0.0.1 instead of 0.0.0.0, or -p is missing.
  • Image too big. Multi-stage build, alpine/distroless, .dockerignore.
  • Slow builds. Put COPY package.json and the dependency install before COPY . to use the layer cache.

Q&A

  • CMD vs ENTRYPOINT? ENTRYPOINT is the fixed executable, CMD is the default args (overridable).
  • COPY vs ADD? Prefer COPY. ADD also extracts tars and fetches URLs.
  • Image vs container? Template vs running instance.
  • Volumes vs bind mounts? Docker-managed storage vs host path.
  • Reduce attack surface? Minimal base, non-root, scan with Trivy, pin tags (no latest).

4. CI/CD

GitHub Actions example

name: ci-cd
on:
  push: { branches: [main] }
  pull_request:

permissions:
  id-token: write      # OIDC, no stored cloud keys
  contents: read

jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test
      - run: docker build -t app:${{ github.sha }} .
      - name: Scan
        uses: aquasecurity/trivy-action@master
        with: { image-ref: "app:${{ github.sha }}", severity: "HIGH,CRITICAL", exit-code: 1 }

  deploy:
    needs: build-test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production      # manual approval gate
    steps:
      - run: echo "kubectl set image deploy/web web=app:${{ github.sha }}"
Enter fullscreen mode Exit fullscreen mode

Jenkinsfile (declarative)

pipeline {
  agent any
  stages {
    stage('Build')  { steps { sh 'docker build -t app:${BUILD_NUMBER} .' } }
    stage('Test')   { steps { sh 'npm test' } }
    stage('Deploy') {
      when { branch 'main' }
      steps { input 'Approve prod?'; sh './deploy.sh' }
    }
  }
  post { failure { echo 'Notify team' } }
}
Enter fullscreen mode Exit fullscreen mode

Terraform in CI (common flow)

PR → fmt/validate/tflint/checkov → plan (comment on PR) → merge → approval → apply

Troubleshooting

Problem Fix
Flaky pipeline Isolate flaky tests, check runner resources, registry rate limits
Auth failure to cloud Expired creds, OIDC trust or role mismatch
Jenkins agent offline Network, disk full, Java version, agent service
Deploy "green" but app broken Add smoke tests and health gating, then auto-rollback
Slow builds Cache deps and layers, parallel stages
Secret leaked in logs Use masked secrets, rotate immediately

Q&A

  • Describe your pipeline. Build → test → scan → push image → deploy dev → approval → prod → smoke test → notify.
  • Secrets in CI? Vault or secret manager, masked variables, OIDC, never in the repo.
  • Deployment strategies? Rolling (default), blue/green (instant switch back), canary (gradual %).
  • GitOps? Git is the source of truth, and ArgoCD/Flux pulls and reconciles the cluster state.
  • CI vs CD vs Continuous Deployment? Integrate and test / deliver with a manual gate / fully automated to prod.

5. MONITORING AND LOGGING

Golden signals: Latency, Traffic, Errors, Saturation

PromQL examples

# Error rate %
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100

# p95 latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Pod restarts
increase(kube_pod_container_status_restarts_total[1h]) > 3
Enter fullscreen mode Exit fullscreen mode

Alert rule

groups:
- name: app
  rules:
  - alert: HighErrorRate
    expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
    for: 5m
    labels: { severity: critical }
    annotations:
      summary: "5xx rate above 5%"
      runbook: "https://wiki/runbooks/high-5xx"
Enter fullscreen mode Exit fullscreen mode

Q&A

  • Prometheus architecture? Server scrapes exporters, stores a TSDB, Alertmanager routes alerts, Grafana visualizes.
  • Metrics vs logs vs traces? What is happening / why / where in the request path.
  • ELK flow? App → Filebeat/Fluent Bit → Logstash → Elasticsearch → Kibana.
  • Good alert? Actionable, symptom-based, has a runbook, avoids fatigue.
  • High CPU on server? Check the dashboard, top, find the process, correlate with deploys or traffic, then scale or fix.
  • Pushgateway? For short-lived batch jobs only.

6. SECURITY

  • Least privilege: only the permissions needed.
  • Shift-left: secret scanning, SAST, dependency scan, image scan (Trivy), IaC scan (checkov/tfsec).
  • K8s: RBAC, NetworkPolicy, non-root pods, Pod Security Standards, secrets via External Secrets/Vault.

Example NetworkPolicy (default deny)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
Enter fullscreen mode Exit fullscreen mode
  • Leaked credentials? Revoke and rotate, audit logs for misuse, remove from Git history, add secret scanning.

7. BACKUP, DR, PATCHING

  • RPO is the max data loss, RTO is the max downtime.
  • DR tiers: Backup & Restore → Pilot Light → Warm Standby → Active/Active.
  • K8s backup: Velero plus etcd snapshots.
  • Patching: Test in non-prod, roll in batches, keep a rollback path, use immutable images (Packer).
  • Rule: an untested backup is not a backup, so run restore drills.

8. INCIDENT MANAGEMENT AND RCA

Flow: Detect → Triage → Communicate → Mitigate → Resolve → RCA → Prevent

First response: What changed recently? If a deploy caused it, roll back first and investigate later.

RCA template

Impact:      30 min outage, 20% of users
Timeline:    10:00 deploy → 10:05 alerts → 10:12 rollback → 10:30 stable
Root cause:  Missing DB index; query timeouts under load
5 Whys:      ...
Fix:         Added index
Prevention:  Load test in CI, slow-query alert (owner, date)
Enter fullscreen mode Exit fullscreen mode

STAR story template

Situation → Task → Action (what you did) → Result (numbers) → Learned.


9. RAPID-FIRE TROUBLESHOOTING

Scenario First steps
Pod CrashLoopBackOff logs --previous, describe, check config/probes
App slow after deploy Compare metrics, check logs and DB, roll back
Terraform apply failed midway Read the error, state is partial, fix and re-apply
Pipeline can't push the image Registry creds, rate limit, network
Intermittent 5xx Check readiness, resource limits, pod restarts, upstream deps
Disk full on Linux df -h, du -sh /*, rotate logs, extend volume
DNS issue in cluster nslookup from a debug pod, check CoreDNS pods and logs

Linux quick-hit commands

top / htop / free -m / df -h / du -sh *
ss -tulnp               # open ports
journalctl -u <svc> -f
curl -v / dig / traceroute
systemctl status <svc>
Enter fullscreen mode Exit fullscreen mode

10. FINAL TIPS

  • Answer pattern: concept → short example → how you'd troubleshoot.
  • Say your process aloud: "First I'd check X, then Y."
  • Have 3 stories ready: an outage, an automation, an optimization.
  • Practice hands-on: break a pod, fix it; break Terraform state, recover it.

Want me to run a mock interview (I ask, you answer, I score you), or go deeper on Terraform labs or K8s scenarios?

Top comments (0)