InstaDevOps

Posted on Nov 16 • Originally published at instadevops.com

Container Security: From Development to Production

#security #docker #kubernetes #devops

Introduction

Containers have revolutionized application deployment, but they've also introduced new security challenges. While containers provide isolation, they're not a security silver bullet. A misconfigured container can expose your entire infrastructure to attack, leak sensitive data, or become a foothold for lateral movement within your network.

In this comprehensive guide, we'll explore container security best practices across the entire software development lifecycle—from writing Dockerfiles to running containers in production. Whether you're running Docker locally or managing thousands of containers in Kubernetes, these practices will help you build a robust security posture.

Understanding Container Security Risks

Before diving into solutions, let's understand the unique security challenges containers present:

Image Vulnerabilities

Container images often contain outdated packages with known vulnerabilities. A study by Snyk found that 75% of Docker Hub images contain at least one critical vulnerability.

Overprivileged Containers

Many containers run as root by default, violating the principle of least privilege. If compromised, attackers gain root access within the container.

Insecure Image Registries

Public registries contain malicious images disguised as legitimate software. Even private registries can be misconfigured to allow unauthorized access.

Secrets in Images

Developers accidentally commit credentials, API keys, and certificates into container images, exposing them to anyone with image access.

Container Escape

While rare, vulnerabilities in container runtimes can allow attackers to escape containers and access the host system.

Supply Chain Attacks

Base images from untrusted sources might contain backdoors or malware, compromising your entire application stack.

Secure Docker Image Development

Start with Minimal Base Images

Use minimal base images to reduce attack surface. Distroless or Alpine images contain far fewer packages than full OS images.

# Bad: Full Ubuntu image (78MB, hundreds of packages)
FROM ubuntu:22.04

# Better: Alpine image (5MB, minimal packages)
FROM alpine:3.18

# Best: Distroless image (smallest, no shell, no package manager)
FROM gcr.io/distroless/nodejs18-debian11

Multi-Stage Builds for Smaller Images

Use multi-stage builds to keep build tools out of production images:

# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build

# Production stage
FROM gcr.io/distroless/nodejs18-debian11
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package*.json ./
USER nonroot
CMD ["node", "dist/index.js"]

Never Run as Root

Create and use a non-root user in your Dockerfile:

FROM node:18-alpine

# Create app user
RUN addgroup -g 1001 -S appuser && \
    adduser -u 1001 -S appuser -G appuser

# Set ownership
WORKDIR /app
COPY --chown=appuser:appuser . .

# Install dependencies as root (needed for npm install)
RUN npm ci --only=production

# Switch to non-root user
USER appuser

# Run as non-root
CMD ["node", "index.js"]

Scan Images for Vulnerabilities

Integrate vulnerability scanning into your CI/CD pipeline:

# Using Trivy
docker run aquasec/trivy image myapp:latest

# Using Snyk
snyk container test myapp:latest

# Using Grype
grype myapp:latest

# GitHub Actions example
name: Container Security Scan

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

      - name: Fail on high severity vulnerabilities
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          exit-code: '1'
          severity: 'CRITICAL,HIGH'

Don't Store Secrets in Images

Never bake secrets into images. Use environment variables, secret management tools, or mount secrets at runtime:

# Bad: Secret hardcoded in image
ENV API_KEY="sk-1234567890abcdef"

# Bad: Secret copied into image
COPY secrets.env /app/

# Good: Secret passed at runtime
# docker run -e API_KEY="$API_KEY" myapp

# Better: Secret from secrets management
# Kubernetes Secret, AWS Secrets Manager, HashiCorp Vault

# Kubernetes: Mount secrets as environment variables
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: API_KEY
      valueFrom:
        secretKeyRef:
          name: myapp-secrets
          key: api-key

Use .dockerignore

Prevent sensitive files from being copied into images:

# .dockerignore
.git
.env
.env.local
*.log
node_modules
npm-debug.log
secrets/
*.pem
*.key
.aws
.gcp

Pin Image Versions

Always use specific image versions, never latest:

# Bad: Unpredictable, can break without warning
FROM node:latest

# Bad: Still somewhat unpredictable
FROM node:18

# Good: Specific version
FROM node:18.17.1

# Best: Specific version + digest (immutable)
FROM node:18.17.1-alpine@sha256:f1657204d3463bce763cefa5b25e48c28af6eb183b4b6b06f7d64a8e2f18314

Limit Image Layer Count

Combine RUN commands to reduce layers and image size:

# Bad: Many layers
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y git
RUN apt-get clean
RUN rm -rf /var/lib/apt/lists/*

# Good: Single layer
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl git && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Container Runtime Security

Use Read-Only Filesystems

Make container filesystems read-only to prevent tampering:

# Docker Compose
services:
  app:
    image: myapp:latest
    read_only: true
    tmpfs:
      - /tmp
      - /var/run

# Kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: tmp
    emptyDir: {}

Drop Unnecessary Capabilities

Linux capabilities allow fine-grained privilege control. Drop all capabilities, then add only what's needed:

# Kubernetes Pod with minimal capabilities
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      capabilities:
        drop:
        - ALL
        add:
        - NET_BIND_SERVICE  # Only if binding to ports <1024
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      runAsUser: 1001
      seccompProfile:
        type: RuntimeDefault

Resource Limits

Prevent container resource exhaustion attacks:

# Docker Compose
services:
  app:
    image: myapp:latest
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M
        reservations:
          cpus: '0.5'
          memory: 256M

# Kubernetes
resources:
  requests:
    memory: "256Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"
    cpu: "1000m"

Use Security Profiles

AppArmor and Seccomp restrict system calls containers can make:

# Kubernetes with AppArmor and Seccomp
apiVersion: v1
kind: Pod
metadata:
  name: myapp
  annotations:
    container.apparmor.security.beta.kubernetes.io/app: runtime/default
spec:
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      seccompProfile:
        type: RuntimeDefault

Network Segmentation

Isolate containers with network policies:

# Kubernetes NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: myapp-netpol
spec:
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
  - Ingress
  - Egress
  ingress:
  # Only allow traffic from specific pods
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  # Only allow traffic to database
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  # Allow DNS
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53

Registry Security

Use Private Registries

Host images in private registries with access control:

# AWS ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  123456789.dkr.ecr.us-east-1.amazonaws.com

# Google Container Registry
gcloud auth configure-docker

# Azure Container Registry
az acr login --name myregistry

Sign Images

Use Docker Content Trust or Notary to sign images:

# Enable Docker Content Trust
export DOCKER_CONTENT_TRUST=1

# Push signed image
docker push myregistry.com/myapp:v1.0

# Verify signature on pull
docker pull myregistry.com/myapp:v1.0

Implement Image Scanning in Registry

Many registries provide built-in vulnerability scanning:

# AWS ECR: Enable scanning on push
aws ecr put-image-scanning-configuration \
  --repository-name myapp \
  --image-scanning-configuration scanOnPush=true

# Google Artifact Registry: Enable vulnerability scanning
gcloud artifacts repositories update myrepo \
  --location=us-central1 \
  --enable-scanning

Use Admission Controllers

Prevent unsigned or vulnerable images from running:

# OPA Gatekeeper policy: Only allow images from approved registries
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: allowed-repos
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    - apiGroups: ["apps"]
      kinds: ["Deployment", "StatefulSet"]
  parameters:
    repos:
    - "myregistry.com/"
    - "gcr.io/myproject/"

Kubernetes-Specific Security

Pod Security Standards

Enforce pod security policies cluster-wide:

# Restricted Pod Security Standard (most secure)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

RBAC for Least Privilege

Grant minimal permissions to service accounts:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: myapp-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["myapp-secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-rolebinding
  namespace: production
subjects:
- kind: ServiceAccount
  name: myapp-sa
  namespace: production
roleRef:
  kind: Role
  name: myapp-role
  apiGroup: rbac.authorization.k8s.io

Secrets Encryption at Rest

Encrypt Kubernetes secrets in etcd:

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <BASE64_ENCODED_SECRET>
  - identity: {}

Audit Logging

Enable comprehensive audit logging:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log admin actions
- level: RequestResponse
  users: ["admin"]
# Log secret access
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets"]
# Log pod exec/attach
- level: Request
  verbs: ["exec", "attach"]
  resources:
  - group: ""
    resources: ["pods/exec", "pods/attach"]

Runtime Security Monitoring

Use Falco for Threat Detection

Falco monitors kernel events to detect anomalous behavior:

# Falco rule: Detect shell in container
- rule: Terminal Shell in Container
  desc: A shell was spawned in a container
  condition: >
    spawned_process and
    container and
    proc.name in (shell_binaries)
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname
    cmdline=%proc.cmdline)
  priority: WARNING

# Detect outbound connections to suspicious IPs
- rule: Outbound Connection to Suspicious IP
  desc: Container connected to known malicious IP
  condition: >
    outbound and
    container and
    fd.sip in (suspicious_ips)
  output: >
    Suspicious outbound connection
    (container=%container.name
    destination=%fd.sip:%fd.sport
    command=%proc.cmdline)
  priority: CRITICAL

Container Runtime Security Tools

# Deploy Falco as DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: falco
  namespace: falco
spec:
  selector:
    matchLabels:
      app: falco
  template:
    metadata:
      labels:
        app: falco
    spec:
      serviceAccount: falco
      hostNetwork: true
      hostPID: true
      containers:
      - name: falco
        image: falcosecurity/falco:latest
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /host/var/run/docker.sock
          name: docker-socket
        - mountPath: /host/dev
          name: dev-fs
        - mountPath: /host/proc
          name: proc-fs
          readOnly: true
      volumes:
      - name: docker-socket
        hostPath:
          path: /var/run/docker.sock
      - name: dev-fs
        hostPath:
          path: /dev
      - name: proc-fs
        hostPath:
          path: /proc

Secrets Management

External Secrets Operator

Sync secrets from external vaults into Kubernetes:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-password
    remoteRef:
      key: prod/myapp/db-password
  - secretKey: api-key
    remoteRef:
      key: prod/myapp/api-key

Sealed Secrets

Encrypt secrets in Git for GitOps workflows:

# Encrypt secret
echo -n 'supersecret' | kubectl create secret generic mysecret \
  --dry-run=client --from-file=password=/dev/stdin -o yaml | \
  kubeseal -o yaml > mysealedsecret.yaml

# Commit encrypted secret to Git
git add mysealedsecret.yaml
git commit -m "Add sealed secret"

Security Scanning and Compliance

Comprehensive Security Scanning Pipeline

# Complete security scanning pipeline
name: Security Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # 1. Scan source code for secrets
      - name: Secret scanning
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./

      # 2. SAST - Static Application Security Testing
      - name: Run Semgrep
        uses: returntocorp/semgrep-action@v1

      # 3. Dependency scanning
      - name: Run Snyk
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

      # 4. Build image
      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .

      # 5. Container scanning
      - name: Run Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'

      # 6. IaC scanning
      - name: Run Checkov
        uses: bridgecrewio/checkov-action@master
        with:
          directory: k8s/
          framework: kubernetes

      # 7. License compliance
      - name: License scanning
        run: |
          npm install -g license-checker
          license-checker --production --onlyAllow 'MIT;Apache-2.0;BSD-3-Clause'

Compliance and Hardening

CIS Benchmark Compliance

Follow CIS Docker and Kubernetes benchmarks:

# Run Docker Bench Security
docker run --rm --net host --pid host --userns host --cap-add audit_control \
  -v /var/lib:/var/lib \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /etc:/etc \
  docker/docker-bench-security

# Run kube-bench for Kubernetes
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench

Incident Response

Container Forensics

When a container is compromised:

# 1. Don't delete the container immediately
# Pause it to preserve state
docker pause <container_id>

# 2. Export filesystem for analysis
docker export <container_id> > compromised-container.tar

# 3. Inspect container configuration
docker inspect <container_id> > container-config.json

# 4. Check logs
docker logs <container_id> > container-logs.txt

# 5. Review network connections
docker exec <container_id> netstat -antp

# 6. Check running processes
docker top <container_id>

# 7. After analysis, kill and remove
docker kill <container_id>
docker rm <container_id>

Best Practices Checklist

Development Phase

[ ] Use minimal base images (Alpine, Distroless)
[ ] Implement multi-stage builds
[ ] Run containers as non-root user
[ ] Pin image versions with digests
[ ] Use .dockerignore to exclude sensitive files
[ ] Never commit secrets to images
[ ] Scan images for vulnerabilities
[ ] Keep images under 500MB
[ ] Document security considerations

Build Phase

[ ] Automated vulnerability scanning in CI/CD
[ ] SAST scanning for code vulnerabilities
[ ] Dependency scanning for vulnerable packages
[ ] Secret scanning to prevent leaks
[ ] License compliance checking
[ ] Sign images before pushing to registry

Registry Phase

[ ] Use private registries with authentication
[ ] Enable vulnerability scanning on push
[ ] Implement image retention policies
[ ] Regular security audits of registry access
[ ] Enable image signing and verification

Runtime Phase

[ ] Use read-only filesystems
[ ] Drop unnecessary Linux capabilities
[ ] Enable security profiles (AppArmor/Seccomp)
[ ] Implement resource limits
[ ] Use network policies for isolation
[ ] Enable audit logging
[ ] Deploy runtime security monitoring (Falco)
[ ] Regular security updates and patching

Kubernetes Phase

[ ] Enforce Pod Security Standards
[ ] Implement RBAC with least privilege
[ ] Encrypt secrets at rest
[ ] Use admission controllers
[ ] Enable audit logging
[ ] Regular security updates
[ ] Network policies for pod isolation
[ ] Use external secrets management

Conclusion

Container security is not a one-time task but an ongoing process that spans the entire software development lifecycle. By implementing security best practices from development through production, you can significantly reduce your attack surface and protect your infrastructure.

Key takeaways:

Shift Security Left: Catch vulnerabilities early in development
Defense in Depth: Layer multiple security controls
Principle of Least Privilege: Grant minimal necessary permissions
Automate Security: Make security checks part of CI/CD
Monitor Continuously: Detect and respond to threats in real-time
Stay Updated: Regularly patch and update dependencies

Start with the basics—minimal images, non-root users, and vulnerability scanning—then progressively add advanced controls like admission controllers, runtime security monitoring, and comprehensive audit logging.

Need help implementing container security? InstaDevOps provides expert consulting and implementation for container security, Kubernetes hardening, and compliance. Contact us for a free consultation.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups and scale-ups build production-ready infrastructure without the overhead of a full-time DevOps team.

Our Services: