Security Considerations for Multi-Cluster Cloud Architecture (HA EKS with Databases)

Running a highly available multi-cluster EKS architecture brings powerful benefits—zero downtime, disaster recovery, and global scalability. But it also multiplies your security challenges.

Securing a single EKS cluster is already complex. Add multiple clusters across regions, databases with sensitive data, and cross-cluster communication, and the attack surface grows significantly. One misconfigured security group or exposed secret can compromise your entire infrastructure.

This guide covers essential security considerations for multi-cluster architectures: network isolation, encryption, IAM management, secrets handling, and incident response. We'll focus on practical measures that protect your infrastructure without sacrificing performance or availability.

Let's build a secure, highly available system.

1. Network Security & Isolation

VPC Architecture

• Separate VPCs per cluster or use shared VPC with isolated subnets
• Private subnets for EKS nodes and databases (no direct internet access)
• Public subnets only for load balancers and NAT gateways
• Implement VPC peering or AWS Transit Gateway for inter-cluster communication
• Use separate VPCs per environment (dev, staging, production)

Network Segmentation

• Production VPC-1 (Region A)

  • Public Subnets: ALB only
  • Private Subnets: EKS Nodes
  • Database Subnets: RDS/Aurora (isolated)

• Production VPC-2 (Region B)

  • Public Subnets: ALB only
  • Private Subnets: EKS Nodes
  • Database Subnets: RDS/Aurora (isolated)

Security Groups

• Principle of least privilege - only allow necessary ports
• Database security groups: Only allow traffic from EKS node security groups
• EKS control plane: Restrict API access to specific CIDR ranges
• No 0.0.0.0/0 rules except for outbound NAT traffic
• Document and regularly audit security group rules

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-to-db
spec:
  podSelector:
    matchLabels:
      app: backend
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: database
    ports:
    - protocol: TCP
      port: 5432

---

2. Identity & Access Management (IAM)

Cluster Access Control

• AWS IAM authentication for cluster access via aws-auth ConfigMap
• Never use permanent credentials in pods or containers
• Implement IAM Roles for Service Accounts (IRSA) for pod-level permissions
• Use AWS SSO/IAM Identity Center for human access
• Separate IAM roles for different teams/applications
• Enable MFA for all human users

IRSA (IAM Roles for Service Accounts)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/app-role
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      serviceAccountName: app-sa
      containers:
      - name: app
        image: myapp:latest

Kubernetes RBAC

• Role-Based Access Control (RBAC) for fine-grained permissions
• Namespace isolation - separate namespaces per team/application
• Principle of least privilege - minimal permissions needed
• ClusterRoles for cluster-wide resources (use sparingly)
• Roles for namespace-scoped resources
• Regular RBAC audits

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: app-prod
rules:
- apiGroups: ["", "apps"]
  resources: ["pods", "deployments", "services"]
  verbs: ["get", "list", "watch"]
# Read-only access, no delete/update

Database Access

• IAM Database Authentication for RDS/Aurora (where possible)
• Avoid hardcoded credentials - use Secrets Manager or Parameter Store
• Rotate credentials regularly (automated rotation)
• Separate database users per application/service
• Read-only replicas for non-critical workloads

---

3. Secrets Management

Never Store Secrets in Code or ConfigMaps
❌ Bad: Secrets in environment variables or ConfigMaps
✅ Good: External secrets management

AWS Secrets Manager / Parameter Store

• Use External Secrets Operator or Secrets Store CSI Driver
• Automatic rotation enabled
• Encryption at rest with KMS
• Audit access via CloudTrail

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
  target:
    name: db-secret
  data:
  - secretKey: password
    remoteRef:
      key: prod/db/password

Alternative: HashiCorp Vault

• Centralized secrets management across clusters
• Dynamic secrets generation
• Lease-based credentials
• Fine-grained access policies

---

4. Encryption

Data at Rest

• EKS etcd encryption using AWS KMS
• EBS volumes encrypted (gp3 with KMS)
• RDS/Aurora encryption enabled with KMS
• S3 encryption (SSE-S3 or SSE-KMS)
• Use customer-managed KMS keys for compliance requirements
• Separate KMS keys per environment/cluster

Data in Transit

• TLS/SSL everywhere:

  • ALB → Pods (via Ingress with TLS)
  • Pod → Pod (service mesh or mTLS)
  • Application → Database (SSL/TLS enforced)

• Certificate management with AWS Certificate Manager or cert-manager

• mTLS with service mesh (Istio, Linkerd, AWS App Mesh)
  

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
spec:
  tls:
  - hosts:
    - app.example.com

---

5. Pod Security

Pod Security Standards

• Enforce restricted pod security standards
• No privileged containers unless absolutely necessary
• Read-only root filesystem where possible
• Non-root users for containers
• Drop all capabilities and add only required ones

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL

Image Security

• Scan images for vulnerabilities (Amazon ECR scanning, Trivy, Snyk)
• Use minimal base images (distroless, Alpine)
• Pin image versions - never use :latest
• Sign and verify images (Sigstore/Cosign, Notary)
• Private container registry (Amazon ECR with VPC endpoints)

• Image pull secrets for private registries
  Admission Controllers

• OPA/Gatekeeper or Kyverno for policy enforcement

• Enforce security policies:

• No privileged pods

• Required resource limits

• Approved registries only

• Required security contexts
  

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged
spec:
  validationFailureAction: enforce
  rules:
  - name: check-privileged
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Privileged containers are not allowed"
      pattern:
        spec:
          containers:
          - securityContext:
              privileged: false

---

6. Multi-Cluster Security

Cluster Isolation

• Separate clusters for different security zones (DMZ, internal, data)
• Separate clusters per environment (never share prod and non-prod)
• Separate AWS accounts per environment (AWS Organizations)
• Service Control Policies (SCPs) to restrict actions at account level

Cross-Cluster Communication

• Service mesh for secure cross-cluster communication (Istio multi-cluster)
• VPC peering or Transit Gateway with strict security groups
• mTLS for service-to-service authentication
• API Gateway or Internal Load Balancer as entry points
• Zero-trust networking - verify every request

DNS Security

• Private Route53 hosted zones for internal services
• DNSSEC where applicable
• Avoid DNS-based service discovery across clusters (security risk)

---

7. Database Security

RDS/Aurora Security

• Multi-AZ deployment for availability
• Private subnets only - no public access
• Encryption at rest (KMS) and in transit (SSL/TLS enforce)
• Automated backups with encryption
• Point-in-time recovery enabled
• Enhanced monitoring enabled
• Performance Insights with encryption

Connection Security

• RDS Proxy for connection pooling and IAM authentication
• SSL/TLS enforcement on database side
• Certificate validation on client side
• No hardcoded connection strings

# Database connection with SSL
DATABASE_URL: "postgresql://user@host:5432/db?sslmode=require"

Database Access Control

• Separate database users per service
• Minimal privileges (SELECT only for read-only services)
• No superuser access from applications
• Parameter groups to enforce security settings
• Audit logging enabled (PostgreSQL pgaudit, MySQL audit log)

Database Activity Monitoring

• AWS Database Activity Streams for real-time monitoring
• Alert on suspicious queries or access patterns
• Log all DDL and privilege changes

---

8. Logging & Monitoring

Comprehensive Logging

• EKS Control Plane logs to CloudWatch (API, audit, authenticator)
• Application logs via FluentBit/Fluentd to centralized location
• Database logs (query logs, error logs, slow query logs)
• VPC Flow Logs for network traffic analysis
• CloudTrail for all API calls
• Immutable logs - prevent tampering

Security Monitoring

• Amazon GuardDuty - threat detection
• AWS Security Hub - centralized security findings
• Amazon Detective - security investigation
• Falco - runtime security monitoring in Kubernetes
• Prometheus + Grafana for metrics and alerting

Audit Logging

# Enable EKS audit logs
apiVersion: v1
kind: ConfigMap
metadata:
  name: audit-policy
data:
  policy.yaml: |
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
      omitStages:
      - RequestReceived

Critical Alerts

• Failed authentication attempts
• Privileged container creation
• Security group changes
• Database connection failures
• Unusual API calls
• Resource exhaustion

---

9. Compliance & Governance

Compliance Frameworks

• AWS Config - track configuration changes
• AWS Audit Manager - compliance reporting
• CIS Kubernetes Benchmark - security hardening
• PCI-DSS, HIPAA, SOC 2 compliance where required
• Regular penetration testing

Policy as Code

• AWS Organizations with SCPs
• CloudFormation/Terraform for infrastructure
• GitOps for cluster configuration (ArgoCD/FluxCD)
• OPA/Kyverno for admission control
• Version control and peer review for all changes

Tagging Strategy

• Mandatory tags: Environment, Owner, Project, CostCenter
• Enforce tagging via AWS Config rules
• Use tags for resource-level IAM policies
• Cost allocation and chargeback

---

10. Disaster Recovery & Backup

Backup Strategy

• Automated RDS snapshots (daily, 7-30 day retention)
• Cross-region snapshot copies for DR
• EBS snapshots for persistent volumes
• etcd backups (Velero for cluster backups)
• GitOps - cluster configuration in Git

Disaster Recovery

• Multi-region setup for critical applications
• RTO/RPO requirements documented
• Failover procedures tested regularly
• Regular DR drills (quarterly minimum)
• Automated failover where possible (Route53 health checks)

---

11. Supply Chain Security

Container Supply Chain

• Verify base images from trusted sources
• SBOM (Software Bill of Materials) for dependencies
• Vulnerability scanning in CI/CD pipeline
• Sign images (Cosign/Notary)
• Admission controller to verify signatures

Dependency Management

• Dependabot or Renovate for automated updates
• Regular security patching
• Monitor CVEs for used software
• Minimal dependencies principle

---

12. Incident Response

Preparation

• Incident response plan documented
• Runbooks for common scenarios
• On-call rotation defined
• Communication channels established
• Post-mortem process defined

Detection & Response

• Automated alerting for security events
• Isolate compromised pods/nodes immediately
• Forensics capability (preserve logs and state)
• Contact AWS Support for suspected breaches
• Notify stakeholders per incident severity

---

13. API Gateway & Service Mesh

API Security

• AWS API Gateway or Kong/Envoy for API management
• Rate limiting to prevent abuse
• API keys or OAuth2 for authentication
• WAF (Web Application Firewall) rules
• DDoS protection via AWS Shield

Service Mesh Benefits

• mTLS for all service-to-service communication
• Zero-trust networking model
• Fine-grained authorization policies
• Observability and traffic monitoring
• Circuit breaking and fault injection

---

14. Update & Patch Management

EKS Updates

• Regular cluster updates (Kubernetes version support: ~14 months)
• Test in non-prod first
• Blue-green cluster upgrades for zero downtime
• Node group rolling updates

Security Patching

• Automated node updates (EKS managed node groups)
• Bottlerocket OS for minimal attack surface
• Container image updates (rebuild regularly)
• Database patching during maintenance windows

---

Security Checklist

• Private subnets for EKS nodes and databases
• Security groups with least privilege
• Network policies enforced
• IRSA configured for all pods needing AWS access
• No hardcoded credentials anywhere
• Secrets Manager/Parameter Store with rotation
• All encryption at rest enabled (etcd, EBS, RDS, S3)
• TLS/SSL enforced everywhere
• Pod security standards enforced (restricted)
• Image scanning in CI/CD
• Admission controllers (OPA/Kyverno) configured
• GuardDuty and Security Hub enabled
• Comprehensive logging to CloudWatch
• CloudTrail enabled in all regions
• VPC Flow Logs enabled
• Regular backups with cross-region copies
• Multi-factor authentication enforced
• RBAC properly configured
• Regular security audits scheduled
• Incident response plan documented

Summary
Security in multi-cluster architectures requires a defense-in-depth approach:

• Network isolation at every layer
• Zero-trust model - verify everything
• Encryption everywhere (at rest and in transit)
• Least privilege access for humans and workloads
• Continuous monitoring and alerting
• Regular audits and compliance checks
• Automation for consistency and reliability

Security is not a one-time setup but an ongoing process requiring continuous improvement and vigilance.