DEV Community: Elijah Chimera

[Boost]

Elijah Chimera — Thu, 25 Jun 2026 00:39:38 +0000

Elijah Chimera

Jun 23

Real-Time Runtime Security and Automated Incident Response for EKS

28 min read

Real-Time Runtime Security and Automated Incident Response for EKS

Elijah Chimera — Tue, 23 Jun 2026 22:34:06 +0000

Project Overview

In Part 1 of this series, I focused entirely on the Shift-Left paradigm. I spent time hardening Dockerfiles, enforcing IAM Roles for Service Accounts (IRSA) via Terraform, and setting up static security gates in our CI/CD pipeline using tfsec and Trivy. That setup ensured that known vulnerabilities or infrastructure misconfigurations were caught before they could ever reach our Amazon Elastic Kubernetes Service (EKS) cluster.

But as a Solutions Architect or DevSecOps Engineer, you must always adopt an assume-breach mentality.

What happens if an unknown, zero-day vulnerability (like Log4j) bypasses your static scanning gates? What happens if an insider threat or a compromised administrative credential executes malicious code directly inside a live, production container? Static security scanners cannot help you at 3:00 AM when a container starts scanning your internal VPC network or spinning up a cryptominer. For that, you need continuous, real-time visibility into your running workloads and an automated mechanism to neutralize threats instantly.

In this article, we will architect and deploy an end-to-end Runtime Security Defense Engine and an out-of-band Automated Incident Response Solution for EKS.

By combining the Cloud Native Computing Foundation (CNCF) open-source detection engine Falco with lightweight, serverless AWS infrastructure (Amazon CloudWatch Logs, Amazon EventBridge, AWS Lambda, and Amazon SNS), we will implement a highly responsive, cost-effective detection and remediation lifecycle.

Architecture Blueprint & Core Design Decisions

Our architecture is entirely event-driven and decoupled from the cluster's data plane:

Kernel-Level Detection: Falco runs on our EKS worker nodes as a DaemonSet, leveraging modern eBPF (Extended Berkeley Packet Filter) technology to intercept and inspect system calls (syscalls) directly within the Linux kernel.
Out-of-Cluster Telemetry Stream: When an anomaly matches a custom security rule, Falco formats the alert into a JSON payload and writes it to stdout. A logging agent ships this stream immediately to Amazon CloudWatch Logs.
Serverless Event Routing: Amazon EventBridge monitors the CloudWatch log stream using an explicit event pattern filter, capturing only high-severity threats (Critical and Emergency).
Out-of-Band Remediation: EventBridge invokes an isolated AWS Lambda function. This function uses the AWS API plane to securely authenticate against the EKS control plane and forcefully terminate the compromised pod.

Architectural Rationale & Trade-offs

Falco vs. Amazon GuardDuty for EKS: GuardDuty for EKS is a managed service that analyzes Kubernetes audit logs and runtime execution via an agent. While excellent, it incurs additional managed service costs. Falco operates at the kernel level via open-source eBPF rules, adding zero line-item costs to your AWS bill during your Proof of Concept (PoC) phase while remaining completely customizable.
Decoupled vs. In-Cluster Remediation: Instead of running heavy, highly-privileged management pods inside the cluster to kill compromised workloads, we stream alerts safely out of the cluster. This out-of-band automation model prevents an attacker who has achieved a container escape from blinding, manipulating, or deleting our security response logic.

Prerequisites & Local Machine Setup

Before we begin building, we must ensure your local workstation has the necessary tooling installed. This guide assumes you are using Fedora Linux, but the steps adapt readily to any Linux distribution.

1. Install the AWS CLI & Authenticate

Open your terminal and install the AWS command-line interface to interact with your cloud account:

sudo dnf install -y awscli2

Verify the installation:

aws --version

Configure the CLI with your sandbox account access keys. Ensure this identity has administrative privileges:

aws configure

2. Install Terraform

Add the official HashiCorp repository and install the Terraform binary:

sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
sudo dnf install -y terraform

Verify the installation:

terraform version

3. Install Helm

Helm acts as the package manager for Kubernetes applications. Install it via dnf:

sudo dnf install -y helm

Verify the installation:

helm version

4. Install kubectl

kubectl is the primary command-line tool used to send instructions to a Kubernetes cluster:

sudo dnf install -y kubernetes-client

Verify the installation:

kubectl version --client

⚠️ Cost Warning: Running a managed Amazon EKS cluster incurs a standard control plane fee of $0.10 per hour (~$73/month), along with the standard EC2 infrastructure charges for your worker nodes. Ensure you execute the complete cleanup steps at the very end of this lab to tear down all resources and avoid unexpected charges.

Creating the Project Workspace Directory Structure

We will build this multi-layered project completely from scratch. Run the following chained terminal command to create your root folder workspace, generate the entire sub-directory hierarchy, and create blank placeholder files:

mkdir -p eks-runtime-defense/{terraform,falco,lambda,k8s} && \
cd eks-runtime-defense && \
touch terraform/main.tf \
      terraform/variables.tf \
      terraform/outputs.tf \
      terraform/providers.tf \
      terraform/falco.tf \
      terraform/falco-response-infra.tf \
      falco/custom-rules.yaml \
      falco/falco-values.yaml \
      lambda/response_engine.py \
      lambda/falco_bridge.py \
      k8s/vulnerable-pod.yaml \
      .gitignore && \
echo -e ".terraform/\n*.tfstate*\n.env\n__pycache__/\n*.pyc" > .gitignore

Expected Directory Layout

Your VS Code workspace explorer will display this folder and file hierarchy:

eks-runtime-defense/
├── terraform/
│   ├── main.tf                  # Core infrastructure (VPC, EKS, IAM)
│   ├── variables.tf             # Input variables
│   ├── outputs.tf               # Output values
│   ├── providers.tf             # AWS, Kubernetes, Helm providers
│   ├── falco.tf                 # Falco Helm deployment
│   └── falco-response-infra.tf  # CloudWatch -> Lambda subscription
├── falco/
│   ├── custom-rules.yaml        # Custom Falco security rules
│   └── falco-values.yaml        # Falco Helm values (post-fix)
├── lambda/
│   ├── response_engine.py       # Pod deletion Lambda (heavily improved)
│   └── falco_bridge.py          # CloudWatch -> EventBridge bridge
└── k8s/
    └── vulnerable-pod.yaml      # Test target pod

Step 1: Provisioning the EKS Cluster and Networking via Terraform

We will use infrastructure-as-code to deploy a fresh, dedicated Virtual Private Cloud (VPC) and a hardened, managed EKS cluster.

Open your terraform/variables.tf file and paste the variable definitions:


variable "aws_region" {
  type        = string
  description = "The target AWS Region for all infrastructure resources"
  default     = "us-east-1"
}

variable "secops_email" {
  type        = string
  description = "The real email address to receive high-fidelity security alerts"
  default     = "your-real-email@example.com" # CHANGE THIS to your own email address
}

· type = string: Enforces type safety—Terraform will reject non-string values.
· description: Self-documenting variables are critical for team collaboration and code review.
· default = "us-east-1": Sets a sensible default while allowing override via terraform.tfvars or CLI flags.
· default = "your-real-email@example.com": Must be changed before apply. Terraform will prompt if no value is provided and no default exists.

Next, open your terraform/main.tf file and paste the comprehensive layout code:


terraform {
  required_version = ">= 1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.31.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.12.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Fetch active availability zones dynamically within the region
data "aws_availability_zones" "available" {}

# Provision a clean, isolated VPC network
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0.0"

  name                 = "runtime-defense-vpc"
  cidr                 = "10.0.0.0/16"
  azs                  = slice(data.aws_availability_zones.available.names, 0, 2)
  public_subnets       = ["10.0.1.0/24", "10.0.2.0/24"]
  private_subnets      = ["10.0.11.0/24", "10.0.12.0/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
}

# Provision a standard, security-hardened managed EKS Cluster
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "19.21.0"
  cluster_name    = "runtime-security-cluster"
  cluster_version = "1.30"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    security_nodes = {
      min_size       = 1
      max_size       = 2
      desired_size   = 1
      instance_types = ["t3.medium"] # Cost-conscious compute profile for testing labs
    }
  }
}

# Create a centralized CloudWatch Log Group for container telemetry
resource "aws_cloudwatch_log_group" "falco_alerts" {
  name              = "/aws/eks/runtime-security-cluster/falco-alerts"
  retention_in_days = 7
}

# Create an Amazon SNS Topic for real-time alerting notifications
resource "aws_sns_topic" "secops_alerts" {
  name = "secops-runtime-alerts-topic"
}

resource "aws_sns_topic_subscription" "email_sub" {
  topic_arn = aws_sns_topic.secops_alerts.arn
  protocol  = "email"
  endpoint  = var.secops_email
}

· required_version = ">= 1.5.0": Ensures compatibility with modern Terraform features like moved blocks and improved error messages.
· required_providers: Explicitly pins provider versions to prevent breaking changes from automatic updates.
· source = "hashicorp/aws": Uses the official HashiCorp AWS provider (not a community fork).
· version = "~> 5.31.0": Allows patch updates (5.0.x) but not minor (5.1.0), balancing stability and bug fixes.
· data "aws_availability_zones": Queries AWS dynamically for available AZs in the region—no hardcoding needed.
· source = "terraform-aws-modules/vpc/aws": Uses the battle-tested community VPC module (millions of downloads, actively maintained).
· version = "~> 5.0.0": Pins to 5.0.x for stability.
· cidr = "10.0.0.0/16": Provides 65,536 IP addresses—sufficient for most workloads.
· slice(..., 0, 2): Dynamically selects first 2 AZs. Using slice instead of hardcoding ensures the code works in regions with only 2 AZs (like us-west-1).
· enable_nat_gateway = true: Allows private subnet outbound internet access for EKS nodes (required for pulling container images).
· single_nat_gateway = true: Cost optimization for lab environments. Production should use one per AZ for high availability.
· enable_dns_hostnames = true: Required for EKS private endpoint resolution.
· cluster_version = "1.30": Uses a recent, supported Kubernetes version. Always check EKS version support lifecycle.
· subnet_ids = module.vpc.private_subnets: Places EKS nodes in private subnets—security best practice.
· cluster_endpoint_public_access = true: Required for this lab (access from local workstation). Production should disable this and use AWS VPN/SSM/private endpoints.
· cluster_enabled_log_types: Enables control plane audit logging—essential for forensic analysis and compliance (SOC 2, PCI-DSS).
· min_size = 1, max_size = 2: Cost-conscious auto-scaling. desired_size = 1 starts minimal.
· instance_types = ["t3.medium"]: Burstable instance with 2 vCPU, 4GB RAM—sufficient for Falco eBPF + test workloads.
· retention_in_days = 7: Cost optimization for lab. Production typically uses 30-90 days or streams to S3/Glacier.
· protocol = "email": Simplest notification channel. Production should add PagerDuty, Slack, or OpsGenie integrations.

Now, populate your terraform/outputs.tf file to output useful strings for later steps:


output "cluster_name" {
  value       = module.eks.cluster_name
  description = "The assigned name of the EKS cluster resource"
}

output "cluster_arn" {
  value       = module.eks.cluster_arn
  description = "The Amazon Resource Name of the EKS cluster control plane"
}

output "cluster_endpoint" {
  value       = module.eks.cluster_endpoint
  description = "The EKS Kubernetes API endpoint"
}

output "sns_topic_arn" {
  value       = aws_sns_topic.secops_alerts.arn
  description = "The Amazon Resource Name of the SecOps communication topic"
}

output "falco_log_group" {
  value       = aws_cloudwatch_log_group.falco_alerts.name
  description = "CloudWatch Log Group receiving Falco alerts"
}

· Outputs expose computed values for downstream use. They appear in terraform output CLI and can be consumed by other Terraform workspaces via remote state.
· description: Critical for documentation and terraform output self-documentation.

Create the provider configuration for Kubernetes and Helm. Without this, Terraform cannot authenticate to EKS for deploying Falco via Helm. This prevents Kubernetes cluster unreachable errors.

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes = {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

· data "aws_eks_cluster": Queries the EKS API for cluster metadata (endpoint, CA cert).
· data "aws_eks_cluster_auth": Generates a short-lived authentication token (valid ~15 minutes) for the current AWS identity.
· host = data.aws_eks_cluster.cluster.endpoint: The Kubernetes API server URL.
· cluster_ca_certificate = base64decode(...): Decodes the cluster's CA certificate for TLS verification. Without this, Terraform cannot verify the API server's identity.
· token = data.aws_eks_cluster_auth.cluster.token: The bearer token for authentication. Using data source ensures it's refreshed on every apply.
· provider "helm" { kubernetes = { ... } }: Helm provider needs identical auth configuration. Note the = sign—older Helm provider versions use kubernetes = {} (map syntax), newer versions use kubernetes {} (block syntax). Match your provider version.

Why use data sources for auth tokens instead of hardcoding kubeconfig files?
Data sources are ephemeral, rotate automatically, and don't leak credentials to state files.

Execution & Cluster Verification

Run the initialization and deployment steps in your terminal:

cd terraform
terraform init
terraform apply -auto-approve

Note: Provisioning an enterprise-grade managed Kubernetes cluster from scratch can take between 10 to 15 minutes to configure networking, control planes, and node pools.

Once the apply command finishes successfully, run the following command to download the cluster credentials and bind your local terminal context to the new EKS API manager:

aws eks update-kubeconfig --region us-east-1 --name runtime-security-cluster

Verify you can communicate with your new cloud resource by fetching the node state:

kubectl get nodes

What you should see: Your terminal will output a row displaying your single t3.medium worker instance with a status of Ready.

Step 2: Authoring the Incident Response Lambda Function

With our base infrastructure standing, we need an out-of-band remediation serverless component. If Falco flags a critical runtime thread breach, our Lambda function will handle the threat directly via the control plane API.

Appending Lambda IaC Blocks

Open your terraform/main.tf file and append the following block to the bottom of the file to provision the Lambda resource and its rigid IAM security boundary:


# Create an execution identity for the remediation lambda function
resource "aws_iam_role" "lambda_remediation_role" {
  name = "secops-remediation-lambda-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

# Define a restrictive policy to inspect the cluster and ship execution logs
resource "aws_iam_policy" "lambda_eks_policy" {
  name        = "LambdaEKSPodMitigationPolicy"
  description = "Allows the mitigation lambda function to read and terminate workloads in EKS"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["eks:DescribeCluster"]
        Resource = [module.eks.cluster_arn]
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = ["arn:aws:logs:*:*:*"]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_attach" {
  role       = aws_iam_role.lambda_remediation_role.name
  policy_arn = aws_iam_policy.lambda_eks_policy.arn
}

# Generate a minimal ZIP archive of the deployment script source code for compilation
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/response_engine.py"
  output_path = "${path.module}/lambda_function.zip"
}

# Declare the Lambda resource instance
resource "aws_lambda_function" "response_engine" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "secops-pod-remediation-engine"
  role             = aws_iam_role.lambda_remediation_role.arn
  handler          = "response_engine.lambda_handler"
  runtime          = "python3.11"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 30

  environment {
    variables = {
      CLUSTER_NAME = module.eks.cluster_name
    }
  }
}

Creating the Response Automation Script

The original code I deployed expected output_fields at the top level of CloudWatch log events. In production, Fluent Bit wraps Falco JSON in a {'_p': 'F', 'data': {...}, 'kubernetes': {...}, 'log': '...'} envelope. The original Lambda silently failed with 'No pod info found, skipping.' The fixed code handles** three nested formats** and includes comprehensive logging for debugging.

Move back to your project directories (cd ..) and open lambda/response_engine.py. Paste this fully working remediation code:

import base64
import json
import gzip
import boto3
import os
import urllib3
import ssl
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest

def lambda_handler(event, context):
    # Decode CloudWatch Logs payload (base64 + gzip)
    cw_data = base64.b64decode(event['awslogs']['data'])
    cw_data = gzip.decompress(cw_data)
    log_event = json.loads(cw_data)

    print(f"Received {len(log_event.get('logEvents', []))} log events")

    for log in log_event.get('logEvents', []):
        message = log.get('message', '')
        print(f"Processing message (first 300 chars): {message[:300]}")

        pod_name, namespace = extract_pod_info(message)
        if pod_name:
            print(f"Found pod: {pod_name} in namespace: {namespace} — DELETING")
            delete_pod(pod_name, namespace)
        else:
            print("No pod info found, skipping")

    return {'statusCode': 200}



def extract_pod_info(message):
    pod_name = None
    namespace = 'default'

    # FORMAT 1: Fluent Bit wrapper with 'data' -> 'output_fields'
    try:
        wrapper = json.loads(message)

        if 'data' in wrapper and isinstance(wrapper['data'], dict):
            fields = wrapper['data'].get('output_fields', {})
            pod_name = fields.get('k8s.pod.name')
            namespace = fields.get('k8s.ns.name', 'default')
            if pod_name:
                return pod_name, namespace

        # FORMAT 2: Top-level output_fields (raw Falco JSON)
        fields = wrapper.get('output_fields', {})
        pod_name = fields.get('k8s.pod.name')
        namespace = fields.get('k8s.ns.name', 'default')
        if pod_name:
            return pod_name, namespace

        # FORMAT 3: Escaped JSON in 'log' field
        raw_log = wrapper.get('log', '')
        if raw_log:
            try:
                falco_data = json.loads(raw_log)
                fields = falco_data.get('output_fields', {})
                pod_name = fields.get('k8s.pod.name')
                namespace = fields.get('k8s.ns.name', 'default')
                if pod_name:
                    return pod_name, namespace
            except json.JSONDecodeError:
                pass

    except json.JSONDecodeError:
        pass

    # FORMAT 4: Plain text fallback
    if 'k8s.pod.name=' in message:
        pod_name = message.split('k8s.pod.name=')[1].split()[0].strip()


def delete_pod(pod_name, namespace):
    cluster_name = os.environ.get('CLUSTER_NAME', 'runtime-security-cluster')
    region = os.environ.get('AWS_REGION', 'us-east-1')

    print(f"Authenticating to EKS cluster: {cluster_name}")

    # Get EKS cluster info (endpoint + CA certificate)
    eks = boto3.client('eks', region_name=region)
    cluster = eks.describe_cluster(name=cluster_name)['cluster']

    endpoint = cluster['endpoint']
    ca_cert = base64.b64decode(cluster['certificateAuthority']['data'])

    with open('/tmp/ca.crt', 'wb') as f:
        f.write(ca_cert)

    # Generate EKS IAM auth token
    token = generate_eks_token(cluster_name, region)
    print(f"Generated EKS token (first 50 chars): {token[:50]}...")

    # Configure HTTPS client with CA verification
    http = urllib3.PoolManager(
        cert_reqs=ssl.CERT_REQUIRED,
        ca_certs='/tmp/ca.crt'
    )

    # Construct Kubernetes API URL for pod deletion
    url = f"{endpoint}/api/v1/namespaces/{namespace}/pods/{pod_name}"
    headers = {
        'Authorization': f'Bearer {token}',
        'Content-Type': 'application/json'
    }

    print(f"DELETE {url}")

    # Execute deletion with immediate termination
    resp = http.request(
        'DELETE',
        url,
        headers=headers,
        body=json.dumps({
            "apiVersion": "v1",
            "kind": "DeleteOptions",
            "propagationPolicy": "Foreground"
        })
    )

    print(f"K8s API response status: {resp.status}")
    print(f"Body: {resp.data.decode()[:500]}")


def generate_eks_token(cluster_name, region):
    session = boto3.session.Session()
    creds = session.get_credentials().get_frozen_credentials()

    # Build an unsigned STS GetCallerIdentity request
    req = AWSRequest(
        method='GET',
        url=f'https://sts.{region}.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15',
        headers={'x-k8s-aws-id': cluster_name}
    )

    # Sign the request with SigV4 using the Lambda's IAM credentials
    SigV4Auth(creds, 'sts', region).add_auth(req)

    url = req.url
    headers = [{'key': k, 'value': v} for k, v in req.headers.items()]

    # Encode the signed URL and headers as base64url
    payload = json.dumps({'url': url, 'headers': headers})
    token = base64.urlsafe_b64encode(payload.encode()).decode().rstrip('=')
    return f'k8s-aws-v1.{token}'

· base64.b64decode(event['awslogs']['data']): CloudWatch Logs subscriptions encode payloads in base64.
· gzip.decompress(...): CloudWatch further compresses with gzip for efficiency.
· json.loads(...): Converts the decompressed JSON string to a Python dictionary.
· log_event.get('logEvents', []): Safely retrieves the list of log events. Using .get() with a default prevents KeyError if the key is missing.
· message[:300]: Truncates long messages for CloudWatch Logs line limits while preserving enough context for debugging.

· namespace = 'default': Default fallback. Kubernetes pods without explicit namespace assignment land here.
· isinstance(wrapper['data'], dict): Type guard prevents AttributeError if data exists but is a string or null.
· wrapper['data'].get('output_fields', {}): Nested safe access. Returns empty dict if output_fields is missing.
· fields.get('k8s.pod.name'): Returns None (falsy) if the field is missing—triggers the next fallback.
· json.loads(raw_log): Handles the case where Fluent Bit puts the actual Falco JSON inside a string-valued log key.
· json.JSONDecodeError: Gracefully handles malformed JSON instead of crashing the entire Lambda.
· Text parsing fallback: Regex-free string splitting for simple key=value formats. Fast but fragile—used only when JSON parsing fails.

· os.environ.get('CLUSTER_NAME', 'runtime-security-cluster'): Reads from Lambda environment variables. Uses default for local testing.
· boto3.client('eks', region_name=region): Creates a regional EKS client. Always specify region_name—Lambda may run in a different region than the cluster.
· base64.b64decode(...): EKS returns the CA certificate as a base64-encoded string.
· ssl.CERT_REQUIRED: Enforces TLS certificate verification. Never use CERT_NONEin production—it opens you to man-in-the-middle attacks.
· /api/v1/namespaces/{namespace}/pods/{pod_name}: The Kubernetes Core API endpoint for pod lifecycle management.
· propagationPolicy: 'Foreground': Ensures dependent resources (like ReplicaSets) are cleaned up before the pod is considered deleted.
· resp.data.decode()[:500]: Decodes the binary response and truncates for logging.
· get_frozen_credentials(): Locks credentials to prevent refresh mid-request. Critical for SigV4 signing where the signature includes a timestamp.
· x-k8s-aws-id: EKS-specific header that binds the token to a specific cluster. Prevents token replay attacks against other clusters.
· SigV4Auth(creds, 'sts', region): Uses AWS Signature Version 4—the same algorithm AWS SDKs use for all API calls.
· base64.urlsafe_b64encode: Uses URL-safe base64 (replaces + with - and / with _). Required because the token is passed in HTTP headers.
· .rstrip('='): Removes padding. Kubernetes API server accepts unpadded base64url.

Defensive programming (multiple fallbacks, type checking, exception handling) is essential in production because log formats can change when Fluent Bit, Falco, or EKS versions update.

We don't use aws-iam-authenticator binary in Lambda. Lambda has a 250MB deployment package limit and cold start constraints. Generating the token in pure Python eliminates binary dependencies and reduces package size.

Click the lambda/falco bridge.py folder and write the following lines of code:

import json
import base64
import gzip
import boto3
import os

def lambda_handler(event, context):
    region = os.environ.get('AWS_REGION', 'us-east-1')
    eventbridge = boto3.client('events', region_name=region)

    # Decode CloudWatch Logs payload (base64 + gzip)
    cw_data = base64.b64decode(event['awslogs']['data'])
    cw_logs = json.loads(gzip.decompress(cw_data))
    log_events = cw_logs.get('logEvents', [])

    forwarded = 0
    for log_event in log_events:
        message = log_event.get('message', '')
        if not message.startswith('{'):
            continue

        try:
            falco_event = json.loads(message)
        except json.JSONDecodeError:
            continue

        # Filter for high-severity events only
        priority = falco_event.get('priority', '').lower()
        if priority not in ['critical', 'emergency']:
            continue

        # Forward to EventBridge for downstream processing
        eventbridge.put_events(
            Entries=[{
                'Source': 'falco.runtime.security',
                'DetailType': 'Falco Runtime Threat Detected',
                'Detail': json.dumps(falco_event),
                'EventBusName': 'default'
            }]
        )
        forwarded += 1
        print(f"Forwarded {falco_event.get('rule')} alert to EventBridge.")

    return {'status': 'success', 'forwarded_events': forwarded}

· if not message.startswith('{'): Fast pre-filter. Falco JSON output always starts with {. This skips Fluent Bit internal logs, timestamps, and metadata lines.
· priority not in ['critical', 'emergency']: Severity gate. Prevents alert fatigue by filtering out Notice/Warning/Error level events.
· 'Source': 'falco.runtime.security': Custom event source. EventBridge rules filter on this to distinguish Falco alerts from other events.
· 'Detail': json.dumps(falco_event): EventBridge requires Detail to be a JSON string, not a dict. This is a common source of MalformedDetail errors.
· 'EventBusName': 'default': Uses the account's default event bus. Production might use custom event buses for multi-tenant isolation.

Deploy the new lambda layer configurations immediately using Terraform:

cd terraform
terraform apply -auto-approve

Lets grant our cluster admin privileges first before installing Fluent Bit:

# Step 1: Figure out who you are in AWS
CURRENT_ARN=$(aws sts get-caller-identity --query Arn --output text)
echo "Your current ARN: $CURRENT_ARN"

# Step 2: If you're using an assumed IAM Role, strip the session suffix
# EKS Access Entries need the ROLE ARN, not the session ARN
# Example: arn:aws:sts::123:assumed-role/MyRole/session -> arn:aws:iam::123:role/MyRole
if echo "$CURRENT_ARN" | grep -q "assumed-role"; then
  PRINCIPAL_ARN=$(echo "$CURRENT_ARN" | sed 's|arn:aws:sts:|arn:aws:iam:|' | sed 's|:assumed-role/|:role/|' | sed 's|/.*||')
  echo "Converted to Role ARN: $PRINCIPAL_ARN"
else
  PRINCIPAL_ARN="$CURRENT_ARN"
fi

# Step 3: Create the access entry
aws eks create-access-entry \
  --cluster-name runtime-security-cluster \
  --principal-arn "$PRINCIPAL_ARN" \
  --type STANDARD

# Step 4: Grant cluster-admin privileges
aws eks associate-access-policy \
  --cluster-name runtime-security-cluster \
  --principal-arn "$PRINCIPAL_ARN" \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

# Step 5: Wait 30 seconds for IAM propagation, then test
sleep 30
kubectl get nodes

Now that we have granted admin privileges to our cluster lets install Fluent Bit.

# After terraform apply finishes:
aws eks update-kubeconfig --region us-east-1 --name runtime-security-cluster

helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-for-fluent-bit eks/aws-for-fluent-bit \
  --namespace kube-system \
  --set cloudWatch.enabled=true \
  --set cloudWatch.logGroupName=/aws/eks/runtime-security-cluster/falco-alerts \
  --set cloudWatch.region=us-east-1 \
  --set cloudWatch.logStreamPrefix=falco- \
  --set firehose.enabled=false \
  --set kinesis.enabled=false

Step 3: Terraform Lambda Infrastructure

Lets break down the terraform file in our main.tf in respect to Lambda.

IAM Role and Policies

resource "aws_iam_role" "lambda_remediation_role" {
  name = "secops-remediation-lambda-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

resource "aws_iam_policy" "lambda_policy" {
  name        = "LambdaEKSRemediationPolicy"
  description = "Allows Lambda to read EKS, send logs, publish SNS, and emit EventBridge events"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["eks:DescribeCluster"]
        Resource = [module.eks.cluster_arn]
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = ["arn:aws:logs:*:*:*"]
      },
      {
        Effect   = "Allow"
        Action   = ["sns:Publish"]
        Resource = [aws_sns_topic.secops_alerts.arn]
      },
      {
        Effect   = "Allow"
        Action   = ["events:PutEvents"]
        Resource = ["*"]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_attach" {
  role       = aws_iam_role.lambda_remediation_role.name
  policy_arn = aws_iam_policy.lambda_policy.arn
}

resource "aws_iam_role_policy_attachment" "lambda_basic" {
  role       = aws_iam_role.lambda_remediation_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

· Principal = { Service = 'lambda.amazonaws.com' }: Only Lambda service can assume this role. Prevents EC2, ECS, or other services from using it.
· eks:DescribeCluster: Minimum permission needed to get the cluster endpoint and CA certificate. The Lambda doesn't need broader EKS permissions.
· Resource = [module.eks.cluster_arn]: Scoped to specific cluster. Follows least-privilege principle.
· AWSLambdaBasicExecutionRole: AWS-managed policy providing CloudWatch Logs access. Always attach this or your Lambda logs won't appear.

EKS Access Entry (Modern Auth — Replaces aws-auth ConfigMap)

Production Improvement: The original main.tf I wrote used the legacy aws-auth ConfigMap for IAM-to-Kubernetes RBAC mapping. EKS now supports Access Entries a native AWS API for managing cluster access without modifying ConfigMaps. This is more secure, auditable, and doesn't risk locking yourself out with malformed YAML.

resource "aws_eks_access_entry" "lambda" {
  cluster_name  = module.eks.cluster_name
  principal_arn = aws_iam_role.lambda_remediation_role.arn
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "lambda" {
  cluster_name  = module.eks.cluster_name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy"
  principal_arn = aws_iam_role.lambda_remediation_role.arn

  access_scope {
    type       = "namespace"
    namespaces = ["default"]
  }

  depends_on = [aws_eks_access_entry.lambda]
}

· type = 'STANDARD': Standard access entry (not EC2_LINUX or FARGATE_LINUX which are for managed nodes).
· AmazonEKSAdminPolicy: AWS-managed policy granting full admin access within the scoped namespace. For production, use AmazonEKSViewPolicy or custom policies.
· access_scope { type = 'namespace', namespaces = ['default'] }: Critical security boundary. The Lambda can only manage resources in the default namespace. If compromised, it cannot affect kube-system, security, or other namespaces.
· depends_on: Ensures the access entry exists before associating policies. Prevents race conditions.

Comparison between Access Entries vs. aws-auth. Access Entries are API-driven (Terraform-friendly), support fine-grained policies, and don't require cluster access to modify. aws-auth requires kubectl access and is prone to syntax errors that can lock out all users.

Lambda Deployment Packages

data "archive_file" "response_lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/response_engine.py"
  output_path = "${path.module}/response_engine.zip"
}

data "archive_file" "bridge_lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/falco_bridge.py"
  output_path = "${path.module}/falco_bridge.zip"
}

· data "archive_file": Terraform data source that creates a zip file during the plan phase.
· ${path.module}: Resolves to the directory containing the current Terraform file. Ensures paths work regardless of where Terraform is executed from.
· output_path: The zip is created in the Terraform working directory (.terraform/ is gitignored).

Lambda Function Definitions

resource "aws_lambda_function" "response_engine" {
  filename         = data.archive_file.response_lambda_zip.output_path
  function_name    = "secops-pod-remediation-engine"
  role             = aws_iam_role.lambda_remediation_role.arn
  handler          = "response_engine.lambda_handler"
  runtime          = "python3.11"
  source_code_hash = data.archive_file.response_lambda_zip.output_base64sha256
  timeout          = 30

  environment {
    variables = {
      CLUSTER_NAME  = module.eks.cluster_name
      SNS_TOPIC_ARN = aws_sns_topic.secops_alerts.arn
    }
  }

  depends_on = [aws_iam_role_policy_attachment.lambda_basic]
}

resource "aws_lambda_function" "falco_bridge" {
  filename         = data.archive_file.bridge_lambda_zip.output_path
  function_name    = "falco-cloudwatch-bridge"
  role             = aws_iam_role.lambda_remediation_role.arn
  handler          = "falco_bridge.lambda_handler"
  runtime          = "python3.11"
  source_code_hash = data.archive_file.bridge_lambda_zip.output_base64sha256
  timeout          = 10

  depends_on = [aws_iam_role_policy_attachment.lambda_basic]
}

· source_code_hash: Forces Terraform to redeploy the Lambda when the Python code changes. Without this, Terraform sees the same filename and skips updates.
· timeout = 30: Response Lambda needs time for EKS API calls (describe cluster + delete pod). Bridge Lambda is simpler (10s).
· environment { variables = { ... } }: Passes configuration without hardcoding. Allows the same code to run against different clusters by changing environment variables.

EventBridge Rule and Targets

resource "aws_cloudwatch_event_rule" "falco_critical_rule" {
  name        = "falco-critical-alerts-filter"
  description = "Intercepts high-severity Falco runtime security events"

  event_pattern = jsonencode({
    "source" : ["falco.runtime.security"],
    "detail" : {
      "priority" : ["Critical", "Emergency", "CRITICAL", "EMERGENCY"]
    }
  })
}

resource "aws_cloudwatch_event_target" "lambda_target" {
  rule      = aws_cloudwatch_event_rule.falco_critical_rule.name
  target_id = "TriggerRemediationLambda"
  arn       = aws_lambda_function.response_engine.arn
}

resource "aws_cloudwatch_event_target" "sns_target" {
  rule      = aws_cloudwatch_event_rule.falco_critical_rule.name
  target_id = "SendSecOpsNotification"
  arn       = aws_sns_topic.secops_alerts.arn
}

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.response_engine.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.falco_critical_rule.arn
}

· event_pattern: JSON Path matching. Only events where source equals falco.runtime.security AND detail.priority matches the list trigger the rule.
· target_id: Human-readable identifier for the target. Appears in CloudTrail and debugging.
· aws_lambda_permission: Grants EventBridge service principal permission to invoke the Lambda. Without this, EventBridge cannot trigger the Lambda—the invocation will fail silently.
· statement_id: Unique identifier for the permission policy statement. Must be unique within the Lambda's policy.

CloudWatch Logs -> Bridge Lambda Integration

resource "aws_lambda_permission" "allow_cloudwatch" {
  statement_id  = "AllowExecutionFromCloudWatchLogs"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.falco_bridge.function_name
  principal     = "logs.amazonaws.com"
  source_arn    = "${aws_cloudwatch_log_group.falco_alerts.arn}:*"
}

resource "aws_cloudwatch_log_subscription_filter" "falco_filter" {
  name            = "falco-critical-filter"
  log_group_name  = aws_cloudwatch_log_group.falco_alerts.name
  filter_pattern  = "?CRITICAL ?EMERGENCY"
  destination_arn = aws_lambda_function.falco_bridge.arn

  depends_on = [aws_lambda_permission.allow_cloudwatch]
}

· filter_pattern = "?CRITICAL ?EMERGENCY": CloudWatch Logs filter syntax. ? means OR. Only log lines containing 'CRITICAL' or 'EMERGENCY' are forwarded. This reduces Lambda invocations and costs.
· depends_on: Ensures the Lambda permission exists before creating the subscription. Without this, CloudWatch Logs cannot verify it has permission to invoke the Lambda, and creation fails.
· source_arn = "${...}:*": The :* suffix allows the permission to apply to all log streams within the log group.

Step 4: Deploying Custom Falco Security Rules via Helm

Now let's configure the runtime detection layer. We will create custom rules that monitor system calls inside our cluster, checking specifically for interactive terminal attachments (kubectl exec) and malicious network connections.

Navigate to falco/custom-rules.yaml and paste the rule configurations:


# Enable the modern eBPF driver instead of compiling legacy kernel modules
driver:
  kind: ebpf

# Configure structural output format to print machine-readable JSON strings to stdout
falco:
  json_output: true
  stdout_output: true

# Custom rule definitions designed to stop specific adversarial attacks
customRules:
  rules_user.yaml: |
    - rule: Terminal Shell Spawned Within Container
      desc: Detects an interactive terminal shell attachment or spawn attempt inside a live container.
      condition: container.id != host and proc.name = bash or proc.name = sh and spawn_process
      output: "CRITICAL: Interactive Shell Spawned (user=%user.name pod=%k8s.pod.name ns=%k8s.ns.name image=%container.image.repository proc=%proc.name cmdline=%proc.cmdline)"
      priority: CRITICAL
      tags: [container, shell, mitre_execution]

    - rule: Cryptominer Network Pattern Detected
      desc: Detects outbound communication attempts targeting common mining pools.
      condition: fd.sport = 4444 or fd.dport = 4444 or fd.sip = "pool.supportxmr.com"
      output: "CRITICAL: Cryptominer Outbound Connection Detected (pod=%k8s.pod.name ns=%k8s.ns.name destination=%fd.sip port=%fd.sport process=%proc.name)"
      priority: EMERGENCY
      tags: [network, mining, mitre_impact]

· driver: kind: modern_ebpf: Uses the modern eBPF probe (Falco 0.35+). More portable than kernel module or legacy eBPF. Works on most kernels without headers.
· json_output: true: Outputs alerts as JSON. Required for structured parsing by Lambda.
· json_include_output_property: true: Includes the human-readable output field in JSON. Useful for SNS emails.
· stdout_output: enabled: true: Writes to stdout where Fluent Bit captures it.
· container.id != host: Ensures the rule only fires for containers, not host processes.
· proc.name in (bash, sh): Matches common interactive shells. Extend with zsh, ash if needed.
· fd.rport in (...): Matches known mining pool ports. These are TCP ports commonly used by XMRig and similar miners.

When deploying Falco 0.44.1 via Helm, we encountered a fatal Runtime error: cannot register plugin /usr/share/falco/plugins/libcontainer.so in inspector: found another plugin with name container. Aborting.
Root Cause: Falco 0.44+ ships the containerplugin *built into the binary *(source plugin), but the Helm chart's falcoctl also installs it as an external .soplugin. Both register with the same name, causing a collision.
Fix: Disable falcoctl artifact installation in Helm values.

# falco/falco-values.yaml
# CRITICAL: Disables falcoctl to prevent duplicate container plugin

driver:
  kind: ebpf

falcoctl:
  artifact:
    install:
      enabled: false
    follow:
      enabled: false

· falcoctl.artifact.install.enabled: false: Prevents falcoctl from downloading and installing plugins at startup. The built-in container plugin is sufficient.
· falcoctl.artifact.follow.enabled: false: Disables automatic plugin updates. In production, you want controlled updates, not automatic ones.

This is a classic example of version drift between components. Falco upgraded to built-in plugins, but the Helm chart lagged behind. Always check component compatibility matrices and test in a non-production environment first

Let us deploy helm in falco.tf

resource "helm_release" "falco" {
  name       = "falco"
  namespace  = "security"
  repository = "https://falcosecurity.github.io/charts"
  chart      = "falco"
  version    = "4.17.0"

  create_namespace = true

  values = [
    file("${path.module}/../falco/falco-values.yaml"),
    file("${path.module}/../falco/custom-rules.yaml")
  ]
}

· version = "4.17.0": Pins the Helm chart version. Chart versions are independent of Falco binary versions.
· create_namespace = true: Creates the security namespace if it doesn't exist. Equivalent to kubectl create namespace security.
· values = [file(...), file(...)]: Loads multiple values files. Later files override earlier ones. falco-values.yaml sets base config; custom-rules.yaml adds custom rules.

Lets create the falco-response-infra.tf because my original architecture routed CloudWatch -> Bridge Lambda -> EventBridge -> Response Lambda it failed. In response, we added a direct CloudWatch -> Response Lambda subscription for faster remediation (bypassing EventBridge latency) and as a fallback if EventBridge fails.

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}

resource "aws_lambda_permission" "allow_cloudwatch_response" {
  statement_id  = "AllowExecutionFromCloudWatchLogs"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.response_engine.function_name
  principal     = "logs.amazonaws.com"
  source_arn    = "arn:aws:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/eks/runtime-security-cluster/falco-alerts:*"
}

resource "aws_cloudwatch_log_subscription_filter" "falco_response" {
  depends_on      = [aws_lambda_permission.allow_cloudwatch_response]
  name            = "falco-alerts-to-response"
  log_group_name  = "/aws/eks/runtime-security-cluster/falco-alerts"
  filter_pattern  = ""
  destination_arn = aws_lambda_function.response_engine.arn
}

· data "aws_region" "current": Queries the current AWS region. Used to construct ARNs without hardcoding.
· data "aws_caller_identity" "current": Queries the current AWS account ID. Essential for cross-account or multi-environment templates.
· filter_pattern = "": Empty pattern matches ALL log events. Used here because the Response Lambda has its own filtering logic (checks for pod names). The Bridge Lambda uses?CRITICAL ?EMERGENCY for selective forwarding.
· source_arn: Must exactly match the log group ARN format. The :* suffix covers all log streams.

Deploying the Falco Engine to EKS

Run these commands from your terminal to connect the open-source Helm repository and launch the deployment inside a dedicated namespace:

# Add the official cloud-native CNCF Falco charts stable registry
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install the Falco chart using our custom security rules and eBPF configuration
helm install falco falcosecurity/falco --namespace security --create-namespace -f ../falco/custom-rules.yaml

Track the rollout status until the security pods are running:

kubectl get pods -n security

Step 5: Deploying a Sample Vulnerable Application Pod

To evaluate our setup, we need an asset inside our cluster we can break into. Let's create a simulation workspace environment.

Navigate to your k8s/vulnerable-pod.yaml manifest file and paste this pod definition:

# k8s/vulnerable-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: dynamic-web-app-pod
  namespace: default
  labels:
    app: public-ingress-service
spec:
  containers:
  - name: web-server
    image: ubuntu:22.04 # Bloated base image containing standard tools (curl, package managers)
    command: ["/bin/bash", "-c", "sleep 3600"]
    securityContext:
      privileged: false
      allowPrivilegeEscalation: true
      runAsUser: 0 # Running explicitly as ROOT user (The dangerous anti-pattern from Part 1)

· image: ubuntu:22.04: Bloated base image with many pre-installed tools (curl, apt). Attackers love this—more tools to exploit.
· command: ["/bin/bash", "-c", "sleep 3600"]: Keeps the pod alive for 1 hour. The sleep command holds the container open.
· allowPrivilegeEscalation: true: Allows the container to gain more privileges than its parent process. Critical anti-pattern.
· runAsUser: 0: Runs as root. Combined with allowPrivilegeEscalation, this is a container escape risk.

Launch this vulnerable asset into your cluster:

kubectl apply -f ../k8s/vulnerable-pod.yaml

Verify it is active before proceeding:

kubectl get pods

Simulated Attack Scenarios (Red Team Phase)

Now, let's step into the role of an adversary who has discovered a vulnerability on your public app. We will trigger the detection logs manually.

Scenario A: Spawning an Interactive Shell

Run this command on your Fedora terminal to force your way inside the live running container via a bash terminal injection simulation:

kubectl exec -it dynamic-web-app-pod -- /bin/bash

What you should see:
Your terminal prompt will change to root@dynamic-web-app-pod:/#. You are now executing commands natively inside the live container instance running on AWS.

Scenario B: Cryptomining Network Reconnaissance

While inside your active root container shell, run these commands to download network analysis tools and simulate an outbound query targeting an offensive mining pool network configuration:

apt-get update && apt-get install -y curl
curl http://pool.supportxmr.com:4444

🎯 Why port 4444?
We configured Falco to detect any outbound connection to port 4444 (and other mining ports). The curl command will attempt a TCP connection to port 4444, triggering Falco's connect syscall probe. The connection does not need to succeed-Falco sees the attempt.

Once executed, step out of the shell connection by typing:

exit

Step 6: Verification and Event Remediation Integration

Let's trace how the end-to-end security system catches and routes this breach.

1. Read Falco's Local Cluster Streams

Run this command to inspect the raw stdout JSON engine outputs generated directly inside the cluster:

kubectl logs -n security -l app.kubernetes.io/name=falco | grep CRITICAL

Expected Telemetry Record Output:

{"hostname":"falc-node-x7r","output":"CRITICAL: Interactive Shell Spawned (user=root pod=dynamic-web-app-pod ns=default image=ubuntu proc=bash cmdline=bash)","priority":"Critical","rule":"Terminal Shell Spawned Within Container","time":"2026-06-11T18:12:45.123456Z","output_fields":{"container.id":"a8b7c6d5e4f3","k8s.ns.name":"default","k8s.pod.name":"dynamic-web-app-pod","proc.name":"bash"}}

2. Verify the CloudWatch Log Stream

Check that logs arrived in CloudWatch:

aws logs tail /aws/eks/runtime-security-cluster/falco-alerts --follow

You should see the Falco JSON records appearing in real time.

3. Verify EventBridge and Lambda Invocation

Check the Lambda CloudWatch logs to see the automation in action:

aws logs tail /aws/lambda/falco-cloudwatch-bridge --follow
aws logs tail /aws/lambda/secops-pod-remediation-engine --follow

4. Confirm Email Subscription:

Go to the inbox of the email address you configured in variables.tf. Look for an AWS Notification email and click Confirm Subscription. If you already confirmed earlier, you will now receive an alert email with the subject [CRITICAL] Falco Alert: Terminal Shell Spawned Within Container.

5. Observe the Automated Remediation

Re-trigger the exploit execution

kubectl exec -it dynamic-web-app-pod -- /bin/bash

Within seconds of typing commands inside the shell, your terminal connection will abruptly freeze or drop with an exit message like connection reset by peer. This occurs because the Lambda script successfully intercepted the kernel-level alert and deleted the compromised container.

Inspect the cluster state:

kubectl get pods

Expected output: Your terminal will show that the compromised pod has completely disappeared or is stuck in a Terminating status state, proving your automated out-of-band response engine functions flawlessly!

Troubleshooting Guide for Common Errors

1. Error: Helm installation failed: security namespace already exists

This happens if you run the helm install command multiple times or if a previous installation failed midway.

The Fix: Completely purge the existing installation before re-running the launch:

helm uninstall falco -n security

2. Error: aws-auth configmap update failed

Your local AWS identity credentials might have drifted since initializing your cloud resources.

The Fix: Re-download your authorization tokens directly using the AWS CLI tool layer:

aws eks update-kubeconfig --region us-east-1 --name runtime-security-cluster

3. Error: Falco CrashLoopBackOff: 'found another plugin with name container

Pod restarts continuously. Logs show plugin registration error.

Root Cause: Falco 0.44+ built-in container plugin conflicts with falcoctl-installed plugin.

The Fix: Add to falco-values.yaml:

falcoctl:
  artifact:
    install:
      enabled: false
    follow:
      enabled: false

4. Terraform: 'Kubernetes cluster unreachable'

terraform apply fails when deploying Falco via Helm.

Root Cause: Kubernetes/Helm providers not configured with EKS auth.

The Fix: Add providers.tf with data.aws_eks_cluster and data.aws_eks_cluster_auth.

5. Lambda: 'No pod info found, skipping'

Lambda invokes but doesn't delete pods. Logs show parsing failure.

Root Cause: CloudWatch log format differs from expected (Fluent Bit wrapper vs. raw JSON).

The Fix: Update response_engine.py extract_pod_info() to handle multiple formats:

· wrapper['data']['output_fields'] (Fluent Bit envelope)
· wrapper['output_fields'] (raw Falco)
· wrapper['log'] (escaped JSON string)
· Text fallback for legacy output

6. Lambda: 'Unauthorized' or 403 from K8s API

Lambda reaches K8s API but deletion fails.

Root Cause:

· EKS Access Entry not created
· Access Policy Association missing
· Lambda IAM role not mapped in EKS

The Fix:

aws eks list-access-entries --cluster-name runtime-security-cluster
aws eks list-associated-access-policies \
  --cluster-name runtime-security-cluster \
  --principal-arn arn:aws:iam::ACCOUNT:role/secops-remediation-lambda-role

7. CloudWatch Subscription Filter Not Triggering Lambda

No Lambda invocations despite logs appearing in CloudWatch.

Root Cause:

· aws_lambda_permission missing or source_arn mismatch
· Subscription filter created before permission
· Filter pattern too restrictive

The Fix:

aws logs describe-subscription-filters \
  --log-group-name /aws/eks/runtime-security-cluster/falco-alerts
aws lambda get-policy --function-name secops-pod-remediation-engine

Complete Project Cleanup Step

To clean up your AWS account and avoid unexpected sandbox charges for EKS or compute resources, run this command inside your terraform/ directory to safely destroy every infrastructure asset built during this lab project:

terraform destroy -auto-approve

Summary of What I Built

Part 2 of this advanced cloud security series is completed! By combining runtime detection engines with cloud-native automation patterns, I built a highly robust system:

Kernel Telemetry Monitoring: We deployed Falco with an optimized eBPF driver configuration to inspect runtime system calls directly within the Linux kernel, minimizing compute overhead.
Decoupled Architecture: We routed security alerts completely out of the Kubernetes cluster, streaming data up to Amazon CloudWatch Logs to prevent log tampering.
Automated Event Remediation: We deployed an event-driven Amazon EventBridge mapping rule to identify high-severity threats and instantly trigger an AWS Lambda function to isolate and eliminate compromised workloads.

Final Architectural questions

1. Why eBPF over kernel modules?

eBPF is safer (verified by kernel verifier), more portable (no kernel headers needed), and doesn't require privileged access to load.

2. Why out-of-band remediation?

If an attacker escapes the container, they can't kill the security agent because it lives outside the cluster as serverless functions.

3. How do you handle false positives?

The Bridge Lambda filters by priority (Critical/Emergency only). Production would add allowlists, anomaly scoring, and SOAR integration.

4. What if the Lambda fails?

CloudWatch Logs retains all events. Failed Lambda invocations trigger CloudWatch Alarms. SNS provides human notification as fallback.

5. How do you update Falco rules?

Rules are in Git. CI/CD pipeline runs terraform apply to update the ConfigMap/Helm values. Rolling update restarts Falco pods gracefully.

6. Why not GuardDuty?
GuardDuty is excellent for audit log analysis but doesn't see syscalls. Falco sees the actual system calls—what the process is doing, not just what the API recorded.

Bonus: NotebookLM Study Guide

Core System Configuration Comparison

Use this comparison matrix to evaluate your runtime security configurations during architecture reviews or engineering design sessions.

Architectural Matrix	Vulnerable Setup	Hardened Static Base (Project 1)	Active Runtime Defense (Project 2)
Primary Focus	Speed of delivery; rapid prototyping.	Source Code, Build-time checking, and Static configurations.	Live execution state, system call analysis, and active compromise mitigation.
Detection Coverage	None.	Catches known older packages and broad IaC infrastructure design misconfigurations.	Catches live human zero-day exploits, unexpected shell spawns, and reverse-shells.
Remediation Speed	Manual intervention following discovery or system failure.	Blocks pull requests; stops deployments at the deployment gate phase.	Event-driven automation; handles live threats and shuts down compromised pods within seconds.
AWS Identity Cost	Broad Node Policies ($0).	Highly Specific Least-Privilege IRSA Mappings ($0).	Decoupled Out-of-Band Lambda Controls ($0 Layer Free-Tier).

Threat Modeling Decision Tree

Threat Event Discovered
   │
   ├── Is it an outdated library or base image vulnerability?
   │     └── Yes -> Handle via Static Scanning Gates (Trivy / PR Block)
   │
   └── Is it malicious behavior on a running container (e.g., shell spawn, network scan)?
         └── Yes -> Handle via Runtime Protection Framework
               │
               ├── Local Event Log Capture (Falco eBPF telemetry)
               ├── Out-of-Band Centralization (Amazon CloudWatch Logs)
               └── Pattern Trigger Remediation (EventBridge Event -> AWS Lambda Eviction)

By completing both Part 1 and Part 2, you have evolved from simple perimeter security to building a resilient, defense-in-depth cloud-native application infrastructure. Go forth and secure your clusters.

Designing a Secure-by-Default DevSecOps Pipeline on AWS.

Elijah Chimera — Mon, 15 Jun 2026 20:50:17 +0000

Project Overview

Hello! it has been a while since I last posted , I have been working on and learning how to architect production grade CI/CD pipeline in a two part series I call "The AWS Defense in Depth Architecture: From Pipeline to Runtime". As cloud architectures grow in complexity, the traditional lines between engineering, operations, and security have vanished. You cannot effectively design an enterprise AWS environment without understanding exactly how an adversary might exploit it.

In this two-part series, we are bridging the gap between offensive security tactics and defensive cloud architecture. We aren't just going to look at diagrams; we are going to build a workload, deliberately break into it to understand the blast radius, and then engineer robust solutions to keep attackers out.

Part 1 (This Article): Focuses on the Shift-Left paradigm. We will build an EKS pipeline, simulate a container breakout, and harden the architecture using IaC scanning, CI/CD gates, and strict IAM least privilege.

Part 2: Focuses on the Assume Breach mindset. We will architect a zero-cost, event-driven runtime defense engine using Falco, Amazon EventBridge, and AWS Lambda to detect and automatically remediate active threats inside our cluster.

In the modern era a Solutions Architect's role extends far beyond simply connecting cloud services; they are the gatekeepers of an organization's blast radius. Modern cloud architecture demands that security is not an afterthought handled by a separate silo, but a foundational element baked into the Infrastructure as Code (IaC) and deployment pipelines.

I have not yet secured a role but I once worked in an incubator startup right here in Nairobi for less than a month , I frequently saw the same anti-patterns:

The "Just Works" Anti-Pattern: Rushing to production prioritizing functionality over security, resulting in overly permissive roles and unpatched containers.
Node-Level Over-Permissioning: Making the beginner mistake of attaching massive blast-radius policies (like AmazonS3FullAccess) to the node's environment rather than restricting permissions at the application pod level.
Bloated Container Images: Using generic, heavy Docker images that contain hundreds of unnecessary OS packages, drastically increasing the attack surface.
Lack of Automated Gates: Relying on basic "Build -> Push -> Deploy" pipelines without automated checks, which requires manual review overhead and allows misconfigurations to incur operational risk or downtime.

In this article, we will build a realistic, production-grade AWS Elastic Kubernetes Service (EKS) workload, deliberately leave it vulnerable, act as the red team to exploit it, and finally, harden it using industry-standard DevSecOps practices.

Blast radius this is a term used in software development to refer to the potential impact of a failure within a system caused by a security breach, software bug or a misconfiguration.

**Disclaimer:* The vulnerable configurations presented here are for educational lab environments only. Never deploy them to production.*

Architecture Overview

In enterprise environments, a single overly permissive IAM role or a vulnerable base image can lead to a catastrophic breach. When reviewing pipelines and architectures, Solutions Architects should use the following framework:

Platform and Orchestration Trade-offs: For this workload, we choose Amazon EKS over ECS. While ECS is simpler and tightly integrated, EKS is the enterprise standard and enables powerful native controls like OIDC-backed IAM Roles for Service Accounts (IRSA).
IaC and Pipeline Tooling: Standardize on tools like Terraform for reliable state management and GitHub Actions to enforce automated security gates while minimizing external dependencies.
Shift-Left Automation: Continuous scanning must be placed at the Pull Request (PR) stage. You must physically prevent developers from merging misconfigured code by enforcing failure states (e.g., exit-code: '1') using tools like TruffleHog, tfsec, and Trivy.
Defense in Depth: Verify that security controls are layered across the IaC layer, the pipeline layer, the container layer, and the AWS API layer.

Prerequisites & Lab Setup

To follow along in your own AWS account, ensure you have:

An active AWS Account (Note: EKS incurs a ~$73/month control plane fee. Remember to destroy resources when finished).
AWS CLI installed and configured with Administrator access.
Terraform (v1.5+), kubectl, and Docker installed locally.

Let's get your local workspace ready. Open your terminal and run:

# Step 1: Create your project workspace
mkdir my-eks-devsecops-lab
cd my-eks-devsecops-lab

# Step 2: Create the required folder structure
mkdir infrastructure src k8s .github
mkdir -p .github/workflows

# Step 3: Verify AWS authentication
aws sts get-caller-identity

Phase 1: Building the Vulnerable Baseline (Common Anti-Patterns)

We begin with a scenario I see frequently: code that "just works." We will deploy an EKS cluster and a Python app that reads data from S3. However, we are making a classic beginner mistake: giving the underlying EC2 node full access to S3, rather than restricting permissions to the specific application pod.

1. Provision the Insecure Infrastructure.

Create a file inside the infrastructure folder named main.tf. This sets up a basic EKS cluster but attaches a massive blast-radius policy (AmazonS3FullAccess) to the node group.

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      # LOCK PRODUCER: Pin to an older v5 release before they removed legacy EKS properties
      version = "5.31.0" 
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# Vulnerable IAM Role for EKS Nodes
resource "aws_iam_role" "insecure_node_role" {
  name = "insecure-eks-node-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}

# The Anti-Pattern: Giving the whole node full S3 access
resource "aws_iam_role_policy_attachment" "s3_full_access" {
  role       = aws_iam_role.insecure_node_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}

# EKS Cluster (Kept at v19 natively matching your workspace cache)
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "19.21.0"
  cluster_name    = "devsecops-lab-cluster"
  cluster_version = "1.30"

  vpc_id          = "vpc-12345678"                # Replace with your Default VPC ID
  subnet_ids      = ["subnet-123", "subnet-456"]  # Replace with your subnets


# --- ADD THESE TWO LINES TO ENABLE PUBLIC ENDPOINT ACCESS ---
  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true
  # -----------------------------------------------------------

eks_managed_node_groups = {
    vulnerable_nodes = {
      min_size     = 1
      max_size     = 2
      desired_size = 1
      iam_role_arn = aws_iam_role.insecure_node_role.arn
    }
  }
}

This Terraform configuration provisions the networking and compute resources for our Kubernetes cluster. The critical security flaw here is the aws_iam_role_policy_attachment. By attaching AmazonS3FullAccess to the insecure_node_role, we are granting every pod running on this EC2 instance full administrative control over all S3 buckets in the AWS account.

Deploy the infrastructure (this takes ~15 minutes):

cd infrastructure
terraform init
terraform apply -auto-approve
cd ..

2. Build the Vulnerable Application

Next, we write a simple script that grabs data from S3, assuming permissions are handled by the node's environment. Create src/app.py:

# src/app.py
import boto3
from flask import Flask

app = Flask(__name__)

@app.route('/')
def read_s3():
    # ANTI-PATTERN: Relies entirely on the node's underlying IAM role
    s3 = boto3.client('s3', region_name='us-east-1')
    try:
        response = s3.list_buckets()
        buckets = [bucket['Name'] for bucket in response['Buckets']]
        return f"I can see all these buckets: {', '.join(buckets)}"
    except Exception as e:
        return str(e)

if __name__ == '__main__':
    # ANTI-PATTERN: Running as root on all interfaces
    app.run(host='0.0.0.0', port=8080)

Notice that the boto3.client call does not pass any explicit AWS credentials. It blindly trusts the overarching environment (the EC2 Node) to provide access. Furthermore, the app binds to 0.0.0.0, exposing it to the entire network

Create src/requirements.txt:

flask==2.2.2
boto3==1.26.0
werkzeug==2.2.2

Create the insecure Dockerfile in the src folder:

# src/Dockerfile
# ANTI-PATTERN: Bloated image, running as root
FROM python:3.9 

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

EXPOSE 8080
CMD ["python", "app.py"]

3. Deploy to EKS.

Build, tag, and push the image to AWS ECR, and deploy it to Kubernetes.

Build the Docker image

bash
# 1. Move up one level into your project root directory
cd secure-eks-pipeline

# 2. Run your docker build command again
docker build -t insecure-app:latest ./src

Refresh the local terminal access token to bind securely to the cluster's public DNS address in short we are connecting to EKS :


aws eks update-kubeconfig --region us-east-1 --name devsecops-lab-cluster

Instantiate a private cloud registry using Amazon ECR, to authenticate the local Docker daemon to the AWS API plane, and push the securely tagged image directly to AWS:


# Create the private cloud registry repository
aws ecr create-repository --repository-name insecure-app --region us-east-1


# Authenticate local Docker to the remote AWS ECR endpoint
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $(aws sts get-caller-identity --query "Account" --output text).dkr.ecr.us-east-1.amazonaws.com


# Tag and ship the local binary layers to the cloud
export ECR_URI="$(aws sts get-caller-identity --query "Account" --output text).dkr.ecr.us-east-1.amazonaws.com/insecure-app"

docker tag insecure-app:latest $ECR_URI:latest
docker push $ECR_URI:latest

Open your blank k8s/deployment.yaml file in VS Code and paste this deployment configuration to spin up your vulnerable Python app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: insecure-app-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: insecure-app
  template:
    metadata:
      labels:
        app: insecure-app
    spec:
      containers:
      - name: web-app
        # ENTER YOUR EXACT COPIED STRING HERE WITHOUT ALTERATIONS:
        image: <YOUR_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/insecure-app:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080

Apply this manifest using kubectl:

kubectl apply -f k8s/deployment.yaml

Architectural Insight:
The python:3.9 image contains hundreds of unnecessary OS packages, increasing the attack surface. Furthermore, by not specifying a USER, the container runs as root.

Phase 2: Threat Modeling & The Red Team Practices.

As a Solution Architect, part of threat modeling is understanding how an attacker navigates your environment. Let's simulate a Server-Side Request Forgery (SSRF) or Remote Code Execution (RCE) vulnerability that gives an attacker a shell inside our pod.

1. Simulate the Breach.

Open the standard terminal on your laptop (the same one you used to run terraform apply or docker build earlier). Ensure you are connected to your AWS environment.

Type this to see a list of your running applications:

kubectl get pods

You are still in your local terminal. You are now going to tell Kubernetes to open a live connection from your laptop directly into the running container inside AWS. Type this (replacing the placeholder with the name you just copied.

kubectl exec -it deployment/insecure-app-deployment -- /bin/bash

The terminal prompt will instantly change. It will go from looking like your local computer (e.g., chimera1@fedora:~$) to looking like this:

You are no longer typing commands on your laptop. You are now physically "inside" the compromised container running in the AWS cloud. Every command you type now executes as if you were sitting at the keyboard of that specific pod.

2. Check Privileges

root@pod-1234:/app# whoami
root

Impact: Because we are root, we can install new packages (like curl or nmap) to map the internal network.

3. Abuse the Metadata Service (IMDSv1)

Because we didn't block access to the EC2 metadata service or enforce IMDSv2, we can steal the Node's IAM credentials directly from inside the container.The IP address 169.254.169.254 is a special AWS address that only works when you are inside an AWS environment.

During infrastructure verification, we noted that the cluster nodes strictly enforce IMDSv2 (Instance Metadata Service Version 2). Traditional, unauthenticated IMDSv1 GETrequests to the link-local address (http://169.254.169.254) are dropped by default, preventing basic extraction.

To bypass this securely, we had to mimic legitimate node behaviors by programmatically requesting a temporary cryptographic session token via a PUT request:


TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

Attempting to pull credentials from a static role name returned a 404 Not Found because the AWS EKS module automatically generates randomized names for node groups.

To overcome this, we queried the base security directory while passing our valid IMDSv2 session token in the request header to discover the exact, real-world identity assigned to the EC2 host:

curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/

By appending the dynamically discovered role name to our metadata request string, the endpoint immediately dumped the underlying security credentials payload:


root@pod-1234:/app# curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/vulnerable_nodes-eks-node-group-20260615172216169600000004

4. The Blast Radius Expansion

The output reveals the AccessKeyId, SecretAccessKey, and Token. An attacker can configure these on their local machine. Because the node role has AmazonS3FullAccess, the attacker bypasses the Kubernetes boundary entirely and can download or delete every S3 bucket in your AWS account.

To leave the compromised container and return to your local laptop, simply type:


exit

Phase 3: The Hardening (Secure Design Patterns)

We will now implement the "Shift-Left" paradigm. We will harden the container, implement IAM least privilege using IRSA, and build a CI/CD pipeline that blocks vulnerable code.

1. Secure the Dockerfile

Open src/Dockerfile and replace it with this hardened, minimal version:

# src/Dockerfile (HARDENED)
# SECURE: Minimal base image reduces attack surface
FROM python:3.9-slim

# SECURE: Create a non-root user and group
RUN groupadd -r appgroup && useradd -r -g appgroup appuser

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

# SECURE: Hand over ownership and switch to non-root user
RUN chown -R appuser:appgroup /app
USER appuser

EXPOSE 8080
CMD ["python", "app.py"]

Using python:3.9-slim drastically cuts down the number of pre-installed OS utilities, reducing the attack surface. By explicitly creating appuser and ending the file with USER appuser, we physically prevent the container from executing as root. If an attacker gains a shell now, they will receive "Permission Denied" errors when trying to install network mapping tools.

2. Implement IAM Roles for Service Accounts (IRSA)

Instead of giving the EC2 node permissions, we give permissions only to the specific Kubernetes ServiceAccount used by our app via OIDC. Add this to your infrastructure/main.tf:

# infrastructure/main.tf (Additions)

# SECURE: Create a strict, least-privilege policy
resource "aws_iam_policy" "app_s3_policy" {
  name        = "StrictAppS3Policy"
  description = "Allows reading only from the specific app bucket"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:GetObject"]
      Resource = ["arn:aws:s3:::my-specific-app-bucket/*"] # Restricted to ONE bucket
    }]
  })
}

# SECURE: Map the IAM Role to the Kubernetes Service Account via OIDC
module "iam_eks_role" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name = "SecureAppRole"

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["default:secure-app-sa"]
    }
  }

  role_policy_arns = {
    policy = aws_iam_policy.app_s3_policy.arn
  }
}

This block decouples application permissions from infrastructure permissions. It creates an IAM policy scoped down to a single S3 bucket, then uses the AWS OIDC (Open ID Connect ) provider to map that policy strictly to the secure-app-sa Kubernetes ServiceAccount. The underlying EKS node remains entirely unprivileged.

You also need to create the Kubernetes ServiceAccount and annotate it with the IAM role ARN. Create k8s/serviceaccount.yaml:\

# k8s/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: secure-app-sa
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<YOUR_ACCOUNT_ID>:role/SecureAppRole

Replace <YOUR_ACCOUNT_ID> with your actual AWS account ID. You can also use Terraform outputs to dynamically inject this value.

Then update your deployment to use this ServiceAccount:


# k8s/secure-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      serviceAccountName: secure-app-sa  # <-- CRITICAL: Use the IRSA ServiceAccount
      containers:
      - name: web-app
        image: <YOUR_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/secure-app:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080

3. Build the Automated Security Gates

We cannot trust developers to always remember these rules; we must enforce them programmatically. Create .github/workflows/main.yml:

# .github/workflows/main.yml
name: DevSecOps Pipeline

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

jobs:
  security-scanning:
    runs-on: ubuntu-latest
    permissions:
      contents: read        # Required for checkout
      security-events: write # Required for SARIF upload
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      # GATE 1: Secret Scanning
      - name: TruffleHog Secret Scan
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.repository.default_branch }}
          head: HEAD
          extra_args: --debug --only-verified

      # GATE 2: Scan Terraform for misconfigurations before deployment
      - name: Checkov IaC Scan
        uses: bridgecrewio/checkov-action@master
        with:
          directory: ./infrastructure
          framework: terraform
          output_format: cli
          soft_fail: false
          # Fails the build on CRITICAL/HIGH misconfigurations

      # GATE 3: Build the Docker image
      - name: Build Image
        run: docker build -t my-app:latest ./src

      # GATE 4: Scan the Docker image for Vulnerabilities
      - name: Trivy Vulnerability Scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'my-app:latest'
          format: 'table'
          exit-code: '1'          # Fails the build on HIGH/CRITICAL CVEs
          ignore-unfixed: true
          severity: 'CRITICAL,HIGH'
          vuln-type: 'os,library'

*Architectural Insight: *
By placing tools like Checkov and Trivy in the CI/CD pipeline, we enforce "Shift-Left" automation. If these tools detect a misconfiguration (like an overly permissive IAM role) or a vulnerable package, they trigger an exit-code: '1', physically preventing developers from merging vulnerable code into the main branch.

Phase 4: Validating the Security Gates (Planned Pipeline Failure)

To prove that our Shift-Left automation handles threats before they reach production, we pushed our code to GitHub to trigger the DevSecOps workflow. The pipeline initiated, cleared the TruffleHog secret check, logged infrastructure misconfigurations via tfsec using a warning threshold, and built our container image.

However, upon reaching GATE 3 (Trivy Vulnerability Scanner), the pipeline intentionally failed with an exit code 1.

Looking at the logs, Trivy intercepted 25 OS vulnerabilities (including Critical heap buffer overflows in openssl) and 10 Python application-layer vulnerabilities (such as a session cookie disclosure flaw in Flask 2.2.2). Because our workflow enforces a strict security ceiling on CRITICAL,HIGH anomalies, the pipeline automatically broke, preventing an unhardened, insecure workload from ever spinning up inside our AWS production environment. This is exactly how automated preventative controls protect enterprise cloud platforms.

Real CVE Context: Flask 2.2.2 and Werkzeug 2.2.2 are known to have multiple vulnerabilities including CVE-2023–30861 (session cookie disclosure), CVE-2023–23934 (cookie parsing issue), CVE-2023–25577 (DoS via multipart parsing), and CVE-2024–34069 (RCE via debugger). Upgrading to Flask 2.3.x+ and Werkzeug 3.0.3+ resolves these.

Validation, Monitoring & Teardown

In Phase 2 we successfully stole the EC2 node's credentials using curl. We fix this at the infrastructure level using Terraform. By enforcing IMDSv2 and setting the "hop limit" to 1, we ensure that a pod running on the node cannot reach the metadata service. Security is an ongoing operational task. To ensure long-term visibility:

Enforce IMDSv2 Block IMDSv1:

Enforce IMDSv2 via launch templates. Require session tokens and set a HttpPutResponseHopLimit of 1 to prevent pods from accessing node metadata.

1. Open your code editor: Open the infrastructure/main.tf file you created in Phase 1.

2. Modify the Node Group: Scroll down to your eks_managed_node_groups block. You are going to add a metadata_options section to enforce IMDSv2. Update the block to look exactly like this:

eks_managed_node_groups = {
    vulnerable_nodes = { # Or secure_nodes if you renamed it
      min_size     = 1
      max_size     = 2
      desired_size = 1
      iam_role_arn = aws_iam_role.insecure_node_role.arn

      # ADD THIS BLOCK: Enforce IMDSv2 and block pod access
      metadata_options = {
        http_endpoint               = "enabled"
        http_tokens                 = "required" # This forces IMDSv2
        http_put_response_hop_limit = 1          # Prevents pods from reading it
      }
    }
  }

While requiring http_tokens enforces IMDSv2, the http_put_response_hop_limit = 1 is the ultimate safeguard. This setting ensures that metadata network packets can only travel 1 network "hop." Because Kubernetes pods run on a secondary virtual network interface inside the EC2 node, accessing the metadata service requires at least 2 hops. Setting the limit to 1 effectively kills the packet before it can reach the pod, completely neutralizing SSRF credential theft.

3. Apply the changes in your local terminal: Open your terminal on your laptop, ensure you are in the infrastructure folder, and apply the secure configuration to AWS:

cd infrastructure
terraform apply -auto-approve

Architectural Insight :
If you try to run that same malicious curl command inside the container now, it will return a 401 Unauthorized or timeout error. The vulnerability is successfully patched.

cd ..

# Verify you are in the right place (it should show 'src' and 'infrastructure')
ls

# Build the hardened Docker image
docker build -t secure-app:latest ./src

# Update kubeconfig
aws eks update-kubeconfig --region us-east-1 --name devsecops-lab-cluster

# Authenticate to ECR
aws ecr get-login-password --region us-east-1 |   docker login --username AWS --password-stdin   $(aws sts get-caller-identity --query "Account" --output text).dkr.ecr.us-east-1.amazonaws.com

# Tag and push the hardened image
export ECR_URI="$(aws sts get-caller-identity --query "Account" --output text).dkr.ecr.us-east-1.amazonaws.com/secure-app"

docker tag secure-app:latest $ECR_URI:latest

aws ecr create-repository \
  --repository-name secure-app \
  --region us-east-1 \
  --image-scanning-configuration scanOnPush=true

docker push $ECR_URI:latest

# Apply the ServiceAccount and secure deployment
cd k8s
kubectl apply -f serviceaccount.yaml
kubectl apply -f secure-deployment.yaml

# Verify pods are running
kubectl get pods

# Test the hardening: try to exec into the pod and access metadata
kubectl exec -it deployment/secure-app-deployment -- /bin/bash

# Inside the container, try the old attack:
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token"   -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/insecure-eks-node-role

# You expect to receive nothing in return - it shows the architecture has been hardened.
exit

You expect to receive nothing in return it shows it has been hardened.

Deploying Runtime Security:

Deploy tools like Falco or AWS GuardDuty for EKS to monitor for anomalous behavior.Though Part 2 is dedicated to building out the Falco architecture. However, if you wanted to see the literal commands of how a Solutions Architect deploys it into this cluster right now, it is done using Helm (a package manager for Kubernetes) from your local laptop terminal.


# Add the Falco Helm repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install Falco with eBPF probe (modern, efficient kernel tracing)
helm install falco falcosecurity/falco   --namespace falco   --create-namespace   --set driver.kind=ebpf   --set tty=true

Crucial Step: Lab Teardown

As a responsible cloud practitioner, always clean up your environment to avoid unexpected AWS billing. Open your terminal:

cd infrastructure
terraform destroy -auto-approve

Key Architectural Lessons

Mastering these concepts separates you from the pack. By walking through this architecture, you demonstrate:

Blast Radius Reduction: Decoupling application permissions from infrastructure permissions.
Shift-Left Automation: Integrating SCA, IaC scanning, and vulnerability checks into CI/CD to reduce manual review overhead.
Defense in Depth: Applying security at the IaC layer, pipeline layer, container layer, and AWS API layer.
Metadata Service Protection: Using IMDSv2 with hop limit 1 to prevent container credential theft.
Minimal Container Images: Reducing attack surface by using slim base images and non-root users.

Conclusion & Extensions

A Secure-by-Default architecture requires acknowledging that developers will make mistakes, and building systems that catch those mistakes before they become breaches. To extend this architecture, consider implementing AWS Organizations with Service Control Policies (SCPs) to establish hard boundaries across multiple accounts, or implementing Cosign to digitally sign your container images before pushing them to Amazon ECR.

Bonus: DevSecOps Decision Matrix

Keep this matrix handy for your next architecture review or certification exam.

Security Domain	Vulnerable Configuration (Anti-Pattern)	Secure Configuration (Best Practice)	SA Justification & Impact
IAM Strategy	EKS Node Group roles with broad permissions (`AmazonS3FullAccess`).	IAM Roles for Service Accounts (IRSA) with OIDC.	Limits blast radius. A compromised pod only yields permissions specific to its app.
Container Build	Running as `root`, using bloated base images (e.g., `python:3.9`).	`USER nonroot`, minimal/distroless base images.	Reduces attack surface. Attackers cannot easily install external tools if a breakout occurs.
Metadata Security	IMDSv1 enabled, accessible from pods.	Require IMDSv2, Hop Limit = 1.	Prevents Server-Side Request Forgery (SSRF) attacks from extracting EC2 credentials.
Pipeline Security	Build -> Push -> Deploy (No automated checks).	TruffleHog -> tfsec -> Trivy -> Deploy.	Catches vulnerabilities at the PR stage before they incur operational risk or downtime.

Agentic DevOps: Automating Security Remediation on AWS Using AWS DevOps Agent.

Elijah Chimera — Tue, 05 May 2026 18:50:33 +0000

Project Overview

The evolution of cloud operations has reached a critical inflection point characterized by the transition from reactive infrastructure management to autonomous, self-healing systems. As distributed computing architectures scale in complexity, the cognitive load placed on Site Reliability Engineering (SRE) and cybersecurity teams has dramatically exceeded the capabilities of manual telemetry correlation, traditional heuristic-based alerting, and static posture management. The general availability of the AWS DevOps Agent as of March 31, 2026, marks the maturation of "frontier agents" which are autonomous artificial intelligence systems capable of end-to-end incident ownership, sophisticated root cause analysis (RCA), and proactive remediation across hybrid, multi-cloud, and on-premises environments.

This comprehensive technical report details the architectural paradigms, security implications, and operational workflows introduced by the AWS Dev Ops Agent. It specifically addresses the integration of the Model Context Protocol (MCP), an open-source standard that enables the agentic reasoning engine to securely interface with localized telemetry data sources, such as Linux-based development environments which in my case I will be running Fedora.

To demonstrate the practical application of these technologies, this report presents an exhaustive, step-by-step walk-through focused on architecting a self-healing security environment. The implementation involves constructing a continuous integration and continuous deployment (CI/CD) pipeline for a web application using Infrastructure as Code (IaC) via Terraform.

The infrastructure is intentionally deployed with specific security misconfigurations such as overly permissive Identity and Access Management (IAM) roles and publicly exposed Amazon Simple Storage Service (S3) buckets. By simulating sophisticated cyber incidents, including brute force attacks and unauthorized configuration drift, the analysis illustrates how the AWS DevOps Agent bridges the gap between pen-testing methodologies and autonomous Dev Ops remediation.

The evidence establishes that integrating topology-aware frontier agents reduces mean time to resolution (MTTR) by up to 75%, shifting the organizational posture from reactive firefighting to proactive, AI-driven resilience.

Key Concepts

To establish a foundational understanding of the interconnected technologies governing autonomous cloud operations in this architecture, the following critical concepts must be defined:

Architectural Framework of Autonomous Operations

The structural design of the AWS Dev Ops Agent is predicated on a dual-console model, which strictly segregates administrative governance from operational execution. The Administration Console, housed within the AWS Management Console, facilitates the configuration of integrations, permissions, and security perimeters by platform administrators. Conversely, the Operations Web App provides a dedicated workspace for SRE and cybersecurity teams to interact with the agent, review investigation journals, and approve mitigation plans via natural language interfaces.

Agent Spaces and Multi-Layered Security Boundaries

Central to the architecture of the AWS Dev Ops Agent is the Agent Space. An Agent Space functions as an isolated logical container that determines the agent's operational boundary. The configuration of an Agent Space requires a precise balance; an overly restrictive configuration deprives the agent of the telemetry required to synthesize an accurate root cause analysis, whereas an excessively broad space increases computational overhead and introduces unnecessary security exposure.

The security of the AWS Dev Ops Agent is maintained through a multi-layered defense-in-depth strategy. Even if broader permissions are granted to the agent's underlying IAM role, the agent enforces internal access controls to limit the scope of its actions. The agent relies on regional processing capabilities, meaning it retrieves operational data from AWS regions across all AWS accounts granted access within the configured Agent Space. While the underlying Amazon Bedrock models automatically select optimal regions for inference processing to maximize compute availability, the architectural design guarantees that customer data remains stored only in the region where the Agent Space was instantiated, ensuring data residency compliance.

The Investigation Journal and Immutable Audit Trails

When an incident is triggered whether via an automated web hook from an Amazon CloudWatch alarm, a ServiceNow ticket, or manual invocation the agent initiates a systematic diagnostic workflow. The core operational artifact produced during this process is the Investigation Journal.

The Investigation Journal provides a transparent, real-time, and immutable chronicle of the agent's reasoning process. It records the initial hypothesis generation, documenting every telemetry query, metric evaluation, and log parsing action executed. Because these logs are integrated with AWS CloudTrail, they serve as a cryptographically verifiable audit trail. This transparency is critical for cybersecurity compliance, allowing human operators to retrospectively analyze exactly why the AI deemed a specific IAM role overly permissive or why it concluded that a database connection failure was the result of a Transit Gateway routing anomaly.

The Model Context Protocol (MCP): Bridging Hybrid and Local Environments

While the AWS Dev Ops Agent possesses native integration with centralized AWS services, enterprise environments operate complex hybrid topologies incorporating on-premises data centers, private code repositories, localized developer workstations, and third-party observability platforms. To extend the agent's topological awareness into these disparate domains, the architecture leverages the Model Context Protocol (MCP).

MCP resolves the fragility of bespoke API engineering by implementing a standardized client-server architecture. The AI application acts as the MCP Client, requesting access to external data, while the MCP Server exposes specific tools, resources, and capabilities.

Transport Mechanisms and Linux Implementations

The protocol operates over two primary transport mechanisms: stdio (Standard Input/Output) and Streamable HTTP.

The stdio mechanism is optimized for localized execution. For example, a security analyst or system administrator operating a Linux workstation can deploy the linux-mcp-serverlocally. The MCP server runs as a subprocess, inheriting the granular file system permissions of the host operator. This architecture enables the cloud-based AWS DevOps Agent to query the local Linux machine's system state, review journalctl logs, analyze running processes, and diagnose performance bottlenecks directly from the local environment. By providing the agent direct access to system information, manual extraction and pasting of terminal output into a web interface are eliminated, transforming the AI into an active participant in the troubleshooting process.

For remote or enterprise-grade deployments, MCP utilizes Streamable HTTP with Server-Sent Events (SSE) to facilitate real-time streaming. This mechanism is critical for connecting the AWS DevOps Agent to persistent infrastructure tools such as self-hosted GitLab instances, private Terraform registries, or containerized observability stacks running on Amazon Elastic Container Service (ECS) with AWS Fargate.

Private Connections and Amazon VPC Lattice

A critical security consideration when bridging the AWS DevOps Agent with internal enterprise tools is network isolation. Exposing internal databases or localized MCP servers to the public internet violates fundamental zero-trust principles. To resolve this, the architecture utilizes Amazon VPC Lattice to establish secure "Private Connections".

VPC Lattice operates as an application networking layer that establishes a secure, encrypted transit path between the Agent Space and the target resource residing in an isolated Virtual Private Cloud (VPC). The connection process involves deploying service-managed Elastic Network Interfaces (ENIs) directly into the specified subnets. Traffic from the DevOps Agent is routed through these ENIs which act as a resource gateway ensuring that requests to private MCP servers, self-hosted Grafana dashboards, or internal documentation APIs never traverse the public internet. The protocol strictly enforces HTTPS communication with a minimum of TLS 1.2, guaranteeing data-in-transit encryption.

Walkthrough: Deploying the Self-Healing Security Architecture

To effectively demonstrate the capabilities of the AWS DevOps Agent in mitigating vulnerabilities and configuration drift, this section details the deployment of a controlled laboratory environment. The objective is to construct a CI/CD pipeline for a web application utilizing Terraform, deliberately introducing security misconfigurations—specifically, an overly permissive IAM role and an exposed S3 bucket. Subsequently, the environment is monitored by the AWS DevOps Agent, which is configured to detect, investigate, and autonomously remediate simulated cyber incidents.

1. Local Environment Preparation on Fedora Linux

The initialization phase requires configuring the local developer workstation running Linux. This involves installing the requisite MCP server to allow the AWS DevOps Agent to read local development configurations and interact with the local Terraform state during the investigation phase.First, the system dependencies are updated and installed via the package manager it could be apt opr dnf depending on your distribution:


shell
Bash
# Update local Fedora repositories and install Node.js
sudo dnf update -y
sudo dnf install nodejs npm 
brew install awscli 
brew tap hashicorp/tap
brew install hashicorp/tap/terraform



# Verify installations
node --version
terraform --version
aws --version

Next, the linux-mcp-server is deployed. This utility grants the AI agent context regarding the local operating system environment, which is highly beneficial when analyzing hybrid configurations or diagnosing connectivity issues originating from the developer's workstation.Execute this directly in your terminal:

Bash
# Execute the Linux MCP server via NPX for local standard I/O transport
curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install linux-mcp-server
linux-mcp-server

Simultaneously, the HashiCorp Terraform MCP server must be configured to provide the AWS DevOps Agent with real-time access to the public Terraform Registry, allowing the agent to reference accurate, up-to-date documentation when generating remediation code. The server is initialized locally using the Streamable HTTP mode to facilitate remote connections from the cloud-based Agent Space.Open a seperate terminal window and run:


Bash
# Start the Terraform MCP server on port 8080
brew install terraform-mcp-server
terraform-mcp-server streamable-http \
  --transport-port 8080 \
  --transport-host 0.0.0.0 \
  --log-level info \
  --toolsets terraform,registry

2.Infrastructure as Code Configuration

The foundational infrastructure is codified using Terraform. This ensures that the deployment is reproducible and that subsequent remediations generated by the AWS DevOps Agent can be seamlessly integrated into the version control system. The following configuration defines the core networking, the deliberately vulnerable S3 bucket, and the highly permissive IAM role.

The implementation utilizes the hashicorp/aws provider for standard resources and the hashicorp/awscc provider to interact with the AWS Cloud Control API, which is required to provision the awscc_devopsagent_agent_spaceresource. Create a file named main.tf and paste the following code:


terraform {
  required_providers {
    aws   = { source = "hashicorp/aws", version = "~> 5.0" }
    awscc = { source = "hashicorp/awscc", version = "~> 1.0" }
  }
}

provider "aws" {
  region = "us-east-1"
}

provider "awscc" {
  region = "us-east-1"
}

data "aws_caller_identity" "current" {}

# -------------------------------------------------------------------------
# 1. IAM ROLE FOR EC2 (CloudWatch Logs Shipping)
# -------------------------------------------------------------------------
resource "aws_iam_role" "ec2_logs_role" {
  name = "EC2LogShippingRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cw_agent_policy" {
  role       = aws_iam_role.ec2_logs_role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

resource "aws_iam_instance_profile" "ec2_profile" {
  name = "EC2LogShippingProfile"
  role = aws_iam_role.ec2_logs_role.name
}

# -------------------------------------------------------------------------
# 2. SECURITY GROUP & EC2 INSTANCES
# -------------------------------------------------------------------------
resource "aws_security_group" "lab_sg" {
  name = "free-tier-lab-sg"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_instance" "vulnerable_target" {
  ami                    = "ami-0c101f26f147fa7fd"
  instance_type          = "t3.micro"
  iam_instance_profile   = aws_iam_instance_profile.ec2_profile.name
  vpc_security_group_ids = [aws_security_group.lab_sg.id]

  user_data = <<-EOF
              #!/bin/bash
              echo "root:password123" | chpasswd
              sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
              sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
              systemctl restart sshd

              dnf install -y amazon-cloudwatch-agent
              cat <<ETX > /opt/aws/amazon-cloudwatch-agent/bin/config.json
              {
                "logs": {
                  "logs_collected": {
                    "files": {
                      "collect_list": [
                        {
                          "file_path": "/var/log/secure",
                          "log_group_name": "ssh-auth-logs",
                          "log_stream_name": "{instance_id}"
                        }
                      ]
                    }
                  }
                }
              }
              ETX
              /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
              EOF

  tags = { Name = "Victim-Server" }
}

resource "aws_instance" "attacker_node" {
  ami                    = "ami-0c101f26f147fa7fd"
  instance_type          = "t3.micro"
  vpc_security_group_ids = [aws_security_group.lab_sg.id]

  user_data = <<-EOF
              #!/bin/bash
              dnf install -y hydra
              echo -e "password123\n123456\nadmin\nroot" > /home/ec2-user/common-passwords.txt
              EOF

  tags = { Name = "Attacker-Hydra" }
}

# -------------------------------------------------------------------------
# 3. CLOUDWATCH LOG GROUP
# -------------------------------------------------------------------------
resource "aws_cloudwatch_log_group" "ssh_logs" {
  name              = "ssh-auth-logs"
  retention_in_days = 1
}

# -------------------------------------------------------------------------
# 4. LAMBDA DISPATCHER
# -------------------------------------------------------------------------
resource "aws_iam_role" "lambda_role" {
  name = "LambdaDispatcherRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_basic" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

resource "aws_lambda_function" "incident_dispatcher" {
  filename         = "lambda_function.zip"
  function_name    = "DevOpsAgent-Dispatcher"
  role             = aws_iam_role.lambda_role.arn
  handler          = "index.handler"
  runtime          = "python3.11"
  source_code_hash = filebase64sha256("lambda_function.zip")

  environment {
    variables = {
      AGENT_SPACE_ID = awscc_devopsagent_agent_space.security_lab_space.id
    }
  }
}

resource "aws_lambda_permission" "allow_cloudwatch" {
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.incident_dispatcher.function_name
  principal     = "logs.us-east-1.amazonaws.com"
  source_arn    = "${aws_cloudwatch_log_group.ssh_logs.arn}:*"
}

resource "aws_cloudwatch_log_subscription_filter" "brute_force_filter" {
  name            = "detect-failed-login"
  log_group_name  = aws_cloudwatch_log_group.ssh_logs.name
  filter_pattern  = "Failed password"
  destination_arn = aws_lambda_function.incident_dispatcher.arn
}

# -------------------------------------------------------------------------
# 5. INTENTIONAL VULNERABILITIES (S3 + Overly Permissive Role)
# -------------------------------------------------------------------------
resource "aws_s3_bucket" "app_data_bucket" {
  bucket = "vulnerable-app-data-lab-2026"
}

resource "aws_s3_bucket_public_access_block" "app_data_public_access" {
  bucket                  = aws_s3_bucket.app_data_bucket.id
  block_public_acls       = false
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}

resource "aws_s3_bucket_policy" "allow_public_read" {
  bucket = aws_s3_bucket.app_data_bucket.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "PublicRead"
        Effect    = "Allow"
        Principal = "*"
        Action    = "s3:GetObject"
        Resource  = "${aws_s3_bucket.app_data_bucket.arn}/*"
      }
    ]
  })
}

resource "aws_iam_role" "app_execution_role" {
  name = "AppExecutionRole-OverlyPermissive"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "admin_access_attachment" {
  role       = aws_iam_role.app_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}

# -------------------------------------------------------------------------
# 6. DEVOPS AGENT IAM ROLE 
# -------------------------------------------------------------------------
resource "aws_iam_role" "devops_agent_role" {
  name = "DevOpsAgentMonitoringRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "aidevops.amazonaws.com"
        }
        Action = "sts:AssumeRole"
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = data.aws_caller_identity.current.account_id
          }
          ArnLike = {
            "aws:SourceArn" = "arn:aws:aidevops:us-east-1:${data.aws_caller_identity.current.account_id}:agentspace/*"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "devops_agent_policy" {
  role       = aws_iam_role.devops_agent_role.name
  policy_arn = "arn:aws:iam::aws:policy/AIDevOpsAgentAccessPolicy"
}

resource "aws_iam_role_policy" "devops_agent_inline" {
  name = "DevOpsAgentExtraPermissions"
  role = aws_iam_role.devops_agent_role.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "iam:CreateServiceLinkedRole"
        ]
        Resource = "arn:aws:iam::*:role/aws-service-role/resource-explorer-2.amazonaws.com/AWSServiceRoleForResourceExplorer"
        Condition = {
          StringEquals = {
            "iam:AWSServiceName" = "resource-explorer-2.amazonaws.com"
          }
        }
      }
    ]
  })
}

# -------------------------------------------------------------------------
# 7. AWS DEVOPS AGENT CONFIGURATION
# -------------------------------------------------------------------------
resource "awscc_devopsagent_agent_space" "security_lab_space" {
  name        = "self-healing-security-lab"
  description = "Agent Space dedicated to monitoring and remediating the vulnerable web application pipeline."
}

resource "awscc_devopsagent_association" "account_association" {
  agent_space_id = awscc_devopsagent_agent_space.security_lab_space.id
  service_id     = "aws"

  configuration = {
    aws = {
      assumable_role_arn = aws_iam_role.devops_agent_role.arn
      account_id         = data.aws_caller_identity.current.account_id
      account_type       = "monitor"
      resources          = []
    }
  }

  depends_on = [
    awscc_devopsagent_agent_space.security_lab_space,
    aws_iam_role.devops_agent_role,
    aws_iam_role_policy_attachment.devops_agent_policy,
    aws_iam_role_policy.devops_agent_inline
  ]
}

3. The Dispatcher Logic file index.py

Because CloudWatch sends its logs in a compressed format, the Lambda function requires a small Python script to "unpack" the data before transmitting it to the AWS DevOps Agent. Create a file named index.py, compress it into lambda_function.zip, and place it in the same directory as your Terraform file:


import json
import gzip
import base64
import os

def handler(event, context):
    # 1. Decode and decompress the CloudWatch log data
    cw_data = event['awslogs']['data']
    compressed_payload = base64.b64decode(cw_data)
    uncompressed_payload = gzip.decompress(compressed_payload)
    payload = json.loads(uncompressed_payload)

    print(f"Brute force detected in logs! Events: {len(payload['logEvents'])}")

    # 2. Logic to notify DevOps Agent
    # In a lab, we log the intent. In prod, you'd POST to the Agent endpoint.
    return {
        "status": "incident_reported",
        "agent_space": os.environ,
        "source_ip": "Extracted from log line"
    }

The infrastructure is then deployed to the AWS environment using standard Terraform execution commands:


Bash
terraform init
terraform plan -out=tfplan
terraform apply "tfplan"

Upon successful execution, the intentionally vulnerable S3 bucket, the permissive IAM role, and the AWS DevOps Agent Space are active within the specified AWS region.

Simulating Security Incidents and Configuration Drift

With the foundational architecture established, the next phase involves simulating cybersecurity incidents that trigger the autonomous response capabilities of the AWS DevOps Agent. This methodology aligns directly with pentesting practices, where understanding the mechanics of an exploit is prerequisite to engineering an automated defense.

Incident Simulation 1: Configuration Drift

Configuration drift occurs when the operational state of the infrastructure diverges from the intended state defined in the Terraform configuration. In this scenario, a simulated malicious actor or a negligent administrator accesses the AWS Management Console and manually alters the S3 bucket policy to grant s3:PutObject(write access) to the public, escalating the risk from a data leak to a potential data corruption or malware hosting vector.

Furthermore, the administrator manually attaches a secondary inline policy to the AppExecutionRole-OverlyPermissive granting cross-account STS assume role capabilities, an action that was not codified in the main.tf file.

Step-by-Step Simulation:

Sign in to the AWS Management Console.
Search for S3 in the top search bar and open the S3 service.
Locate and select the bucket named vulnerable-app-data-lab-2026.
Navigate to the Permissions tab.
Click Edit on the Bucket Policy. Alter the JSON payload to explicitly grant the s3:PutObject(write access) action to Principal: "*". Click Save changes. This escalates the risk from a potential data leak to a data corruption/malware hosting vector.
Next, search for IAM in the top search bar and open the IAM console.
Click on Roles in the left navigation pane and search for AppExecutionRole-OverlyPermissive.
You will see two permissions created from our code as follows:

Incident Simulation 2: SSH Brute Force Attack

To validate the agent's capability to respond to active threats, a brute force attack is simulated against the EC2 instance associated with the application.

Step-by-Step Simulation:

Using AWS Systems Manager (SSM) Session Manager or SSH, log into the Attacker-Hydra EC2 instance created by Terraform.

 ssh -i "your-key.pem" ec2-user"ec2 instance public ip address"

The user data script has already installed Hydra and created a dictionary file. Execute the following command against the Victim-Server private IP address.

 Bash
  # Simulating a brute force attack using Hydra from the attacker node
  hydra -l root -P /home/ec2-user/common-passwords.txt ssh://<victim-server-private-ip>

If hydra did not install run the following commands first :

  # 1. Update system
  sudo dnf update -y 
 # 2. Install dependencies (very important)
  sudo dnf install -y gcc make libssh-devel openssl-devel libidn-devel \  pcre-devel libpq-devel libmemcached-devel libpcap-devel \                     git 
  # 3. Download Hydra source code
 git clone https://github.com/vanhauser-thc/thc-hydra.git
  cd thc-hydra

  # 4. Compile and install
 ./configure
  make
  sudo make install

  # 5. Verify installation
 hydra -h

Navigate to the CloudWatch console, then Log Management then Log groups , and open ssh-auth-logs. You will see the victim server actively shipping its authentication failure logs.

The Subscription Filter catches the "Failed password" regex, immediately triggering the DevOpsAgent-Dispatcher Lambda function. This zero-cost mechanism subsequently fires the secure webhook, initiating the agent's autonomous RCA workflow.

Autonomous RCA and Pull Request Orchestration

Upon receiving the alert trigger, the AWS DevOps Agent transitions into the active investigation phase. Operating independently of human interaction, the agent correlates the disparate telemetry streams to formulate a cohesive understanding of the incident.

The Investigation Workflow

The agent's activities are transparently recorded in the Operations Web App's Investigation Journal.To follow along with the agent's logic:

Step-by-Step RCA Observation

In the AWS Management Console, search for AWS DevOps Agent and open the service.
Click on the Agent Spaces menu and select the self-healing-security-lab space you deployed earlier.
Select Configure Web App and leave everything as it is and proceed to Open the Operations Web App and navigate to the Incident Response tab to view active investigations.
Click on the incidents tab. You are now viewing the Incidence Response Dashboard. Click Start investigation and fill in the following in the blank spaces and Click Start investigating :

Investigation details
Please investigate a recent multi-vector security incident. First, determine the source and impact of repeated SSH brute force attempts against our EC2 instance. Second, investigate unauthorized configuration drift that granted public write access to the vulnerable-app-data-lab-2026 S3 bucket and escalated the permissions of the AppExecutionRole-OverlyPermissive IAM role.
Investigation starting point
Start by analyzing the ssh-auth-logs CloudWatch log group for "Failed password" events. Then, cross-reference AWS CloudTrail management events to identify who recently modified the S3 bucket policy and who attached the secondary STS inline policy to the IAM role.
Name your investigation
(You can replace the default timestamp with something more readable, such as:) Incident-Security-Drift-Brute-Force

1. Investigation findings
When the investigation completes, the agent populates the Investigation summary and Root cause tabs within the Operations Web App. What makes the DevOps Agent so powerful is its ability to contextually differentiate between intended Infrastructure as Code (IaC) deployments and malicious or negligent out-of-band changes..

a. The Root Causes (Attributing the Drift)
The agent successfully maps the multi-vector incident back to two primary root causes:
Unauthorized Manual Actions:
It explicitly flags that the IAM user Chimera bypassed the Terraform pipeline and used the Firefox browser (from IP 196.96.170.247) to escalate the S3 bucket policy to public-read-write and attach the dangerous sts:AssumeRole inline policy. Crucially, the agent notes that this user session was created without MFA authentication, identifying weak access controls as a core vulnerability.
Insecure Infrastructure Baseline:
The agent accurately determines that the foundational infrastructure was deployed via Terraform by a different user (ECR-Access). However, it points out that this baseline was inherently insecure to begin with—it featured a security group allowing SSH access from 0.0.0.0/0, disabled S3 Public Access Blocks, and an overly permissive AdministratorAccess managed policy attached to an EC2 role.
2. Key Findings (The Brute Force Attack)
The agent correctly identifies the mechanics of the simulated attack. It notes that the Attacker-Hydra EC2 instance executed a deliberate brute force using the Hydra password cracking tool. Because both instances resided on the same subnet with a shared, wide-open security group, the attack succeeded against the weak password123 credentials defined in the EC2 user data.
3. Investigation Gaps (The Observability Moment)
Perhaps the most impressive part of the investigation is the agent's ability to identify what it couldn't see. While investigating the SSH attack, it flagged three critical observability gaps:
Broken Telemetry: The ssh-auth-logs log group existed but contained 0 bytes of data, meaning the CloudWatch agent on the Victim-Server failed to successfully ship the logs or was disrupted.
Missing Network Visibility: It attempted to query DescribeFlowLogs but returned an empty array, highlighting that VPC Flow Logs were entirely disabled, depriving the environment of network-level traffic evidence.
Missing Threat Detection: Finally, it checked for Amazon GuardDuty detectors and found none, noting that without GuardDuty, there was no AWS-native threat intelligence available for the account.
2. Topology Mapping:

The topology below shows a simulated AWS security scenario where an attacker EC2 instance (**Attacker-Hydra**) is directly connected to a target EC2 instance (**Victim-Server**), indicating that network access most likely SSH on port 22 is allowed between them. The victim is protected by a security group (**free-tier-lab-sg**), which appears to permit inbound access broadly (a common misconfiguration), making it vulnerable to brute-force login attempts. This setup represents a typical attack path where an exposed service is reachable from an external or malicious host.
At the same time, the architecture includes a logging pipeline: the victim instance is attached to an instance profile (**EC2LogShippingProfile**) and IAM role (**EC2LogShippingRole**) that allow it to send authentication logs to a CloudWatch log group (**ssh-auth-logs**). This means the system is instrumented for monitoring and detection of suspicious activity like repeated failed SSH logins, but not necessarily protected against them.

Generating the Remediation

To achieve this, click the Go to root cause button in the timeline.

Having completed the Root Cause Analysis (RCA), the agent formulates a mitigation strategy. The objective is twofold: halt the immediate brute force threat and permanently remediate the underlying configuration drift through codified infrastructure changes.

Once the root cause is identified, clicking the Generate mitigation plan button reveals exactly why frontier agents are replacing static runbooks. Instead of just offering a generic suggestion to "fix the S3 bucket," the AWS DevOps Agent generates a comprehensive, five-phase SRE mitigation runbook of AWS CLI commands.

Here is a breakdown of the generated mitigation plan and why each step matters for enterprise operations:

Phase 1: Prepare (State Capture)

Before modifying any resources, the agent instructs you to capture the current, drifted state. This is a critical safety mechanism.

Step 1.1 & 1.2:

It provides aws iam get-role-policy and aws s3api get-bucket-policy commands to save the exact JSON payloads of the drifted policies.

Step 1.3:

It captures the aws s3api get-bucket-policy-status to establish a baseline for post validation.

Why this matters: If reverting the drift breaks an unexpected dependency, you have the exact JSON needed to undo the change immediately.

Phase 2: Pre-Validate (Impact Assessment)

A human engineer might rush to delete the overly permissive policy, accidentally breaking a legitimate workload. The agent prevents this by validating preconditions.

Step 2.1:

It runs aws iam list-role-policies to confirm the rogue STSAssumeRole policy actually still exists before attempting deletion.

Step 2.2:

It runs aws ec2 describe-instances filtered by the target IAM profile. This is genius. It checks if any live EC2 instances are currently using the compromised role, allowing you to coordinate with stakeholders before pulling the plug.

Step 2.3 & 2.4:

It uses aws iam generate-service-last-accessed-details to see if the dangerous sts:AssumeRole permission has actually been exploited recently.

Phase 3: Apply (The Tactical Fix)

Having validated the environment, the agent provides the exact, surgical CLI commands to revert the drift without touching the underlying, legitimate IaC.

Step 3.1:

It runs aws iam delete-role-policy to explicitly drop the manual STSAssumeRole inline policy, neutralizing the lateral movement threat.

Step 3.2:

It runs aws s3api put-bucket-policy with a newly generated, hardened JSON payload that explicitly strips the s3:PutObject permission, locking the bucket back down to public-read-only to match the Terraform baseline.

Phase 4: Post-Validate (Verification)

The agent doesn't assume the commands worked; it verifies them.

Steps 4.1 & 4.2:

It reruns the commands from Phase 1 (list-role-policies and get-bucket-policy) and asks you to confirm that the output no longer contains the malicious permissions.

Phase 5: Rollback

Finally, the agent provides a complete Rollback script using aws iam put-role-policy and aws s3api put-bucket-policy, populated with the exact variables needed to restore the drifted state if the fix causes an unexpected outage.

By automating the creation of this peer-reviewable, highly defensive CLI runbook, the AWS DevOps Agent collapses the mitigation lifecycle from hours of stressful, manual engineering to mere minutes of execution.

Environment Cleanup

To avoid incurring any unexpected charges and to cleanly shut down your lab environment, follow these detailed steps to delete all AWS resources and local configurations.

Step 1: Empty the S3 Bucket

Terraform cannot destroy an S3 bucket if it contains objects (like Terraform state files or application data)

.Open the AWS Management Console and navigate to S3.Select the vulnerable-app-data-lab-2026 bucket.Click Empty, type permanently delete to confirm, and click Empty.

Step 2: Destroy Terraform Infrastructure

In yourterminal, navigate to the directory containing your main.tf file and execute the destruction command:


terraform destroy -auto-approve

Note: If you manually created the AWS DevOps Agent Space in the console earlier because it failed to deploy, you will need to manually delete it from the AWS DevOps Agent console.

Step 3: Clean Up Local Environment

Stop any running MCP servers and remove the installed packages to clean up your local workstation.

Stop Processes: Switch to the terminal windows running your linux-mcp-server and terraform-mcp-server and press Ctrl+C to terminate the processes.

Uninstall Linux MCP Server: Since this was installed using uv, remove it by running:


uv tool uninstall linux-mcp-server

Uninstall Terraform MCP Server: Depending on how you installed it, remove it via Homebrew or by deleting the binary:


# If installed via Homebrew
brew uninstall terraform-mcp-server

# If installed via binary download
sudo rm /usr/local/bin/terraform-mcp-server

Remove Lab Files: Finally, delete the Terraform state and lab files from your directory to ensure a clean slate:



rm -rf.terraform terraform.tfstate terraform.tfstate.backup tfplan lambda_function.zip index.py main.tf

Conclusion and Future Implications

The integration of agentic AI into the operational fabric of cloud infrastructure fundamentally disrupts traditional paradigms of software engineering, systems administration, and cybersecurity. From a project management and strategic cybersecurity perspective, the empirical evidence derived from this implementation is profound. The deployment of the AWS DevOps Agent yields an MTTR reduction of up to 75%, increases root cause accuracy to 94%, and accelerates overall incident resolution speeds by a factor of 3 to 5.

However, the most significant implication is the radical reduction of operational toil. By offloading the "undifferentiated heavy lifting" of log correlation, anomaly detection, configuration drift remediation, and HCL syntax generation to an autonomous system, the cognitive load on human engineers is drastically alleviated. This cognitive liberation allows cybersecurity professionals and Project Managers to pivot their focus from reactive firefighting and manual patching to strategic architectural planning, advanced threat modeling, and proactive innovation.

Furthermore, the implementation of the Model Context Protocol democratizes access to these agentic capabilities. By standardizing the interface between the AI reasoning engine and the underlying data layer, organizations are no longer constrained by monolithic, vendor-specific integrations. They can deploy lightweight, highly specialized MCP servers tailored to their proprietary internal systems whether that is a localized Fedora environment, a custom CI/CD pipeline, or a legacy on-premises database ensuring that the frontier agent has comprehensive, unobstructed visibility across the entire technological estate.

Ultimately, the adoption of agentic operations is not merely an enhancement of existing DevOps workflows; it is a fundamental architectural evolution. As demonstrated by the capacity to automatically detect security misconfigurations, synthesize root causes, and generate deployable, peer-reviewable infrastructure code, the AWS DevOps Agent establishes the foundation for true self-healing systems a requisite advancement for managing the scale, security, and sustainability demands of the next decade of cloud computing.

aws #devops #terraform #cybersecurity