Real-Time Runtime Security and Automated Incident Response for EKS

Project Overview

In Part 1 of this series, I focused entirely on the Shift-Left paradigm. I spent time hardening Dockerfiles, enforcing IAM Roles for Service Accounts (IRSA) via Terraform, and setting up static security gates in our CI/CD pipeline using tfsec and Trivy. That setup ensured that known vulnerabilities or infrastructure misconfigurations were caught before they could ever reach our Amazon Elastic Kubernetes Service (EKS) cluster.

But as a Solutions Architect or DevSecOps Engineer, you must always adopt an assume-breach mentality.

What happens if an unknown, zero-day vulnerability (like Log4j) bypasses your static scanning gates? What happens if an insider threat or a compromised administrative credential executes malicious code directly inside a live, production container? Static security scanners cannot help you at 3:00 AM when a container starts scanning your internal VPC network or spinning up a cryptominer. For that, you need continuous, real-time visibility into your running workloads and an automated mechanism to neutralize threats instantly.

In this article, we will architect and deploy an end-to-end Runtime Security Defense Engine and an out-of-band Automated Incident Response Solution for EKS.

By combining the Cloud Native Computing Foundation (CNCF) open-source detection engine Falco with lightweight, serverless AWS infrastructure (Amazon CloudWatch Logs, Amazon EventBridge, AWS Lambda, and Amazon SNS), we will implement a highly responsive, cost-effective detection and remediation lifecycle.

---

Architecture Blueprint & Core Design Decisions

Our architecture is entirely event-driven and decoupled from the cluster's data plane:

1. Kernel-Level Detection: Falco runs on our EKS worker nodes as a DaemonSet, leveraging modern eBPF (Extended Berkeley Packet Filter) technology to intercept and inspect system calls (syscalls) directly within the Linux kernel.
2. Out-of-Cluster Telemetry Stream: When an anomaly matches a custom security rule, Falco formats the alert into a JSON payload and writes it to stdout. A logging agent ships this stream immediately to Amazon CloudWatch Logs.
3. Serverless Event Routing: Amazon EventBridge monitors the CloudWatch log stream using an explicit event pattern filter, capturing only high-severity threats (Critical and Emergency).
4. Out-of-Band Remediation: EventBridge invokes an isolated AWS Lambda function. This function uses the AWS API plane to securely authenticate against the EKS control plane and forcefully terminate the compromised pod.

Architectural Rationale & Trade-offs

• Falco vs. Amazon GuardDuty for EKS: GuardDuty for EKS is a managed service that analyzes Kubernetes audit logs and runtime execution via an agent. While excellent, it incurs additional managed service costs. Falco operates at the kernel level via open-source eBPF rules, adding zero line-item costs to your AWS bill during your Proof of Concept (PoC) phase while remaining completely customizable.
• Decoupled vs. In-Cluster Remediation: Instead of running heavy, highly-privileged management pods inside the cluster to kill compromised workloads, we stream alerts safely out of the cluster. This out-of-band automation model prevents an attacker who has achieved a container escape from blinding, manipulating, or deleting our security response logic.

---

Prerequisites & Local Machine Setup

Before we begin building, we must ensure your local workstation has the necessary tooling installed. This guide assumes you are using Fedora Linux, but the steps adapt readily to any Linux distribution.

1. Install the AWS CLI & Authenticate

Open your terminal and install the AWS command-line interface to interact with your cloud account:

sudo dnf install -y awscli2

Verify the installation:

aws --version

Configure the CLI with your sandbox account access keys. Ensure this identity has administrative privileges:

aws configure

2. Install Terraform

Add the official HashiCorp repository and install the Terraform binary:

sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
sudo dnf install -y terraform

Verify the installation:

terraform version

3. Install Helm

Helm acts as the package manager for Kubernetes applications. Install it via dnf:

sudo dnf install -y helm

Verify the installation:

helm version

4. Install kubectl

kubectl is the primary command-line tool used to send instructions to a Kubernetes cluster:

sudo dnf install -y kubernetes-client

Verify the installation:

kubectl version --client

⚠️ Cost Warning: Running a managed Amazon EKS cluster incurs a standard control plane fee of $0.10 per hour (~$73/month), along with the standard EC2 infrastructure charges for your worker nodes. Ensure you execute the complete cleanup steps at the very end of this lab to tear down all resources and avoid unexpected charges.

---

Creating the Project Workspace Directory Structure

We will build this multi-layered project completely from scratch. Run the following chained terminal command to create your root folder workspace, generate the entire sub-directory hierarchy, and create blank placeholder files:

mkdir -p eks-runtime-defense/{terraform,falco,lambda,k8s} && \
cd eks-runtime-defense && \
touch terraform/main.tf \
      terraform/variables.tf \
      terraform/outputs.tf \
      terraform/providers.tf \
      terraform/falco.tf \
      terraform/falco-response-infra.tf \
      falco/custom-rules.yaml \
      falco/falco-values.yaml \
      lambda/response_engine.py \
      lambda/falco_bridge.py \
      k8s/vulnerable-pod.yaml \
      .gitignore && \
echo -e ".terraform/\n*.tfstate*\n.env\n__pycache__/\n*.pyc" > .gitignore

Expected Directory Layout

Your VS Code workspace explorer will display this folder and file hierarchy:

eks-runtime-defense/
├── terraform/
│   ├── main.tf                  # Core infrastructure (VPC, EKS, IAM)
│   ├── variables.tf             # Input variables
│   ├── outputs.tf               # Output values
│   ├── providers.tf             # AWS, Kubernetes, Helm providers
│   ├── falco.tf                 # Falco Helm deployment
│   └── falco-response-infra.tf  # CloudWatch -> Lambda subscription
├── falco/
│   ├── custom-rules.yaml        # Custom Falco security rules
│   └── falco-values.yaml        # Falco Helm values (post-fix)
├── lambda/
│   ├── response_engine.py       # Pod deletion Lambda (heavily improved)
│   └── falco_bridge.py          # CloudWatch -> EventBridge bridge
└── k8s/
    └── vulnerable-pod.yaml      # Test target pod

---

Step 1: Provisioning the EKS Cluster and Networking via Terraform

We will use infrastructure-as-code to deploy a fresh, dedicated Virtual Private Cloud (VPC) and a hardened, managed EKS cluster.

Open your terraform/variables.tf file and paste the variable definitions:

variable "aws_region" {
  type        = string
  description = "The target AWS Region for all infrastructure resources"
  default     = "us-east-1"
}

variable "secops_email" {
  type        = string
  description = "The real email address to receive high-fidelity security alerts"
  default     = "[e-mail]" # CHANGE THIS to your own email address
}

· type = string: Enforces type safety—Terraform will reject non-string values.
· description: Self-documenting variables are critical for team collaboration and code review.
· default = "us-east-1": Sets a sensible default while allowing override via terraform.tfvars or CLI flags.
· default = "[e-mail]": Must be changed before apply. Terraform will prompt if no value is provided and no default exists.

Next, open your terraform/main.tf file and paste the comprehensive layout code:

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.31.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.12.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

# Fetch active availability zones dynamically within the region
data "aws_availability_zones" "available" {}

# Provision a clean, isolated VPC network
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0.0"

  name                 = "runtime-defense-vpc"
  cidr                 = "10.0.0.0/16"
  azs                  = slice(data.aws_availability_zones.available.names, 0, 2)
  public_subnets       = ["10.0.1.0/24", "10.0.2.0/24"]
  private_subnets      = ["10.0.11.0/24", "10.0.12.0/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
}

# Provision a standard, security-hardened managed EKS Cluster
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "19.21.0"
  cluster_name    = "runtime-security-cluster"
  cluster_version = "1.30"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    security_nodes = {
      min_size       = 1
      max_size       = 2
      desired_size   = 1
      instance_types = ["t3.medium"] # Cost-conscious compute profile for testing labs
    }
  }
}

# Create a centralized CloudWatch Log Group for container telemetry
resource "aws_cloudwatch_log_group" "falco_alerts" {
  name              = "/aws/eks/runtime-security-cluster/falco-alerts"
  retention_in_days = 7
}

# Create an Amazon SNS Topic for real-time alerting notifications
resource "aws_sns_topic" "secops_alerts" {
  name = "secops-runtime-alerts-topic"
}

resource "aws_sns_topic_subscription" "email_sub" {
  topic_arn = aws_sns_topic.secops_alerts.arn
  protocol  = "email"
  endpoint  = var.secops_email
}

· required_version = ">= 1.5.0": Ensures compatibility with modern Terraform features like moved blocks and improved error messages.
· required_providers: Explicitly pins provider versions to prevent breaking changes from automatic updates.
· source = "hashicorp/aws": Uses the official HashiCorp AWS provider (not a community fork).
· version = "~> 5.31.0": Allows patch updates (5.0.x) but not minor (5.1.0), balancing stability and bug fixes.
· data "aws_availability_zones": Queries AWS dynamically for available AZs in the region—no hardcoding needed.
· source = "terraform-aws-modules/vpc/aws": Uses the battle-tested community VPC module (millions of downloads, actively maintained).
· version = "~> 5.0.0": Pins to 5.0.x for stability.
· cidr = "10.0.0.0/16": Provides 65,536 IP addresses—sufficient for most workloads.
· slice(..., 0, 2): Dynamically selects first 2 AZs. Using slice instead of hardcoding ensures the code works in regions with only 2 AZs (like us-west-1).
· enable_nat_gateway = true: Allows private subnet outbound internet access for EKS nodes (required for pulling container images).
· single_nat_gateway = true: Cost optimization for lab environments. Production should use one per AZ for high availability.
· enable_dns_hostnames = true: Required for EKS private endpoint resolution.
· cluster_version = "1.30": Uses a recent, supported Kubernetes version. Always check EKS version support lifecycle.
· subnet_ids = module.vpc.private_subnets: Places EKS nodes in private subnets—security best practice.
· cluster_endpoint_public_access = true: Required for this lab (access from local workstation). Production should disable this and use AWS VPN/SSM/private endpoints.
· cluster_enabled_log_types: Enables control plane audit logging—essential for forensic analysis and compliance (SOC 2, PCI-DSS).
· min_size = 1, max_size = 2: Cost-conscious auto-scaling. desired_size = 1 starts minimal.
· instance_types = ["t3.medium"]: Burstable instance with 2 vCPU, 4GB RAM—sufficient for Falco eBPF + test workloads.
· retention_in_days = 7: Cost optimization for lab. Production typically uses 30-90 days or streams to S3/Glacier.
· protocol = "email": Simplest notification channel. Production should add PagerDuty, Slack, or OpsGenie integrations.

Now, populate your terraform/outputs.tf file to output useful strings for later steps:

output "cluster_name" {
  value       = module.eks.cluster_name
  description = "The assigned name of the EKS cluster resource"
}

output "cluster_arn" {
  value       = module.eks.cluster_arn
  description = "The Amazon Resource Name of the EKS cluster control plane"
}

output "cluster_endpoint" {
  value       = module.eks.cluster_endpoint
  description = "The EKS Kubernetes API endpoint"
}

output "sns_topic_arn" {
  value       = aws_sns_topic.secops_alerts.arn
  description = "The Amazon Resource Name of the SecOps communication topic"
}

output "falco_log_group" {
  value       = aws_cloudwatch_log_group.falco_alerts.name
  description = "CloudWatch Log Group receiving Falco alerts"
}

· Outputs expose computed values for downstream use. They appear in terraform output CLI and can be consumed by other Terraform workspaces via remote state.
· description: Critical for documentation and terraform output self-documentation.

Create the provider configuration for Kubernetes and Helm. Without this, Terraform cannot authenticate to EKS for deploying Falco via Helm. This prevents Kubernetes cluster unreachable errors.

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

provider "helm" {
  kubernetes = {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

· data "aws_eks_cluster": Queries the EKS API for cluster metadata (endpoint, CA cert).
· data "aws_eks_cluster_auth": Generates a short-lived authentication token (valid ~15 minutes) for the current AWS identity.
· host = data.aws_eks_cluster.cluster.endpoint: The Kubernetes API server URL.
· cluster_ca_certificate = base64decode(...): Decodes the cluster's CA certificate for TLS verification. Without this, Terraform cannot verify the API server's identity.
· token = data.aws_eks_cluster_auth.cluster.token: The bearer token for authentication. Using data source ensures it's refreshed on every apply.
· provider "helm" { kubernetes = { ... } }: Helm provider needs identical auth configuration. Note the = sign—older Helm provider versions use kubernetes = {} (map syntax), newer versions use kubernetes {} (block syntax). Match your provider version.

Why use data sources for auth tokens instead of hardcoding kubeconfig files?
Data sources are ephemeral, rotate automatically, and don't leak credentials to state files.

Execution & Cluster Verification

Run the initialization and deployment steps in your terminal:

cd terraform
terraform init
terraform apply -auto-approve

Note: Provisioning an enterprise-grade managed Kubernetes cluster from scratch can take between 10 to 15 minutes to configure networking, control planes, and node pools.

Once the apply command finishes successfully, run the following command to download the cluster credentials and bind your local terminal context to the new EKS API manager:

aws eks update-kubeconfig --region us-east-1 --name runtime-security-cluster

Verify you can communicate with your new cloud resource by fetching the node state:

kubectl get nodes

What you should see: Your terminal will output a row displaying your single t3.medium worker instance with a status of Ready.

---

Step 2: Authoring the Incident Response Lambda Function

With our base infrastructure standing, we need an out-of-band remediation serverless component. If Falco flags a critical runtime thread breach, our Lambda function will handle the threat directly via the control plane API.

Appending Lambda IaC Blocks

Open your terraform/main.tf file and append the following block to the bottom of the file to provision the Lambda resource and its rigid IAM security boundary:

# Create an execution identity for the remediation lambda function
resource "aws_iam_role" "lambda_remediation_role" {
  name = "secops-remediation-lambda-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

# Define a restrictive policy to inspect the cluster and ship execution logs
resource "aws_iam_policy" "lambda_eks_policy" {
  name        = "LambdaEKSPodMitigationPolicy"
  description = "Allows the mitigation lambda function to read and terminate workloads in EKS"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["eks:DescribeCluster"]
        Resource = [module.eks.cluster_arn]
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = ["arn:aws:logs:*:*:*"]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_attach" {
  role       = aws_iam_role.lambda_remediation_role.name
  policy_arn = aws_iam_policy.lambda_eks_policy.arn
}

# Generate a minimal ZIP archive of the deployment script source code for compilation
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/response_engine.py"
  output_path = "${path.module}/lambda_function.zip"
}

# Declare the Lambda resource instance
resource "aws_lambda_function" "response_engine" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "secops-pod-remediation-engine"
  role             = aws_iam_role.lambda_remediation_role.arn
  handler          = "response_engine.lambda_handler"
  runtime          = "python3.11"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  timeout          = 30

  environment {
    variables = {
      CLUSTER_NAME = module.eks.cluster_name
    }
  }
}

Creating the Response Automation Script

The original code I deployed expected output_fields at the top level of CloudWatch log events. In production, Fluent Bit wraps Falco JSON in a {'_p': 'F', 'data': {...}, 'kubernetes': {...}, 'log': '...'} envelope. The original Lambda silently failed with 'No pod info found, skipping.' The fixed code handles** three nested formats** and includes comprehensive logging for debugging.

Move back to your project directories (cd ..) and open lambda/response_engine.py. Paste this fully working remediation code:

import base64
import json
import gzip
import boto3
import os
import urllib3
import ssl
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest

def lambda_handler(event, context):
    # Decode CloudWatch Logs payload (base64 + gzip)
    cw_data = base64.b64decode(event['awslogs']['data'])
    cw_data = gzip.decompress(cw_data)
    log_event = json.loads(cw_data)

    print(f"Received {len(log_event.get('logEvents', []))} log events")

    for log in log_event.get('logEvents', []):
        message = log.get('message', '')
        print(f"Processing message (first 300 chars): {message[:300]}")

        pod_name, namespace = extract_pod_info(message)
        if pod_name:
            print(f"Found pod: {pod_name} in namespace: {namespace} — DELETING")
            delete_pod(pod_name, namespace)
        else:
            print("No pod info found, skipping")

    return {'statusCode': 200}



def extract_pod_info(message):
    pod_name = None
    namespace = 'default'

    # FORMAT 1: Fluent Bit wrapper with 'data' -> 'output_fields'
    try:
        wrapper = json.loads(message)

        if 'data' in wrapper and isinstance(wrapper['data'], dict):
            fields = wrapper['data'].get('output_fields', {})
            pod_name = fields.get('k8s.pod.name')
            namespace = fields.get('k8s.ns.name', 'default')
            if pod_name:
                return pod_name, namespace

        # FORMAT 2: Top-level output_fields (raw Falco JSON)
        fields = wrapper.get('output_fields', {})
        pod_name = fields.get('k8s.pod.name')
        namespace = fields.get('k8s.ns.name', 'default')
        if pod_name:
            return pod_name, namespace

        # FORMAT 3: Escaped JSON in 'log' field
        raw_log = wrapper.get('log', '')
        if raw_log:
            try:
                falco_data = json.loads(raw_log)
                fields = falco_data.get('output_fields', {})
                pod_name = fields.get('k8s.pod.name')
                namespace = fields.get('k8s.ns.name', 'default')
                if pod_name:
                    return pod_name, namespace
            except json.JSONDecodeError:
                pass

    except json.JSONDecodeError:
        pass

    # FORMAT 4: Plain text fallback
    if 'k8s.pod.name=' in message:
        pod_name = message.split('k8s.pod.name=')[1].split()[0].strip()


def delete_pod(pod_name, namespace):
    cluster_name = os.environ.get('CLUSTER_NAME', 'runtime-security-cluster')
    region = os.environ.get('AWS_REGION', 'us-east-1')

    print(f"Authenticating to EKS cluster: {cluster_name}")

    # Get EKS cluster info (endpoint + CA certificate)
    eks = boto3.client('eks', region_name=region)
    cluster = eks.describe_cluster(name=cluster_name)['cluster']

    endpoint = cluster['endpoint']
    ca_cert = base64.b64decode(cluster['certificateAuthority']['data'])

    with open('/tmp/ca.crt', 'wb') as f:
        f.write(ca_cert)

    # Generate EKS IAM auth token
    token = generate_eks_token(cluster_name, region)
    print(f"Generated EKS token (first 50 chars): {token[:50]}...")

    # Configure HTTPS client with CA verification
    http = urllib3.PoolManager(
        cert_reqs=ssl.CERT_REQUIRED,
        ca_certs='/tmp/ca.crt'
    )

    # Construct Kubernetes API URL for pod deletion
    url = f"{endpoint}/api/v1/namespaces/{namespace}/pods/{pod_name}"
    headers = {
        'Authorization': f'Bearer {token}',
        'Content-Type': 'application/json'
    }

    print(f"DELETE {url}")

    # Execute deletion with immediate termination
    resp = http.request(
        'DELETE',
        url,
        headers=headers,
        body=json.dumps({
            "apiVersion": "v1",
            "kind": "DeleteOptions",
            "propagationPolicy": "Foreground"
        })
    )

    print(f"K8s API response status: {resp.status}")
    print(f"Body: {resp.data.decode()[:500]}")


def generate_eks_token(cluster_name, region):
    session = boto3.session.Session()
    creds = session.get_credentials().get_frozen_credentials()

    # Build an unsigned STS GetCallerIdentity request
    req = AWSRequest(
        method='GET',
        url=f'https://sts.{region}.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15',
        headers={'x-k8s-aws-id': cluster_name}
    )

    # Sign the request with SigV4 using the Lambda's IAM credentials
    SigV4Auth(creds, 'sts', region).add_auth(req)

    url = req.url
    headers = [{'key': k, 'value': v} for k, v in req.headers.items()]

    # Encode the signed URL and headers as base64url
    payload = json.dumps({'url': url, 'headers': headers})
    token = base64.urlsafe_b64encode(payload.encode()).decode().rstrip('=')
    return f'k8s-aws-v1.{token}'

· base64.b64decode(event['awslogs']['data']): CloudWatch Logs subscriptions encode payloads in base64.
· gzip.decompress(...): CloudWatch further compresses with gzip for efficiency.
· json.loads(...): Converts the decompressed JSON string to a Python dictionary.
· log_event.get('logEvents', []): Safely retrieves the list of log events. Using .get() with a default prevents KeyError if the key is missing.
· message[:300]: Truncates long messages for CloudWatch Logs line limits while preserving enough context for debugging.

· namespace = 'default': Default fallback. Kubernetes pods without explicit namespace assignment land here.
· isinstance(wrapper['data'], dict): Type guard prevents AttributeError if data exists but is a string or null.
· wrapper['data'].get('output_fields', {}): Nested safe access. Returns empty dict if output_fields is missing.
· fields.get('k8s.pod.name'): Returns None (falsy) if the field is missing—triggers the next fallback.
· json.loads(raw_log): Handles the case where Fluent Bit puts the actual Falco JSON inside a string-valued log key.
· json.JSONDecodeError: Gracefully handles malformed JSON instead of crashing the entire Lambda.
· Text parsing fallback: Regex-free string splitting for simple key=value formats. Fast but fragile—used only when JSON parsing fails.

· os.environ.get('CLUSTER_NAME', 'runtime-security-cluster'): Reads from Lambda environment variables. Uses default for local testing.
· boto3.client('eks', region_name=region): Creates a regional EKS client. Always specify region_name—Lambda may run in a different region than the cluster.
· base64.b64decode(...): EKS returns the CA certificate as a base64-encoded string.
· ssl.CERT_REQUIRED: Enforces TLS certificate verification. Never use CERT_NONEin production—it opens you to man-in-the-middle attacks.
· /api/v1/namespaces/{namespace}/pods/{pod_name}: The Kubernetes Core API endpoint for pod lifecycle management.
· propagationPolicy: 'Foreground': Ensures dependent resources (like ReplicaSets) are cleaned up before the pod is considered deleted.
· resp.data.decode()[:500]: Decodes the binary response and truncates for logging.
· get_frozen_credentials(): Locks credentials to prevent refresh mid-request. Critical for SigV4 signing where the signature includes a timestamp.
· x-k8s-aws-id: EKS-specific header that binds the token to a specific cluster. Prevents token replay attacks against other clusters.
· SigV4Auth(creds, 'sts', region): Uses AWS Signature Version 4—the same algorithm AWS SDKs use for all API calls.
· base64.urlsafe_b64encode: Uses URL-safe base64 (replaces + with - and / with _). Required because the token is passed in HTTP headers.
· .rstrip('='): Removes padding. Kubernetes API server accepts unpadded base64url.

Defensive programming (multiple fallbacks, type checking, exception handling) is essential in production because log formats can change when Fluent Bit, Falco, or EKS versions update.

We don't use aws-iam-authenticator binary in Lambda. Lambda has a 250MB deployment package limit and cold start constraints. Generating the token in pure Python eliminates binary dependencies and reduces package size.

Click the lambda/falco bridge.py folder and write the following lines of code:

import json
import base64
import gzip
import boto3
import os

def lambda_handler(event, context):
    region = os.environ.get('AWS_REGION', 'us-east-1')
    eventbridge = boto3.client('events', region_name=region)

    # Decode CloudWatch Logs payload (base64 + gzip)
    cw_data = base64.b64decode(event['awslogs']['data'])
    cw_logs = json.loads(gzip.decompress(cw_data))
    log_events = cw_logs.get('logEvents', [])

    forwarded = 0
    for log_event in log_events:
        message = log_event.get('message', '')
        if not message.startswith('{'):
            continue

        try:
            falco_event = json.loads(message)
        except json.JSONDecodeError:
            continue

        # Filter for high-severity events only
        priority = falco_event.get('priority', '').lower()
        if priority not in ['critical', 'emergency']:
            continue

        # Forward to EventBridge for downstream processing
        eventbridge.put_events(
            Entries=[{
                'Source': 'falco.runtime.security',
                'DetailType': 'Falco Runtime Threat Detected',
                'Detail': json.dumps(falco_event),
                'EventBusName': 'default'
            }]
        )
        forwarded += 1
        print(f"Forwarded {falco_event.get('rule')} alert to EventBridge.")

    return {'status': 'success', 'forwarded_events': forwarded}

· if not message.startswith('{'): Fast pre-filter. Falco JSON output always starts with {. This skips Fluent Bit internal logs, timestamps, and metadata lines.
· priority not in ['critical', 'emergency']: Severity gate. Prevents alert fatigue by filtering out Notice/Warning/Error level events.
· 'Source': 'falco.runtime.security': Custom event source. EventBridge rules filter on this to distinguish Falco alerts from other events.
· 'Detail': json.dumps(falco_event): EventBridge requires Detail to be a JSON string, not a dict. This is a common source of MalformedDetail errors.
· 'EventBusName': 'default': Uses the account's default event bus. Production might use custom event buses for multi-tenant isolation.

Deploy the new lambda layer configurations immediately using Terraform:

cd terraform
terraform apply -auto-approve

Lets grant our cluster admin privileges first before installing Fluent Bit:

# Step 1: Figure out who you are in AWS
CURRENT_ARN=$(aws sts get-caller-identity --query Arn --output text)
echo "Your current ARN: $CURRENT_ARN"

# Step 2: If you're using an assumed IAM Role, strip the session suffix
# EKS Access Entries need the ROLE ARN, not the session ARN
# Example: arn:aws:sts::123:assumed-role/MyRole/session -> arn:aws:iam::123:role/MyRole
if echo "$CURRENT_ARN" | grep -q "assumed-role"; then
  PRINCIPAL_ARN=$(echo "$CURRENT_ARN" | sed 's|arn:aws:sts:|arn:aws:iam:|' | sed 's|:assumed-role/|:role/|' | sed 's|/.*||')
  echo "Converted to Role ARN: $PRINCIPAL_ARN"
else
  PRINCIPAL_ARN="$CURRENT_ARN"
fi

# Step 3: Create the access entry
aws eks create-access-entry \
  --cluster-name runtime-security-cluster \
  --principal-arn "$PRINCIPAL_ARN" \
  --type STANDARD

# Step 4: Grant cluster-admin privileges
aws eks associate-access-policy \
  --cluster-name runtime-security-cluster \
  --principal-arn "$PRINCIPAL_ARN" \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

# Step 5: Wait 30 seconds for IAM propagation, then test
sleep 30
kubectl get nodes

Now that we have granted admin privileges to our cluster lets install Fluent Bit.

# After terraform apply finishes:
aws eks update-kubeconfig --region us-east-1 --name runtime-security-cluster

helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-for-fluent-bit eks/aws-for-fluent-bit \
  --namespace kube-system \
  --set cloudWatch.enabled=true \
  --set cloudWatch.logGroupName=/aws/eks/runtime-security-cluster/falco-alerts \
  --set cloudWatch.region=us-east-1 \
  --set cloudWatch.logStreamPrefix=falco- \
  --set firehose.enabled=false \
  --set kinesis.enabled=false

---

Step 3: Terraform Lambda Infrastructure

Lets break down the terraform file in our main.tf in respect to Lambda.

IAM Role and Policies

resource "aws_iam_role" "lambda_remediation_role" {
  name = "secops-remediation-lambda-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

resource "aws_iam_policy" "lambda_policy" {
  name        = "LambdaEKSRemediationPolicy"
  description = "Allows Lambda to read EKS, send logs, publish SNS, and emit EventBridge events"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["eks:DescribeCluster"]
        Resource = [module.eks.cluster_arn]
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = ["arn:aws:logs:*:*:*"]
      },
      {
        Effect   = "Allow"
        Action   = ["sns:Publish"]
        Resource = [aws_sns_topic.secops_alerts.arn]
      },
      {
        Effect   = "Allow"
        Action   = ["events:PutEvents"]
        Resource = ["*"]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_attach" {
  role       = aws_iam_role.lambda_remediation_role.name
  policy_arn = aws_iam_policy.lambda_policy.arn
}

resource "aws_iam_role_policy_attachment" "lambda_basic" {
  role       = aws_iam_role.lambda_remediation_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

· Principal = { Service = 'lambda.amazonaws.com' }: Only Lambda service can assume this role. Prevents EC2, ECS, or other services from using it.
· eks:DescribeCluster: Minimum permission needed to get the cluster endpoint and CA certificate. The Lambda doesn't need broader EKS permissions.
· Resource = [module.eks.cluster_arn]: Scoped to specific cluster. Follows least-privilege principle.
· AWSLambdaBasicExecutionRole: AWS-managed policy providing CloudWatch Logs access. Always attach this or your Lambda logs won't appear.

EKS Access Entry (Modern Auth — Replaces aws-auth ConfigMap)

Production Improvement: The original main.tf I wrote used the legacy aws-auth ConfigMap for IAM-to-Kubernetes RBAC mapping. EKS now supports Access Entries a native AWS API for managing cluster access without modifying ConfigMaps. This is more secure, auditable, and doesn't risk locking yourself out with malformed YAML.

resource "aws_eks_access_entry" "lambda" {
  cluster_name  = module.eks.cluster_name
  principal_arn = aws_iam_role.lambda_remediation_role.arn
  type          = "STANDARD"
}

resource "aws_eks_access_policy_association" "lambda" {
  cluster_name  = module.eks.cluster_name
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy"
  principal_arn = aws_iam_role.lambda_remediation_role.arn

  access_scope {
    type       = "namespace"
    namespaces = ["default"]
  }

  depends_on = [aws_eks_access_entry.lambda]
}

· type = 'STANDARD': Standard access entry (not EC2_LINUX or FARGATE_LINUX which are for managed nodes).
· AmazonEKSAdminPolicy: AWS-managed policy granting full admin access within the scoped namespace. For production, use AmazonEKSViewPolicy or custom policies.
· access_scope { type = 'namespace', namespaces = ['default'] }: Critical security boundary. The Lambda can only manage resources in the default namespace. If compromised, it cannot affect kube-system, security, or other namespaces.
· depends_on: Ensures the access entry exists before associating policies. Prevents race conditions.

Comparison between Access Entries vs. aws-auth. Access Entries are API-driven (Terraform-friendly), support fine-grained policies, and don't require cluster access to modify. aws-auth requires kubectl access and is prone to syntax errors that can lock out all users.

Lambda Deployment Packages

data "archive_file" "response_lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/response_engine.py"
  output_path = "${path.module}/response_engine.zip"
}

data "archive_file" "bridge_lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/../lambda/falco_bridge.py"
  output_path = "${path.module}/falco_bridge.zip"
}

· data "archive_file": Terraform data source that creates a zip file during the plan phase.
· ${path.module}: Resolves to the directory containing the current Terraform file. Ensures paths work regardless of where Terraform is executed from.
· output_path: The zip is created in the Terraform working directory (.terraform/ is gitignored).

Lambda Function Definitions

resource "aws_lambda_function" "response_engine" {
  filename         = data.archive_file.response_lambda_zip.output_path
  function_name    = "secops-pod-remediation-engine"
  role             = aws_iam_role.lambda_remediation_role.arn
  handler          = "response_engine.lambda_handler"
  runtime          = "python3.11"
  source_code_hash = data.archive_file.response_lambda_zip.output_base64sha256
  timeout          = 30

  environment {
    variables = {
      CLUSTER_NAME  = module.eks.cluster_name
      SNS_TOPIC_ARN = aws_sns_topic.secops_alerts.arn
    }
  }

  depends_on = [aws_iam_role_policy_attachment.lambda_basic]
}

resource "aws_lambda_function" "falco_bridge" {
  filename         = data.archive_file.bridge_lambda_zip.output_path
  function_name    = "falco-cloudwatch-bridge"
  role             = aws_iam_role.lambda_remediation_role.arn
  handler          = "falco_bridge.lambda_handler"
  runtime          = "python3.11"
  source_code_hash = data.archive_file.bridge_lambda_zip.output_base64sha256
  timeout          = 10

  depends_on = [aws_iam_role_policy_attachment.lambda_basic]
}

· source_code_hash: Forces Terraform to redeploy the Lambda when the Python code changes. Without this, Terraform sees the same filename and skips updates.
· timeout = 30: Response Lambda needs time for EKS API calls (describe cluster + delete pod). Bridge Lambda is simpler (10s).
· environment { variables = { ... } }: Passes configuration without hardcoding. Allows the same code to run against different clusters by changing environment variables.

EventBridge Rule and Targets

resource "aws_cloudwatch_event_rule" "falco_critical_rule" {
  name        = "falco-critical-alerts-filter"
  description = "Intercepts high-severity Falco runtime security events"

  event_pattern = jsonencode({
    "source" : ["falco.runtime.security"],
    "detail" : {
      "priority" : ["Critical", "Emergency", "CRITICAL", "EMERGENCY"]
    }
  })
}

resource "aws_cloudwatch_event_target" "lambda_target" {
  rule      = aws_cloudwatch_event_rule.falco_critical_rule.name
  target_id = "TriggerRemediationLambda"
  arn       = aws_lambda_function.response_engine.arn
}

resource "aws_cloudwatch_event_target" "sns_target" {
  rule      = aws_cloudwatch_event_rule.falco_critical_rule.name
  target_id = "SendSecOpsNotification"
  arn       = aws_sns_topic.secops_alerts.arn
}

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.response_engine.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.falco_critical_rule.arn
}

· event_pattern: JSON Path matching. Only events where source equals falco.runtime.security AND detail.priority matches the list trigger the rule.
· target_id: Human-readable identifier for the target. Appears in CloudTrail and debugging.
· aws_lambda_permission: Grants EventBridge service principal permission to invoke the Lambda. Without this, EventBridge cannot trigger the Lambda—the invocation will fail silently.
· statement_id: Unique identifier for the permission policy statement. Must be unique within the Lambda's policy.

CloudWatch Logs -> Bridge Lambda Integration

resource "aws_lambda_permission" "allow_cloudwatch" {
  statement_id  = "AllowExecutionFromCloudWatchLogs"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.falco_bridge.function_name
  principal     = "logs.amazonaws.com"
  source_arn    = "${aws_cloudwatch_log_group.falco_alerts.arn}:*"
}

resource "aws_cloudwatch_log_subscription_filter" "falco_filter" {
  name            = "falco-critical-filter"
  log_group_name  = aws_cloudwatch_log_group.falco_alerts.name
  filter_pattern  = "?CRITICAL ?EMERGENCY"
  destination_arn = aws_lambda_function.falco_bridge.arn

  depends_on = [aws_lambda_permission.allow_cloudwatch]
}

· filter_pattern = "?CRITICAL ?EMERGENCY": CloudWatch Logs filter syntax. ? means OR. Only log lines containing 'CRITICAL' or 'EMERGENCY' are forwarded. This reduces Lambda invocations and costs.
· depends_on: Ensures the Lambda permission exists before creating the subscription. Without this, CloudWatch Logs cannot verify it has permission to invoke the Lambda, and creation fails.
· source_arn = "${...}:*": The :* suffix allows the permission to apply to all log streams within the log group.

---

Step 4: Deploying Custom Falco Security Rules via Helm

Now let's configure the runtime detection layer. We will create custom rules that monitor system calls inside our cluster, checking specifically for interactive terminal attachments (kubectl exec) and malicious network connections.

Navigate to falco/custom-rules.yaml and paste the rule configurations:

# Enable the modern eBPF driver instead of compiling legacy kernel modules
driver:
  kind: ebpf

# Configure structural output format to print machine-readable JSON strings to stdout
falco:
  json_output: true
  stdout_output: true

# Custom rule definitions designed to stop specific adversarial attacks
customRules:
  rules_user.yaml: |
    - rule: Terminal Shell Spawned Within Container
      desc: Detects an interactive terminal shell attachment or spawn attempt inside a live container.
      condition: container.id != host and proc.name = bash or proc.name = sh and spawn_process
      output: "CRITICAL: Interactive Shell Spawned (user=%user.name pod=%k8s.pod.name ns=%k8s.ns.name image=%container.image.repository proc=%proc.name cmdline=%proc.cmdline)"
      priority: CRITICAL
      tags: [container, shell, mitre_execution]

    - rule: Cryptominer Network Pattern Detected
      desc: Detects outbound communication attempts targeting common mining pools.
      condition: fd.sport = 4444 or fd.dport = 4444 or fd.sip = "pool.supportxmr.com"
      output: "CRITICAL: Cryptominer Outbound Connection Detected (pod=%k8s.pod.name ns=%k8s.ns.name destination=%fd.sip port=%fd.sport process=%proc.name)"
      priority: EMERGENCY
      tags: [network, mining, mitre_impact]

· driver: kind: modern_ebpf: Uses the modern eBPF probe (Falco 0.35+). More portable than kernel module or legacy eBPF. Works on most kernels without headers.
· json_output: true: Outputs alerts as JSON. Required for structured parsing by Lambda.
· json_include_output_property: true: Includes the human-readable output field in JSON. Useful for SNS emails.
· stdout_output: enabled: true: Writes to stdout where Fluent Bit captures it.
· container.id != host: Ensures the rule only fires for containers, not host processes.
· proc.name in (bash, sh): Matches common interactive shells. Extend with zsh, ash if needed.
· fd.rport in (...): Matches known mining pool ports. These are TCP ports commonly used by XMRig and similar miners.

When deploying Falco 0.44.1 via Helm, we encountered a fatal Runtime error: cannot register plugin /usr/share/falco/plugins/libcontainer.so in inspector: found another plugin with name container. Aborting.

Root Cause: Falco 0.44+ ships the containerplugin *built into the binary *(source plugin), but the Helm chart's falcoctl also installs it as an external .soplugin. Both register with the same name, causing a collision.
Fix: Disable falcoctl artifact installation in Helm values.

# falco/falco-values.yaml
# CRITICAL: Disables falcoctl to prevent duplicate container plugin

driver:
  kind: ebpf

falcoctl:
  artifact:
    install:
      enabled: false
    follow:
      enabled: false

· falcoctl.artifact.install.enabled: false: Prevents falcoctl from downloading and installing plugins at startup. The built-in container plugin is sufficient.
· falcoctl.artifact.follow.enabled: false: Disables automatic plugin updates. In production, you want controlled updates, not automatic ones.

This is a classic example of version drift between components. Falco upgraded to built-in plugins, but the Helm chart lagged behind. Always check component compatibility matrices and test in a non-production environment first

Let us deploy helm in falco.tf

resource "helm_release" "falco" {
  name       = "falco"
  namespace  = "security"
  repository = "https://falcosecurity.github.io/charts"
  chart      = "falco"
  version    = "4.17.0"

  create_namespace = true

  values = [
    file("${path.module}/../falco/falco-values.yaml"),
    file("${path.module}/../falco/custom-rules.yaml")
  ]
}

· version = "4.17.0": Pins the Helm chart version. Chart versions are independent of Falco binary versions.
· create_namespace = true: Creates the security namespace if it doesn't exist. Equivalent to kubectl create namespace security.
· values = [file(...), file(...)]: Loads multiple values files. Later files override earlier ones. falco-values.yaml sets base config; custom-rules.yaml adds custom rules.

Lets create the falco-response-infra.tf because my original architecture routed CloudWatch -> Bridge Lambda -> EventBridge -> Response Lambda it failed. In response, we added a direct CloudWatch -> Response Lambda subscription for faster remediation (bypassing EventBridge latency) and as a fallback if EventBridge fails.

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}

resource "aws_lambda_permission" "allow_cloudwatch_response" {
  statement_id  = "AllowExecutionFromCloudWatchLogs"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.response_engine.function_name
  principal     = "logs.amazonaws.com"
  source_arn    = "arn:aws:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/eks/runtime-security-cluster/falco-alerts:*"
}

resource "aws_cloudwatch_log_subscription_filter" "falco_response" {
  depends_on      = [aws_lambda_permission.allow_cloudwatch_response]
  name            = "falco-alerts-to-response"
  log_group_name  = "/aws/eks/runtime-security-cluster/falco-alerts"
  filter_pattern  = ""
  destination_arn = aws_lambda_function.response_engine.arn
}

· data "aws_region" "current": Queries the current AWS region. Used to construct ARNs without hardcoding.
· data "aws_caller_identity" "current": Queries the current AWS account ID. Essential for cross-account or multi-environment templates.
· filter_pattern = "": Empty pattern matches ALL log events. Used here because the Response Lambda has its own filtering logic (checks for pod names). The Bridge Lambda uses?CRITICAL ?EMERGENCY for selective forwarding.
· source_arn: Must exactly match the log group ARN format. The :* suffix covers all log streams.

Deploying the Falco Engine to EKS

Run these commands from your terminal to connect the open-source Helm repository and launch the deployment inside a dedicated namespace:

# Add the official cloud-native CNCF Falco charts stable registry
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install the Falco chart using our custom security rules and eBPF configuration
helm install falco falcosecurity/falco --namespace security --create-namespace -f ../falco/custom-rules.yaml

Track the rollout status until the security pods are running:

kubectl get pods -n security

---

Step 5: Deploying a Sample Vulnerable Application Pod

To evaluate our setup, we need an asset inside our cluster we can break into. Let's create a simulation workspace environment.

Navigate to your k8s/vulnerable-pod.yaml manifest file and paste this pod definition:

# k8s/vulnerable-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: dynamic-web-app-pod
  namespace: default
  labels:
    app: public-ingress-service
spec:
  containers:
  - name: web-server
    image: ubuntu:22.04 # Bloated base image containing standard tools (curl, package managers)
    command: ["/bin/bash", "-c", "sleep 3600"]
    securityContext:
      privileged: false
      allowPrivilegeEscalation: true
      runAsUser: 0 # Running explicitly as ROOT user (The dangerous anti-pattern from Part 1)

· image: ubuntu:22.04: Bloated base image with many pre-installed tools (curl, apt). Attackers love this—more tools to exploit.
· command: ["/bin/bash", "-c", "sleep 3600"]: Keeps the pod alive for 1 hour. The sleep command holds the container open.
· allowPrivilegeEscalation: true: Allows the container to gain more privileges than its parent process. Critical anti-pattern.
· runAsUser: 0: Runs as root. Combined with allowPrivilegeEscalation, this is a container escape risk.

Launch this vulnerable asset into your cluster:

kubectl apply -f ../k8s/vulnerable-pod.yaml

Verify it is active before proceeding:

kubectl get pods

---

Simulated Attack Scenarios (Red Team Phase)

Now, let's step into the role of an adversary who has discovered a vulnerability on your public app. We will trigger the detection logs manually.

Scenario A: Spawning an Interactive Shell

Run this command on your Fedora terminal to force your way inside the live running container via a bash terminal injection simulation:

kubectl exec -it dynamic-web-app-pod -- /bin/bash

What you should see:
Your terminal prompt will change to root@dynamic-web-app-pod:/#. You are now executing commands natively inside the live container instance running on AWS.

Scenario B: Cryptomining Network Reconnaissance

While inside your active root container shell, run these commands to download network analysis tools and simulate an outbound query targeting an offensive mining pool network configuration:

apt-get update && apt-get install -y curl
curl http://pool.supportxmr.com:4444

🎯 Why port 4444?
We configured Falco to detect any outbound connection to port 4444 (and other mining ports). The curl command will attempt a TCP connection to port 4444, triggering Falco's connect syscall probe. The connection does not need to succeed-Falco sees the attempt.

Once executed, step out of the shell connection by typing:

exit

---

Step 6: Verification and Event Remediation Integration

Let's trace how the end-to-end security system catches and routes this breach.

1. Read Falco's Local Cluster Streams

Run this command to inspect the raw stdout JSON engine outputs generated directly inside the cluster:

kubectl logs -n security -l app.kubernetes.io/name=falco | grep CRITICAL

Expected Telemetry Record Output:

{"hostname":"falc-node-x7r","output":"CRITICAL: Interactive Shell Spawned (user=root pod=dynamic-web-app-pod ns=default image=ubuntu proc=bash cmdline=bash)","priority":"Critical","rule":"Terminal Shell Spawned Within Container","time":"2026-06-11T18:12:45.123456Z","output_fields":{"container.id":"a8b7c6d5e4f3","k8s.ns.name":"default","k8s.pod.name":"dynamic-web-app-pod","proc.name":"bash"}}

2. Verify the CloudWatch Log Stream

Check that logs arrived in CloudWatch:

aws logs tail /aws/eks/runtime-security-cluster/falco-alerts --follow

You should see the Falco JSON records appearing in real time.

3. Verify EventBridge and Lambda Invocation

Check the Lambda CloudWatch logs to see the automation in action:

aws logs tail /aws/lambda/falco-cloudwatch-bridge --follow
aws logs tail /aws/lambda/secops-pod-remediation-engine --follow

4. Confirm Email Subscription:

Go to the inbox of the email address you configured in variables.tf. Look for an AWS Notification email and click Confirm Subscription. If you already confirmed earlier, you will now receive an alert email with the subject [CRITICAL] Falco Alert: Terminal Shell Spawned Within Container.

5. Observe the Automated Remediation

Re-trigger the exploit execution

kubectl exec -it dynamic-web-app-pod -- /bin/bash

Within seconds of typing commands inside the shell, your terminal connection will abruptly freeze or drop with an exit message like connection reset by peer. This occurs because the Lambda script successfully intercepted the kernel-level alert and deleted the compromised container.

Inspect the cluster state:

kubectl get pods

Expected output: Your terminal will show that the compromised pod has completely disappeared or is stuck in a Terminating status state, proving your automated out-of-band response engine functions flawlessly!

---

Troubleshooting Guide for Common Errors

1. Error: Helm installation failed: security namespace already exists

This happens if you run the helm install command multiple times or if a previous installation failed midway.

The Fix: Completely purge the existing installation before re-running the launch:

helm uninstall falco -n security

2. Error: aws-auth configmap update failed

Your local AWS identity credentials might have drifted since initializing your cloud resources.

The Fix: Re-download your authorization tokens directly using the AWS CLI tool layer:

aws eks update-kubeconfig --region us-east-1 --name runtime-security-cluster

3. Error: Falco CrashLoopBackOff: 'found another plugin with name container

Pod restarts continuously. Logs show plugin registration error.

Root Cause: Falco 0.44+ built-in container plugin conflicts with falcoctl-installed plugin.

The Fix: Add to falco-values.yaml:

falcoctl:
  artifact:
    install:
      enabled: false
    follow:
      enabled: false

4. Terraform: 'Kubernetes cluster unreachable'

terraform apply fails when deploying Falco via Helm.

Root Cause: Kubernetes/Helm providers not configured with EKS auth.

The Fix: Add providers.tf with data.aws_eks_cluster and data.aws_eks_cluster_auth.

5. Lambda: 'No pod info found, skipping'

Lambda invokes but doesn't delete pods. Logs show parsing failure.

Root Cause: CloudWatch log format differs from expected (Fluent Bit wrapper vs. raw JSON).

The Fix: Update response_engine.py extract_pod_info() to handle multiple formats:

· wrapper['data']['output_fields'] (Fluent Bit envelope)
· wrapper['output_fields'] (raw Falco)
· wrapper['log'] (escaped JSON string)
· Text fallback for legacy output

6. Lambda: 'Unauthorized' or 403 from K8s API

Lambda reaches K8s API but deletion fails.

Root Cause:

· EKS Access Entry not created
· Access Policy Association missing
· Lambda IAM role not mapped in EKS

The Fix:

aws eks list-access-entries --cluster-name runtime-security-cluster
aws eks list-associated-access-policies \
  --cluster-name runtime-security-cluster \
  --principal-arn arn:aws:iam::ACCOUNT:role/secops-remediation-lambda-role

7. CloudWatch Subscription Filter Not Triggering Lambda

No Lambda invocations despite logs appearing in CloudWatch.

Root Cause:

· aws_lambda_permission missing or source_arn mismatch
· Subscription filter created before permission
· Filter pattern too restrictive

The Fix:

aws logs describe-subscription-filters \
  --log-group-name /aws/eks/runtime-security-cluster/falco-alerts
aws lambda get-policy --function-name secops-pod-remediation-engine

---

Complete Project Cleanup Step

To clean up your AWS account and avoid unexpected sandbox charges for EKS or compute resources, run this command inside your terraform/ directory to safely destroy every infrastructure asset built during this lab project:

terraform destroy -auto-approve

---

Summary of What I Built

Part 2 of this advanced cloud security series is completed! By combining runtime detection engines with cloud-native automation patterns, I built a highly robust system:

• Kernel Telemetry Monitoring: We deployed Falco with an optimized eBPF driver configuration to inspect runtime system calls directly within the Linux kernel, minimizing compute overhead.
• Decoupled Architecture: We routed security alerts completely out of the Kubernetes cluster, streaming data up to Amazon CloudWatch Logs to prevent log tampering.
• Automated Event Remediation: We deployed an event-driven Amazon EventBridge mapping rule to identify high-severity threats and instantly trigger an AWS Lambda function to isolate and eliminate compromised workloads.

Final Architectural questions

1. Why eBPF over kernel modules?

eBPF is safer (verified by kernel verifier), more portable (no kernel headers needed), and doesn't require privileged access to load.

2. Why out-of-band remediation?

If an attacker escapes the container, they can't kill the security agent because it lives outside the cluster as serverless functions.

3. How do you handle false positives?

The Bridge Lambda filters by priority (Critical/Emergency only). Production would add allowlists, anomaly scoring, and SOAR integration.

4. What if the Lambda fails?

CloudWatch Logs retains all events. Failed Lambda invocations trigger CloudWatch Alarms. SNS provides human notification as fallback.

5. How do you update Falco rules?

Rules are in Git. CI/CD pipeline runs terraform apply to update the ConfigMap/Helm values. Rolling update restarts Falco pods gracefully.

6. Why not GuardDuty?
GuardDuty is excellent for audit log analysis but doesn't see syscalls. Falco sees the actual system calls—what the process is doing, not just what the API recorded.

---

Bonus: NotebookLM Study Guide

Core System Configuration Comparison

Use this comparison matrix to evaluate your runtime security configurations during architecture reviews or engineering design sessions.

Architectural Matrix	Vulnerable Setup	Hardened Static Base (Project 1)	Active Runtime Defense (Project 2)
Primary Focus	Speed of delivery; rapid prototyping.	Source Code, Build-time checking, and Static configurations.	Live execution state, system call analysis, and active compromise mitigation.
Detection Coverage	None.	Catches known older packages and broad IaC infrastructure design misconfigurations.	Catches live human zero-day exploits, unexpected shell spawns, and reverse-shells.
Remediation Speed	Manual intervention following discovery or system failure.	Blocks pull requests; stops deployments at the deployment gate phase.	Event-driven automation; handles live threats and shuts down compromised pods within seconds.
AWS Identity Cost	Broad Node Policies ($0).	Highly Specific Least-Privilege IRSA Mappings ($0).	Decoupled Out-of-Band Lambda Controls ($0 Layer Free-Tier).

Threat Modeling Decision Tree

Threat Event Discovered
   │
   ├── Is it an outdated library or base image vulnerability?
   │     └── Yes -> Handle via Static Scanning Gates (Trivy / PR Block)
   │
   └── Is it malicious behavior on a running container (e.g., shell spawn, network scan)?
         └── Yes -> Handle via Runtime Protection Framework
               │
               ├── Local Event Log Capture (Falco eBPF telemetry)
               ├── Out-of-Band Centralization (Amazon CloudWatch Logs)
               └── Pattern Trigger Remediation (EventBridge Event -> AWS Lambda Eviction)

By completing both Part 1 and Part 2, you have evolved from simple perimeter security to building a resilient, defense-in-depth cloud-native application infrastructure. Go forth and secure your clusters.