Monitoring EKS audit logs with Falco security

Amazon EKS control plan logging provide audit and diagnostic logs directly from the Amazon EKS control plan to CloudWatch Logs in AWS account. The following cluster control plan log types correspond components of the Kubernetes control plane:

• API server: which exposes the Kubernetes API.
• Audit: Kubernetes audit logs provide record of individual users, administrators, or system components.
• Authenticator: This logs is unique to Amazon ESK, represent the control plane component, cluster management.
• Controller manager: Manage core control loops and watch the state of cluster through apiserver and attemping to move the current state towards desired state, kube-controller-manager.
• Scheduler: scheduler component manages when and where to run Pods.

Monitoring Amazon Elastic Kubernetes (Amazon EKS) audit Logs is an essential practice for enhancing the security and visibility of your Kubernetes clusters.

At a high-level overview, Falco is comprised of the following components:

• Event sources (drivers, Kubernetes audit events).
• A set of rules.
• Output system integration.

Currently, the Falco support two types of drivers: kernal module and eBPF probe.

Monitoring the EKS audit logs is beneficial for detect anomalous behavior (e.g. unauthorized access attempts, privilege escalations, abnormal resource usage). In addition, Falco also allow to custom rules tailored to your specific security needs.

Install Falco

Before Falco version 0.33.0, Falco allows to tracks the changed of Kubernetes audit event basing rule defined in k8s audit rule. However, there was no direct component allowing to monitor EKS audit event. There were two options at that time:

1. Deploy an EKS CloudWatch component to retrieve logs from AWS CloudWatch and push to Falco.
2. Using Falco EKS audit bridge, which is using AWS Kinesis Firehose and S3 for transferring data.

Now, Falco have introduced a plugin framework which allows to pull logs from AWS CloudWatch Logs Stream.

Prerequisite

Before deploying Falco, you need to ensure:

• Amazon EKS Cluster set up and running.
• kubectl installed in your local workstation or jump-host server where allowing to connect to the cluster.
• Helm package manager installed.

Install Falco

Installing Falco will go through the steps:

• Prepare IAM role service account.
• Prepare Falco helm values.
• Deploy Falco on the EKS cluster.

Create IAM Role service account

To allow Kubernetes Pods in EKS connect to AWS services we have to create an IAM role which have Assume Role With Web Identity and associated to Kubernetes service accounts where the pods using. In this case, the Falco need to pull the AWS CloudWatch EKS audit logs so it has to associated to IAM role having permission to access AWS CloudWatch logs.

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"ReadAccessToCloudWatchLogs",
      "Effect":"Allow",
      "Action":[
        "logs:Describe*",
        "logs:FilterLogEvents",
        "logs:Get*",
        "logs:List*"
      ],
      "Resource":[
        "arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:*"
      ]
    }
  ]
}

From the code block above, replace the following parameters:

• REGION: where the EKS cluster deploy.
• ACCOUNT_ID: AWS account id.
• CLUSTER_NAME: name of EKS cluster which deploy the Falco. We also need to create a assume role for web identity with the following information:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}"
        }
      }
    }
  ]
}

From the code block, replace the following parameters:

• ACCOUNT_ID: AWS account id.
• OIDC_PROVIDER: OpenID Connect (OIDC) which can copy from AWS EKS console.
• NAMESPACE: The namespace where Falco running.
• SERVICE_ACCOUNT: The Kubernetes service account which is associated to Kubernetes pods used to interact AWS CloudWatch logs. Follow the document Kubernetes service account IAM role to deploy the role to AWS. Or using terraform code to deploy the IAM. Clone the source code https://github.com/galiops/galireview/tree/main/blogs/falco/infra to local, then adding values for the variables in variables.tf file. After that, run the following command to deploy to AWS:

# cd into the codebase
terraform init
terraform plan
terraform deploy

After deploy, we need to copy the IAM role to use for deploy Falco helm chart. The format of the IAM role is

arn:aws:iam::<AWS_ACCOUNT_ID>:role/system-galireview-falco-worker-role

You can also get the IAM from AWS console -> IAM -> Role -> search for the role name: system-galireview-falco-worker-role -> copy the ARN.

Prepare Falco helm charts

An official Falco helm charts can be found from falcosecurity/falco ArtifactHub.
From local create an folder name falco and create the following files and folder:

.
|-charts/
|-Chart.yaml
|_values.yaml

cd into the charts folder and run the following command to download the latest Falco chart (3.4.1).

cd charts
wget https://github.com/falcosecurity/charts/releases/download/falco-3.4.1/falco-3.4.1.tgz

Adding the following information for Chart.yaml file:

name: falco
version: 3.1.1
appVersion: "0.34.1"
description: Falco

The values.yaml can be found from the link https://github.com/galiops/galireview/blob/main/blogs/falco/ops/values.yaml. Make sure change the two values:

• ROLE_ARN: The IAM role created from the previous step.
• CLUSTER_NAME: EKS cluster name.

Deploy Falco on the EKS cluster

You already setup and prepare things for deploying Falco security which allow pull audit log from AWS CloudWatch to Falco for monitoring. To deploy the Falco to EKS, follow the steps below:

# using kubectl to connect to EKS cluster
aws eks update-kubeconfig --name CLUSTER_NAME --region REGION_ID
# create falco namespace
kubectl create namespace falco
# cd in helm chart folder
helm install falco . -n falco -f values.yaml

Verification

After deploying Falco on Kubernetes, it's crucial to verify that it is functioning correctly and monitoring your cluster for security events as expected. Here are some verification steps you can take:

• Check Falco deployment status

kubectl get po -n falco

• Verify Falco logs

kubectl logs <falco-pod-name> -n falco

• The logs should not show critical issue.
• You can also generate some test events by follow the link https://github.com/falcosecurity/event-generator/tree/main.

Conclusion

In conclusion, the installation of Falco on Kubernetes is a straightforward yet powerful process that empowers organizations to bolster the security of their containerized applications and Kubernetes clusters. By following the step-by-step guide provided above, you can seamlessly integrate Falco into EKS environment and gain real-time security insights and protection. Check out the How to deploy Falco on Kubernetes (EKS) or Deploying Falco on Kubernetes (EKS) using Helm (Update 2023) for more detail and other instructions when working with Falco.