Automated threat detection on Amazon EKS using Terraform and GuardDuty

zoltan — Wed, 27 Sep 2023 06:22:30 +0000

Effective monitoring of Kubernetes audit logs offers several benefits. First and foremost, it helps in identifying unauthorized access attempts, unusual or suspicious activities, and potential security breaches. This enables administrators to take immediate action to mitigate risks and prevent further damage.

However, having logs just for the sake of it is a waste of money and effort if no one is checking them. This is where Amazon GuardDuty can easily help with its new feature that offers automated monitoring for these the audit logs.

Example findings

Couple of examples that it can spot for you:

CredentialAccess:Kubernetes/MaliciousIPCaller - An API commonly used to access credentials or secrets in a Kubernetes cluster was invoked from a known malicious IP address.
PrivilegeEscalation:Kubernetes/PrivilegedContainer - A privileged container with root level access was launched on your Kubernetes cluster.
Policy:Kubernetes/ExposedDashboard - The dashboard for a Kubernetes cluster was exposed to the internet
Or my personal favourite: Policy:Kubernetes/AnonymousAccessGranted - The system:anonymous user was granted API permission on a Kubernetes cluster.

More can be found here.

To use this new feature, there are 2 things you need to do:

Enable audit logs on your EKS cluster (if you haven't done already)
Setup EKS Audit log monitoring in GuardDuty

Enable audit logs on your EKS cluster using Terraform

Using aws_eks_cluster resource

If you are using the aws_eks_cluster resource, then you basically need to add a single line to it to enable Audit Logging.

resource "aws_eks_cluster" "example" {
  # ... other configurations ...

  enabled_cluster_log_types = ["audit"] # Additional types: api, authenticator, controllerManager, scheduler

  # ... other configurations ...
}

Using AWS's EKS Terraform module

If you are using the official EKS Terraform module, then you will need to make sure that audit is added to the cluster_enabled_log_types input parameter as below. (By default, it is already configured)

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"
  # ... other configurations ...
  cluster_enabled_log_types = ["audit"]
  # ... other configurations ...

Setup GuardDuty using Terraform for EKS Audit log

With the official aws_guardduty_detector resource this is fairly simple and it only takes a few lines.

resource "aws_guardduty_detector" "example" {
  enable = true

  datasources {
    # ... other configurations ...
    kubernetes {
      audit_logs {
        enable = false
      }
    }
    # ... other configurations ...
  }
}

Once you set it up, you should start receiving alerts from GuardDuty when it detects suspicious activities. For the fill list of findings, check out the official documentation.

Dynamic number of SNS subscribers using AWS CloudFormation ForEach

zoltan — Sat, 05 Aug 2023 19:15:28 +0000

CloudFormation is an AWSome service that empowers users to define and provision AWS resources using simple JSON or YAML templates. It allows for the automated deployment and management of complex cloud infrastructures, ensuring consistency and reducing manual errors. With CloudFormation, you can describe cloud-based applications and services in a declarative way, making it easier to update, delete, and recreate resources as needed. By providing infrastructure as code capabilities, CloudFormation streamlines the process of building and maintaining AWS environments, promoting scalability and reliability in cloud deployments.

The problem arises when you start using it at scale. One of the most common use-cases for CloudFormation is to simply deploy some basic templates/resources. While Terraform is well-suited for complex projects, CloudFormation, in my opinion, is superior when you only need to accomplish a simple task. For instance, I often need to set up some basic alerting in customer environments, such as budget alerts or basic CloudWatch alarms.

One of the most frustrating issues I have encountered is that it became increasingly difficult to create an SNS topic with a dynamic number of subscribers without modifying the actual CloudFormation template (thus avoiding the addition of extra Parameters, for example). The only way to achieve this was via Custom Resources. While this approach works fine, it necessitates the creation of additional Lambda functions as part of your template, which requires IAM roles and code maintenance over time, even when all you needed was to add another email address as a subscriber to your SNS topic.

The solution below is not flawless, and it introduces a different bottleneck that can complicate your template, but it simplifies setups like these. So, let's explore the full solution first, and then we can delve into the details of why certain parts are as they are.

AWSTemplateFormatVersion: 2010-09-09
Transform: 
  - 'AWS::LanguageExtensions'
  - 'AWS::Serverless-2016-10-31'
Description: Sample Cloudformation file for SNS subscriptions
Parameters:
  Subscribers:
    Type: String
    Description: Comma separated list of subscribers
    Default: ''
  SubscriberIndexes:
    Type: String
    Description: |
      Comma separated list of indexes.
      Should start from 0 and should be in sequence upto the number of subscribers.

      It is used as a helper to create unique logical ids for subscriptions.
    Default: ''
Resources:
  SampleTopic:
    Type: AWS::SNS::Topic
    Properties:
      DisplayName: Sample Topic
      TopicName: sample-topic
  Fn::ForEach::Subscriptions:
      - EmailIndex
      - !Split [',', !Ref SubscriberIndexes]
      - SampleTopicSub${EmailIndex}:
          Type: AWS::SNS::Subscription
          Properties:
            Endpoint: !Select [!Sub "${EmailIndex}", !Split [',', !Ref Subscribers]]
            Protocol: email
            TopicArn: !Ref SampleTopic

The first thing you notice, is that there are 2 parameters.

Subscribers
SubscriberIndexes

The param Subscribers speaks for itself. It is what we need after all. A comma (,) separated list of email addresses that should be subscribed to our new SNS topic.

So let's look at what SubscriberIndexes is for. It should be a list of "indexes" that we will need to use unfortunately, to get around some of the limitations of the Fn::ForEach function.

If you look at the documentation for it, you may notice that we need to set the key dynamically. Unfortunately, this key has some limitations as it can only contain alphanumeric characters. As you probably correctly recall, an at sign (@) or the dot (.) in the email does not count as "alphanumeric". So we need a way to generate a new index.

To get around this problem, we will actually build our loop on a new parameter called SubscriberIndexes instead of the original list of emails.

...
  Fn::ForEach::Subscriptions:
      - EmailIndex
      - !Split [',', !Ref SubscriberIndexes]
      - SampleTopicSub${EmailIndex}:
...

Now, this is all good, but we still need to get the actual emails into the resource definition. If you think about the comma separated list of emails, you will notice that they can be converted into an array using the Fn::Split function the following way:

!Split [',', !Ref Subscribers]

Once that's done, we now just need to reference the right items. This is where we will use the original EmailIndex variable. Users need to make sure that it's been set correctly, as it will fail otherwise. I don't cover it here, but it would be possible to setup some validation Rules by following the documentation.

After that, we jut need to access the right items by using the Fn::Select function. For example in this case:

!Select [!Sub "${EmailIndex}", !Split [',', !Ref Subscribers]]

Putting al this together, we will get the following template. (just make sure you set the index params right during deployment)

TLDR:

AWSTemplateFormatVersion: 2010-09-09
Transform: 
  - 'AWS::LanguageExtensions'
  - 'AWS::Serverless-2016-10-31'
Description: Sample Cloudformation file for SNS subscriptions
Parameters:
  Subscribers:
    Type: String
    Description: Comma separated list of subscribers
    Default: ''
  SubscriberIndexes:
    Type: String
    Description: |
      Comma separated list of indexes.
      Should start from 0 and should be in sequence upto the number of subscribers.

      It is used as a helper to create unique logical ids for subscriptions.
    Default: ''
Resources:
  SampleTopic:
    Type: AWS::SNS::Topic
    Properties:
      DisplayName: Sample Topic
      TopicName: sample-topic
  Fn::ForEach::Subscriptions:
      - EmailIndex
      - !Split [',', !Ref SubscriberIndexes]
      - SampleTopicSub${EmailIndex}:
          Type: AWS::SNS::Subscription
          Properties:
            Endpoint: !Select [!Sub "${EmailIndex}", !Split [',', !Ref Subscribers]]
            Protocol: email
            TopicArn: !Ref SampleTopic