DEV Community: Lorenz Vanthillo

Handling Automated Jira Tickets for New EventBridge Schema Discoveries

Lorenz Vanthillo — Mon, 26 Aug 2024 17:11:52 +0000

Let me start at the beginning. In my role as an AWS Cloud Engineer at one of my previous clients, an event-driven architecture was used where third parties constantly sent many events to our AWS environment via EventBridge. For each third party, we provided an event bus with various EventBridge rules.

The challenge here was keeping track of the event structure—how it was organized. Events were frequently updated, leading to many meetings to clarify things.

At the end of 2019, a large part of our problem was solved thanks to EventBridge Schema Discovery. By enabling this on the event buses, schemas were automatically generated based on the received events. This allowed us to generate code bindings from these schemas, which was a great help in our object-oriented environment.

Below you can see a very basic example event of a third party.

{
  "version": "0",
  "id": "ef21d5fc-a5ba-e2c6-fc4b-a8807455c64d",
  "detail-type": "orderType",
  "source": "com.company.A",
  "account": "xxx",
  "time": "2024-08-22T08:04:26Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "orderId": 123456789,
    "customer": {
      "customerId": "C001",
      "name": "John Doe"
    },
    "orderDate": "2024-08-22"
  }
}

AWS discovered a schema for these types of events:

By using the AWS Toolkit for Visual Studio Code, we could easily represent our events as strongly typed objects in our code.

Below is a very basic example on how we used the code bindings.

Output:

123456789
C001
<class 'schema.com_company_a.ordertype.OrderType.OrderType'>
<class 'dict'>

This improved our way of working, but we still encountered an issue. From time to time, third parties would add new attributes to their events. EventBridge would discover these changes, but developers often forgot to update the code bindings for the new schema. Although our implementation was robust enough to prevent breakages when new attributes were added, it resulted in new data that we weren’t utilizing. We had to rely on developers to remember to update their code bindings occasionally, and there was no clear process in place to manage this.

Sometimes a code binding wasn’t updated for months, and occasionally, two developers would update it simultaneously, leading to conflicts or duplicate work.

To handle this better, we decided to build a solution that automatically creates a Jira ticket whenever a third party updates their event and a new schema is discovered.

The solution is available in CloudFormation on my GitHub. Check the README.

The first step was to create an EventBridge rule on our default bus that would trigger whenever a new schema or a schema version update was discovered. This event was then sent to an SQS queue to serve as input for an EventBridge Pipe. Here, we could add additional filtering (optional in this example) and enrich our event using a Lambda function.

For enrichment, we used the describe_schema using boto3.

data = event[0]["input"]["detail"]

try:
    response = client.describe_schema(
        RegistryName=data["RegistryName"],
        SchemaName=data["SchemaName"],
        SchemaVersion=data["Version"],
    )
except ClientError as e:
    raise e

return_data = {
    "SchemaName": response["SchemaName"],
    "SchemaVersion": response["SchemaVersion"],
    "SchemaArn": response["SchemaArn"],
    "Content": json.loads(response["Content"]),
}

After enriching our data, we sent it to a Step Function workflow. This workflow, in turn, triggered the AWS-provided AWS-CreateJiraIssue SSM Automation, which automatically created a Jira ticket.

The ticket included details such as the schema name, the new schema version, and the ARN of the schema. (Additional content from the event can also be added if needed.)

+----------------+      +--------+      +-------------------------+      +----------------+      +-------------------------+
| EventBridge    | ---> | SQS    | ---> | EventBridge Pipe        | ---> | Step Function  | ---> | SSM Automation Document |
| Rule           |      |        |      | (Filtering & Enrichment)|      |                |      |                         |
+----------------+      +--------+      +-------------------------+      +----------------+      +-------------------------+

Let's demo this solution. Here you see an updated event, based on the original. The attribute status is new.

{
  "version": "0",
  "id": "dffbd38b-9258-d028-21f3-da0ba3c9e314",
  "detail-type": "orderType",
  "source": "com.company.A",
  "account": "xxx",
  "time": "2024-08-22T08:04:26Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "orderId": 123456789,
    "status": "Completed",
    "customer": {
      "customerId": "C001",
      "name": "John Doe"
    },
    "orderDate": "2024-08-22"
  }
}

A new schema will be discovered. This will trigger the whole solution. After the Lambda enriched our event, that updated event will be used as input for our Step Function.

The input event of our Step Function is enriched and looks like this.

[
  {
    "statusCode": 200,
    "data": {
      "SchemaName": "com.company.A@OrderType",
      "SchemaVersion": "2",
      "SchemaArn": "arn:aws:schemas:eu-west-1:xxx:schema/discovered-schemas/com.company.A@OrderType",
      "Content": {
        "openapi": "3.0.0",
        "info": {
          "version": "1.0.0",
          "title": "OrderType"
        },
        "paths": {},
        "components": {
          "schemas": {
            "AWSEvent": {
              "type": "object",
              "required": [
                "detail-type",
                "resources",
                "detail",
                "id",
                "source",
                "time",
                "region",
                "version",
                "account"
              ],
              "x-amazon-events-detail-type": "orderType",
              "x-amazon-events-source": "com.company.A",
              "properties": {
                "detail": {
                  "$ref": "#/components/schemas/OrderType"
                },
                "account": {
                  "type": "string"
                },
                "detail-type": {
                  "type": "string"
                },
                "id": {
                  "type": "string"
                },
                "region": {
                  "type": "string"
                },
                "resources": {
                  "type": "array",
                  "items": {
                    "type": "object"
                  }
                },
                "source": {
                  "type": "string"
                },
                "time": {
                  "type": "string",
                  "format": "date-time"
                },
                "version": {
                  "type": "string"
                }
              }
            },
            "OrderType": {
              "type": "object",
              "required": [
                "orderId",
                "orderDate",
                "customer",
                "status"
              ],
              "properties": {
                "customer": {
                  "$ref": "#/components/schemas/Customer"
                },
                "orderDate": {
                  "type": "string",
                  "format": "date"
                },
                "orderId": {
                  "type": "number"
                },
                "status": {
                  "type": "string"
                }
              }
            },
            "Customer": {
              "type": "object",
              "required": [
                "customerId",
                "name"
              ],
              "properties": {
                "customerId": {
                  "type": "string"
                },
                "name": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    }
  }
]

The Step Function workflow will, in turn, trigger the SSM automation and a create a Jira Ticket.

For convenience, I kept the ticket content brief. However, since the Content is also sent as input to the Step Function, it could also be included in the ticket. In that way, you can directly mention the new attributes or changes to the schema in your ticket.

This solution will also be triggered when a completely new event is discovered, as it will create a version 1 of the new event, causing the EventBridge rule to fire.

In this way, we were informed about updates and could schedule them into our sprint. This leads to an acceleration of our development cycle.

I’m aware that this is quite a specific case, but similar solutions can be built by setting up EventBridge rules to trigger on the desired events, and then using enrichment and Step Functions in combination with SSM to create further automations.

How I got my new bicycle thanks to AWS!

Lorenz Vanthillo — Thu, 06 Oct 2022 16:40:27 +0000

I bought my first bicycle in 2019. The pandemic had just started and during that period all team sports were stopped. The bicycle was already a few years old and I bought it for a few hundred euros. I really started to enjoy cycling and and I joined a small cycling team.

During the summer of 2021, I started looking around for a new bike. It was time to trade my old one for a new one. I became interested in a Canyon Endurance CF SL 8 but unfortunately they were always all sold out. In September 2021, I subscribed to the mailing list. I would get an email if the model was back in stock.

After a few months I started to wonder why I never got an email. I saw that similar bikes were sold and after doing a little investigation on Reddit, I read that the notification system wasn't working properly. The emails were probably sent in small badges, and it didn't take more than a few minutes before all Canyon Endurance CF SL 8s were sold out again.

It was time to look for another method. I read about a tool called Distrill, which could be used to check the site every so many seconds. The free version was working with a browser plugin. To be able to use it without your browser you needed a paid plan.

So I decided to create something similar which was cheaper.
I decided to develop a Lambda function to scrape* the web page and check if the bicycle in my size was still unavailable. I checked the content of the div.

I made use of the Python module BeautifulSoup to scrape the webpage and find a match for the Coming soon text.
If the text would change, I would get notified by email using SNS.

""" Scrape Canyon site."""
import requests
import os

import boto3
from bs4 import BeautifulSoup

client = boto3.client("sns")

url = "https://www.canyon.com/xxx"

def lambda_handler(event, context):
    """Main."""
    page = requests.get(url)
    results = BeautifulSoup(page.content, "html.parser")

    items = []
    for div in results.findAll(
        "div", attrs={"class": "productConfiguration__availabilityMessage"}
    ):
        text = div.text
        items.append(text.strip())

    # size small is 4th of the list
    small_item = items[3]
    print("item: " + small_item)

    if "Soon" not in small_item:
        print("alert!")
        client.publish(
            TopicArn=os.environ["TOPIC"],
            Message="Time to buy a Canyon!",
            Subject="Time to buy a Canyon!",
        )

The code (and actually the whole solution) is really basic. I didn't need advanced integrations or checks.
I used a simple EventBridge rule to trigger the Lambda every minute (except when I need to sleep!).

  Event:
    Type: AWS::Events::Rule
    Properties:
      Description: Trigger every minute
      Name: ScraperEvent
      # Run every minute when I don't sleep
      ScheduleExpression: cron(0/1 6-23 * * ? *)
      Targets:
        - Arn: !GetAtt Scraper.Arn #Lambda Arn
          Id: canyon-scraper
  LambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt Scraper.Arn
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt Event.Arn

SNS was used to send notifications.

  Topic:
    Type: AWS::SNS::Topic
    Properties: 
      DisplayName: canyon-topic
      Subscription: 
        - Endpoint: me@mail.com
          Protocol: email
      TopicName: canyon-topic

I also made use of Lambda layers to make the function as fast and lightweight as possible. I used Docker to build my layer and I uploaded it to S3.

$ docker run --rm \
--volume=$(pwd):/lambda-build \
-w=/lambda-build \
lambci/lambda:build-python3.8 \
pip install -r requirements.txt --target python

$ zip -vr python.zip python/

$ aws s3 cp python.zip s3://xxx-layers/python.zip

I configured my Lambda to make use of this layer.

  Scraper:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: canyon-scraper
      CodeUri: src/
      Handler: lambda.lambda_handler
      Runtime: python3.8
      Role: !GetAtt ScraperRole.Arn
      Environment:
        Variables:
          TOPIC: !Ref Topic
      Layers:
        - !Ref libs
  libs:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: python-lib
      Description: Dependencies for the canyon scraper
      ContentUri: s3://xxx-layers/python.zip
      CompatibleRuntimes:
        - python3.8

The function was fast enough to run with the minimum amount of memory and with a timeout of a few seconds.

This basic solution was monitoring the website ever minute. It's important to note that Cron expressions that lead to rates faster than 1 minute are not supported. If you need a faster solution, then check out this blog of a fellow Community Builder! Be sure that you're allowed to scrape and that you're not flooding the server!

Now I just had to wait, and after a month I got an email...

And I was able to order my favorite bicycle! It gave me great satisfaction. Like many, I'm also interested in new AWS features, fancy integrations and big setups, but sometimes you don't need the new fancy stuff to accomplish your needs.

All code is available here. Feel free to fork it and adapt it to your needs.

*Web scraping is legal if you follow the rules (avoid scraping personal data or intellectual property, check copyrights and robots.txt of the website, you're not allowed to flood the servers, ...!

How we use AWS Config and Security Hub for Cloud Governance

Lorenz Vanthillo — Wed, 17 Aug 2022 11:49:00 +0000

Introduction

As a freelance Cloud & DevOps engineer I come into contact with different companies and different types of solutions. At one of my clients I'm part of the Cloud Governance team. The company has a lot of teams which all have their own AWS accounts. They follow the principle of YBIYRI (You Build It, You Run It). A team gets complete freedom to choose which IaC tool or programming language they will use to create their solutions.
There are more than 200 AWS accounts which all contain different tools and applications. Quickly it became clear that there was need for some sort of governance for security improvements, tagging compliance and cost optimization.

That's why we initially developed the following solution.

We used a parent Lambda which retrieved all active AWS account IDs and put them on an SQS queue. This parent Lambda ran once each night. A child Lambda was triggered at least once for each account. The child Lambda assumed a role within each target account. We created this role in each account of our organization using Organizational StackSets. The role could be assumed from our central governance account and was used to retrieve the necessary information from within the target account.
At last the lambda was writing the data to DynamoDB. We had to query this DynamoDB table and warn the teams from which their accounts were not compliant to our rules.

The problems with this solution were:

We had to warn teams manually by sending emails.
The solution was not "real-time" because it only ran once each night.
There was a lack of visibility for the teams to see for which rules they were or weren't compliant.
The child Lambda execution time was high for big accounts.

There was clearly a need for a service which offered more visibility and where the rules could run not only scheduled but also event based. A combination of AWS Config & Security Hub seemed like the perfect fit. Teams were already familiar with Security Hub for general security findings. That's why we were very happy when AWS where announced the integration between AWS Config and Security Hub. We started the investigation to see if we could convert our old solution to AWS Organizational Config Rules. Often an Organizational AWS Managed Config Rule seemed a great fit.

AWS Organizational Managed Config Rule

The following managed rule is much easier and much more efficient than our old solution. It will check if each loggroup has a retention period set. It's a managed rule offered by AWS.

AWSTemplateFormatVersion: "2010-09-09"
Description: "Deploy Organizational AWS Managed Config Rule."

Resources:
  BasicOrganizationConfigRule:
      Type: "AWS::Config::OrganizationConfigRule"
      Properties:
          OrganizationConfigRuleName: "OrganizationConfigRuleName"
          OrganizationManagedRuleMetadata:
              RuleIdentifier: "CW_LOGGROUP_RETENTION_PERIOD_CHECK"
              Description: "Check if CW Loggroup has a retention period set."

We deploy the following CloudFormation stack in our aws-config-delegated-admin account. This will deploy the rule in each account of our organization. In our aws-security-hub-delegated-admin account we can see all results combined in Security Hub which increases the visibility for the teams.

Some nice-to-have feature would be to setup an integration between Security Hub and AWS Chatbot. This is something we will investigate in the near future. Now let's conclude with some general remarks about AWS Config Managed Rules.

Advantages:

Easy to implement and schedule.
Many managed rules offered by AWS.
No need to deploy a config role in each account.

Disadvantages:

Not possible to do customizations.
Not all trigger types are possible.

AWS Organizational Custom Config Rule backed by Lambda

To allow us to implement some more advanced customizations we are also using AWS Custom Config Rules backed by Lambda.

  CustomConfigRuleApplicationTag:
    Type: "AWS::Config::OrganizationConfigRule"
    Properties:
        OrganizationConfigRuleName: "custom-config-rule"
        OrganizationCustomRuleMetadata:
            Description: "Validate whether xxx."
            LambdaFunctionArn: !GetAtt LambdaFunction.Arn
            OrganizationConfigRuleTriggerTypes:
            # Run Rule each hour
              - "ScheduledNotification"
            # Run rule on each EC2 config change 
              - "ConfigurationItemChangeNotification"
            MaximumExecutionFrequency: "One_Hour"
            ResourceTypesScope:
              - "AWS::EC2::Instance"

We can trigger config rules on a scheduled base or when a resource change occurs (or both). We use the code below when the rule is triggered by a config change. We try to collect the most recent change of the affected resource using get_resource_config_history.

def get_configuration(resource_type, resource_id, configuration_capture_time):
    result = AWS_CONFIG_CLIENT.get_resource_config_history(
        resourceType=resource_type,
        resourceId=resource_id,
        laterTime=configuration_capture_time,
        limit=1)
    configurationItem = result['configurationItems'][0]
    return convert_api_configuration(configurationItem)

For our scheduled events we use list_discovered_resources to get the affected resources in a particular account.

def list_config_discovered_resources_per_resource_type(
    aws_config_client, resource_type
):
    """Get list of current resources in Config per resource type"""
    resources = []
    ldr_pagination_token = ""
    while True:
        discovered_resources_response = aws_config_client.list_discovered_resources(
            resourceType=resource_type,
            includeDeletedResources=False,
            nextToken=ldr_pagination_token,
        )
        resources.extend(discovered_resources_response["resourceIdentifiers"])
        if "nextToken" in discovered_resources_response:
            ldr_pagination_token = discovered_resources_response["nextToken"]
        else:
            break
    return resources

Complete and similar examples can be found in the official docs.

Advantages:

Using Lambda we can monitor nearly everything we want.
It can be triggered based on a schedule expression or by a configuration change.
Custom Rules also integrate with Security Hub.

Disadvantages:

We need to deploy an IAM Role in each account (using StackSets) which can be assumed from within the Lambda.
A Lambda can become pretty complex (especially when we need some combination of scheduled executions and executions triggered by a resource configuration change).

AWS Organizational Custom Config Rule backed by Guard

Currently, we're also exploring the brand new AWS Config rules backed by guard. Now you can write rules using guard which is a policy-as-code language. Below you can find some example of a Guard Rule which we are testing.

let resource_whitelist = [
    ...
    "AWS::EC2::EIP",
    "AWS::EC2::Instance",
    "AWS::EC2::SecurityGroup",
    "AWS::S3::Bucket",
    "AWS::SNS::Topic",
    "AWS::SQS::Queue",
    ...
]

rule resource_is_tagged when
    resourceType in %resource_whitelist {
        tags["Application"] !empty or
        tags["application"] !empty or
        tags["App"] !empty or
        tags["app"] !empty
        tags["Stage"] !empty or
        tags["stage"] !empty
}

This rule can be used to monitor our tagging compliancy. All resources in the resource_whitelist should be tagged with ((A/a)pplication OR (A/a)pp) AND (S/s)tage. If a resources is not part of the whitelist, it will be ignored.
We use the cfn-guard CLI to test the integration.

We store AWS Config events in a .yaml file. Below you can find an example of an AWS Config event for an AWS::EC2::SecurityGroup. Next we define what we expect to happen when we apply the rule on the event.

- name: TagCase1
  input: {
    "relatedEvents": [],
    "relationships": [
        {
            "resourceId": "vpc-xxx",
            "resourceName": null,
            "resourceType": "AWS::EC2::VPC",
            "name": "Is contained in Vpc"
        }
    ],
    "configuration": {
        "description": "test",
        "groupName": "test",
        "ipPermissions": [],
        "ownerId": "xxx",
        "groupId": "sg-xxx",
        "ipPermissionsEgress": [
            {
                "ipProtocol": "-1",
                "ipv6Ranges": [],
                "prefixListIds": [],
                "userIdGroupPairs": [],
                "ipv4Ranges": [
                    {
                        "cidrIp": "0.0.0.0/0"
                    }
                ],
                "ipRanges": [
                    "0.0.0.0/0"
                ]
            }
        ],
        "tags": [
            {
                "key": "app",
                "value": "something"
            },
            {
                "key": "stage",
                "value": "something"
            }
        ],
        "vpcId": "vpc-xxx"
    },
    "supplementaryConfiguration": {},
    "tags": {
        "app": "something",
        "stage": "something"
    },
    "configurationItemVersion": "1.3",
    "configurationItemCaptureTime": "2022-08-11T19:37:42.753Z",
    "configurationStateId": 111,
    "awsAccountId": "xxx",
    "configurationItemStatus": "OK",
    "resourceType": "AWS::EC2::SecurityGroup",
    "resourceId": "sg-xxx",
    "resourceName": "test",
    "ARN": "arn:aws:ec2:eu-west-1:xxx:security-group/sg-xxx",
    "awsRegion": "eu-west-1",
    "availabilityZone": "Not Applicable",
    "configurationStateMd5Hash": "",
    "resourceCreationTime": null,
    "CONFIG_RULE_PARAMETERS": {}
}
  expectations:
    rules:
      resource_is_tagged: PASS

- name: TagCase2
  input: {
    "relatedEvents": [],
    "relationships": [
        {
            "resourceId": "vpc-xxx",
            "resourceName": null,
            "resourceType": "AWS::EC2::VPC",
            "name": "Is contained in Vpc"
        }
    ],
    "configuration": {
        "description": "test",
        "groupName": "test",
        "ipPermissions": [],
        "ownerId": "xxx",
        "groupId": "sg-xxx",
        "ipPermissionsEgress": [
            {
                "ipProtocol": "-1",
                "ipv6Ranges": [],
                "prefixListIds": [],
                "userIdGroupPairs": [],
                "ipv4Ranges": [
                    {
                        "cidrIp": "0.0.0.0/0"
                    }
                ],
                "ipRanges": [
                    "0.0.0.0/0"
                ]
            }
        ],
        "tags": [
            {
                "key": "app",
                "value": "something"
            },
            {
                "key": "stagee",
                "value": "something"
            }
        ],
        "vpcId": "vpc-xxx"
    },
    "supplementaryConfiguration": {},
    "tags": {
        "app": "something",
        "stagee": "something"
    },
    "configurationItemVersion": "1.3",
    "configurationItemCaptureTime": "2022-08-11T19:37:42.753Z",
    "configurationStateId": 111,
    "awsAccountId": "xxx",
    "configurationItemStatus": "OK",
    "resourceType": "AWS::EC2::SecurityGroup",
    "resourceId": "sg-xxx",
    "resourceName": "test",
    "ARN": "arn:aws:ec2:eu-west-1:xxx:security-group/sg-xxx",
    "awsRegion": "eu-west-1",
    "availabilityZone": "Not Applicable",
    "configurationStateMd5Hash": "",
    "resourceCreationTime": null,
    "CONFIG_RULE_PARAMETERS": {}
}
  expectations:
    rules:
      resource_is_tagged: FAIL

The rule will pass or fail depending on the tag values. We expect the second case to fail because there is no stage tag but a stagee tag which is not valid.

$ cfn-guard test -r rule.guard -t tests.yaml
Test Case #1
Name: "TagCase1"
  PASS Rules:
    resource_is_tagged: Expected = PASS, Evaluated = PASS

Test Case #2
Name: "TagCase2"
  PASS Rules:
    resource_is_tagged: Expected = FAIL, Evaluated = FAIL

We deploy this rule in our organization. You can enable some useful debug logs to verify the events.

Now we add the correct tags to one of the security groups.

Advantages:

It's easy to learn the Guard Rules syntax.
Guard rules are easier to maintain than a custom lambda.
No need to deploy a config role in each account.
No programming knowledge required.
Custom Rules also integrate with Security Hub.

Disadvantages:

My rule is not triggered for SNS or SQS changes (opened a case w AWS).
API endpoint to deploy an organizational Config Rule backed by Guard is not working for SDK/CLI in eu-west-1 .
No option to define a retention period for the CloudWatch debug log group.
No CloudFormation support (yet)
Can not be triggered on a scheduled base.

Conclusion

Guard Rules seems promising but not fully production ready for our use case. In the future they can replace most of our Custom Config Rules backed by Lambda but not all of them. Some of these Lambda backed Custom Config Rules need to connect with non-AWS systems. AWS Managed Config Rules help us to increase our velocity to release new rules and allowed us to remove a lot of custom code. Thanks to AWS Config we have made great strides in our Cloud Governance journey.

All names and examples referenced in the story have been changed for demo purposes. Also screenshots are taken from demo accounts

Automate tagging of vulnerable Docker images in ECR

Lorenz Vanthillo — Sun, 19 Jun 2022 17:12:44 +0000

Amazon Elastic Container Registry is a fully-managed Docker container registry. It makes it easy for developers to store and manage Docker images inside their AWS environment.
ECR supports two types of image scanning. Enhanced image scanning requires an integration with Amazon Inspector. It will scan your repositories continuously. Basic image scanning will use the Common Vulnerabilities and Exposures (CVEs) database (open-source Clair) to find vulnerabilities in your images. You can trigger scans on image push or manually.

Although it's a very useful feature to find vulnerabilities in your images, it's still up to you to do something with this information.
By using Amazon Eventbridge, you can configure some action in response to the result of a scan.

In this case we will automatically tag images which contain CRITICAL vulnerabilities with a vulnerable prefix. Images with this prefix will be removed from your repository in 5 days. You can not give the same tag to multiple images in one repository. That's why we will use prefixes.

The Eventbridge rule (*) to create this condition:

EventTrigger:
  Type: "AWS::Events::Rule"
  Properties:
    Description: Trigger Lambda if a scan contains critical vulnerabilities
    EventPattern:
      source:
        - "aws.ecr" 
      detail-type:
        - "ECR Image Scan" 
      detail:
        finding-severity-counts:
          CRITICAL:
            - exists: true
            - numeric: [ ">", 0 ]
    State:  "ENABLED"
    Targets:
    - Arn: !GetAtt LambdaFunction.Arn
      Id: scan-ecr

(*) I was not able to make this working with an "OR" condition to tag the image when it contains critical "OR" high vulnerabilities. If you need an OR condition you can create multiple rules (with the risk to trigger your Lambda twice when you have both high and critical vulnerabilities). Another solution could be to forward all scan events and do the filtering inside the Lambda.

As you can see we've configured a LambdaFunction as target of our rule. To make this work we still need to provide the permission for our Eventbridge rule to trigger the Lambda.

PermissionForEventsToInvokeLambda: 
  Type: "AWS::Lambda::Permission"
  Properties: 
    FunctionName: !GetAtt LambdaFunction.Arn
    Action: "lambda:InvokeFunction"
    Principal: "events.amazonaws.com"
    SourceArn: !GetAtt EventTrigger.Arn

The Lambda will add the vulnerable tag prefix by using the ecr client its put_image function. By using this function we can add a tag to the image without the need to pull or push the Docker image.

def add_image_tag(repo, manifest, digest):
    """Add 'vulnerable' tag prefix to image."""
    try:
        response = client.put_image(
            repositoryName=repo,
            imageManifest=manifest,
            imageTag="vulnerable-" + digest[7:],
            imageDigest=digest,
        )
    except ClientError as error:
        logging.error("Unexpected error")
        raise error

    LOGGER.info("Image tag 'vulnerable' added!")
    return response

As you can see, we only assign the following policies to our Lambda role.

"ecr:BatchGetImage"
"ecr:PutImage"

We don't need to assign any additional policies like ecr:InitiateLayerUpload or ecr:UploadLayerPart which are required when you really want to push a Docker image.

To test our setup we can pull, tag and push an image to our registry.

$ docker pull node:11-stretch-slim
$ docker tag node:11-stretch-slim 123456789012.dkr.ecr.eu-west-1.amazonaws.com/node:11-stretch-slim
$ docker push 123456789012.dkr.ecr.eu-west-1.amazonaws.com/node:11-stretch-slim

Just after the push, the scan is still running.

After the scan the image is tagged with a vulnerable prefix

We can push some additional images to test the implementation

The solution is working! Still there are some "missing" features. As already mentioned, the OR Condition in the Eventbridge rule is not working. So we're currently only filtering for CRITICAL images. Next it would be nice if we could use Docker image tag prefixes in IAM policies, so we can deny that vulnerable images are being pulled. In my solution I'm removing images with this tag prefix after 5 days.

{
  "rulePriority": 1,
  "description": "Remove vulnerable images after 5 days",
  "selection": {
    "tagStatus": "tagged",
    "tagPrefixList": ["vulnerable"],
    "countType": "sinceImagePushed",
    "countUnit": "days",
    "countNumber": 5
  },
  "action": { "type": "expire" }
}

That being said, I hope you enjoyed reading this post and that you will start thinking about vulnerabilities in Docker images! The full solution is available on my GitHub.

New ECR Pull Through Cache Repositories

Lorenz Vanthillo — Sun, 12 Dec 2021 17:14:49 +0000

Just before AWS Re:invent 2021, AWS announced Pull Through Cache Repositories for Amazon Elastic Container Registry. This new feature allows you to keep your ECR registry in sync with the upstream registry.
It's important to note that there is only support for upstream repositories hosted on Quay.io and ECR Public. The most popular registry Docker Hub isn't supported but there is a way to work around this problem. Another recent announcement confirmed that Docker Official images are available on ECR Public. This means you should be able to use this new "Pull Through Cache" feature for Docker official images. Your flow would look like this:

ECR Pull Through Cache → ECR Public → Docker Hub (official images)

These two new features can solve the following issues:

There is no rate limit on ECR Public, so you can pull Docker official images as often as you need without the Docker Hub rate limits.
(They are rate limited if not authenticated with an Amazon account).
ECR Public is replicated across all AWS regions, so pulls are local to the region you pull from, which reduces latency.
By using Pull Through Cache you don't have to worry about keeping images in sync.

Now let's test this new "Pull Through Cache" feature. The first test case is pretty basic. I'm using an EC2 instance on which I've installed Docker and the instance is deployed in a private subnet which is connected to a NAT Gateway. I'll pull the docker/library/alpine:latest Docker image.

The instance role has the AmazonEC2ContainerRegistryFullAccess and AmazonElasticContainerRegistryPublicFullAccess policies attached. I think there is some room for improvement here. Pulling images wasn't working with the PowerUser roles.

Now I'll create the following "Pull Through Cache" rule:

I connect to my EC2 instance (deployed in a private subnet) and I'll pull the docker/library/alpine:latest image:

$ aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public

$ docker pull 123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/alpine:latest
latest: Pulling from ecr-public/docker/library/alpine
59bf1c3509f3: Pull complete
Digest: sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
Status: Downloaded newer image for 123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/alpine:latest
123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/alpine:latest

The image is pulled. That was easy. The "Pull Through Cache" feature will sync future versions to our repository.
Now let me verify the connection to ECR. I'm using a route through a NAT Gateway so the connection should go over the public internet.

nslookup 123456789101.dkr.ecr.eu-west-1.amazonaws.com
...
Address: 63.33.82.70

63.33.82.70 represents the public IP of my ECR registry.

Now for the second case I will remove the NAT gateway and corresponding routes. I will also configure the following VPC endpoints:

ecr.api (Interface endpoint)
ecr.dkr (Interface endpoint)
s3 (Gateway endpoint)

These endpoints are connected to my VPC. After configuring the security groups I'm able to establish a private connection from my EC2 to ECR.

Now I'll pull a new image so I can be sure no cache is being used.

$ docker pull 123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/node:lts
lts: Pulling from ecr-public/docker/library/node
c4cc477c22ba: Pulling fs layer
077c54d048f1: Pulling fs layer
0368544993b2: Pulling fs layer
dd9d1af71976: Waiting
eb8ae1e274f4: Waiting
d77ded8fde28: Waiting
7e078f8be34b: Waiting
eaf05a96d7a3: Waiting
701ddb919de1: Waiting
error pulling image configuration: Get "https://d2glxqk2uabbnd.cloudfront.net/78e9cb-903773873662-febe5bc4-51ed-c6fb-e028-9449267842e2/e6e9695a-7d27-44bc-8bbe-f6aa12aa29d1?CallerAccountId=400469424169&Expires=1639329365&Signature=DIiMLgpQuPHYXkfaOswdB0Y90RNx1oUCYqtptqx-vSevE7-l7YHlEFxs~--Ila~ihIPWz-qYEkTPk0MBqgRFDYdKbt5sveRSfoXSx2KNFtMFzl0MFS3NecNSKs-e9OfQXiaAfMcBn8vv6He2Zr3lFMzf8KKDxALEL3bYeVXvJQuWXA901HXPm44~O3Iq8BDrcxXc7PKfUjkS9XBtZ-HQY9EYGjxyPpck3fFMNNoWFfLz6H2-DKM1D6PplY9CvbYAhTmHwpDDls2Tw8k7WL3tY9bULBX25gBR4WptdeMLFXzneWWSpPkriMHlFqTLqQVkjGlFeZSG2coUQBVswi0mMw__&Key-Pair-Id=KSPLJWGOARK62": 
dial tcp 13.224.227.216:443: i/o timeout

The pull command times out. I see the new repository is created but the image is not there yet:

After some time I see that the image appears. ECR was still pulling the image in the background.

Now the image is available in my ECR registry, I can retry to pull it.

docker pull 123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/node:lts
lts: Pulling from ecr-public/docker/library/node
c4cc477c22ba: Pull complete
077c54d048f1: Pull complete
0368544993b2: Pull complete
dd9d1af71976: Pull complete
eb8ae1e274f4: Pull complete
d77ded8fde28: Pull complete
7e078f8be34b: Pull complete
eaf05a96d7a3: Pull complete
701ddb919de1: Pull complete
Digest: sha256:89b59ce49929d8a8e230946bdb1b58c14cdbbb86c9a7397610afcecfce1be035
Status: Downloaded newer image for 123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/node:lts
123456789101.dkr.ecr.eu-west-1.amazonaws.com/ecr-public/docker/library/node:lts

Now the pull works! I was a little surprised by this behaviour but it's described in the docs. Also check the other considerations.

When an image is pulled using a pull through cache rule for the first time, if you've configured Amazon ECR to use an interface VPC endpoint using AWS PrivateLink then you need to create a public subnet in the same VPC, with a NAT gateway, and then route all outbound traffic to the internet from their private subnet to the NAT gateway in order for the pull to work. Subsequent image pulls don't require this.

The reason behind this behaviour is described here. During my testing I discovered that if the image is small enough, it can work from the first time. Only bigger images seem to result in this issue.

Many container orchestration tools will retry image pulls so this shouldn't be a big issue. Still it would be nice if the very first pull would always work.

To conclude, I'm pretty happy with these improvements. There is no need anymore to create a mechanism keeping your images in sync.
Still there is room for some improvements:

Support (all?) Docker Hub images, not only the official ones
Support additional registries
Make the initial image pull succeed when you're using a private connection to ECR
CloudFormation support
Support "Pull Through Cache" for private repositories (authentication)
Cached images are only checked once per 24 hours
Make clear which policies are needed + Update AWS managed policies

I have no doubt that AWS will make some improvements soon.
I hope you enjoyed it!

Access private containers on Amazon ECS using PrivateLink and Terraform

Lorenz Vanthillo — Sun, 21 Nov 2021 12:03:19 +0000

Many companies are using container orchestration services (like ECS or EKS) to host their microservice environment. Those microservices can offer APIs which need to be accessible for other customers.
If those customers are also using AWS then the best solution would be to keep all communication privately inside AWS. The services inside the VPC of the customer should be able to communicate with the containers hosted in the VPC of the provider which offers the APIs. In this demo I'll use ECS to host the API.

I'll use one AWS account but two VPCs. One VPC for the API provider and one VPC for the customer (= API consumer). In reallity they will both have a separate account but that does not make a big difference in this setup (see later). Another important remark is that I'll serve my API over HTTP to keep things basic. It's recommended to use HTTPS.

The API provider is hosting their containers in ECS using Fargate (serverless) and uses a private Application Load Balancer (ALB) in front of it. To allow a private connection between the VPC of the customer and the VPC of the API provider we need to setup AWS PrivateLink. AWS PrivateLink can provide a private connectivity between VPCs or AWS services. It's not possible to connect PrivateLink with an Application Load Balancer. It requires a Network Load Balancer (NLB). This means our architecture will look like this:

The sample API being used in this demo is a task API which can print all tasks and specific tasks based on id.

$ docker run -d -p 8080:8080 lvthillo/python-flask-api
$ curl localhost/api/tasks
[{"id": 1, "name": "task1", "description": "This is task 1"}, {"id": 2, "name": "task2", "description": "This is task 2"}, {"id": 3, "name": "task3", "description": "This is task 3"}]
$ curl localhost/api/task/2
{"id": 2, "name": "task2", "description": "This is task 2"}

To set up the infrastructure I'm using Terraform. The code is available on my GitHub. We need to create two VPCs. I've created a basic module which will create a VPC which contains four subnets (in two AZs) and deploy a NAT Gateway and an Internet Gateway. Also the route tables are configured properly.

module "network" {
  source                 = "./modules/network"
  vpc_name               = "vpc-1"
  vpc_cidr               = var.vpc
  subnet_1a_public_cidr  = var.pub_sub_1a
  subnet_1b_public_cidr  = var.pub_sub_1b
  subnet_1a_private_cidr = var.priv_sub_1a
  subnet_1b_private_cidr = var.priv_sub_1b
}

Next we need to deploy the ECS cluster, service and task definition inside the VPC of the API provider. The ALB will be deployed in front of it.
The ecs module will deploy two sample API containers.

module "ecs" {
  source               = "./modules/ecs"
  ecs_subnets          = module.network.private_subnets
  ecs_container_name   = "demo"
  ecs_port             = var.ecs_port # we use networkMode awsvpc so host and container ports should match
  ecs_task_def_name    = "demo-task"
  ecs_docker_image     = "lvthillo/python-flask-api"
  vpc_id               = module.network.vpc_id
  alb_sg               = module.alb.alb_sg
  alb_target_group_arn = module.alb.alb_tg_arn
}

module "alb" {
  source         = "./modules/alb"
  alb_subnets    = module.network.private_subnets
  vpc_id         = module.network.vpc_id
  ecs_sg         = module.ecs.ecs_sg
  alb_port       = var.alb_port
  ecs_port       = var.ecs_port
  default_vpc_sg = module.network.default_vpc_sg
  vpc_cidr       = var.vpc # Allow inbound client traffic via AWS PrivateLink on the load balancer listener port
}

The security group of the ECS service only allows connections from the ALB.

resource "aws_security_group_rule" "ingress" {
  type                     = "ingress"
  description              = "Allow ALB to ECS"
  from_port                = var.ecs_port
  to_port                  = var.ecs_port
  protocol                 = "tcp"
  security_group_id        = aws_security_group.ecs_sg.id
  source_security_group_id = var.alb_sg
}

An NLB has no security group so we can not reference it in the security group of the ALB. We need to find another way to restrict ALB access.
The alb module uses the VPC subnet - in which we will create our NLB - as source in the ALB security group. This means everything inside the API provider its VPC can communicate with the ALB.
In a production environment it can be useful to deploy the NLB in separate NLB subnets and restrict to those subnet CIDR ranges.
Now let's deploy this private NLB in front of our private ALB.

module "nlb" {
  source            = "./modules/nlb"
  nlb_subnets       = module.network.private_subnets
  vpc_id            = module.network.vpc_id
  alb_arn           = module.alb.alb_arn
  nlb_port          = var.nlb_port
  alb_listener_port = module.alb.alb_listener_port
}

Here I'm using some new AWS magic. Since September 2021 it's possible to register an ALB as target of an NLB.
This was not possible in the past. I created some Terraform module which deployed a Lambda to monitor the IPs of the ALB and updated the NLB target group when needed. I'm very happy that it's not needed anymore!
Now we can define alb as target_type for our NLB target group.

resource "aws_lb_target_group" "nlb_tg" {
  name        = var.nlb_tg_name
  port        = var.alb_listener_port
  protocol    = "TCP"
  target_type = "alb"
  vpc_id      = var.vpc_id
}

Next we can deploy our AWS PrivateLink-powered service (referred to as an endpoint service). In a real production environment you'll have to add allowed_principals to whitelist the account ID from the customer.

resource "aws_vpc_endpoint_service" "privatelink" {
  acceptance_required        = false
  network_load_balancer_arns = [module.nlb.nlb_arn]
  #checkov:skip=CKV_AWS_123:For this demo I don't need to configure the VPC Endpoint Service for Manual Acceptance
}

Everything is ready from API provider perspective. We have deployed the VPC, ECS infrastructure, ALB, NLB and PrivateLink.
The customer its VPC can be deployed using the same network module. The customer needs a VPC endpoint to connect to PrivateLink.
The VPC endpoint Fully Qualified Domain Name (FQDN) will point to the AWS PrivateLink endpoint FQDN.
This creates an elastic network interface to the VPC endpoint service that the DNS endpoints can access.

resource "aws_vpc_endpoint" "vpce" {
  vpc_id             = module.network_added.vpc_id
  service_name       = aws_vpc_endpoint_service.privatelink.service_name
  vpc_endpoint_type  = "Interface"
  security_group_ids = [aws_security_group.vpce_sg.id]
  subnet_ids         = module.network_added.public_subnets
}

At last I'm deploying an EC2 in the customer VPC (public subnet) to test the integration. Here for I'm using this Terraform module.
Update the keyvariable in variables.tf with the name of your existing AWS SSH key.

module "ec2_instance_added" {
  source                      = "terraform-aws-modules/ec2-instance/aws"
  version                     = "~> 3.0"
  name                        = "test-instance-vpc-2"
  associate_public_ip_address = true
  ami                         = "ami-05cd35b907b4ffe77" # eu-west-1 specific
  instance_type               = "t2.micro"
  key_name                    = var.key
  vpc_security_group_ids      = [aws_security_group.ssh_sg_added.id]
  subnet_id                   = element(module.network_added.public_subnets, 0)
}

Deploy the Terraform stack to test the setup.

$ git clone https://github.com/lvthillo/aws-ecs-privatelink.git
$ cd aws-ecs-privatelink
$ terraform init
$ terraform apply

Check the Terraform outputs and SSH to the EC2 instance deployed in the customer VPC. Then try to curlthe VPC endpoint.

$ ssh -i your-key.pem ec2-user@54.247.23.167
$ curl http://vpce-07715651ecce291f6-x6r0avfj.vpce-svc-0beea6830e4c68cd2.eu-west-1.vpce.amazonaws.com/api/tasks
[{"id": 1, "name": "task1", "description": "This is task 1"}, {"id": 2, "name": "task2", "description": "This is task 2"}, {"id": 3, "name": "task3", "description": "This is task 3"}]
$ curl http://vpce-07715651ecce291f6-x6r0avfj.vpce-svc-0beea6830e4c68cd2.eu-west-1.vpce.amazonaws.com/api/task/2
{"id": 2, "name": "task2", "description": "This is task 2"}

That was it! Our customer is able to access our API hosted in ECS using PrivateLink. All communication remains inside AWS.
There are some important caveats to remember which are described at the bottom of this post. Don't forget to destroy this demo stack using terraform destroy!

I hope you enjoyed it!

Build Serverless Applications using CDK and SAM

Lorenz Vanthillo — Fri, 21 May 2021 15:56:56 +0000

AWS recently announced the public preview of Serverless Application Model (SAM) support for CDK. SAM is an open-source framework that can be used to build, test and deploy serverless applications on AWS. It provides a Lambda-like execution environment that lets you locally build, test, and debug applications. Previously this could only be defined by SAM templates but now it is also possible through the AWS Cloud Development Kit (CDK)!

I will guide you through a small demo project to demonstrate how to build a serverless application with AWS CDK and test it locally with AWS SAM.

We will build a simple REST API which shows the current bid or ask price of a certain cryptocurrency on Binance (exchange), expressed in the value of Bitcoin.

The API expects two query parameters:

coin: (ETH, DOG, LINK, DOT, ...)
type: (bid or ask price)

Example of the API call:

$ curl "http://127.0.0.1:3000/crypto?type=ask&coin=ETH"
{"coin": "ETH", "price": 0.066225}

The setup in AWS will also be pretty straight forward.
We will set up a Lambda proxy integration in API Gateway

To get started, we need to install the AWS CDK CLI and create a new CDK project. I use Python as client language.

$ npm install -g aws-cdk
$ cdk init app --language python

The project structure looks like this:

.
├── README.md
├── app.py
├── cdk.json
├── requirements.txt
├── sam_cdk_demo
│   ├── __init__.py
│   └── sam_cdk_demo_stack.py
└── setup.py

The file sam_cdk_demo/sam_cdk_demo_stack.py should contain our code to define the AWS cloud resources we need but first let's start with writing our Lambda.

Create a folder inside the root of the project called "lambda" and add a handler.py. The ccxt library is used by our Lambda to interact with the Binance API. The Lambda itself is very basic on purpose.

import ccxt
import json


# use CCXT library to connect with Binance API
exchange = getattr(ccxt, 'binance')({
    'timeout': 3000,
    'enableRateLimit': True
})

def get_current_price(coin_name, price_type):
    # fetch latest ticker data for coin pair xxx/BTC
    ticker = exchange.fetch_ticker('{}/BTC'.format(coin_name))
    # get ask/bid price from ticket data
    current_price = ticker[price_type]
    return current_price

def lambda_handler(event, context):
    # get values from query string parameters
    coin = event['queryStringParameters']['coin']
    price = event['queryStringParameters']['type']

    # CCXT exchange expects coin in uppercase
    valid_coin = coin.upper()

    # get current price based on coin name and price type (ask/bid)
    current_price = get_current_price(valid_coin, price)

    return {
        'statusCode': 200,
        'headers': { 
            'Content-Type': 'application/json'
        },
        'body': json.dumps({
            'coin': valid_coin,
            'price': current_price,
        })
    }

Don't forget to add a requirements.txt inside the folder to make the ccxt library available to the Lambda.

ccxt==1.50.13

The Lambda is ready! Now we will use AWS CDK to define our AWS infrastructure. We need to deploy the Lambda and create an API Gateway in front of it. Update the file demo/demo_stack.py. We keep the code pretty basic again.

from aws_cdk import (
    aws_lambda as _lambda,
    aws_apigateway as apigw,
    core,
)

class CdkLambdaSamStack(core.Stack):

    def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # creating Lambda function that will be triggered by the API Gateway
        get_price_handler = _lambda.Function(self,'CryptoFunction',
            handler='handler.lambda_handler',
            runtime=_lambda.Runtime.PYTHON_3_8,
            code=_lambda.Code.asset('lambda'),
            timeout=core.Duration.seconds(30),
        )

        # create REST API
        api = apigw.RestApi(self, 'crypto-api')

        # add resource /crypto
        resource = api.root.add_resource('crypto')

        # create Lambda integration 
        get_crypto_integration = apigw.LambdaIntegration(get_price_handler)

        # add method which requires two query string parameteres (coin and type)    
        resource.add_method(
            http_method='GET',
            integration=get_crypto_integration,
            request_parameters={
                'method.request.querystring.coin': True,
                'method.request.querystring.type': True
            },
            request_validator_options=apigw.RequestValidatorOptions(validate_request_parameters=True)
        )

Update the requirements.txt in the project root with the necessary modules.

aws-cdk.core
aws-cdk.aws_lambda
aws-cdk.aws_apigateway

Start the Python virtual environment which is created by CDK and install the modules.

$ source .venv/bin/activate
(.venv)$ pip3 install -r requirements.txt

We will use AWS SAM to test our setup locally. It's important to mention that you need to have Docker installed. We will use Docker to build our code. The Lambda will also run inside as a Lambda-like Docker container.

Prepare the deployment artifact.

(.venv)$ sam-beta-cdk build --use-container

Start the local API Gateway.

$ sam-beta-cdk local start-api
...
* Running on http://127.0.0.1:3000/

We can use a tool like Postman (or curl or just your browser) to perform calls against our API.

It takes a few seconds to execute the function because AWS SAM is spinning up a Docker container to execute our code. After the execution the container is destroyed.

When everything looks fine we can deploy it to AWS.

(.venv)$ cdk bootstrap
(.venv)$ cdk deploy -a .aws-sam/build

Now test against the deployed API.

We were able to test our API and Lambda using the new Serverless Application Model integration with CDK! You can find all code on my GitHub. Be aware that this feature is in preview. Feel free to do more extensive testing. You can report bugs and submit feature requests to the SAM opensource repository.