DEV Community: Ziad Osman

Log CloudTrail events to DynamoDB using AWS State Machine

Ziad Osman — Sat, 10 Aug 2024 16:56:21 +0000

Introduction

If you’re ever browsing through the different AWS services and offerings, you might have come across AWS State Machines and gotten scared by the name. In reality, the state machine service allows you to chain events easily and seamlessly. In addition to its usage to chain other AWS services together, State Machines have some cool integrations up their sleeves, which can save you from writing your own logic. Less moving parts means less stuff to break!

This will be a two part blog, where each part will focus on solving a real world issue using State Machines.

For this blog, I will be using a trick that allows a state machine to take in an input, and write it directly to DynamoDB, without you having to write your own database insert logic. This can be used for a real world scenario where you need to log certain events from CloudTrail to a DynamoDB table.

AWS Resources needed

In order to complete this demo, I’m going to need to create a few AWS resources:

A DynamoDB table: this will be used to save the event.
A state machine: for obvious reasons :).
An EventBridge rule: this will be used to transfer the events from CloudTrail to our state machine.

1 - DynamoDB

First off, we need to create our DynamoDB table. For my example, I'm logging the event as it is, and I’m using the EventID, which is readily available as part of the event, as my Partition Key.

Here is the terraform snippet of my DyanmoDB table. Please be aware that on-demand mode is probably not the most economical billing mode:

resource "aws_dynamodb_table" "state_machine_demo_table" {
  name           = "state-machine-demo-table"
  tags = var.tags
  billing_mode   = "PAY_PER_REQUEST"  # On-demand
  hash_key       = "EventID"


  attribute {
    name = "EventID"
    type = "S"
  }
}

2 - state machine

Our state machine definition will be quite simple. Let’s look at it and break it down.

{
  "Comment": "transfer CloudTrail events to DynamoDB.",
  "StartAt": "WriteToDynamoDB",
  "States": {
    "WriteToDynamoDB": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:putItem",
      "Parameters": {
        "TableName": "state-machine-demo-table",
        "Item": {
          "EventID": {
            "S.$": "$.detail.eventID"
          },
          "EventData": {
            "S.$": "States.JsonToString($.detail)"
          }
        }
      },
      "End": true
    }
  }
}

The state machine only has one task, which is to write to our DB. We use the states:::dynamodb:putItem resource, which is the resource responsible for the native integration between the state machine and DynamoDB.

In the parameters sections, we specify the table name, and the Item. Each Item element will become a column in our DB row. Since EVENTID is our Partition key, this one is mandatory. Note that you can create as many items as you want depending on your need.

If you look closely at the annotation, you’ll see this : "S.$": "$.detail.eventID". What this means is that We’re going to insert an item of type string (hence the S at the beginning). All CloudTrail events nest their eventID under the “detail” section, which is why we’re retrieving it there. That is to say that you can perform logic on your Json object in the state machine so that you only retrieve the data you want. This is a really powerful feature, and a necessary one if you plan on inserting complex data.

The Other item is our Event data, turned to a string via the built-in function States.JsonToString().

3 - EventBridge

Quite simply, this is an event bridge rule that will have the state machine as a target. The event pattern can be whatever events you would like to catch. For the sake of this example, I added 2 events (CreateSubnet and DeleteSubnet).

This is what my event pattern looks like:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventName": [
      "CreateSubnet",
      "DeleteSubnet",
    ]
  }
}

When creating your EventBridge rule, remember to associate it to the Default EventBus. This will ensure that the eventbridge rule picks up the CloudTrail events. Also, remember to set your state machine as the target of the rule.

Conclusion

With this, we have just created an AWS State Machine that takes in CloudTrail events as an input, and directly writes them to a DynamoDB table. This can be changed to a more complex architecture where your data is coming in from another source, and where you retrieve only certain data in your state machine.

Part 2

In part 2 of this blog, I'm going to show you how to chain multiple lambda functions, so that the output of one is fed as an input to the other. More importantly, I’m going to show you how you can trigger parallel executions of your lambda function, such that for a list of objects, a lambda is triggered for each one. This trick will help you in processing data in parallel. Check it out here!

Parallel Lambda execution with AWS State Machine

Ziad Osman — Sat, 10 Aug 2024 16:56:10 +0000

This is part 2 of a 2 part blog about real life scenarios that can be solved with AWS State Machine. In part 1, we discussed how to log CloudTrail events to DynamoDB by using a State Machine to write directly to DynamoDB. If you’re interested in that part, please refer to this link.

In this blog, we’re going to be using a State Machine to feed the output of one lambda function to the other. Furthermore, this output is going to be a list of objects. And, for each object, we’re going to trigger a lambda function in parallel. This is very useful when you have a scenario where you need to process a lot of objects at the same time.

Prerequisites

In addition to the state machine, I’m going to be creating two lambda functions. The first will be called OutputLambda, and the second will be called InputLambda.

We’re going to go back to the code of the lambda functions in a second, but first, let’s look out our State Machine.

1 - State machine

The state machine is going to have two Steps. The first step is the OutputLambda, which will trigger step two upon completion. The second step will be of type Map, which means that it will trigger once for each element in a list, in parallel.

Here is what our state machine definition looks like:

{
  "Comment": "parallel execution of lambda functions demo",
  "StartAt": "FirstState",
  "States": {
    "FirstState": {
      "Type": "Task",
      "Resource": "<OutputLambdaArn>",
      "Next": "IterateOverList"
    },
    "IterateOverList": {
      "Type": "Map",
      "ItemsPath": "$.list",
      "Iterator": {
        "StartAt": "SecondState",
        "States": {
          "SecondState": {
            "Type": "Task",
            "Resource": "<InputLambdaArn>",
            "End": true
          }
        }
      },
      "End": true
    }
  }
}

Let’s break down what we got:

The first step is very simple, as it specifies a lambda function as a resource and sets IterateOverList as its next step.

IterateOverList is defined as type Map, which is what we need to achieve the parallelism as specified before. It specifies the ItemsPath as being named “list”. This means that OutputLambda needs to return an object named “list” for it to be caught by this state (we'll get back to this in a second).

The Iterator section is where the magic happens. This is simply a foreach loop, that loops over the elements of our object named “list”, and triggers the task we defined for each element. Finally, inside the Iterator section, we have defined our task as being the InputLambda.

Now, let’s move on to look at what our lambda functions look like.

2 - OutputLambda

This function only needs to return a list of objects. As defined in the ItemsPath section of our state machine definition, the name of this list of objects should be “list”. Here is what the lambda function could look like, using Python3:

def lambda_handler(event, context):
    fruits = ["apple","orange","pear"]
    return {'list': fruits}

Next up, let's look at our InputLambda.

3 - InputLambda

This lambda function will be triggered three times at the same time by our state machine (since in our example, the list has three items). Each instance of it triggering will have access to a different element of the list. Again, using Python3, here is how to retrieve that element in the lambda_handler.

def lambda_handler(event, context):
    print(f"fruit of the day: {event}.")

Yep, as simple as that! The list element is available as the event.

Conclusion

In this blog, I showed you how you can use a state machine to chain multiple lambda functions together. We saw how we can feed the output of one lambda function as the input of another. We also saw how to trigger a lambda function multiple times in parallel, which is useful to process data in parallel at the same time.

CloudFront 101: API Gateway and S3 SPA origins

Ziad Osman — Thu, 06 Jul 2023 12:10:58 +0000

I was recently thrown into the deep end with CloudFront when I had to configure an AWS account for a client. The account had a CloudFront distribution with multiple origins, one of which is a single page application S3 static website, and the others apis in api gateway. Because of the scarce and/or dispersed information I found, I decided to aggregate all the points that I found challenging into a single blog.

Unlike my other blogs, this one is more of a “what-to-do” and less of a “step by step how-to”. If you find it too long and too condensed to be read whole, I would recommend referring to the specific points you’re stuck on.

The points that will be covered are:

1 - CloudFront VS edge optimized api gateway
2 - api default endpoint vs custom domain name
3 - CloudFront multi origin default behavior and ordered behavior
4 - Cloudfront with an SPA (paths, 403 redirects and lambda@edge)

Introduction

Cloudfront is AWS’s offering of a CDN, or content delivery network. In short, its job is to cache your content through a worldwide network of edge locations to improve latency and user experience.

On the other hand, Api Gateway, as the name suggests, is a service that provides an entry point to your apis.

And finally S3, which is simply a storage solution that AWS offers. What’s special about S3 is that you can use it to host static websites (think read-only sites, or sites that can use apis to interact with the database rather than having a full-on backend framework).

CloudFront Vs edge optimized api gateway

Edge optimized endpoints

Suppose you need to leverage the power of edge locations to decrease the latency of your apis. In that case, you need to use CloudFront. Luckily, you don’t need to figure out how CloudFront works to use it with your apis, since Api Gateway offers a nifty integration to CF.

All you need to do is go into your api settings and switch the endpoint type to edge optimized.
This will create a CloudFront distribution for you and link it to your api.

So why would you ever use a manually created CF distribution if this option already exists?

Linking your existing CloudFront distribution to your api

While edge optimized endpoints are good enough for the majority of use cases, the CF distribution that is created for you cannot be modified and is in fact managed by AWS. If your use case involved some tinkering with CF, you might have to opt into the second option: creating your own CF distribution.

To add your api to your CF distribution, you just simply add your api as an origin, while specifying your api’s default endpoint as an origin domain.

If you happen to have a websocket api, make sure to add the following headers in an origin request policy. (for more information, consult the official documentation)
Sec-WebSocket-Key
Sec-WebSocket-Version
Sec-WebSocket-Protocol
Sec-WebSocket-Accept
Sec-WebSocket-Extensions

If you happen to have a lambda authorizer in place for your api, be sure to add the Authorize header in the origin behavior in CF

But that's not all. Remember, your api has a stage defined (often, stages will be the names of the environments, so /dev or /prod). If your api has a /dev stage, then you need to add a /dev to the end of your default endpoint url in CF. This can be done by specifying an origin path of “/dev” to your origin, or by adding a behavior to your cloudfront distribution.

If you decide to go for the latter approach, you specify the path pattern as “/dev”, and associate your api origin to it. Now, when someone accesses the url: your-cloudfront-domain-name/dev, they will be redirected to your-default-endpoint/dev, which will correctly point to your api stage.

Great, now you’re all set! But…what’s a default endpoint anyway?

api default endpoint vs custom domain name

A default endpoint is the url that is created along with your api, and it is how you in fact access said api. The default endpoint is an amazon url, and it looks like this : api-id.execute-api.region.amazonaws.com. And while this endpoint is perfectly usable, it might not work for all use cases. If for some reason you need your api endpoint url to not be an amazon url, you need to use custom domains.

Custom domains are quite simply a custom domain name that points to your api.
To create a custom domain you need to:

Go to ACM and create a certificate for the domain in question
and validate the certificate in route 53
Create the custom domain in the api console.
Under api mapping, link the custom domain to an api, and to a stage in the api
Go to route 53 and create an alias to an api gateway. (the api endpoint to specify can be found in the custom domain page in the api console, under “API Gateway domain name”).

You’re all set, now you can access your api using the custom domain. Don’t forget to disable the default endpoint in the api settings if you’re not using it anymore.

Important note when using CloudFront :
Since the custom domain name already points to a stage, you don't need to add any paths in CloudFront. This means that, unlike with default endpoints, you do not specify an origin path, or a behavior.

Alternatively, if you have to specify a behavior because you have multiple origins, you can add a path to the custom domain name. By adding a “/dev” path to the api custom domain name, you can now keep the CF behavior of “/dev” as well.

But how do you use multiple origins in CloudFront?

CloudFront multi origin default behavior and ordered behavior

CloudFront can handle multiple origins, and they don’t even have to have the same type! As an example, you can have 2 rest api origins, 1 websocket origin and an S3 bucket origin.

They Way CloudFront knows which request to forward to what origin is by using behaviors. In the behaviors you specify a path that, when matched, the request is passed to the origin.

So for example, you can specify a behavior path of “/api” to forward to your api origin, and a default behavior to forward to your S3 website.

In this example, if you go to the url cloudfront-domain-name/api, you’re forwarded to the api. and if you go to any other path, (including cloudfront-domain-name/ base path), you’re forwarded to the S3 website.

If you happen to be using Terraform, you need to note the following:

To specify an ordered behavior, you have to first specify a default behavior
The ordered behaviors are created in the same order as in the code. This has an effect on the precedence (the order in which the behaviors get evaluated by CloudFront)

Now that we know how to add origins, let's look at the specifics of adding an SPA website hosted on S3.

Cloudfront with an SPA

Reaching your S3 static website

Before we go into the specifics of an SPA, let's first discuss how to reach your website hosted on S3.

First you need to create an origin, and put the S3 bucket url an origin domain.

Second, create an origin access control for CloudFront to have permissions to reach the S3 bucket. In terraform, this might look like this



resource "aws_cloudfront_origin_access_control" "CloudFrontOriginAccessControl" { 
  name                              = "CF-access-control"
  description                       = "reach s3 static website"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

Third, make sure the bucket permissions allow CF to access the bucket. The statement for the bucket policy might look like this:

{ "Sid": "AllowCloudFrontServicePrincipal", "Effect": "Allow", "Principal": { "Service": "cloudfront.amazonaws.com" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::your-s3-bucket-name/*", "Condition": { "StringEquals": { "AWS:SourceArn": "arn:aws:cloudfront::account-id:distribution/distribution-id" } } }

Important note about paths: CloudFront always redirects the path as is. So for example, if you have set the behavior path of “/static-website” to redirect to your s3 static website origin, then you have to make sure your website code is in a folder named “static-website“ in S3. Generally, it's best to keep the behavior to “/”, and to keep the website code in the root of your S3 bucket.

Redirecting 403 errors in an SPA

Apparently, CloudFront has some weird behavior with single page application websites, which causes it to throw 403 errors. A solution for this is to redirect any 403 errors to the index page.

If your CF distribution only has one origin, then this is fairly straightforward. You just need to add an error page to your distribution that redirects 403 errors to a 200 (which is the http code for an ‘OK’ response), and to display the index page.

However, when your CF distribution has multiple origins, it gets a little more complicated. This error page that we just created is applied to all origins, which means that in the case where our api origin was to throw a 403, then it would get redirected to the index page of the static website.

If we want the redirects to only apply to one of our origins, then we need to add logic to our Cloudfront. To do That, we need to use lambda@edge. The steps are:

The lambda needs to be created in us-east-1, and here is what the code could look like (Nodejs runtime)



'use strict';
exports.handler = (event, context, callback) => {
       const response = event.Records[0].cf.response;
    if (response.status === '403') {
        response.status = 302;
        response.statusDescription = 'page found';
        /* Drop the body*/
        response.body = '';
        response.headers['location'] = [{ key: 'Location', value: '/index.html' }];
    }
    callback(null, response);
};

in the lambda console, under action, press deploy to lambda@edge. Specify your distribution and and deploy it as an “origin response”
in CF, in your origin behavior for the S3 origin, scroll down. Under origin response, add the lambda arn

You’re all set!

Conclusion

This blog was a non-exhaustive list of the problems you might face when working with CloudFront and api gateway, as well as when working with CloudFront and S3 static websites. My attempts to be brief still resulted in an 8 page blog, and I still feel I can add more detail, so feel free to ask any questions you have in the comments and I will be sure to get to them.

CodePipeline live feedback on slack - with git tags

Ziad Osman — Tue, 30 Aug 2022 10:03:00 +0000

Introduction

This guide is for you if you have ever wanted to get live feedback on how your Pipelines on AWS are going. Additionally, if you keep track of your application version on git tags, I’m going to also show you how to retrieve them in CodeBuild and, as a bonus, how to also send them as slack messages.

Credits

While mathmaticians and phycisits stand on the shoulders of giants, software engineers stand on the shoulders of other software engineers. This guide would not be possible without Wesley’s charles’ contribution, as he made the script for the slack bot. As for how to retrieve git tags from CodeBuild, that as well would not have been possible without the contribution of Timothy Jones who wrote a fantastic script to do so.

Pre requisites

A configured AWS CLI
AWS SAM CLI (we will be deploying our slack bot script with the help of AWS SAM)
A working AWS CodePipeline (including CodeCommit, CodeBuild, and CodeDeploy)
Sufficient credentials for the AWS User

Cloning the slack bot script

The first thing we need to do is clone the slack bot repo to a local machine.

After cloning the repo, we should have these files inside of our directory
│ .gitignore │ build.gif │ LICENSE │ Pipfile │ README.md │ template.yml │ └───src .pylintrc build_info.py message_builder.py notifier.py requirements.txt slack_helper.py

Configuring slack

Creating a slack app

First up we need to create a slack app. For that head on to https://api.slack.com/apps.
Click on Create New App

Click “From Scratch”

For App Name, it doesn’t matter. I picked “Pipeline progress”. For workspace, choose your workspace from the drop-down list.

After your app is created, in the Sidebar, go to OAuth & Permissions.

Now Scroll down to scope and add the following permissions one by one: channels:history, channels:manage, channels:read, chat:write, chat:write.customize, chat:write.public, groups:history, groups:read, im:read, links:write, mpim:read

Now, scroll to the top of the page, generate a Bot User OAuth Token

Note: make note of the OAuth Token, as we will need it soon.

Finally, press the button : install the app to your workspace. This is what the button will look like after a successful installation.

Add app to slack

Go to slack and create a new channel called builds. This is the default channel name that the script takes. If you name your channel anything else, I will show you in a later step where to specify it.

Inside your builds channel, in the dialog box, press /.
This opens up a search box. Search for apps and click on add apps to this channel

On the next page, search for your app by its name and add it to the channel. In my case, the name is Pipeline progress.
This is what it should look like after adding it to the channel.

Deploying the script to AWS

Note: For this step, make sure you have the AWS CLI installed and configured, along with the SAM CLI.
To deploy, open a CMD in the script directory, and first run:
sam build
followed by:
sam deploy --guided
this will open up an interactive prompt.

For stack name, I chose to name it as “aws-codepipeline-slack”.
Stack Name [sam-app]: aws-codepipeline-slack

For region, if the default is fine press enter, otherwise specify the region
AWS Region [eu-west-1]:

For SlackBotUserOAuthAccessToken, paste the OAuth token we created in a previous step. Note that this is a hidden field, meaning that what you paste won’t show on the screen.
Parameter SlackBotUserOAuthAccessToken:

For SlackChannel, if you kept the channel name as build, then just press enter. Otherwise, specify the channel name
Parameter SlackChannel [builds]:

For SlackBotName, this is the name of the bot that will send pipeline updates. I left it at default and pressed enter.
Parameter SlackBotName [PipelineBuildBot]:

For SlackBotIcon, this is the icon of the bot that will send pipeline updates. I left it at default and pressed enter.
Parameter SlackBotIcon [:robot_face:]:

For show resource change, if you select y, it will show you changes of resources and prompt you to accept them before it deploys on each time you run SAM deploy.
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy Confirm changes before deploy [y/N]:

For SAM permissions, keep it at the default Y
#SAM needs permission to be able to create roles to connect to the resources in your template Allow SAM CLI IAM role creation [Y/n]:

For Disable rollback, choose according to your use-case, it won’t matter much if you’re only planning on deploying once.
#Preserves the state of previously provisioned resources when an operation fails Disable rollback [y/N]:

For authorization, select Y
Notifier Function Url may not have authorization defined, Is this okay? [y/N]: Y

For save arguments to config file, select Y. this will make future deployments faster since the default values are saved.
Save arguments to configuration file [Y/n]: Y

Press enter to keep the default config file as samconfig.toml
SAM configuration file [samconfig.toml]:

Press enter to keep the configuration environment as default
SAM configuration environment [default]:

This should be enough to successfully deploy the script. If you selected Y for “Confirm changes before deploy” then you’ll get an additional prompt confirming if you want to deploy. Select Y on it and you’re good to go.

With this, we’re done! you are now able to get updates on the states of your pipelines.
You can check out your new lambda function if you go into your AWS console and go to lambda

Adding support for git tags

If you happen to keep track of your application version using git tags, and you’d like to also get notified of the version that’s getting deployed on slack. Then please read along and follow these steps.
We need to enable function url on our newly deployed lambda function, so that we can call the function from CodeBuild and pass to it the git tag. We will also need to add some permissions to the CodeBuild service role. Finally , we will need to add a bash script to our CodeCommit repo that enables CodeBuild to clone the repo and retrieve the git tag from it. If you want to learn more about why we need to do this workaround instead of just retrieving the git tags from our CodeCommit repo, then I highly suggest reading Timothy Jones’s article, the person behind the bash script we will be using.

Enabling function url

To enable function url on our slack script. Open the function code locally on your favorite IDE. Navigate to template.yml, and uncomment these two lines. (line 28 and 29).

And that’s it! Now follow the same steps above to deploy
sam build
followed by:
sam deploy --guided
Now if you got back to your AWS console, and go to your lambda function, you should be able to find your function url under configurtation -> function url

Note: make note of your function URL as we will need it in the next step

adding the bash script to CodeCommit

as explained, we will be using a bash script in CodeBuild to help us retrieve the git tag.
Add this file to your CodeCommit repo’s base directory. Make sure the file name is: codebuild-git-wrapper.sh

Adding the necessary permissions to CodeBuild

You will need to add 2 policies to your CodeBuild service role.
The first one will enable CodeBuild to perform git pull on the CodeCommit repo. This is the policy template. Make sure to add your repo ARN under Resource.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "codecommit:GitPull", "Resource": "YOUR_REPO_ARN" } ] }

The second policy will give CodeBuild permission to invoke the lambda function url. This is the policy template. Make sure to replace the resource with your lambda function ARN.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "lambda:InvokeFunctionUrl", "Resource": "YOUR_FUNCTION_ARN" } ] }

Modify your buildspec.yml

Finally, we will be adding commands to buildspec.yml to clone the repo, retrieve the git tag, and pass the git tag to our function url.
Go to your buildspec.yml file, and in the first step of the build stage, add the following commands.
build: commands: - echo get version - /bin/bash codebuild-git-wrapper.sh YOUR_REPO_URL YOUR_BRANCH_NAME #get release version from git tag - RELEASE_VERSION=$(git tag --points-at HEAD) #send git version to slack - curl YOUR_FUNCTION_URL/?git-tag=$RELEASE_VERSION -o /dev/null
Make sure to change the following fields:

YOUR_REPO_URL: your code commit repository url. You can retrieve this by going to CodeCommit, and clicking on HTTPS under Clone URL

YOUR_BRANCH_NAME: the name of the branch the CodePipeline gets triggered on. Normally this should be: main. but check your pipeline configuration to be sure.
YOUR_FUNCTION_URL: the lambda function url we created and took note of in a previous step.

Security considerations

As it stands, your function url can be invoked by anyone that has the url and send messages to your slack channel. You can secure your function by either using IAM authentication or putting your lambda function behind API Gateway and using api keys.

Done!

Congratulations! You now have a slack bot that will give you updates on the state of your CodePipeline pipelines. And if you followed the additional steps, it will also send you a message of your release version via git tags.