DEV Community: desawsume

Create AWS Nested Stack

desawsume — Thu, 24 Nov 2022 06:38:03 +0000

Alternative way of working Lambda layer in CDK

desawsume — Wed, 03 Aug 2022 02:09:00 +0000

What if your Lambda functions doesn't have the module or packages in AWS, and you wish to run this Lambda in the Cloud

Simple answer is using Lambda Layer.

CDK has the native way of handling it like below

layer = aws_lambda.LayerVersion(self, 'lambda-layer',
                code = code,
                layer_version_name='layer-bundle',
                compatible_runtimes=[
                    aws_lambda.Runtime.PYTHON_3_7,
                    aws_lambda.Runtime.PYTHON_3_8,
                    aws_lambda.Runtime.PYTHON_3_9
                ]
            )

The other way of do this is:

Deploy Python Lambda functions with .zip file archives.

- cd lambda

- python3 -m venv .venv-zip

- source .venv-zip/bin/activate

- pip install -r requirements.txt

- deactivate

- cd .venv-zip/lib/python3.9/site-packages

- zip -r ../../../../lambda_handler.zip .

- cd ../../../../

- zip -g lambda_handler.zip *.py requirements.txt

using the from_asset() method point to your zip file location

Get the python version and clean up the Mac residue python installed version

desawsume — Fri, 29 Jul 2022 08:51:00 +0000

we might have situation when the python install in below path
Library/Frameworks/Python.framework

And we will go through the step to clean up and remove the path as well as install the brew version

First, let's uninstall previous Python versions:

rm -rf /Library/Frameworks/Python.framework
rm -rf /usr/local/bin/python3

Second, remove the previous frameworks from the $PATH variable:

it can either on
~/.bash_profile
~/.zshrc

brew install python3

Pyenv failed to install specific version on Mac M1

desawsume — Thu, 21 Jul 2022 23:43:47 +0000

Cross-account S3 Origin Setup

desawsume — Tue, 12 Jul 2022 03:45:58 +0000

It is possible to setup cross account S3 Bucket for CloudFront

Let's take a look of the solution first.

Master account - This is where you created your CDN

Sub-account - This is where you have the S3

Sub-account

You will need to setup s3 bucket policy to allow OAI access from the Master account

S3 bucket policy looks like below

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <OAI ID>"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucket-name-of-the-sub-account>/*"
        }
    ]
}

Master Account

Create a S3 Origin using the S3 endpoint

Format:

<bucket-name-from-the-sub-account>.s3.<aws-region>.amazonaws.com

abc is the bucket name from the sub-account

Select the Origin access identity from the Master account.

Last but not least, Create a path pattern that suit your s3 origin behaviour if you have multiple origins

Identifying HTTP request

desawsume — Mon, 04 Jul 2022 22:29:07 +0000

http://www.domain.com:12345/page101?t=win&s=chess#para5 these would be:

Scheme: http
Authority: www.domain.com:12345
User Information: not present
Host: www.domain.com
Port: 12345
Path: /page101
Query String: t=win&s=chess
Fragment: para5

QueryString also called URL parameters or URL query parameters

It refers to the portion of the URL that comes after a question mark (?) and consist of a key and a value, separated by an equal sign (=), multiple parameters are each then separated by an ampersand (&)

? : query string begins
= : value separator
& : parameter separator

for example:

https//www.domain.com/page?key1=value1&key2=value2

Making use of Lambda@Edge outside of us-east-1 in CDK

desawsume — Mon, 04 Jul 2022 04:58:06 +0000

If you are deploying Lambda@Edge functions and CloudFront at us-east-1 it is straightforward to get all the resources stood up properly

Background

ap-southeast-2 is my home region,
us-east-1 is where I deploy CDN and other Global resources.

CloudFront got deployed from a stack in ap-southeast-2, as my OAI and S3 will need to be in my home region (Even though my CDN comes from the ap-southeast stack, but it will get stay in N.Virginia region without problem)

Note: Lambda@Edge functions must be created in the us-east-1 region, regardless of the region of the CloudFront distribution and stack

When I look closer, if you deploy the Edge function in us-east-1, you can take advantage of using the normal lambda construct,

func = lambda_.Function(self, "EdgeFunction",
    runtime=lambda_.Runtime.NODEJS_12_X,
    handler="index.handler",
    code=lambda_.Code.from_asset(<path_of_your_lambda@edge function>)
)

If you have the stack structures like me, you would need to upload ARN or relevant reference to the SSM and let the resources passing from region to region

Thumb up to below method

EdgeFunction

edge_lambda_function = cloudfront.experimental.EdgeFunction(self, "edge_lambda_function",
    runtime=lambda_.Runtime.NODEJS_12_X,
    handler="index.handler",
    code=lambda_.Code.from_asset(<path_of_your_lambda@edge function>),
)

So now, on the same stack, you can reference it in the main menthod.

cdn_distribution = cloudfront.Distribution(
self, 
"cdn_distribution",
default_behavior=cloudfront.BehaviorOptions(origin=my_origin),
    additional_behaviors={
        "images/*": cloudfront.BehaviorOptions(
            origin=my_origin,
            edge_lambdas=[cloudfront.EdgeLambda(
                function_version=edge_lambda_function.current_version,
                event_type=cloudfront.LambdaEdgeEventType.VIEWER_REQUEST,
                include_body=True
            )
            ]
        )
    }
)

Problem

When using the CloudFront EdgeFunction with a role specified attached to the function,

edge_lambda_role = iam.Role(
               self,
                "edge-lambda-iam-role",
             assumed_by=iam.CompositePrincipal(
                    iam.ServicePrincipal('lambda.amazonaws.com'),
                    iam.ServicePrincipal('edgelambda.amazonaws.com')
                ),
                managed_policies=[
                    iam.ManagedPolicy.from_aws_managed_policy_name('service-role/AWSLambdaBasicExecutionRole')
                ]
            )

Error -

Adding this dependency ("edge-lambda-stack-xxxxxxxxxxxxxxxxxxxxxxxxx/edge-fucntion/Resource" depends on "cloudfront/edge-iam-role/Resource") would create a cyclic reference.

This is the most tricky part, and took me a while to understand what's going on.

Solution

After some research, someone mentioned:

The created us-east-1 stack must have a dependency on the "main" stack, as the main stack can't be created first; doing so would mean the function ARN isn't available to read and that the Distribution would not deploy correctly. Creating the role in the main stack certainly leads to a circular dependency.

So I have to remove it, and let the Lambda@Edge construct to create it on the fly (I think it does better than the IAM role we defined, I will explain that in a bit)

Under the hood, the EdgeFunction method does this:

create the IAM role, which will have enough permission to do all the magic but it is not permissive

a custom resource also upload the version arn to param store (CloudFront want the version arn instead of function arn)

It took me 2 hours to get this all going, enjoy finding the solution and also hope this you find helpful as well

How to shrink the EBS Volume

desawsume — Sat, 02 Jul 2022 22:54:54 +0000

## TL;DR

Stop the target EC2 instance (Target machine is Windows server 2019)
Snapshot the root EBS volume.
Create a brand new EBS volume and make sure it is at the same AZ.
Detach the source volume from the EC2 and attach it as non-root volume to the worker EC2 below
Launch a new EC2 worker instance in the same availability zone as the target instance. Here I launched a small Amazon Linux 2 box to do this demo

Main steps

Fetch the epel to install ntfsprogs

amazon-linux-extras install epel -y

yum install ntfsprogs

List the attached volume

fdisk -l
Note down the target volume device name

source volume - /dev/xvdf (The original volume)

target volume - /dev/xvdg (The smaller size of the EBS volume)

Dump current source disk partition info to a file

sfdisk -d /dev/xvdf > xvdf.info

Find out the minimum size the filesystem can be reduced to

ntfsresize --info /dev/xvdf
ntfsresize --info /dev/xvdf1

Resize the filesystem to the suggested number. Enter ‘y’ when prompted to proceed.

In this example it is 30Gb
ntfsresize -s 30000M /dev/xvdf

Install pv, which is a tool for monitoring the progress of data through a pipeline.

yum install pv

Find out the total MB we need to copy from the source disk to the new smaller disk. Here we need to cover the reserved sectors (2048 sectors, or 1MB, from step 6 above) at the front of the disk, to the end of the filesystem.

echo $((30000*2048))

alway + 1 of the total of your drive

Block copy from xvdf to xvdg, with 1MB block size and total count
dd bs=1M if=/dev/xvdf count=30001 | pv -s 30001m | dd of=/dev/xvdg bs=1M

Dump the new target disk xvdg partition info to a file. Edit the file by updating the xvdg1 size to the new size in sector

sfdisk -d /dev/xvdg > sfdisk-d.xvdg

Detach the xvdg and attach it as root back to the original instance

Wait for few mins and check!

How to upgrade AWS CDK stacks from CDKv1 to v2

desawsume — Sat, 02 Jul 2022 10:09:20 +0000

Background

On June 1, 2023, AWS CDK version 1 will reach the end-of-support. AWS CDK Maintenance Policy

The Old Way

With CDKv1, we defined our CDK app packages through a requirements.txt file

aws-cdk.core=={cdk_version1}",
aws-cdk.aws_codebuild=={cdk_version1}",
aws-cdk.aws_cloudtrail=={cdk_version1}",
aws-cdk.aws_codepipeline=={cdk_version1}",
aws-cdk.aws_codepipeline_actions=={cdk_version1}",
aws-cdk.aws_events=={cdk_version1}",
aws-cdk.aws_events_targets=={cdk_version1}",
aws-cdk.aws_iam=={cdk_version1}",
aws-cdk.aws_lambda_event_sources=={cdk_version1}",
aws-cdk.aws_lambda=={cdk_version1}",
aws-cdk.aws_s3=={cdk_version1}",
aws-cdk.aws_kms=={cdk_version1}
aws-cdk.aws-ec2==cdk_version1
aws-cdk.aws-elasticloadbalancingv2==cdk_version1
aws-cdk.aws-lambda==cdk_version1

boto3
pytest
-e .

It could potentially be blocked by the package manager and we might get overwhelmed by the aws-cdk packages name.

CDK version 2

Now, CDK and all stable constructs are now combined in 1 package/module. Experimental modules still need to be installed one by one.

It makes things much easier with aws-cdk-lib

Preparation

To make changes easier, I would create a virtual environment isolated it from my CDKv1, because most my CDK apps are written in python.

python3 -m venv .cdkv2

And activate your new virtual environment and run pip install to install the cdk v2 lib

source .cdkv2/bin/activate

Note: Move all the old aws-cdk v1 package to another txt file, for example - requirements-dev.txt

The requirement.txt look similar to below depending on your CDK version, use cdk --version to find out.

aws-cdk-lib==2.15.0
constructs>=10.0.0,<11.0.0

The cdk.json needs to be adjusted as well, some of the options in CDK v1 are now deprecated, which also simplify a lot in the cdk.json file

overall, it looks something like below:

{
  "app": "python app.py",
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": false,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": false,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": false,
    "@aws-cdk/core:stackRelativeExports": false,
  }
}

## Change the imports

All the mature and stable release are in the aws-cdk-lib. Most cdk apps only need to change import statements

Example that I have -

from aws_cdk import core
from aws_cdk.core import Tags
import aws_cdk.aws_iam as iam
import aws_cdk.aws_lambda as _lambda

Change to

from constructs import Construct

from aws_cdk import (
    aws_iam as iam,
    aws_lambda as _lambda,
    Stack,
    CfnOutput,
    Tags
)

now re-bootstrap the env to the account by simply run

cdk bootrap

Lukcy that I don't have any error, I saw other people have issue with cdk bootrap version conflict.