DEV Community: Ken Choong

How to provision APIgateway StageVariables and Deployment using AWS CDK?

Ken Choong — Sat, 08 May 2021 01:54:11 +0000

In previous post, you know all the basic concept about how APIGateway, Lambda, Lambda version, alias, environment work together.

Now, we start provision the APIgateway using the CDK python.

In this post, we already learned that provision Lambda and Lambda Layer, so we will use back the code here, if you new here, just read the post, cause I mention a lot details about Lambda and Lambda Layer here

Now your MySampleAppStack file look like this:


from aws_cdk.aws_lambda_python import PythonFunction, PythonLayerVersion


class MySampleStack(core.Stack):

        def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
           super().__init__(scope, id, **kwargs)

           # All the aws resources define here

            # Here define a Lambda Layer 
            boto3_lambda_layer = PythonLayerVersion(
               self, 'Boto3LambdaLayer',
               entry='lambda/boto3Folder',
               compatible_runtimes=[lambda_.Runtime.PYTHON_3_8],
               description='Boto3 Library',
               layer_version_name='Whatever name you want'
           )

          my_first_lambda_function=PythonFunction(
                 self, 'MyFirstLambda',
                 entry='lambda/MyFirstLambdaFunction',
                 index='app.py',
                 runtime=lambda_.Runtime.PYTHON_3_8,
                 layers=[boto3_lambda_layer], 
                 handler='MyFirstLambdaHandler',
                timeout=core.Duration.seconds(10)
          )

Alright, the Lambda already done, continue from here, we provision our API first.

First

pip install aws_cdk.aws_apigateway

This install all the necessary package.

As mention in this post, we need 2 API stage nameddev and prod,

dev is for us to develop and testing
prod is for our production user

Then we need to set the StageVaribles for each API stage which is

dev stage have Stage Variables of "lambdaAlias": "dev"
prod stage have stage variables of "lambdaAlias": "prod"

As you can see both API stage also have lambdaAlias, just the value is different. Therefore we will use this value to differentiate the API call from frontend is either developer or production user.

Again, I talk a lot of details here, I strongly recommend you read it, this will help you better understand throughout the whole series.

In order to achieve that, we do the code below, then I explain step by step.

from aws_cdk import (
    aws_lambda as lambda_,
    aws_apigateway as apigateway, #add this
    core
)

class MySampleStack(core.Stack):

        def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
           super().__init__(scope, id, **kwargs)

           # ... all the lambda code above 

           # define a dev deployment
           dev_deployment = apigateway.StageOptions(
              stage_name="dev",
              variables={
                "lambdaAlias": "dev" # this is stage variable 
            })

           # define an API 
           api = apigateway.LambdaRestApi(
               self, "MyAPI", # give it a name
               handler=my_first_lambda_function, # the lambda above
               deploy_options=dev_deployment
           )

           #define a prod deployment
           prod_deployment = apigateway.Deployment(
              self, "ProdDeployment08052021", # change the date here
              api=api,
              retain_deployments=True,
              # Change the description here as well for our reference
              description="Production API deployment as 08-05-2021 Live version"
        )

          # Production stage
          prod_stage = apigateway.Stage(
            self, "ProdStage",
            deployment=prod_deployment,
            stage_name="prod",
            variables={
                "lambdaAlias": "prod"
          })

Some explanation for our code above

Dev deployment
dev_deployment: This is our dev deployment. We define a stage named dev and the stage variable key value "lambdaAlias": "dev"

API
api : This is our main API object, it will use my_first_lambda_function that we define above as handler. Means that, when this API endpoint is hit, it will trigger my_first_lambda_function by default(Lambda is not specially set to a method). We specify deploy_options to dev_deployment which means everytime we deploy our CDK, it will only update the dev stage and effect to prod stage, which is what we want. In other words, dev stage is the default stage of our API.

Prod deployment
prod_deployment : This is our prod stage deployment. The ID ProdDeployment08052021 in the code above, after we make sure everything is ok, and we like to push the stack the our production user, we change the date 08052021, by this, CDK will create a brand new deployment for all the code your define in your stack.

Remember, we only want update dev_deployment for testing, then everything ok then only push to production?

This is exactly what we do, let's say we 3 new method to the API, at the meantime, we just testing in dev_deployment, prod_deployment is not effected. At 11-05-2021 we done all the testing, everything ok, is time to push to product, we change to ProdDeployment11052021, this will create a new deployment for all 3 method we defined before.

Prod stage
prod_stage: This is the production stage of our API. All production user will hit this API stage. It will always use prod_deployment, so everytime a new prod_deployment created, it will use that. Therefore change the date in the ID, then new deployment is created, prod_stage will use that. This is how we update our production API stage.

All right, for now the basic of our API is completed.

Now we can add some API method to our API. We can do it like this:

# ... continue with the code above

store = api.root.add_resource("store") 
store.add_method("GET") # GET /store
store.add_method("POST") # POST /store

profile = api.root.add_resource("profile")
profile.add_method("GET") # GET /profile
profile.add_method("POST") # POST /profile

product = api.root.add_resource("product")
product.add_method("GET") # GET /product
product.add_method("DELETE") # DELETE /product

Alright, so now you have 3 API root, which is store, profile and product with all the different CRUD method.

For now, if you ok for all the method you defined, you should change the date in ProdDeployment<Your date> and description in prod_deployment.

For summarize:
In this tutorial, you learned about:
1) How to create different API stage using CDK
2) How to create different API deployment using CDK
3) How to define stage variable for each API stage using CDK
4) How to define API method using CDK

I hope all this make sense to you. Have a good day.

In next part of the series:
For now, all this 3 API root(store, profile , product), no matter what CRUD method, it also will only trigger my_first_lambda_function that we define above.

In next part, I will make different API root, will trigger different lambda function. This is called Lambda Integration.

I also will tell how we can build our CDK Constructs, what is it and why we should use it.

Before you go, if you like this series or find this useful consider to buy me a coffee 😊🤞 for 5 USD or more.
I will prepare a GitHub repo for this whole tutorial series and arrange into separate commit for each part.
This will only available for my supporter cause I spent a lot of time to prepare this. Anyway, I appreciate you here. Have a good day.

Shout out to me on Twitter: @upupkenchoong

Concept for how to use ApiGateway StageVariables, Lambda Version, Alias, Environment variable, all together

Ken Choong — Sun, 02 May 2021 09:56:28 +0000

In previous partof this series, we know how to set environment variables inside our Lambda.

In this part, I will talk about how we can use Apigateway StageVariables, Lambda Version, Alias, Environment variable all together.

We have to understand how the whole process work first, then only we can create this easily using CDK code.

Let's imagine this scenario

You have an API that will use Lambda function
You make a Lambda function
This function will in production which used by real world user
But you also want to edit this Lambda or continuously develop new stuff for this Lambda
Now what if you have some error when editing the Lambda code?

Then this will effect you Real world user right? The app will crash, then user leave, give you some nasty feedback.
NO GOOD!
Can you imagine how painful and stressful when developing stuff like this?

Then this is why Lambda Version and Alias come in handy.

How Lambda version work?
Imagine this:

You make a Lambda function
By default, every Lambda function have a $Latest version, this is where you keep editing/developing your code, this only use by you.
When everything ok, you publish a new Version for the function, let say version:1. The real world user will access to this version:1.
We will keep repeating the process which is edit code in $Latest, publish a new version every time when everything ok for production.

By doing this, you keep editing code in $Latest, any changes in $Latest will not affect to version:1, so any error will not annoy your real world user.

Now you have 2 version for your Lambda, $Latest and version:1, then how we can set the production user will access version:1 and developer(us) will access $Latest?

This is why Lambda Alias come in handy.

The process of alias and version look like this:

Now we need to create 2 alias for our Lambda named dev and prod
dev alias -> point to -> $Latest version
prod alias -> point to -> version:1(or 2,3,4,5, depends)

Alias is done, but now your frontend application still don't know which Lambda version to access.

This is why APIGateway StageVariables come in play.

Now we can create 2 Stage for your API named dev and prod

This will give us 2 URL which look like this:

https://{APP-ID}.execute-api.{aws-region}.amazonaws.com/dev
https://{APP-ID}.execute-api.{aws-region}.amazonaws.com/prod

Now set a Stage Variables for each API stage like:

dev stage: lambdaAlias:dev
prod stage: lambdaAlias:prod

Then the whole process will look like this:

API dev stage -> Lambda dev alias -> Lambda `$Latest` version

API prod stage -> Lambda prod alias -> Lambda `version:1`

Then inside the Lambda function we get back the StageVariables and use it with the environment variables that we set in previous part:

import boto3
import os

def MyFirstLambdaHandler(event, context):
    client = boto3.client('dynamodb')

    # Get the stageVariable lambdaAlias
    stage = event['stageVariables']['lambdaAlias']

    # Get the environment variable depends on stage variable
    if stage == 'prod':
        tableName = os.environ['production_db_url']

    if stage == 'dev':
        tableName = os.environ['development_db_url']

    # Then here you can access different table depends on environment

Now we connecting the dots:

API dev -> Lambda dev alias -> Lambda version:$Latest -> environment variables 

API prod -> Lambda prod alias -> Lambda version:1 -> environment variables

For summary:
TLDR, everything in development stay in development, which wont affect anything in production.

Now you have some basic concept for Lambda Version, Alias, APIGateway stage variables and environment variables.

You now know how to continuously develop your Lambda, but at the same time wont fu**ked up your whole production app when some error happened when developing.

In next part of the series, I will talk about:

How to create Lambda Version using CDK
How to create Lambda Alias and hit different Lambda Version using CDK
How to create different Deployment stage of API gateway and set stage variables using CDK

In case you cant wait that long, here is my previous work on this topic:

Shout out to me on Twitter: @upupkenchoong

How to set environment variable for Lambda using CDK

Ken Choong — Sat, 01 May 2021 21:46:02 +0000

In the previous post, I show how to make a Lambda function using CDK. In this post, I will show we can set the environment variable, so we can use it inside the Lambda function.

When to use environment variable?
You want to access different value of a variable depending on production or development environment, but this value is very important, you need to keep it secret, not comfortable to keep it in version control(eg. github).

Some example use case:

Access different database depending on production environment
Access different Stripe credential depending on environment.

Steps:
In this example, we access different database URL depending on environment.

Use back the previos Lambda function



my_first_lambda_function=PythonFunction(
    self, 'MyFirstLambda',
    entry='lambda/MyFirstLambdaFunction',
    index='app.py',
    runtime=lambda_.Runtime.PYTHON_3_8,
    layers=[boto3_lambda_layer],
    environment={ # ADD THIS, FILL IT FOR ACTUAL VALUE 
        "production_db_url": "YOUR_ACTUAL_PROD_DB_URL",
        "development_db_url": "YOUR_ACTUAL_DEV_DB_URL"
    },
    handler='MyFirstLambdaHandler',
    timeout=core.Duration.seconds(10)
)

Now, we already define the environment variable, now we need to access this inside our Lambda function.



import boto3
import os

def MyFirstLambdaHandler(event, context):
    client = boto3.client('dynamodb')

    # You get the value back like this
    prod = os.environ['production_db_url']
    dev = os.environ['development_db_url']

    # later you other stuff using the above value

Ok, for now you know how to define the environment variable and use it inside the Lambda function. You can put in any value inside environment, in key value depending on you need.

May be you not sure why you need to set environment variable for now, but is ok, next part of the series, you will connecting the dots.

In next part of the series:
We will talk about version, alias for the Lambda function, APIgateway with stageVariables, and use different environment value depending on the Lambda alias

Shout out to me on Twitter: @upupkenchoong

How to provision Lambda and Lambda Layer using CDK

Ken Choong — Thu, 18 Mar 2021 01:32:18 +0000

Why using CDK?

CDK let you provision all your aws service using the programming language you use.

In this example, I will define a Lambda and Lambda Layers using CDK python.

First, make a new project

cdk init my-sample-app --language python

Then activate the virtual env

.env\Scripts\activate.bat

Set up the project structure



my-sample-app
|-lambda
|--boto3Folder
|--MyFirstLambdaFunction
|---app.py
|--All the lambda folder here
|-MySampleAppStack
|--my_sample_stack.py

What is Lambda Layer?
In my own words:
Lambda Layers is the place you install your library in cloud. Normally you just pip install some-library for your python project right? Lambda Layers is just do that, you install a library inside a local folder, then CDK help you package all file inside that folder and upload it to cloud, then you can use it elsewhere.

First install boto3 library inside lambda/boto3Folder



pip install --target=d:\yourLocalPath\my-sample-app\lambda\boto3Folder boto3

By this, you install boto3 library into my-sample-app/lambda/boto3Folder, then you should see some file and folder inside boto3Folder

Then inside my_sample_stack.py, define the Lambda Layer as follow:



from aws_cdk.aws_lambda_python import PythonFunction, PythonLayerVersion


class MySampleStack(core.Stack):

        def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
           super().__init__(scope, id, **kwargs)

           # All the aws resources define here

            # Here define a Lambda Layer 
            boto3_lambda_layer = PythonLayerVersion(
               self, 'Boto3LambdaLayer',
               entry='lambda/boto3Folder',
               compatible_runtimes=[lambda_.Runtime.PYTHON_3_8],
               description='Boto3 Library',
               layer_version_name='Whatever name you want'
           )

In this few lines of code, you defined a Lambda Layer, tell CDK where is the folder in entry, in our case is lambda/boto3Folder and other necessary information.

So now you have a Lambda layers, now we want to use this Lambda Layers, inside a Lambda function.

In MySampleStack above add this to define a Lambda Function:



my_first_lambda_function=PythonFunction(
         self, 'MyFirstLambda',
         entry='lambda/MyFirstLambdaFunction',
         index='app.py',
         runtime=lambda_.Runtime.PYTHON_3_8,
         layers=[boto3_lambda_layer], # here you use the lambda layer above 
         handler='MyFirstLambdaHandler',
         timeout=core.Duration.seconds(10)
)

This you define a Lambda function that will use the Lambda layers. In your MyFirstLambdaFunction/app.py you can use boto3 as usual.

Inside MyFirstLambdaFunction/app.py



import boto3

def MyFirstLambdaHandler(event, context):
    client = boto3.client('dynamodb')

    # do all the stuff with the client



cdk deploy

All the resources will start provision in the cloud.

Why using Lambda Layers?
1) Avoid install the same library over and over again.
Lets say you have 10 Lambda function all will using 2 same library, then you will need to install 10 separate times inside 10 Lambda. Use Lambda Layers, install 1 time, you it by 10 different Lambda.

2) Reduce size of your Lambda function.
If your Lambda function is more that 10MB, you wont able to edit it in Aws Lambda console.

Summary
In this article, you learned

How to create a Lambda Layer and Lambda function
How to use Lambda Layer inside a Lambda
Why you need to use a Lambda Layer?

I hope you learned something. Have a good day.

In next part of series, I will talk about set up environment variable for a Lambda function using CDK.

Shout out to me on Twitter: @upupkenchoong

[Part 1]What is Amazon EventBridge and How to provision Logging S3 bucket, and bucket policy using AwsCDK

Ken Choong — Fri, 30 Oct 2020 03:29:44 +0000

As mentioned in last part

We already have a S3 bucket which is Amplify set up for us
Now when a file uploaded to that bucket, we need to notify a lambda function, which we will use other Aws service inside it

What is Amazon EventBridge?

Amazon EventBridge is a serverless event bus service that makes it easy to connect your applications with data from a variety of sources. EventBridge was formerly called Amazon CloudWatch Events. (From aws docs)

Based on my understanding:

Anything happen in your resource, will emit an events
events is actually a log files in json format produce by CloudTrail.
EventBridge is the watcher for this events
You tell EventBrigde about what events you want to watch(In our case, when a file uploaded to our S3 bucket)
You tell EventBridge when that events happen what you want to do(In our case, invoke a Lambda Function)

What we need TODO:

Define S3 bucket to store the event (log files produce by CloudTrail)
Define the bucket policy for that S3 bucket (Bucket policy)
Tell EventBridge what pattern of event you want to watch (EventRule)
Tell EventBridge if that pattern of event happened, what you want it to do. (EventTarget)

Ok, how we do that?
In this blog post, we will provisioned all the resource defined above using AWS CDK.

Why we use CDK?
Think about this, AWS have 160+ services. Every time when we need something, we have to define it 1 by 1 in the AWS management console. Very time consuming, need to find service by service in the console.

No, we don't do that. We write some code using AWS CDK. Then we run a short command cdk deploy, then everything get set up in the cloud.

Nice.

Let's get started

First set up the cdk project using Python:

Make a directory

$ mkdir mycdkapp && cd mycdkapp

Set up a CDK app

$ cdk init app --language python

Activate the virtual environment

$ .env\Scripts\activate.bat

Install all the dependencies needed

$ pip install -r requirements.txt

Cool, so we done setting up a CDK app for our project.

Now you will see mycdkapp_stack.py in your directory. This is where we define all the resource need here.

Refer to the existing S3 bucket and create a new Logging bucket for CloudTrail

Install the library needed:

$ pip install aws_cdk.aws_s3

This will install the L1 construct which represent a single type of aws service. All this create by aws, so every time you need a particular service, you can find the library here

In this case, we install the aws_s3 library.

Now paste the code below into mycdkapp_stack.py

from aws_cdk import (
    aws_s3 as s3,
    core
)


class MycdkappStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # refer the existing s3 bucket, substitute <your-existing-s3-bucket-arn> with your bucket arn
        existing_bucket=s3.Bucket.from_bucket_arn(self, "BucketByArn", "arn:aws:s3:::<your-existing-s3-bucket-arn>")

        # this create a new s3 bucket with name s3LoggingBucket
        logging_bucket = s3.Bucket(self,"s3LoggingBucket", bucket_name="s3loggingbucket")

With just 2 lines of code, we already get the existing_bucket and created a new bucket named s3LoggingBucket.

Now run this

cdk synth > template.yaml

You will see a template.yaml in your directory.

Next we need to set up Bucket policy for this Logging bucket.

Make a new directory name contruct inside mycdkapp
Create a new file named logging_policy.py
pip install aws_cdk.aws_iam
Paste this below code into logging_policy.py

from aws_cdk import (
    aws_s3 as s3,
    aws_iam as iam,
    core,
)

class LoggingBucketPolicy(core.Construct):

    def __init__(self,scope: core.Construct, id: str, _bucket: s3.IBucket, **kwargs):
        super().__init__(scope,id, **kwargs)

        cloudtrail_service_principal = iam.ServicePrincipal('cloudtrail.amazonaws.com')
        get_bucket_policy = iam.PolicyStatement(
            actions=["s3:GetBucketAcl"],
            resources=[_bucket.bucket_arn],
            principals=[cloudtrail_service_principal]
            )

        put_object_policy = iam.PolicyStatement(
                actions=['s3:PutObject'],
                resources=[f"arn:aws:s3:::{_bucket.bucket_name}/AWSLogs/{core.Environment.account}/*"],
                principals=[cloudtrail_service_principal]
            )

        _bucket.add_to_resource_policy(get_bucket_policy)
        _bucket.add_to_resource_policy(put_object_policy)

What is this code above do?

We define a Construct by inherit core.Construct
This let us group together all the component which doing the same thing(In this case, this construct will define policy to a bucket)
We will insert s3.IBucket into this construct with variable _bucket
Define 2 iam policy get_object_policy and put_object_policy
Allow both policy can use by cloudtrail.amazonaws.com
Then the policy to the _bucket

Now in our mycdkapp_stack.py add this line:

LoggingBucketPolicy(self, 's3Policy', logging_bucket)

Then run

cdk synth > template.yaml

Now check template.yaml, you will see something like this:

s3LoggingBucketPolicyEF0E9CBA:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket:
        Ref: s3LoggingBucket06C9F6F3
      PolicyDocument:
        Statement:
          - Action: s3:GetBucketAcl
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Resource:

Done.

*Now you successfully: *

Understand what is Amazon Eventbridge and why we use it.
Understand what is Aws CDK and why we use it.
Refer the existing bucket into our newly created cdk app.
Create a new Logging bucket for CloudTrail using CDK
Apply all bucket policy to the new bucket using CDK

*Next we will talking about, *

How to define a Event rule/pattern for Amazon Eventbrige?
When the certain pattern in Eventbridge occurred, what should do next?

All using CDK.

Stay tuned.

[Intro] How to trigger a Lambda function when a file is uploaded to an existing S3 bucket using Amazon EventBridge and AWS CDK

[Part 1]What is Amazon EventBridge and How to provision Logging S3 bucket, and bucket policy using AwsCDK

Coming soon
[Part 2] How to define EventRule, EventTarget for Amazon Eventbridge using CDK.

Follow me on Twitter: @upupkenchoong

My upcoming product(If you interested): @sarah_assistant

[Intro] How to trigger a Lambda function when a file is uploaded to an existing S3 bucket using Amazon EventBridge and AWS CDK

Ken Choong — Thu, 29 Oct 2020 10:44:12 +0000

*In this series of blog post, what we gonna to do is: *

Observe a file is uploaded to an existing S3 bucket using Amazon EventBridge
When Amazon EventBridge notice a file is uploaded to S3 bucket, it will trigger a Lambda function.
Provision our Lambda Function, Amazon EventBridge, Policy and all other resources using AWS CDK.

Scenario

You using Aws Amplify for Android app.

Now you need storage
so you can store the images, video or audio your user upload from you app.

So you do this:

amplify add storage

Answer a few question in Amplify CLI like showing here

Amplify provisioned a S3 bucket for you. Nice and easy. Now you can upload your user file into this bucket.

Follow with this scenario:

Now you have an existing S3 bucket which generated by Aws Amplify
When user upload the file to this bucket, you want to trigger a Lambda function which you can use other AWS service.

For example,

When an image is uploaded to the bucket, you want to use AWS Rekognition to do face recognition
When an audio file uploaded to the bucket, you want to use AWS Transcribe to transcribe the audio file to words.

The problem is how?

Stay Tuned.

[Part 1]What is Amazon EventBridge and How to provision Logging S3 bucket, and bucket policy using AwsCDK

Follow me on Twitter: @upupkenchoong

My upcoming product(If you interested): @sarah_assistant