DEV Community: JP Scriven

Test your AWS Lambda function locally

JP Scriven — Mon, 23 Aug 2021 09:16:36 +0000

Using AWS SAM to run your Lambda function locally

What is AWS SAM?
SAM stands for Serverless Application Model and it is a framework that is used for developing and deploying serverless applications. The idea is that you can create a very simple SAM YAML file which would then generate more complex CloudFormation templates.
All the configuration is done in YAML and it supports anything from CloudFormation like Parameters, Resources, Mapping, Outputs, etc...

SAM gives you the ability to help you run Lambda, API Gateway, DynamoDB locally. Where you can then test your serverless solution before deploying to AWS.

Differences between SAM and CloudFormation:
You must include the Transform Header within your template and it indicates it is a SAM template.

Transform: 'AWS::Serverless-2016-10-31'

Then you have the following types for resources:

AWS::Serverless::Function
AWS::Serverless::Api
AWS::Serverless::SimpleTable

Important commands to know:

sam build: Fetches dependencies and creates local deployment artifacts
sam package: Package and upload to Amazon S3 and generates a CloudFormation template
sam deploy: Deploy to CloudFormation

Simple HelloWorld SAM template to get you started:

First create a folder for your application and then you can use VS Code or any editor of your choice to open that folder.

Next we will run the sam init command, which will initialize a new serverless application with a SAM template.
We will select option 1 - AWS Quick Start Templates to get us started quickly with a template.
Then it will ask us which package type we want and for this example we will select option 1, which will zip the artifact and upload to Amazon S3.
Now we need to select a runtime that we want to use for our serverless function. It supports a wide range of runtimes so you can select any version that you are comfortable with. I will select option 2 here and go with python 3.8.
Choose a Project name or leave blank to accept the default of sam-app.
Next you will be provided with a list of different templates, we want just the basic Hello World example so we will go with option 1 again.
As you can see now it is busy generating the application for us and creating the folder and file structure with the template that we need to get going.

Folder structure:

After it is done generating all the files your folder structure should look similar to this.
Our focus will be on the hello_world folder, which is where our Lambda function code is stored and then the template.yaml file, which is our AWS SAM template.

The template.yaml file:

As we can see at the top the Transform: AWS::Serverless-2016-10-31 is included, which has to be included within the SAM template.
Next under Resources we can see that we have a Lambda function and it's type is AWS::Serverless::Function, which you include with your Lambda function in a SAM template.

Under properties next to CodeUri, we need to specify where our lambda files will be, which in this case is under the folder hello_world. The handler will be the file name where the handler function is located (app.py) and our handler name (lambda_handler), which is why it is added as app.lambda_handler. The runtime is what we selected during the initializing stage so depending on what you choose this might be different for you.
With this Hello World example the also included an event attached to our function, which is an API and it is a get method.

Run this template locally for testing

IMPORTANT: To be able to test this template locally you will need to install Docker first as it will use Docker to run your function locally Download Docker here

Now we can run sam build which will fetch the dependencies and create local deployment artifacts.
Next we want to invoke our function, so we need to run the following command sam local invoke
It will mount our build function to a container and this is where Docker will be used. You don't have to know Docker to be able to use your function locally, you just need to have it installed and running, SAM does the rest for us.
You will receive an output with the following if the function was invoke successfully. Shows you the same status you will receive as when you run your function from the AWS console. From here you can view details as the Duration, the billed duration, Max memory used etc. With the response from our Lambda function.

Conclusion:

This is a quick way to get up and running with AWS SAM so you can test out, edit and deploy functions from your local machine. You can add code to your function and test the changes and play around with some other features of SAM. This quick short template once deployed to AWS will generate a more complex CloudFormation template and then deploy the stack to your AWS account.

What is the Lambda Execution context and why use it?

JP Scriven — Sat, 14 Aug 2021 10:22:30 +0000

What is the AWS Lambda Execution Context?

When your Lambda function gets called, Lambda will then invoke your function in an execution environment, which is an isolated runtime environment. The execution context is a temporary runtime environment that initializes any external dependencies of your lambda code. As your lambda function can be invoke numerous times and scale the execution context is maintained for some time in anticipation of another Lambda function invocation. When that happens it can "reuse" the context to execution time which will save time on your function. The execution context includes a /tmp directory, that provides 512mb of temporary storage for your functions.

What can it be used for?

There are a number of things the execution context can be used for but some great use cases for example is database connection, HTTP clients, SDK clients, etc...
Best practice to not put any connections for example inside your handler, because that will mean that every time your function gets invoked, it will create a new database connection, which will cause problems in the future. You want to be able to create that database connection outside of your handler and then just use that connection within the handler of your Lambda function.

Do not initialize your Database connections like this:

This will cause a new connection to your DB every time your function gets invoked.

Rather create your function as follows:

This is best practice and your Lambda function can now reuse the database connection in your handler, without the need to recreating the database connection if your function gets invoked multiple times in a short period.

/tmp space:

When you need to work with a big file inside of your Lambda function, you can use the /tmp space to temporarily save the file to use, if it's not bigger than 512MB. When the execution context is frozen, the directory content remains, which is helpful for multiple invocations. If you need more permanent objects that will not temporarily be used, then rather use something like AWS S3 for persistence of the object.

Why to use CloudFormation, while learning AWS

JP Scriven — Tue, 03 Aug 2021 08:47:59 +0000

What is CloudFormation?
CloudFormation is a IaaC(Infrastructure as Code), which lets you create, update and destroy AWS resources on the fly. Instead of having to build up server racks on-premises and then install network cables, routers, switches, firewalls, etc... You create a template and define your resources and configuration you need to deploy for your application and/or infrastructure.
For Example you can do the following with a CloudFormation Template:

Create a Security Group
Then, create 2 EC2 instances which will use this security group
Create 2 Elastic IP's that these EC2 will use
Create a bucket in AWS S3
As well as create a load balancer (ex. Application Load Balancer) in front of these machines

CloudFormation will then know exactly in which order to create these resources and create then in the right order for you and with the configuration that you specified in the template.

Why use CloudFormation when starting out with AWS?
When you start out using and exploring AWS, you will mostly use the console, which is really great and a nice way to get familiar with the 100's of AWS services they provide. That is where you should start as well to get a feel for the service, how the configuration works, and how it is setup. Once you done this and comfortable with the console, you will realize that using the console is not a very efficient way to create/update and destroy resources. That is where CloudFormation comes in, specially when you are testing out your application at the start and don't want to pay for resources you are not using anymore. CloudFormation will help you create all the resources you need/define in code and does the rest, while you wait for your resources to be created on the fly. Now you can test you application and when you are done, you can delete your entire stack and all the resources in the template will be destroyed, saving you $$$ on resources not being used anymore. Then the next day you can create the resources again for further testing.

Now for an example:

CloudFormation can be written in YAML and JSON, but as you can see YAML is easier to use and read on CloudFormation templates especially on more advanced templates.

We start with the Resources section, this is required for the template and also where you will declare all your resources to be created.
Now you can declare a name for the resource to be created, which is up to you. Ex. MyInstance for the EC2 and MyEIP for your Elastic IP, SSHSecurityGroup for the Security group 1.
Now we can specify what resource we actually want to create by specifying the Type. Here you can find a whole list of resources you can use: AWS Resources Type
Next, we create properties specific to the resource we want to create, you can also read in the documentation as to which property fields are available to which resource.

Uploading your CloudFormation template

Log into your AWS console. And then search for CloudFormation in the top search bar.
From here we can click on Create stack.
From here select the following: Upload a template file and choose your YAML CloudFormation template file that you want to upload.
Click on Next. You will be asked to provide a Stack name, you can make that anything you want. And then click next and leave everything as default for simplicity. And then your resources will be created.
After everything ran successful you should see a screen similar to this one:
When you are done testing/using the resources, you can click on Delete, which will then remove all the resources the stack just created in the right order.

Conclusion
As you can see with this simple YAML template file we were able to create and destroy the resources easily and we are also to reuse this template multiple times, we can use it within other accounts. This was just a basic high-level overview of CloudFormation and you can do some pretty amazing things with it once you master all the basics it has to offer. You will never want to manually create resources yourself again after using CloudFormation!

#ACloudGuru ElastiCache Challenge

JP Scriven — Sun, 01 Aug 2021 08:56:21 +0000

ACloudGuru ElastiCache Challenge

I decided to take this challenge when I saw that they released it, because I am planning on writing my AWS Developer Associate Certification soon and wanted some practical projects to work on while preparing for the exam. I recently build my NextJS portfolio website which I am hosting as a static website on AWS S3 and then using CloudFront as my distribution and HTTPS provider pointing to my Static website. Then by using AWS Lambda and DynamoDB, I created a function to increment my page visits every time the website gets viewed. I wanted more exposure to CloudFormation templates and decided to do this challenges in CloudFormation templates instead of the AWS console or CLI.

CloudFormation Template

I started with the parameters, to have some flexibility and to focus on referencing them in the template. I decided for testing purposes to use my EC2 keypair and the master database username as parameters. I wanted to keep the master database password a bit secure, so I used SSM secure string to keep my master password. There are many ways one can approach this. In this challenge I wanted to test and see how I could reference a parameter coming from somewhere else in AWS instead of CloudFormation.

Next I started working on the Security Groups, I allowed HTTP and the flask port to my EC2 server to be able to browse to my app from my local computer. Then with the RDS and ElastiCache security groups I referenced the SG of my EC2 server, so that only my EC2 server can communicate to those services.

The next part creates a RDS PostgreSQL database which will be in the free tier if you keep the setup as is. Then for the master password I decided to use SSM SecureString to store my Database password, encryption will be handled by AWS KMS, but I wanted to test out as well how to incorporate the SSM store within my template, as there are a few ways you can configure this for secure password storage. For the MasterUsername we the intrinsic function !Ref which returns the value of our RDSDatabaseUsername parameter.

The next part we will have a look at our EC2 configuration. It is also setup to be within the free tier in AWS. We also use the intrinsic function !Ref to get our KeyName value and also our Security Group for the EC2. In the user data section we use a bootstrap script which will only run once when our EC2 gets created, this is to setup the EC2 the way we need it to run without going in after and setup everything manually. You can basically setup anything on the UserData section to config your EC2 the way you need it to be.

Here we will create our Redis ElastiCache which we will use in our Flask app to use to cache our results. This will also be within the free tier of AWS.

The next section we just output our Public DNS name of our EC2 and then as well our URL of our RDS PostgreSQL database that we can use.

Flask Application

Now that our service on AWS is up and configured, we can edit our Flask app so that we check if the results are cached in our ElastiCache and if it is it will return the cached result, which will return a response quickly and if it doesn't exist in the cache then we query the RDS server and will return a slower response.

Here we visit our web app for the first time which would have no results in the ElastiCache and would need to query our RDS database and will return a slower response.

Now that result will be saved in our cache, which means if we refresh our page again the next few times we will have a very quick response time and quick page load.

As you can see our next 3 page loads, loaded much faster than our first time when we loaded the page. The cache TTL is configured to last 20sec(for development purposes), after that it will query our database again and save the new results to cache.

Conclusion

I learned a great deal on this challenge and also read a great deal of the AWS whitepapers and resources for research into CloudFormation and ElastiCache especially. These challenges by ACloudGuru has been a great help in challenging myself to build and use AWS. The resume challenge and then this one was fun in helping me prepare for the AWS Developer Associate exam and looking forward to seeing them bring out more AWS challenges going forward.

Setup AWS S3 Buckets to Host Your Django Static Files

JP Scriven — Thu, 18 Mar 2021 05:34:49 +0000

Setup AWS S3 Buckets to Host Your Django Static Files

When thinking about deploying your static files for your Django Web Application, there are a few options to choose from:

Amazon S3
Apache Liquid
Azure Storage
Digital Ocean
Dropbox
FTP
Google Cloud Storage
SFTP

Today we will be looking at Amazon S3 and how to configure it to be ready to host your Static Files from Django.

First you will have to go create an AWS account, you can signup for the free tier account and it is a great place to start playing around and testing out your solutions. You will be asked to provide a credit card but you will not be charged until you have used up the free tier resources, so have a look at your billing dashboard and keep an eye out that your usage does not reach your free tier limit.

Once you are registered and logged into your AWS management console, go to the very top and type in S3 and select the first option, this will take you to the S3 service.

You can click on create a bucket, which will take you to the configuration page. From here:

Choose a name for your bucket, this name will have to be unique so you might have to try a few options.
Select your region you want your S3 bucket to be hosted in.
Uncheck the option that states, Block all public access.
The rest we can leave at defaults for now.

Now our Bucket is created but will still not be able to work fully with our Django app. We have to create a bucket policy first to allow a public read and getObject to our S3 bucket.

Now click on you bucket name and then navigate to the permission tab.

Once there you can scroll down and edit the bucket policy with the following:

Save your configuration and now at the top under your bucket name you will see that it states Publicly accessible in red, which means now we are able to access that content in the bucket outside of our AWS environment, which means our Django app will now be able to read the objects within our buckets.

How to pass your AWS Cloud Practitioner Certification

JP Scriven — Sun, 28 Feb 2021 17:47:31 +0000

Why to get your AWS Cloud Practitioner and How to Pass it the first time.

I started looking into AWS about just over a year ago and decided to get AWS Cloud certified. They have made some great services and serverless technologies in the last few years and continuing to do so. They also have some great AI/ML services for you to get started with. I was between going immediately for the AWS Developer/SysOp Associate directly and/or to do the AWS Cloud Practitioner first.

Think the AWS Cloud Practitioner is great for everyone, even if you are already a cloud engineer. You don't have to study for this exam too long but gives you a great overview of what AWS can offer and as well goes into Billing and Costs and their different plans. You will get more comfortable with AWS and this will help you when you go for your associates certificates.

If you have been working on AWS for a long time already and now looking to get AWS certified I would suggest then to go directly for the Associate levels instead of the Cloud Practitioner.

How long should I study for the AWS Cloud Practitioner?

I would suggest anywhere between 2-4 weeks depending on how you're on time and how many hours you can commit. Try to get real in-depth knowledge of all the services within the Exam Guideline
Specially get comfortable with EC2, S3, RDS, DynamoDB, CloudWatch, Billing and Costs, RI vs Savings Plans, Shared Responsibility Model.

Resources to use for studying:

Udemy
Udemy Practice Test
AWS Whitepapers as listed on the AWS Cloud Practitioner site
This should be enough to pass the certificate.

Build some real-world projects:

I started by building some simple projects on AWS to get a more hands-on experience and then went on building a little bit more complex projects. This was great to answer some questions on the exam you might not have been 100% sure on. Spend allot of time understanding EC2 instances and S3 Buckets, after you get a better understanding of them go over to the database side of AWS which is RDS, DynamoDB, Redshift, EMR etc. To get a high level of the functions of each of these. Spend some time reading the docs on Billing and Costs as this can be tricky to simulate on your account if you do not want to pay a monthly fee for services. The docs and whitepapers are more than enough to pass this exam.

Good luck!