DEV Community: Simon Green

Integrating 'Session Counts'

Simon Green — Wed, 27 Mar 2024 13:35:32 +0000

Current status

Shortly after completing the Cloud Resume Challenge I decided to revisit it and choose a selection of the suggested modifications to continue adding development features into the application, ie. resume page.

One of the key features of the project is the 'visitor counter', an integer that is retrieved from a DynamoDB table through a number of processes, this number is simply incrementing by 1 each time the page is reloaded or refreshed.

Counting distinct sessions

I wanted to update this feature so that it counts the perceived browser sessions on the page, rather than the page 'load count'. To do this I needed to get and pass the ip_address and user_agent retrieved by API Gateway into the Lambda/Python function for evaluation.

The Python function needs to:

Concatenate these two string values
Add them to the database
Count the distinct values of these concatenated strings and
Return the unique value as the session_count

Counting distinct active sessions

Using this data and taking it a step further, I also wanted to count the active sessions within a 1 hour window.

To do this, together with the session_id I would need to add a timestamps for first_accessed (using 'datetime.now()' from Python) and also calculate the last_accessed timestamp value of the session_id.

This involved 3 key steps and some pretty interesting logic that I have summarised below:

1) Check if the unique session_id already exists in the database:

If no >> add to the database a new row containing 1) session_id, 2) first_accessed timestamp as 'now', and 3) last_accessed timestamp also as 'now',

If yes >> update the 'last_accessed' timestamp as current time/date

2) Distinct sessions: Count the number of distinct session_ids in the database:

3) Active sessions: Count the number of sessions within the last hour:

-- Scan the table, filtering out rows where the last_accessed timestamp is greater than '1 hour ago'

-- Retrieve and count all remaining session_ids

The Python code for this can be seen below:

# SESSION COUNT ADDED - NON-GDPR COMPLIANT

import json
import boto3
from datetime import datetime, timedelta
import hashlib

# Initialize DynamoDB client
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('session_count_hash_table')

def lambda_handler(event, context):

    ################### CREATE UNIQUE IDENTIFIER ###################

    ip_address = event.get('requestContext', {}).get('identity', {}).get('sourceIp', 'Unknown IP')
    user_agent = event.get('requestContext', {}).get('identity', {}).get('userAgent', 'Unknown UserAgent')

    # Generate a unique session ID based on IP address and user agent
    session_id = f"{ip_address}-{user_agent}"


    ################### ACTIVE SESSION COUNT ###################

    # Check if the session exists in DynamoDB
    response = table.get_item(Key={'session_id': session_id}) # sends a request to DynamoDB to retrieve the item that matches the specified key
    session_data = response.get('Item')

    if session_data:
        # Update the session timestamp
        table.update_item(
            Key={'session_id': session_id},
            UpdateExpression='SET last_accessed = :val', # sets contain only unique elements
            ExpressionAttributeValues={':val': datetime.now().isoformat()}
        )
    else:
        # Create a new session entry
        table.put_item(
            Item={
                'session_id': session_id,
                'first_accessed': datetime.now().isoformat(),
                'last_accessed': datetime.now().isoformat()
            }
        )

    # Count the number of active sessions within the last hour
    hour_ago = datetime.now() - timedelta(hours=1)
    response_active = table.scan(
        FilterExpression='last_accessed > :val',
        ExpressionAttributeValues={':val': hour_ago.isoformat()}
    )
    active_sessions = len(response_active['Items'])


    ################### DISTINCT SESSION COUNT ###################

    # Retrieve all session IDs from the database
    response_all = table.scan()
    session_ids = set([item['session_id'] for item in response_all['Items']])

    # Calculate the distinct count of session IDs
    unique_session_count = len(session_ids)


    ################### WHAT TO RETURN (json string)  ###################

    response = {
        "statusCode": 200,
        # HTTP headers that will be included in the response sent back to the client
        "headers": {
            "Content-Type": "application/json",
            "Access-Control-Allow-Origin": "*",  # Allows requests from any origin
            "Access-Control-Allow-Credentials": "true",  # Required for cookies, authorization headers with HTTPS 
            "Access-Control-Allow-Methods": "OPTIONS,GET,PUT,POST,DELETE",  # Allowed request methods
            "Access-Control-Allow-Headers": "Content-Type,Authorization",  # Allowed request headers
        },
        "body": json.dumps({  # converts a Python dictionary to a JSON-formatted string
            # Your response body
            'message': 'Session counts updated successfully',
            'unique_session_count': unique_session_count,
            'active_sessions': active_sessions   
        }),
    }

    return response

After some configuration and testing this worked well and updated a new DynamoDB table (session_count_table) updating the relevant columns where and when needed, as shown below.

Making it GDPR compliant

As an IP address can potentially be used for location tracking, it is possible that collecting the IP address together with the browser user agent data could be a breach of GDPR.

GDPR (General Data Protection Regulation) is a European Union regulation on information privacy in the European Union and the European Economic Area. The GDPR is an important component of EU privacy law and human rights law, in particular Article 8 of the Charter of Fundamental Rights of the European Union.

To eliminate this issue I replaced the session_id string with a consistent hash, (ie. for each distinct session_id, a constant hash is generated) removing the need to store IP addresses.

The session_hash is created using the SHA-256 cryptographic hash function provided by the hashlib module and provides a hexadecimal string as the hash, which will be consistent for the same input.

The important line of Python code added is to create the session_hash, as seen below:

import hashlib ~ session_hash = hashlib.sha256(session_id.encode()).hexdigest()

And from that point on replace session_id with session_hash so as to perform calculations on the distinct hash values, as seen below.

# SESSION COUNT ADDED - GDPR COMPLIANT

import json
import boto3
from datetime import datetime, timedelta
import hashlib

# Initialize DynamoDB client
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('session_count_hash_table')

def lambda_handler(event, context):

    ################### CREATE UNIQUE IDENTIFIER ###################

    ip_address = event.get('requestContext', {}).get('identity', {}).get('sourceIp', 'Unknown IP')
    user_agent = event.get('requestContext', {}).get('identity', {}).get('userAgent', 'Unknown UserAgent')

    # Generate a unique session ID based on IP address and user agent
    session_id = f"{ip_address}-{user_agent}"

    # Convert the session id into a consistant sha256 hash, and use session_hash from here on
    session_hash = hashlib.sha256(session_id.encode()).hexdigest()


    ################### ACTIVE SESSION COUNT ###################

    # Check if the session exists in DynamoDB
    response = table.get_item(Key={'session_hash': session_hash}) # sends a request to DynamoDB to retrieve the item that matches the specified key
    session_data = response.get('Item')

    if session_data:
        # Update the session timestamp
        table.update_item(
            Key={'session_hash': session_hash},
            UpdateExpression='SET last_accessed = :val',
            ExpressionAttributeValues={':val': datetime.now().isoformat()}
        )
    else:
        # Create a new session entry
        table.put_item(
            Item={
                'session_hash': session_hash,
                'first_accessed': datetime.now().isoformat(),
                'last_accessed': datetime.now().isoformat()
            }
        )

    # Count the number of active sessions within the last hour
    hour_ago = datetime.now() - timedelta(hours=1)
    response_active = table.scan(
        FilterExpression='last_accessed > :val',
        ExpressionAttributeValues={':val': hour_ago.isoformat()}
    )
    active_sessions = len(response_active['Items'])


    ################### DISTINCT SESSION COUNT ###################

    # Retrieve all session IDs from the database
    response_all = table.scan()
    session_ids = set([item['session_hash'] for item in response_all['Items']])

    # Calculate the distinct count of session IDs
    unique_session_count = len(session_ids)


    ################### WHAT TO RETURN (json string)  ###################

    response = {
        "statusCode": 200,
        # HTTP headers that will be included in the response sent back to the client
        "headers": {
            "Content-Type": "application/json",
            "Access-Control-Allow-Origin": "*",  # Allows requests from any origin
            "Access-Control-Allow-Credentials": "true",  # Required for cookies, authorization headers with HTTPS 
            "Access-Control-Allow-Methods": "OPTIONS,GET,PUT,POST,DELETE",  # Allowed request methods
            "Access-Control-Allow-Headers": "Content-Type,Authorization",  # Allowed request headers
        },
        "body": json.dumps({  # converts a Python dictionary to a JSON-formatted string
            # Your response body
            'message': 'Session counts updated successfully',
            'unique_session_count': unique_session_count,
            'active_sessions': active_sessions   
        }),
    }

    return response

An example of the session_count_hash_table can be seen below:

To complete this part of the project I needed to configure the following:

Update AWS IAM settings to allow Lambda to connect with the new DynamoDB table
Add a new endpoint to the existing API Gateway to connect with the new Lambda service for session counts
Update Terraform API Gateway and Lambda configurations to include these updates as Infrastructure as Code
Update the frontend webpage to invoke and receive data from API Gateway and display that data on the page.

The frontend update can be seen below:

With this update in place, I can now see additional metrics that loosely relate to visitors to the webpage and have gained additional experience in implementing additional features to an existing application.

Part 4b: Problem solving and debugging

Simon Green — Fri, 16 Feb 2024 11:30:41 +0000

Included in this chapter...

Objective
How I resolve technical issues...
Issue 1: API Gateway Internal Server Error
Issue 2: Malformed Lambda proxy response
Issue 3: Git push, file exceeding limit
Issue 4: GitLab CI Variables
Issue 5: CloudFront - Failed to contact the origin
Issue 6: API Gateway: Missing Authentication Token
Issue 7: CSS intermittently not loading
Conclusion

Objective

This post is the second part of my previous post (Part 4a: Terraform IaC, GitLab CI and automated testing) where I explain seven key technical issues I faced during the development of the AWS / Terraform IaC part of the challenge and I explain how I was able to resolve each.

My process for resolving technical issues...

Issues can arise when adding a new implementation or as a result of making changes to an existing configuration.

My approach to handling an issue goes something like this:

Gather the available error logs
Understand the information flow and where in that flow the error is being generated
Research
Testing
Documenting

Research can take the form of reading the output logs from various sources (or creating them if necessary), reading official documentation, blogs, community forums etc. The latter options often times can be old and it’s good to understand that the technology in question is changing fast, however they can often times provide solutions or clues that can be investigated further.

When resolving issues, I typically make one change at a time and test frequently to help identify the cause of the issue, and take steps to be able to roll back to a previous state if necessary.

The research and testing steps are typically a continuous cycle until the problem is resolved. Sometimes breaking the issue down into smaller elements for testing works, sometimes I find that a different approach is needed to achieve the required outcome, but "all" issues can be fixed with persistence and patience.

With that said, here are some issues I experienced...

Issue 1: API Gateway Internal Server Error

With the API Gateway and Lambda created as IaC, I tried to invoke the Lambda function from the API Gateway console using the ‘Test Resource’ feature and was continuously receiving a “500 Internal Server Error” response code, as seen below:

The logs indicate there are Invalid permissions on the Lambda function, indicating a possible IAM issue to resolve in the Terraform config.

Issue 1: Solution

From the Terraform documentation:

execution_arn - Execution ARN part to be used in lambda_permission's source_arn when allowing API Gateway to invoke a Lambda function, e.g., arn:aws:execute-api:eu-west-2:123456789012:z4675bid1j, which can be concatenated with allowed stage, method and resource path.

After some testing I was able to resolve this issue by simply concatenating a ‘/*’ to the end of the API Gateway ARN output value, with this change it now allows invocation from any API Gateway stage, method or resource path.

Before:

output "api_gateway_source_arn" {
 value = aws_api_gateway_rest_api.api_gateway_tf.execution_arn
}

Corrected:

output "api_gateway_source_arn" {
  value = join("",[aws_api_gateway_rest_api.api_gateway_tf.execution_arn,"/*"])
}

With this updated API Gateway could successfully invoke the Lambda function however this then led to issue 2, detailed below.

Issue 2: Malformed Lambda proxy response

With this issue again when trying to invoke the Lambda function I was receiving a 502 Bad Gateway, configuration error due to a *Malformed Lambda proxy response.

From the logs I could see that the request was being sent successfully (as it returned the ‘count’ value) however it was failing at the response.

Issue 2: Solution

This took quite a bit of trial and error to find the solution but I managed to resolve this by updating the api_gateway.tf file and under the aws_api_gateway_integration resource by changing the ‘type’ argument/configuration from type = ‘AWS_PROXY’ (for Lambda proxy integration) to type = ‘AWS’ (for AWS services).

With this updated this returns the updated visitor count without any error.

resource "aws_api_gateway_integration" "integration" {
  rest_api_id                   = aws_api_gateway_rest_api.api_gateway_tf.id
  resource_id                   = aws_api_gateway_resource.resource.id
  http_method                   = aws_api_gateway_method.method.http_method
  integration_http_method       = "POST"
  # type                        = "AWS_PROXY"       # <--- OLD
  type                          = "AWS"         # <--- UPDATED
  uri                           = var.lambda_function_arn
}

NOTE: I have later updated this config to use integration type "AWS_PROXY", to resolve the issue I needed to update the lambda/Python response to match the requirements of the API Gateway integration requirements. I now understand that this is an ideal way to receive a response in API Gateway when invoking Lambda.

Issue 3: Git push, file exceeding limit

For Terraform to work correctly with AWS it downloads in the background an AWS Provider Binary file (409MB) to the local repo. When pushing the latest updates to GitLab this file exceeds the upper limit of what you can upload.

Issue 3: Solution

Using the Terminal I searched for large files within the project folder using this command:

find cloud-resume-challenge -type f -size +200M

I then added these files to the .gitignore file, essentially preventing the upload of the specified files.

If you were to clone the repo onto another machine, these binary files would automatically be downloaded anyway.

Issue 4: GitLab CI Variables

Whilst configuring the GitLab automation pipeline in the ´.gitlab-ci.yml´ file, within the GitLab CI/CD settings → Variables settings I had configured the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY however I was continually seeing errors like:

upload failed: ./foo.txt to s3://gitlab-test-bucket-sg/foo.txt An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.

…or…

upload failed: ./foo.txt to s3://gitlab-test-bucket-sg/foo.txt Unable to locate credentials

Issue 4: Solution

I found the solution was simply to uncheck the “Protect Variable” option within the GitLab UI in the Variable configuration. The documentation states:

“Protect variable Optional. If selected, the variable is only available in pipelines that run on protected branches or protected tags.”

With the "Protect variable" option unchecked the pipeline worked 🙂

Issue 5: CloudFront - Failed to contact the origin

To help test the functionality of the CloudFront configuration, The CloudFront console provides a 'Distribution domain name' URL that should, in my case open the 'index.html' file in the browser.

In my case, this Terraform created CloudFront service was opening an error page, indicating that it has Failed to contact the origin.

Issue 5: Solution

The resolution to this issue was a two-fold solution.

For the first I found there was a syntax error in the formatting of the local value in cloudfront.tf file, before the declared region variable I had used a hyphen, whereas this should be formatted with a period, see below for the example.

Issue 1 of 2

locals {
    # Before and after:
    # s3_domain_name = "${var.bucket_name}.s3-website-${var.region}.amazonaws.com"
    s3_domain_name = "${var.bucket_name}.s3-website.${var.region}.amazonaws.com"
}

Issue 2 of 2

With this syntax error corrected I was still unable to open the index.html file using the 'Distribution domain name', so after further investigation I found out that for the aws_cloudfront_distribution resource configuration, the origin_protocol_policy should be set to http-only and not https-only.

resource "aws_cloudfront_distribution" "cloudfront_dist_tf" {
  origin {
    origin_id       = local.s3_origin_id
    domain_name        = local.s3_domain_name
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      # origin_protocol_policy = "https-only" # <--- Before
      origin_protocol_policy = "http-only" # <--- Updated
      origin_ssl_protocols   = ["TLSv1"]
    }
  }

With this second issue now corrected, the CloudFront Distribution domain name now successfully opened the index.html file.

Issue 6: API Gateway: Missing Authentication Token

With the Terraform structure mostly created, the API Gateway and Lambda were good and working well, however when trying to open the API Gateway Invoke URL in the browser, I was receiving the error {"message":"Missing Authentication Token"}.

In my case the expected result should return the raw, updated visitor count number into the browser, this is what is passed to the index.html file, see below for the expected result after Invoking the URL:

Issue 6: Solution

After a fair amount of investigation and trial and error with all types of API Gateway configuration I figured out that the provided 'Invoke URL' provided by API Gateway simply needed to be updated to match my Terraform settings.

In creating the Terraform aws_api_gateway_resource, I named the path_part as "resource".

resource "aws_api_gateway_resource" "resource" {
  path_part   = "resource"
  parent_id   = aws_api_gateway_rest_api.api_gateway_tf.root_resource_id
  rest_api_id = aws_api_gateway_rest_api.api_gateway_tf.id
}

So to be able to open the Invoke URL, I needed to concatenate '/resource' to the end of the URL for the output, which I did in the following way:

output "invoke_url_full" {
  value = "${aws_api_gateway_deployment.deployment.invoke_url}/resource"
}

In addition, I also configured Terraform to output this updated URL locally to a text file which would be uploaded together with the other website files to the GitLab repo. Like this, as this URL is expected to change, the index.html file refers to this text file for the most recent Invoke URL, even if the Terraform resource is destroyed and re-created.

With these two changes made, the updated URL finally returns the count in the browser.

Issue 7: CSS intermittently not loading

This one was a front end issue I was experiencing where after uploading the website files to S3, opening the domain in the browser, sometimes it initially open with CSS but subsequent refreshes it would load with no CSS.

Testing locally it worked fine, confirming the styles.css was good.

Testing online and assuming it was a caching issue, I tried hard page reloads and testing in other browsers but it still resulted in the html page with no CSS. Also checking the console output in Chrome Developer Tools, it showed that the CSS file was loaded with status code = 200, however no CSS was being loaded.

Issue 7: Solution

Digging further, I did a curl request that confirmed the styles.css file type was “text/html”, as this is how it was configured when uploaded to the bucket by Terraform.

curl -I https://www.simon-green.uk/styles.css | grep -i content-type

In Terraform I updated the file type for the styles.css file to be 'text/css' when uploaded to S3, and thankfully this resolved the issue.

Output before:
content-type: text/html

Output after:
content-type: text/css

However it's worth noting that to make this a more robust flow, I ended up updating the Terraform configuration so that it no longer uploads any website files to the bucket, instead Terraform creates an empty bucket and the files are then added using Git (after passing Cypress testing).

Conclusion

In this post, I've shared my approach to resolving technical issues I encountered during the AWS/Terraform Infrastructure as Code (IaC) part of challenge. It's all about methodical problem-solving, starting with gathering information, researching, and testing solutions.

From API Gateway errors to CloudFront misconfigurations, each challenge required persistence and patience. By making one change at a time and testing frequently, I was able to pinpoint the root causes and implement effective solutions.

Ultimately, navigating technical hurdles demands a blend of technical know-how and perseverance. By embracing a systematic approach and leveraging available resources, we can overcome any obstacle in our development journey.

Part 4a: Terraform IaC, GitLab CI and automated testing

Simon Green — Wed, 14 Feb 2024 15:10:48 +0000

Included in this chapter...

Technology Used
Objective
Preparation
Execution
Architecture Diagram
Summary

Technology Used

AWS
Cypress
GitLab CI
Terraform
Visual Studio Code / bash / environment variables…

Objective

At the end of this step, the frontend website will look no different however the backend configuration will be completely reformed. The key differences will be:

The entire AWS configuration will have been set up as Infrastructure as Code using Terraform, all deployable with only a few clicks.
All configurations will live in a GitLab repository dedicated to this project.
If any changes have been made to the index.html file, when pushed to the GitLab repo, this will trigger a pipeline that will run tests to ensure that updated content meets predefined rules.

Preparation

I took a comprehensive deep-dive course on Terraform where I learnt how to use it from the ground up. Afterwards, with a good understanding of how it works, I began to apply this knowledge to create the Terraform configuration in parallel with the AWS console created infrastructure.

The order of my creation was:

DynamoDB
IAM
Lambda
API Gateway
S3
CloudFront
Certificate Manager / CloudFlare

I also took courses in GitLab and Cypress so I was later able to configure a GitLab repo where I would store all the code as well as setting up an automation pipeline using GitLab CI and Cypress to ensure the index.html file meets certain testing criteria before uploading to the S3 bucket.

I have separated this blog into two posts, this one you are reading, with an overview of how I solved this part of the Cloud Resume Challenge and a second post (Part 4b: Problem solving and debugging) detailing several issues I had and how I was able to overcome each of them.

Execution

Within Terraform I decided to set up the configuration using a modular structure as opposed to adding everything to the main.tf file. The project file structure I used is shown below. I set up the configuration this way as it allows for greater flexibility and readability.

Project file structure:

├── cypress
│   └── integration
│       └── index-resume-test-tf.cy.js
├── terraform_config
│   ├── acm_module
│   │   └── acm.tf
│   ├── api_gateway_module
│   │   └── api_gateway.tf
│   ├── cloudfront_module
│   │   └── cloudfront.tf
│   ├── dynamodb_module
│   │   └── dynamodb.tf
│   ├── iam_module
│   │   └── iam_role.tf
│   ├── lambda_module
│   │   ├── lambda_code.py
│   │   ├── lambda_code.zip
│   │   └── lambda.tf
│   ├── s3_module
│   │   └── s3.tf
│   └── main.tf
├── website_files
├── .gitlab-ci.yml
└── cypress.config.js

During the development of the Terraform IaC stage I have become comfortable using the Terraform Registry for creating each service to match my AWS console created version.

This part of the challenge involved a fairly solid chunk of work and along the way I encountered several issues that slowed down my progress but with a considerable amount of testing and persistence to see it through I was able to overcome each challenge.

This has been by far the most complicated part of the Challenge, the learning level was steep however I have learnt a lot about using and configuring Terraform, setting up automation pipelines and inflight testing using GitLab and Cypress and I have successfully built an end-to-end application using these and other important cloud technologies.

The end result can be seen here 🙂

www.simon-green.uk

Architecture Diagram

The diagram shown below illustrates the final overall end-to-end architecture that I created for this section and the project.

Summary

I have completed the Cloud Resume Challenge and in doing so I have gained a wealth of knowledge and hands-on experience in developing a cloud based application on AWS. I have encountered many issues but I have solved each in a logical and methodical manner to overcome each challenge.

It has been a fulfilling learning experience which I aim to continue building on through additional projects and ultimately a career move.

Don't forget to check out my next post Part 4b: Problem solving and debugging to read about a selection of the issues I faced and how I was able to resolve them.

Part 3: Frontend / backend integration

Simon Green — Mon, 23 Oct 2023 10:45:45 +0000

Included in this chapter...

Step 1: API Gateway - Deploy API
Step 2: Update index.html (hello JavaScript!)
Step 3: API Gateway - Enable CORS
Step 4: API Gateway - Re-deploy API
Step 5: Testing in the browser
Step 6: Throttling
Summary

Ok, so we have the frontend created, and the backend configured and working, now it's time to connect the two together!

For this section I have broken it down into five steps

1. Deploy the API onto a stage
2. Update landing page index.html file with the necessary JavaScript code to invoke the API when the page is loaded and return a value
3. Enabled CORS on the API
4. Re-deployed the API
5. Testing

Step 1: API Gateway - Deploy API

Following on from the previous tutorial and now with the API Gateway configured and triggering the Lambda function, it can now be deployed.

In the API Gateway console, navigate to the 'API' section and select the previously created API, in this case titled 'api_lambda_update_visitor_count', this will open the API's Resource section.

In the 'Resources' section, click on the orange 'Deploy API' button and a dialogue box will open requesting the staging details of where to deploy the API.

Stage = 'New stage'
Stage name = add stage name (for eg. 'stage_visitor_count_prod')
Click 'Deploy'

With the API now deployed to its stage, the following 'Stage' details will appear. Notice the provided 'Invoke URL', this will be needed shortly.

Step 2: Update index.html (hello JavaScript!)

++++++++++++++++++++++++++++++++++++++++
JavaScript side-note

I have been using Python for around three years now but I have never needed to use JavaScript. I understood that for this challenge only a few lines of JavaScript code were required to be inserted into the index.html file to make this work. I did some background reading on the language and found the following two pages helpful that provided an overview and and advice on how to implement what I need, which turned out to be a *'fetch()' request**...

JavaScript language overview
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Language_overview
Fetch API – How to Make a GET Request and POST Request in JavaScript
- https://www.freecodecamp.org/news/how-to-make-api-calls-with-fetch

++++++++++++++++++++++++++++++++++++++++

The following JavaScript snippet contains a fetch method and two then methods which form a kind of step-function.

I edited the following JavaScript in a text editor and for the 'fetch()' method I inserted the Invoke URL taken from the recently deployed API stage (see previous step).

        fetch('https://xxxxxxxxx.execute-api.eu-west-3.amazonaws.com/stage_visitor_count_prod')
        .then(response => response.json())
        .then((data) => {
        document.getElementById('visitor_counter').innerText = data.Count
        })

With this bit of code updated, I inserted it between the head tags in the index.html file.

When the page is loaded from the browser, this JavaScript will invoke the API and return an artefact / value for the 'visitor_counter', ready for insertion into the page.

Now I have retrieved the value, I need to indicate where within the html I want it to be displayed, to do this I used to place the value in the desired place:

<p>You are visitor number: <span id="visitor_counter" /></p>

With the index.html file updated, it can now be re-uploaded to the S3 bucket containing the website files to replace the existing version.

Step 3: API Gateway - Enable CORS

Back in API Gateway, under the 'Resources' > 'Resource details' section, click on the white 'Enable CORS' button.

In the page that opens, I updated the sections to configure the 'Gateway responses' and 'Access-Control-Allow-Methods' as shown below and clicked 'Save'.

Step 4: API Gateway - Re deploy API

With CORS enabled I needed to redeploy the API again so back under 'Resources' click again on the orange 'Deploy API' button

In the 'Deploy API' dialogue box, select the previously created stage (ie. 'stage_visitor_count_prod') and click 'Deploy'.

Step 5: Testing in the browser

Now the basic config is complete, opening the live website URL I now see the visitor counter displaying correctly on the page. If the page is refreshed the value updates by 1, success!

Step 6: Throttling

As the API is public facing I wanted to add an additional precaution to prevent any abuse from any excessive API calls, to do this I applied some throttling conditions to the API Gateway stage.

Open API Gateway in the console and click on 'Stages'. As shown in this screen, the 'Rate' and 'Burst' indicate the current throttling limits, which for this case is too high. Click 'Edit' to make the changes.

On the next screen scroll down to the 'Additional settings' section to find the throttling settings. As I don't expect a huge impact for this given use case I reduced the limits considerably.

I can now see that these settings have been applied and the API is now much more resilient to attacks.

Summary

This was an interesting section, mainly focussed around the config, I now have a more solid understanding of how the individual components work and what is needed to get them to work together, as well as learning about and using JavaScript!

With this solution primarily built using the AWS console, the next step of the challenge will be to represent the cloud resources as configuration and implement a continuous integration (CI) deployment pipeline. For this I will be using a combination of GitLab and Terraform to create my Infrastructure as Code (IaC).

This is great stuff and I'm really enjoying what I'm learning, onwards and upwards!

Thanks for reading!

Part 2: Configure the backend

Simon Green — Wed, 04 Oct 2023 11:52:59 +0000

Included in this chapter...

Objective
AWS services used
Step 1: Create the DynamoDB table
Step 2: Create the IAM Role
Step 3: Create the AWS Lambda function
Step 4: Create the API Gateway configuration
Summary

Objective

For this part of implementing the website’s visitor counter I will be working on several steps as follows:

Create a DynamoDB table that will contain a single row of two columns, the latter column will contain the current visitor count value.
Create the IAM policy and role to allow Lambda to access and update DynamoDB.
Create a Lambda function that will 1) access DynamoDB and get the current visitor count, increase it by 1 and 2) replace replace the value back into DynamoDB.
Create an API Gateway that will invoke the above Lambda function, the API will later be configured to CloudFront and triggered when the resume page is opened.

AWS services used

DynamoDB
IAM
AWS Lambda
API Gateway

Step 1: DynamoDB - create the table

Open DynamoDB from the AWS console
Select the region as ‘Europe (Paris)’, ‘eu-west-3’- Click on ‘Create table’

In the following 'Create table' page, enter the following details:

Table details
- Add the table name
- Partition key
  - Add ‘id’ and select ‘String’
Table settings
- Select ‘Customize settings’
Read/write capacity settings
- Select ‘On-demand’
Scroll to bottom
- and click on ‘Create table’

The below image shows how the table will look once set up. Note that the row count will not increase, only value under 'visitor_count' will be updated.

Step 2: IAM - Create the Role

To enable Lambda to securely access DynamoDB I will need to create a policy and attach it to an IAM role, the role will later be attached to the Lambda function. The role will be based on a policy of allowing ‘read’, ‘write’ and ‘update’ operations when Lambda connects to DynamoDB.

Open the IAM console
Click on ‘Policies’ and then ‘Create Policy’

Policy creation (AWS step 1 of 2): Specify permissions

Click on the ‘JSON’ tab and enter the below code into the ‘Policy editor’

IAM policy for Lambda so it can access DynamoDB:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "dynamodb:DescribeTable"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:dynamodb:eu-west-3:************:table/visitor_count_table"
        }
    ]
}

Policy creation (AWS step 2 of 2): Review and create

Add a name for the policy (for example 'access_dynamodb_from_lambda')

The Policy should now be created. You can click on the ‘+/-’ button on the left of the policy name to view more details.

Next, on the left-hand side of the console, click on 'Roles' and 'Create role'. Here I'll be attaching the policy to the role.

Role creation (AWS step 1 of 3): Select trusted entity

On the ‘Select trusted entity’ page, select ‘AWS service’ and under Use case select ‘Lambda’, then click on ‘Next’

Role creation (AWS step 2 of 3): Add permissions

On the ‘Add permissions’ screen add the following two permissions by searching for:

AWSLambdaBasicExecutionRole
access_dynamodb_from_lambda (as created in the above step)

Then scroll to the bottom and click ‘Create role’

Role creation (AWS step 3 of 3): Name, review, and create

Enter a name for the role (for example: 'lambda_dynamodb_role'), scroll to the bottom and click ‘Create role’.

Step 3: Create the AWS Lambda function

In this step I´ll be 1) setting up the Lambda function that will connect to DynamoDB, 2) get a value, 3) perform a calculation and 4) return (replace) it back into the database. Let's get started...

Open AWS Lambda in the AWS console
Select the region as ‘Europe (Paris)’, ‘eu-west-3’
Click on ‘Create function’

In the ‘Create function’ section I selected the following:

‘Author from scratch’
'Basic information'
- ‘Function name’
  - Crc_lambda_func_dynamodb_update
- Runtime
  - ‘Python 3.11’
Permissions
- Expand ‘Change default execution role’
- Under ‘Execution role’, select ‘Use an existing role’
- Under ‘Existing role’ locate the IAM role created above, in my case it is ‘lambda_dynamodb_role.
Then click the orange ‘Create function’ button at the bottom.

When created, the following screen will appear:

An overview of the Python script

As a reminder the DynamoDB table has been set up like this:

The following Python code that I developed was pasted into the ‘lambda_function.py’ file.

import json
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('visitor_count_table')

def lambda_handler(event, context):
# The 'event' object contains information from the invoking service
# The 'context' object is passed to your function by Lambda at runtime
    try:
        # 1. Get Item
        response = table.get_item(Key={
                'id':'visitors'
        })
        visitor_count = response['Item']['visitor_count']
        visitor_count += 1
        # timestamp
        date_time = datetime.now()
        str_date_time = date_time.strftime("%Y-%m-%d, %H:%M:%S")
        # print("Current timestamp", str_date_time)
        # 2. Put Item
        response = table.put_item(Item={
                'id':'visitors',
                'visitor_count': visitor_count,
                'timestamp_utc': str_date_time
        })
        return{'Count':visitor_count}
        status_code = response["ResponseMetadata"]["HTTPStatusCode"]
        print("HTTPStatusCode = {}".format(status_code))
        print("Successfully updated - visitor_count now = {}".format(visitor_count))
    except:
        # timestamp
        date_time = datetime.now()
        str_date_time = date_time.strftime("%Y-%m-%d, %H:%M:%S")
        # print("Current timestamp", str_date_time)
        # If the item doesn't exist, create it
        table.put_item(
            Item={
                'id': 'visitors',
                'visitor_count': 1,
                'timestamp_utc': str_date_time
            }
        )
        return{'Count':visitor_count}
        status_code = response["ResponseMetadata"]["HTTPStatusCode"]
        print("HTTPStatusCode = {}".format(status_code))
        print("visitor_count did not exist, recreated, visitor_count now equals: 1")

Here is an overview of what it does:

Declares that the Lambda function will be working with DynamoDB
States the DynamoDB table 'visitor_count_table'
Function:
- get_item: return the row where 'id' = 'visitors'
  - Declare 'visitor_count' from the DynamoDB table as a variable
  - Increase the count by 1
  - Put_item: replace the visitors row with the new value
  - Returns the updated visitor_count value
  - Print the httpstatus code to the console
  - Print confirmation that operation has been performed with new count

Step 4: Create the API Gateway configuration

Open API Gateway in the AWS console
Select the region as 'Europe (Paris)', 'eu-west-3'
Click on 'Create API'
Choose the API type 'REST API' and click 'Build'

Create REST API

API details
- ‘New API’
Add an API name -(for example: 'api_lambda_update_visitor_count')
API endpoint should be ‘Regional’
Click the orange ‘Create API’ button.

Resources

In the ‘Methods’ section, click on ‘Create method’

Here I need to create a ‘GET’ method that I did with these steps:

For ‘Method type’, select ‘GET’
For ‘Integration type’, select ‘Lambda function’
Select the Availability Zone, the same as used for the Lambda function (eu-west-3)
From the drop down menu, select the Lambda function that we previously created
Click ‘Create method’

Back on the ‘Resources’ Page

Open up the ‘Test’ tab
Then click on the orange ‘Test’ button
This should invoke the Lambda function

Checking the DynamoDB table, the visitor_count value should now increment by 1 each time the API is triggered.

Summary

This was a challenging section to complete, mainly focussed around getting the Lambda function to replace the updated value back into the database. As a result I now have a much better understanding of how this works.

Aside from this, I found it just a simple case of connecting the individual components together.

Next step, connecting the frontend and the backend!

Thanks for reading!

Part 1: Building the frontend

Simon Green — Wed, 04 Oct 2023 09:16:01 +0000

Included in this chapter...

Objective
AWS services used
S3 bucket
AWS Certificate Manager (ACM)
CloudFlare (www.cloudflare.com)
AWS CloudFront
Testing
Summary

Objective

To begin the first part of the project I needed to create a simple landing page that end users will be able to open from the web, this will act as my single-page online resume.

I used a html/css template that I used to display and stylise my resume details. The two files were then stored in an S3 bucket enabled for static website hosting.

One of the requirements for this section is that the S3 website URL should default to use https, to enable this I needed to use AWS CloudFront.

I also purchased a domain name (simon-green.uk) from a third-party registrar (CloudFlare) that I would later need to configure to point to CloudFront.

The following is the step-by-step process I used to overcome this part of the challenge.

AWS services used for this part:

Amazon S3
CloudFront
Amazon Certificate Manager (ACM)

S3 bucket

With the website files added to the bucket, initial testing was done locally and using the S3 website endpoint URL that can be found on the bucket page, 'Properties' tab, at the bottom in the 'Static website hosting' section.

One important thing to note here is that the bucket had to be named with the 'www' subdomain and the root domain, ie 'www.simon-green.uk', otherwise the functionality would not work.

AWS Certificate Manager (ACM)

I purchased a domain name from CloudFlare which I then needed to route to CloudFront by configuring certificates and DNS settings, CloudFront here will point to the S3 bucket and will also be used to enforce an https connection.

The below diagram shows the process to request a Public Certificate using Amazon Certificate Manager (ACM)

To request a Certificate:

Open Amazon Certificate Manager (ACM) and click on 'Request a certificate'

Request 'Public certificate'
Add 'Fully qualified domain name' (simon-green.uk)
Validation method should be DNS validation (and not 'Email validation)'
Click the orange 'Request' button at the bottom

Now under ACM → List certificates, I see the requested certificate with Status = 'Pending Validation'. This won't resolve until I complete the next step/s.

Click on the blue 'Certificate ID' and in the 'Domains' section, I'm interested in the string values under 'CNAME name' and the 'CNAME value'. These will be used to create DNS records on CloudFlare.

CloudFlare

In the third-party CloudFlare console I navigated to the 'Domain name management console and on the left-hand side clicked on 'DNS', here under DNS management for '[Domain Name]' - click on 'Add record' which will expand the box where you can input details.

Enter the following details:

Type = CNAME
Name = pasted in from ACM, 'CNAME name'
Target = pasted in from ACM, 'CNAME value'
Proxy status = OFF
TTL = Auto
Add comment if needed
Click ‘Save’

With this done, back in ACM, the previously requested certificate will soon show 'Status = Issued' (in my experience this typically took less than 5 minutes.)

AWS CloudFront

Now I need to create a CloudFront distribution, this will act as the public accessible endpoint (that CloudFlare will point to), the distribution will then server content from the S3 bucket hosting the website using https.

In CloudFront, click 'Create distribution' and update the following four sections:

1. (Create distribution) - Orign

For the 'Origin domain' drop down, select the S3 bucket where the website is hosted
There will be a warning box appear suggesting to 'Use website endpoint' rather than the 'bucket endpoint'
Click the button 'Use website endpoint'
Origin path = leave empty
Name = leave as is
Origin access = Public
Enable Origin Shield = No

2. (Create distribution) - Default cache behavior

Compress objects automatically = No
Viewer Protocol Policy = Redirect HTTP to HTTPS
Cache key and origin requests
- Select 'Cache policy and origin request policy (recommended)'
- Under 'Cache Policy' select 'Caching disabled'

3. (Create distribution) - Web Application Firewall (WAF)

Select 'Do not enable security protections'

4.Create distribution) - Settings

Select 'Use only North America and Europe'
In section 'Alternate domain name (CNAME)'
- Click on the 'Add item' button
- Enter domain name ('simon-green.uk')
- Again click ‘Add item’ and add 'www.simon-green.uk'
  - Under 'Custom SSL certificate - optional'
- Click on 'Choose certificate' box
- Click the drop-down and under the heading 'ACM certificates', select the certificate that was created in above step (it starts with the domain name), see below screenshot, in my case it is the only option.
Scroll to the bottom and click 'Create Distribution'

With the distribution created

The CloudFront distribution will be in a 'Deploying' status for around 5 minutes until 'Deploying' is replaced with a 'Last modified date'.

The final step is to point the domain name on CloudFlare to the CloudFront distribution's URL.

Once the distribution has finished 'Deploying', open it and under 'General' →' 'Details' click to copy the 'Distribution domain name'.

Now navigate one more time to the CloudFlare DNS section and click 'Add record' and complete it with the following details:

Type = CNAME
Name = domain name (simon-green.uk)
Target = paste in the CloudFront 'Distribution domain name' URL, from above.
- Important, you need to remove the network protocol ('https://')
Click 'Save'

Testing

After some five minutes to allow for propagation, I tested the following and worked well on various browsers and devices.

The following addresses opened the site (resolving to https)
- simon-green.uk
- www.simon-green.uk
If an http request was served (http://www.simon-green.uk), it automatically resolved to https (https://www.simon-green.uk)

Also when confirming the Domain name / SLL certificate (using dnschecker.org) it shows it was successfully issued by Amazon.

Summary

Initially I had some issues with incorrect config, where the webpage was displaying correctly on some devices but was returning 403 (Service Unavailable) on others.

The solution was in understanding the correct configuration of DNS certificates and DNS settings on both ACM and CloudFlare.

This was an interesting part of the project, I now have a much better understanding of the components and how the configuration works (especially with a third-party involved). Also the testing and resolution process was

Next step, configure the back end...

Into the Cloud...

Simon Green — Mon, 02 Oct 2023 12:25:57 +0000

Welcome to this series of blog posts where I will be documenting my journey into the cloud, in particular the completion of the Cloud Resume Challenge!

My background

I have over 10 years of broad data and analytical experience, with a core background in Trust and Safety and Product Analytics roles at letgo/OLX. I feel comfortable using IT, I have solid experience using SQL, Python and other analytics and automation tools to solve complex business problem.

I have loved building and fixing things since I was a child, always taking things apart and learning how they work before putting them back together.

In 2018 I began using SQL to help detect fraudulent account and analyse rule performance from 'big data' Redshift databases. In 2019 I began learning Python and soon developed automation workflows saving hours of manual work. I then became interested in machine learning and created a text detection model using PyTorch that could detect with good accuracy prohibited contact details within product listing photos.

Towards the end of 2023 I decided I wanted to change career direction and focus more on cloud computing especially involving AWS.

My new direction

I decided to focus on AWS as it is used by both Letgo and OLX as their cloud computing platform, also AWS has the majority global market share so it was the logical platform to begin focussing on.

After some months of studying I achieved the intermediate level AWS Certified Developer – Associate certification in September 2023. With this in hand I immediately started using AWS to build personal projects, this is where the fun begins.

The Cloud Resume Challenge

To gain some solid experience using AWS I decided to do the The Cloud Resume Challenge, developed by Forrest Brazeal who is an 'AWS Serverless Hero' who previously worked as 'Head of Developer Media' at Google Cloud. The resume challenge he devised tells you what you need to do, but not how to do it, therefore the challenger needs to learn and figure out how to accomplish each of the steps with very little guidance except for the expected output.

Below is a summary of the Cloud Resume Challenge:

Building the frontend:
- Create a simple html page and host it on S3
- Purchase a domain name
- Use AWS CloudFront to only allow https connections
- Configure all certificates etc
Configure the backend (visitor counter):
- Build the website visitor counter API using DynamoDB, Lambda (Python) and API Gateway
- All config and logic will be achieved using these services, nothing templated.
Frontend / backend integration:
- Connecting everything together
Infrastructure as Code:
- Configure the AWS services for this project as IaC
Automation / CI
- Introducing testing and automation

To briefly skip to the end of the project, the following diagram shows an overview of what I have built:

The following series of blog posts documents the actions I took to successfully complete the Cloud Resume Challenge, partly for my benefit (so I can recall the steps at a later date) and partly to share with anyone who may be interested.

A lot of reading, learning, tutorials/courses, testing and trial and error has happened from my side before this documentation could been written. My main aim here was to learn as much as possible through hands-on experience. So although I get straight into how I did it, in the majority of cases there was a fairly steep, yet overall enjoyable learning curve.

Parts 1-3 involved clicking around in the AWS console, whereas part 4 involved configuring the entire backend as IaC using Terraform.

So onto the next post: Part 1: Building the frontend