DEV Community: Vimal Paliwal

Building a Secure, Serverless Multi-Tenant RAG Chatbot with Amazon Bedrock and Lambda

Vimal Paliwal — Sun, 04 Jan 2026 15:23:24 +0000

What is RAG?

Retrieval-Augmented Generation (RAG) combines the large language model such as Anthropic Claude, OpenAI GPT, Google Gemini, etc with a vector search layer to generate a response based on the information retrieved from a knowledge base.

Why multi-tenant RAG?

The solution we worked with has a SaaS product that is used by enterprise clients and is architected to provide each client their dedicated set of infrastructure resources and the chatbot was to be integrated with the same product hence it must comply with the same rules.

As per the AWS article, there are multiple ways to architect a multi-tenant RAG solution and we decided to go with pool pattern which consists of a single knowledge base, vector store and data bucket and ensures data segregation using metadata files stored in the data bucket.

The reason behind opting for the pool pattern over others is that majority of the data provided to the knowledge base is generated internally by the product team. This allows us to avoid data duplication, keep the architecture simple and avoid admin overheads while still complying with the existing policies.

Additionally, we applied isolation at the compute layer by using the lambda tenant isolation feature.

Fig 1. Architecture Diagram

Enough talking. I believe the best way to learn is by doing it, so let's get our hands dirty. We will start with creating Bedrock Knowledge Base.

Bedrock Knowledge Base

Navigate to the Amazon Bedrock service and use the left panel to switch to Knowledge Bases to get started. Let's create a knowledge base with unstructured database.

Let AWS create a new service role for the knowledge base and choose S3 as the data source.

Fig 2.1 Bedrock Knowledge Base Setup

In the next step, select the S3 bucket that contains documents you want Bedrock to parse, generate embeddings, store it in a vector store and generate responses. Additionally, we need to select the appropriate option for parsing the source data to extract and structure the information. Let's select Amazon Bedrock Data Automation as parser but you can even choose a different foundation model if that fits your requirements.

Fig 2.2 Bedrock Knowledge Base Setup

Next step is about selecting the LLM model that you want to use for creating embeddings and the vector store that you want to use for storing those embeddings. I prefer Amazon Titan Text Embeddings model and S3 as the vector store for keeping the cost at minimum. Before opting for vector make sure to understand limitations associated with each of them to help you pick the right one.

Fig 2.3 Bedrock Knowledge Base Setup

On the review screen, confirm all the inputs and proceed with creating the knowledge base. It will take a few minutes to setup the knowledge base and other related resources.

Fig 2.4 Bedrock Knowledge Base Setup

Before we can test the knowledge base, we need to upload some files in the source bucket. I downloaded a few PDF files about Stranger Things series from Wikipedia. Along with these files you will notice metadata files too. These metadata files will help us ensure tenant-level data separation while generating response using Bedrock.

Fig 3. Bedrock KB Data Source

Stranger_Things_season_1.pdf.metadata.json

{
    "metadataAttributes" : {
      "tenantId" : "demo"
    }
}

List_of_Stranger_Things_episodes.pdf.metadata.json

{
    "metadataAttributes" : {
      "tenantId" : "test"
    }
}

Note: The metadata file name must be same as the source file and must end with the extension .metadata.json.

Pro Tip: You can include additional attributes such as user/group id to further restrict access to certain documents by a user/group.

Next step is to sync the data source. During this process, files are fetched from S3 bucket, parsed, and split into chunks to create embeddings and store them in the vector store. This step can take a few minutes depending on the amount of data you are syncing.

Fig 4. Bedrock KB Data Sync

Let's do a quick check to confirm if the knowledge base is working before we proceed to launch next services.

Fig 5. Bedrock KB Test

Awesome! It seems to be working. Let's move ahead to see how we integrated the knowledge base as a multi-tenant RAG chatbot within the SaaS product.

IAM Roles

Let's start with creating two IAM roles: one for our Authorizer Lambda function (optional) and another one for the Bedrock Agent Lambda function.

We used Lambda Authorizer to validate the request and extract tenant id from the token. This approach over relying on host header or request parameter ensures tenant id is not spoofed. If you do not require a Lambda Authorizer, you can skip creating role and policy for it.

IAM role for AuthZ lambda function (optional):

Step 1: Create Role

aws iam create-role --role-name rag-apigw-authz-lambda \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "lambda.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }'

Step 2: Attach Policy

aws iam attach-role-policy --role-name rag-apigw-authz-lambda --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

IAM role for Bedrock Agent lambda function:

Step 1: Create Role

aws iam create-role --role-name rag-bedrock-lambda \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "lambda.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }'

Step 2: Attach Policies

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "bedrock:Retrieve",
                "bedrock:RetrieveAndGenerate"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:bedrock:AWS_REGION:ACCOUNT_ID:knowledge-base/BEDROCK_KB_ID"
            ]
        },
        {
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:GetInferenceProfile"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:bedrock:AWS_REGION:ACCOUNT_ID:inference-profile/us.anthropic.claude-sonnet-4-20250514-v1:0",
                "arn:aws:bedrock:*::foundation-model/anthropic.claude-sonnet-4-20250514-v1:0"
            ]
        }
    ]
}

Note: Make sure to replace the placeholders BEDROCK_KB_ID and AWS_REGION in the above IAM policy with their respective values.

To use an LLM model other than Claude Sonnet 4 or in a region other than US, change the foundation model id and inference profile id. You can find them in AWS doc.

aws iam put-role-policy --role-name rag-bedrock-lambda --policy-name rag-bedrock --policy-document file://policy-lambda.json

aws iam attach-role-policy --role-name rag-bedrock-lambda --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Now that we have the IAM roles ready, let's deploy both the lambda functions.

Lambda Functions

AuthZ Lambda Function (Optional)

For simplicity, the below snippet extracts tenant/client id from the Authorization header passed as plain text. In actual, this must be extracted from a verified JWT to enforce tenant authenticity and prevent spoofing.

Tenant ID is added to the context block so that it is made available to the API Gateway Integration request step for further use.

bedrock_apigw_authz.py:

import os
import boto3

sts = boto3.client("sts")


def handler(event: dict, context: dict) -> dict:
    """
    Entry point for the lambda function
    """

    aws_region = os.environ.get("AWS_REGION")
    account_id = sts.get_caller_identity().get("Account")

    # extract tenant/client id from authz header
    tenant_id = event["headers"].get("Authorization").split(" ")[1]
    print(f"tenant_id: {tenant_id}")

    return {
        "principalId": "user",
        "policyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "execute-api:Invoke",
                    "Effect": "Allow",
                    "Resource": f"arn:aws:execute-api:{aws_region}:{account_id}:*",
                }
            ],
        },
        "context": {"tenantId": tenant_id},
    }

Note: Make sure to replace the placeholders AWS_REGION and ACCOUNT_ID in the above python file with their respective values. To further restrict the policy please refer to the AWS doc.

Zip the above python file and create the lambda function:

zip bedrock_apigw_authz.zip bedrock_apigw_authz.py

aws lambda create-function --function-name bedrock-apigw-authz --runtime python3.13 --role AUTHZ_IAM_ROLE_ARN --handler bedrock_apigw_authz.handler --timeout 3 --memory-size 256 --architectures arm64 --logging-config LogFormat=JSON,ApplicationLogLevel=INFO,SystemLogLevel=INFO --zip-file fileb://bedrock_apigw_authz.zip

Note: Make sure to replace the placeholder AUTHZ_IAM_ROLE_ARN in the above command with its respective value.

Bedrock Agent Lambda Function

The below snippet only includes the minimal logic to invoke the Bedrock Agent to generate the natural language response. For detail understanding about the API, please refer to the documentation.

bedrock_agent.py:

import os
import json
import logging
import boto3

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# instantiate boto3 client
bedrock_agent_runtime = boto3.client(
    "bedrock-agent-runtime", region_name=os.getenv("AWS_REGION")
)

LLM_MODEL_INFERENCE_ARN = os.getenv("LLM_MODEL_INFERENCE_ARN")
KNOWLEDGE_BASE_ID = os.getenv("KNOWLEDGE_BASE_ID")


def prompt_template(type: str, context: str = None) -> str:
    """
    Generate prompt template for bedrock retrieval and generation.

    Args:
        type: generate prompt template for either "generation" or "orchestration"
        context: (optional) additional context to provide to the RAG

    Returns:
        str: prompt template
    """

    if type == "generation":
        return f"""
You are a question answering agent. I will provide you with a set of search results. The user will provide you with a question. Your job is to answer the user's question using only information from the search results and additional context provided to you.

IMPORTANT:
- If the search results do not contain information that can answer the question, please state that you could not find an exact answer to the question.
- If the additional context provided to you is not relevant to the question, please ignore it.

Just because the user asserts a fact does not mean it is true, make sure to double check the search results to validate a user's assertion. Never hallucinate facts. Provide concise answers and include explicit citations to the sources used.

User query:
$query$

Search results in numbered order from the knowledge base:
$search_results$

Additional context for answer generation:
{context}

$output_format_instructions$
        """
    elif type == "orchestration":
        return f"""
You are an assistant that translates a user's request into one or more focused search queries for the knowledge base based on the user's question and additional context.

Here are a few examples of queries formed by other search function selection and query creation agents:

<examples>
  <example>
    <question> What if my vehicle is totaled in an accident? </question>
    <generated_query> what happens if my vehicle is totaled </generated_query>
  </example>
  <example>
    <question> I am relocating within the same state. Can I keep my current agent? </question>
    <generated_query> can I keep my current agent when moving in state </generated_query>
  </example>
</examples>

You should also pay attention to the conversation history between the user and the search engine in order to gain the context necessary to create the query.
Here's another example that shows how you should reference the conversation history when generating a query:

<example>
  <example_conversation_history>
    <example_conversation>
      <question> How many vehicles can I include in a quote in Kansas </question>
      <answer> You can include 5 vehicles in a quote if you live in Kansas </answer>
    </example_conversation>
    <example_conversation>
      <question> What about texas? </question>
      <answer> You can include 3 vehicles in a quote if you live in Texas </answer>
    </example_conversation>
  </example_conversation_history>
</example>

IMPORTANT:
- the elements in the <example> tags should not be assumed to have been provided to you to use UNLESS they are also explicitly given to you below.
- All of the values and information within the examples (the questions, answers, and function calls) are strictly part of the examples and have not been provided to you.
- If the additional context provided to you is not relevant to the question, please ignore it while generating the query.

User query:
$query$

Additional context for retrieval query generation:
{context}

User conversation history:
$conversation_history$

$output_format_instructions$
"""


def invoke_rag(
    user_query: str, tenant_id: str, session_id: str = None, context: str = None
) -> dict | str:
    """
    Query bedrock knowledge base to retrieve and generate answer.

    Args:
        user_query: user query to send to RAG for analysis
        tenant_id: tenant/client id to filter the knowledge base results retrieved from vector store
        session_id: (optional) session id to maintain context and knowledge from previous interactions
        context: (optional) context to provide additional information to the RAG

    Returns:
        dict: response from RAG if successful, error message if failed
    """

    try:
        # ref: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent-runtime/client/retrieve_and_generate.html
        req_args = {
            "input": {"text": user_query},
            "retrieveAndGenerateConfiguration": {
                "knowledgeBaseConfiguration": {
                    "knowledgeBaseId": KNOWLEDGE_BASE_ID,
                    "modelArn": LLM_MODEL_INFERENCE_ARN,
                    "retrievalConfiguration": {
                        "vectorSearchConfiguration": {
                            "numberOfResults": 5,
                            "overrideSearchType": "SEMANTIC",
                            # allows retrieval of data that belong to the tenant
                            "filter": {
                                "equals": {"key": "tenantId", "value": tenant_id}
                            },
                        }
                    },
                    "generationConfiguration": {
                        "promptTemplate": {
                            "textPromptTemplate": prompt_template("generation", context)
                        },
                    },
                    "orchestrationConfiguration": {
                        "promptTemplate": {
                            "textPromptTemplate": prompt_template(
                                "orchestration", context
                            )
                        },
                        "queryTransformationConfiguration": {
                            "type": "QUERY_DECOMPOSITION"
                        },
                    },
                },
                "type": "KNOWLEDGE_BASE",
            },
        }

        # assign session id to maintain context and knowledge from previous interactions
        if session_id is not None:
            req_args["sessionId"] = session_id

        # query rag to generate response
        response = bedrock_agent_runtime.retrieve_and_generate(**req_args)

        return {"statusCode": 200, "message": response}
    except Exception:
        logger.exception("failed to generate response")
        return {
            "statusCode": 500,
            "message": {"error": "failed to generate response"},
        }


def handler(event: dict, context: dict) -> dict:
    """
    Entry point for the lambda function
    """
    req_body = json.loads(event["body"])

    # prepare request arguments
    req_args = {"user_query": req_body["user_query"], "tenant_id": context.tenant_id}

    # load session id to maintain context and knowledge from previous interactions
    if "session_id" in req_body:
        req_args["session_id"] = req_body["session_id"]

    # load context to provide additional information to the RAG
    if "context" in req_body:
        req_args["context"] = req_body["context"]

    response = invoke_rag(**req_args)

    return {
        "statusCode": response["statusCode"],
        "body": json.dumps(response["message"]),
        "headers": {
            "Content-Type": "application/json",
        },
    }

Note: Both the prompts used in the above code are sourced from AWS.

The above snippet starts with extracting values from request body such as user_query, tenant_id and optionally session_id and context which are used as inputs along with a few other values to generate natural language response.

The important point to focus is the use of filter parameter within vectorSearchConfiguration block while preparing request body for bedrock_agent_runtime.retrieve_and_generate method. This filter parameter ensures that when Bedrock fetches data from the vector store it only refers to the data owned by the respective tenant.

"retrievalConfiguration": {
    "vectorSearchConfiguration": {
        ...
        "filter": {
            "equals": {
                "key": "tenantId", 
                "value": tenant_id
            }
        },
    }
}

In case you are separating client data at bucket or folder level, you can filter using S3 prefixes which is a built-in metadata. This avoids maintaining custom metadata files.

"retrievalConfiguration": {
    "vectorSearchConfiguration": {
        ...
        "filter": {
            "startsWith": {
                "key": "x-amz-bedrock-kb-source-uri", 
                "value": f"s3://$bucket_name/"
            }
        },
    }
}

Note: startsWith and stringContains is not supported for S3 vector store. Please refer to the AWS doc to learn more about using the filters.

Time to zip the above python file and create the lambda function with tenant-isolation mode enabled to ensure client data segregation at compute layer.

zip bedrock_agent.zip bedrock_agent.py

aws lambda create-function --function-name bedrock-agent --runtime python3.13 --role BEDROCK_AGENT_LAMBDA_ROLE_ARN --handler bedrock_agent.handler --timeout 15 --memory-size 256 --architectures arm64 --logging-config LogFormat=JSON,ApplicationLogLevel=INFO,SystemLogLevel=INFO --tenancy-config TenantIsolationMode=PER_TENANT --zip-file fileb://bedrock_agent.zip --environment 'Variables={LLM_MODEL_INFERENCE_ARN=xxxx,KNOWLEDGE_BASE_ID=xxxx}'

Note: Make sure to replace the placeholder BEDROCK_AGENT_LAMBDA_ROLE_ARN in the above command with its respective value and update values for both the environment variables LLM_MODEL_INFERENCE_ARN and KNOWLEDGE_BASE_ID.

It's time to deploy the final piece to complete the puzzle.

REST API Gateway

At the time of writing this article, HTTP API Gateway cannot be used with a multi-tenant enabled lambda function because it does not support overriding the X-Amz-Tenant-Id header, hence we create REST API Gateway. However, if you are deploying a lambda function without tenant-isolation mode, you can surely use HTTP API Gateway.

openapi-schema.json

{
  "openapi" : "3.0.1",
  "info" : {
    "title" : "bedrock-rag-chatbot",
    "description" : "Bedrock RAG chatbot REST API Gateway",
    "version" : "1.0"
  },
  "paths" : {
    "/genai/query" : {
      "post" : {
        "security" : [ {
          "apigw-lambda-authz" : [ ]
        } ],
        "x-amazon-apigateway-integration" : {
          "type" : "aws_proxy",
          "httpMethod" : "POST",
          "uri" : "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/BEDROCK_LAMBDA_FUNCTION_ARN/invocations",
          "requestParameters" : {
            "integration.request.header.X-Amz-Tenant-Id" : "context.authorizer.tenantId"
          },
          "responses" : {
            "default" : {
              "statusCode" : "200"
            }
          }
        }
      }
    }
  },
  "components" : {
    "securitySchemes" : {
      "apigw-lambda-authz" : {
        "type" : "apiKey",
        "name" : "Unused",
        "in" : "header",
        "x-amazon-apigateway-authtype" : "custom",
        "x-amazon-apigateway-authorizer" : {
          "identitySource" : "method.request.header.Authorization,context.httpMethod,context.path",
          "authorizerUri" : "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/AUTHZ_LAMBDA_FUNCTION_ARN/invocations",
          "authorizerResultTtlInSeconds" : 300,
          "type" : "request"
        }
      }
    }
  }
}

Note: Do not forget to update the placeholder BEDROCK_LAMBDA_FUNCTION_ARN and AUTHZ_LAMBDA_FUNCTION_ARN in the above OpenAPI file with their actual values.

In case you are not using Lambda Authorizer to extract tenant id and add it to the context, follow the steps provided in the AWS blog to extract it from http request.

Step 1: Create/Import REST API

aws apigateway import-rest-api --body 'fileb://openapi-schema.json'

Step 2: Deploy API

aws apigateway create-deployment --rest-api-id REST_API_ID --stage-name demo

Next, we need to update resource policy for both the lambda functions to allow API gateway to invoke them.

Fetch Lambda Authorizer ID

aws apigateway get-authorizers --rest-api-id REST_API_ID

Permission for Authoriser Lambda

aws lambda add-permission --function-name bedrock-apigw-authz --action lambda:InvokeFunction --statement-id apigw-invoke --principal apigateway.amazonaws.com --source-arn arn:aws:execute-api:AWS_REGION:ACCOUNT_ID:REST_API_ID/authorizers/REST_API_AUTHORIZER_ID

Permission for Agent Lambda

aws lambda add-permission --function-name bedrock-agent --action lambda:InvokeFunction --statement-id apigw-invoke --principal apigateway.amazonaws.com --source-arn arn:aws:execute-api:AWS_REGION:ACCOUNT_ID:REST_API_ID/*/POST/*

Fun time

It's finally time to validate if the entire setup is working as expected. I'm using curl tool for making the API calls but you can use tools such as Postman Client as well.

We will test for the following cases:

Client test has access to List_of_Stranger_Things_episodes.pdf, so must be able to list episodes for all of the seasons.
Client demo can only access Stranger_Things_season_1.pdf, so can give information about season 1 but nothing about any other season.
Invalid client dummy should not be able to retrieve any sort of information.

Fig 6.1 Client test

Fig 6.2.1 Client demo - Unable to fetch information about season 2 episodes

Fig 6.2.2 Client demo - Showing information about season 1 episodes

Fig 6.3 Invalid Client dummy

Mission accomplished! We have successfully created a secure multi-tenant RAG chatbot.

Caveats

This article does not cover about applying guardrails to the knowledge base which is recommended to safeguard the content generated by LLM models against sensitive data, profanity and hallucination. Additionally, AWS WAF should be applied to the REST API Gateway to protect applications against Layer 7 attacks.

Last but not the least, rigorously test the responses generated by various models available within Bedrock and tweak the values passed to the Bedrock API to better meet the requirements for your use-case.

Stop Malware at the Door: Automated S3 File Scanning with AWS GuardDuty

Vimal Paliwal — Thu, 27 Nov 2025 19:46:54 +0000

Photo by Ed Hardie on Unsplash

Original Source: https://skildops.com/blog/stop-malware-at-the-door-automated-s3-file-scanning-with-aws-guardduty

Amazon S3 is widely used to store various kind of files and most of the times we do not scan these files for malwares thus exposing ourselves and our clients to risk. It was the same case for one of projects we were working on. It was business as usual and we were on a prep call before the official ISO-27001 audit process kicks off. We had managed to tick all the boxes except the one when we were asked if we are scanning the files that we upload to the S3 bucket and our answer was a straight no. I could see on the face of the evaluator that she wasn’t very happy with our answer and asked if we can manage to get it sorted.

The security audit was just a week later and we knew the time is not enough to implement a solution within the timeframe. Luckily, a temporary exception was approved for the solution because all the files distributed to the clients were generated internally by the data engineers. The exception got us enough time to implement a solution to mitigate the risk of distributing malicious files to the clients.

In this article, we will implement a simple serverless solution to automatically mitigate such risk in real-time using AWS GuardDuty.

Key Services Involved:

Let’s start by taking a look at the architecture diagram to understand how the solution functions.

Architecture Overview

Fig 1. Architecture Diagram

It all starts with enabling the GuardDuty malware protection plan for the source S3 bucket either via console, API or CLI. Once you enable the malware protection plan, S3 event notifications are configured to invoke the scan whenever an object is uploaded to the source bucket.

After the scan, GuardDuty will optionally tag the object and publish the result to an EventBridge rule for further processing.

Once the scan results are published to the EventBridge rule, they are forwarded to an SNS topic to achieve a fan-out solution. The message is then pushed to an SQS queue which is consumed by a Lambda function for final processing.

Note:

As of writing this article, only one source bucket can be associated per protection plan. Hence, you need to create a dedicated protection plan for every bucket you want to secure.

S3 bucket must be in the same region as GuardDuty.

Business Logic

Once a message is pulled by the Lambda function from SQS, it starts with extracting the scan result and file details from the message generated by GuardDuty and pushed to the EventBridge to identify the action that it needs to perform. The action that will be executed depends on the scan result and the action configured for a result type.

There are two ways to configure the action: either by setting the environment variable GD_SCANNED_FILE_ACTION for the Lambda function through the respective Terraform variable or by adding a tag to the object uploaded to the bucket. Details about how to tag an S3 object can be found in the GitHub repo readme.

Tag attached to the object always gets the highest priority if an action is configured using both the above mentioned options whereas if an action is not configured via either of the supported methods, default action is chosen based on the scan result.

In case, actions are configured partially via either or both the supported methods, deep merge is performed to generate the final result with S3 object tag having the highest priority followed by the lambda function environment variable and default action with the least priority.

Post identifying and executing the action, optionally a notification will be triggered via AWS SNS topic.

Default action:

{
  "threat": {
    "action": "delete",
    "notify_user": True,
  },
  "no_threat": {
    "action": "do_nothing",
    "notify_user": False,
  },
  "failed": {
    "action": "do_nothing",
    "notify_user": True,
  },
}

For a detailed guide on how to deploy and configure the solution, please refer to the GitHub repo that contains the source code, a terraform module and a detailed readme to deploy the solution on AWS cloud.

How we saved $20k annually by self-hosting a map server

Vimal Paliwal — Mon, 07 Apr 2025 13:21:33 +0000

A bit of history…

For the last few years, the project I have been working on was using Mapbox to render maps with custom data until one day when the team decides to move away from it to an open-source solution. The project have grown big over the years and with that have grown the licensing cost for Mapbox. Hence, switching to an alternate and open-source solution was a wise option, so we decide to switch to TileServer.

About TileServer

TileServer GL is an open-source map server that supports both serving vector tiles as well as raster tiles (through Mapbox GL Native).

Maptiler Data hosts vector files that can be used with the TileServer but as of writing this article, the file is available for free only for non-commercial/educational purpose. Otherwise, a license must be purchased. Alternatively, the vector files can be generated from scratch by following the instructions mentioned in the OpenMapTiles git repo.

Beginning of the journey

The dev team started working on the POC on their local machine and in few days they were able to confirm that TileServer is a great drop-in replacement to Mapbox but little they knew about the challenges that we were about to face while deploying TileServer to AWS for production use.

Let’s begin the real journey.

Generating Vector Files - A Nightmare

We face our initial challenge during the process of generating the planet.mbtiles vector file from scratch. We first try generating the file on a local Mac that has 32GB memory. After few hours, we encounter our first error and realise that we need a more powerful machine, so we decide to spin up an EC2 machine with 64GB memory and start the process to run in the background using nohup and stream the logs to a file.

It's been a few hours and we had managed to successfully run the first few steps/scripts. Somewhere during midday, we start the next script in the background and before logging off for the day, we look at the logs and the script still seems to be running, so leave the instance running overnight.

The next day, we check the logs and notice an error "No space left on the device". Not a good start of the day! We decided to double the storage to 200GB and re-ran the script. Periodically checking to make sure everything is working fine. Another day comes to an end and the script is still running.

Day 3 and the script is still running, surprising yet glad because of no errors. Unfortunately, the happiness stayed only for an hour or so before we hit the same error "No space left on the device". As frustrating it is, we upgrade the storage to 500GB and run the script once again.

Day 4 passed and on Day 5 we hit the same error once again. This time before increasing the disk space, I decide to check the disk usage metrics using "df -h" command and what I notice is that there's still around 200GB remaining. After brainstorming for sometime, we realised that we have come across similar case earlier as well where psql complains about "No space left on the device" and we decide to upgrade the instance type this time to have 128GB memory. We run the script again and after 2 days, we finally achieve success.

Wait, the struggle isn't over yet.

We still had 2-3 scripts left to run to generate the mbtiles file. The remaining scripts were very quick and by the end of the day we had managed to generate our mbtiles file.

We start celebrating the success but soon the bubble bursts and we realise that the generated vector file isn’t valid.

The team starts investigating what went wrong during the entire process and during this we come across a reddit post that turned out to be a saviour for us. We don't need to generate the file anymore from scratch as this kind hearted person has done the heavy lifting for us. This person hosts the file on a public server to allow people to simply download and use it.

We were at the top of the mountain but before we can use the file, we had to make sure that it is safe to use the file as it is generated by a third person. The scan took few mins given the large file size but it came clean and we were excited by this.

The journey of setting up infrastructure starts now.

Onto the implementation part

I will be focusing mostly only on the architecture specific implementation details rather than generic steps about deploying a particular service to keep the article concise.

Entire source code is available in the GitHub repository. Feel free to customise it as per your use case and contributions are always appreciated.

A quick overview of all the components involved:

CloudFront: Acts as an entry point to the incoming traffic and forwards the traffic to API Gateway along with a custom header.
WAF: Protects against application layer attacks and validates the incoming request.
CloudFront Function: Associated to Viewer Response event, adds CORS and security response headers. This is optional because you can use response headers policy feature of CloudFront to manage the same.
HTTP API Gateway: Responsible for routing incoming requests to the ECS service via VPC Private Link.
Lambda Authorizer: To validate and authorize requests based on origin, custom header from CloudFront and the bearer token.
Cloud Map: Keeps track of private IP addresses assigned to the privately hosted ECS service and share it with API Gateway.
ECS: To deploy nginx reverse proxy and TileServer containers.
S3 & EFS: To store the mbtiles file and tileserver config files. These are use-case specific and hence optional.

Let’s dive into individual components and understand the configuration.

CloudFront

I’ll skip walking through the setup process of the CloudFront for two reasons: firstly, because it’s a standard setup with HTTP API Gateway as origin and a default behaviour to forward all traffic to the origin server except introducing the custom header part and second, this has already discussed in detail in my previous article.

In brief, within the origin section, there’s an option to forward custom header and we have used this feature so that Lambda Authorizer attached to HTTP API Gateway can validate all the incoming requests are routed via CloudFront as this ensures that all the incoming requests are scanned by WAF.

Fig 1. CloudFront Overview

Fig 2. CloudFront Origin

Fig 3. CloudFront Behavior

WAF

By default, we have set the action to block all the traffic and only allow the traffic if it has passed all the rules. To fulfil our requirement, we have used a combination of AWS managed rules and custom rules.

AWS Managed rules:

AWSManagedRulesCommonRuleSet
AWSManagedRulesAmazonIpReputationList
AWSManagedRulesKnownBadInputsRuleSet
AWSManagedRulesAnonymousIpList
AWSManagedRulesSQLiRuleSet
AWSManagedRulesLinuxRuleSet

Custom rules:

Block Sanctioned Countries
Block Traffic from AWS default domain
Whitelist traffic from CORS domain

Block traffic from Sanctioned Countries

rule {
  name     = "block-countries-rule"
  priority = 0

  action {
    block {}
  }

  statement {
    geo_match_statement {
      country_codes = [
        "CU", # Cuba
        "IR", # Iran
        "KP", # N. Korea
        "RU", # Russia
        "SY"  # Syria
      ]
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "tileserver-block-countries-rule-waf-acl"
    sampled_requests_enabled   = true
  }
}

Block traffic from AWS default domain

rule {
  name     = "block-aws-default-domain"
  priority = 1

  action {
    block {}
  }

  statement {
    byte_match_statement {
      field_to_match {
        single_header {
          name = "host"
        }
      }
      positional_constraint = "ENDS_WITH"
      search_string         = ".cloudfront.net"
      text_transformation {
        type     = "NONE"
        priority = 0
      }
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "tileserver-block-aws-default-domain-waf-acl"
    sampled_requests_enabled   = true
  }
}

Whitelist traffic from product domain

rule {
  name     = "whitelist-cors-domain"
  priority = 200

  action {
    allow {}
  }

  statement {
    byte_match_statement {
      positional_constraint = "EXACTLY"
      search_string         = "xxxxxxxx.xxx"

      field_to_match {
        single_header {
          name = "origin"
        }
      }

      text_transformation {
        priority = 0
        type     = "NONE"
      }
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "tileserver-whitelist-cors-domain-rule-waf-acl"
    sampled_requests_enabled   = true
  }
}

Fig 4. WAF Rules

CloudFront Function

It is not necessary to utilise CloudFront Function to add CORS and Security headers because the same can be achieved by attaching response header policy to the CloudFront distribution.

We decided to use CloudFront Function because we had to dynamically generate the CORS access-control-allow-origin response header based on the origin request header and only generate the CORS response headers if the origin header is present in the list of whitelisted sub-domains.

async function handler(event) {
    var request = event.request;
    var response = event.response;
    var headers = response.headers;

    // Set CORS headers
    // Since JavaScript doesn't allow for hyphens in variable names, we use the dict["key"] notation.
    const originRegex = /^https:\/\/(?:[a-zA-Z0-9-]+\.)+abc\.example\.com$/;
    if (originRegex.test(request.headers['origin'].value)) {
        console.log("Adding CORS headers");
        response.headers['access-control-allow-origin'] = {
            value: request.headers['origin'].value
        };
        response.headers['access-control-allow-credentials'] = {
            value: 'true'
        };
        response.headers['access-control-allow-methods'] = {
            value: 'GET, OPTIONS'
        };
        response.headers['access-control-allow-headers'] = {
            value: 'Authorization'
        };
    }
    else {
        console.log("Origin does not match regex. Origin: " + request.headers['origin'].value);
    }

    // Set HTTP security headers
    // Since JavaScript doesn't allow for hyphens in variable names, we use the dict["key"] notation
    headers['strict-transport-security'] = {
        value: 'max-age=63072000; includeSubdomains; preload'
    };
    headers['x-content-type-options'] = {
        value: 'nosniff'
    };
    headers['x-frame-options'] = {
        value: 'DENY'
    };
    headers['x-xss-protection'] = {
        value: '1; mode=block'
    };
    headers['referrer-policy'] = {
        value: 'no-referrer-when-downgrade'
    };

    // Remove the Server header
    delete headers['server'];

    // Return the response to viewers
    return response;
}

Fig 5. CloudFront Function

HTTP API Gateway

Wondering why HTTP API Gateway instead of REST API Gateway?

Simple answer: Cost effective and serves all our requirements.

Nothing complicated with the setup. Just a single proxy path "/{path+}" that forwards all the incoming requests to the ECS service via VPC Private Link. Additionally, we have implemented Lambda AuthZ to validate the bearer token to ensure the request is coming from an authenticated user and the custom authZ token received from the CloudFront distribution to ensure the request is coming via CloudFront.

Fig 6. HTTP API Gateway

Fig 7. VPC Private Link

Fig 8. Lambda Authorizer

Cloud Map

HTTP API Gateway cannot directly discover the physical address of ECS service and given the self-healing nature and auto-scaling capability, it is not a wise option to hardcode the private IP of the service(s), so we use Cloud Map which provides the capability of service discovery. We need to make sure to configure service discovery while creating the ECS service to use this feature. Moreover, Cloud Map perform health checks and only registers healthy ECS tasks to the service.

There’s a detailed article by the AWS team on exposing ECS service via HTTP API Gateway using Private Link that you can read to understand all about the moving pieces involved.

Fig 9. CloudMap

S3 & EFS

S3 serves as a single source of truth for maintaining the mbtiles file and custom configuration for TileServer. The custom config is maintained in the GitHub repo and to sync the changes to S3 we have a GitHub workflow about which I won’t be covering in this article. The process to update mbtiles file is currently manual given that we have to download the file from a third-party server, scan it for any security risk before pushing it to S3 for production use. If you aren’t planning on customising the TileServer and serving just a specific small region, then you may skip maintaining the bucket.

Provided we are running multiple replicas of the service, EFS was our obvious choice. We implemented access points and mapped them to non-root user along with mandating encryption and IAM auth via a resource policy for enhanced security.

Fig 10. EFS Access Points

Fig 11. EFS Access Policy

ECS

It’s time to finally deploy TileServer. I initially chose to use EC2 as the capacity provider and Bottlerocket as the AMI but soon I hit a roadblock because as of writing this article, Bottlerocket do not support EFS with IAM authentication and encryption mode enabled, so we decided to go with Fargate as the capacity provider given the enhanced security, ease of deployment and less management overhead.

That being said, I have subscribed to the GitHub issue to stay up-to-date with any progress because I’m keen to switch to using EC2 with Bottlerocket AMI given the OS is specially designed to run containers securely with bare minimum essentials softwares resulting in reduced attack-surface compared to standard OS and at scale can achieve cost savings too.

Anyway, let’s get into the implementation details. As seen in the architecture diagram, the task comprises of 2 containers: nginx and tileserver. Along with these containers we also created an init containers for each of the services. These init containers were used to generate configuration and pull some data required to run the services.

Nginx

nginx act as a reverse-proxy because when running TileServer behind a proxy or load balancer, we need to ensure that X-Forwarded-* headers are forwarded to generate the URLs with correct protocol and domain. The init container is responsible for generating the custom nginx routing config.

TileServer

TileServer is the epicentre of this entire implementation. Before running the tileserver container we run an init container to pull the planet.mbtiles file from S3 and store it in EFS along with some custom configuration to style the UI. Given the planet.mbtiles file is around 90GB, choosing EFS over local storage seems wise to avoid startup delay every time a container starts. If you are planning to simply host map for a specific city/region, then you may just choose local/EBS storage given the file size is small and save some money.

Fig 12. ECS Cluster

Fig 13. TileServer

Triumph mode on!

After weeks of struggle, we finally managed to host the TileServer on AWS using all serverless services and save around $20k annually compared to Mapbox.

But don’t worry, you will not have to spend weeks to get it up and running. All the source code is available in my GitHub repository that will help you to host TileServer on AWS within just few minutes or hours.

Demystifying an interesting relation between ECR and S3

Vimal Paliwal — Wed, 11 Dec 2024 16:09:51 +0000

Photo by Giorgio Trovato on Unsplash

Original Source: https://skildops.com/blog/demystifying-an-interesting-relation-between-ecr-and-s3

Preface

Did you know ECR images are stored in S3? Well, I discovered this fact after the pods running on EKS cluster started failing to pull the images from ECR throwing 403 Forbidden error.

You may be puzzled about how are these 2 scenarios even related? Glad you asked this.

Basically, I created an S3 Gateway Endpoint so that VPC resources will avoid using the expensive Internet/NAT gateway route to interact with S3 objects and rather than using an overly-permissive resource policy for the gateway endpoint, I decided to restrict the use of endpoint by the services in a particular account by using aws:PrincipalAccount condition.

Gateway Endpoint Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "*",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalAccount": "NUMERIC_AWS_ACCOUNT_ID"
                }
            }
        }
    ]
}

Discovery Phase

Few hours after the S3 gateway endpoint was deployed, the development team triggers a deployment pipeline to test new backend changes and realises the pipeline is broken and reaches out to the DevOps team.

The task gets assigned to me and I start the investigation. Upon looking at the Kubernetes events, I notice an ImagePullBack error with reason 403 Forbidden. Initially, I’m bit confused because no infrastructure changes were deployed in the past few hours that modified permissions of either the IAM role associated to the EKS pod or the resource policy of ECR registry. Still, to be 100% sure, my first course of action was to verify permissions for both ECR resource policy and IAM role attached to the EKS pod. As I thought earlier, the permissions were intact.

As my next step, I start looking at the recent changes made to the infrastructure and the only change pushed in the last few hours was the implementation of S3 Gateway Endpoint. At first, it does not make sense to me that why and how is S3 Gateway Endpoint linked with ECR image pull issue.

After few mins, it strikes my mind that may be ECR uses S3 to store the images and may be the restrictive policy of gateway endpoint is the reason, so I decided to temporarily switch to the default overly-permissive resource policy to validate the hypothesis. Surprisingly, the pod can now pull the image from ECR.

Even though the issue was resolved by changing the policy, we cannot call it a victory because we the default resource policy does not follow the least-privilege principle. Hence, we decided to dive deeper into the problem and uncover the relation between these two services so that we can write a restrictive policy and also allow our pods and other services to download the container images from ECR.

Troubleshooting Phase

Continuing with the investigation, I decide to revert back the endpoint resource policy to the original restrictive one before doing anything else.

We have a bastion host running within the VPC, so I SSH into it and pull the ECR image with a hope to get some additional information regarding the error that can help me resolve the mystery.

Brilliant! I get a detailed error that explains it all.

Error:

Error pulling image configuration: download failed after attempts=1: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>User: arn:aws:sts::026506400584:assumed-role/storage-access-role-prod-eu-west-1/s8-fe-dub-prod-2-12021.dub2.amazon.com is not authorized to perform: s3:GetObject on resource: \"arn:aws:s3:::prod-eu-west-1-starport-layer-bucket/bf8750-096813982275-babc1bd1-8c7f-a85b-5fdf-78508ffe501c/a07f425b-261f-4cae-a9ba-1993d84f9d58\" because no VPC endpoint policy allows the s3:GetObject action</Message><RequestId>B740Z04JBRFNX95Q</RequestId><HostId>kjSd+f8pT2rKPt6HVOV6H8+7rKubuD34xeEnwVpmVbqJ2Df3ibnrLOsxmpkxb+YMmoSrCJSW25GhD2t2BHDm6NiM1ypxTqXv</HostId></Error>"

The error makes it clear that an AWS managed IAM role is trying to download image/artifacts from an S3 bucket managed by AWS.

This brings me one step closer to finding the right solution.

Solution Phase

After doing a bit of research I come across an AWS documentation that talks about the exact problem and its solution. It was time to update the endpoint policy to allow AWS to use the gateway endpoint to pull files from this particular bucket.

Updated Gateway Endpoint Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "*",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalAccount": "NUMERIC_AWS_ACCOUNT_ID"
                }
            }
        },
        {
            "Sid": "AccessToAWSEcrBucket",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::prod-{AWS_REGION_ID}-starport-layer-bucket/*"
        }
    ]
}

Note: If you plan to use the above policy, make sure to replace the placeholders NUMERIC_AWS_ACCOUNT_ID and AWS_REGION_ID with the actual values

Understanding the basics

Q: What is the difference between S3 gateway endpoint and interface endpoint?
A: In both cases, your traffic stays on the Amazon network so that your resources within the VPC can bypass Internet/NAT gateway to interact with S3. Few noticeable differences include creation of ENIs in case of interface endpoints within your VPC and use private IP space to interact with S3 API whereas in case of gateway endpoint, route table is updated to direct the traffic to the endpoint if traffic is destined to S3 assigned public IP. Moreover, use of gateway endpoint is free but interface endpoint incurs additional cost. To learn more differences between both the endpoints, please visit AWS doc.

Q: What is the difference between VPC Endpoint and PrivateLink?
A: VPC Endpoint enables you to connect to AWS services privately from within your VPC whereas with the help of PrivateLink, other AWS accounts can interact with privately hosted services in your account.

Q: How do I know if VPC endpoint is used?
A: There are multiple ways to validate if traffic is flowing via a VPC endpoint and it depends on the type of endpoint you are using. You can use network utility scripts like ping, traceroute to identify the IP and path or you can use the VPC flow logs to validate the traffic flow.

AWS Client VPN Caveat: Knowing this can save you hours of troubleshooting

Vimal Paliwal — Thu, 24 Oct 2024 09:04:42 +0000

Image by Dan Nelson from Pixabay

Original Source: https://skildops.com/blog/aws-client-vpn-caveat-knowing-this-can-save-you-hours-of-troubleshooting

I’m assuming you already know about AWS Client VPN but if not, it is a fully-managed remote access VPN solution that supports both certificate and SAML based authentication to securely access resources within both AWS and your on-premises network.

Let’s set the context

For the last few years, I have been working on a project that hosts a single-tenant SaaS application on AWS in a single region. Many of our services are hosted in private network, so we use AWS Client VPN to interact with these services.

Few months ago, we decided to refactor our IaC (Terraform) code so that we can deploy our SaaS application in multiple regions based on client’s request to fulfil data-residency and compliance requirements. Being a single-tenant application, each client has their own set of resources like KMS key, S3 buckets, database instance, etc. Along with these resources, there’s also a central/shared database cluster which contains generic data and this shared database cluster resides in our primary region unlike client databases that are scattered across different regions. We wanted to make sure that client database clusters can talk to the central/shared database cluster, so we decided to setup VPC Peering between the regions which was quick and straightforward.

Note: The above diagram is just for understanding purpose and is a very simplified representation of the actual architecture.

Now, you may be thinking that this sounds like a pretty common design and there doesn’t seem to be any problem. You’re absolutely correct but this is not the full picture yet. Let’s talk about the problem now.

The Problem

Earlier, I mentioned that we use AWS Client VPN to connect to internal resources which was all smooth until we introduced support for multiple region deployments. We did not want to have separate Client VPN in each region, so we decided to modify the configuration of our existing Client VPN to interact with resources deployed across various regions. So, this is where the problem starts.

This was a new challenge for me so I turn to AWS documentation to find out how do I accomplish this. After going through the AWS Client VPN doc carefully, I updated the route table and authorization rule. Part one complete. Next, I had to update the route tables and NACLs for VPC and security groups for internal resources to accept connection from the VPC CIDR hosting the Client VPN.

Boom!! It’s all working smoothly and I get super excited.

But wait, looks like I got excited little early because I did these changes via console and not via Terraform and this is where I experience the problem. I replicated all the changes I did in the console in Terraform, reverted back the changes in console and fired terraform apply command.

Image generated using DALL-E-3

In an ideal scenario, this should have worked because all the settings were properly configured but in reality I wasn’t able to connect to any of the internal resources in regions other than primary.

Time To Troubleshoot

Sad and shocked, I start troubleshooting the config changes both in Terraform and AWS console. Everything looks exactly the same as I did earlier via the console. After troubleshooting for around an hour, I go back to AWS Client VPN doc and read the steps 2-3 times and compare my changes to figure out in case I have messed up anything but it all looks absolutely fine. At this time, I’m starting to get a bit annoyed and exhausted, so I go for a walk to grab my favourite Flat White ☕️ to refresh my mind.

Photo by Nathan Dumlao on Unsplash

After a short break, I decide to check if the bastion host present in the primary region is able to connect to internal resources hosted in another region and it can. This confirms my VPC peering and NACL rules are correct and something is wrong with AWS Client VPN configuration, so I switch back to Client VPN console and start staring at authorization rules and route tables config but couldn’t figure out anything with either of them. Destination CIDR, Group ID, Access Type and Target Subnet looks all properly configured, so I give up after an hour and decide to end my day 😴.

Image generated using DALL-E-3

The Solution

Next day morning, I start fresh and after an hour of troubleshooting with a networking expert colleague I manage to fix the issue. The problem wasn’t with the configuration. All the values were correct.

While troubleshooting with the networking colleague, we tried multiple changes but nothing worked out and I reverted all the manual changes. By this time, I had already invested a lot of time in figuring out the issue but unfortunately, could not figure out problem so I thought to raise AWS tech support ticket rather than investing more time on this issue by myself. This is when suddenly a thought crossed my mind and I decided to give it a try.

The previous day, when for the first time I was making config changes via console, I remember adding new entries under Route Tables tab before adding Authorization Rule for AWS Client VPN. I thought may be it’s the order in which these two settings are applied, it was a wild guess but I decided to act on it. So I delete the Routes and Authorization rules for the non-primary region and add them back in the same order as I did the day before.

Guess what happens? Bingo, It works!! After hours of troubleshooting I can finally connect to internal resources in non-primary region via Client VPN. Before celebrating the victory, I had to add explicit dependency between Authorization Rule and Route Table resources in Terraform and test it one final time.

Image generated using Microsoft Bing

My Thoughts

Even though I was relived and happy figuring out the root cause after hours to troubleshooting, I was also a bit disappointed after realising the reason behind the issue. I feel there should be a validation in place that prevents or warns users while adding Authorization Rule if the respective CIDR block is not already present in the Route Table.

I wonder if this is by design or a bug that hasn’t been flagged yet. What do you guys think?

Understanding the basics

Q: What is the difference between AWS Client VPN and AWS VPN Gateway?
A: AWS Client VPN is a fully-managed service that is used to connect to internal resources either on AWS or on-premise whereas VPN Gateway is used setup connectivity between on-premise and AWS cloud to setup hybrid infrastructure.

Q: What are the limitations of AWS client VPN?
A: Here are some of the limitations:

As of writing this article, Client VPN only supports IPv4 traffic.
The client CIDR cannot be changed after the Client VPN is created.
Client VPN CIDR cannot overlap with VPC CIDR block.
Client VPN is not FIPS compliant.

Above list is not exhaustive. To know about all the limitations and best practices navigate to AWS doc.

Q: What is the difference between AWS Client VPN and OpenVPN?
A: AWS Client VPN is a fully-managed VPN service to interact with services hosted in private network either on-premise or on AWS cloud. In contrast, you need to install OpenVPN on an EC2 instance, configure it, make it HA, patch it periodically yourself. To avoid the administration overhead, you can use AWS Client VPN. AWS Client VPN supports both certificate and federated authentication to seamlessly integrate with your existing IAM infrastructure.

Leveraging Lambda@Edge to seamlessly manage frontend maintenance window like a pro

Vimal Paliwal — Tue, 10 Sep 2024 13:32:58 +0000

Photo by Bermix Studio on Unsplash

Original Source: https://skildops.com/blog/leveraging-lambda-edge-to-seamlessly-manage-frontend-maintenance-window-like-a-pro

In today’s time, zero-downtime deployment is preferred for introducing updates/patches for our applications but exceptions arise and we may need to put on maintenance window before a rollout in certain situations. While working on a project, one such situation occurred and that’s when we decided to implement a simple and quick mechanism to manage maintenance window for our frontend application to deal with such situation.

We decided to use Lambda@Edge rather than CloudFront functions to achieve the end goal. In case you are wondering why we did not opt for CloudFront functions, let’s hold that thought for sometime and come back to that at a later point.

Basically, we used 4 components to achieve this task:

S3: To host our frontend application files
CloudFront: To serve our frontend application
Lambda: To handle maintenance window logic
GitHub Action: To toggle maintenance window

Let’s go through each of the components and understand how we executed it.

S3 Bucket

Let’s setup an S3 bucket to store the files. To keep it simple, we will upload just two files:

home.html
maintenance.html

Fig 1. S3 Bucket

home.html

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Home Page</title>
    <style>
        /* Set height to 100% for both html and body to make centering work */
        html, body {
            height: 100%;
            margin: 0;
            font-family: Arial, sans-serif;
            display: flex;
            justify-content: center; /* Centers horizontally */
            align-items: center; /* Centers vertically */
            background-color: #f0f0f0;
        }

        .welcome-message {
            background-color: #ffffff;
            padding: 20px 40px;
            border-radius: 8px;
            box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2);
            text-align: center;
        }

        h1 {
            margin: 0;
            font-size: 2rem;
            color: #333333;
        }
    </style>
</head>

<body>
    <div class="welcome-message">
        <h1>Welcome to My Home Page!</h1>
    </div>
</body>

</html>

maintenance.html

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Maintenance Mode</title>
    <style>
        html, body {
            height: 100%;
            margin: 0;
            font-family: 'Arial', sans-serif;
            display: flex;
            justify-content: center; /* Center horizontally */
            align-items: center; /* Center vertically */
            background-color: #f8d7da; /* Light red background to indicate maintenance */
            color: #721c24; /* Dark red text color */
        }

        .maintenance-message {
            background-color: #ffffff;
            padding: 30px 50px;
            border-radius: 10px;
            box-shadow: 0 6px 12px rgba(0, 0, 0, 0.3);
            text-align: center;
            max-width: 600px;
        }

        h1 {
            margin: 0;
            font-size: 2.5rem;
        }

        p {
            margin-top: 15px;
            font-size: 1.2rem;
        }
    </style>
</head>

<body>
    <div class="maintenance-message">
        <h1>We're Under Maintenance</h1>
        <p>Our website is currently undergoing scheduled maintenance. We should be back shortly. Thank you for your patience!</p>
    </div>
</body>

</html>

Now, let’s setup the Lambda function that will contain the logic to show maintenance page when it is switched on.

Lambda Function

The logic we are using to display the maintenance page is quite straightforward. If the maintenance-on.html file is present in the bucket, read its content and return it as HTML body. Otherwise, return the original request block.

Before we create the Lambda function, we will need an IAM role that has required permissions to write to CloudWatch logs and read the files from S3 bucket and appropriate trust relationship policy to be associated to CloudFront distribution, so let’s create the role first.

trust-relationship-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "edgelambda.amazonaws.com",
                    "lambda.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

lambda-iam-role-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}

Note: Make sure to update the placeholder BUCKET_NAME in the above IAM policy with your own bucket name.

Important points to consider before creating the lambda function:

It must be in us-east-1 region
And, the architecture must be x86_64

It’s time to create the lambda function using the above IAM role and the below Python code.

import boto3

from botocore.exceptions import ClientError

s3 = boto3.resource('s3')

def lambda_handler(event, context):
    try:
        body = s3.Object('maintenance-mode-demo', 'maintenance-on.html').get()['Body'].read()
        print('maintenance mode is switched on')
        return {
            'status': '200',
            'statusDescription': 'ok',
            'headers': {
                'content-type': [
                    {
                        'key': 'Content-Type',
                        'value': 'text/html'
                    }
                ]
            },
            'body': body
        }
    except ClientError:
        print('maintenance mode is switched off')
        return event['Records'][0]['cf']['request']

Once you have created the Lambda function, publish a version of it by navigating to Versions tab. We will need this while associating lambda function with the CloudFront distribution.

Fig 2. Lambda Function

Next step is to setup a CloudFront distribution to serve the static pages stored in S3 bucket and associate the lambda function to it.

CloudFront

I’m assuming you’re aware about how to set up CloudFront distribution to service a static website, if not you can follow this AWS article. It’s a simple 5 min process.

Fig 3. CloudFront Distribution

Lambda function can be associated either during CloudFront creation time or after it is created. I decided to choose the later option so that you get to learn both the ways.

Associating Lambda Function:

Navigate to Behaviors tab
Select the listed behavior in the table and click the Edit button
Scroll down to the Function associations section
For viewer request, select Lambda@Edge from dropdown, provide published lambda function version ARN, ignore the Include body checkbox and save the changes

We selected viewer request so that as soon as the request is received by the CloudFront distribution, we run the maintenance window logic and take an appropriate action.

Fig 4. Lambda function association

That’s it. We are ready to test our setup.

We won’t be creating GitHub actions workflow in this article because all it does is rename the maintenance.html file to maintenance-on.html to enable the maintenance window and vice-versa.

To begin testing, let’s navigate to CloudFront domain to confirm if we can load the home page. This will also confirm that maintenance mode is currently disabled.

Fig 5. Home Page

Now, let’s head to the S3 bucket and rename maintenance.html file to maintenance-on.html to enable the maintenance window.

Fig 6. S3 Bucket

Now, refresh the CloudFront domain and you should see the maintenance page.

Fig 7. Maintenance Page

Congratulations! You have successfully implemented a mechanism to manage maintenance mode for your frontend application.

Before concluding, let me explain why we chose Lambda@Edge over CloudFront Functions. At the time of writing, CloudFront Functions does not support modifying the response body or importing the boto3 library, which is necessary for interacting with S3.

Lastly, don't forget to delete all the resources if you followed along, as leaving them active could result in additional costs.

Understanding the basics

What is Lambda@Edge?
Lambda@Edge is a Amazon CloudFront feature that enables you to run Lambda functions at edge locations closer to your customers, improving performance and reducing latency.

When to use CloudFront Functions vs Lambda@Edge?
Both CloudFront Functions and Lambda@Edge runs your code at edge location to improve performance but CloudFront Functions can be used only simple operations like modifying headers, redirecting requests, etc using Javascript whereas Lambda@Edge is used to more complex operations where you need to interact with other AWS or third-party services and can be written in either NodeJS or Python.

Does Lambda@Edge have cold start?
Yes, Lambda@Edge is prone to cold start and the size of your Lambda function also plays a vital role in the cold start duration.

Optimise and Secure AWS HTTP API Gateway by locking down direct access

Vimal Paliwal — Wed, 01 May 2024 14:53:33 +0000

Photo by Growtika on Unsplash

Original Source: https://skildops.com/blog/optimise-and-secure-aws-http-api-gateway-by-locking-down-direct-access

AWS HTTP API Gateway let’s you deploy RESTful API quickly in the most affordable way without compromising on basic security, performance, scalability and observability but unlike REST API Gateway lacks many advanced features such as WAF attachment, resource policy, API key management, caching, canary deployments, request body transformation, X-Ray tracing, etc.

AWS WAF is a vital resource to secure publicly exposed endpoints from various types of attacks and because HTTP API Gateway does not support WAF association natively, we need to create a CloudFront distribution and use it as an entry point to the HTTP API Gateway.

Even after implementing the above resources, we aren’t fully secure because if a bad actor can discover our HTTP API Gateway URL, they can bypass WAF and our API becomes highly vulnerable to attacks. Hence, in this article we will learn how to lock down direct access to HTTP API Gateway URL. The implementation is very simple and straightforward so let’s get started.

HTTP API Gateway

Let’s navigate to API Gateway in AWS management console to launch a new HTTP API Gateway and click on the Build button available with the HTTP API container.

Fig 1. API Gateway Dashboard

Upon clicking on the Build button we will be taken to a page that consists multiple steps before we can spin up an API gateway.

Step 1 - Create API:

Click on Add Integration button
Select HTTP as the integration type and GET as the method type
Under URL endpoint, I am using a mock endpoint that I created using beeceptor which returns a fixed response
Finally, give your API gateway a name and click Next

Fig 2. HTTP API - Create API

Step 2 - Configure Routes:

Select GET as method and / as resource path
From the target dropdown, select the Integration that we created in the Step 1 and click Next

Fig 3. HTTP API - Routes

Step 3 - Define stages

We can leave $default as the stage name and auto-deploy enabled and click Next

Fig 4. HTTP API - Define stages

In the last step, review all the inputs and click Create button if everything looks good.

Fig 5. HTTP API - Review

Our API Gateway is now ready to accept incoming connections but we want to use CloudFront as the entry point to our API as it provides a global endpoint rather than a regional endpoint so let’s create a CloudFront distribution.

CloudFront

Let’s navigate to CloudFront console and create a distribution for our API Gateway.

Required values for each parameter is mentioned below per section. Optional fields can be left blank or filled as per your requirement.

Section 1: Origin

Origin domain: API Gateway domain
Protocol: HTTPS
Minimum SSL protocol: TLSv1.2 (default)

Under Add custom header, click on Add header button. We will add a custom header which CloudFront will forward to API Gateway for every request and we will validate the header for every request using Lambda Authorizer that we will create in the next step.

Header name can be x-cf-api-gateway-token and value will be a long random string making it difficult for hackers to guess. It is not mandatory to choose the header name I suggested earlier so feel free to change it as per your convenience.

Section 2: Default cache behaviour

Compress objects automatically: Yes
Viewer protocol policy: Redirect HTTP to HTTPS
Allowed HTTP methods: GET, HEAD
Restrict viewer access: No
Cache policy: CachingDisabled
Origin request policy: AllViewerExceptHostHeader

Note: We can skip the Function associations, WAF and Settings sections completely and stick to the default values for these sections for this demo and create the distribution.

Fig 6. CloudFront Distribution

Let’s also store the above created token in SSM Parameter store as a SecretString so that we can validate it later using Lambda Authorizer.

Fig 7. SSM Parameter

We are almost close to achieving our target. Our last step is to create a Lambda Authorizer for our API Gateway that will validate the token received in the custom header for every request and decide whether to allow the request or not.

Lambda Authorizer

Before we create a Lambda function, we need to create an IAM role that will allow the function to read the SSM parameter that we created earlier.

assume-role-policy-doc.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

aws iam create-role --role-name api-cf-demo-role --assume-role-policy-document file://assume-role-policy-doc.json

Now, let’s create an inline policy that will allow the role to read value from SSM parameter.

ssm-access-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:GetParameter",
            "Resource": "arn:aws:ssm:AWS_REGION:AWS_ACCOUNT_ID:parameter/SSM_PARAMETER_NAME"
        }
    ]
}

aws iam put-role-policy --role-name api-cf-demo-role --policy-name ssm-access --policy-document file://ssm-access-policy.json

Note: Make sure to replace the placeholders in the above policy with their actual values before creating the inline policy.

Along with this inline policy, we also need to attach an AWS managed policy to this role so that lambda can write logs to CloudWatch logs.

aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole --role-name api-cf-demo-role

Fig 8. Lambda IAM Role

Cool, we now have the IAM role that we need for our Lambda function so let’s navigate to Lambda console and create the function.

Feel free to give an appropriate name to the function, select Python 3.11 as the function runtime, x86_64 for the architecture and for execution role select the IAM role that we just created.

Fig 9. Lambda Function

Once the function is created, replace the Lambda code with the below provided code. The below code will read token from the SSM parameter and validate it with the token received from the CloudFront. If both matches, it will return a JSON object that will instruct API Gateway to allow the request.

Next, we need to add an environment variable to the Lambda function so go to Configuration > Environment variables and add a new variable with key as SSM_PARAMETER_NAME and value as the name of SSM Parameter that we created earlier to store the token.

Fig 10. Lambda Environment Variables

Great, our Lambda function is ready to be attached to the API Gateway as an authorizer so let’s navigate back to API Gateway and do the same by following the below steps.

On the API Gateway page, click on Authorization in the left panel and switch to Manage authorizers tab.
Click on Create authorizer and select Lambda as the type.
Give the authorizer a name and select the region you created the lambda function and the lambda function from the dropdown. In case you don’t see a dropdown, start typing the lambda function name and the dropdown should appear.
The Payload format version and Response mode can stay as it is but let’s disable Authorizer caching and remove the pre-filled Identity source.
Under Invoke permissions section, make sure the switch is enabled so that API gateway can automatically create a resource policy and attach it to the lambda function. This policy will allow API gateway to invoke the lambda function.

Fig 11. API Gateway Lambda Authorizer

Note: Sometimes API gateway might not add resource policy to the lambda function and because of this we won't get the desired result hence, before proceeding further go the lambda function and verify if the resource policy is attached to it. In case it isn't, run the following command to attach the required resource policy:
aws lambda add-permission --function-name FUNCTION_NAME --source-arn arn:aws:execute-api:AWS_REGION:AWS_ACCOUNT_ID:API_GW_ID/authorizers/AUTHORIZER_ID --principal apigateway.amazonaws.com --statement-id AllowAPIGatewayAuthorizerAccess --action lambda:InvokeFunction

Fig 12. Lambda Function Resource Policy

Next, let’s attach the authorizer to the route that we create in the beginning so let’s switch to the Attach authorizers to routes tab, select the GET path within the route and attach the lambda authorizer that we just created.

Fig 13. API Gateway Route Authorizer

This completes our implementation of locking down direct access to API gateway and preventing users from bypassing WAF implemented on CloudFront. Let’s test the implementation before saying adios.

It’s very simple to test the implementation. Simply visit both the API Gateway and the CloudFront URL and you will notice that API Gateway throws Forbidden error whereas CloudFront returns a message.

Fig 14. API Gateway Test URL

Fig 15. CloudFront Test URL

Bravo! Mission accomplished!

Covering the basics

How do I protect my HTTP API gateway?
There are various ways to protect AWS HTTP API Gateway such as attaching SSL certificates, using the latest TLS protocol version, applying API requests throttling, using authorizers, associating WAF and so on.

How do I protect my AWS API Gateway from DDoS?
AWS API Gateway by default includes basic DDoS protection which is provided via AWS Shield and you can further uplift the security of your APIs by attaching WAF and for highest level of protection you can subscribe to AWS Shield Advanced.

Which authentication methods can I use to secure API Gateway?
AWS HTTP Gateway supports three different ways for authenticating the requests. You can use IAM authorizer and manage access to the API for the users that can access to AWS account. For wider audience, you can create either a JWT authorizer or a lambda authorizer type for custom authentication logic.