DEV Community: shimo

Manage AWS Lambda Environment Variables with CLI Shell Script

shimo — Fri, 19 May 2023 00:12:05 +0000

In this blog post, I introduce a script that allows you to efficiently manage the environment variables of an AWS Lambda function.

This script provides the capability to:

Register a new variable: Add a new environment variable to the Lambda function.
Update an existing variable: Modify the value of an existing environment variable.
Delete a variable: Remove an environment variable from the Lambda function.

Important Note on Updating Environment Variables

When using the aws lambda update-function-configuration command to update the existing environment variables, it is crucial to pass both the environment variable you want to add or update and all existing environment variables. Failing to do so will result in the deletion of the existing environment variables. The script provided in this blog post takes this into consideration and ensures the proper handling of environment variables during the update process.

Parameters

function_name: The name of the Lambda function.

region: The AWS region where the Lambda function is deployed.

env_vars_upsert: Key-value pairs of environment variables to be upserted (updated or inserted).

During the upsert process, the environment variable will be updated if it exists; otherwise, a new one will be registered.
You can include any number of key-value pairs.

env_vars_upsert=(
  "MY_ENV1" "value1"
  "MY_ENV2" "value2"
)

env_vars_remove: Keys of environment variables to be removed.

If the specified key does not exist, nothing will happen.

env_vars_remove=("MY_ENV3" "MY_ENV4")

Output

The script will response in JSON format. The "Environment" section will display the updated state of the environment variables.

{
    "FunctionName": "xxxx",
    ...

    "Environment": {
        "Variables": {
            "MY_ENV2": "value2",
            "MY_ENV1": "value1"
        }
    },
    ...
}

Summary

I introduced a shell script for managing AWS Lambda environmental variables, enabling users to register, update, and delete variables with ease. This script also takes into consideration the prevention of accidental deletions, ensuring safe and efficient management of your Lambda function's environmental variables.

Caution

Before using this script in a production environment, please create a test Lambda function and test it thoroughly to ensure it works as expected.

StepFunctions Approval Action with One-Time HTML Page Confirmation

shimo — Sun, 07 May 2023 08:19:06 +0000

In this blog post, I will discuss an improved architecture for approval actions using AWS StepFunctions, featuring a one-time HTML page for confirmation. This enhanced design aims to make the approval process more efficient and user-friendly.

Motivation

Previous version

In my previous blog post, (Lambda Function URLs for Approval Actions in StepFunctions), I explained how using Lambda function URLs can eliminate the need for API Gateway.

This works well. But I'm thinking of making it more useful. For example, the email notification contained long URLs (like below), and users could accidentally approve or reject requests with a single click, without any confirmation.

Newer version

This time I added some resources to confirm with multiple steps. Here are the specifications of the new architecture.

First Lambda Function
- Sends an email to the user containing a URL to a one-time HTML page for approval actions.
- Generates a unique HTML page for each approval action, accessible via S3 static hosting.
User Interaction
- Users access the HTML page and click either the "Approve" or "Reject" button, triggering the second Lambda function.
Second Lambda Function
- Reports the approval result to the StepFunctions.
- Deletes the HTML page.

Workflow

Start State Machine: When you start a state machine, an email is sent to the user.

Check email

Access Approval Page: By clicking the link in the email, users are directed to the one-time HTML page containing "Approve" and "Reject" buttons.

Access approval page

Confirmation: After clicking the "Approve" button, a confirmation message box appears.

Confirmation

By selecting OK, you can finally approve!

Summary

In this blog post, I introduced an improved approval action architecture using AWS StepFunctions. By incorporating a one-time HTML page for confirmation, users can avoid making accidental decisions and benefit from a more streamlined approval process.

Note: Please refer to my repository for the CDK code examples to explore this architecture further.

Find AWS Lambda Functions Executed in a Specific Time Range Using Boto3 and CloudWatch Logs

shimo — Sun, 30 Apr 2023 21:41:45 +0000

Motivation

At times, we need to identify which Lambda functions were executed within a specific time range, primarily for troubleshooting or monitoring purposes.

In this blog, I'll share a Python Boto3 script that allows you to search and discover which Lambda functions have execution results in their CloudWatch logs.

(Note: You can also use CloudWatch Logs Insights to query the logs. If the number of target log groups is smaller than the maximum limit of 50, using Logs Insights on the Console might be more convenient.)

Code

Example results

This is the example of the results. I've found that the Lambda function test-my2 was executed twice (2 events) between 2023-04-30 20:10:00+09:00 and 2023-04-30 21:10:00+09:00.

Start. 2023-04-30 21:20:37
===
From 2023-04-30 20:10:00+09:00
To 2023-04-30 21:10:00+09:00
Check 65 log groups... wait...
---
/aws/lambda/test-my2
2 events
===
Finished.  2023-04-30 21:21:17

Appendix

You can also use logGroupNamePattern instead of logGroupNamePrefix. (These are mutually exclusive. ) Note that boto3 version after 1.26.17 is required.

Refer to the boto3 document.

Create selectable PDF files with Lambda Python and ReportLab

shimo — Sun, 23 Apr 2023 23:02:41 +0000

Motivation

There are times when I need to create PDF files for reporting from data points using an AWS Lambda function.

This blog post was inspired by a blog post from Classmethod (in Japanese), which demonstrates how to create PDF files with a Lambda function using WeasyPrint + Jinja. In this post, I want to share an alternative approach to create PDF files with Python Lambda using the ReportLab library.

Here is the image of the PDF file.

(Note: We want the text in the PDF files to be selectable and copyable, so we won't be using Matplotlib this time.)

Architecture

Step by step deployment

Note: Python 3.9

Create S3 Bucket and SNS Topic

Create the S3 Bucket
- The default setting is OK. (No public access).
- your-bucket-name will be used later.
- aws-doc
Create the SNS Topic
- arn of the topic will be used later.
- aws-doc

Create Lambda Layer

Start from any directory on your terminal.

Install the reportlab library to a directory named python (Note: the name must be python).

pip install reportlab -t python

Zip python directory with the name my_lambda_layer.zip (Note: arbitrary name).

zip -r my_lambda_layer.zip python

Upload this zip file in the step of creating a layer.

You will use the arn of the layer later.

Create Lambda function

On the Lambda console, Create Lambda (Python).

Configuration

General configuration
- The default 3 sec timeout might be short. Change it a little longer, like 10 sec.
Environmental variables
- SNS_ARN: arn of the topic you have
- BUCKET_NAME: name of the bucket you have
IAM Policy
- Select Permission, and Role name. Add an inline policy like the one below for sending SNS and uploading a file to S3. Change the arn of your SNS topic and S3 bucket name.
- Make sure you don't change the IAM policy while your pre-singed URL will be used.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sns:Publish",
            "Resource": "arn:aws:sns:REGION:ACCOUNT_ID:your-sns-topic-arn"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name/*",
                "arn:aws:s3:::your-bucket-name"
            ]
        }
    ]
}

Code

Add Lambda Layer

In the Code tab, add the Layer you've created. Specify the ARN of the layer you've created.

Write code

In this code
- Percentage of the year passed and left days are calculated.
- The gauge for the percentage is generated with draw_gauge().
- Create an A4 PDF file. Upload it to the S3 bucket.
- Set the pre-assigned URL to the file in the bucket
- Send the URL with SNS.
As the unit I used mm. inch is available by changing unit = mm to inch, but some modifications are needed with x, y.
on_grid parameter in create_pdf_days_passed_left() is for designing layouts. (See Appendix)

import io
import os
import uuid
from datetime import datetime

import boto3
from reportlab.lib import colors
from reportlab.lib.pagesizes import A4, landscape
from reportlab.lib.units import inch, mm
from reportlab.pdfgen import canvas


def percentage_of_year_passed(start_of_year, end_of_year, now):
    total_days = (end_of_year - start_of_year).days + 1
    days_passed = (now - start_of_year).days + 1
    percentage_passed = int((days_passed / total_days) * 100)
    return percentage_passed


def calculate_days():
    now = datetime.now()
    start_of_year = datetime(now.year, 1, 1)
    end_of_year = datetime(now.year, 12, 31, 23, 59, 59)

    percentage_passed = percentage_of_year_passed(start_of_year, end_of_year, now)
    days_left = (end_of_year - now).days + 1

    return percentage_passed, days_left


def draw_grid(c, width, height, unit):
    step = 25 if unit == mm else 1
    c.setLineWidth(1 / unit)
    c.setStrokeColor(colors.grey)
    c.setFont("Helvetica", 25 / unit)

    for i in range(0, int(height) + 1, step):
        c.line(0, i, width, i)
        c.drawString(0, i, f"{i}")

    for i in range(0, int(width) + 1, step):
        c.line(i, 0, i, height)
        c.drawString(i, 0, f"{i}")


def draw_gauge(c, x, y, radius, percentage, base_color, fill_color):
    """
    Draw gauge with two arches: base 180 deg arch and filled with percentage.
    """
    # Draw base half-circle
    c.setLineWidth(15)
    c.setStrokeColor(base_color)
    c.arc(x - radius, y - radius, x + radius, y + radius, 180, -179.9)

    # Draw filled half-circle
    c.setStrokeColor(fill_color)
    gauge = -180 * percentage / 100 + 0.01
    c.arc(x - radius, y - radius, x + radius, y + radius, 180, gauge)


def draw_str(c, x, y, item, font_size, font_color="black"):
    c.setFont("Helvetica", font_size)
    c.setFillColor(font_color)
    c.drawCentredString(x, y, f"{item}")


def create_pdf_days_passed_left(page_size, x, y, radius, on_grid = False):

    unit = mm  # mm or inch
    buffer = io.BytesIO()
    c = canvas.Canvas(buffer, pagesize=page_size)
    c.scale(unit, unit)

    if on_grid:
        width = page_size[0] / unit
        height = page_size[1] / unit
        draw_grid(c, width, height, unit)

    base_color = colors.HexColor("#c8c8c8")  # light grey
    fill_color = colors.HexColor("#1f77b4")  # light blue

    percentage_passed, days_left = calculate_days()

    message = f'Created at {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}'
    draw_str(c, x + 50, y + 55, message, 8)
    draw_gauge(c, x, y, radius, percentage_passed, base_color, fill_color)
    draw_str(c, x, y, f"{percentage_passed}%", font_size=20)
    draw_str(c, x, y - 25, "This year passed", font_size=10)

    draw_str(c, x + 100, y, days_left, font_size=20, font_color="green")
    draw_str(c, x + 100, y - 25, "days left", font_size=10)

    c.showPage()
    c.save()
    buffer.seek(0)
    return buffer


def generate_n_char_id(n: int):
    unique_id = uuid.uuid4()
    short_id = str(unique_id)[:n]
    return short_id


def generate_presigned_url(bucket_name, object_name, expiration_sec=3600):
    s3_client = boto3.client("s3")
    response = s3_client.generate_presigned_url(
        "get_object",
        Params={"Bucket": bucket_name, "Key": object_name},
        ExpiresIn=expiration_sec,
    )

    return response


def send_sns_message(topic_arn, message):
    sns_client = boto3.client("sns")
    response = sns_client.publish(
        TopicArn=topic_arn,
        Message=message,
    )

    return response


def lambda_handler(event, context):
    s3 = boto3.client("s3")
    bucket_name = os.environ["BUCKET_NAME"]
    sns_topic_arn = os.environ["SNS_ARN"]

    dt = datetime.now().strftime("%Y%m%d-%H%M%S")
    uuid = generate_n_char_id(8)
    filename = f"demo-{dt}-{uuid}.pdf"

    page_size = landscape(A4)
    x, y = 100, 125  # Center of gauge. Use this point as anchoring
    radius = 40  # radius of gauge

    pdf_buffer = create_pdf_days_passed_left(page_size, x, y, radius)
    s3.upload_fileobj(pdf_buffer, bucket_name, filename)

    url = generate_presigned_url(bucket_name, filename, expiration_sec=3600)
    if url:
        print(f"Generated presigned URL: {url}")
        message = f"Download the PDF file here: {url}"
        send_sns_message(sns_topic_arn, message)
    else:
        print("Failed to generate presigned URL")

    return {
        "statusCode": 200,
        "body": "PDF created and uploaded to S3. Presigned url has sent with SNS.",
    }

Result

You'll receive an email to download URL.

Summary

I've demonstrated creating a PDF file using a Lambda Python function, using Reportlab library.

Appendix

This is the example PDF file with unit = mm and grid_on = True, useful for designing the page. The dimension of the A4 landscape is 210 x 297 mm.

Receive Slack Notification of CodePipeline with SNS and Lambda

shimo — Mon, 27 Mar 2023 22:20:58 +0000

Motivation

I want to receive Slack notifications from CodePipeline through Amazon SNS and Lambda. By using the Lambda function, formatting and filtering of messages are available. (Therefore, not Chatbot.) In this blog, I'll share how to achieve this.

Example of Slack notification:

(Note: This is useful for other architectures' notifications if SNS and Lambda are used.)

Create Slack

Create Webhook
- Ref: This post describe how to use webhooks in Lambda function.
Memo the webhook URL for Lambda environmental variables

Create Lambda

Create a Lambda function of Py3.9
Set Lambda environmental variables
- Key: WEBHOOK_URL_SLACK, Value: https://hooks.slack.com/services/xxxxxxx
In this Python script, get an event from SNS.
- If the CodePipeline status is SUCCEED, do nothing.
- If the CodePipeline status is FAIL, send a message to Slack.
Permission: BasicExecutionRole will be OK.

import json
import os

import urllib3

http = urllib3.PoolManager()

WEBHOOK_URL_SLACK = os.environ["WEBHOOK_URL_SLACK"]


def gen_message_from_codepipeline_event(event_dict):
    """
    Return message according to the CodePipeline state.
    """

    # No message when SUCCEEDED
    if event_dict["detail"]["state"] == "SUCCEEDED":
        return ""

    if event_dict["detail"]["state"] == "FAILED":
        message = "CodePipeline state: FAILED"

        failed_stage = event_dict.get("additionalAttributes", {}).get("failedStage")
        stage_info = (
            f"Failed Stage: {failed_stage}"
            if failed_stage
            else "Failed Stage: N/A"
        )
        message += f"\n{stage_info}"

        failed_actions = event_dict.get("additionalAttributes", {}).get("failedActions")
        if failed_actions:
            # Send only the last try info
            info_last_action = failed_actions[-1]['additionalInformation']
            message += f"\nInformation: {info_last_action}"

        return message


def lambda_handler(event, context):
    """
    Handle CodePipeline notifications and send messages to Slack.
    """

    try:
        event_str = event["Records"][0]["Sns"]["Message"]
    except (KeyError, IndexError):
        print("Error: Event is missing required data")
        return

    event_dict = json.loads(event_str)

    # generate message
    message = gen_message_from_codepipeline_event(event_dict)
    if not message:
        print({"statusCode": 200, "body": "No message to return."})
        return
    region = event_dict["region"]
    pipeline = event_dict["detail"]["pipeline"]
    pipeline_url = f"https://{region}.console.aws.amazon.com/codesuite/codepipeline/pipelines/{pipeline}/view?region={region}"

    # Send Slack webhook
    text = f"{message}\n<{pipeline_url}|Visit CodePipeline>"
    msg = {
        "text": text,
    }
    encoded_msg = json.dumps(msg).encode("utf-8")
    resp = http.request("POST", WEBHOOK_URL_SLACK, body=encoded_msg)
    print({"statusCode": resp.status, "body": "Send message."})

    return

Create CodePipeline

You can use any CodePipeline. If you want a simple sample, you can use a CloudFormation template I've created from the AWS document with the simple S3 and EC2 pipeline. Upload the zip file found in the section of this AWS document.
If you want to deploy to Windows Server instances using CodeDeploy, download the sample application here: SampleApp_Windows.zip.

Create notification

AWS doc is here.

On the pipeline page, select "Notify".
- Create notification rules
- Set Notification name
- Detail type: Full
- Select events: Pipeline execution Failed, Succeeded
- Targes
- Create target: SNS Topic
- Use existing SNS Topic --> Need to set access policy

My settings is as below.

As a target, the SNS topic is used here. When you use an existing SNS Topic, you may need to add access policy. *The Principal is codestar-notifications.amazonaws.com. *

If you create a target topic here, the access policy is included.

  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codestar-notifications.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:REGION:ACCOUNT_ID:TOPIC_NAME"
    },

Subscribe the Lambda function to this SNS Topic.

Run and fail

Fail scenarios:

Start Pipeline without the zip file in the proper S3 bucket.
Start Deploy section with EC2 instances are stopped.

Then you'll receive the Slack notification.

Note that the Information part is from the additionalInformation in the JSON message. And it appears on the console like the below image.

Summary

I've demonstrated how to receive Slack notifications from CodePipeline.

AWS Budgets: Update alert thresholds unlimitedly with Lambda

shimo — Tue, 27 Dec 2022 23:21:29 +0000

Motivation

When I use AWS Budgets, I can receive notifications when the billing is above a threshold.

The facts are that:

We can set only 10 alert thresholds for one budget.
Two budgets are free but we are charged for using more budgets.

For personal use, usually, I'm not charged so much (around 10 USD per month). I want to be notified every 1 USD but there is a limitation of 10 alerts like above fact.

I this post, I share how to update the alert threshold incremently when AWS Budgets triggers.

Note from the pricing page:

Your first two action-enabled budgets are free (regardless of the number of actions you configure per budget) per month. Afterwards each subsequent action-enabled budget will incur a $0.10 daily cost.

Architecture

EventBridge triggers Lambda(setBudget) at the 1st of months. This Lambda delete and re-create the budget to initialize for a month and create 10 alert thresholds: 1, 2, 3, ... 10 USD thresholds, for example.
When the billing is above the threshold, the user is notified via SNS email. At the same time, the second Lambda function is triggered and updates the threshold. If the triggered threshold is 1 USD, the Lambda function update the threshold to 11 USD.

CloudFormation template

Deploy from the Console. Create stack -> Upload a template -> Choose this file.
Enter Stack name and parameters: budget name, email, increment.
Leave the rest default. Choose Next, Next, check IAM acknowledge, and Submit.
When starting in the middle of the month, manually run the SetBudgetHandler Lambda function. Note that you'll get notified of all alerts of the current used cost.

AWSTemplateFormatVersion: "2010-09-09"
Description: "Increment alert threshold of AWS Budgets"
Parameters:
  budgetname:
    Type: String
    Default: increment-notification
  increment:
    Type: String
    Default: 1
  email:
    Type: String
    AllowedPattern: "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"
    ConstraintDescription: Must be a valid email address.
Resources:
  MyTopic:
    Type: AWS::SNS::Topic
  MyTopicTokenSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Protocol: email
      TopicArn: !Ref MyTopic
      Endpoint: !Ref email
  MyTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      PolicyDocument:
        Statement:
          - Action: sns:Publish
            Effect: Allow
            Principal:
              Service: budgets.amazonaws.com
            Resource: !Ref MyTopic
            Sid: "0"
        Version: "2012-10-17"
      Topics:
        - !Ref MyTopic
  UpdateBudgetHandlerServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  UpdateBudgetHandlerServiceRoleDefaultPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action: budgets:*
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: UpdateBudgetHandlerServiceRoleDefaultPolicy
      Roles:
        - !Ref UpdateBudgetHandlerServiceRole
  UpdateBudgetHandler:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          import os
          import re

          import boto3

          budget_name = os.environ['BUDGET_NAME']
          increment = os.environ['INCREMENT']

          def handler(event, context):
              print(event)
              client = boto3.client('budgets')
              account_id = context.invoked_function_arn.split(":")[4]
              print("account_id", account_id)

              message = event['Records'][0]['Sns']['Message']
              print("message", message)

              # catch 1 from "Alert Threshold: > $1.00"
              match = re.search(r"Alert Threshold: > \$(\d{1,})\.00", message)
              if not match:
                  print("No Budget Notification. Exit.")
                  return

              current_value_str = match.groups()[0]
              print("current_value", current_value_str)

              current_value = int(current_value_str)
              next_value = current_value + 10 * int(increment)

              response = client.update_notification(
                  AccountId=account_id,
                  BudgetName=budget_name,
                  OldNotification={
                      "NotificationType": "ACTUAL",
                      "ComparisonOperator": "GREATER_THAN",
                      "Threshold": current_value,
                      "ThresholdType": "ABSOLUTE_VALUE",
                  },
                  NewNotification={
                      "NotificationType": "ACTUAL",
                      "ComparisonOperator": "GREATER_THAN",
                      "Threshold": next_value,
                      "ThresholdType": "ABSOLUTE_VALUE",
                  },
              )

      Role: !GetAtt UpdateBudgetHandlerServiceRole.Arn
      Environment:
        Variables:
          BUDGET_NAME: !Ref budgetname
          INCREMENT: !Ref increment
      Handler: index.handler
      Runtime: python3.9
      Timeout: 10
    DependsOn:
      - UpdateBudgetHandlerServiceRoleDefaultPolicy
      - UpdateBudgetHandlerServiceRole
  UpdateBudgetHandlerEventInvokeConfig:
    Type: AWS::Lambda::EventInvokeConfig
    Properties:
      FunctionName: !Ref UpdateBudgetHandler
      Qualifier: $LATEST
      MaximumRetryAttempts: 0
  UpdateBudgetHandlerAllowInvokeStackMyTopic:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt UpdateBudgetHandler.Arn
      Principal: sns.amazonaws.com
      SourceArn: !Ref MyTopic
  UpdateBudgetHandlerMyTopic:
    Type: AWS::SNS::Subscription
    Properties:
      Protocol: lambda
      TopicArn: !Ref MyTopic
      Endpoint: !GetAtt UpdateBudgetHandler.Arn
  SetBudgetHandlerServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  SetBudgetHandlerServiceRoleDefaultPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action: budgets:*
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: SetBudgetHandlerServiceRoleDefaultPolicy
      Roles:
        - !Ref SetBudgetHandlerServiceRole
  SetBudgetHandler:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          import os
          import boto3

          budget_name = os.environ['BUDGET_NAME']
          sns_arn = os.environ['SNS_ARN']
          increment = os.environ['INCREMENT']

          client = boto3.client("budgets")


          def handler(event, context):
              account_id = context.invoked_function_arn.split(":")[4]

              # Delete if the budget exists
              try:
                  client.delete_budget(AccountId=account_id, BudgetName=budget_name)
                  print("Deleted the old budget.")
              except client.exceptions.NotFoundException:
                  pass

              # Create new budget
              client.create_budget(
                  AccountId=account_id,
                  Budget={
                      "BudgetName": budget_name,
                      "BudgetLimit": {"Amount": "100.0", "Unit": "USD"},
                      "CostTypes": {
                          "IncludeTax": True,
                          "IncludeSubscription": True,
                          "UseBlended": False,
                          "IncludeRefund": False,
                          "IncludeCredit": False,
                          "IncludeUpfront": True,
                          "IncludeRecurring": True,
                          "IncludeOtherSubscription": True,
                          "IncludeSupport": True,
                          "IncludeDiscount": True,
                          "UseAmortized": False,
                      },
                      "TimeUnit": "MONTHLY",
                      "BudgetType": "COST",
                  },
                  NotificationsWithSubscribers=[
                      {
                          "Notification": {
                              "NotificationType": "ACTUAL",
                              "ComparisonOperator": "GREATER_THAN",
                              "Threshold": threshold,
                              "ThresholdType": "ABSOLUTE_VALUE",
                              "NotificationState": "OK",
                          },
                          "Subscribers": [
                              {"SubscriptionType": "SNS", "Address": sns_arn},
                          ],
                      }
                      for threshold in range(1, 1 + 10 * int(increment), int(increment))
                  ],
              )
              print("Created a new budget.")
      Role: !GetAtt SetBudgetHandlerServiceRole.Arn
      Environment:
        Variables:
          BUDGET_NAME: !Ref budgetname
          SNS_ARN: !Ref MyTopic
          INCREMENT: !Ref increment
      Handler: index.handler
      Runtime: python3.9
      Timeout: 10
    DependsOn:
      - SetBudgetHandlerServiceRoleDefaultPolicy
      - SetBudgetHandlerServiceRole
  SetBudgetHandlerEventInvokeConfig:
    Type: AWS::Lambda::EventInvokeConfig
    Properties:
      FunctionName: !Ref SetBudgetHandler
      Qualifier: $LATEST
      MaximumRetryAttempts: 0
  ScheduleRule:
    Type: AWS::Events::Rule
    Properties:
      ScheduleExpression: cron(0 0 1 * ? *)
      State: ENABLED
      Targets:
        - Arn: !GetAtt SetBudgetHandler.Arn
          Id: Target0
  ScheduleRuleAllowEventRuleStackSetBudgetHandler:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt SetBudgetHandler.Arn
      Principal: events.amazonaws.com
      SourceArn: !GetAtt ScheduleRule.Arn

Summary

I've shared how to update AWS Budgets threshold incrementally.

Suspend any Lambda functions using a single CloudWatch Alarm

shimo — Mon, 19 Dec 2022 23:24:24 +0000

(Note: This post does not include the update of CloudWatch alarm in 2022/Dec.)

Motivation

When using Lambda functions, there are some risks that too many executions are invoked unintentionally. For example, self-looping of the Lambda functions or DDoS attack on the Lambda function URLs. A user in Japan reported the incident with 2.8 billion invocations in 10 days (Japanese blog).

So it would be nice to set some thresholds to suspend the uncontrollable functions. CloudWatch can detect such behaviors and trigger alarms. Surely this works, but we usually need to set one alarm for one Lambda function. The alarm costs 0.1 USD per month per alarm. If we use many Lambda functions, it should be costly.

In this post, I share the idea that a single CloudWatch Alarm takes care of all of the Lambda functions in the region. (So, 0.1 USD per month!)

How it works

The image below shows the steps of this architecture.

Suppose Lambda(s) is running repeatedly too much. Here we don't care which function or how many functions are involved. Only the sum of invocations is used as the metric.
When the sum of invocations reached first threshold, CloudWatch Alarm is triggered. This alarm tells that "SUM of the invocation reached threshold."
Lambda-throttle is invoked by CloudWatch alarm. This function queries which functions and how many times were invoked at a high rate.
When the invocations of the Lambda functions are beyond second threshold, Lambda-throttle sets the concurrency to 0 of them.
Lambda-throttle sends a message to the user via Amazon SNS.

Code

(Find complete CDK code in my repository.)

First, this is the snapshot of the CloudWatch alarm setting, which is just a normal setting with AWS/Lambda namespace. The first threshold for the sum of invocations is set to 100 this time.

Next, let's see the Lambda code.

The second threshold is used for determining whether to suspend each function or not.
The most fun part of this post is MetricDataQueries. In this query, Lambda query with Metrics Insights sum of invocations for each lambda functions. (I write twice because it's important: Not sum of all functions, but the sum of each function.)
- Query range is 5 minutes. Counts for every 1 minutes are obtained for this range.
- Option: Adding "LIMIT 10", for example, in the Expression narrows the result.
After the metrics query, compare the sum of invocations for a function and the second threshold (threshold_lambda_stop).
When suspending the function, put_function_concurrency with ReservedConcurrentExecutions=0 works.
("Failed to throttle." is just a verbose part in the case.)

import os
from datetime import datetime, timedelta

import boto3


def send_sns(message, subject):
    client = boto3.client("sns")
    topic_arn = os.environ["SNS_ARN"]
    client.publish(TopicArn=topic_arn, Message=message, Subject=subject)


def get_invocation_top_functions():
    """
    Check which functions are invoked many times
    """
    range_minutes = 5
    cloud_watch = boto3.client("cloudwatch")

    response = cloud_watch.get_metric_data(
        MetricDataQueries=[
            {
                "Id": "q1",
                "Expression": """
                    SELECT SUM(Invocations)
                    FROM SCHEMA(\"AWS/Lambda\", FunctionName)
                    GROUP BY FunctionName
                    ORDER BY SUM() DESC
                    """,
                "Period": 60,
                "Label": "Invocation top",
            },
        ],
        StartTime=datetime.now() - timedelta(minutes=range_minutes),
        EndTime=datetime.now(),
    )

    return response


def handler(event, context):

    threshold_lambda_stop = int(os.environ["THRESHOLD_LAMBDA_STOP"])

    response = get_invocation_top_functions()

    # Count invocation in range_minutes for each function
    # If the count is more than threshold, throttle the function
    for fn in response["MetricDataResults"]:
        count = sum(fn["Values"])
        fn_name = fn["Label"].split()[-1]

        if count >= threshold_lambda_stop:
            client = boto3.client("lambda")
            response = client.put_function_concurrency(
                FunctionName=fn_name, ReservedConcurrentExecutions=0
            )

            # Notify
            if response["ResponseMetadata"]["HTTPStatusCode"] == 200:
                message = f"Lambda: {fn_name} was throttled. Count in 5 minutes: {count}."
                subject = "Lambda throttled."
                send_sns(message, subject)
            else:  # Verbose
                message = f"Failed throttling Lambda: {fn_name}. Count in 5 minute: {count}."
                subject = "Failed to throttle."
                send_sns(message, subject)

Try

For a test, I've set first threshold in the CloudWatch alarm to 10, and the second alarm to 5. Then manually run a Lambda function more than 10 times in a minute.

I received an SNS message like this.

Note

When testing, make sure not to suspend your critical Lambda functions.
As the resources are region specific, one alarm for a region is required.

Summary

I have shared how to detect and throttle all Lambda functions in a single CloudWatch Alarm.

Setting two thresholds properly is quite essential. One is for catching the sum of the invocations, and the other is for determining to suspend each function. They depend on your system.

Appendix

Find complete CDK code in my repository.

Trigger precisely at hh:mm:00 with EventBridge and Lambda

shimo — Sat, 26 Nov 2022 03:40:40 +0000

Motivation

In Amazon EventBridge Rules and EventBridge Scheduler, the precision of time is one minute. This is the same as the usual cron jobs.

... the minimum precision for a schedule is one minute. Your scheduled rule runs within that minute, but not on the precise 0th second. (aws-doc)

Sometimes we want to trigger precisely at zero seconds, for example, at 00:00:00.

In this post, I share the way to trigger precisely using the Lambda function with the EventBridge Schedule.

Architecture

EventBridge invokes the Lambda function 1 minute before the desired trigger time 00:00:00. That means the EventBridge Schedule is set to invoke the Lambda function at 23:59. As written above, we don't know what seconds in 23:59 the time Lambda function will be invoked. (It might be 23:59:16 or 23:59:38...)
The Lambda function calculates the time left to 00:00:00 and sleeps until then.
At 00:00:00, the Lambda function triggers some actions, events or resources.

Lambda function

Here is the example code for the Python Lambda function. You can just create a new function and paste this.

Note that the timeout of the Lambda function should be more than 1 minute.

import time
from datetime import datetime, timedelta


def lambda_handler(event, context):

    time_called = datetime.now()
    print("time_called", time_called)

    # ceiling minute
    target = time_called - timedelta(
        minutes=-1,
        seconds=time_called.second,
        microseconds=time_called.microsecond
    )

    diff = target - time_called
    wait_sec = diff.seconds + diff.microseconds / 1000000
    time.sleep(wait_sec)

    time_zero_sec = datetime.now()
    print("time_zero_sec", time_zero_sec)

    # Write here what you want to trigger at xx:xx:00.

Two print() are used to compare the times before and after wait.

Result

Let's try and see the result. At this time, desired time is 23:00. EventBridge Scheduler was set to invoke the Lambda function at 22:59.

Here is the capture of CloudWatch logs of the Lambda function.

22:59:16.345072 is from first print(), when EventBridge triggered the Lambda function.
2022-11-25 23:00:00.044251 is from second print(), when the Lambda function waited until 23:00:00 as intended.

Good! This is what I wanted to do here.

Summary

I've shared how to trigger events precisely in second order using EventBridge and Lambda.

Note

There is a parameter at expression at(yyyy-mm-ddThh:mm:ss) in EventBridge Scheduler cli/boto3, but ss part does not work.

DEV Community: shimo

Manage AWS Lambda Environment Variables with CLI Shell Script

Parameters

Output

Summary

Caution

StepFunctions Approval Action with One-Time HTML Page Confirmation

Motivation

Previous version

Newer version

Workflow

Check email

Access approval page

Confirmation

Summary

Find AWS Lambda Functions Executed in a Specific Time Range Using Boto3 and CloudWatch Logs

Motivation

Code

Example results

Appendix

Create selectable PDF files with Lambda Python and ReportLab

Motivation

Architecture

Step by step deployment

Create S3 Bucket and SNS Topic

Create Lambda Layer

Create Lambda function

Configuration

Code

Add Lambda Layer

Write code

Result

Summary

Appendix

Receive Slack Notification of CodePipeline with SNS and Lambda

Motivation

Create Slack

Create Lambda

Create CodePipeline

Create notification

Subscribe

Run and fail

Summary

AWS Budgets: Update alert thresholds unlimitedly with Lambda

Motivation

Architecture

CloudFormation template

Summary

Suspend any Lambda functions using a single CloudWatch Alarm

Motivation

How it works

Code

Try

Note

Summary

Appendix

Trigger precisely at hh:mm:00 with EventBridge and Lambda

Motivation

Architecture

Lambda function

Result

Summary

More to read

Note