DEV Community: Jason Shen

Monitoring SES Email Events though Configuration Set with SNS, SQS, Lambda and CloudWatch Log Groups

Jason Shen — Fri, 01 Sep 2023 11:28:03 +0000

When you are sending emails through Amazon SES service, you need to track sending events like Bounce and Complaint. You can use those events to adjust your mail list and eventually maintain reputation of your SES account, so you don't have to face unwanted status of your SES account like sending pause and review situation.

Amazon SES provides sending events through email feedback and event notification. When Configuration Set feature is available, now you can publish more events to multiple destinations. For example, you can publish Subscription event to CloudWatch metric.

In this article, I will show you how to set up a serverless workflow to monitor sending events of your SES account by using the following AWS services.

Amazon SES
Amazon Simple Notification Service (SNS)
Amazon Simple Queue Service (SQS)
AWS Lambda
Amazon CloudWatch log groups

1#. Follow SES document "Set up email sending with Amazon SES " and setup your SES account. If your SES account is in sandbox, you will need to verify both sender and recipient email addresses.

2#. Follow SES document "Creating configuration sets in SES" and create a Configuration Set.

3#. Follow SQS document "Create a queue (console)" and create SQS queue with standard queue type.

4#. Follow SNS document "Getting started with Amazon SNS" and create SNS topic.

5#. Create a SNS subscription with Protocol type "SQS", put SQS ARN of SQS queue created in step 3.

6#. Update Access Policy of SQS queue to allow SNS topic to publish events.

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AWS_Account_ID>:root"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-west-2:<AWS_Account_ID>:<SQS_Queue_Name>"
    },
    {
      "Sid": "__sender_statement",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-west-2:<AWS_Account_ID>:<SQS_Queue_Name>"
    }
  ]
}

7#. Create a Lambda function using Python version > 3.9

The function will be triggered by SQS queue and store events into CloudWatch log groups based on SES message ID. The function will create CloudWatch log group if not existing.

import json
import boto3
from datetime import datetime
from time import time
import random

def lambda_handler(event, context):
    #print(event)
    for record in event['Records']:
        print("record")
        payload = record["body"]
        print(payload)
        # get lambda running region
        region = context.invoked_function_arn.split(":")[3]
        # get lambda running version
        version = context.function_version
        # get lambda running name
        name = "ses-notification-" + region
        mail = json.loads(payload)['mail']
        #print("mail")
        #print(mail)
        messageID = mail['messageId']
        print(messageID)
        logGroupNamePrefix="/aws/lambda/" + name
        # define CloudWatch log group client in the region
        client = boto3.client('logs', region_name=region)
        # check cloudwatch log group with name "ses-notification-cw-logs" existence
        response = client.describe_log_groups(logGroupNamePrefix=logGroupNamePrefix)
        # if log group not found, create it
        if len(response["logGroups"]) == 0:
            client.create_log_group(logGroupName=logGroupNamePrefix)
            print("Log group created")
        # get current date in form as year/month/day
        currentDay = str(datetime.now().day)
        currentMonth = str(datetime.now().month)
        currentYear = str(datetime.now().year)
        date = currentYear + "/" + currentMonth + "/" + currentDay
        logStreamsName = date + "/[" + version + "]" + messageID
        # create CloudWatch log stream
        responseLogStream=client.describe_log_streams(logGroupName=logGroupNamePrefix,logStreamNamePrefix=logStreamsName)
        if len(responseLogStream["logStreams"]) == 0:
            client.create_log_stream(logGroupName=logGroupNamePrefix, logStreamName=logStreamsName)
            print("Log stream created")
        # put payload into CloudWatch log stream with current timestamp
        #payload = "this is test payload"
        client.put_log_events(logGroupName=logGroupNamePrefix, logStreamName=logStreamsName, logEvents=[{"timestamp": int(time() * 1000), "message": payload}])
        print("Payload put into log stream")

8#. The IAM role of the Lambda function must have the following permission in IAM managed policy.

CloudWatchLogsFullAccess
AWSLambdaSQSQueueExecutionRole

9#. In the SQS queue, you need to follow "Configuring a queue to trigger an AWS Lambda function (console)" and add the Lambda function as trigger.

10#. Create Event Destination in Configuration Set created in step 2 by following "Set up an Amazon SNS event destination for event publishing.

11#. Now you can send an email by using SMTP, AWS CLI or API through your SES account. Remember that you need to specify the Configuration Set in your sending action.

Amazon S3 - Web Based Upload Object with POST request and Presigned URL in Python Boto3

Jason Shen — Tue, 22 Aug 2023 13:21:41 +0000

In this article, I will show you how to generate S3 Presigned URL for HTTP POST request with AWS SDK for Boto3(Python). The unique part of this article is that I will show you how to apply Server Side Encryption with KMS key, Tagging objects, Updating Object Metadata and more with S3 Presigned URL for HTTP POST.

When using S3, there is a scenario about "Broswer-Based Uploads Using HTTP POST". However, it is required to calculate AWS SigV4 Signature to follow the section.

Instead of calculating the signature by you own codes, you can actually use AWS Boto3 SDK with method "generate_presigned_post" to generate S3 PreSigned URL. This is not only saving your time to debug "Signature Mismatch" error with your own codes, you don't have to figure out requirements of crypto modules used by your codes to generate right signature. It will all be handled by AWS SDK.

For example, you owns an S3 bucket in your account. One customer of yours is running a business to allow users of the customer to upload images. The images will be directly uploaded from the customer's website into your S3 bucket. The customer is not familiar with Amazon S3 service and does not own an AWS account, so you need to provide your customer an easy method uploading objects from the customer website directly into your S3 bucket. At the meantime, you don't need to make your bucket public for uploading objects.

This is where S3 Presigned URL is needed. You can generate the S3 Presigned URL for HTTP POST from AWS Lambda function by having these benefits. Then you can provide the S3 Presigned URL with your customer to integrate into the customer's website.

But you might ask this question:

Why are you not using S3 Presigned URL for PutObject API call?

S3 Presigned URL for HTTP POST from broswer-based uploads provides a unique feature. You can define "starts-with" condition in the policy. You and your customers can both have some controls on requirements of the uploaded objects.

For example, you only want your customer to upload text files, so you can use the following "start-with" condition to restrict value of "Content-Type" starting with "plain" in uploading request. The uploading request is created from your customer's website. The value of "Content-Type" request header is set when a file is being uploaded from your customer's website by using your S3 Presigned URL.

["starts-with", "$Content-Type", "plain"],

In document of AWS SDK for Boto3, it did not share much information regarding how to use "Fields" and "Conditions" parameters mentioned at "generate_presigned_post". It took me some time to figure it out, so I added my understanding in the code example.

I hope they will save your time in your code development.

Here is the Python Code Example. Before you test it, you will need to update the constants to match your resources.

import boto3
import requests
from botocore.config import Config

ACCESS_KEY="AKIAIOSFODNN7EXAMPLE"
SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

BUCKET_NAME="example-bucket-name"
OBJECT_NAME="example-key-name"
REGION_LOCATION="ap-southeast-2"

KMS_KEY_ARN="arn:aws:kms:<region>:<account-id>:key/<key-id>"

EXPIRATION_TIME = 60*60 * 12 # 12 hours

TEST_FILE_NAME="/Absolute/Path/To/Local/FileName"

my_config = Config(
    region_name = REGION_LOCATION,
    signature_version = 'v4',
    retries = {
        'max_attempts': 10,
        'mode': 'standard'
    }
)

# define S3 client in ap-southeast-2 region
s3=boto3.client('s3',
                 aws_access_key_id=ACCESS_KEY,
                 aws_secret_access_key=SECRET_ACCESS_KEY,
                 config=my_config)


fields={
    "tagging": "<Tagging><TagSet><Tag><Key>type</Key><Value>test</Value></Tag></TagSet></Tagging>",
    "x-amz-storage-class": "STANDARD_IA",
    "Cache-Control": "max-age=86400",
    "success_action_status": "200",
    "x-amz-server-side-encryption": "aws:kms",
    "x-amz-server-side-encryption-aws-kms-key-id": KMS_KEY_ARN,
    "x-amz-server-side-encryption-bucket-key-enabled": "True"
    # "acl": "public-read"
    }


conditions=[
    {
        "x-amz-storage-class": "STANDARD_IA"
    },
    ["starts-with", "$Content-Type", "plain"],
    {
        "tagging": "<Tagging><TagSet><Tag><Key>type</Key><Value>test</Value></Tag></TagSet></Tagging>"
    },
    {
        "Cache-Control": "max-age=86400"
    },
    {
        "success_action_status": "200"
    },
    {
        "x-amz-server-side-encryption": "aws:kms"
    },
    {
        "x-amz-server-side-encryption-aws-kms-key-id": KMS_KEY_ARN
    },
    # {
    #     "acl": "public-read"
    # },
    {
        "x-amz-server-side-encryption-bucket-key-enabled": "True"
    }
]

# generate S3 Presigned URL for HTTP POST Request
response_presigned_url_post=s3.generate_presigned_post(
    BUCKET_NAME,
    OBJECT_NAME,
    Fields=fields,
    Conditions=conditions,
    ExpiresIn=EXPIRATION_TIME
)
print(response_presigned_url_post)

# User requests.post to test the URL
post_fields=response_presigned_url_post['fields']

# files={'file': open(TEST_FILE_NAME, 'rb')}
# you will see 403 error
#comment the following line and uncomment the second following line, you will see 200 successful

post_fields["Content-Type"]="application/octet-stream"
#post_fields["Content-Type"]="plain/text"

# file key must be the last key in the "files" parameter(form)
# it is defined at https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html#RESTObjectPOST-requests-form-fields

post_fields["file"]=open(TEST_FILE_NAME, 'rb')
print(post_fields)

# making POST Request
response_post_request=requests.post(response_presigned_url_post['url'], files=post_fields)

# print response, by default status code is 204,
# "success_action_status": "200" change it to 200

print(f'Response Status of POST request with S3 Presigned URL: {response_post_request.status_code}')
print(f'Response Headers of POST request with S3 Presigned URL: {response_post_request.headers}')
print(f'Response Body of POST request with S3 Presigned URL: {response_post_request.text}')

CloudFront Edge Computing - Dynamic At Edge with CloudFront Functions

Jason Shen — Mon, 14 Aug 2023 06:53:53 +0000

Edge Computing feature is definitely one of the things that I like the most with CloudFront service. When CloudFront Functions is launched, it makes the feature even better.

Recently I saw someone asked a question about the following scenario in re:Post. I think it is a perfect scenario for CloudFront Functions.

Files in the S3 origin:

JSON files are under content folder(prefix), e.g. s3://example_bucket/content/a.json
Image files are under a sub-folder of content folder (prefix) e.g. s3://example_bucket/content/image/a.png

The files are requested by viewer request like:

JSON file: https://text.example.com/abc/a.json
PNG file: https://image.example.com/a.png

What is going to happen when there was no edge computing in CloudFront service?

For the JSON file, there might be no other options but move the objects under 'content/abc/' folder. Then origin path will be 'content/' in origin setting of the distribution.

For the PNG file, it will require a separate origin setting in the distribution configuration other than the JSON file because they are not going to share the same origin path. The PNG file will need origin path as 'content/image'.

The JSON file and PNG files will also need different behaviors because they need to call different origin settings.

It looks like lots of work.

With CloudFront Functions, a few lines of codes will resolve the problem.

function handler(event) {
    var request = event.request;
    var uri = request.uri;
    var file_name = uri.split('/').pop();

    // Check whether the URI is missing a file name.
    if ( file_name.endsWith('.json')) {
        request.uri = "/content/" + file_name;
        return request;
    } else if (file_name.endsWith('.png')) {
        request.uri = '/content/image/' + file_name;
        return request;
    }

    return request;
}

In the behavior, the function will be added as Viewer Request trigger in Default behavior with an S3 origin pointing to "s3://example_bucket".

That is it!

And if any further requirement is required, e.g. changing path based on the requesting domain name, only a few modifications in codes will meet the requirement.

function handler(event) {
    var request = event.request;
    var uri = request.uri;
    var file_name = uri.split('/').pop();
    var host = request.headers.host.value;

    // Check whether the URI is missing a file name.
    if ( host === "text.example.com" ) {
        request.uri = "/content/" + file_name;
        return request;
    } else if ( host === "image.example.com" )) {
        request.uri = '/content/image/' + file_name;
        return request;
    }

    return request;
}

CloudFront Functions are perfect to process those 'small' modifications in viewer requests and responses.

The code I am using is modified from this code example in CloudFront public document.

I am seeing a trend that generative AI tools like ChatGPT or Amazon CodeWhisperer will help to writ those scripts much easier than ever!

Like Lambda@Edge, there are some restrictions on runtime of CloudFront Functions that you should have a read before wring your codes.

The Differences In Sending Email Actions Between SES Version 1 and Version 2 APIs

Jason Shen — Mon, 07 Aug 2023 03:05:28 +0000

If you have been using Amazon SES service for a while, it might not be new to you that Amazon SES is having two versions of API actions available at the moment, API Reference (version 1) and API v2 Reference. I tried to find the announcement of the API v2 in What's New with AWS? page but I did not have the luck.

Amazon SES service API calls can be separated into two types:

Management API actions, which can be recorded in CloudTrail Event History
Sending API actions, which are not recorded in CloudTrail Event History

In this article, you will read about how I took a peek at the sending API actions in Amazon SES API v1 and v2.

What are these API actions?

I can find the API action names in SES API documents. But I find that I can also get the list of API calls by creating an example IAM policy from IAM console.

Here is the list of actions I got when I was creating an IAM policy with service name 'SES' and 'SES v2' from IAM console.

v1 IAM actions

"Action": [
"ses:SendBounce",
"ses:SendBulkTemplatedEmail",
"ses:SendCustomVerificationEmail",
"ses:SendEmail",
"ses:SendRawEmail",
"ses:SendTemplatedEmail"
]
v2 IAM actions

"Action": [
"ses:SendBulkEmail",
"ses:SendCustomVerificationEmail",
"ses:SendEmail"
]

What are the differences?

I used SES SDK Boto3 Document in the following comparison.

To use Amazon SES v2 APIs, the service client must be boto3.client('sesv2')

SES v2 API "SendEmail" provides the same functions in v1 APIs "SendEmail", "SendRawEmail" and "SendTemplatedEmail". But only the API "SendEmail" in SES v2 supports the feature "list management". List Management can help to reduce the chance of get complaints or even hard bounces by sending emails to recipients who have opted out of their previous subscriptions to some email services, e.g. newsletter email.
SES v2 API "SendBulkEmail" has different names of parameters in sending requests with SES v1 "SendBulkTemplatedEmail". But values of those parameters should be no difference.
SES v2 API action "SendCustomVerificationEmail" has no difference with the action in SES v1 API.
SES v2 API actions do not have a definition of action "SendBounce". The action "SendBounce" is used to generate and send a bounce message to the sender of an email which I received through Amazon SES. There is a restriction that we can only use this API action on an email up to 24 hours after receiving it.

How do I restrict usage in SES API v1 and v2?

SES public document has given an example about "Allowing Access to only SES API version 2". Relatively, I could modify the condition forcing to use SES API v1.

Why would I want to do that? One thing in the same document could provide an answer:

The SES SMTP interface uses SES API version 2 of ses:SendRawEmail.

As an IAM user can convert AWS credentials into SMTP username and password, I can use the trick to restrict the IAM user to use or not use SMTP client to send emails through Amazon SES.

One more difference is the message size between using SES API v1 and v2. While size of message (after base64 encoding and including attachments) is 10 MB using SES API v1, the size is 40MB in SES API v2.

SES BYODKIM - Streamline Private and Public Key by Python Script

Jason Shen — Tue, 01 Aug 2023 05:26:28 +0000

In Amazon SES document, it states the following:

You have to delete the first and last lines (-----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY-----, respectively) of the generated private(public) key. Additionally, you have to remove the line breaks in the generated private key. The resulting value is a string of characters with no spaces or line breaks.

source

If the private or public is not streamlined, you won't be able to use them with SES BYODKIM.

At the meantime, you also need to generate a random string as selector in TXT DNS record publishing the public key

Name	Type	Value
selector._domainkey.example.com	TXT	p=yourPublicKey

Replace selector with a unique name that identifies the key.
Here is the Python Script to complete

source

Here is python script can do the both.

import sys, string, random

def streamline_key(keyLocation):
   keyLines=open(keyLocation).readlines()
   keyStream=[]
   for line in keyLines[1:-1]:
      keyStream.append(line.replace('\n', ''))
   key=''.join(keyStream)
   return key

print(sys.argv)

key1Location=sys.argv[-3]
print(key1Location)

key2Location=sys.argv[-2]
print(key2Location)

domain=sys.argv[-1]
print(domain)

key1Streamline=streamline_key(key1Location)
print(key1Location+" streamline:\n" + key1Streamline )
print("\n")
key2Streamline=streamline_key(key2Location)
print(key2Location+" streamline:\n" + key2Streamline)
print("\n")
selector = ''.join(random.choice(string.ascii_lowercase + string.digits) for i in range(32))
print("public key TXT record name:\n" + selector+'._domainkey.'+domain)

Save the code with name Run the code 'remove-newline-in-key.py' in the same folder storing private key and public key. Then run the script as following format in command line. My public key name is public.key.

% python remove-newline-in-key.py private.key public.key example.com

You will get the following result

['remove-newline-in-key.py', 'private.key', 'public.key', 'example.com']
private.key
public.key
example.com
private.key streamline:
MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAMdPWfzVMYahtkvTvBfXsVr52O2CHUok8JzeoCt1Ou3t1SmDmV3755+ztxGj7nFwUCVFmrT5ZmaaDZ5u7Jd856KmejtlIeuPHBt9wuoaiwI1IohXWZAMGLi+qo+FX1kHk+nKj5nMLNq9dSOE8xXXfmtPcz+B4LACpuQRXNGhqCLlAgMBAAECgYEAkXTq8qdQtrXMSfij3C6xI/kVhPihkZv18jZTZIPw1vXszJhbVIjkWNwarggam7Vg+GKc7pjZT+X8LHU9u60Pio22vi6ZNBQwqe0DlpMx1MtJIht4EwH63CZDSU6jijZUjvdTyKqtoqMHiqUaLz2Iom8LYikmrKImMr6S9PqgBsECQQDsJ+8N4asakc0uUKZkxgQNpoM7fykuFmF9TJcq3K9JHfx8HpvMN9UWNyGDfQqIo/4oFD3LxeheeyltETCNqE91AkEA2A7OE2r+D9uMpAnNyt3SmIRQZzVn+ZHB+0fICFB8L17rt6TnuH2AU6ceuoVzr8vtSWGZ+/sotUGvaIbZvwkHsQJAfoTafv5i8+YfHewZaS3pKAMIlcyHnGhjLITnDBCVXD/TcA/Z+iwDXlaE/vPzu8bYOFK31L8fwdaMGCG4eHwurQJAO2CeO/Hsjrkcxrw3BWi/BtFeM28W+xyWvhM1IyvTZUVl7JtyX16GVPcZ19LzPz4BIWikY/7baiz6IvTkhL7bkQJAZhwo33EVKRcDoavSOWshcWEsp6SNychsdT9R17uEzsZq1RgrB9XNqskTveJvzLeT/aRtuoqB+mpJ1R3ux3VLYw==

public.key streamline:
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHT1n81TGGobZL07wX17Fa+djtgh1KJPCc3qArdTrt7dUpg5ld++efs7cRo+5xcFAlRZq0+WZmmg2ebuyXfOeipno7ZSHrjxwbfcLqGosCNSKIV1mQDBi4vqqPhV9ZB5Ppyo+ZzCzavXUjhPMV135rT3M/geCwAqbkEVzRoagi5QIDAQAB

public key TXT record name:
w2hajm6q1zoe0gw1q993shhmqopj5auy._domainkey.example.com