Monitoring SES Email Events though Configuration Set with SNS, SQS, Lambda and CloudWatch Log Groups

When you are sending emails through Amazon SES service, you need to track sending events like Bounce and Complaint. You can use those events to adjust your mail list and eventually maintain reputation of your SES account, so you don't have to face unwanted status of your SES account like sending pause and review situation.

Amazon SES provides sending events through email feedback and event notification. When Configuration Set feature is available, now you can publish more events to multiple destinations. For example, you can publish Subscription event to CloudWatch metric.

In this article, I will show you how to set up a serverless workflow to monitor sending events of your SES account by using the following AWS services.

• Amazon SES
• Amazon Simple Notification Service (SNS)
• Amazon Simple Queue Service (SQS)
• AWS Lambda
• Amazon CloudWatch log groups

1#. Follow SES document "Set up email sending with Amazon SES " and setup your SES account. If your SES account is in sandbox, you will need to verify both sender and recipient email addresses.

2#. Follow SES document "Creating configuration sets in SES" and create a Configuration Set.

3#. Follow SQS document "Create a queue (console)" and create SQS queue with standard queue type.

4#. Follow SNS document "Getting started with Amazon SNS" and create SNS topic.

5#. Create a SNS subscription with Protocol type "SQS", put SQS ARN of SQS queue created in step 3.

6#. Update Access Policy of SQS queue to allow SNS topic to publish events.

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AWS_Account_ID>:root"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-west-2:<AWS_Account_ID>:<SQS_Queue_Name>"
    },
    {
      "Sid": "__sender_statement",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-west-2:<AWS_Account_ID>:<SQS_Queue_Name>"
    }
  ]
}

7#. Create a Lambda function using Python version > 3.9

The function will be triggered by SQS queue and store events into CloudWatch log groups based on SES message ID. The function will create CloudWatch log group if not existing.

import json
import boto3
from datetime import datetime
from time import time
import random

def lambda_handler(event, context):
    #print(event)
    for record in event['Records']:
        print("record")
        payload = record["body"]
        print(payload)
        # get lambda running region
        region = context.invoked_function_arn.split(":")[3]
        # get lambda running version
        version = context.function_version
        # get lambda running name
        name = "ses-notification-" + region
        mail = json.loads(payload)['mail']
        #print("mail")
        #print(mail)
        messageID = mail['messageId']
        print(messageID)
        logGroupNamePrefix="/aws/lambda/" + name
        # define CloudWatch log group client in the region
        client = boto3.client('logs', region_name=region)
        # check cloudwatch log group with name "ses-notification-cw-logs" existence
        response = client.describe_log_groups(logGroupNamePrefix=logGroupNamePrefix)
        # if log group not found, create it
        if len(response["logGroups"]) == 0:
            client.create_log_group(logGroupName=logGroupNamePrefix)
            print("Log group created")
        # get current date in form as year/month/day
        currentDay = str(datetime.now().day)
        currentMonth = str(datetime.now().month)
        currentYear = str(datetime.now().year)
        date = currentYear + "/" + currentMonth + "/" + currentDay
        logStreamsName = date + "/[" + version + "]" + messageID
        # create CloudWatch log stream
        responseLogStream=client.describe_log_streams(logGroupName=logGroupNamePrefix,logStreamNamePrefix=logStreamsName)
        if len(responseLogStream["logStreams"]) == 0:
            client.create_log_stream(logGroupName=logGroupNamePrefix, logStreamName=logStreamsName)
            print("Log stream created")
        # put payload into CloudWatch log stream with current timestamp
        #payload = "this is test payload"
        client.put_log_events(logGroupName=logGroupNamePrefix, logStreamName=logStreamsName, logEvents=[{"timestamp": int(time() * 1000), "message": payload}])
        print("Payload put into log stream")

8#. The IAM role of the Lambda function must have the following permission in IAM managed policy.

• CloudWatchLogsFullAccess
• AWSLambdaSQSQueueExecutionRole

9#. In the SQS queue, you need to follow "Configuring a queue to trigger an AWS Lambda function (console)" and add the Lambda function as trigger.

10#. Create Event Destination in Configuration Set created in step 2 by following "Set up an Amazon SNS event destination for event publishing.

11#. Now you can send an email by using SMTP, AWS CLI or API through your SES account. Remember that you need to specify the Configuration Set in your sending action.