How to Set Up Cost-Effective Email Solutions with AWS SES and Terraform

🐙 GitHub

Setting Up Cost-Effective Email Solutions on AWS

In this article, I'll show you how to set up email addresses for your AWS domain at almost no cost. I've successfully implemented this technique for my domain, radzion.com, as well as for two of my projects: increaser.org and georgiancitizen.com. This method is straightforward, requiring only a single AWS Lambda function and no additional costs as you add more domains. You can find everything needed to deploy this solution, including both the Terraform infrastructure and the Lambda function source code, in the RadzionKit repository—a comprehensive toolkit designed to jumpstart full-stack projects within a monorepo.

Automating Email Infrastructure with Terraform

To minimize manual operations on the AWS console, we utilize Terraform to automate the infrastructure setup. This approach simplifies adding email solutions for new addresses without requiring much additional effort. Our infrastructure hinges on just four variables:

• name: Use this to prefix your resources. Choose something unique and descriptive.
• forward_to: This should be your personal email address, such as your Gmail, where you wish to receive emails.
• domain: A JSON list of objects, each containing a domain name and AWS zone ID.
• sentry_key: Optional. Use this to send error reports to Sentry.

variable "name" {}

variable "forward_to" {}

variable "domains" {
  type = list(object({
    domain_name : string
    zone_id     : string
  }))
}

variable "sentry_key" {}

To manage the Terraform state effectively, I store it in an S3 bucket. Before executing any Terraform commands, it's essential to set up environment variables. These include AWS credentials, a description of the S3 bucket, and the necessary Terraform variables. All Terraform variables should be prefixed with TF_VAR_.

export AWS_SECRET_ACCESS_KEY=
export AWS_ACCESS_KEY_ID=
export AWS_REGION=

# optional, only if you want to store terraform state in S3
export TF_VAR_remote_state_bucket=
export TF_VAR_remote_state_key=
export TF_VAR_remote_state_region=

export TF_VAR_name=
# e.g. [e-mail]
export TF_VAR_forward_to=
# a JSON string, e.g.
# '[{"domain_name":"radzion.com","zone_id":"A1026834LOTQUY1CVV2S"},{"domain_name":"increaser.org","zone_id":"Z1QT1BOR8JUIVM"}]'
export TF_VAR_domains=
export TF_VAR_sentry_key=

With the environment variables set, we can proceed to initialize the Terraform project and apply the infrastructure changes. Since I use an S3 backend for storing Terraform state, it is necessary to specify the bucket name, key, and region in the terraform init command.

terraform init \
  -backend-config="bucket=${TF_VAR_remote_state_bucket}" \
  -backend-config="key=${TF_VAR_remote_state_key}" \
  -backend-config="region=${TF_VAR_remote_state_region}"

terraform apply

Establishing SES Domain Identity and Email Reception

First, we need to set up the SES domain identity, which is essential for proving ownership of your domain to AWS. This step is crucial as it allows you to manage email sending and receiving capabilities securely under your domain's name. This verification is accomplished by iterating over each domain in the domains variable, creating an aws_ses_domain_identity resource for each. Subsequently, a aws_route53_record resource is configured with a verification token from domain_identity to verify the domain.

provider "aws" {
}

terraform {
  backend "s3" {
  }
}

resource "aws_ses_domain_identity" "domain_identity" {
  for_each = { for idx, domain in var.domains : idx => domain }

  domain = each.value.domain_name
}

resource "aws_route53_record" "amazonses_verification_record" {
  for_each = { for idx, domain in var.domains : idx => domain }

  zone_id = each.value.zone_id
  name    = "_amazonses.${each.value.domain_name}"
  type    = "TXT"
  ttl     = 600
  records = [aws_ses_domain_identity.domain_identity[each.key].verification_token]
}

resource "aws_route53_record" "amazonses_receiving_record" {
  for_each = { for idx, domain in var.domains : idx => domain }

  zone_id = each.value.zone_id
  name    = each.value.domain_name
  type    = "MX"
  ttl     = 600
  records = ["10 inbound-smtp.${data.aws_region.current.name}.amazonaws.com"]
}

resource "aws_s3_bucket" "emails_storage" {
  bucket = "tf-${var.name}-emails-storage"
}

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

data "archive_file" "local_zipped_lambda" {
  type        = "zip"
  source_dir  = "${path.module}/lambda"
  output_path = "${path.module}/lambda.zip"
}

resource "aws_s3_object" "zipped_lambda" {
  bucket = aws_s3_bucket.lambda_storage.id
  key    = "lambda.zip"
  source = data.archive_file.local_zipped_lambda.output_path
}

resource "aws_s3_bucket" "lambda_storage" {
  bucket = "tf-${var.name}-storage"
}

resource "aws_lambda_function" "ses_forwarder" {
  function_name = "tf-${var.name}"

  s3_bucket = aws_s3_bucket.lambda_storage.bucket
  s3_key    = "lambda.zip"

  handler     = "src/index.handler"
  runtime     = "nodejs20.x"
  timeout     = 50
  memory_size = 1600
  role        = aws_iam_role.ses_forwarder.arn

  environment {
    variables = {
      SENTRY_KEY     = var.sentry_key
      FORWARD_TO     = var.forward_to
      EMAILS_BUCKET  = aws_s3_bucket.emails_storage.bucket
    }
  }
}

resource "aws_iam_role" "ses_forwarder" {
  name = "tf-${var.name}"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action    = "sts:AssumeRole"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Effect = "Allow"
        Sid    = ""
      }
    ]
  })
}

resource "aws_cloudwatch_log_group" "ses_forwarder" {
  name = "tf-${var.name}"
}

resource "aws_iam_policy" "ses_forwarder" {
  name = "tf-${var.name}"
  path = "/"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      },
      {
        Effect   = "Allow"
        Action   = "ses:SendRawEmail"
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = [
          "s3:GetObject",
          "s3:PutObject"
        ]
        Resource = "arn:aws:s3:::${aws_s3_bucket.emails_storage.bucket}/*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ses_forwarder" {
  role       = aws_iam_role.ses_forwarder.name
  policy_arn = aws_iam_policy.ses_forwarder.arn
}

data "aws_iam_policy_document" "emails_storage" {
  statement {
    sid = "GiveSESPermissionToWriteEmail"

    effect = "Allow"

    principals {
      identifiers = ["ses.amazonaws.com"]
      type        = "Service"
    }

    actions = ["s3:PutObject"]

    resources = ["${aws_s3_bucket.emails_storage.arn}/*"]
  }
}

resource "aws_s3_bucket_policy" "emails_storage" {
  bucket = aws_s3_bucket.emails_storage.id
  policy = data.aws_iam_policy_document.emails_storage.json
}

resource "aws_lambda_permission" "allow_ses" {
  statement_id  = "AllowExecutionFromSES"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.ses_forwarder.function_name
  source_account = data.aws_caller_identity.current.account_id
  principal     = "ses.amazonaws.com"
}

resource "aws_ses_receipt_rule_set" "rule_set" {
  rule_set_name = "tf-${var.name}-rule-set"
}

resource "aws_ses_active_receipt_rule_set" "rule_set" {
  rule_set_name = aws_ses_receipt_rule_set.rule_set.rule_set_name
}

locals {
  domain_names = [for domain in var.domains : domain.domain_name]
}

resource "aws_ses_receipt_rule" "receipt_rule" {
  name          = "tf-${var.name}-rule"
  rule_set_name = aws_ses_receipt_rule_set.rule_set.rule_set_name
  recipients    = local.domain_names
  enabled       = true
  scan_enabled  = true

  s3_action {
    bucket_name = aws_s3_bucket.emails_storage.bucket
    position    = 1
  }

  lambda_action {
    function_arn    = aws_lambda_function.ses_forwarder.arn
    invocation_type = "Event"
    position        = 2
  }
}

This Terraform resource, aws_route53_record named "amazonses_receiving_record", configures MX records for each domain specified in the domains variable to enable email receiving. It sets the mail exchange server to AWS's SMTP interface, specifying the AWS region dynamically, ensuring that emails directed to your domain are correctly routed to AWS for processing.

The Terraform resource aws_s3_bucket named "emails_storage" creates an S3 bucket to store all incoming emails, ensuring they are securely archived in AWS.

Configuring AWS Lambda for Email Forwarding

Once an email is received and stored in the S3 bucket, an AWS Lambda function, ses_forwarder, notifies and forwards the email to a personal address specified in the forward_to variable. Initially, a placeholder ("dummy") Lambda function code is uploaded using the aws_s3_object resource to store it in the "lambda_storage" bucket. This setup allows us to establish the infrastructure without the final Lambda code. Later, we replace the dummy code with the actual functionality. The dummy code, located in lambda/index.js within our Terraform module, is prepared by archiving it locally with the archive_file before uploading to S3.

Setting Permissions and Receipt Rules for Email Handling

In the Lambda policy we grant it access for making logs, sending an email and reading/writing to the S3 bucket. We also allow SES to write emails to the S3 bucket by creating an IAM The Lambda function is granted permissions to log actions, send emails, and read/write to the S3 bucket via a Lambda policy. Additionally, an IAM policy document allows SES to store incoming emails directly in the S3 bucket. To ensure the Lambda function triggers upon receiving an email, the aws_lambda_permission resource allows SES to invoke the function. Finally, the aws_ses_receipt_rule resource establishes a receipt rule for each domain listed in the domains variable, directing the handling of incoming emails to the specified S3 bucket and Lambda function.

Final Steps and Troubleshooting the Email Setup

We've covered the setup of the infrastructure needed for handling emails via AWS. It's common to encounter errors when applying Terraform changes, particularly with Lambda deployments. If an error occurs, deploying the Lambda function manually and re-running terraform apply should resolve the issue. Additionally, you must manually add your forward_to email address to the SES identities. This step isn't automated through Terraform because it requires verification by clicking a link in an email sent to that address, which ensures the security and ownership of the email account.

Understanding the Lambda Function Code and Environment Management

Now, let's examine the Lambda function code in the lambda.ts file. We begin by setting up Sentry for error tracking to ensure any issues are promptly reported and handled. The function processes each record in the SES event by iterating over them and forwarding the emails to the specified address using the processSesEventRecord function.

import { AWSLambda } from "@sentry/serverless"
import { SESEvent } from "aws-lambda"
import { processSesEventRecord } from "./processSesEventRecord"
import { getEnvVar } from "./getEnvVar"

AWSLambda.init({
  dsn: getEnvVar("SENTRY_KEY"),
})

export const handler = AWSLambda.wrapHandler(async (event: SESEvent) => {
  await Promise.all(event.Records.map(processSesEventRecord))
})

In my projects, I incorporate the getEnvVar function to manage environment variables effectively. This function checks for the presence of required environment variables and throws an error if any are missing, preventing runtime issues due to misconfiguration.

type VariableName = "SENTRY_KEY" | "FORWARD_TO" | "EMAILS_BUCKET"

export const getEnvVar = (name: VariableName): string => {
  const value = process.env[name]
  if (!value) {
    throw new Error(`Missing ${name} environment variable`)
  }

  return value
}

Detailed Workflow of the Lambda Email Processing Function

The processSesEventRecord function executes a series of steps to handle each SES event record. If an error occurs during this process, it calls the reportError function, passing the problematic record and a detailed explanation to Sentry for more effective troubleshooting and context.

import { SESEventRecord } from "aws-lambda"
import { reportError } from "@lib/lambda/reportError"
import { getEmailFromStorage } from "./getEmailFromStorage"
import { formatEmail } from "./formatEmail"
import { getEnvVar } from "./getEnvVar"
import { forwardEmail } from "./forwardEmail"

export const processSesEventRecord = async (record: SESEventRecord) => {
  console.log("Processing SES event record", { record })
  try {
    const {
      mail,
      receipt: { recipients },
    } = record.ses
    const message = await getEmailFromStorage(mail.messageId)
    if (recipients.length > 1) {
      throw new Error("Multiple recipients are not supported")
    }

    const [recipient] = recipients

    const formattedEmail = formatEmail({
      message,
      recipient,
      forwardTo: getEnvVar("FORWARD_TO"),
    })

    return forwardEmail({
      forwardTo: getEnvVar("FORWARD_TO"),
      message: formattedEmail,
      sendFrom: recipient,
    })
  } catch (err) {
    reportError(err, { record, msg: "Error processing SES event record" })
  }
}

The first step in processSesEventRecord involves retrieving the email from the S3 bucket using the getEmailFromStorage function. This function initializes an S3 client and executes a GetObjectCommand with the provided messageId to fetch the email. If the email cannot be found, the function throws an error. Additionally, since the returned Body from S3 is a blob, it is converted into a string for further processing.

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3"
import { getEnvVar } from "./getEnvVar"

export const getEmailFromStorage = async (messageId: string) => {
  const s3 = new S3Client({})

  const command = new GetObjectCommand({
    Bucket: getEnvVar("EMAILS_BUCKET"),
    Key: messageId,
  })

  const { Body } = await s3.send(command)

  if (!Body) {
    throw new Error(`Email ${messageId} not found`)
  }

  return Body.transformToString()
}

Next, the function checks the recipients of the email. I operate under the assumption that emails with multiple recipients are likely to be spam, and therefore, I throw an error in such cases. If you prefer to handle emails with multiple recipients, you can set an environment variable containing a list of your domains. The function can then check if any recipient address matches one of your domains and forward the email accordingly.

Once the recipient is determined, the formatEmail function processes the email to prepare it for forwarding. This function ensures SES compliance by reformatting headers: it updates the "From" header to a verified domain, sets the "To" header to the intended recipient, and manages the "Reply-To" setting. Additionally, it removes headers like "Return-Path" and "DKIM-Signature" that could cause delivery issues.

import { ErrorWithContext } from "@lib/utils/errors/ErrorWithContext"

type FormatEmailInput = {
  message: string
  recipient: string
  forwardTo: string
}

export const formatEmail = ({
  message,
  recipient,
  forwardTo,
}: FormatEmailInput): string => {
  const parsedMessage = message.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m)
  if (!parsedMessage) {
    throw new ErrorWithContext("Failed to parse email", {
      message,
    })
  }
  let header = parsedMessage[1]
  const body = parsedMessage[2]

  // Add "Reply-To:" with the "From" address if it doesn't already exists
  if (!/^reply-to:[\t ]?/im.test(header)) {
    const [, from] =
      header.match(/^from:[\t ]?(.*(?:\r?\n\s+.*)*\r?\n)/im) || []
    if (from) {
      header = `${header}Reply-To: ${from}`
    }
  }

  // SES does not allow sending messages from an unverified address,
  // so replace the message's "From:" header with the original
  // recipient (which is a verified domain)
  header = header.replace(
    /^from:[\t ]?(.*(?:\r?\n\s+.*)*)/gim,
    (_, from) =>
      `From: ${from.replace("<", "at ").replace(">", "")} <${recipient}>`
  )

  // Replace original 'To' header with a manually defined one
  header = header.replace(/^to:[\t ]?(.*)/gim, () => `To: ${forwardTo}`)

  // Remove the Return-Path header.
  header = header.replace(/^return-path:[\t ]?(.*)\r?\n/gim, "")

  // Remove Sender header.
  header = header.replace(/^sender:[\t ]?(.*)\r?\n/gim, "")

  // Remove Message-ID header.
  header = header.replace(/^message-id:[\t ]?(.*)\r?\n/gim, "")

  // Remove all DKIM-Signature headers to prevent triggering an
  // "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
  // These signatures will likely be invalid anyways, since the From
  // header was modified.
  header = header.replace(/^dkim-signature:[\t ]?.*\r?\n(\s+.*\r?\n)*/gim, "")

  return `${header}${body}`
}

Finally, the forwardEmail function sends the processed email to our personal address using the recipient's address. This function creates an SES v2 client and executes the SendEmailCommand, which requires the destination address (forwardTo), the email content, and the FromEmailAddress (the verified sender address).

import { SESv2Client, SendEmailCommand } from "@aws-sdk/client-sesv2"

type ForwardEmailInput = {
  forwardTo: string
  message: string
  sendFrom: string
}

export const forwardEmail = ({
  forwardTo,
  message,
  sendFrom,
}: ForwardEmailInput) => {
  const client = new SESv2Client({})

  const command = new SendEmailCommand({
    Destination: {
      ToAddresses: [forwardTo],
    },
    Content: {
      Raw: {
        Data: new TextEncoder().encode(message),
      },
    },
    FromEmailAddress: sendFrom,
  })

  return client.send(command)
}

Deploying and Building the Lambda Function

Now we can deploy our lambda by building the project, packaging it into a zip file, and uploading it to S3. After uploading, we update the Lambda function's code using the AWS CLI to point to the new package in the S3 bucket.

yarn build
cd dist

BUCKET=tf-radzion-email-storage
BUCKET_KEY=lambda.zip
FUNCTION_NAME=tf-radzion-email

zip -r ./$BUCKET_KEY *

aws s3 cp $BUCKET_KEY s3://$BUCKET/$BUCKET_KEY
aws lambda update-function-code --function-name $FUNCTION_NAME --s3-bucket $BUCKET --s3-key $BUCKET_KEY

cd ..

The project's build process starts with the clean script that removes any previous build artifacts from the dist directory and deletes the lambda.zip file. Then, the transpile script uses esbuild to bundle, minify, and transpile the lambda.ts file into index.js, targeting Node.js environments and storing the output in the dist directory.

{
  "scripts": {
    "transpile": "esbuild lambda.ts --bundle --minify --sourcemap --platform=node --target=es2020 --outfile=dist/index.js",
    "build": "yarn clean & yarn transpile",
    "clean": "rm -rf ./dist lambda.zip"
  }
}

Testing Email Reception and Configuring Outbound Settings

With the infrastructure and Lambda function configured, you're now ready to test sending emails to your domain. For example, with the radzion.com domain, you could send emails to addresses like [e-mail] or [e-mail]. Our setup is designed to handle all variations, ensuring that each email is correctly processed and forwarded.

Now that you've configured receiving emails, the next step is to set up the ability to send emails from your domain using your email provider. For Gmail users, navigate to "Settings," then "See all settings," and click on the "Accounts and Import" tab. Here, select "Add another email address." This process requires SMTP server details, a username, and a password, which you will retrieve from AWS SES.

In the AWS SES console, under SMTP settings, create SMTP credentials. This will provide you with the necessary server name, port, and SMTP credentials to enter in Gmail's setup, allowing you to send emails as if from your domain.