Do you really need MailChimp? (transactional emails with AWS SES, Terraform and Lambda)

What are transactional emails?

Many applications need to send emails. Usually these emails are triggered in code in response to a given event, such as a product purchase. For this reason they are often called Transactional Emails. But an app or backend service can’t just send an email, it needs to have a SMTP server setup. Setting up email servers can be tricky, so a number of services exist for sending emails with API requests. The main players are: MailChimp (Mandrill), Sendgrid, MailGun, and SendinBlue.

Recently however more developers have been using Amazon’s “simple email service” or SES. It’s not as famous as the others but is developer focused and fits nicely into any existing AWS infrastructure. Plus, a lot of the time SES is easier and cheaper to use. Here is a simple guide for setting up an SES domain, sending emails, and receiving emails. Later, we’ll go over testing your email actions with real email addresses.

If you'd like to use a powerful alternative built on SES try MailSlurp.

Building the infrastructure in AWS

Registering a domain

First off, you need a domain to send and receive email from. If you already have one that’s great, if not go ahead and get one. You also need an AWS account.

Adding Route53 domain records

Next we need to add MX records for this domain to Route53. I’m going to use Terraform to set up my AWS infrastructure because it is a lot more maintainable in the long run.

First let’s create some variables in Terraform:

variable "zone_id" {
  default = "your-route-53-domain-zone-id"
}

variable "domain" {
  default = "your-domain-here"
}

Now let’s create an SES domain identity and the Route53 records associated with it. This will let us send emails from the domain.

# ses domain
resource "aws_ses_domain_identity" "ms" {
  domain = "${var.domain}"
}

resource "aws_route53_record" "ms-domain-identity-records" {
  zone_id = "${var.zone_id}"
  name = "_amazonses.mailslurp.com"
  type = "TXT"
  ttl = "600"

  records = [
    "${aws_ses_domain_identity.ms.verification_token}",
  ]
}

# ses dkim
resource "aws_ses_domain_dkim" "ms" {
  domain = "${aws_ses_domain_identity.ms.domain}"
}

resource "aws_route53_record" "ms-dkim-records" {
  count = 3
  zone_id = "${var.zone_id}"
  name = "${element(aws_ses_domain_dkim.ms.dkim_tokens, count.index)}._domainkey.mailslurp.com"
  type = "CNAME"
  ttl = "600"

  records = [
    "${element(aws_ses_domain_dkim.ms.dkim_tokens, count.index)}.dkim.amazonses.com",
  ]
}

# ses mail to records
resource "aws_route53_record" "ms-mx-records" {
  zone_id = "${var.zone_id}"
  name = "${var.domain}"
  type = "MX"
  ttl = "600"

  records = [
    "10 inbound-smtp.us-west-2.amazonses.com",
    "10 inbound-smtp.us-west-2.amazonaws.com",
  ]
}

resource "aws_route53_record" "ms-spf-records" {
  zone_id = "${var.zone_id}"
  name = "${var.domain}"
  type = "TXT"
  ttl = "600"

  records = [
    "v=spf1 include:amazonses.com -all",
  ]
}

Setting up SES receipt rules

Great, now we can send and receive from the domain we registered. But we still need to tell SES to use this domain with a set of email addresses. These configurations are called receipt rule sets. Let’s create a rule that catches all inbound emails to any address in our domain and saves them to S3 and notifies an SNS topic.

# ses rule set
resource "aws_ses_receipt_rule_set" "ms" {
  rule_set_name = "ms_receive_all"
}

resource "aws_ses_active_receipt_rule_set" "ms" {
  rule_set_name = "${aws_ses_receipt_rule_set.ms.rule_set_name}"

  depends_on = [
    "aws_ses_receipt_rule.ms",
  ]
}

# lambda catch all
resource "aws_ses_receipt_rule" "ms" {
  name = "ms"
  rule_set_name = "${aws_ses_receipt_rule_set.ms.rule_set_name}"

  recipients = [
    "${var.domain}",
  ]

  enabled = true
  scan_enabled = true

  s3_action {
    bucket_name = "${aws_s3_bucket.ms.bucket}"
    topic_arn = "${aws_sns_topic.ms2.arn}"
    position = 1
  }

  stop_action {
    scope = "RuleSet"
    position = 2
  }

  depends_on = ["aws_s3_bucket.ms", "aws_s3_bucket_policy.ms_ses", "aws_lambda_permission.with_ses"]
}

We might also want to catch email errors. Let’s add a rule for that and send them to a cloudwatch log.

resource "aws_ses_configuration_set" "ms" {
  name = "ms-ses-configuration-set"
}

resource "aws_ses_event_destination" "ses_errors" {
  name = "ses-error-sns-destination"
  configuration_set_name = "${aws_ses_configuration_set.ms.name}"
  enabled = true

  matching_types = [
    "reject",
    "reject",
    "send",
  ]

  sns_destination {
    topic_arn = "${aws_sns_topic.ms2_ses_error.arn}"
  }
}

resource "aws_ses_event_destination" "ses_cloudwatch" {
  name = "event-destination-cloudwatch"
  configuration_set_name = "${aws_ses_configuration_set.ms.name}"
  enabled = true

  matching_types = [
    "reject",
    "reject",
    "send",
  ]

  cloudwatch_destination = {
    default_value = "default"
    dimension_name = "dimension"
    value_source = "emailHeader"
  }
}

How SNS fits in

So in the above rulesets we defined a catch all for all inbound email. It’s actions were to save the email to S3 and then notify an SNS topic. SNS is a simple notification service that we can subscribe to within our application or with a lambda to handle inbound emails. The event we receive in these handlers will contain a link to the S3 bucket item containing the email. Here is how we can set up an SNS topic in Terraform with a lambda to handle the event.

resource "aws_sns_topic" "ms2" {
  name = "ms2-receipt-sns"
}

resource "aws_sns_topic_subscription" "ms2_receive" {
  topic_arn = "${aws_sns_topic.ms2.arn}"
  protocol = "lambda"
  endpoint = "${aws_lambda_function.ms_receive_mail.arn}"
}

resource "aws_sns_topic" "ms2_ses_error" {
  name = "ms2-ses-error"
}

Create a Lambda to handle your emails

You could at this point do anything you like with the SNS topic to handle emails. You could subscribe to it within your application, or you could hook it up to a Lambda. I’ll show you how you might right a Lambda for this kind of event and act on inbound emails.

Here’s what a Lambda might look like:

import os
from botocore.vendored import requests

# handle the event here
# (the object will contain a url to the S3 item containing the full email)
def handler(event, context):
    print(event)

And here is how we can deploy the Lambda using Terraform.

# this points to your lambda python script and zips it during terrafom apply
data "archive_file" "ms_receive_mail" {
  type = "zip"
  source_file = "${path.module}/${var.receive_dist}"
  output_path = "${path.module}/dist/receive.zip"
}

# receive mail lambda definition uses the data archive file
resource "aws_lambda_function" "ms_receive_mail" {
  role = "${aws_iam_role.lambda.arn}"
  handler = "lambda.handler"
  runtime = "python3.6"
  filename = "${data.archive_file.ms_receive_mail.output_path}"
  function_name = "ms_receive_mail"
  source_code_hash = "${base64sha256(file(data.archive_file.ms_receive_mail.output_path))}"
  timeout = "${var.receive_lambda_timeout}"
}

# allow sns to invoke the lambda
resource "aws_lambda_permission" "sns_notify_ms_receive" {
  statement_id = "AllowExecutionFromSNS"
  action = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.ms_receive_mail.function_name}"
  principal = "sns.amazonaws.com"
  source_arn = "${aws_sns_topic.ms2.arn}"
}

Using SES

Recieving an email

We now have Route53, SES, SNS and a Lambda in place to handle inbound emails. We can now send an email from some email client to “[e-mail]” and watch our Lambda get invoked. From this point it’s up to you how you want to handle inbound email events.

Sending a transactional email

Now to the crux of the article: sending transactional emails. With the setup we created above we can now send emails from any address on our registered domain with a simple API call to AWS. For this, the best approach is to use the AWS SDK in your given language. I use a lot of Kotlin for MailSlurp so I’ll show you my approach using the AWS SES SDK for Java.

// gradle dependencies
dependencies {
        compile "com.amazonaws:aws-java-sdk-ses:${awsVersion}"
}


// lets create a client for sending and receiving emails with SES
@Service
class SESClient {

    @Autowired
    lateinit var appConfig: AppConfig

    lateinit var client: AmazonSimpleEmailService

    @PostConstruct
    fun setup() {
        client = AmazonSimpleEmailServiceClientBuilder.standard().withRegion(appConfig.region).build()
    }
}

And here is how you would send an email with the given client.

@Service
class SESService : MailService {

    @Autowired
    private lateinit var appConfig: AppConfig

    @Autowired
    private lateinit var sesClient: SESClient

    override fun sendEmail(emailOptions: SendEmailOptions) {
        // validate options
        if (emailOptions.to.isEmpty()) {
            throw Error400("No `to address` found for send")
        }
        // build message
        val content = Content().withCharset(emailOptions.charset).withData(emailOptions.body)
        val body = if (emailOptions.isHTML) {
            Body().withHtml(content)
        }
        else {
            Body().withText(content)
        }
        val message = Message()
                .withBody(body)
                .withSubject(Content().withCharset(emailOptions.charset).withData(emailOptions.subject))
        val request = SendEmailRequest()
                .withDestination(Destination()
                        .withToAddresses(emailOptions.to)
                        .withBccAddresses(emailOptions.bcc)
                        .withCcAddresses(emailOptions.cc))
                .withMessage(message)
                .withReplyToAddresses(emailOptions.replyTo.orElse(appConfig.defaultReplyTo))
                .withSource(emailOptions.from.orElseThrow { Error400("Missing `from address` for email send") })

        // send
        sesClient.client.sendEmail(request)
    }
}

Pros and cons of AWS SES

SES is great but there are some downsides:

• There is no GUI for non-coders to send and receive emails
• There is no transactional email contact management
• Lacking many features that make other providers great

Positives include:

• Cheaper than other providers
• More control
• Fits with existing infrastructure

Testing your transactional emails

So all this infrastructure wouldn’t be much use if we we’re sure of it’s reliability. That’s why end-to-end testing email functionality is so important. We can test whether our infra receives and handles real emails using MailSlurp!

How does email testing work?

Basically, MailSlurp is an API that let’s you create new email addresses for use with testing. You can create as many as you want and send and receive emails from them via an API or SDKs.

Writing a test for inbound emails

First you need to sign up for MailSlurp and obtain a free API Key. Then let’s use one of the SDKs to test our domain.

We can install the SDK for javascript using npm install mailslurp-client. Then in a test framework like Jest or Mocha we can write:

// javascript jest example. other SDKs and REST APIs available
import * as MailSlurp from "mailslurp-client";
const api = new MailSlurp({ apiKey: "your-api-key" });

test("my app can receive and handle emails", async () => {
  // create a new email address for this test
  const inbox = await api.createInbox();

  // send an email from the new random address to your application
  // the email will by sent from something like '[e-mail]`
  await api.sendEmail(inbox.id, { to: "[e-mail]" });

  // assert that app has handled the email in the way that we chose
  // this function would be written by you and validate some action your lambda took
  expect(myAppReceivedEmail()).resolves.toBe(true);
});

If we run this test and it passes, that means that all our infrastructure is working and our Lambda is correctly handling our events.

We just tested our inbound email handling code with a real email address and real emails.

Testing transactional emails (are they actually sent?)

Lastly let’s test whether our app can actually send emails and whether they are properly received. We could do this by sending an email to our own personal account each time and manually confirming its existance but that won’t fit well into an automated CD test suite. With MailSlurp we can test the receiving or real transactional emails with real addresses during automated testing. This ensures that key parts of our application (emails) are actually working.

What does it look like?

// can your app send emails properly
import * as MailSlurp from "mailslurp-client"
const api = new MailSlurp({ apiKey: "your-api-key" })

test('my app can send emails', async () => {
    // create a new email address for this test
    const inbox = await api.createInbox()

    // trigger an app action that sends an email
    // to the new email address. this might be a new sign-up
    // welcome email, or a new product payment for example
    await signUpForMyApp(inbox.emailAddress)

    // fetch welcome email from the inbox we created
    const emails = await api.getEmails(inbox.id, { minCount: 1 })

    // assert that the correct email was sent
    expect(emails[0].length).toBe(1)
    expect(emails[0].content).toBe(expectedContent)

    // profit!
}

Now if this test passes we can rest assured that our application will send transactional emails are the correct times and that those emails are actually delivered! This is fantastic!

Wrapping it all up

In summary, transactional emails are an important part of many applications. Doing so with AWS SES is a great way to gain more control over other services like MailChimp and MailGun. However, which ever solution you choose remember to test your transactional email sending and receiving with real email addresses. MailSlurp makes that easy and automated so you know for sure that your app is sending and receiving what it should.

I hope that helped!

Thanks,

Jack from MailSlurp