⚠️ Hey! This blog post was initially published on my own blog. Check it out from the source: https://cloudnature.net/blog/building-a-serverless-newsletter-your-guide-to-aws-and-amazon-ses
Discover the secrets of creating a serverless newsletter using AWS and Amazon SES. Dive into SST, Middy with Typescript, and master everything you need to know about SES for a seamless newsletter service.
Introduction
Have you ever wrestled with Amazon SES, firing off emails left and right, dealing with bounces, and handling complaints? I sure have. Then, a lightbulb moment hit 💡, I want to build my own newsletter infrastructure! Sure, there are quicker routes out there, but where's the fun in that?
So, buckle up for this blog post. We're delving into serverless setups and everything there is to know about Amazon SES. Of course, we're making things work at scale!
And hold tight, because one section is titled "Lambda? Nope, thank you!". Can serverless exists without Lambda functions? 🤔
Project Overview
Creating a newsletter infrastructure is quite a puzzle. It needs to be quick, handle heavy traffic, and stay operational no matter what. So, why did I opt for a complete serverless approach? It's straightforward: serverless computing is the superhero of performance, resilience, and scalability 🦸.
And hey, it's also budget-friendly. Newsletters don't need to be active 24/7; they wake up once a day, get the job done, and then rest until the next day. Let's break down the reasons behind choosing a serverless setup for this newsletter. But before we get there, let's tackle the elephant in the room by exploring three ways to create a newsletter.
The Fastest Way
Do I really have to spell it out? Numerous newsletter providers exist, so many choices, really. 😐
The Simple Way
Setting up a basic serverless newsletter infrastructure on AWS is a breeze, requiring just a few essential components:
- Validate your sender identity in Amazon SES
- Implement a straightforward AWS Step Function for sending emails with AWS
- Set up a single web service to publish the newsletter
With these in place, you can kick off your newsletter distribution. The beauty of this solution? In just 8-16 hours, you've built your custom newsletter infrastructure.
But hold on, we're not aiming for a quick fix here, are we? We're on a mission to dedicate an entire weekend just to the HTML and CSS 🙄. Don't worry, I'm not trying to spook anyone. I believe in your champion-level skills, capable of centering a div better than I ever could.
Don't worry, I'll spare you the HTML and CSS details of the newsletter templates.
Let's shift our focus to the complete AWS infrastructure, which is the reason we're all gathered here 🚀.
The Complete Way
Building a serverless infrastructure doesn't have to be a complete mess. I'm not a fan of unnecessary complexity. Instead, let's keep it simple by assembling basic components, creating completeness, and minimizing confusion.
So, what are these foundational bricks? Let's break them down:
- Setting up Amazon SES
- Additional Amazon SES configurations for Bounces, Complaints, Clicks (don't worry, we'll dive into this shortly)
- API Web Services (AWS 😜) for seamless newsletter operations
- AWS Step function workflow for sending emails
These bricks, when glued together, form the backbone of our newsletter.
Now, let's delve into each element while keeping things clear and straightforward.
Amazon SES Setup
Setting up SES (Simple Email Service) is the starting point for our newsletter infrastructure. It's the AWS service that powers our email delivery. Let's look at the infrastructure needed to send emails with AWS.
Domain Validation
Before we jump into the technicalities, let's get our domain validated. Why? Email providers want to know who owns that email address. The drill? Domain validation through DNS (basically, adding a few CNAME records in your DNS provider). In my case, a quick validation of https://cloudnature.net in Route53, and we're good to go.
Find the full identity validation process in the Infrastructure As Code section at the bottom of the GitHub repository.
But hold on, validation alone is not enough for email providers. To be top-tier among other email providers, we need a bit more from Amazon SES:
- DKIM (DomainKeys Identified Mail)
- Custom MAIL FROM (cue SPF authentication and DMARC compliance)
- BIMI compliance
Now, let's look into all of them. The names might be a mouthful, but they're harmless, I promise 😜
DomainKeys Identified Mail (DKIM)
Ensures email authenticity by adding a digital signature. This is your email's passport to a trustworthy inbox. DKIM is a security standard, ensures that an email claiming to be from a specific domain is genuinely authorized by the domain owner.
Opting for Easy DKIM means Amazon SES takes the lead. It generates a public-private key pair and includes a DKIM signature in every message sent from that identity. Your job? Well, nothing much, Amazon SES has got it covered!
Custom MAIL FROM
Ever wondered about the addresses behind the emails you send? There's the From address, visible to the recipient, and then there's the MAIL FROM address, indicating the message's origin. By default, Amazon SES assigns a MAIL FROM domain for your outgoing messages, using a subdomain of amazonses.com.
But what if you want to add a personal touch to your MAIL FROM address? As seen in the picture above, adding your own MAIL FROM means you get rid of that ugly "via amazonses.com" tag near your name!
For this, you simply need to comply with the Sender Policy Framework (SPF), an email validation standard designed to prevent email spoofing. While at it, let's add another protocol: Domain-based Message Authentication, Reporting, and Conformance (DMARC). DMARC, fancy name, simple job: it helps detect email spoofing, allowing you to guide the email provider on how to act.
To check your DMARC compliance, visit 🔗 DMARC Inspector.
Brand Indicators for Message Identification (BIMI)
Give your emails a visual identity. Let's set up BIMI and become a recognized face in your recipient's inbox.
At this point, we're almost there. But wouldn't it be nice to have a logo as your email profile image? That's where BIMI comes in. If you've followed the previous steps, your Amazon SES configuration is nearly ready.
All we need are two things:
- Create an SVG for our profile image.
- Add the BIMI record to our DNS registrar.
To convert your SVG logo (modern SVGs may not be sufficient for BIMI), download a simple yet efficient program in GitHub repository https://github.com/authindicators/svg-ps-converters/tree/master.
Afterward, upload your logo and add a DNS record like so:
default._bimi.dev.cloudnature.net TXT v=BIMI1;l=https://cloudnature.net/logos/logo.svg;
And there you have it! Check your BIMI compliance at 🔗 Bimi Group.
Email Sending Challenges
Sending emails may seem like a straightforward task, but the journey from your server to your recipient's inbox can be a bumpy ride. In this section, we'll navigate through common challenges encountered while sending emails.
We need to be careful because AWS policies are strict, so manual checking may not be enough. Here's how our automatism flows:
- Some triggering event arrives
- Amazon SES triggers Amazon SNS
- Amazon SNS triggers AWS Lambda
- AWS Lambda run custom logic to decide if the email address needs to be removed from the newsletter list (for example if the email is unexistent), if so, it remove the record from the DynamoDB table
Let's see briefly one case at the time 👇.
Handling Bounces
Bounces in the email world are annoying but more common than you could think. They happen when an email cannot be delivered to the recipient, either due to an invalid email address or a temporary issue. In our serverless setup, dealing with bounces is crucial for maintaining a clean and effective newsletter infrastructure.
So, what's the drill when a bounce event pops up? There are different types of bounces, but for the hard ones, we remove that email address from our newsletter list.
Addressing Complaints
Not every email recipient is thrilled to receive newsletters, and complaints may arise. Complaints occur when a recipient marks your email as spam or unwanted. Managing complaints is key to maintaining a positive sender reputation and ensuring your emails land in the inbox.
Now, what happen when a complaint event lands in our logs? Developers don't pay attention to warnings, so treat it like a yellow card and investigate why it got the spam label. If it becomes a recurring event, consider removing that email address from your subscribers list.
To check if you are following every best practices, use this website: Mail Tester
Troubleshooting Errors
In the intricate world of email sending, errors can pop up unexpectedly. From server glitches to misconfigurations, troubleshooting errors is an essential part of maintaining a reliable newsletter infrastructure.
Always keep a watchful eye on error logging. It's like your troubleshooting toolkit, it helps you identify issues worth fixing.
Tracking Clicks Per Link
Knowing how recipients engage with your newsletters is gold. Tracking clicks per links gives you the insight on the effectiveness of your email campaigns. While Amazon SES doesn't offer this out of the box, we've got to add to our serverless architecture an additional SNS Topic and an AWS Lambda. This duo saves records in our Amazon DynamoDB database, tracking every piece of information about a specific link.
A big shoutout to Guillermo Ojeda for enlightening me on the significance of this metric during our conversation about must-have metrics for newsletters. Thanks Guillermo 🙏.
Newsletter API Web Services
Building the AWS serverless infrastructure was a piece of cache (sorry, I couldn't resist). Let's break down the stack:
- SST with CDK integration
- Typescript
- Middy
These tools are the unsung heroes that took my developer experience to a whole new level. I won't dive too deep here; you can find the full repository with all the goodies at the bottom.
Now, which APIs are we talking about?
- CRUD APIs for admin operations on newsletter items
- Newsletter publish and unpublish APIs
- Newsletter subscribe and unsubscribe APIs
Pretty standard stuff, right? For the Amazon DynamoDB connection, I opted for dynamodb-toolbox: a library that proved its worth and is likely to feature in my future projects too.
Take a peek at a snippet of the Postman APIs developed for this project. One API in the mix serves as the documentation. If you're authenticated and give it a buzz, it'll hand you the JSON Postman collection, all decked out with documentation ready to use (just remember to include environment variables).
AWS Lambda functions? No, Thank You!
Alright, let's kick off this chapter by addressing the elephant in the serverless room: AWS Lambda functions. Now, you might wonder, "Why didn't we use Lambda functions?" Well, the answer is as simple as our serverless setup: a single AWS Step Function took the reins and orchestrated the entire show. Say goodbye to the need for countless Lambda functions or a single AWS Lambda function doing all the dirty work. Interested? Let's explore why one not-so-big AWS Step function stole the spotlight.
This overview divides our step function into three main parts:
- Workflow that sends the first email: a critical step to ensure everything is functioning as expected. An initial test to our own email address helps us validate the integrity of our setup.
- List every subscriber from Amazon DynamoDB: an essential task where we fetch the list of subscribers and organize them into batches of 50 email addresses each.
- For each batch, send out emails with Amazon SES: the final leg of our journey involves the systematic sending of emails in batches using Amazon SES.
Now, let's dive into each step and explore the challenges that may arise when building this intricate workflow with AWS Step Functions.
Sending Test Email
This segment involves a straightforward process. We retrieve the newsletter item from Amazon DynamoDB, process and convert it into a valid JSON, and then proceed to send the email using the Amazon SES SendBulkEmail V2 API.
SendBulkEmail stands out as the best practice, ensuring that each recipient sees only their own name and email address in the To header of the messages they receive.
This step proves to be exceptionally useful to know how the newsletter will truly appear in our email inbox. The simplicity and effectiveness of this approach allow for quick identification and correction if any HTML issues arise. Following this, we can unpublish and reschedule the newsletter publish date.
Processing Subscribers
This phase proved to be the most intricate to develop! Pagination for Amazon DynamoDB works differently compared to many NoSQL or SQL databases. You simply inform the database about the last item you retrieved. Using that as a starting point, we can list through every single subscriber.
The objective here is to produce a list of subscribed email addresses. However, to create that list, we need to go through a few rounds of Pass states. In my case, this was the workflow:
- First Step: Query elements from Amazon DynamoDB (not scan)
- Second Step: Create an array of items that merges previous items and new ones: States.Array($.dynamodbConfig.items[ * ], $.subscribers.Items[ * ].email.S)
- Third Step: The result is pretty ugly:
"items": [
[
"emailaddress3@example.com"
],
[
"emailaddress@example.com",
"emailaddress1@example.com",
"emailaddress2@example.com"
]
]
This means we need to flatten this list like so: $.dynamodbConfig.items[ * ][ * ]
- Fourth Step: Check if there are more elements. If not, we are going to close this process. If, in fact, there are more subscribers, we save the items in another object and start all over again.
Sending Out Emails
Now, we are almost ready to "hit" that send button and dispatch our newsletters. The workflow here is comparatively simpler than the previous one, but we need to make some adjustments to ensure we can call the Amazon SES SendBulkEmail at scale.
The process begins with this input:
[
"emailaddress@example.com",
"emailaddress1@example.com",
"emailaddress2@example.com",
"emailaddress3@example.com",
...
]
Here's what we need to achieve:
- Create a valid object with parameters like so:
[
{
"Destination": {
"ToAddresses": [
"emailaddress@example.com"
]
}
},
...
]
While this task is relatively straightforward, we need to loop through each email address and add the payload as shown above.
- Divide the object containing email addresses into batches of 50 items (due to Amazon SES hard quotas). Even better, we can use the AWS Step Function intrinsic function like so States.ArrayPartition($.toAddressDestinations, 50)
- For each batch, call the Amazon SES SendBulkEmail with the correct parameter structure.
One thing that proved super useful was getting the parameter from outside the Map state. This is possible using ItemSelector like so:
{
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value",
"toAddressDestinations.$": "$.toAddressDestinations",
"templateData.$": "$.templateData"
}
Finally, our workflow is done. Before wrapping everything up, we need to set up a proper error flow. In this case, if there are any errors, my workflow stops, and I get alerted via email.
See? A complete workflow without AWS Lambda functions!
Challenges
During the process of building the serverless AWS infrastructure, I encountered several challenges, but I'm glad I was able to complete the AWS Step Function workflow without the need for AWS Lambda functions.
Let's explore some of the challenges I faced (and no, I won't talk about the enormous time spent on HTML/CSS 😜):
Unfortunately, there is no built-in utility in AWS Step Functions to enable marshalling and unmarshalling results to and from Amazon DynamoDB. This means you need to handle .S or .N every time.
You can identify errors from SendBulkEmail by examining this object: $.BulkEmailEntryResults[ * ].Error. In my case, I used this object to list every error. If any were found, I raised an error, stopping the entire workflow.
Flattening the DynamoDB output requires two steps. Unfortunately, it's not possible to accomplish this in a single, straightforward step.
It's not possible to use AWS Step Function intrinsic functions within the Choice state. Consequently, in many cases, a new Pass step needed to be added.
Metrics
Visualization of metrics is crucial, and SES offers two main capabilities:
- Metrics for tracking Bounces, Complaints, Sends, and more.
- Virtual Deliverability Manager (VDM) for Amazon SES.
The default metrics are fairly standard. Now, let's look into VDM functionality.
Virtual Deliverability Manager for Amazon SES
VDM provides a lot of metrics, including soft and hard bounce rates, open rates for emails, and more. While it lacks a "click per link" metric, which we had to build that ourselves. Apart from this, there aren't many fancy or custom metrics available; however, the standard insights provided are fundamental for any newsletter.
Additionally, you can download a CSV file containing every single metric available in Amazon SES, which proves to be quite useful.
Total Pricing
Let's quickly summarize the pricing for Amazon SES and AWS Step Functions. Here's a comparison:
Amazon SES:
- 1,000 emails: ~ 0.3$
- 10,000 emails: ~ 3$
- 1,000,000 emails: ~ 30$
AWS Step Functions:
- 1,000 emails: ~ 2,150 executions = free
- 10,000 emails: ~ 21,500 executions = ~ 0.5$
- 1,000,000 emails: ~ 2,150,000 executions = ~ 55$
⚠️ The price could potentially be reduced to 3$ for AWS Step Functions by incorporating the "Compute to bulk destination" Map into the DynamoDB processing. This could be a notable improvement for the future.
Conclusion
Creating a serverless newsletter with Amazon SES and AWS Step Functions is a potent and cost-efficient solution. By leveraging SES for reliable email delivery and Step Functions for orchestration, we've built a scalable system without traditional Lambda functions.
Well now you know how to create a serverless newsletter on AWS, the only thing I didn't show you is how it turns out!
Check the results 👉 https://cloudnature.net/newsletters/subscribe
I'll post monthly, covering the latest AWS trends, community blog posts, and Cloud-related news. If you're an AWS Cloud Architect, Cloud Engineer, DevOps, or AI/ML Developer, you'll find valuable content.
You can find the repository here: https://github.com/Depaa/newsletter-manager-template 😉.
If you enjoyed this article, please let me know in the comment section or send me a DM. I'm always happy to chat! ✌️
Thank you so much for reading! 🙏 Keep an eye out for more AWS-related posts, and feel free to connect with me on LinkedIn 👉 https://www.linkedin.com/in/matteo-depascale/.
References
- https://docs.aws.amazon.com/ses/
- https://middy.js.org/
- https://docs.sst.dev/
- https://www.mail-tester.com/
Disclaimer: opinions expressed are solely my own and do not express the views or opinions of my employer.
Top comments (0)