Amazon Simple Email Service (SES) is a powerful tool for sending emails at scale, but implementing it effectively comes with challenges, as with any technology. The service is used by Amazon at scale (over 100+ billion emails per year!), along with Netflix, Reddit, and DuoLingo.
Recently, while working on an SES-based project, I encountered several issues that taught me valuable lessons about testing and optimizing email solutions. The solution involved users emailing into SES (Inbound emails), these emails to be classified and if the email contained any issues (i.e. the email subject, the file attachment format etc) then to email back the client via SES (Outbound email) with the error for it to be addressed and resent. A high-level diagram of this solution can be found below:
In this post, I’ll share my key learnings and provide actionable recommendations for building robust email systems with SES.
Key Learnings
1. Email Size Limits
One of the initial challenges we encountered was the size limitation imposed by Amazon SES. At first glance, we believed the maximum email size was 40 MB (using the SES v2 API). However, the devil is in the detail: the documentation specifies this limit is imposed "after base64 encoding". This effectively means the raw email size must be kept under ~30-35 MB. Exceeding this limit can lead to email rejections.
Tip: Always account for base64 encoding when calculating the email size, particularly when sending large attachments. Be sure to review (more than once!) the list of SES Service Quotas.
2. Email Address & Email Attachment Case Sensitivity
Another issue we observed during live usage was that case sensitivity in file extensions (e.g., .pdf vs .PDF) and/or email addresses (e.g., hello@example.com vs HELLO@EXAMPLE.COM) can cause unexpected errors and impact functionality.
Tip: Make sure to account for this within your testing so that your application can deal with such differences.
3. Dash Characters in Email Subjects
In our use case, we also incorporated automation based on the subject the user provided. As part of this, we used the "-" (dash) delimiter to split up categories of data which can then be automated to perform different actions based on the values provided. However, upon further usage in a Production context, we started to see that there are multiple dash symbols (who would have known?!). These include; Hyphen-minus (-), En dash (–) and Em dash (—) symbols.
These symbols can accidentally appear in email subjects, especially when dealing with actual people drafting emails (where copy-paste easily causes this mishap). Not accounting for these different types of characters can cause inconsistencies in processing.
Tip: Be sure to either be strict on these types of characters or include all of these different types of the same symbol within a list of accepted delimiters (which we used DynamoDB to store and retrieve).
4. Auto-Replies (Preventing Loops)
Emails sent to recipients with auto-reply settings can create loops if not managed properly. For instance, one auto-reply could trigger another, leading to unnecessary system strain.
Tip: Implement logic to identify and ignore auto-replies.
5. Handling Multiple Attachments
Test scenarios involving high volumes of files to ensure stability and proper handling when sending or receiving emails with multiple attachments.
_Tip: Be sure to test the limits of your application by testing scenarios whereby a large number of email attachments are attached. _
Side note: If you are also using Step Functions to process incoming emails, be sure to include a large number of attachments within your tests and that this does not exceed the Step Function Payload Size Limit of 256KB.
6. Source & Destination Address Handling
For Inbound emails into SES, AWS provide you with an incoming email which includes several attributes, however, they do not recommend the usage of specific attributes. For example, relying on email headers (mail headers object) for parsing email addresses (e.g., source and destination) introduced errors due to special characters like apostrophes, semi-colons etc being included in the email address field. Instead, we found it more reliable to use the details included within the SES events' mail source/destination object.
Tip: Make sure to test your application for incoming and outgoing emails with multiple email addresses from different emailing sources (human/machine generated).
7. Multiple Destination Addresses
Sending or receiving emails to multiple recipients, especially in scenarios with programmatically generated addresses, requires thorough testing to ensure all addresses are processed correctly.
Additionally, for inbound emails, please note that the mail"destination" can contain multiple addresses. Be careful when configuring your application to take this into account when multiple recipients are involved.
Tip: See point 6!
8. Corrupt Attachments
Corrupt or incomplete attachments (e.g., damaged PDFs) caused solution failures. The system should handle these gracefully and provide meaningful error messages to users.
Tip: Be sure to include corrupt attachments within your testing (based on the file types which is being accepted/supported by your application).
9. UTF Encoding
We encountered issues with emails encoded in UTF-7, which caused errors when converting EML files to PDFs. This highlighted the importance of robust encoding support for email processing.
Tip: Be sure to include testing which includes UTF-7/other encoding so that your application can handle these differences and respond accordingly.
Recommendations for Testing and Implementation
Based on these learnings, here are my top recommendations for anyone implementing SES:
Comprehensive Testing - Test all edge cases, including large attachments, multiple email addresses, encoding variations, symbol/case variations, and unusual address formats.
Understand SES Limitations - Familiarize yourself with SES’s size limits, encoding requirements, and other constraints before designing your solution.
Error Handling - Build robust error-handling mechanisms to deal with corrupt files, encoding issues, and size limits.
Scalability - Monitor payload limits in workflows like AWS Step Functions and design for scalability.
Prevention of Loops - Implement safeguards to detect and ignore auto-replies or other system-generated emails.
Final Thoughts
SES is a powerful tool, but its full potential can only be realized through meticulous planning and testing. These lessons have strengthened my approach to building scalable and resilient email systems. I hope this blog helps others navigate SES implementation more effectively.
If you’ve had similar experiences or have tips of your own, I’d love to hear about them - let’s discuss them in the comments!
Top comments (0)