DEV Community: jeff-goldstein

Archiving Emails: A How-To Guide for Tracking Sent Mail

jeff-goldstein — Wed, 20 Sep 2017 13:05:36 +0000

Not every email sender needs the ability to archive their emails, but for some segments, like Financial or Health Care, tracking sent emails is often a requirement. So I decided to create a sample program that does just that. As with most of my blog posts, I have a corresponding Github repository you can take a look at if you’re interested.

You are more than welcome to use this repository as a starting point to your project or modify and update the repository for others to leverage.

There are a couple of different ways to support archiving. For this post, I decided to focus on just one method, the “Archive function within the SparkPost SMTP call. Another approach would be to store the template and the data needed to merge into that template. I may address that approach with some sample code in a future post.

But let’s get back to the approach that we’re covering today. Here is a graphic of the three distinct steps in this process; starting with the email creation down to the archiving of the email body and supporting components.

At a 32,000 foot level, those steps manifest into three development steps:

Set up an inbound relay Webhook within SparkPost. Relay Webhooks are a way to instruct SparkPost to accept inbound emails on your behalf and forward it to you over HTTP for your own consumption.
Create a tool that will receive the Relay Webhook information (the inbound email) in the form of a JSON payload.
Send an email via the SMTP protocol adding the SparkPost SMTP API header. The SMTP API header will contain the email address of that you will use for your archiving domain.

Step 1 and 2 share the “chicken or the egg syndrome. They’re so intertwined you can’t do one without the other so it may not matter which you do first, but I’m going address the setup first because you can’t do full unit testing without getting DNS and the Webhooks in place.

“Enabling Inbound Email Relaying & Relay Webhooks will walk you through the Relay Webhook setup process, but I have a few helpful hints to add. I’ll walk through the steps of the document at a high level, then add some examples that you may find useful.

Set up an Inbox Relay Webhook

The first step in the document mentioned above is to set up mx records to have the email for a given domain forwarded to SparkPost for processing. The second step in the aforementioned document has you creating an Inbound Domain using the SparkPost inbound-domain RESTful API. The body of the call is straightforward with one key/value pair, with the key named “domain”. For my example, the value matches my MX entry for the domain named geekwithapersonality.com (no judgements on the domain name, please). Thus the body of the API call is simply:

{
    "domain": "geekwithapersonality.com"
}

There is no UI functionality within SparkPost to set this up, so you must use a client like Postman or an application like CURL. SparkPost does have a Postman collection that you can leverage, which makes this step extremely easy. Next, you need to tie that inbound-domain to where your application is running. As a footnote, there can only be one relay-webhook application per inbound-domain. In this step you will call the relay-webhooks RESTful API, with a body similar to this one:

{
  "name": "Replies Webhook",
  "target": "http://www.geekwithapersonality.com/cgi-bin/inbox-relay.php",
  "auth_token": "",
  "match": {
    "protocol": "SMTP",
    "domain": "geekwithapersonality.com"
    }
}

Notice how the domain key/value pair uses the same domain value, geekwithapersonality, as the entry employed in the RESTful API inbound-domain call. This step is the one that ties the two together. So any emails that go to the domain geekwithapersonality.com will be directed to SparkPost, then translated into a JSON structure for processing. Again, there is no SparkPost UI component for this step, so you need to use something like Postman or CURL to execute this step. Unless you’re an ESP processing inbound messages for many clients, this is typically a very rare task.

Create a Collector to Process and Store the Archive Email

Even though the code is sitting in Github, I’ll describe the components and the logic for each piece. Overall, this code was fairly straightforward to write. I decide to archive the following components of each email in separate files:

All the headers used to send the email. Because the archive email went to a different location than the original email, these headers are NOT 100% equivalent to the original email, but they are close.
The full JSON structure.
The text body with the original To email address, along with the From and Subject appended to the top of the text file.
The HTML body with the original To email address along with the From and Subject appended to the top of the HTML file.
The rfc822 compliant email. This allows an application to grab the whole file and resend it if necessary.

In this sample code, I placed each group of files in their own directory and logged each Archive email processed. Beyond placing the data into a directory, I am NOT archiving these items into something more enterprise level like S3 or HP Autonomy. I’ll leave that process for another time, but there are so many CMS or archiving tools out there that creating a dozen different hooks into archiving tools would only scratch the surface. I also want the process to be as fast as possible, so I think the archiving tool should only be loosely coupled and either run off an index/queue/stack of processed items or perhaps the archiving tool will simply look for new directories and relocate them to a more permanent location.

In order to create a unique directory for each archive, I created directories with a combination of the name of the original email recipient address along with a timestamp.

The structure of the inbound webhook structure may change over time but at the moment the current structure can be found within the Relay Webhooks Example Payloads documentation of SparkPost.

Here are the json values that I grabbed in order to either store them separately or process them further:

msys/relay_message/content/to: This is the email address for the original target of the content.
msys/relay_message/rcpt_to: This is the email address used to send the archive message.
msys/relay_message/friendly_from: This is the from address presented in the original email address.
msys/relay_message/content/subject: Subject of the email.
msys/relay_message/content/headers: Once pulled, I format these into a nice html (ok, semi-nice) format for storage and reference.
msys/relay_message/content/html: The HTML body of the email.
msys/relay_message/content/text: The text body of the email.
msys/relay_message/content/email_rfc822: The full email body in rfc822 standard. It’s save as an eml file which allows most email clients to open directly.

Once I get all of this data, I process the headers into an html format. From there, I add the To, From and Subject to the HTML and Text bodies and save everything. Finally, I create a log entry for that specific entry.

Send an Archived Copy of the Email

The Archiving functionality can only be initiated by using the SMTP API (which, by the way, is a horribly named industry standard for an added header for your SMTP call). For SparkPost, the SMTP API is actually a JSON formatted header called X-MSYS-API. In that header, you are able to set items like campaign id, metadata, IP Pools, CC and BCC email addresses in addition to a list of Archiving email addresses. Here is a sample X-MSYS-API header:

X-MSYS-API:
{
  "campaign_id": "Jeffs BBQ Campaign",
  "metadata" : {
            "home_address_region": “West”,
            "smoker": "SMB"
  },
  "archive": [
             “archive@mail.geekwithapersonality.com”
  ],
"options" : {
            "open_tracking": false,
            "click_tracking": true,
            "transactional": false,
            "ip_pool": "sp_shared",
}
}

This isn’t a complete sample with all of the options available, but illustrates what a typical header looks like. Notice that the original recipient’s email address isn’t in the X-MSYS-API header, but you’ll get it in the Reply Webhook.

In the Github repository, I have added a test PHP application that calls SMTP by using the PHPMailer framework. While calling SMTP and adding in the X-MSYS-API header with an archive email address is probably the best approach for thorough unit testing, it isn’t exactly necessary. Any email sent to the domain you are targeting will get processed the same way as an archive email. So I did most of my unit testing by sending an email to my geekwithapersonality domain, either by a Postman transmission SparkPost API call or simply sending an email via my MAC email client. Neither is really an archive, but the inbox relay doesn’t care and treats all emails the same.

Voila! Archiving Emails is Simple!

Those are the three steps to set up archiving. If you are using SMTP to send emails to SparkPost, you’re probably already using the X-MSYS-API headers, so just adding the archive entry will be easy for you. Then all you need to do is point email for your archiving domain to SparkPost and create a simple app to capture the data.

I hope this was helpful – if you have any questions on archiving emails, email templates, etc, please feel free to reach out to me directly.

Happy Sending!

The post Archiving Emails: A How-To Guide for Tracking Sent Mail appeared first on SparkPost.

Our Top 5 Email Template Hacks

jeff-goldstein — Wed, 09 Aug 2017 13:01:35 +0000

Today, I thought I would point out a few of my favorite tricks when creating templates. Let’s call them email template hacks. I’ve briefly mentioned some of these in previous blogs, so today I’ll expand on those ideas and add in some use cases and examples.

1. Leveraging Substitution Fields in CSS

The first trick is to use substitution fields for your CSS values. Colors, fonts, height, pretty much anything can be substituted. Many email clients leverage header styles while others don’t. However you choose to tackle formatting your emails across various clients, you can leverage substitutions both inline, in the header block or both.

There are a few different approaches available if you want to inject CSS in the header block in order to maintain standards through all of your templates. The first method is to simply replace CSS values with substitution data. For example, if modifying the text color for an html <a> tag within the <style> block I would modify the color field to reference a substitution field. Here’s an example:

a {
  text-decoration:none;
  color:{{a_normal_color}};
}

Then, in the transformations JSON call, I would have the corresponding substitution field:

"a_normal_color" : "#378cd2"

This is the easiest way to modify the look and feel without having to rummage through hundreds of template and make changes. This also assumes that you are holding those values somewhere on your server and can retrieve them fairly easily for the transmission JSON creation.

Building this example up, you may use a dynamic_html and/or dynamic_text blocks. This allows you to make wholesale changes by bringing in large CSS blocks that you may be storing in a repository of standards. Change the blocks in that repository, and you change all of the templates that get the dynamic html/text referenced.

Please keep in mind that dynamic_html/text blocks are held within the global substitution_data blocks, now the recipient substitution_data blocks. A CSS block within the Transmissions dynamic_html structure may look something like this:

"substitution_data": {
 "dynamic_html": {
"css" : "<style type='text/css'> /* resets + globals */
* { 
    margin:0;
    padding:0;
    font-family: arial, sans-serif;
}

body {
    -webkit-font-smoothing:antialiased;
    -webkit-text-size-adjust:none;
    width: 100%;
    height: 100%;
}

img { 
    max-width: 100%; 
}

/* general styles */
body {background-color:#fff;}

a {
  text-decoration:none;
  color:#378cd2;    /* blue */
}
a:hover {color:#82b450; /* green */}

/* body styles */
p {
font-family: arial,helvetica,sans-serif; font-size: 1.2rem; line-height: 1.6rem; color: #3e3e3e; font-weight: normal; margin-top: 0px; margin-bottom: 1.8rem; max-width: 500px;
}

li {
font-family: arial,helvetica,sans-serif; font-size: 1.2rem; line-height: 1.4rem; color: #3e3e3e; font-weight: normal; margin-top: 0px; margin-bottom: .5rem;
}

ul {
padding-left:1.3rem; margin-top:0px; margin-bottom:2rem;
}

h3 {
font-family: arial,helvetica,sans-serif; font-size: 1.3rem; line-height: 1.6rem; color: #378cd2; font-weight: bold; margin-top: 1.1rem; margin-bottom: 1.8rem;
}"}}

This code block is no different than if you had it placed in the HTML template itself. Now it’s just easier to update when referenced into the template via the transmission which pulls from a central repository. The template itself will use this code block with the following entry in the template somewhere in the html <header> block:

{{render_dynamic_content(dynamic_html.css)}}

If you want to go all out, dynamic rendering even allows for recursive substitution fields. This means that the dynamic code block can have substitution fields as well. A good use case for this is when you OEM or White label your product. In the following example, the CSS block is similar to the one above, but there are substitution fields for the CSS values. Those substitution fields are then placed in the global substitution block of the transmission. For example:

"substitution_data": {
"dynamic_html": {
"css" : "<style type='text/css'> /* resets + globals */
* { 
    margin:{{global_margin}};
    padding:{{global_padding}};
    font-family: {{global_font_family}};
}

body {
    -webkit-font-smoothing:antialiased;
    -webkit-text-size-adjust:none;
    width: {{body_width}};
    height: {{body_height}};
}

img { 
    max-width: {{image_max_width}}; 
}

/* general styles */
body {background-color: {{body_background_color}};}

a {
  text-decoration:none;
  color:{{a_normal_color}}; /* blue */
}
a:hover color:{{a_hover_color}}; /* green */}

/* body styles */
p {
{{p_style}};
}

li {
{{li_style}};
}

ul {
{{ul_style}};
}

h3 {
{{h3_style}};
}"},

"global_margin" : 0,
"global_padding": 0,
"global_font_family" : "arial, sans-serif",
"body_width" : "100%",
"body_height" : "100%",
"image_max_width" : "100%",
"body_background_color" : "#378cd2",
"a_normal_color" : "#378cd2",
"a_hover_color" : "#82b450",
"p_style" : "font-family: arial,helvetica,sans-serif; font-size: 1.2rem; line-height: 1.6rem; color: #3e3e3e; font-weight: normal; margin-top: 0px; margin-bottom: 1.8rem; max-width: 500px",
"li_style" : "font-family: arial,helvetica,sans-serif; font-size: 1.2rem; line-height: 1.4rem; color: #3e3e3e; font-weight: normal; margin-top: 0px; margin-bottom: .5rem",
"ul_style" : "padding-left:1.3rem; margin-top:0px; margin-bottom:2rem",
"h3_style" : "font-family: arial,helvetica,sans-serif; font-size: 1.3rem; line-height: 1.6rem; color: #53e00d; font-weight: bold; margin-top: 1.1rem; margin-bottom: 1.8rem
},

But wait, what if I want to use inline CSS, you ask? Well, just change those big blocks of CSS settings and make them substitution fields.

2. Not sending the email at all if certain data does not exist

As emails become more personalized, they start to look and feel like transactional emails. This is a great trend, but it opens up a greater possibility for unfinished emails. Let’s say a job board sends lists of job opportunities to all active members each morning. Hopefully there are checks and balances that stop the application from requesting for an email to go out if there are no matches for that user, but as a template designer, I don’t want to rely on that. So a little trick that I use is to check for a specific substitution field or array that must be present; if it doesn’t exist, I call a non-existent function with the following line somewhere in the template (I tend to put this on the bottom). This will automatically kill that email from being generated so the bad email doesn’t make it out the door. In the example below, I’m checking for the ProductList array, and if it doesn’t exist, I call the function ‘crash’ with the parameter ‘onpurpose’.

{{if !ProductList}} {{ crash(onpurpose) }}{{end}}

3. Validating data before creating the table for arrays

SparkPost has a very powerful feature that allows a template to loop through an array in order to display a set of information, like jobs, real estate listings or products. In fact, SparkPost even supports having loops within loops within loops. Often, multiple tables, rows and columns get created order to display this information in the fashion the content creator wishes. But what if there is no data to show in one of those loops, but you still want the email to go out, just without the empty rows or tables? In many of my retail templates, I display a list of products that have ‘x’ number of features for each product. If there are no features, I still want the product to be displayed but I may not need to create some of the corresponding html code for the empty features list. In order to keep the email looking good, I check both the existence and the value of my arrays before building unnecessary tables, rows or columns that would house this information. This trick is solved with a simple two part if statement:

{{ if empty(myarray) or !myarray}}

4. Using substitution fields in the subject line

So you built your template and placed it onto the SparkPost server. Don’t forget to create a dynamic subject line. One of the easiest way’s to get your email clipped by Gmail or buried into a huge thread is to use the same subject line over and over. So go ahead and use substitution fields for your subject. I often put the person’s name, along with a subject field. My Subject field in the SparkPost UI is set to something like:

{{first_name}}, {{subject}}

5. Use dynamic_html for headers and footers

So, continuing on the theme of standards, I like to send my headers and footers to each template via the Transmission API in the form of a dynamic_html entry. This allows me to keep a library of headers and footers that can be easily modified without having to change every template. Because SparkPost doesn’t have the ability to store those headers and footers on the server then insert them into the targeted template, it’s a good practice to have those standard headers and footers in another content management system that marketing can change without developments help. When the transmission json is getting built, the appropriate headers and footers are placed into the dynamic_html section which is then pulled into the template before sending.

Upcoming Email Template Hacks

So those are some of my favorite template tricks. In my next blog, I’m going to tackle the trick of validating that the transmission payload has all of the data the template is expecting to receive. The blog will be accompanied by a full PHP code sample for this validation step.

Happy Sending!

This post was originally published on sparkpost.com