Archiving Emails: A How-To Guide for Tracking Sent Mail

Not every email sender needs the ability to archive their emails, but for some segments, like Financial or Health Care, tracking sent emails is often a requirement. So I decided to create a sample program that does just that. As with most of my blog posts, I have a corresponding Github repository you can take a look at if you’re interested.

You are more than welcome to use this repository as a starting point to your project or modify and update the repository for others to leverage.

There are a couple of different ways to support archiving. For this post, I decided to focus on just one method, the “Archive function within the SparkPost SMTP call. Another approach would be to store the template and the data needed to merge into that template. I may address that approach with some sample code in a future post.

But let’s get back to the approach that we’re covering today. Here is a graphic of the three distinct steps in this process; starting with the email creation down to the archiving of the email body and supporting components.

At a 32,000 foot level, those steps manifest into three development steps:

1. Set up an inbound relay Webhook within SparkPost. Relay Webhooks are a way to instruct SparkPost to accept inbound emails on your behalf and forward it to you over HTTP for your own consumption.
2. Create a tool that will receive the Relay Webhook information (the inbound email) in the form of a JSON payload.
3. Send an email via the SMTP protocol adding the SparkPost SMTP API header. The SMTP API header will contain the email address of that you will use for your archiving domain.

Step 1 and 2 share the “chicken or the egg syndrome. They’re so intertwined you can’t do one without the other so it may not matter which you do first, but I’m going address the setup first because you can’t do full unit testing without getting DNS and the Webhooks in place.

“Enabling Inbound Email Relaying & Relay Webhooks will walk you through the Relay Webhook setup process, but I have a few helpful hints to add. I’ll walk through the steps of the document at a high level, then add some examples that you may find useful.

Set up an Inbox Relay Webhook

The first step in the document mentioned above is to set up mx records to have the email for a given domain forwarded to SparkPost for processing. The second step in the aforementioned document has you creating an Inbound Domain using the SparkPost inbound-domain RESTful API. The body of the call is straightforward with one key/value pair, with the key named “domain”. For my example, the value matches my MX entry for the domain named geekwithapersonality.com (no judgements on the domain name, please). Thus the body of the API call is simply:

{
    "domain": "geekwithapersonality.com"
}

There is no UI functionality within SparkPost to set this up, so you must use a client like Postman or an application like CURL. SparkPost does have a Postman collection that you can leverage, which makes this step extremely easy. Next, you need to tie that inbound-domain to where your application is running. As a footnote, there can only be one relay-webhook application per inbound-domain. In this step you will call the relay-webhooks RESTful API, with a body similar to this one:

{
  "name": "Replies Webhook",
  "target": "http://www.geekwithapersonality.com/cgi-bin/inbox-relay.php",
  "auth_token": "",
  "match": {
    "protocol": "SMTP",
    "domain": "geekwithapersonality.com"
    }
}

Notice how the domain key/value pair uses the same domain value, geekwithapersonality, as the entry employed in the RESTful API inbound-domain call. This step is the one that ties the two together. So any emails that go to the domain geekwithapersonality.com will be directed to SparkPost, then translated into a JSON structure for processing. Again, there is no SparkPost UI component for this step, so you need to use something like Postman or CURL to execute this step. Unless you’re an ESP processing inbound messages for many clients, this is typically a very rare task.

Create a Collector to Process and Store the Archive Email

Even though the code is sitting in Github, I’ll describe the components and the logic for each piece. Overall, this code was fairly straightforward to write. I decide to archive the following components of each email in separate files:

1. All the headers used to send the email. Because the archive email went to a different location than the original email, these headers are NOT 100% equivalent to the original email, but they are close.
2. The full JSON structure.
3. The text body with the original To email address, along with the From and Subject appended to the top of the text file.
4. The HTML body with the original To email address along with the From and Subject appended to the top of the HTML file.
5. The rfc822 compliant email. This allows an application to grab the whole file and resend it if necessary.

In this sample code, I placed each group of files in their own directory and logged each Archive email processed. Beyond placing the data into a directory, I am NOT archiving these items into something more enterprise level like S3 or HP Autonomy. I’ll leave that process for another time, but there are so many CMS or archiving tools out there that creating a dozen different hooks into archiving tools would only scratch the surface. I also want the process to be as fast as possible, so I think the archiving tool should only be loosely coupled and either run off an index/queue/stack of processed items or perhaps the archiving tool will simply look for new directories and relocate them to a more permanent location.

In order to create a unique directory for each archive, I created directories with a combination of the name of the original email recipient address along with a timestamp.

The structure of the inbound webhook structure may change over time but at the moment the current structure can be found within the Relay Webhooks Example Payloads documentation of SparkPost.

Here are the json values that I grabbed in order to either store them separately or process them further:

1. msys/relay_message/content/to: This is the email address for the original target of the content.
2. msys/relay_message/rcpt_to: This is the email address used to send the archive message.
3. msys/relay_message/friendly_from: This is the from address presented in the original email address.
4. msys/relay_message/content/subject: Subject of the email.
5. msys/relay_message/content/headers: Once pulled, I format these into a nice html (ok, semi-nice) format for storage and reference.
6. msys/relay_message/content/html: The HTML body of the email.
7. msys/relay_message/content/text: The text body of the email.
8. msys/relay_message/content/email_rfc822: The full email body in rfc822 standard. It’s save as an eml file which allows most email clients to open directly.

Once I get all of this data, I process the headers into an html format. From there, I add the To, From and Subject to the HTML and Text bodies and save everything. Finally, I create a log entry for that specific entry.

Send an Archived Copy of the Email

The Archiving functionality can only be initiated by using the SMTP API (which, by the way, is a horribly named industry standard for an added header for your SMTP call). For SparkPost, the SMTP API is actually a JSON formatted header called X-MSYS-API. In that header, you are able to set items like campaign id, metadata, IP Pools, CC and BCC email addresses in addition to a list of Archiving email addresses. Here is a sample X-MSYS-API header:

X-MSYS-API:
{
  "campaign_id": "Jeffs BBQ Campaign",
  "metadata" : {
            "home_address_region": “West”,
            "smoker": "SMB"
  },
  "archive": [
             “[e-mail]”
  ],
"options" : {
            "open_tracking": false,
            "click_tracking": true,
            "transactional": false,
            "ip_pool": "sp_shared",
}
}

This isn’t a complete sample with all of the options available, but illustrates what a typical header looks like. Notice that the original recipient’s email address isn’t in the X-MSYS-API header, but you’ll get it in the Reply Webhook.

In the Github repository, I have added a test PHP application that calls SMTP by using the PHPMailer framework. While calling SMTP and adding in the X-MSYS-API header with an archive email address is probably the best approach for thorough unit testing, it isn’t exactly necessary. Any email sent to the domain you are targeting will get processed the same way as an archive email. So I did most of my unit testing by sending an email to my geekwithapersonality domain, either by a Postman transmission SparkPost API call or simply sending an email via my MAC email client. Neither is really an archive, but the inbox relay doesn’t care and treats all emails the same.

Voila! Archiving Emails is Simple!

Those are the three steps to set up archiving. If you are using SMTP to send emails to SparkPost, you’re probably already using the X-MSYS-API headers, so just adding the archive entry will be easy for you. Then all you need to do is point email for your archiving domain to SparkPost and create a simple app to capture the data.

I hope this was helpful – if you have any questions on archiving emails, email templates, etc, please feel free to reach out to me directly.

Happy Sending!

The post Archiving Emails: A How-To Guide for Tracking Sent Mail appeared first on SparkPost.