Receiving and Parsing Email in Rails 5

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user’s responses to email or action items quicker.

Another aspect of email within a SaaS application is receiving mail. While this is far less normal or used in comparison to sending, it can be a great way to make end user’s responses to email or action items quicker.

---

This is an excerpt from my book Build a SaaS App in Ruby on Rails 5. I have some sample chapters and discounts available for the pre-sale. The book is poised to be finished soon! With that being said, you may see some code blocks in this post that reference a model or object that is pertinent to the book, but does not detract from how you may interface with incoming mail in Rails.

---

In the case of the Standup App we are building, we can have directions to respond to an Email Reminder to create a new standup right from their email response! Some of the tools that we will be using are Mailgun’s email routing service and ngrok, an HTTP tunneling service. ngrok is very useful for a few solution throughout the remainder of this book. It allows you to have a web accessible URL, which tunnels(connects) to ports within your local machine.

Meaning, that you will have a http://somesubdomain.ngrok.com that will forward to your computer, and a specific port specified when you start ngrok. This allows you to test with external services such as Mailgun, Stripe(later), Github(later) and more!

Let’s get started with some setup:

• Download ngrok
• Unzip and move the executable file to where you would like.
• In *nix OS’s, open ngrok with: path/to/ngrok HTTP start 3000. This will be dependent of the port you are using for your Rails server. ngrok will now fire up a tunnel service with a randomly generated URL.
• Optionally, if you upgrade to a paid version of ngrok, you can set a subdomain, so you do not have to change your settings elsewhere every time you restart ngrok.
• In Mailgun go to the Routes tab to create the email route(modeled after MVC routes like Rails) that will send the email. Enter the following settings:
• Chose Match Recipient
• Enter development.standup.*@app.yourdomain.com for the recipient field. Mailgun routes allow wildcard character matching, which will allow you input extra characters in the email's reply-to. Meaning, adding something like a user's hash_id to have identifiable information from the incoming email.
• Check forward and enter http://yoursubdomain.ngrok.io/email_processor as the destination of that forward.
• You can leave the priority alone and give the route a name, before pressing the submit button.

Now with that done, we can begin to modify the application. The application changes will consist of three main parts. First, a new gem(and a companion adapter gem) will be added. griddler is the main library to handle incoming email easily; griddler-mailgun is the Mailgun specific adapter that allows the griddler functionality to work with Mailgun as an incoming mail router. Next, Griddler will need to add a few parts of configuration application-wide to make sure some basic configuration is met. Lastly, an email processor file will be added to handle receiving and parsing the email.

The great part about adding Griddler to your current application is that if you are using the Gemfile from Chapter 3, you already have it installed. If not, just add gem 'griddler' and gem 'griddler-mailgun' to your Gemfile and run a bundle install.

Next to configure and setup Griddler in the application you will need to add a new file in the initializer folder setting a few configuration values. Then, add a quick line to add the default Griddler routing into the routes.rb file.

First the Griddler configuration:

    Griddler.configure do |config|
      config.reply_delimiter = '-- REPLY ABOVE THIS LINE --'
      config.email_service = :mailgun
    end

This will set the text the griddler library will look for in the email and tell it that it will use the installed Mailgun adapter.

Next, add a line to mount the library based routes into your application. Adding the line right above the root to: is fine:

    Rails.application.routes.draw do
      ...

      # mount using default path: /email_processor
      mount_griddler

      root to: 'activity#mine'
    end

By mounting Griddler with that syntax, it will automatically add a route to your application that will route from a specified endpoint to a Griddler based controller:

    email_processor POST /email_processor(.:format)    griddler/emails#create

There are settings in Griddler’s GitHub documentation to change the defaults, but unless you want to get creative with route paths or email processor class names, it can be unnecessary.

The default settings expect a class EmailProcessor to exist and to handle parsing the incoming email with a method process. Griddler, however, does not care where the actual file is placed, but that the class exists and is loaded. Personally, I find that email processing fits most into the services definition and can be placed there.

To allow Griddler and its files to capture text from incoming replies, there are a few changes that will be needed.

First, we will update the EmailReminderMailer to create a unique reply-to address and include that email address as part of the outgoing email:

class EmailReminderMailer < ApplicationMailer
  def reminder_email(user, team)
    @user = user
    @team = team
    reply_to = "'Standup App' <#{'development.' if Rails.env.development?}\
    standup.#{@user.hash_id}@app.yourdomain.com>"
    mail(
      to: @user.email,
      subject: "#{team.name} Standup Reminder!",
      reply_to: reply_to
    )
  end
end

Here we are using building the reply_to string by adding development if the current Rails environment is your local Rails application. This way, the separate Mailgun route made for development can route different from a production route you will add before deploying your application.

Next, we will update the mailer template to have ##- Please type your reply above this line -## and some text letting the email recipient know they can add a standup by replying:

doctype html
html xmlns="http://www.w3.org/1999/xhtml"
  head
    meta content="width=device-width" name="viewport" /
    meta content=("text/html; charset=UTF-8") http-equiv="Content-Type" /
    title= "#{@team.name} Reminder!"
    css:
      | *{margin:0;padding:0;font-family:"Open Sans",Helvetica,Helvetica,Arial,
      sans-serif;box-sizing:border-box;font-size:14px}img{max-width:100%}body{-
      webkit-font-smoothing:antialiased;-webkit-text-size-adjust:none;width:100
      %!important;height:100%;line-height:1.6}table td{vertical-align:top}body{
      background-color:#f6f6f6}.body-wrap{background-color:#f6f6f6;width:100%}.
      container{display:block!important;max-width:800px!important;margin:0
      auto!important;clear:both!important}.content{max-width:800px;margin:0
      auto;display:block;padding:20px}.main{background:#fff;border:1px solid #e
      9e9e9;border-radius:3px}.content-wrap{padding:20px}.content-block{padding
      :0 0 20px}.header{width:100%;margin-bottom:20px}.footer{width:100%;clear:
      both;color:#999;padding:20px}.footer a{color:#999}.footer a,.footer
      p,.footer td,.footer unsubscribe{font-size:12px}h1,h2,h3,a,th,td{font-
      family:"Open Sans",Helvetica,Arial,"Lucida Grande",sans-serif;color:#
      000;margin:40px 0 0;line-height:1.2;font-weight:400}h1{font-size:32px;
      font-weight:500}h2{font-size:24px}h3{font-size:18px}h4{font-size:14px;
      font-weight:600}ol,p,ul{margin-bottom:10px;font-weight:400}ol li,p li,ul
      li{margin-left:5px;list-style-position:inside}a{color:##3c8dbc;text-decor
      ation:underline}.btn-primary{text-decoration:none;color:#
      FFF;background-color:##3c8dbc;border:solid ##3c8dbc;border-width:5px 10px
      ;line-height:2;font-weight:700;text-align:center;cursor:pointer;display:
      inline-block;border-radius:5px;text-transform:capitalize}.last{margin-
      bottom:0}.first{margin-top:0}.aligncenter{text-align:center}.alignright{
      text-align:right}.alignleft{text-align:left}.clear{clear:both}.alert{font
      -size:16px;color:#
      fff;font-weight:500;padding:20px;text-align:center;border-radius:3px 3px
      0 0}.alert a{color:#fff;text-decoration:none;font-weight:500;font-size:16
      px}.alert.alert-warning{background:#f8ac59}.alert.alert-bad{background:#e
      d5565}.alert.alert-good{background:##3c8dbc}.invoice{margin:40px
      auto;text-align:left;width:80%}.invoice td{padding:5px 0}.invoice .
      invoice-items{width:100%}.invoice .invoice-items td{border-top:#eee 1px
      solid}.invoice .invoice-items .total td{border-top:2px solid #
      333;border-bottom:2px solid #333;font-weight:700}@media only screen and (
      max-width:640px){h1,h2,h3,h4{font-weight:600!important;margin:20px 0 5px!
      important}h1{font-size:22px!important}h2{font-size:18px!important}h3{font
      -size:16px!important}.container{width:100% !important}
      .content,.content-wrap{padding:10px !important}.invoice{width:10
      0% !important}
  body
    div style="color: #b5b5b5;text-align:center;"
      | ##- Please type your reply above this line -##
    table.body-wrap style="width:100%"
      tr
        td
        td.container width="800"
          .content
            table.main cellpadding="0" cellspacing="0" width="100%"
              tr
                td.content-wrap
                  table cellpadding="0" cellspacing="0" style="width:100%"
                    tr
                      td.aligncenter
                        | Standup App
                    tr
                      td.content-block
                        h3= "#{@team.name} Reminder!"
                    tr
                      td.content-block
                        = "Just wanted to remind you to add your standup for \
                        the team: #{@team.name}"
                    tr
                      td.content-block.aligncenter
                        = link_to "Add Your Standup", new_standup_url(), \
                        {class:"btn-primary", style: "width:95%"}
                    tr
                      td.content-block
                        = "You can quickly submit your standup by replying to \
                          this email in the format:"
                        pre
                          pre
                          = "[d] This is a done item\n[t] This is a todo item\n\
                          [b] This is a blocker"
            .footer
              table width="100%"
                tr
                  td.aligncenter.content-block
        td

Lastly, we will need to add an extra column to the Standups table to track the Message-ID coming from the Mailgun routed emails. As you can not count on an email service to provide "just once" delivery, we will need to track these unique IDs ourselves on the Standup table.

    rails g migration AddMessageIdToStandups message_id

Next, before the end of the newly created migrations change method, you will want to add add_index :standups, :message_id. This index will allow quick lookups as the Standups table grows. Finally, migrate the actual change:

    bin/rails db:migrate

With those changes out of the way we can now add the new EmailProcessor class that will parse the incoming email:

class EmailProcessor

    attr_reader :email

    def initialize(email)
     @email = email
    end

    TASK_TYPE_HASH = {
        '[d]' => 'Did',
        '[t]' => 'Todo',
        '[b]' => 'Blocker'
      }

    def process
        if Rails.env.development?
          Rails.logger.info '-----------EMAIL-------------'
          Rails.logger.info email.to.first[:token]
          Rails.logger.info email.body
          Rails.logger.info email.headers["Message-ID"]
          Rails.logger.info '-----------EMAIL-------------'
        end

    # Get a user hash_id from reploy-to or bail
        reply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.
        split('.')&.last
        return if reply_user.blank?

    # Find a user by the hash_id or bail
        user = User.find_by(hash_id: reply_user)
        return if user.nil?

    # Bail if standup with incoming message-id exists
        return if Standup.exists?(message_id: email.headers["Message-ID"])

    # Bail if a standup for today exists
        today = Date.today.iso8601
        return if Standup.exists?(standup_date: today)

    # Get content or bail
        tasks_from_body = email.body.scan(/(\[[dtb]{1}\].*)$/)
        return if tasks_from_body.blank? || tasks_from_body.empty?

    build_and_create_standup(
          user: user,
          tasks: tasks_from_body,
          date: today,
          message_id: email.headers["Message-ID"]
        )
      end

    private

    def build_and_create_standup(user:, tasks:, date:, message_id:)
        standup = Standup.new(
          user_id: user.id,
          standup_date: date,
          message_id: message_id
        )

    tasks.each do |task|
          task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
          standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
        end

    standup.save
      end
    end

The class is relatively simple, but let’s go over it section by section:

class EmailProcessor

    attr_reader :email

    def initialize(email)   
     @email = email
    end

    TASK_TYPE_HASH = {
        '[d]' => 'Did',
        '[t]' => 'Todo',
        '[b]' => 'Blocker'
      }

    ...

Here we are initializing the object when the class is called by Griddler, setting the email to a local email variable. Additionally, we are creating a hash to later use in the text content to Task type conversion.

    ...

    def process
        if Rails.env.development?
          Rails.logger.info '-----------EMAIL-------------'
          Rails.logger.info email.to.first[:token]
          Rails.logger.info email.body
          Rails.logger.info email.headers["Message-ID"]
          Rails.logger.info '-----------EMAIL-------------'
        end

    ...

This just adds some additional logging if the mail is processed in the local
development environment.

    ...

    # Get a user hash_id from reply-to or bail
        reply_user = email.to.first[:token]&.split('<')&.last&.split('@')&.first&.
        split('.')&.last
        return if reply_user.blank?

    # Find a user by the hash_id or bail
        user = User.find_by(hash_id: reply_user)
        return if user.nil?

    # Bail if standup with incoming message-id exists
        return if Standup.exists?(message_id: email.headers["Message-ID"])

    # Bail if a standup for today exists
        today = Date.today.iso8601
        return if Standup.exists?(user_id: user.id, standup_date: today)

    # Get content or bail
        safe_body = Rails::Html::WhiteListSanitizer.new.sanitize(email.body)
        tasks_from_body = safe_body.scan(/(\[[dtb]{1}\].*)$/)
        return if tasks_from_body.blank? || tasks_from_body.empty?

    ...

Here we are grabbing some information used in the parsing, as well as giving the process method chances to exit early if the incoming email is not sufficient for processing and Standup creation. In the first section, the incoming email address is parsed to find the user's hash_id. That string is then used to find a User. If there is no user, the method returns without adding a Standup.

The next line will exit the method early if there is already a standup with the current Message-ID. Again, this is making sure to guard against email providers not guaranteeing “just once delivery. That is followed up by generating a variable for the current date and making sure there is no standup with the current user and current time.

Finally, the actual email content is parsed with a regular expression. Regular Expression is a programming language that allows you to pattern match on a string and even capture parts of the pattern matching. This particular pattern(which you can get a more thorough syntax explanation here) searches for lines that begin with [d], [t] or [r]. If those are present, it captures the content to the end of the line. The .scan method on the content's body, allows it to catch all occurrences of the above pattern. If the scan's output is empty, the process method exits.

    build_and_create_standup(
          user: user,
          tasks: tasks_from_body,
          date: today,
          message_id: email.headers["Message-ID"]
        )
      end

    private

    def build_and_create_standup(user:, tasks:, date:, message_id:)
        standup = Standup.new(
          user_id: user.id,
          standup_date: date,
          message_id: message_id
        )

    tasks.each do |task|
          task_type, task_body = task.first.scan(/(\[[dtb]\])(.*)$/).flatten
          standup.tasks << Task.new(type: TASK_TYPE_HASH[task_type], title: task_body)
        end

    standup.save
      end
    end

The last section here is a culmination of all of the stored information so far to be saved into a new Standup. The user, tasks_from_body, today and message_id are all passed into a method that will hand the actual save. The build_and_create_standup method creates a new Standup, with the user's ID, date and message_id. Once the object is created, the tasks strings are iterated over to build the Task with a type and assigned as child objects with the << syntax.
Finally the new Standup object with children Tasks will be saved with standup.save

Lastly, you can test this all works if you reply back to an email(that was sent through Mailgun SMTP and not letter_opener) with the following text:

    [d] Did a thing
    [d] And Another
    [t] Something to do
    [b] Something in the way. Some really long line about something or another

Testing this will require a new spec file with quite a few it blocks to test all the branches the EmailProcessor may encounter.

First, it would be best if we create a factory to be able to quickly generate an email to be used within the EmailProcessor's spec. This way the email can have defaults and then we can use the FactoryGirl .build commands to create a new email object with any different attributes when needed to test the processor.

The factory itself is pretty simple:

FactoryGirl.define do
  factory :email, class: OpenStruct do
    # Assumes Griddler.configure.to is :hash (default)
    to [
      {
        full: '[e-mail]',
        email: '[e-mail]',
        token: 'to_user',
        host: 'email.com',
        name: nil
      }
    ]
    from(
      token: 'from_user',
      host: 'email.com',
      email: '[e-mail]',
      full: 'From User <[e-mail]>',
      name: 'From User'
    )
    subject 'email subject'
    body '[d] Did a thing\n[t] Doing a thing\n[b] Blocked by a thing'
    headers {'Message-ID <[e-mail]>'}
  end
end

Now, with a factory available, the email_processor_spec, will be able to easily spin up new email objects as needed with specific changes to test all of the processors' conditional branches.

require 'rails_helper'

describe EmailProcessor do
  subject(:email_processor) { EmailProcessor }
  let(:user) { FactoryGirl.create(:user) }
  let(:email) do
    FactoryGirl.build(:email,
      to: [
        {
          email: "standup.#{user.hash_id}@app.buildasaasappinrails.com",
          token: "standup.#{user.hash_id}@app.buildasaasappinrails.com"
        }
      ]
    )
  end

  describe 'processes incoming email' do

    it 'works as intended' do
      expect { email_processor.new(email).process }
        .to change(Standup, :count).by(1)
    end

    it 'fails on bad to' do
      bad_to = FactoryGirl.build(
        :email,
        to: [{ token: nil, email: '[e-mail]' }]
      )
      expect { email_processor.new(bad_to).process }
        .to change(Standup, :count).by(0)
    end

    it 'fails on no user' do
      bad_to = FactoryGirl.build(
        :email,
        to: [
          {
            token: '[e-mail]',
            email: '[e-mail]'
          }
        ]
      )
      expect { email_processor.new(bad_to).process }
        .to change(Standup, :count).by(0)
    end

    it 'only saves one per message-id' do
      expect do
        email_processor.new(email).process
        email_processor.new(email).process
      end.to change(Standup, :count).by(1)
    end

    it 'only saves one per date' do
      email2 = FactoryGirl.build(:email, headers: { 'message-id': '123' })
      expect do
        email_processor.new(email).process
        email_processor.new(email2).process
      end.to change(Standup, :count).by(1)
    end

    it 'fails on empty or bad body' do
      email = FactoryGirl.build(:email, body: '90ioqwhdk.qhdu')
      email2 = FactoryGirl.build(:email, body: '')
      expect do
        email_processor.new(email).process
        email_processor.new(email2).process
      end.to change(Standup, :count).by(0)
    end
  end
end

While long and containing six examples, this spec is actually pretty straightforward. It is first testing the happy path where everything is set up and working. Then tests each failing path that doesn’t create a standup, in order as those paths appear in the EmailProcessor's .process method.

A quick run of the whole rspec suite should show no failing tests and nearly perfect test/code coverage:

rspec spec ........................................................................................................................................................

    Finished in 25.78 seconds (files took 10.16 seconds to load) 152 examples, 0 failures Coverage report generated for RSpec to standup_app/coverage. 428 / 431 LOC (99.3%) covered.