DEV Community: Opeyemi Obembe

IMAP: new messages since last check

Opeyemi Obembe — Thu, 02 May 2019 06:52:19 +0000

I have been working on SharedBox in the last couple of months. The idea is simple: allow teams send and receive emails in Slack. I had to allow email connection via IMAP and doing this means it is important to be able to get only new emails at every check.

Message IDs and other stories

Let’s start with some basic understanding of the message ids—sequence number and uid. It’s going to be important. These numbers are how messages are identified¹ in a mailbox. (Note that a mailbox refers to a message folder—inbox, sent, draft…and not the full email box).

The message sequence number is a sequential numbering of messages in a mailbox. This means that the first message in the mailbox gets 1, the next gets 2, and so on. If a message is deleted, the number is reassigned. For example, given a mailbox with 10 messages, if message 7 is deleted, message with sequence number 8 now becomes 7, 9 becomes 8 and 10 becomes the new 9. So this [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] becomes this [1, 2, 3, 4, 5, 6, 7, 8, 9]. What this says is that the number is not unique and can always change.

UID is similar to sequence number but “more” unique. It is not affected by message deletes. In the example above, even though message with sequence number 8 is now 7, its UID will remain 8. The new array of message UID will then be [1, 2, 3, 4, 5, 6, 8, 9, 10]. Does this mean UIDs never change? No. Things may happen on the server that will reset message UIDs. Like if the mailbox is recreated. The good news though is that when UIDs change, there is a way to know. Enter UIDVALIDITY.

UIDVALIDITY is an additional value returned when you select a mailbox. This value should be the same for every time the message UIDs are unique. If at any mailbox selection (session) the value is not the same as what it was in the last session, then know that your message UIDs have changed. It is therefore important you store this value at every session so you are able to compare at next check to know if the message UIDs have changed.

So where do sequence numbers and uids come in when pulling messages? We use them to specify what message or range of messages to pull. To pull a range of messages, we need to specify a sequence set of the message id (uid or sequence number). This can take the following formats:

4 (only the message with id 4)
2,4,9 (messages 2, 4 and 9)
4:7 (messages 4 to 7, i.e 4, 5, 6, 7)
12:* (messages 12 to the last message in the mailbox. If the last message in the mailbox is 14, that is 12, 13, 14. If the last message in the box is however less than 12, say 10 for example, then it is that number to 12, i.e 10, 11, 12)
2,4:7,9,12:* (a combination of the formats above)

Now that we have a basic idea of what message ids are, it’s easy to start pulling emails. I will show some examples using the Javascript library emailjs-imap-client. Then we will get to the difficult part, pulling new messages (after last check).

(The library can be used in browser but I will be using it in Node.js instead. So as the first step, install: npm install —save emailjs-imap-client)

Let’s start with connection to the IMAP server and box selection. And see what the response of the select action looks like.

(async function (){
  // Connect to the imap server
  const imap = new ImapClient.default('imap.mail.yahoo.com', 993, {
          auth: {
            user: 'awesomeme@yahoo.com',
            pass: 'ninjaninja'
          }
      });
  await imap.connect();

  // Select the "mailbox" you want to "interact" with
  const box = await imap.selectMailbox('INBOX');
  console.log(box);
})()

This should give you a response like this:

{
  "readOnly": false,
  "exists": 1,
  "flags": [
    "\\Answered",
    "\\Flagged",
    "\\Draft",
    "\\Deleted",
    "\\Seen",
    "$NotPhishing",
    "$Phishing"
  ],
  "permanentFlags": [
    "\\Answered",
    "\\Flagged",
    "\\Draft",
    "\\Deleted",
    "\\Seen",
    "$NotPhishing",
    "$Phishing",
    "\\*"
  ],
  "uidValidity": 1,
  "uidNext": 686,
  "highestModseq": "108661"
}

Notice the uidValidity and uidNext fields. Also note highestModseq. We will get to it. Another parameter you may be interested in is exists. It returns the number of emails currently available in the mailbox. Even though the mailbox might have received a lot of emails, only one is currently left in the mailbox.

Let’s extend our example to pull message with sequence number 1:

(async function (){
  // ...
  const messages = await imap.listMessages('INBOX', '1', ['body[]']);
})()

We can also pull message with UID 686:

(async function (){
  // ...
  const messages = await imap.listMessages('INBOX', '686', ['body[]'], {byUid: true});
})()

Pulling all emails from the mailbox is easy. All you need to do is specify a message sequence of 1:*. (This may be a bad idea as the number of messages in the mailbox may choke your application. But you can always split the process 1:500, 500:1000 and so on). The tricky part comes when you want to only pull new emails (mails after your last pull) from the server. And if you think one way syncs are tricky, wait till you attempt two-way syncs.

HighestModseq and ChangedSince

highestModseq returned when the mailbox is selected as you’ve seen above is the highest sequence number value of all messages in the mailbox. Once you select a mailbox and this number is greater than at last check, you can assume that there has been changes to the mailbox. You can then use the last value you have to pull all new messages.

Let’s assume the first time we checked the user’s mailbox, highestModseq was 100. The next time, it’s 120. This tells us there has been changes to the mailbox. We can then fetch new messages from when our highestModseq was 100.

(async function (){
  // ...
  const messages = await imap.listMessages('INBOX', '1:*', ['body[]'], {changedSince: '100'});
})()

This is easy and works. There is just one problem though. Not all servers support highestModseq.

\Recent? \Seen?

There is a recent flag that can be used to get “recent” messages from the server. The issue with this though is that the definition of “recent” by the server is relative. Here is what I mean:

You disconnected from the server at 9:00pm
2 new messages come in at 9:02pm. The server marks these messages with the recent flag.
You connect again at 9:05pm to check for new emails using the recent flag and you get the 2 new messages.
You disconnect shortly after and the server removes the recent flag on the messages
A new message comes in 9:07pm and is marked recent
Another mail client that is not you, connects to the server to pull mails
The recent flag is removed from the message
You connect 9:10pm using the remove flag. You get zero messages even though there has been a new message since you last checked.

The seen flag is similar but also goes through the same fate. If another client opens the message, the flag is removed. Trying to get “unseen” messages after another client has “seen” them will return nothing.

Search Since

We can combine IMAP’s search function with a since parameter to get new messages since our last check. And this would have been a great solution—store the last time we checked and use that to get new messages since then. But there is a limitation to IMAP. The since parameter only takes date and not time.

uidValidity + uidNext

Can we use the knowledge of what the next UID will be (taking into consideration if uidValidity has changed or not) to do this? Absolutely. If at first pull, uidValidity is 1 and uidNext is 686 then we can pull new messages since last pull with the sequence set: 686:* if uidValidity is still 1.

(async function (){
  // ...
  const messages = await imap.listMessages('INBOX', '686:*', ['body[]'], {byUid: true});
})()

What if uidValidity has changed? Then we can assume that there has been a major change to the mailbox—it has been recreated or so. We just need to assume we are starting our sync again—we store the new uidValidity and the use the new uidNext as our sequence set.

https://tools.ietf.org/html/rfc3501#section-2.3.1 ↩

Exploring APIs for likes and bookmarks

Opeyemi Obembe — Sun, 05 Aug 2018 07:01:34 +0000

I built a service that reminds people of things they’ve liked or bookmarked (tweets, repos, articles, pictures, etc). Here are notes exploring some APIs for these data.

Twitter (likes)

First love. I’ve built many things on Twitter’s API. I’ve also been burnt many times, but then, what’s a bookmark reminder tool without Twitter likes?
API still on OAuth 1.0a. All other services I reviewed are on OAuth 2.0.
By the way, Twitter now has bookmarks but I doubt people really use it in place of “Like”. One idea is to also allow people connect their Twitter bookmarks to Waytab. Unfortunately, Twitter doesn’t have an API for it as at writing this. (And I doubt they will).

Unsplash (likes)

Photo resources come with color value that represents the dominant colour of the photo. This is useful in many ways. An example is using this as background colour while loading photo.
App needs to be submitted for review. Process is smooth if you adhere to their use policy though.

Pocket (unread articles)

Unlike other OAuth 2.0 APIs, requests requires your consumer_key alongside the access_token.
What’s also weird is that the HTTP method for getting [unread] articles is POST.

Github (starred repos)

Only issue I have with Github is that only one authorisation callback URL is allowed. This means to test locally, I have to create a separate app that can have a localhost callback URL.

Pinterest (user pins)

All authorisation callback URL must be on SSL including localhost. Easier thing to do is to use an app like ngrok to tunnel to your localhost.
App needs to be submitted for review.

The services that didn’t make it:

Pinboard (bookmarks)

I wanted to include at least one bookmark service. Pinboard was the one that came to mind (what are the others people use these days?). Unfortunately, the API is non OAuth so I dropped it. I can’t deal with handling user passwords.

Youtube (watch later)

Youtube deprecated their watch later API. As that’s the most interesting list for me, I dropped adding the service. I may add “liked videos” later but only if I get considerable request for it. Another list that may be interesting is unwatched videos from subscribed channels. I am yet to find a way to pull that though.

Dribbble (likes)

Dribble is not granting access to their like API at the moment :/

Spotify (likes)

I will have to create a “commercial app” on Spotify. (“if… your app or product charges a fee to purchase, promotes using ads or utilizes a freemium model for upsells; you are legally protected as a corporate entity or company”). As there was no clear cost implication, I dropped it.

Instagram (likes)

Instagram’s API is so messed up at the moment. It’s a big no-no for now. Plus nothing in the Facebook documentation talks about likes.

Sending bulk emails with Mailgun lists

Opeyemi Obembe — Thu, 26 Apr 2018 18:37:48 +0000

Beyond transactional emails, sometimes we need to send generic emails to all users or a segment of users. The way this is mostly done is to use email marketing services like MailChimp. For something custom, one option is to send to the emails one after the other using a loop, or a queue. But an even easier way to go is to use Mailgun Mailing Lists.

Mailgun Mailing Lists allow you create an email address you can subscribe members to. Any email sent to the address will then be sent to each subscribed member.

Creating the Mailing List

Let’s start by creating our example mailing list. We can do this from the dashboard or Mailgun API.

Notes:

It is important to leave the access level at read only. This ensures that only you can send mail to the list through the API.

Your domain’s MX records must point to Mailgun so that Mailgun can receive and process emails sent to the alias address (the Mailing List email address). If you are using another email provider for your emails, you can create a separate subdomain and use that for your Mailing List.

Adding Members

Now that we have our Mailing List, we can start adding members.

Endpoint: https://api.mailgun.net/v3/lists/{address}/members
Method: POST
Authentication: HTTP Basic (api as username, API Key as password)
Parameters:

address: email address, e.g. Alice <alice@example.com> or alice@example.com
name: Member name. Optional.
vars: JSON-encoded dictionary string with arbitrary parameters, e.g. {"gender":"female","age":27}
subscribed: yes to add as subscribed (default), no as unsubscribed
upsert: yes to update member if present, no to raise error in case of a duplicate member (default)

You will want to automatically do this from your app when the necessary event is triggered. If for example your list is targeted at your paid subscribers, you will want to call the Add Member API once a payment event is triggered on your site.



stripe.charges.create({
  amount: charge * 100,
  currency: 'usd',
  customer: user.customer.id
}, (err, charge) => {
  // Error and charge checks
  // ...

  // Subscribe user to list
  request.post({
    url: 'https://api.mailgun.net/v3/lists/customers@lists.suet.co/members',
    gzip: true,
    auth: {
      user: 'api',
      pass: 'secretpassword',
      sendImmediately: false
    },
    form: {
      address: user.email
    }
  }, (err, response, body) => {
    // Checks
    // ...
  });
})

You can also add existing users to the list using the bulk upload option under Mailing List in the Mailgun Dashboard.

Or use the bulk insert API.

Endpoint: https://api.mailgun.net/v3/lists/{address}/members.json
Method: POST
Authentication: HTTP Basic (api as username, API Key as password)
Parameters:

members: JSON-encoded array. Elements can be either addresses, . ["bob@example.com", "alice@example.com"], or JSON objects, e.g. [{“address": "bob@example.com", "name": "Bob", "subscribed": false}, {"address": "alice@example.com", "name": "Alice"}]. Custom variables can also be provided, e.g. [{"address": "Bruce <bruce@jl.dc>", "vars": {"superpower": "money"}},{"name": "Barry", "address": "barry@jl.dc", "vars": {"superpower": "speed"}}]. 1000 emails per call.
upsert: yes to update member if present, no to raise error in case of a duplicate member (default)



// ...
// users => array of retrieved users from db

let chunk = [];
while(users.length) {
  chunk.push(users.splice(0, 1000));
}

Promise.all(chunk.map(d => {
  return new Promise((resolve, reject) => {
    request.post({
      url: 'https://api.mailgun.net/v3/lists/customers@lists.suet.co/members.json',
      gzip: true,
      auth: {
        user: 'api',
        pass: 'secretpassword',
        sendImmediately: false
      },
      form: {
        members: JSON.stringify(d)
      }
    }, (err, response, body) => { 
      // ...
      return resolve(); 
      // Or reject() depending or response
    })
  })
}))
.then(() => {
  // ...
}).catch(reason => {
  // ...
})

Removing Members

There are two ways to remove people from our list. One is to delete them. This completely removes them from the list. The other is to unsubscribe them. They will still remain in the list but emails sent to the list will not be sent to them. Let’s take a look at both options. We will be completely deleting them when they no longer fit our segment (in this case, when they are no longer a paid user) and will let them unsubscribe if they are no longer interested in the mails we send.

Let’s start with deleting the user.



stripe.customers.del(user.customer.id, (err, status) => {
  // Error and checks
  // ...

  // Remove user to list
  request.delete({
    url: `https://api.mailgun.net/v3/lists/customers@lists.suet.co/members/${user.email}`,
    gzip: true,
    auth: {
      user: 'api',
      pass: 'secretpassword',
      sendImmediately: false
    }
  }, (err, response, body) => {
    // Checks
    // ...
  });
})

For canceling subscriptions, we can include the unsubscribe variable %mailing_list_unsubscribe_url% in our mail. This will be replaced with an unsubscribe link by Mailgun and Mailgun will handle the rest. We can as well create a custom unsubscribe page we can link to from our mail. Within our email, our unsubscribe link will look like this http://domain.tld/unsubscribe/%recipient_email%. The %recipient_email% variable will automatically be replaced with the user’s email.



// An express route
app.get('/unsubscribe/:email', (req, res) => {
  let email = req.params.email;

  request.put({
    url: `https://api.mailgun.net/v3/lists/customers@lists.suet.co/members/${email}`,
    gzip: true,
    auth: {
      user: 'api',
      pass: 'secretpassword',
      sendImmediately: false
    },
    form: {
      subscribed: 'no'
    }
  }, (err, response, body) => {
    // Checks
    // ...

    res.send('You have been successfully unsubscribed :(');
  });
})

Sending the mail

Because our list’s access level is read-only, the only way to post to it is through the API. Let’s create a test mail. It can be a simple text email.



Hi, Opeyemi here.

I am happy to announce a new feature - Quick Mail.

Quick Mail allows you to quickly send emails to any of your users. This is really helpful when you need to do a quick follow up. Simply provide a name and email to send as (it is recommended you use the same email and name you send with within your app), subject, body and send!

I look forward to feedback from you.
Thanks.

%mailing_list_unsubscribe_url%

Or a simple HTML template.

Source here



request.post({

  url: 'https://api.mailgun.net/v3/lists.suet.co/messages',

  auth: {

    user: 'api',

    pass: process.env.EMAIL_KEY

  },

  sendImmediately: false,

  form: {

    from: process.env.EMAIL_FROM,

    subject: 'New feature: Quick Mail',

    html: html, // Content of our html template

    to: 'customers@lists.suet.co', // Our list address

    text: text // Text version of our email

  }

}, (err, response, body) => {

  // Status and checks

});

Tracking deliveries, opens and clicks

You can easily track email deliveries, opens and clicks from the Mailgun dashboard. If you use Suet, this will also be available on your Suet dashboard.

Conclusion

Using a mailing list makes it easy to send bulk emails to your users. The interesting part is you can create multiple lists and segment your users into them.

Testing [Nodejs] Lambda functions

Opeyemi Obembe — Mon, 26 Mar 2018 11:38:11 +0000

Testing Lambda functions can be tricky. A typical cycle is to assume everything is fine, push up, get stuck, debug and make corrections locally, then push up again. A better approach would actually be to test locally before pushing up.

Let’s consider a typical Nodejs Lambda function

exports.handler = (event, context, callback) => {
  // Do stuff here
  return callback();
}

To be able to test this locally, we need to be able to pass in the required event, context and callback parameters. Interestingly, by looking into the event and context objects and of course knowing callback is just a function passed in to return information, we can create similar objects and use them for our local test.

Let’s start with event. This is used to pass event data to the function. The data will depend on the event source connected to your Lambda function. You can find a list of sample events published by event sources here. Here is the sample event data for SNS for example:

{
  "Records": [
    {
      "EventVersion": "1.0",
      "EventSubscriptionArn": eventsubscriptionarn,
      "EventSource": "aws:sns",
      "Sns": {
        "SignatureVersion": "1",
        "Timestamp": "1970-01-01T00:00:00.000Z",
        "Signature": "EXAMPLE",
        "SigningCertUrl": "EXAMPLE",
        "MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
        "Message": "Hello from SNS!",
        "MessageAttributes": {
          "Test": {
            "Type": "String",
            "Value": "TestString"
          },
          "TestBinary": {
            "Type": "Binary",
            "Value": "TestBinary"
          }
        },
        "Type": "Notification",
        "UnsubscribeUrl": "EXAMPLE",
        "TopicArn": topicarn,
        "Subject": "TestInvoke"
      }
    }
  ]
}

For me though, (I use this on TFP to send data across the workers), two things:

The only value I’m interested in from the event data is event.Records[0].Sns.Message
Message is a stringified JSON object.

Based on these, I can create my event object as this:

var event = {
  Records: [{
    Sns: {
      Message: JSON.stringify(testdata)
    }
  }]
}

Next is the context parameter. We can see all the keys of the context object in this document. Let’s create a similar object.

var context = {
  awsRequestId: Math.random().toString(36).replace(/[^a-z]+/g, '').substr(0, 5), // random string per request
  callbackWaitsForEmptyEventLoop: true,
  getRemainingTimeInMillis: function(){ return 0 },
  functionName: '',
  functionVersion: '',
  memoryLimitInMB: '',
  logGroupName: '',
  logStreamName: '',
  clientContext: null,
  identity: null
}

callback is the easy one.

var callback = function(err, result) {
  if (err)
    console.log(err);
  if (result)
    console.log(result);
  // Terminate execution once done
  process.exit(0);
}

Now that we have the 3 required parameters. We can put everything together.

require('dotenv').config(); // Environmental variables
var lambda = require('./path/to/lambda/function/index.js')
;

// Event
var eventdata = {
    url: 'https://api.twitter.com/1.1/statuses/home_timeline.json'
    , qs: {
      tweet_mode: 'extended'
    }
    , user: '5ab7d745174f534889991a30'
    , oauth: {
      consumer_key: process.env['TWTR_CK']
      , consumer_secret: process.env['TWTR_CS']
      , token: process.env['TWTR_TOKEN']
      , token_secret: process.env['TWTR_SECRET']
    }
  }
var event = {
  Records: [{
    Sns: {
      Message: JSON.stringify(eventdata)
    }
  }]
}
// Context 
var context = {
  awsRequestId: Math.random().toString(36).replace(/[^a-z]+/g, '').substr(0, 5),
  callbackWaitsForEmptyEventLoop: true,
  getRemainingTimeInMillis: function(){},
  functionName: '',
  functionVersion: '',
  memoryLimitInMB: '',
  logGroupName: '',
  logStreamName: '',
  clientContext: null,
  identity: null
}
// Callback
var callback = function(err, result) {
  if (err)
    console.log(err);
  if (result)
    console.log(result);
  // Terminate execution once done
  process.exit(0);
}

// Run
lambda.handler(event, context, callback);

And there we have it. If there is any issue with the function, we can easily know and debug. One tip during test is to do lots of console logging within your function to note execution points and see the data that is passed around.

Formatting tweets: a look at Extended tweets, Retweets and Quotes

Opeyemi Obembe — Sat, 17 Mar 2018 04:06:53 +0000

One thing I’ve found out with thefeed.press is that the conversations (the tweets) surrounding shared links are sometimes more interesting than the link. To place proper emphasis on these tweets mean displaying them wherever necessary; the email digest for example. And displaying them mean formatting them properly.

Introduction

To display a tweet properly, it needs to be well formatted. This means identifying and linking entities like usernames, hashtags and URLs. In simple terms, it is converting a typical tweet object¹ like this:

{
  "created_at": "Mon Mar 05 21:16:46 +0000 2018",
  "id": 970770116043595800,
  "text": "Wish I have some time to curate #WeAreNigerianCreatives. Someone please do.",
  "entities": {
    "hashtags": [{
        "text": "WeAreNigerianCreatives",
        "indices": [32, 55]
      }
    ],
    "symbols": [],
    "user_mentions": [],
    "urls": []
  }
}

to this:

Wish I have some time to curate #WeAreNigerianCreatives. Someone please do.

Notice that the tweet object’s text is plain unformatted text but there is an additional entities object with necessary details for formatting. You probably won’t need to write a library to match and replace the entities in the text though. Twitter provides Twitter Text, an amazing library to do this.

This is a representation in Node.js.

// twitter-text already installed with 
// `npm install twitter-text`
// ...
const twitter = require('twitter-text')
    , tweet = {
        "created_at": "Mon Mar 05 21:16:46 +0000 2018",
        "id": 970770116043595800,
        "text": "Wish I have some time to curate #WeAreNigerianCreatives. Someone please do.",
        "entities": {
          "hashtags": [{
              "text": "WeAreNigerianCreatives",
              "indices": [32, 55]
            }
          ],
          "symbols": [],
          "user_mentions": [],
          "urls": []
        }
      }
    ;

console.log(twitter.autoLinkWithJSON(tweet.text, tweet.entities);

Say hello to extended tweets

For tweets over 140 characters, the tweet object only returns 140 characters of text by default. In this compatibility mode,

text is truncated to 140 characters
truncated is set to true for tweets that are more than 140 characters
entities only include those in the available 140 text range

Here is an example tweet object

{
  "created_at": "Sat Mar 10 18:12:17 +0000 2018",
  "id": 972535628742078500,
  "text": "I kind of hate how with most web development/new frameworks etc., I start out with the intention “I’d like to spend… https://t.co/A10WmSzVeL",
  "truncated": true,
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [{
        "url": "https://t.co/A10WmSzVeL",
        "expanded_url": "https://twitter.com/i/web/status/972535628742078469",
        "display_url": "twitter.com/i/web/status/9…",
        "indices": [
          117,
          140
        ]
      }
    ]
  }
}

Formatting that will give this:

I kind of hate how with most web development/new frameworks etc., I start out with the intention “I’d like to spend… https://twitter.com/i/web/status/972535628742078469 …

compared to the original tweet:

I kind of hate how with most web development/new frameworks etc., I start out with the intention “I’d like to spend 20 minutes learning X today,” and have to invest an additional 60 minutes just setting up the appropriate environment.

Mode: Extended

How to get the full text? Simple. Add the parameter tweet_mode=extended to any endpoint you are querying. So instead of https://api.twitter.com/1.1/statuses/show/972535628742078469.json, let’s try https://api.twitter.com/1.1/statuses/show/972535628742078469.json?tweet_mode=extended

{
  "created_at": "Sat Mar 10 18:12:17 +0000 2018",
  "id": 972535628742078500,
  "full_text": "I kind of hate how with most web development/new frameworks etc., I start out with the intention “I’d like to spend 20 minutes learning X today,” and have to invest an additional 60 minutes just setting up the appropriate environment.",
  "truncated": false,
  "display_text_range": [0, 234],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": []
  }
}

Yeah, that simple. Notice that:

full_text replaces text
truncated is false
display_text_range identifies the start and end of the displayable content of the tweet.

You can then go ahead and format using full_text and entities.

const twitter = require('twitter-text')
    , tweet = {
        "created_at": "Sat Mar 10 18:12:17 +0000 2018",
        "id": 972535628742078500,
        "full_text": "I kind of hate how with most web development/new frameworks etc., I start out with the intention “I’d like to spend 20 minutes learning X today,” and have to invest an additional 60 minutes just setting up the appropriate environment.",
        "truncated": false,
        "display_text_range": [0, 234],
        "entities": {
          "hashtags": [],
          "symbols": [],
          "user_mentions": [],
          "urls": []
        }
      }
    ;

console.log(twitter.autoLinkWithJSON(tweet.full_text, tweet.entities);

Hmmm…retweets

Here is a retweet requested in extended mode.

{
  "created_at": "Sun Mar 11 12:00:27 +0000 2018",
  "id": 972804442667003900,
  "full_text": "RT @jasongorman: As a physics grad, I understand how snooker works at a level I imagine a lot of pro snooker players don't. But I suck at s…",
  "truncated": false,
  "display_text_range": [
    0,
    140
  ],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [
      {
        "screen_name": "jasongorman",
        "name": "jasongorman",
        "id": 18771008,
        "id_str": "18771008",
        "indices": [
          3,
          15
        ]
      }
    ],
    "urls": []
  },
  "retweeted_status": {...}
}

Notice how full_text is truncated even though truncated says false. What could be wrong? Well, texts in retweets are prefixed with RT @username: and if the resulting text is more than 140 characters, it will be truncated.

What to do? Use the retweeted_status instead. The retweeted_status object contains the full text and entities you need.

{
  "created_at": "Sun Mar 11 12:00:27 +0000 2018",
  "id": 972804442667003900,
  "full_text": "RT @jasongorman: As a physics grad, I understand how snooker works at a level I imagine a lot of pro snooker players don't. But I suck at s…",
  "truncated": false,
  "display_text_range": [...],
  "entities": {...},
  "retweeted_status": {
    "created_at": "Sun Mar 11 08:10:46 +0000 2018",
    "id": 972746641957642200,
    "full_text": "As a physics grad, I understand how snooker works at a level I imagine a lot of pro snooker players don't. But I suck at snooker. Understanding != ability.",
    "truncated": false,
    "display_text_range": [0, 155],
    "entities": {
      "hashtags": [],
      "symbols": [],
      "user_mentions": [],
      "urls": []
    },
  }
}

Just check if retweeted_status exist and use that instead.

// Get tweet
// ...
if (tweet.retweeted_status)
  tweet = tweet.retweeted_status;

formatted = twitter.autoLinkWithJSON(tweet.full_text, tweet.entities);

Quotes :/

Quotes are in an entirely different world of their own. You need to see what a quoted tweet looks like to understand.

{
  "created_at": "Sat Dec 16 04:04:36 +0000 2017",
  "id": 941881722685284400,
  "full_text": "Added tweets to the daily newsletter for better context. https://t.co/Q46O3husnz",
  "truncated": false,
  "display_text_range": [0, 56],
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [{
        "url": "https://t.co/Q46O3husnz",
        "expanded_url": "https://twitter.com/thefeedpress/status/941880801087680512",
        "display_url": "twitter.com/thefeedpress/s…",
        "indices": [57, 80
        ]
      }]
  },
  "quoted_status": {...}
}

The full_text does not tell the complete story. It does not include the tweet that was quoted. The quoted tweet is hidden somewhere in quoted_status. And unlike retweets where you can replace the tweet with the retweeted status, you need both the original and additional tweet to make complete sense of a quote. Here is what quoted_status looks like:

{
  "created_at": "Sat Dec 16 04:00:56 +0000 2017",
  "id": 941880801087680500,
  "full_text": "New newsletter screenshot https://t.co/HQmJumZfhN",
  "truncated": false,
  "display_text_range": [0, 25],
  "entities": {...},
  "extended_entities": {...}
}

So what do we do in this case? What we need to achieve is something like this:

Added tweets to the daily newsletter for better context

@thefeedpress:

New newsletter screenshot pic.twitter.com/HQmJumZfhN

And it seems we just need to format the quoted tweet and additional tweet separately and show them together.

const twitter = require('twitter-text')
    ;

// Get tweet
// ..
let text = twitter.autoLinkWithJSON(tweet.full_text, tweet.entities);
if (tweet.quoted_status) {
  let qt = twitter.autoLinkWithJSON(tweet.quoted_status.full_text, 
            tweet.quoted_status.entities);
  text += `<blockquote><a href="https://twitter.com/${tweet.quoted_status.user.screen_name}">@${tweet.quoted_status.user.screen_name}</a>:<br>
            ${qt}
          </blockquote>`;
}

console.log(text);

Added tweets to the daily newsletter for better context. https://twitter.com/thefeedpress/status/941880801087680512 …

@thefeedpress:

New newsletter screenshot pic.twitter.com/HQmJumZfhN

Looks pretty close. But the additional tweet has a link to the embedded quote. Can we remove this link though? Let’s try.

Since we know the link to the quoted status will always end the additional tweet text, we can match end of text for link with format https://twitter.com/[quoted_status_user_username]/status/[0-9]+ and remove. There are a couple of issues with this though. If we match the unformatted text, the url will still be in the format http://t.co/\w+ (unexpanded) and not https://twitter.com/[quoted_status_user_username]/status/[0-9]+ (expanded). If we match after formatting, the link would have been expanded but will contain HTML tags that will break our regular expression².

Well, since we know the link will always end the text, we can remove any ending link in the unformatted text. We can also remove the index from the entities before we then proceed to format the text.

if (tweet.retweeted_status)
  tweet = tweet.retweeted_status;

if (tweet.quoted_status) {
  if (tweet.entities && tweet.entities.urls) {
    let re = new RegExp('https://twitter.com/\\w+/status/'+tweet.quoted_status.id_str);
    tweet.entities.urls = tweet.entities.urls.filter(url => !re.test(url.expanded_url));
  }
  text = twitter.autoLinkWithJSON(tweet.full_text, tweet.entities);
  let qt = twitter.autoLinkWithJSON(tweet.quoted_status.full_text, tweet.quoted_status.entities);
  text = text.replace(/https:\/\/t.co\/[^\/]+$/, '');
  text += `<blockquote><a href="https://twitter.com/${tweet.quoted_status.user.screen_name}">@${tweet.quoted_status.user.screen_name}</a><br>${qt}</blockquote>`;
}
else
    text = twitter.autoLinkWithJSON(tweet.full_text, tweet.entities);

Conclusion

This is all you will probably need. But there is still more to do. What about displaying media (pictures, videos) within the tweet? Quotes within quotes? Threaded replies?

If you really want to do it, formatting tweets can be a complex thing. But you really don’t have to do it if not necessary. You can use embedded tweets instead.

Some items are removed from the tweet object as well as others used in this piece for brevity purpose. ↩
Here is what the formatted HTML for the link https://twitter.com/thefeedpress/status/941880801087680512 looks like <a href="https://t.co/Q46O3husnz" title="https://twitter.com/thefeedpress/status/941880801087680512" rel="nofollow"> https://twitter.com/thefeedpress/status/941880801087680512 …</a> ↩

Working with Mailgun webhooks

Opeyemi Obembe — Tue, 08 Aug 2017 06:43:05 +0000

TL;DR
Mailgun webhooks allow you receive event data for your transactional emails and they are easy to work with. You can use the data to build custom analytics and enrich your applications. For a real world example, check out Suet on Github.

Transactional emails are essentials for most apps. We send welcome emails, password recovery emails, notifications and more. And when we do, we use providers like Mailgun. Sending the emails is cool and all but what about the delivery and performance? Did that user get that password reset email? Was that â€˜credit card expiring’ notification email opened?

Although this data is available in your Mailgun account dashboard, another way you can also get updates about what’s happening to your transactional mails in Mailgun is through webhooks. There is also the API but unlike with the API where you “request” for these updates (Poll), with webhooks, the updates are sent to you (Push). All you need to do is provide the URL of a script that can handle the event data through POST .

Needless to mention, push has some advantage over poll.

You don’t have to make repeated API requests. This means consuming lesser server resources.
Updates are more realtime because it is pushed as soon as available on the server.

Setting it up

There are two ways to set up webhooks in Mailgun. It can either be done through the Mailgun dashboard or API. The more straightforward way to do it is through the dashboard. Once logged into your dashboard, a Webhooks link is available on the navigation bar.

The webhooks page lists the different event types you can receive event data for. By clicking the + button in front of each event, you can set the URL the event data will be sent to.

Handling the data

To handle the event data sent to our webhook URL, we must know what the data look like in the first place. Parameters sent via POST is available in the API documentation. We can go a step further and confirm this by using a test webhook URL that will log the data from Mailgun. We can use Mailgun’s Postbin or requestb.in. These services will generate a unique endpoint we can use in the Mailgun dashboard to get sample event data. I recommend requestbin because it provides more details like the request headers. These headers are important because you will easily miss the fact that Mailgun sends some data using the content-type application/x-www-form-urlencoded and some as multipart/form-data. Missing this little detail changes everything about how you get the event data.

Let’s go ahead to create a test endpoint and see what the event data look like compared to what is in the documentation.

Visit requestb.in and create a bin.
Copy the URL and access the Webhooks section of your Mailgun dashboard.
Paste the URL in the input field and click the Test Webhook link. This will send sample event data to the URL.
Repeat this for all the events you are interested in.
Refresh your requestbin page to see the event data sent.

If you look closely at the requestbin data, you will notice what I said about some data being sent as multipart/form-data.

Now that we know what the parameters are for each event types and the content type they may come in, it is easy to write the code that will handle the sent data. Here is a simple code that will output details of complaints and dropped emails. (I am using multer to handle multipart/form-data)

const express = require('express')
    , bodyParser = require('body-parser')
    , multer = require('multer')
    ;

const app = express();
app.use(bodyParser.urlencoded({extended: false}));
app.listen(process.env.PORT || 3000);

app.post('/webhook', multer().none(), function(req, res) {
  const email = req.body.recipient;
  const event = req.body.event;

  if (event == 'complained') {
    console.log(`${email} complained about your mail`);
  }
  else if (event == 'dropped') {
    console.log(`Mail to ${email} dropped. ${event.description}`);
  }
  else if (event == 'bounced') {
    console.log(`Error ${event.code}: Mail to ${email} bounced. ${event.error}`);
  }

  res.end();
});

Making it secure

Nothing stops anyone that knows our webhook URL from crafting a false event data and sending it to the URL. Luckily, Mailgun signs each request sent and posts the following parameters along:

timestamp (Number of seconds passed since January 1, 1970)
token (Randomly generated string with length 50)
signature (Hexadecimal string generated by HMAC algorithm)

To verify the token;

Concatenate the values of timestamp and token.
Encode the resulting string with HMAC, using your Mailgun API key as the key and Sha256 as the algorithm.
The result should be the same as the signature.

Here is what it looks like in Node.js:

const value = event_data_timestamp+event_data_token;
const hash = crypto.createHmac('sha256', apikey)
                   .update(value)
                   .digest('hex');
if (hash !== event_data_signature) {
  console.log('Invalid signature');
  return;
}

If we add that to our original code example, we will have something like this:

const express = require('express')
    , crypto = require('crypto')
    , multer = require('multer')
    , bodyParser = require('body-parser')
    ;

const app = express();
app.use(bodyParser.urlencoded({extended: false}));
app.listen(process.env.PORT || 3000);

app.get('/webhook', multer().none(), function(req, res) {
  // Validate signature
  const value = req.body.timestamp+req.body.token;
  const hash = crypto.createHmac('sha256', 
                            process.env.API_KEY)
                   .update(value)
                   .digest('hex');
  if (hash !== req.body.signature) {
    console.log('Invalid signature');
    return res.end();
  }

  // Log status of event
  const email = req.body.recipient;
  const event = req.body.event;

  if (event == 'complained') {
    console.log(`${email} complained about your mail`);
  }
  else if (event == 'dropped') {
    console.log(`Mail to ${email} dropped. ${event.description}`);
  }
  else if (event == 'bounced') {
    console.log(`Error ${event.code}: Mail to ${email} bounced. ${event.error}`);
  }

  res.end();
});

We can even step this up and:

For every request, check against a token cache to prevent use of the same token. Every token will be stored there. This will prevent replay attacks.
Check if the timestamp is not too far from the current time.

Making it scalable

If you send lots of emails and you are expecting lots of events, putting your webhook script on a server that can’t scale automatically is a bad idea. Even if you are not expecting lots of events, unexpected things can lead to a surge in events. Having a server that can scale automatically is really useful for instances like this.

Enter Serverless computing. In simple terms, the idea is that you can delegate the execution of your code and everything related to a provider. Because multiple instances of your code can be executed in parallel and you can adjust computing resources like RAM and execution time on the fly, it is highly scalable. You are also charged based on consumed resources and execution time so it can be really cheap.

There are a couple of serverless computing providers. One I use and recommend is Google Cloud Functions because of the ease of setting up HTTP functions. An HTTP function is a code block wrapped as a function that can be triggered by visiting a URL. This is exactly what we need as our webhook.

To create this function, we need to write a JavaScript function that will be exported as a Node.js module. The function takes HTTP-specific arguments: request and response.

exports.webhook = function(request, response) {
  // Handle event data here
  response.send({status:"ok"});
}

Based on the request content-type, the body of the request is automatically passed and available in the body parameter of the request object.

exports.webhook = function(request, response) {
  let event = request.body.event; // delivered
  // Handle event data here
  // ...
  response.send({status:"ok"});
}

This doesn’t work for the content type multipart/form-data though. And as we already know, Mailgun sends some data as multipart/form-data. We can bring in a library like Multer by using require(). However, we need to ensure the dependency is listed in the package.json file.

const multer = require('multer');

exports.webhook = function(request, response) {
    parser(request, response, function(){
    console.log(request.body); // Our event data
    // Handle event data here
    // ...
    response.send({status:"ok"});
    });
}

{
  "dependencies": {
    "multer": "^1.3.0"
  }
}

Next, we can publish the function to Cloud Functions. An easy way to do this is to do it from the Cloud Functions dashboard.

Go to your Google Cloud Console (if you don’t have an account yet, create one).
Enable Cloud Functions in the dashboard.
Click on Create Function.
Enter a name for your function (e.g mailgun-webhook).
In the trigger section, select HTTP trigger. Note the URL, that will be your webhook URL.
Copy your event data handling code to the index.js section of the Cloud function.
Copy the content of your package.json and paste in the package.json section.
Select or create a Stage bucket. The stage bucket is simply where the code is staged. You can use anything here.
In Function to execute, enter the name of your function (e.g webhook).
Save.

You can now use the function’s URL in Mailgun as your webhook URL.

Conclusion

Working with Mailgun’s webhooks is easy. There are many ways the event data can be used to enrich your applications outside Mailgun. If for example you allow your users send emails from your web for any reason and you use Mailgun, you can use this to provide analytics for them. Or maybe you want to send your email analytics to another platform. Or maybe you want to be notified of failures in your Slack account. Or maybe not even that. Maybe you just want more detailed analytics than what is available on the Mailgun dashboard. Whatever the use case is, the event data is available for you.

For a real world example, check out the source of Suet’s webhook file.