Why Your Emails May Be Bouncing Back & What You Can Do About It

Recently at Giant Machines, I was investigating a handful of emails from one of our client projects that were not being properly delivered to specific Internet Service Providers (ISPs). After doing some research on their email settings and configurations, I discovered that an ISP's spam scanner configuration may have been mistakenly blocking harmless emails for its customers. As a result, I had to troubleshoot and debug these emails locally to identify why they were not being delivered. For the local debugging that I did, I used an npm package called spamscanner, which I will be covering in more depth down below.

55% to 85% of all email traffic is due to spam emails! Due to this, email clients are constantly attempting to block potentially harmful spam emails from reaching their users. However, at times, perfectly valid (and non-spam) emails can be flagged incorrectly.

For this post, I will cover how ISPs use spam scanners to try to detect malicious emails. I will also show you how to set up a spam scanner locally that you can use to debug and troubleshoot your own emails. This troubleshooting can be helpful in determining why your emails may not be reaching their intended recipients! This post assumes a basic familiarity with JavaScript, using a terminal, and using a code editor.

Email Basics

The process for sending and receiving email has changed quite a bit since its inception, but its basic principles have remained in place. For the purposes of this post, all that you need to know is that web mail clients (such as Gmail, Outlook, etc.) communicate back and forth with dedicated email servers. For a closer look at how email works under the hood, refer to the following article: "How Does Email Work."

Email Authentication

Due to the abundance of email spam, several protocols have been implemented over the years to try to mitigate spam messages by performing various programmatic checks.

The three main protocols are:

• SPF (Sender Policy Framework): Is the sender who they claim to be?
• DKIM (DomainKeys Identified Mail): Encrypts email headers with a private key; servers then use a publicly available key to decrypt the headers and verify the message.
• DMARC (Domain Message Authentication Reporting and Conformance): Built on top of SPF and DKIM; senders can set policies deciding how to handle SPF/DKIM and what to do for failing checks.

For additional information on these email authentication protocols, refer to the following article: "How Email Authentication Works."

Email Spam Scanners

To detect whether incoming emails are malicious, mail servers also use spam scanners, such as Apache's popular SpamAssasin. The internal workings of these spam scanners can be somewhat complicated (involving Naive Bayes Classifiers on trained, large datasets, for the curious), but the primary takeaway is that these classification systems typically assign a numerical point value to an incoming email to determine the validity of the message. The higher the score, the more likely that the email is spam. For reference, the ISP Optimum states the following regarding their spam filtering:

All incoming emails are evaluated against these spam rules and are assigned a "spam score". This score determines whether the message will be classified as spam. For standard filtering, the threshold is set at 5, meaning any message with a score of 5 or higher is classified as spam. Messages scoring between 5 and 10 will be delivered, but will include a spam notification in the subject of the email so that you can immediately identify and delete these messages.

Different ISPs have different policy configurations on their chosen spam scanner, but the same idea applies.

Services like Litmus provide the ability to see how various spam scanners rank your emails.

As you can see in the screenshot above, the email template that I was investigating received very low scores across the various spam scanners. So what gives? Why were these emails bouncing back, despite having a low score? We will be taking a closer look at this specific issue down below.

First Steps

Before using a spam scanner to investigate and troubleshoot your email templates, there are some quick wins for lowering your score that can be achieved by following some of the recommendations listed in this article.

So… about that local spam scanner setup?

Initial Setup

For installation instructions on the npm package spamscanner, refer to their docs.

Simple Server and Local Email Template

Email clients allow you to download your email messages (with the file extension ".eml"). With these locally saved messages, we can run spamscanner against them to further inspect their contents.

Assuming that you have installed spamscanner and have Node.js locally setup, you may use the following bare-bones script for running the scanner against a locally saved email message:

// in a file called index.js
const fs = require("fs");
const path = require("path");
// Make sure to install spamscanner in your package.json
const SpamScanner = require("spamscanner");

const scanEmail = async () => {
  // For a list of all options & their defaults, see:
  // https://www.npmjs.com/package/spamscanner#api
  const scanner = new SpamScanner({ debug: true });

  // Swap out the "Your_locally_saved_message_here.eml" file with the actual filename in the directory
  // containing this script
  const source = fs.readFileSync(
    path.join(__dirname, "Your_locally_saved_message_here.eml")
  );

  try {
    const scanResult = await scanner.scan(source);

    // For a list of properties available for inspection, see:
    // https://www.npmjs.com/package/spamscanner#scannerscansource
    console.log("Scan results, scanResult.mail:", scanResult.mail);
  } catch (err) {
    console.error("Error in scanEmail:\n", err);
  }
};

scanEmail();

// To run this script, run `node index.js` in your terminal where this script resides.

Note that you can also run the scanner on Strings or Buffers as long as they are a complete SMTP message (i.e., they include headers and the full email contents).

The results of running this script will come back in the following shape:

interface ScanResult {
  is_spam: boolean;
  message: string;
  results: {
    classification: Object;
    phishing: Array;
    executables: Array;
    arbitrary: Array;
  };
  links: Array;
  tokens: Array;
  mail: Object;
}

For a detailed description of these fields, refer to the docs. Typically, the result in the is_spam field should be enough to give you confidence that your email will not be marked as spam. Note that spamscanner does not assign a numerical value but instead opts to return a boolean.

However, different ISPs use different spam scanners, and it may be necessary to investigate your email messages further. To do so, make sure that the "debug" flag is set to true, as per the code sample above. You can then inspect the contents of scanResult.mail, which is an object containing more detailed debugging information regarding the email contents (shown below).

This ".mail" object returns the following shape:

interface ParsedMail {
  attachments: Attachment[];
  bcc?: AddressObject | AddressObject[];
  cc?: AddressObject | AddressObject[];
  date?: Date;
  from?: AddressObject;
  headerLines: HeaderLines;
  headers: Headers;
  html: string | false;
  inReplyTo?: string;
  messageId?: string;
  priority?: "normal" | "low" | "high";
  references?: string[] | string;
  replyTo?: AddressObject;
  subject?: string;
  text?: string;
  textAsHtml?: string;
  to?: AddressObject | AddressObject[];
}

It can be used to get more specific information about the email.

A sample screenshot of the "headers" field that is a part of the ".mail" object is shown below.

In the emails that I was investigating, the spam scanner classifier was marking the email messages as "not spam" but Optimum was appending the following X-Optimum-spam: yes header to the messages as they were incoming:

Some of the headers present in the email message file. Note the Optimum spam header.

This was causing these messages to not only be marked as spam but they were also being blocked/bounced entirely!

When all else fails, try manual debugging.

If your messages are still being blocked despite a low spam scanner score (or is_spam is false if using spamscanner), you may have to take a more manual approach. To do so, I gradually removed parts of the email and re-sent the trimmed-down emails to the ISP that was blocking us. I was eventually able to trace the problem down to this line:

<a href="mailto:[e-mail]">Contact customer support</a>

Specifically the mailto: present in the template caused Optimum's email configuration to flag the email as spam and reject the message outright, despite mailto tags not causing our messages to be flagged as spam by other ISPs.

Additionally, other emails were bouncing back due to the following (modified) copy:

If this email change request wasn't authorized by you, please click the link below to cancel. If you have any questions, you can contact support via [e-mail] or by calling [phone].

Specifically, the +1 present in the template caused Optimum's spam scanner configuration to flag the email as spam and reject the message outright despite being valid and despite not being flagged by other ISPs or SpamAssasin.

Due to Optimum's unique SpamAssassin configuration, we were seeing issues for our customers who had an Optimum domain email and attempted to receive emails with "mailto:" or "+1" present. It is not clear why Optimum chooses to block these emails when other ISPs do not, but it could be the case that their configuration is particularly sensitive and errs on the side of caution in attempting to mitigate potential security risks.

The issues that may be affecting your emails may differ but the techniques used here can help you narrow down why your emails may be bouncing back!

TL;DR

• Email servers accept, forward, and deliver messages.
• If an email has not been properly authenticated, the email server must return a failure "bounce" message.
• Spam scanners typically assign a point ranking to emails to classify them as spam or not spam. Hot dog/not hot dog anyone? 🌭
• You can use the npm package spamscanner locally on your email templates to check whether they are being classified as spam.
• When all else fails, you may have to try a more manual debugging approach to debug ISP-specific edge cases.

Additional Resources

• https://spamassassin.apache.org/
• https://sendgrid.com/blog/10-tips-to-keep-email-out-of-the-spam-folder/
• https://sendgrid.com/docs/glossary/spam/

Have any questions? Comment down below and happy coding!