DEV Community: Sahand Seifi

The Developer's Guide to Google Bulk Sender Requirements

Sahand Seifi — Mon, 05 Feb 2024 17:47:20 +0000

Context

Starting in February 2024, Google enforces new requirements for sending emails to its users.

‍

Does this apply to me?

Yes. The requirements can be split into 3 levels:

Basic requirements for any sender
Bolder new requirements for domains that send more than 5000 emails per day
One additional requirement for domains that send marketing emails

Since it is unclear how Google recognizes your emails as transactional vs. marketing, we recommend that anyone sending more than 5k emails/day follow all the requirements.

‍

Do NotificationAPI users need to take action?

NotificationAPI users are automatically compliant.

‍

Level 1 Requirements:

Any Sender

‍

1. SPF & DKIM

You probably have one or both of these DNS records already setup:

AWS SES: Verified domains use DKIM by default, you may need to verify SPF
SendGrid: DKIM & SPF verified by default
Mailgun: you need to check the "DKIM" option when configuring your domain

How to test:

Send an email to yourself in Gmail. Check your domain's verifications using the "Show Original" option.

‍

2. Spam rate < 0.1% - 0.3%

This refers to how many of your outgoing emails are reported as spam by recipients. Google suggests keeping this below 0.1% (1 in 1000 emails) and avoiding 0.3%.

We recommend signing up for Google Postmaster Tools, which reports on your domain's email reputation and spam rates.

‍

3. Generic requirements: PTR Record, TLS Connection, RFC 5322 Email Formatting

If you use any modern email service, you shouldn't worry about these requirements.

‍

Level 2 Requirements:

Senders with 5000+ emails/day

‍

4. DMARC record

‍DMARC is a TXT record that has many configurations. Simply, it tells recipients how to treat emails from your domain that don't pass SPF/DKIM verification.

Setting a strict DMARC configuration could block your emails. So be careful!

We recommend that you start with a loose DMARC record, such as:

Record Name: _dmarc

‍Record Value: v=DMARC1; p=none;

This record tells recipients that you want to follow DMARC standard v1 (recommended) but not to do anything (p=none) when they encounter an email from your domain that doesn't pass SPF/DKIM.

Over time, you want to change the DMARC record to:

Report back emails with faulty SPF/DKIM using the rua option,
Fix the issues,
And make the DMARC record more strict using the p option

‍

5. DMARC Alignment‍

There are two "from" addresses for every email:

Header From: the regular From address you see on an email, e.g. John Smith john@smith.com
Envelope From: refers to the source of the email. For example, an email from john@smith.com may have an envelope header sendegrid.com.
DMARC Alignment means Header From matching your Envelope From.

Alignment could be 1) relaxed, where one From is the subdomain of the other from, or 2) strict, where the domains exactly match. Google is ok with either.

In the image below, you see a spam email where the Header From differs from the Envelope From, and Gmail is bringing attention to it with the "via" keyword.

‍

Level 3 Requirements:

Marketing emails

Recommended for Transactional Too!
There is no way to know how Google categorizes your emails (marketing vs transactional), so we recommend doing this anyway.

‍

6. One-Click Unsubscribe

First, create an API end-point like the one below. The method must be POST, but the URL can be anything.

Method: POST
URL: https://app.yourdomain.com/unsubscribe?email=user@gmail.com
Body: none
You should unsubscribe the user from your email when this end-point is hit. For example, Google will call this end-point when the user hits the "Unsubscribe" button in Gmail's interface.

The One-Click Unsubscribe mechanism in action
Then, add the following headers to your outgoing emails:

List-Unsubscribe-Post: List-Unsubscribe=One-Click
List-Unsubscribe: https://app.yourdomain.com/unsubscribe?email=user@gmail.com
Remember to replace it with actual values.

‍

Compliance through NotificationAPI

NotificationAPI provides the one-click unsubscribe option at no cost without writing a single line of code. Our account setup process also ensures your emails comply with SPF, DKIM, DMARC, and DMARC Alignment.

So, all NotificationAPI users are compliant without additional effort.

‍

Let us know what you think

Do you have any questions or did we miss anything? Ping us on our contact page, and we'll send you some goodies for your contribution!

‍

Sources

Serverless Notification Service Design in AWS - with diagrams

Sahand Seifi — Mon, 17 Oct 2022 14:12:42 +0000

Background and Motivation

We discussed why some software development teams consider an internal or external notification service in our Ultimate Guide to Notification Services, including reasons such as:

Complexity of notifications (number of channels, number of notifications) demanding a service-oriented approach
Reliability and scalability
Creating internal tools for observability and monitoring

We also proposed a generic Notification Service Design compatible with any cloud provider, with critical technical constraints and business requirements. This article proposes a Serverless design for a notification microservice with AWS services, given the same objectives mentioned in the previous article.

High-Level Architecture

Step-by-Step Architecture:

1. POST /send

The /send end-point is a Lambda function with AWS's new Function URL capability.
The Function URL authorizes the incoming requests based on the IAM_ROLE of the caller. This allows you to skip API Gateway or write any authorization code. So, for example:
- Your code runs on EC2 with a specific IAM Role
- You modify the policy on your EC2 role to allow invoking the /send end-point from your code
- When making an API call to /send from your code, you follow this article to sign your requests
The request to this end-point contains the userId of the user. This allows the service that sends the notification has no knowledge of your user's attributes.
The request also contains a notificationId, which indicates the type of notification, such as alert or new_customer.
The request to this end-point also contains all the channels the service wishes to send. These channels will get filtered in the next step based on user preferences.

2. Match Notification Preferences

A DynamoDB stores records that indicate whether user should receive a specific notification on a specific channel. Record structure: KEY: user_id:notification_id, VALUE: [{channel: "email", state: true}, {channel: "sms", state: false}]‍
If the send end-point reads a "false" value from the records, it will ignore that channel. If a record for a channel does not exist, it means the user has not explicitly set their preferences. In this case, you need to agree to a default.
Notice how we allow users to update the data in this table directly. How does this work? You can allow your front-end to directly update records in a DynamoDB with Fine-Grained IAM Policy for DynamoDB. For example, you only allow users to update records if the KEY contains their userId.

3. Get User Attributes

The /send end-point read the user's email address, phone number, ... from the Cognito User Pool.
Notice that you can only make Cognito's getUser() request 120 times per second. If you wish to send at higher scales, you may store this information in a DynamoDB.

4. Publish to Fanout

The /send end-point forms a message similar to the image above (top-right) and sends it to SNS, only containing channels that the user is subscribed to.
The SNS is configured to fanout this message to multiple queues.
However, SNS is configured to filter destinations based on the list of channels in the message. Example:
- The caller requests sending a notification using email and SMS
- The /send end-point decides that the user is not subscribed to SMS using the DynamoDB, so it only includes the email data in the SNS message
- SNS looks at the channels in the message and only duplicates the message to the email queue

5. Fanout to Job Processors

The messages go into SQS queues until they are processed.
You can configure complicated or straightforward failure and retry mechanisms at this stage.
We recommend setting the retry to 1. We also recommend setting up a DLQ with a 24-hour time-to-live (TTL) window. It will hold onto failed messages to reroute them later when issues are resolved.

6. Process Delivery Jobs

These are simple Lambda functions that take the message from the queue and send it to the appropriate service. For example, the email sender lambda would read the message and send it to SendGrid or a similar service.
You can limit the number of concurrent lambdas to avoid sending too many emails or SMS and getting throttled.

Further Improvements

Here are a few things that are possible but we haven't covered. If you need any of these capabilities, read the next section.

In-App Notifications: They are complex. They require tables, APIs, and UIs
Centralizing the notification UI elements and providing a visual editor on it
Feature flags for notifications
Monitoring and reporting

An external Notification-as-a-Service

Building an internal notification service is a lot of (annoying) work. We know because we have built this a few times in our careers.

That is why we made NotificationAPI: a plug-and-play notification-as-a-service solution. It takes 10 minutes to configure and integrate and gives you much more functionality, including in-app notifications, feature flags, and logs.

If I had to describe NotificationAPI in one word, it would be simple!
-- Jacob Brown, Software Team Lead

Watch Jacob's interview, discussing how they implemented their own notifications, then switch to NotificationAPI.

Notification Service Design - with diagrams

Sahand Seifi — Mon, 09 May 2022 14:38:12 +0000

In our Ultimate Guide on Notification Services, we discussed if and when you should build a Notification Service. This article proposes a notification system architecture that you can use as your in-house notification system.

Motivation

Notifications are a common culprit for code smells, technical debt, and endless HTML/CSS templates lying around your code-base. They usually end up with no tests, no oversight or ownership, and cause pains such as repeated notifications or banned email accounts.

Objective

To design a notification service that can send product-to-user notifications across many channels at scale

Requirements:

Send API: Expose an authenticated end-point so we can trigger sending notifications from any back-end and microservice
Supported Channels: Support sending notifications to any channel that exposes an API, e.g., Email, SMS, Push
User Preferences: Allow users to pick their user preferences on each notification and channel
Respecting downstream service limits: Avoid getting throttled or suspended by your email or SMS service
Scalable: Allow horizontal scaling for (theoretically) unlimited scaling

‍

High-Level Architecture

‍

A Quick Overview

Let's imagine that your code should send a notification. The numbers below correspond with the numbers you see on the diagram.

Your code calls the POST /send endpoint. The request contains the userId of the recipient, the type of the notification, and the contents of the notification for every supported channel.
The /send end-point authenticates the request using OAuth2's Client Credentials Flow.
It then requests the user's notification preferences from the database. The preferences indicate whether the user is subscribed to a particular notification and channel or not.
It will then read the user attributes such as email address or phone number from the database.
This end-point will form a message object containing user attributes from step (4) along with the channels and content for each channel. However, it will exclude disabled channels based on step (3). Finally, the message is sent to a fan out service.
The fanout service is configured to broadcast incoming messages to job queues. However, there is filtering in place to ignore job queues related to channels that are not listed inside the message.
There is a job queue and processor per channel. The processor picks up the job and requests the appropriate service, e.g., a transactional email or SMS service.

‍

Important Architecture Decisions:

POST /sent

You notice that the request to this end-point only contains the userId and not the email or phone number. This allows the services that send notifications have no knowledge of your users.
The end-point is behind a load balancer to ensure scalability.
The end-point is not protected with your regular user-facing authentication. Since the service that makes the request is a "program" itself, you need to use a different authentication mechanism known as the OAuth2 Client Credentials Flow used for server-to-server communication. Here are links to how to do this in Auth0 and Cognito.

Do we need a whole end-point for this?

There will be many parts of your application that will be issuing notifications. Implementing the send function as an end-point behind a load-balancer ensures that it is independently scalable and allows you to use it from virtually anywhere, e.g., from a new codebase or your build pipeline.

‍

User Preferences

Use a highly scalable NoSQL or key/value pair database. Structure the records as: KEY:
sample_user_id:sample_notification_id, VALUE: [{channel: "email", state: true}, {channel: "sms", state: false}]‍
When the send end-point sees "false" values in the records, it will remove the related channel from the message sent to the fanout. If a record for a channel does not exist, it means the user has not explicitly set their preferences. In this case, you need to agree to a default.
The information in the user_preferences table is updated by the user through your UI, through a normal end-point protected by your common authentication mechanisms.

Why do we need user preferences?

Not allowing users to control their notification preferences will only frustrate them and force them to mark your notifications as spam or mute your notifications. This will further damage your user experience and mark your account for suspension by email or SMS delivery services.

‍

Fan Out

Fanout takes a message and duplicates it to various places. They are cheap and highly scalable. In AWS, use SNS. In Google Cloud Platform, use Pub/Sub, and in Azure, use topics and subscriptions.
You can configure filtering between the fanout and job queues to avoid sending unnecessary messages to job queues of channels that have been excluded. For example, in AWS SNS, you can specify that the email job queue should only receive the fanout message if the message contains the "email" property inside the "channels" property.

Why do we need a fanout?

You could write code that puts the same message into the necessary job queues, but fanout is cheaper, and writing less code is good. Another benefit of fanout is being able to easily add/remove queues, thus allowing you to refactor and extend your channels.

‍

Job Processing

Queues hold on to messages until your job processors process them. They are also cheap and highly scalable. Job processors are code that takes messages from the job queues and processes them. They can scale based on the number of messages in the queue.
In our case, the job processor should make an API call to the appropriate service to send out the notification through a transactional email service.
Most email, SMS, or similar delivery services have strict guidelines on the amount and quality of messages you send. You should also carefully review these and put proper systems in place. Here is our guide on how to prevent getting suspended on AWS SES.
You can configure a max number of job processors to avoid hitting the rate limits of the delivery services.

Why do we need job queues and processors?

‍Multiple reasons: A) External delivery services are generally slow. The queue mechanism allows you to process these dead jobs asynchronously from the rest of your code. B) The queue mechanism allows you to control the rate of your jobs, thus avoiding getting throttled. C) These external services could face outages. A job queue mechanism lets you decide what to do in cases of job failure without a single line of code, e.g., retry the job every 30 minutes for a maximum of 3 times.

‍

Further Improvements

Here are a few things that are possible but we haven't covered. If you need any of these capabilities, read the next section.

The architecture of a scalable in-app notification service - they require their own APIs, tables, etc., so they deserve their own article
Removing notification contents from the code and instead allowing your product and design team to edit the notifications visually without code change
Dashboard for your team to enable/disable notifications or specific channels without code changes
Collecting and displaying open/click report

‍

A Quicker Approach

There is so much work that goes into building a scalable notification service. That is precisely why we made NotificationAPI: a plug-and-play notification-as-a-service solution. It takes 5 minutes to set up. It is scalable and has every feature you can imagine for your notifications, such as a dashboard for your team to design and configure notifications visually without developer involvement.
‍

At its core, all engineering comes down to making tradeoffs between the perfect and the workable.
-- Katie Hafner

‍

Conclusion

In this article, we learned about the architecture of a scalable notification service. We used tools available in all the major cloud providers so that you can build your notifications based on this. Alternatively, you also learned of a notification-as-a-service product that could save you all the trouble. Software engineering, like any other engineering discipline, comes down to trade-offs. Perhaps the decision tree and table in our ultimate guide on notification services can help you figure out what trade-offs you make.