Building an Email Marketing Engine: Express.js & SendGrid Part 1

Introduction

In this digital era, businesses thrive on effective communication. Email marketing is a cornerstone in building relationships with customers and keeping them engaged. While there are numerous email marketing platforms available, there's something uniquely satisfying about building your own system, one that's tailored to your specific needs.

Also, it provides you the opportunity to bypass the expensive marketing plans provided by email marketing services by only accessing their API instead of their whole suite.

In this multi-part series, we'll walk through the process of creating a custom email marketing engine using Node.js, focusing on the core functionalities in this first part. We will set up a simple web server using Express.js, design an API to send bulk emails using the SendGrid service, and create a basic web form for our email configuration. We'll also touch upon the topics of error handling, application structure, and data management.

In this initial version, our application will take the ID of a SendGrid contacts list, retreive its email adresses and send a desired email to those contacts.

And of course, we will be hosting and deploying this project on Codesphere, which wont even take 3 minutes to set up from local to live.

If you want to skip the coding and directly create host your own version of our Email Marketing engine to see for yourself just click here. No need for any setup, just define the environment variable for your API key ( SENDGRID_API_KEY="your_key"), install your dependencies (npm install) and run npm start. No more config needed!

Let's dive right in and start setting up our environment!

Setting up the environment

In this tutorial, we are going to use several Node.js packages to facilitate the creation of our custom marketing mail engine:

Dependency	Usage
@sendgrid/mail	Handling mail operations using SendGrid service
express	Serve our application and handle HTTP requests
dotenv	Manage environment variables securely
ndjson	Parse newline JSON data, format provided by the SendGrid API
axios	Promise-based HTTP client for Node.js for handling requests
tailwindcss	CSS framework used for our frontend
@tailwindcss/forms	Provides useful form styles when using Tailwind CSS
autoprefixer, postcss	Required dependencies installed with Tailwind CSS
npm-run-all (dev)	CLI tool to run multiple npm-scripts in parallel or sequential

Once your development environment is set up and your dependencies have been installed, the next step is to establish the base of our application: the server.

This is the foundational step in our journey. We have chosen to use Express.js for this. Express simplifies the process of setting up a server and allows us to handle different types of HTTP requests with ease, essentially creating our own API.

To kick things off, create a new file in the root of our project named index.js. This is where we'll initialize our Express server. Let's start with a full overview of the index.js file. We'll be breaking this down in stages as we progress.

Initially, we import the necessary modules, including our functions for handling API requests and processing the data provided by the SendGrid API. We then initialize our Express application and set up the express.json() middleware to parse incoming JSON payloads correctly.

Next, we establish our /contacts route, the only route we're setting up for this iteration of our series. This route is where our imported functions come into play to fetch and process contact data and dispatch emails.

Remember, as you're coding along, the complete index.js code with detailed comments is available below. I highly recommend referring to it as we navigate through each part of the code.

// Import necessary modules
const express = require('express');
const { getContactsLink, downloadAndDecompress, sendEmails } = require('./server/api');
const { processContacts } = require('./server/utils');

// Initialize express app
const app = express();
const port = 3000;

// Use express built-in middleware for parsing incoming JSON payloads
app.use(express.json());

// Route to handle contact data fetch and send emails
app.post("/contacts", async (req,res) => {
  console.log(req.body);

  const {listId , emailText} = req.body;
  console.log(listId);

  try {
    // Get contacts data URL from SendGrid
    const contactsUrl = await getContactsLink(listId);
    // Download and decompress the contacts data file
    await downloadAndDecompress(contactsUrl);
    // Process the contacts data into groups
    const emailGroups = await processContacts(2);

    // Send emails to each group
    for (let group of emailGroups) {
      const emailAddresses = group.map(contact => contact.email); // Assuming each contact object has an 'email' field
      await sendEmails(emailAddresses, emailText);
    }

    // Send the email groups back in the response
    res.json(emailGroups);
  } catch (error) {
    // Handle errors
    res.status(500).json({ error: "An error occurred while fetching contacts" });
  }
})

// Use express built-in middleware to serve static files
app.use(express.static('public'));

// Start the server
app.listen(port, () => {
  console.log(`Server is running at http://localhost:${port}`);
});

This walkthrough guides you through each step of setting up the server and provides you with a solid understanding of how each component functions in our application. In the next sections, we'll dive into the specific details of our API handling functions and data processing utilities.

Working with the SendGrid API and Email Setup

With our server set up and functional, let's understand how we manage API requests to retrieve the contact data necessary for sending our emails.

To fetch contact data from the SendGrid API, we proceed as follows:

1. Initiate a POST request : We use the fetchContacts function to send a POST request to the SendGrid API, including the ID of the list of contacts we wish to target with our emails. This request yields an API endpoint to retrieve the URL of the JSON file containing contact information.
2. Send a GET request : We use the pollForContactsUrl function to send a GET request to the acquired endpoint. Depending on the JSON file's size, generating the download link might take time, so we continue polling until our response status returns 'ready'.
3. Extract contact links : The getContactsLink function is then employed to run fetchContacts and pollForContactsUrl to extract the URL necessary for downloading the JSON file.
4. Download the JSON file : Finally, we use the downloadAndDecompress function to get our JSON file and handle its data. For this phase, we're not working with a database. Instead, we send a GET request to our JSON file's download link, then leverage Node's fs and path modules to write the JSON file to a specified server location.

In the following section, I'll explain how we process the JSON data using a separate utils.js file, preparing it for sending using the sendEmails function.

But first, a quick overview of the sendEmails function:

It utilizes the imported sgMail module to dispatch the emails. The function accepts an array of email addresses (defined in our utils.js file) and the email content as emailText (sourced from our frontend). To prevent recipients from viewing a long list of email addresses in their email client's "to" field, we construct a personalizationsArray—essentially, it creates a separate envelope for each email, instead of one mass email addressed to all recipients. In this phase, we will not delve deep into the email text since our focus is on establishing the application's groundwork. Therefore, we directly pass the emailText into the HTML inside a paragraph and to the plain text of our msg object.

// Import required modules and packages
require("dotenv").config();
const axios = require('axios');
const config = require('./config');
const fs = require("fs");
const path = require("path");
const sgMail = require('@sendgrid/mail');

const sendGridEndpoint = `${config.SENDGRID_CONTACTS_ENDPOINT}`

// Function to fetch contacts: sends a POST request to the SendGrid API
const fetchContacts = async (listId) => {
    try {
        // Send POST request to the SendGrid API
        const response = await axios.post(sendGridEndpoint, {
            "list_ids": [listId],
            "file_type": "json",
            "max_file_size": 1000
        }, {
            headers: {
                "Authorization":`Bearer ${process.env.SENDGRID_API_KEY}`,
                "Accept":"application/json",
                "Content-Type": "application/json"
            }
        });
        return response.data;
    } catch(error) {
        console.error(`Error fetching contacts ${error}`)
        throw error;
    }
};

// Function to poll for the contacts URL: makes a GET request and waits until the status === ready
const pollForContactsUrl = async (url, interval, maxAttempts) => {
    for (let attempt = 0; attempt < maxAttempts; attempt++) {
        try {
            let response = await axios.get(url, {
                headers: {
                    "Authorization":`Bearer ${process.env.SENDGRID_API_KEY}`,
                    "Accept":"application/json",
                    "Content-Type": "application/json"
                }
            });

            if (response.data.status === 'ready') {
                return response.data;
            }
        } catch (error) {
            console.error(`Error while polling for contacts: ${error}`);
            throw error;
        }

        // Wait for a specified interval before making the next request
        await new Promise(resolve => setTimeout(resolve, interval));
    }

    throw new Error('Max attempts exceeded while polling for contacts');
};

// Function to retrieve contacts link
const getContactsLink = async (listId) => {
    try{
        // Fetch the contacts URL
        const initialReq = await fetchContacts(listId);
        const contactsEndpoint = initialReq._metadata.self;

        // Poll for the contacts URL
        const contactsExport = await pollForContactsUrl(contactsEndpoint,1000,5);
        const contactsUrl = contactsExport.urls;

        return contactsUrl;
    } catch(error) {
        console.error(`Error downloading contacts ${error}`)
        throw error;
    }
}

// Function to download and decompress the JSON file
const downloadAndDecompress = (contactsUrl) => {
  return new Promise((resolve, reject) => {
    axios({
      method: "get",
      url: contactsUrl,
      responseType: "stream",
    })
      .then((response) => {
        const filePath = path.join(__dirname, 'data', 'contacts.json');
        const writer = fs.createWriteStream(filePath);

        response.data.pipe(writer);

        // Event listener when writing is complete
        writer.on("finish", function () {
          resolve();
        });

        // Event listener for handling errors
        writer.on("error", function (err) {
          console.error("Error writing data to contacts.json:", err);
          reject(err);
        });
      })
      .catch((error) => {
        console.error("Error downloading contacts:", error);
        reject(error);
      });
  });
};

// Set up SendGrid
sgMail.setApiKey(process.env.SENDGRID_API_KEY);

// Function to send emails to the list of contacts
const sendEmails = async (emailAddresses, emailText) => {
    let personalizationsArray = emailAddresses.map(email => {
        return { to: [{ email: email }] };
    });

    const msg = {
        from: '[e-mail]',
        subject: 'Hello from Codesphere',
        text: emailText,
        html: `<p>${emailText}</p>`,
        personalizations: personalizationsArray,
    };

    try {
        await sgMail.send(msg);
        console.log('Emails sent successfully');
    } catch (error) {
        console.error('Error sending emails:', error);
        throw error;
    }
};

module.exports = { fetchContacts, getContactsLink, downloadAndDecompress, sendEmails }

Processing the contacts list

Now, let's focus on how we can refine our data to make it more suitable for our sendEmails function.

For this, we incorporate a separate file in our server-side code named utils.js. The main function in this file, processContacts, is tasked with opening a read stream from our contacts.json file using Node's fs module. It then pipes the parsed data into an array for further processing.

You might notice that we're using the ndjson module here. This is because SendGrid utilizes the newline delimited JSON (ndJSON) format for its contact data. Therefore, we need ndjson to parse this particular file format correctly.

We encapsulate our processing within a Promise. This ensures that our data is fully processed before we proceed with other operations. In essence, we don't want to start sending emails before we've finished preparing our recipient list.

Lastly, SendGrid imposes a limit of 1000 recipients per send request. To work around this limitation, we've designed a helper function chunkArray that divides our large contact list into smaller sublists or chunks. Each of these chunks can then be sent individually, ensuring we stay within SendGrid's constraints.

// Require the necessary modules
const fs = require('fs');
const ndjson = require('ndjson');

// Define a function to chunk an array into smaller arrays of a specific size
function chunkArray(array, chunkSize) {
    let results = [];

    // While the array still has elements, splice it into chunks and push the chunks to results
    while (array.length) {
        results.push(array.splice(0, chunkSize));
    }

    return results;
}

// Create a function to process the contacts
function processContacts(chunkSize) {
    return new Promise((resolve, reject) => {
      let data = [];

      console.log("Creating array from JSON");

      // Read the contacts.json file, parse it as ndjson, and push each parsed object to data
      fs.createReadStream('server/data/contacts.json')
        .pipe(ndjson.parse())
        .on('data', function (obj) {
          data.push(obj);
        })
        .on('end', function () {
          // When all data is read and parsed, chunk the data array and resolve the promise with the resulting groups
          let emailGroups = chunkArray(data, chunkSize);
          console.log("New data array", emailGroups);
          resolve(emailGroups); 
        })
        .on('error', function (err) {
          // If an error occurs during the stream, reject the promise
          reject(err); 
        });
    });
  }

// Export the function for use in other modules
module.exports = { processContacts }

Setting up the Frontend

To be able to pass in a list ID and an email text, we need a front end. To make this easier to set up, I used TailwindCSS to be able to leverage TailwindUIs ready made components.

I created a public folder with an index.html and a success.html in it. The index.html contains our form including a small loading animation and an error message in case of a fetch error as well as a script tag to handle our fetch operation on form submit.

The JavaScript creates an Event Listener for our form submit. We then get the form data and put it into a formData object.

This object is then sent sent to the server in a POST request using the fetch API provided with JS. In case of a successful response from the server, the user is redirected to a success.html.

<!-- File: public/contacts.html -->
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Contacts</title>
  <link href="./index.css" rel="stylesheet">
</head>
<body>
<div class="container mx-auto mt-8">
  <form id="emailForm">
    <div class="space-y-12">
      <div class="border-b border-gray-900/10 pb-12">
        <h1 class="mb-4 text-4xl font-extrabold text-gray-900">Configure your Email</h1>
        <p class="mt-1 text-sm leading-6 text-gray-600">Configure your List-Id and Email contents here!</p>

        <div class="mt-10 grid grid-cols-1 gap-x-6 gap-y-8 sm:grid-cols-6">
          <div class="sm:col-span-4">
            <label for="username" class="block text-sm font-medium leading-6 text-gray-900">List ID</label>
            <div class="mt-2">
              <div class="flex rounded-md shadow-sm ring-1 ring-inset ring-gray-300 focus-within:ring-2 focus-within:ring-inset focus-within:ring-indigo-600 sm:max-w-md">
                <input type="text" name="username" id="list-id" autocomplete="username" class="block flex-1 border-0 bg-transparent py-1.5 pl-2 text-gray-900 placeholder:text-gray-400 focus:ring-0 sm:text-sm sm:leading-6" placeholder="6aef6a7c-73dd-45e7-b982-9157ed6e88da">
              </div>
            </div>
          </div>

          <div class="col-span-full">
            <label for="about" class="block text-sm font-medium leading-6 text-gray-900">Email Text</label>
            <div class="mt-2">
              <textarea id="email-text" name="about" rows="3" class="block w-full rounded-md border-0 py-1.5 text-gray-900 shadow-sm ring-1 ring-inset ring-gray-300 placeholder:text-gray-400 focus:ring-2 focus:ring-inset focus:ring-indigo-600 sm:text-sm sm:leading-6"></textarea>
            </div>
            <p class="mt-3 text-sm leading-6 text-gray-600">This will be the content of the email and rendered in the email client inside a &lt;p&gt;.</p>
          </div>

    <div class="mt-6 flex items-center justify-start">
      <button type="submit" class="rounded-md bg-indigo-600 px-3 py-2 text-sm font-semibold text-white shadow-sm hover:bg-indigo-500 focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2 focus-visible:outline-indigo-600">Send Emails</button>
      <p id="error-message" class="mt-3 text-sm leading-6 text-red-600 hidden">Request failed.</p>

      <div role="status">
        <svg aria-hidden="true" id='spinner' class="hidden w-8 h-8 mr-2 text-gray-200 animate-spin dark:text-gray-600 fill-blue-600" viewBox="0 0 100 101" fill="none" xmlns="http://www.w3.org/2000/svg">
            <path d="M100 50.5908C100 78.2051 77.6142 100.591 50 100.591C22.3858 100.591 0 78.2051 0 50.5908C0 22.9766 22.3858 0.59082 50 0.59082C77.6142 0.59082 100 22.9766 100 50.5908ZM9.08144 50.5908C9.08144 73.1895 27.4013 91.5094 50 91.5094C72.5987 91.5094 90.9186 73.1895 90.9186 50.5908C90.9186 27.9921 72.5987 9.67226 50 9.67226C27.4013 9.67226 9.08144 27.9921 9.08144 50.5908Z" fill="currentColor"/>
            <path d="M93.9676 39.0409C96.393 38.4038 97.8624 35.9116 97.0079 33.5539C95.2932 28.8227 92.871 24.3692 89.8167 20.348C85.8452 15.1192 80.8826 10.7238 75.2124 7.41289C69.5422 4.10194 63.2754 1.94025 56.7698 1.05124C51.7666 0.367541 46.6976 0.446843 41.7345 1.27873C39.2613 1.69328 37.813 4.19778 38.4501 6.62326C39.0873 9.04874 41.5694 10.4717 44.0505 10.1071C47.8511 9.54855 51.7191 9.52689 55.5402 10.0491C60.8642 10.7766 65.9928 12.5457 70.6331 15.2552C75.2735 17.9648 79.3347 21.5619 82.5849 25.841C84.9175 28.9121 86.7997 32.2913 88.1811 35.8758C89.083 38.2158 91.5421 39.6781 93.9676 39.0409Z" fill="currentFill"/>
        </svg>
        <span class="sr-only">Loading...</span>
      </div>

    </div>
  </form>
</div>

  <script>
    document.getElementById("emailForm").addEventListener("submit", function(event){
      event.preventDefault();
      console.log("submit logged");

      const listId = document.getElementById('list-id').value
      const emailText = document.getElementById('email-text').value

      const formData = {
        listId:listId,
        emailText:emailText
      }

      document.getElementById('spinner').style.display = 'block';

      fetch('/contacts', {
        method: "POST",
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify(formData)
      })
      .then(response => {
        document.getElementById('spinner').style.display = 'none';

        if(response.ok) {
          window.location.href = "/success.html";
        }
      })
      .catch(error => {
        console.error('Error:', error);
        const errorMessageElement = document.getElementById('error-message');
        errorMessageElement.innerText = 'Sending failed: ' + error.message;
        errorMessageElement.style.display = 'block';

        document.getElementById('spinner').style.display = 'none';
      });
    })
  </script>
</body>
</html>

Bringing it together

I already provided the full index.js above but I want to go over the /contacts route once again to see how all our modules are brought together.

As you can see, once the fetch request from the frontend is made to our server, we create a new object for our listId and emailText using the JS destructuring assignment.

We then use getContactsLink with listId as the argument to retreive the endpoint for our JSON file. We then proceed to run downloadAndCompress to download the JSON file to our server and process the contacts using processContacts. In this case, processContacts takes in a value of 2 as an argument which is used to split the email recipients in chunks of two. This is only for testing purposes. You could enter 1000 here.

After everything is processed, we loop over emailGroups array that houses our chunked email recipients and finally call sendEmails using the our newly created emailAdresses array which contains the email adresses.

app.post("/contacts", async (req,res) => {
  console.log(req.body);

  const {listId , emailText} = req.body;
  console.log(listId);

  try {
    const contactsUrl = await getContactsLink(listId);
    await downloadAndDecompress(contactsUrl);
    const emailGroups = await processContacts(2);

    // Send emails to each group
    for (let group of emailGroups) {
      const emailAddresses = group.map(contact => contact.email); // Assuming each contact object has an 'email' field
      await sendEmails(emailAddresses, emailText);
    }

    res.json(emailGroups);
  } catch (error) {
    res.status(500).json({ error: "An error occurred while fetching contacts" });
  }
})

Conclusion

We've established the groundwork for our email marketing engine, from setting up the Express server, retrieving contacts from the SendGrid API, to chunking our contact list to align with SendGrid's constraints. However, our journey towards building a comprehensive email marketing platform has only just begun.

You can check out the latest status of our Email Marketing Engine here on Github or just host it for free on Codesphere by clicking here. Just define your CI pipeline (npm install, npm start) and env variables ( SENDGRID_API_KEY="your_key")

In upcoming parts of our series, we'll dive deeper into several key functionalities that will improve our application and provide greater flexibility and automation:

Trigger Email Templates : We'll explore how to utilize SendGrid's dynamic email templates, allowing us to tailor our communication for various purposes while maintaining a consistent brand image.

• Trigger Email Templates : We'll explore how to utilize SendGrid's dynamic email templates, allowing us to tailor our communication for various purposes while maintaining a consistent brand image.
• Database for Contact Entries : To better manage our contacts, we'll create a database with entries for each contact, which will include their email address, name, and ID.
• API Endpoint for New Contacts : To continuously grow our audience, we'll build an API endpoint for adding new contacts. This could be used, for example, to integrate a 'Sign Up' form from our front end to our back end.

Comments

[Elias Groll]

Great doc!

[Tobias Ohlsson]

Awesome!