Building a netlify lambda function to turn twitter lists into email digests.

I'm probably not the only one who feels overwhelmed by the overall panic on social media right now.

I've decided to code some sanity back into my life by creating a netlify lambda function that emails me tweets once a day.

Github Link

Tutorial: Setting Up

Things we need

• A free sendgrid account and API key
• A netlify account
• A twitter API key and API secret
• A public twitter list
• A cron job service that allows you to set headers (what I found: cron-job.org — free, easycron.com - free trial) or if you have a server you should be able to set this up yourself.
• Some understanding of Javascript & HTTP.
• An email

We need to create a node project and install dependencies. Run the following commands in your project directory. You should have a package.json/package-lock.json file and a node_modules folder afterwards.

npm init -y
npm i netlify-lambda @sendgrid/mail encoding twitter-lite dotenv-webpack

netlify-lambda - helps with bundling and local development.

@sendgrid/mail - helps us use sendgrid's mail API

twitter-lite - a wrapper for twitter's API

dotenv-webpack - setting local development environment variables (not covered in this post check out the github repo)

encoding - was installed to solve a deployment error I encountered (issue).

Project folder structure

Create the following files according to the structure below. (node modules, package-lock.json and package.json should already be there)

node_modules/ 
src/
  lambda/
    digest.js 
  utils/
    twitter.js 
    sendgrid.js 
    email.js 
netlify.toml 
.gitignore 
package.json 
package-lock.json 

Creating the digest

We have to do 3 things (Only 2 of which require writing code):

1. Get tweets from a twitter list.
2. Send out an email containing those tweets.
3. Schedule emails to be sent every day (or whenever) with cron-job.org

Below is our lambda function (digest.js) which brings together the different components of the function.

The first thing we do is add some basic authorization and validation of the request. A valid request to the function needs to:

• Have a request method of POST
• Have an authorization header
• Have the proper value in its authorization header

NOTE: I will be using process.env to refer to variables that would be set later on when deployed.

I am returning the status code of 404 in order to avoid revealing whether the endpoint exists. My rationale is that 404 - Not Found is more likely to deter some bored spammer/attacker than 401 - Unauthorized.

//digest.js
const twitter = require('../utils/twitter');
const sendGrid = require('../utils/sendgrid');

exports.handler = async (event, context)=>{
         try{
             //Validate the request method and the authorization header
             if(event.httpMethod != 'POST') return {statusCode: 404}
             if(!event.headers.authorization)  return {statusCode: 404};

             //check for valid authorization value
             const basicAuth = (new Buffer(`${process.env.AUTH_USER}:${process.env.AUTH_PASS}`)).toString('base64')
             if(event.headers.authorization.split(' ')[1] !== basicAuth) return {statusCode: 404};

             //Get tweets
             const tweets = await twitter();

             //Send email if there are tweets available
             if(tweets.length > 0) await sendGrid(tweets);

             // success
             return {statusCode: 200}
             
         }catch(err){
             console.log(err)
             //error
             return {statusCode: 500}
         }
}

The rest of the function involves calling the modules we imported at the top. One is a function that authenticates with twitter and gets tweets from our list (twitter.js). The other sends out our email (sendgrid.js).

1) Getting tweets from our twitter list

//File - twitter.js
const Twitter = require('twitter-lite')
module.exports = async () => { 
    //a) authenticate with twitter
    const user = new Twitter({
      consumer_key: process.env.TWITTER_KEY,
      consumer_secret: process.env.TWITTER_SECRET
    });
    const response = await user.getBearerToken();
    const app = new Twitter({
      bearer_token: response.access_token
    });

     //b) get tweets from list
    const data = await app.get('lists/statuses', {
                                list_id: process.env.TWITTER_LIST_ID,
                                tweet_mode: 'extended'
                            })

     //c) get only the necessary data & return tweets
    const tweets = data.map(tweet=>{
        return {
            text: tweet.full_text, 
            user:tweet.user.screen_name,
            url: `https://twitter.com/${tweet.user.screen_name}/status/${tweet.id_str}`
        }
    })

    return tweets;
     
}

a) We installed a package called twitter-lite which is a wrapper for the twitter api. To authenticate we just need our twitter api key (TWITTER_KEY) and twitter api secret (TWITTER_SECRET) passed into the packages Twitter function.

Twitter's API has user authentication and app authentication. We are using app authentication we just need a bearer token to start making requests to twitter's api. The package takes care of basically everything, you can see the full documentation for twitter-lite here

b) We can use twitter-lite to get tweets from a list by calling the get method on "app". The object being passed into app.get( ) contains the query parameters needed for twitter's lists/statuses endpoint.

This includes the id of the twitter list(list_id) and whether we'd like to get the full tweet versus a truncated version of the tweet (tweet_mode). It returns an array of tweet objects when done.

c) Once we have the tweets, we only need a few bits of information (the tweet text, the username of the "tweeter" and the tweet URL). Using the array map method we can create a new array that only contains the necessary data from each tweet.

2) Sending the email.

//sendgrid.js
//a) set sendgrid's api key
const sgMail = require('@sendgrid/mail');
const createHtmlEmail = require('./email');
sgMail.setApiKey(process.env.SENDGRID_KEY); 

module.exports = (tweets) => {
    return sgMail.send({ 
            to: process.env.MY_EMAIL,
            from: process.env.MY_EMAIL,
            subject: 'Your Tweet Digest',
            html: createHtmlEmail(tweets), //b) create html email & send
            });
}

a) Here we import sendgrid's npm package, set the api key for sendgrid and import the module that helps us create the HTML email containing the tweets.

b) sgMail.send( ) takes in an object with some information about the email we need to send. To get the value for the "html" property we call createHtmlEmail and pass in our tweets.

Here is what goes on inside the createHtmlEmail function. We are essentially inserting our tweet data into html and returning a string of html. This is very easy to do thanks to template literals.

//File - email.js
//creates the html email
module.exports = (tweets) => {
    let tweetElements = '';

    //insert tweet data into elements
    tweets.forEach(tweet=>{
       tweetElements += `
                <div style = "width: 100%; border: 1px solid black; margin-bottom:10px; padding: 10px ">
                <p>@${tweet.user}</p>
                <p>${tweet.text}</p>
                <a style="color: white; padding:5px; background: black; text-decoration:none" href="${tweet.url}">Read Tweet </a>
                </div>
               `
    })
    //Wrap tweet elements in a div
    let email = `   
        <div style = "width: 40rem; font-size: 1.2rem; margin: 0 auto">
            ${tweetElements}
        </div>
    
    `;
 
    return email;
}

sgMail.send() returns a promise which is why we use the await keyword in our lamda function(digest.js).

Deploy the function !

Before we deploy, we need to edit a few files

• package.json - adding a build script
• netlify.toml - adds a build command and lets netlify know what folder our function is in (src/lambda)
• .gitignore - removes our node_modules folder from git tracking

//package.json
"scripts": {
    "build:lambda": "netlify-lambda build src/lambda"
}

//netlify.toml
[build]
  Command = "npm run build:lambda"
  Functions = "lambda" 

//.gitignore
node_modules/

Now we can commit the project to github.

git init
git remote add origin <link to git repository>
git add .
git commit -m "commit message"
git push origin master

We can now login to netlify. You should see the option to create a new site from git, once you click that you'll go through a 3 step process where you choose your git provider, authorize netlify and pick a repository. The most important is step 3 where we have build options.

Clicking "Show Advanced" then "new variable" should let you add environment variables (the process.env prefixed stuff throughout the code).

Here is a list of the variables that need to be set

Name:           Description:
TWITTER_KEY     your twitter API key
TWITTER_SECRET  your twitter API secret
AUTH_USER       a username (does not have to be your twitter username, just come up with something)
AUTH_PASS       a password (does not have to be your twitter password, just come up with something)
YOUR_EMAIL      your email
TWTTER_LIST_ID  id of your twitter list, it is simply the numbers at the end of the URL when you are on your twitter list
Example:  <https://twitter.com/i/lists/1234> will have a TWITTER_LIST_ID of 1234

The github repository explains some of these environment variables a bit better. After filling in the environment variables the project can be deployed.

Scheduling the function to run every day.

You should be able to access your function's url like this:

https://[DEPLOY_URL]/.netlify/functions/digest.js

[DEPLOY_URL] represents the site url shown in netlify's overview dashboard for your deployed site. For example if your site's url is example.netlify.com then your function url is:

https://example.netlify.com/.netlify/functions/digest.js

We'll be using a cron job on https://cron-job.org to call the function's url daily. After signing up on the site you should be brought into a members panel where you can click "Cronjobs" on the upper left navigation bar then click "create cronjob".

Now fill out the form

• Title - Any name you want
• Address - your function url (https://....)
• Check "Requires HTTP authentication"
  • User - same as AUTH_USER set previously
  • Password - same as AUTH_PASS set previously
• Schedule - set the schedule for any time interval (I'm using every day)

When you are done with that, click "create cronjob". You'll be redirected to a page listing your current cronjobs, now find the newly created job and click "edit".

In the edit form page you can scroll down to find "Advanced Settings" which allows you to change the request method from GET to POST then save the form.

and.....

That's it!

Results! :D

Looks decent.

There are lots of improvements to be made of course, so if you have some ideas to improve it you can go ahead and contribute to the project on github.

: D

Resources:

Github Repository

Original Blog Post