Super-powered newsletter content with Pocket and Netlify Lambda

#jamstack #serverless #javascript #productivity

An example Netlify Lambda to fetch all “newsletter” posts from Pocket.

Pocket is an application and web service for managing a reading list of articles from the Internet. It's quite widely used and tightly integrated into the Firefox browser. I find that I use it extensively to save articles (usually about development).

For the Enterprise Node.js and JavaScript newsletter I have a “From the Web” section, which is populated from links from the web. Since most links that end up there are at one point or another stored on Pocket, I built a lambda that fetches posts tagged with “newsletter” from Pocket. I then consume that lambda from a Hugo newsletter file generator.

See the lambda code at src/lambda/newsletter.js in the repository github.com/HugoDF/pocket-newsletter-lambda.

Run it at https://pocket-newsletter-lambda.netlify.com/, or even deploy your own on Netlify.

The application is styled using TailwindCSS, you can see a starter project for that at github.com/HugoDF/netlify-lambda-tailwind-static-starter.

Pocket Fetching logic

The bulk of the Pocket-specific logic is the fetchBookmarks function, it does the following:

fetch from Pocket API using consumer key and access token
- passes state: 'all' in order to get both archived and unarchived posts
- uses tag: 'newsletter' to fetch only posts tagged with newsletter
- detailType: 'complete' means the API returns more complete data
convert the response to a flat list of { title, url, excerpts, authors } (all of those fields are strings)
return the list

See the code (full source at github.com/HugoDF/pocket-newsletter-lambda)

async function fetchBookmarks(consumerKey, accessToken) {
  const res = await axios.post('https://getpocket.com/v3/get', {
    consumer_key: consumerKey,
    access_token: accessToken,
    tag: 'newsletter',
    state: 'all',
    detailType: 'complete'
  });
  const {list} = res.data;
  // List is a key-value timestamp->entry map
  const entries = Object.values(list);
  return entries.map(
    ({
      given_title,
      given_url,
      resolved_url,
      resolved_title,
      excerpt,
      authors,
      rest
    }) => ({
      ...rest,
      title: given_title || resolved_title,
      url: given_url || resolved_url,
      excerpt,
      authors: authors
        ? Object.values(authors)
            .map(({name}) => name)
            .filter(Boolean)
            .join(',')
        : ''
    })
  );
}

Netlify Lambda request validation and body-parsing

HTTP verb and payload presence validation

The lambda only supports POST with a body, hence:

if (event.httpMethod !== 'POST') {
  return {
    statusCode: 404,
    body: 'Not Found'
  };
}

if (!event.body) {
  return {
    statusCode: 400,
    body: 'Bad Request'
  };
}

Parsing POST request bodies from form submissions and AJAX/JSON requests

We support both URL-encoded form POST requests (done eg. when JS is disabled on the demo page) and JSON requests.

The body arrives either base64 encoded (if using a URL-encoded form body request) or not. This is denoted by the isBase64Encoded flag on the event.

Parsing a base64-encoded string in Node is done using Buffer.from(event.body, 'base64').toString('utf-8).

To convert the body from URL-encoded form into an object, the following function is used, which works for POSTs with simple fields.

function parseUrlEncoded(urlEncodedString) {
  const keyValuePairs = urlEncodedString.split('&');
  return keyValuePairs.reduce((acc, kvPairString) => {
    const [k, v] = kvPairString.split('=');
    acc[k] = v;
    return acc;
  }, {});
}

Here's the functionality in the lambda:

const {
    pocket_consumer_key: pocketConsumerKey,
    pocket_access_token: pocketAccessToken
  } = event.isBase64Encoded
    ? parseUrlEncoded(Buffer.from(event.body, 'base64').toString('utf-8'))
    : JSON.parse(event.body);

If the consumer key or access token are missing we send a 400:

if (!pocketConsumerKey || !pocketAccessToken) {
  return {
    statusCode: 400,
    body: 'Bad Request'
  };
}

Sending appropriate responses on failure

We attempt to fetchBookmarks, this functionality has been broken down in “Pocket Fetching logic”.

When the Pocket API fails on a request error we want to send back that response's information to the client. If the failure can't be identified, 500. When fetchBookmarks succeeds send a 200 with data.

Thankfully when axios fails, it has a response property on the error. This means that our “Proxy back Pocket API errors” use-case and the other 2 cases are easily fulfilled with:

try {
  const bookmarks = await fetchBookmarks(pocketConsumerKey, pocketAccessToken);

  return {
    statusCode: 200,
    body: JSON.stringify(bookmarks)
  };
} catch(e) {
  if (e.response) {
    return {
      statusCode: e.response.statusCode,
      body: `Error while connecting to Pocket API: ${e.response.statusText}`
    }
  }
  return {
    statusCode: 500,
    body: e.message
  }
}

Sample response

See github.com/HugoDF/pocket-newsletter-lambda#sample-response or try out the application yourself at pocket-newsletter-lambda.netlify.com/.

A Pocket-driven newsletter in the real-world

On codewithhugo.com, the lambda doesn't read the access token and consumer key from the request. Instead it's a GET endpoints that reads the token and key from environment variables.

It's actually simpler to do that. We set POCKET_CONSUMER_KEY and POCKET_ACCESS_TOKEN in the Netlify build configuration. Then update the lambda to the following:

const {POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKEN} = process.env;

// keep fetchBookmarks as is

export async function handler(event) {
  if (event.httpMethod !== 'GET') {
    return {
      statusCode: 404,
      body: 'Not Found'
    };
  }

  const bookmarks = await fetchBookmarks(POCKET_CONSUMER_KEY, POCKET_ACCESS_TOKEN);
  return {
    statusCode: 200,
    body: JSON.stringify(bookmarks)
  };
}

codewithhugo.com runs on Hugo. To generate a new newsletter file, I leverage Hugo custom archetypes (ie. a content type that has a template to generate it).

The archetype looks something like the following:

{{- $title := replace (replaceRE `[0-9]{4}-[0-9]{2}-[0-9]{2}-` "" .Name) "-" " " | title -}}
---
title: {{ $title }} - Code with Hugo
---

{{- $newsletterBookmarks := getJSON "https://codewithhugo.com/.netlify/functions/newsletter" }}
{{ range $newsletterBookmarks }}
[{{ .title }}]({{.url}}) by {{ .authors }}: {{ .excerpt }}
{{ end }}

Once the newsletter has been generated and edited, it gets added to buttondown.email, you can see the result of this approach at the Enterprise Node.js and JavaScript Archives.

unsplash-logo
Mr Cup / Fabien Barral