Bring Push Notification to Diablo Clone with Netlify CMS, Gatsbyjs and Fauna Database

I have always been a fans of Diablo and have been playing the Diablo II Expansion on and off for years. One thing in the Diablo II, Diablo Clone(DC), is definitely the most exciting event in the world of Diablo. I cannot stress enough that how frustrated I was whenever I missed a hunting event.

So.....what could I do for this? huh.....why not make an tiny application for DC notification? How hard could it be? It must be very easy. Let's do it😁

Let's see the result first 😁

and how we add the PWA website as an app-like onto mobile desktop

In this article, I will demonstrate:

1. How to make a Windows Service with C# .Net Core
2. How to use Netlify Lambda function (Powered by AWS Lambda. Simplified by Netlify) as a server-less back-end
3. Convert a Gatsbyjs website to Progressive Web App(PWA)
4. How to integrate Push Notification into the PWA
5. Use Fauna Database to store push subscriptions.

Notice: this article is not a step by step tutorial on topics e.g. converting website to PWA. Those topics have been beautifully written in a succinct approach by others. Instead I will provide my thoughts and helpful references for you to glue the parts together and complete this project.

Initially, I was thinking of flashing a LED light when the DC event is happening and playing a music for selling Stone of Jordan and a different music for Diablo walks the earth. Then setting up my old Arduino components exhausted all my interests on it plus I won't be able to get notified when I'm not at home, so I dropped the LED light idea.

Then I turned around and had a look at what I had: a Gatsby + Netlify CMS website. Interestingly, I even found that the Gatsbyjs Netlify CMS template that I'm using supports Netlify CMS Function~!😍 Then this opens a door for me to use the lambda function as a server-less back-end so that I could do more interesting things.

So this is how the project works:

👉when events of selling Stone of Jordan and Diablo walks the earth happen, a file located under a folder let's say: d:\games\diablo2\dc.txt will be updated with the room name, game server ip, my character's name and the date and time will be recorded inside this file.

👉a Windows Service keeps track of the changes of this file and will notify the server-less Netlify Lambda function

👉the Netlify Lambda function then push out notifications to all related subscribers

So how does this all fit together? Okay, let's have a look what the project look like:

Project Architecture

Section 1: Windows Service

For the Windows Service project, its main task is monitoring the dc.txt file and notify the Netlify Function(NF) once the content of this file is changed.

Why makes it a Windows Service?

Because I would like to have a service running in background and continuously keeps tracking of the file and informs the NF once the events happened. In addition, I am lazy, I do not want to manually run this project every time myself, instead once my laptop started, run the service automatically.

Okay, enough talky talk, let's rock and roll:

Open Visual Studio 2019 and create a Worker Service project

Install these 2 packages:

Microsoft.Extensions.Hosting.WindowsServices

Microsoft.Extensions.Http

Then open the Program.cs, make it look like this:

   public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureServices((hostContext, services) =>
                {
                    services.AddConfiguration<DCMonitorConfiguration>(hostContext.Configuration, "DCMonitor");
                    services.AddHttpClient();
                    services.AddHostedService<Worker>();
                })
                .UseWindowsService();

services.AddConfiguration<DCMonitorConfiguration>(hostContext.Configuration, "DCMonitor"); is not necessary if you don't mind a little bit hard-coding stuff in a personal project.

Now open the Worker.cs, change it to something like this:

public class Worker : BackgroundService
    {
        private readonly ILogger<Worker> _logger;
        private readonly string TargetFile = @"d:\games\Diablo2\dc.txt";
        private readonly string PushEndpoint = "https://yourwebsite.com/.netlify/functions/";
        private readonly string NotifyEmail = "[e-mail]";
        private readonly HttpClient _httpClient;
        private FileSystemWatcher _fileWatcher;

        public Worker(ILogger<Worker> logger, IHttpClientFactory httpClientFactory)
        {
            _logger = logger;

            _httpClient = httpClientFactory.CreateClient();
            _httpClient.DefaultRequestHeaders.Add("User-Agent", "DCMonitorSvc");
            _httpClient.BaseAddress = new Uri(PushEndpoint);
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            try
            {
                _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
                await Task.Delay(1000, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError($"Worker error at {DateTime.Now}{Environment.NewLine}{ex.Message}");
            }
        }

        public override Task StartAsync(CancellationToken cancellationToken)
        {
            try
            {
                //System.Diagnostics.Debugger.Launch();

                _fileWatcher = SetupFileWatcher(TargetFile);
                return base.StartAsync(cancellationToken);
            }
            catch (Exception ex)
            {
                _logger.LogError($"Worker error at {DateTime.Now}{Environment.NewLine}{ex.Message}");
                return Task.FromResult(ex);
            }

        }

        public override Task StopAsync(CancellationToken cancellationToken)
        {
            _fileWatcher.EnableRaisingEvents = false;
            _fileWatcher.Dispose();
            return base.StopAsync(cancellationToken);
        }

        private FileSystemWatcher SetupFileWatcher(string targetFile)
        {
            var watcher = new FileSystemWatcher(targetFile, "*.txt");
            watcher.NotifyFilter = NotifyFilters.LastAccess
                                             | NotifyFilters.LastWrite
                                             | NotifyFilters.FileName
                                             | NotifyFilters.CreationTime
                                             | NotifyFilters.DirectoryName;

            // Add event handlers.
            watcher.Changed += OnChanged;

            // Begin watching.
            watcher.EnableRaisingEvents = true;
            return watcher;
        }

        private async void OnChanged(object source, FileSystemEventArgs e)
        {
            // Specify what is done when a file is changed, created, or deleted.
            Console.WriteLine($"File: {e.FullPath} {e.ChangeType} at {DateTime.Now}");

            if (e.ChangeType == WatcherChangeTypes.Changed)
            {
                try
                {
                    var fileContent = File.ReadAllText(e.FullPath);
                    var parts = fileContent.Split('|');

                    // 2020-08-08 16:59:00|AcA|aca-123|398120|112
                    if (!string.IsNullOrWhiteSpace(fileContent))
                    {
                        await Alert(fileContent);
                    }
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"File is in use.{Environment.NewLine}{ex.Message}");
                }

            }
        }

        private async Task Alert(string notificationContent)
        {
            string url = $"alert?dc={notificationContent}&email={NotifyEmail}";
            var response = await _httpClient.GetAsync(url);
            response.EnsureSuccessStatusCode();
        }
    }

So what I have done here is:

Defining the StartAsync(), StopAsync() and Execute() functions as they map to start/stop the Windows Service we are developing. Most importantly, we create a FileWatcher which keeps tracking of the file(dc.txt) under the game folder and sends out notifications by calling an endpoint which I will cover in the next section.

Once the project is built, register this Windows Service in a command line like this:

# Register your Windows Service
> sc.exe create DCMonitor binPath="to-your-windows-service.exe"

# Delete your Windows Service
> sc.exe delete DCMonitor

Then open press Win button and type Service to open the Services panel and start your very own Windows Service:

You can make it automatic so that it starts once PC starts.

Go to Task Manager, you should be able to find your Windows Service under background processes

......

Section 2: Progressive Web App + Push Notification

In this section, I will provide references on how to set up a Gatsby.js + Netlify CMS website and add support to Service Worker so that we can turn the website into a Progressive Web App(PWA). Then I will set up Push Notification so that the PWA will be able to notify users when the delicious Diablo Clone events happens.

I use this gatsby-starter-netlify-cms template to start building my personal blog website. The reason is very simple, with a few steps of configuration, I am able to have a fully functional blog website. It also has the real-time preview functionality which enables me to see how my blog looks like when I am still in writing.

What is more interesting is, I can even use the Blog editor to add html components on a page on the fly! This means for simple UI changes, I do not even need any IDEs, simply just open my website's admin panel and write a few lines of code, then clicks Publish, the new html components will be immediately added onto my website. How cool it is~! 🤩

To give you a bit of idea about how it looks like:

PWA

Adding Service Worker and convert our normal website to PWA has been made incredibly easy. Digital Ocean has written a great article which helps me a lot on converting my Gatsby website to PWA, I will not repeat what has been written there, so follow the tutorial here:

Making Gatsby a PWA: Service Worker and Web App Manifest.

One thing to notice here, in order for the web app manifest to be cached, we’ll need to list gatsby-plugin-manifestBEFORE gatsby-plugin-offline

Push Notification

Google has written its fantastic tutorials which I found particular useful and easy to follow:

Developing Progressive Web Apps 08.0: Integrating web push

Notes: on section 4, the old approach doesn't work anymore, so don't stop there, go ahead use the VAPID approach, everything will be fine 😉

When talking about push notification, there are 2 actions: Push and Notifying. Once I speak it out, it seems obvious, but back then I was not aware of these. So here is my learning notes:

The Push events happen at server side, the Notifying part happens at client side.

So let's do the client side part. In the project under the staticfolder add a file helpers.js:

async function askForPermission() {
  console.log("ask for permission");
  if (!("Notification" in window)) {
    console.log("This browser does not support notifications!");
    throw new Error("This browser does not support notifications!");
  }

  const status = await Notification.requestPermission();
  console.log("Notification permission status:", status);
  return status === "granted";
}

function urlB64ToUint8Array() {
  const applicationServerPublicKey = "";

  const padding = "=".repeat((4 - (applicationServerPublicKey.length % 4)) % 4);
  const base64 = (applicationServerPublicKey + padding)
    .replace(/\-/g, "+")
    .replace(/_/g, "/");

  const rawData = window.atob(base64);
  const outputArray = new Uint8Array(rawData.length);

  for (let i = 0; i < rawData.length; ++i) {
    outputArray[i] = rawData.charCodeAt(i);
  }
  return outputArray;
}

// Subscribe user is actually subscribing push registration
async function subscribeUser(swRegistration) {
  const applicationServerKey = urlB64ToUint8Array();

  try {
    const subscription = await swRegistration.pushManager.subscribe({
      userVisibleOnly: true,
      applicationServerKey: applicationServerKey,
    });

    console.log("User has subscribed successfully");
    return subscription;

  } catch (subscribeError) {

    if (Notification.permission === "denied") {
      console.warn("Permission for notifications was denied");
    } else {
      console.error("Failed to subscribe the user: ", subscribeError);
    }

    return null;
  }
}

async function getSubscription() {
  try {
    const swRegistration = await navigator.serviceWorker.ready;
    let pushSubscription = await swRegistration.pushManager.getSubscription();

    // if not found from pushManager, then we subscribe the user right now
    if(!pushSubscription) { 
      pushSubscription = await subscribeUser(swRegistration)
    }

    pushSubscription = pushSubscription.toJSON();
    document.getElementById("subkey").textContent = pushSubscription.keys.auth;
    return pushSubscription;
  } catch (error) {
    console.log("getSubscription() error: ", error);
    return null;
  }
}

function validateEmail(email) {
  const re = /^(([^<>()[\]\\.,;:\s@\"]+(\.[^<>()[\]\\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
  const valid = re.test(email);
  return valid;
}

async function updateSubscriptionOnServer(pushSubscriptionObject) {
  let url = "";
  try {
    url = "https://yourwebsite.netlify.app/.netlify/functions/updateSubscription";

    await fetch(url, {
      method: "POST", // *GET, POST, PUT, DELETE, etc.
      mode: "no-cors", // no-cors, *cors, cors, same-origin
      cache: "no-cache", // *default, no-cache, reload, force-cache, only-if-cached
      credentials: "same-origin", // include, *same-origin, omit
      headers: {
        "Content-Type": "application/json;charset=utf-8",
      },
      redirect: "follow", // manual, *follow, error
      referrerPolicy: "no-referrer", // no-referrer, *no-referrer-when-downgrade, origin, origin-when-cross-origin, same-origin, strict-origin, strict-origin-when-cross-origin, unsafe-url
      body: JSON.stringify(pushSubscriptionObject), // body data type must match "Content-Type" header
    });

    return true;

  } catch (ex) {
    console.log("saveReg() catch block:", ex);
    return false;
  }
}

async function updateSubscription() {
  try {
    const allowed = await askForPermission();
    if (!allowed) return;

    let subscription = await getSubscription();
    if (!subscription) return;

    // email
    const email = getEmail();
    if (!email || !validateEmail(email)) {
      alert("huh...so how are you going to receive notifications?");
      return;
    }

    let extra = {
        email: email
    };

    subscription.extra = extra;
    const successful = await updateSubscriptionOnServer(subscription);

    if (successful) alert("you have successfully subscribed to the DC monitor");
    else alert("shit happens, try it later");
  } catch (err) {
    console.log("updateSubscription() failed: ", err);
  }
};

function getEmail() {
  console.log("getEmail()");
  let email = document.getElementById("email").value;

  const pathname = window.location.pathname;
  console.log("the current location is: ", pathname);
  if (pathname.indexOf("/about") >= 0) {
    email = document.getElementById("email").value;
  }

  if (localStorage) {
    if(!email) {
      email = localStorage.getItem("dc_email");
    } else {
      localStorage.setItem("dc_email", email);
    }
  }

  if (email) {
    document.getElementById("email").value = email;
  }

  console.log("getEmail(): ", email);
  return email;
};

So what we do here is:

1. we need to ask user's permission to do push notification.
2. get push subscription for this user from pushManager, if this user hasn't subscribed, then subscribe the user
3. patch the email onto the push subscription object, so that later we can target specific users for different notifications.
4. finally we save the updated push notification object into Fauna Database

In order to make the Javascript code available to the html components we added from the Blog editor, I have to put the script under static folder as all files under this folder will be kept as is. If I put the helpers.js under src folder, then the functions will be uglified then the function names will be simplified and I won't be able to reference them from the html code. Ideally, I would like to hide the public and private keys from the code and use Environment variables, but I have not figured it out. So please let me know if you are sure you know how to import the env variables for files under static folder. I have tried the dotenv, but had no luck.

......

Section 3: Netlify Functions

The features of Netlify Functions(NF) fits well as our server-less back-end. The idea is: our Windows Service calls the serverless functions and the functions will then notify the subscribed users.

Let's open lambda folder and add a file called: dcalert.js

const webPush = require("web-push");
const { getSubscriptions, removeSubscription, getResponse, getSubscriptionsByEmail } = require("./utils/utils.js");

const vapidPublicKey = process.env.VAPID_PUBLIC_KEY;
const vapidPrivateKey = process.env.VAPID_PRIVATE_KEY;
const notificationOptions = {
  TTL: 60,
  // TODO 4.3b - add VAPID details
  vapidDetails: {
    subject: `mailto:${process.env.VAPID_EMAIL}`,
    publicKey: vapidPublicKey,
    privateKey: vapidPrivateKey,
  },
};

async function notifySubscriber(pushSubscription, dc) {
  try {
    const response = await webPush.sendNotification(pushSubscription, dc, notificationOptions);
    if (response.statusCode % 200 <= 1) {
      console.log(`notification successfully sent at ${(new Date()).toString()}`, response);
      return true;
    }
    else {
      console.log(`notification sent at ${(new Date()).toString()} with status code ${response.statusCode}`, response);
      console.log("error: ", err);
      return  false;
    }
  }
  catch (ex) {

    if (ex.statusCode === 410) { // the subscription has expired or gone, then we remove the subscription from our database
      const auth = pushSubscription.keys.auth;
      const deleted = await removeSubscription(auth);
      if (deleted) {
        console.info(`subscription ${auth} has expired or gone, removed from database.`);
        return false;
      }
      else {
        console.error(`failed to remove expired subscription ${auth}`);
        return false;
      }
    }
    else {
      console.log("error: ", ex);
      return false;
    }
  }
}

function validateRequest(event) {
  const userAgent = event.headers["user-agent"];
  const dc = event.queryStringParameters.dc;

  if (!userAgent || !dc)
    return false;

  return true;
}

module.exports.handler = async function(event, context) {
  if (!validateRequest(event))
    return getResponse(400, "bad request");

  const userAgent = event.headers["user-agent"];
  const dc = event.queryStringParameters.dc;
  const email = event.queryStringParameters.email;

  if(!dc) {
    return getResponse(400, 'notification content cannot be empty');
  }

  let subscriptions = [];
  if (!email) {
    subscriptions = await getSubscriptions();
  } else {
    subscriptions = await getSubscriptionsByEmail(email);
  }

  if (!subscriptions) {
    return getResponse(200, 'No subscriber to notify');
  }

  let successfulCounter = 0;

  for (let pushSubscription of subscriptions) {
    const successful = await notifySubscriber(pushSubscription, dc);
    successfulCounter += successful ? 1 : 0;
  }

  if (successfulCounter === subscriptions.length) {
    return getResponse(200, `notification has been sent to all ${subscriptions.length} clients`);
  } else if (successfulCounter === 0) {
    return getResponse(500, `sending notification has failed`);
  } else {
    return getResponse(200, `notification has been sent to ${successfulCounter} out of ${subscriptions.length} clients`);
  }
};

What happens here is, once a Http request hits this api endpoint, it does

1. validation against the request.
2. get subscriptions according to whether we passed in an email or not, via setting the email, we will be able to control sending notifications to all our subscribers or just a particular user.
3. iterate through the subscriptions and send out notifications.

One thing I'd like to point out is, push subscriptions could expire at some time, so we need to clean up those expired subscriptions. How do I handle this is when sending out notification, it will error out if the subscription has gone. I then use it as a chance to clean our the expired subscription. So take a look at catchsection of notifySubscriber().

tl;dr

Also I was thinking that in order to save time storing subscriptions into cache on the server with the lambda functions would be okay to me, then I was stuck at there for many hours.....puffy eyes, sleepless night....😫.

The reason why it doesn't work is each lambda function doesn't share cache, so a subscription created by endpoint saveSubscription() will not be accessible by another function.

In general we need to think of each run of a Netlify Function (or any AWS Lambda function) as being completely stateless. This includes the in-memory data of the running Node.js process. We can put things in memory and read them out of memory freely but only within the context of that single run. So thesaveSubscriptions()may work, but that memory will be erased quickly after that run is over.

Then how about saving them into a text file......not working.... Once again, puffy eyes, sleepless night.....😫

The reason is the file system on Lambda function server is read only.

This pushed me to the final solution: Fauna Database. It has free tier and recommended by Netlify, plus I always wanted to try this kind of NoSql like, non-structural database.

Look back what I have done so far, I have basically tried the JAM Stack with all these fantastic tools, it doesn't harm to add in one more fancy component into my project🤗🤗

......

Section 4: Fauna Database

Go to Fauna website and create an account, then log in it, you will see something like this:

Shell is your friend to quickly try out queries and we can easily convert the FQL query to Javascript code.

This is how the shell looks like:

I have a utils.js file which does all the database CRUD operations. Found out how to do it in Javscript from the Netlify-Faunadb-example Github repository.

Create abootstrap.js script under scripts folder:

/* bootstrap database in your FaunaDB account */
const faunadb = require('faunadb')
const chalk = require('chalk')
const insideNetlify = insideNetlifyBuildContext()
const q = faunadb.query

console.log(chalk.cyan('Creating your FaunaDB Database...\n'))

// 1. Check for required enviroment variables
if (!process.env.FAUNADB_SERVER_SECRET) {
  console.log(chalk.yellow('Required FAUNADB_SERVER_SECRET enviroment variable not found.'))
  console.log(`Make sure you have created your Fauna databse with "netlify addons:create fauna"`)
  console.log(`Then run "npm run bootstrap" to setup your database schema`)
  if (insideNetlify) {
    process.exit(1)
  }
}

// Has var. Do the thing
if (process.env.FAUNADB_SERVER_SECRET) {
  createFaunaDB(process.env.FAUNADB_SERVER_SECRET).then(() => {
    console.log('Fauna Database schema has been created')
    console.log('Claim your fauna database with "netlify addons:auth fauna"')
  })
}

/* idempotent operation */
function createFaunaDB(key) {
  console.log('Create the fauna database schema!')
  const client = new faunadb.Client({
    secret: key
  })

  /* Based on your requirements, change the schema here */
  return client.query(q.Create(q.Ref('classes'), { name: '<your-lovely-database>' }))
    .then(() => {
      return client.query(
        q.Create(q.Ref('indexes'), {
          name: 'all_subs',
          source: q.Ref('classes/<your-lovely-database>')
        }))
    }).catch((e) => {
      // Database already exists
      if (e.requestResult.statusCode === 400 && e.message === 'instance not unique') {
        console.log('Fauna already setup! Good to go')
        console.log('Claim your fauna database with "netlify addons:auth fauna"')
        throw e
      }
    })
}

/* util methods */

// Test if inside netlify build context
function insideNetlifyBuildContext() {
  if (process.env.DEPLOY_PRIME_URL) {
    return true
  }
  return false
}

The above script creates the database for you, just replace the database name and also add FAUNADB_SERVER_SECRET into your .env file like this:

FAUNADB_SERVER_SECRET=your-precious-secret-key

Also you could set up your environment variables on Netlify directly:

Then add a command script into package.json to run the bootstrap script:

"bootstrap":"netlify dev:exec node ./scripts/bootstrap-fauna-database.js"

then run

npm run bootstrap

Your Fauna Database is now ready to be consumed.

One thing needs to mention is: all searches with Fauna DB are done by indexes. So that means if I would like to search push subscriptions by email, then I need to create an index for it.

To give you an idea about how the FQL query looks like and how the converted Javascript code look like:

# FQL Query
Map(
  Paginate(Match(Index("search_by_email"), "[e-mail]")),
  Lambda("sub", Select(["data"], Get(Var("sub"))))
)

# Javascript code
const faunadb = require("faunadb");
const q = faunadb.query;

const client = new faunadb.Client({
  secret: process.env.FAUNADB_SERVER_SECRET,
});
 const response = await client.query(
        q.Map(
          q.Paginate(
            q.Match(q.Index("search_by_email"), param)
          ),
          q.Lambda(
            "sub",
            q.Select(["data"], q.Get(q.Var("sub")))
          )
        )
      );

Now you should have a rough idea about how the whole project works and how these components work together to achieve the instant push notification.

Looking back at the very beginning of this project, the whole idea started as a small push notification for Diablo Clone, I had never thought that this project could expand to across so many technical stacks and took me so many sleepless nights and efforts to complete.

Finally, I hope you enjoy this article and make your very own PWA with Push notification enabled 🤩.