Uptime Monitoring with Firebase

In this second step of the series, I assume that the Firebase project has already been initialized and is ready. Otherwise I recommend to work through the steps from the first part.

The database

I usually like to start with the database design. This helps a lot to have a structured understanding of the application.
Firestore is a NoSQL database that stores data in documents inside collections. One very cool feature is that you can also create subcollections for expressing hierarchical data structures.

In my Project I basically need two Datatypes. The UptimeEntries and the UptimeRequests. An UptimeEntry will be created when we check the site and it will then have one (if the first request is ok) or multiple (if the first request fails) UptimeRequests.

Both have a pretty strict structure defined in TypeScript interfaces:

export interface UptimeRequest {
  id?: string;
  url: string;
  ok: boolean;
  statusCode: number;
  duration: number;
  started: firebaseAdmin.firestore.Timestamp;
  ended: firebaseAdmin.firestore.Timestamp;
}

export interface UptimeEntry {
  id?: string;
  url: string;
  initialResponseOk: boolean;
  responseOk: boolean;
  downtimeMillis: number;
  created: firebaseAdmin.firestore.Timestamp;
  latestCheck: firebaseAdmin.firestore.Timestamp;
}

The idea is now that I will have two collections. The first will just be "entries", a list of UptimeEntry and the second will be a subcollection with the path "entries/{entryId}/requests". This means that one UptimeEntry can have multiple UptimeRequests.

I really like to abstract things away. So for all the database communication I created just one class with a handfull of methods:

import * as firebaseAdmin from "firebase-admin";
firebaseAdmin.initializeApp();

const firestore = firebaseAdmin.firestore();

class Firestore {
  collectionEntries = () => {
    // returns the "entries" collection reference
  };

  collectionRequests = (entryId: string) => {
    // returns the "entries/{entryId}/requests" collection reference
  };

  createEntry = async (data: UptimeEntry) => {
    // creates a new UptimeEntry
  };

  getAllEntries = async () => {
    // returns all UptimeEntries
  };

  getEntry = async (entryId: string) => {
    // returns an UptimeEntry by ID
  };

  update = async (entryId: string, entry: Partial<UptimeEntry>) => {
    // updates an UptimeEntry by ID
  };

  getLatestEntry = async () => {
    // returns the UptimeEntry
  };

  addRequest = async (entryId: string, request: UptimeRequest) => {
    // adds an UptimeRequest to an UptimeEntry
  };
}

The exact implementation of the class can be found here: https://github.com/nico-martin/uptime-slackbot/blob/main/functions/src/utils/Firestore.ts

Check the status

Now that I have my database adapter I can finally start with the monitoring.

Here I actually need two functionalities of Firebase cloud functions. First, I want to periodically (every 5 minutes) check the status of my website.
Second, if a request fails, I want to retry the request until the site is back online.

So my first function should run in a 5 minute interval. Here we can use the functions.pubsub.schedule Function:

const scheduleUptime = functions.pubsub
  .schedule("every 5 minutes")
  .onRun(async () => {
    // ..
  });

export default scheduleUptime;

Inside the function we are going through a couple of steps:

1. we need to make sure that there is not already a failed request/retry ongoing. So we will get the latest entry from the DB. If there is a latest entry and the latest entry is still not ok, don't need to continue (because there already is an ongoing request/retry).
2. after that we will run the request to the URL, if it is not ok, we know that we have downtime
3. After that we will add our Entry to the DB and we can also assign the Request to the Entry

const scheduleUptime = functions.pubsub
  .schedule("every 5 minutes")
  .onRun(async () => {
    const latest = await db.getLatestEntry();

    if (latest && !latest.responseOk) {
      return;
    }
    const check = await createRequest();

    if (!check.ok) {
      functions.logger.log(
        `Uptime Monitor is DOWN: ${check.url} - StatusCode: ${check.statusCode}`
      );
    }

    const createdId = await db.createEntry({
      url: check.url,
      initialResponseOk: check.ok,
      responseOk: check.ok,
      created: firebaseAdmin.firestore.Timestamp.now(),
      latestCheck: firebaseAdmin.firestore.Timestamp.now(),
      downtimeMillis: 0,
    });

    await db.addRequest(createdId, check);
    return;
  });

https://github.com/nico-martin/uptime-slackbot/blob/main/functions/src/scheduleUptime.ts

Recheck if downtime detected

So now we know when our site is down. But what is missing is an indicator when our site is available again. I have tried different ideas back and forth. The following makes the most sense from my point of view.

const requestOnWrite = functions.firestore
  .document("uptime/{uptimeId}/requests/{requestId}")
  .onCreate(async (requestSnapshot, context) => {
    // ...
  });

export default requestOnWrite;

In this function we now have several options.

1. if the status of the request is ok and also the initial request was ok, we don't have to do anything.
2. if the status of the request is ok we know that we are coming from a downtime and the page is now online again. This means that we can update the entry accordingly and log our message.
3. if the status of the request is not ok we are still in a downtime and after a certain time we can start a new attempt.

const requestOnWrite = functions.firestore
  .document("uptime/{uptimeId}/requests/{requestId}")
  .onCreate(async (requestSnapshot, context) => {
    const uptimeEntry = await db.getEntry(context.params.uptimeId);
    const request = requestSnapshot.data() as UptimeRequest;
    if (request.ok && uptimeEntry.initialResponseOk) {
      // is first request of a successful uptime check
    } else if (request.ok) {
      // request successfull after retry
      uptimeEntry.latestCheck = request.started;
      const downtimeMillis = request.started.toMillis() - uptimeEntry.created.toMillis();
      uptimeEntry.responseOk = true;
      uptimeEntry.downtimeMillis = downtimeMillis;
      await db.update(context.params.uptimeId, uptimeEntry);
      functions.logger.log(`Uptime Monitor is UP: ${request.url}. It was down for ${formatSeconds(Math.round(downtimeMillis / 1000))}.`);
    } else {
      // request failed, create new request after 2 sec
      setTimeout(async () => {
        const check = await createRequest();
        await db.addRequest(uptimeEntry.id, check);
      }, 2000);
    }

    return;
  });

export default requestOnWrite;

https://github.com/nico-martin/uptime-slackbot/blob/main/functions/src/requestOnWrite.ts

With this setup we are logging when our site is down and also when it is back up again. Please check the full source code on GitHub since I am also using some helper functions from functions/src/utils/helpers.ts:
https://github.com/nico-martin/uptime-slackbot/blob/main/functions/

Once your functions are done you can export them in your functions/src/index.ts:

export { default as scheduleUptime } from "./scheduleUptime";
export { default as requestOnWrite } from "./requestOnWrite";

And with that you are now ready to deploy your functions:

npx firebase deploy --only functions

Let's get ready for the last step where we create and implement our Slackbot.