DEV Community: maarteNNNN

SocketCluster. The most underrated framework. Part 3: A Pub/Sub example and middleware

maarteNNNN — Thu, 15 Apr 2021 16:41:22 +0000

maarteNNNN / sc-underrated-framework-pubsub

SocketCluster. The most underrated framework. Part 3: A Pub/Sub example and middleware

Introduction

In this part we will make a simple chat example to understand how Pub/Sub works in SocketCluster. The app can be tested across multiple browser windows. We will add some simple middlewares. A chat history and censorship for bad-words.

Setup

Let's setup a blank project by running socketcluster create sc-pubsub and cd sc-pubsub. Let's install nodemon to restart the server automatically npm i -D nodemon. And for our bad-words censorship we will use a package called bad-words from NPM. npm i -s bad-words. The server can be run with npm run start:watch.

Client code setup (don't give much attention to this, just copy and paste)

We will use vanilla JavaScript in HTML like part 2 shipped with SocketCluster in public/index.html. Let's delete everything inside the style tag and replace it with:

* {
  margin: 0;
  padding: 0;
}

html {
  height: 100vh;
  width: 100vw;
}

.container {
  height: 100vh;
  display: flex;
  align-items: center;
  justify-content: center;
  flex-direction: column;
}

.chat-history {
  height: 70vh;
  width: 75%;
  border: 1px solid #000;
  display: flex;
  flex-direction: column;
  overflow-y: auto;
}

.chat-input {
  width: 75%;
  height: 5vh;
  border-left: 1px solid #000;
  border-bottom: 1px solid #000;
  border-right: 1px solid #000;
}

input {
  box-sizing: border-box;
  width: 100%;
  height: 100%;
  border: none;
  padding: 0 1em;
}

strong,
small {
  font-size: 11px;
  color: gray;
}

.message {
  padding: 0.25rem 1rem;
}

and delete eveything inside <div class="container"> tag and replace it with:

<div id="chat-history" class="chat-history"></div>
<div class="chat-input">
  <input placeholder="message" onkeyup="sendMessage(event)" />
</div>

Okay. Now we have a basic chat page. Nothing too fancy. Now we can focus on getting actual logic of our chat application.

The Pub/Sub functionality

Client

Pub/Sub in SocketCluster is something that can work without writing any backend logic. We can create a channel on the client and the server makes this channel available for other clients.

(async () => {
  for await (const data of socket.subscribe('chat')) {
    console.log(data);
  }
})();

and we should create the function that listens to the enter key on the input to send the publish the message.

const sendMessage = async (event) => {
  if (event.keyCode === 13) {
    try {
      await socket.transmitPublish('chat', {
        timestamp: Date.now(),
        message: event.target.value,
        socketId: socket.id,
      });
      event.target.value = '';
    } catch (e) {
      console.error(e);
    }
  }
};

The transmitPublish method does not suspect a return value. If you do want a response you can look at invokePublish.

The transmitPublish sends an object with a timestamp, message and the socketId. The socket.subscribe('chat') async iterable will log any new data being pushed. Open two browser windows next to each other and open the Developer Tools in both windows. If you send a message in one window it should output it in both consoles.

We will display the messages in the #chat-history div by creating a function that creates an element, changes the text, adds a class and appends the element.

const createMessage = ({ socketId, timestamp, message }) => {
  const chatHistoryElement = document.getElementById('chat-history');
  const messageElement = document.createElement('div');
  messageElement.className = 'message';
  messageElement.innerHTML = `<strong>${socketId}</strong> <small>${timestamp}:</small> ${message}`;

  chatHistoryElement.appendChild(messageElement);

  // Always scroll to the bottom
  chatHistoryElement.scrollTop = chatHistoryElement.scrollHeight
};

change the previous console.log(data) inside the socket.subscribe('chat') to createMessage(data).

Now if we send messages it should display them in the HTML instead of the Developer Tools. Pretty neat, huh? Upon to this point we still didn't do any server-side code.

Server-side

There's only one problem with our app. Every new window does not have any older messages. This is where the server comes in. We will create a middleware that pushes every message to an array, for simplicity sake. Another thing the middleware will pick-up is bad-words. We can filter them and replace the characters with a *.

const Filter = require('bad-words');
const filter = new Filter();

...

const history = []

agServer.setMiddleware(
  agServer.MIDDLEWARE_INBOUND,
  async (middlewareStream) => {
    for await (const action of middlewareStream) {
      if (action.type === action.PUBLISH_IN) {
        try {
          // Censor the message
          action.data.message = filter.clean(action.data.message);
        } catch (e) {
          console.error(e.message);
        }
        // Push to the array for history
        history.push(action.data);
      }
      // Allow the action
      action.allow();
    }
  },
);

...

We set an inbound middleware, we pass it an async iterable stream. On every action of the stream we check if the action.type equals the constant provided by SC action.PUBLISH_IN. If the conditional is true we filter the message and allow the action. Alternatively we could action.block() the action if we don't want it to go through. More on middleware here

To implement the history it's pretty simple, we just create a constant const history = [] and push every action.data to it. Like shown in the above code.

To initially get the history we transmit the data upon a socket connection (e.g. a new browser window).

(async () => {
  for await (let { socket } of agServer.listener('connection')) {
    await socket.transmit('history', history);
  }
})();

And create a receiver on the client which uses a loop to create the messages.

(async () => {
  for await (let data of socket.receiver('history')) {
    for (let i = 0; i < data.length; i++) {
      const m = data[i];
      createMessage(m);
    }
  }
})();

I will try to add an article every two weeks.

SocketCluster. The most underrated framework. Part 2: A simple example

maarteNNNN — Tue, 30 Mar 2021 19:16:35 +0000

maarteNNNN / sc-underrated-framework-part2

A repo that represents the code from an article I wrote on dev.to

Part 2: A simple example

Recap

In Part 1 we gave an introduction to what SocketCluster is, does and which projects/companies use it.

Setting up the SocketCluster project.

NOTE
In this article I will focus solely on SocketCluster and not implement Vue yet. We will do some vanilla client js in the index.html file served by SocketCluster.

We'll need the socketcluster-cli to be able to scaffold a project. Install it by executing npm i -g socketcluster. Initiating a project can be done with socketcluster create <project-name>. Let's open our code editor and take a look at what we have. The project consists of two directories kubernetes and public and some files. kubernetes is where all the config files to deploy as a kubernetes service. We will discuss this in a later article. public is a directory served by Express and upon firing up SocketCluster it will be accessible by navigating localhost:8000. Let's fire up SocketCluster and see what we can do by running npm run start. The default port is 8000.

Let's take a look where the magic happens. The server.js file. The server.js consists of some sections, the beginning are mostly constants to environment variables. Then it creates two servers. One is the httpServer and the other is the agServer. The agServer handles the sockets. The httpServer is a HTTP server wrapped in eetase. Which basically adds a request to a queue to be able to run for await (let ... of asyncQueueIterable) { ...logic... } loop on the requests. Then comes the Express part. It defines which directory to serve and provides a /health-check route to see if the server is operational.

// HTTP request handling loop.
(async () => {
  for await (let requestData of httpServer.listener('request')) {
    expressApp.apply(null, requestData);
  }
})();

This piece of code calls every request asynchronously to the expressApp variable and passes the requestData as the first argument, basically handling the request. This is where the eetase comes in handy to queue up requests and carefully execute them one by one.

Let's create an endpoint

We will create an endpoint in the server.js first. This is the entrypoint for every socket connection. Let's add some code:

for await (let { socket } of agServer.listener('connection')) {
...
  for await (let request of socket.procedure('test')) {
    console.log(request.data);
    request.end({ message: 'Data received, thanks client!' });
  }
...
}
...

This will handle any request on the test procedure. Log the given data to the console and respond with the message Data received, thanks client!.

Now let's add some code to actually test this route on the client. Let's use the index.html in the public folder so we can do some basic testing. Add a button below the iframe block in the HTML:

...
<!-- IFRAME BLOCK -->
<button onclick="send()">Test</button>
...

And add some JavaScript logic below let socket = socketClusterClient.create();

const send = async () => {
  console.log('sending to the server...');
  const response = await socket.invoke('test', {
    message: 'This is our first message from the client to the server',
  });
  console.log(response);
};

Restart the server and go to localhost:8000, open your developer console and hit the button Test. Upon clicking you should receive the Data received, thanks client! in the browser and when going to the terminal it should show This is our first message from the client to the server.

Now you succesfully created an endpoint from the client to the server. Now you can do some cool things but let me show you that you can do it the other way around as well. Unlike REST, SocketCluster lets you handle messages both from the server to the client as well as from the client to the server. Let's make a quick example sending an interval to the client.

On the server we will add the interval of 10 seconds:

...
const send = async () => {
  console.log('sending to the server...')
  const response = await socket.invoke(
    'test',
    { message: 'This is our first message from the client to the server' },
  );
  console.log(response)
}

setInterval(async () => {
  console.log('sending data to client...');
  const data = await socket.invoke('from-server', {
    message: 'This is sent from the server to the client',
  });

  console.log(data);
}, 10000);
...

And on the client we will listen to the procedure:

...
const send = async () => {
  console.log('sending to the server...')
  const response = await socket.invoke(
    'test',
    'This is our first message from the client to the server',
  );
  console.log(response)
}

(async () => {
  for await (let request of socket.procedure('from-server')) {
    console.log(request.data);
    // return this message to the server, it could just be request.end() to terminate it
    request.end({ message: 'Thanks server, message received' });
  }
})();
...

Restart the server and refresh the browser window. You should receive the message This is sent from the server to the client every 10 seconds.

SocketCluster. The most underrated framework. Part 1: Intoduction

maarteNNNN — Tue, 30 Mar 2021 19:12:45 +0000

Part 1: An introduction to SocketCluster and its potential

What is SocketCluster?

SocketCluster is a framework which allows you to use the WebSocket protocol to transmit between its backend API and client library. The client can be used both on a backend (E.g. a Raspberry pi) or frontend application.

What you can expect of these articles

These articles should give you a good insight of how to use a scalable SocketCluster application and how to deploy it to a Cloud service. I will try to keep it beginner friendly for everyone that has knowledge in JavaScript.

What SocketCluster solves and which applications it can be used for

Unlike REST where there is a learning curve to fully understand the HTTP Protocol or GraphQL where it's needed to learn the fundamentals of schemas, SocketCluster lets you use its framework without actually knowing how sockets work. Nor does it really need to. The framework perfectly handles sockets in a way you just need to transfer data. You can any type of data e.g. objects, strings, buffers, ... It in some way resembles SocketIO. You could say its SocketIO on steroids. It comes with bulkier tools and scalability out of the box. It has prewritten Kubernetes files.

Its internals

NOTE
This is pretty technical and can be skipped.

The latest version of SocketCluster uses the for await loop which was recently introduced in JavaScript. The internals create queues that consume data asynchronously. Let's take an example where this is useful.

E.g. the EventEmitter in JavaScript might execute a callback twice. A good example is finalizing a delete operation in a SQL Database.

Projects/companies that use SC

Lisk

One of the most famous success stories is the Lisk protocol. A cryptocurrency blockchain written in JavaScript using SocketCluster as its peer-to-peer protocol. At the time of writing Lisk is ranked #99 globally according to Coin Market Cap. Here are [1][2] articles related to SocketCluster on LiskHQ's GitHub.

Redux dev tools

Another example is Redux dev tools. They use SocketCluster to remote monitoring.

HarperDB

Here you can find an article of their reasons why. The difficulties they had over SocketIO and why SocketCluster was a better fit.

What we will be building

We will build a full-fledged application which has much resemblance to the current problems in the world, the pandemic. We will build a contact tracing app which in theory should really scale. The technologies we will be using:

Vue 3 with the Composition API as the frontend
PostgreSQL as the database
Knex.js as the query builder

Additionally I'll provide a deeper insight into Vue 3's Composition API as its fairly new technology.

Google Calendar integration with NodeJS without OAuth 2.0

maarteNNNN — Mon, 01 Jun 2020 03:11:11 +0000

Google Calendar integration

Prerequisites

This was done using a G-Suite account, I'm not sure this can be done with a normal Gmail account.
a Google cloud project
NodeJS and NPM/Yarn installed
Basic knowledge of JavaScript and ES6

Introduction

A client asked me to integrate a scheduling system into her website, to facilitate her appointments. I told her it would be an expensive solution written by hand as she is only a small company. However I'm always up to learn new stuff and looked into ways to make it happen. One of the first things I thought about and is part our daily lives is Google Calendar. It's easy and robust. So I took a look at the APIs and what and/or if it was possible to integrate it.

Headaches and solutions (this part can be skipped)

With the NPM package googleapis you have access to various Google services. However in my opinion the APIs are documented in a strange way, various parts in one place other parts in another, I wasn't always sure if parts were really related to the NPM package.

It took me a long time to understand authentication part. After trying the APIs with OAuth 2.0 my application was working. But the token was short lived. I didn't want the headache of repeating the authentication process. I thought that there should be some other way so I looked into API keys. Google provides this but not for every API, but calendar wasn't supported. The solution finally came after lots of searching and reading. A Google service account. That's what I'm going to share with you guys today. It's actually pretty darn easy.

Setting up a Google Service Account (in the Google Cloud Console)

Obs. This article will not go in to depth setting up a Google Cloud Project

In the console go to IAM And Management -> Service accounts and create a new service account. Give it a name, an ID (this is also an email we'll need later on) and a description. Click Create. This account doesn't need permissions. Click Continue. This account doesn't need a service account user nor admin. But we'll need a JSON key. A file will be downloaded that we'll need later on.

The calendar

We'll now add the service account email (ID) to the calendar. Add it to the Calendar. Click here to know how to. The email should have the following permission Make changes to events.

The calendar ID can be found here, we'll also need it.

Well that was easy wasn't it, not everything needs to be bureaucratic.

The fun part

Now we'll run some basic NodeJS tests, make new project dir with npm init -y and install npm i -s googleapis date-fns to the project. Date-fns we'll use to format and manipulate dates. Copy the credentials json file in to the root of the project directory.

Let's create an get.js with the following code:

const { addWeeks } = require('date-fns')
const { google } = require('googleapis')
const credentials = require('<CREDENTIALS_FILE>.json')

const scopes = ['https://www.googleapis.com/auth/calendar']

const client = google.auth.getClient({
  credentials,
  scopes,
})

client.subject = '<SERVICE_ACCOUNT_EMAIL>'

const calendar = google.calendar({ version: 'v3', auth: client })

calendar.events.list(
  {
    calendarId: '<CALENDAR_ID>',
    timeMin: new Date().toISOString(),
    timeMax: addWeeks(new Date(), 1).toISOString(), // Let's get events for one week
    singleEvents: true,
    orderBy: 'startTime',
  },
  (err, res) => {
    if (err) {
      console.log(`The API returned an error: ${err}`)
    }
    // console.log(res.data.items) // All data
    const appointments = res.data.items.map((appointment) => ({
      start: appointment.start.dateTime || appointment.start.date,
      end: appointment.end.dateTime || appointment.end.date,
      id: appointment.id,
      status: appointment.status,
      creator: appointment.creator,
      description: appointment.description,
    }))
    console.log(appointments)
  },
)

and let's also create a new calendar event:

const { addMinutes } = require('date-fns')
const { google } = require('googleapis')
const credentials = require('<CREDENTIALS_FILE>.json')

const scopes = ['https://www.googleapis.com/auth/calendar']

const client = google.auth.getClient({
  credentials,
  scopes,
})

client.subject = '<SERVICE_ACCOUNT_EMAIL>'

const calendar = google.calendar({ version: 'v3', auth: client })

calendar.events.insert(
  {
    calendarId: '<CALENDAR_ID>',
    resource: {
      start: {
        dateTime: new Date().toISOString(),
        timeZone: '<TIMEZONE>',
      },
      end: {
        dateTime: addMinutes(new Date(), 60).toISOString(),
        timeZone: '<TIMEZONE>',
      },
      summary: 'Test event',
      status: 'confirmed',
      description: 'Test description',
    },
  },
  (err, event) => {
    if (err) console.log('Error', err)
    console.log(event.data)
  },
)

This was a lot of work to find and figure out. I learned a lot, especially that it actually wasn't that hard!

This is my first post, help me with my writing skills and share your thoughts!

Hope it's of good use, happy hacking!