DEV Community: Srushtika Neelakantam

Say hello to Ably Chat: A new product optimized for large-scale chat interactions

Srushtika Neelakantam — Fri, 12 Jul 2024 10:02:11 +0000

TL;DR: Today, we're excited to announce the private beta launch of our new chat product! Ably Chat bundles purpose-built APIs for all the chat features your users need in a range of realtime applications, from global livestreams operating at extreme scale to customer support chats embedded within your apps. It is powered by Ably’s reliable global platform with proven performance guarantees and scalability.

Request a beta invite now to give it a try!

We’ve had the privilege of working with a wide range of customers including global retailers, CRM vendors, sports franchises, creators, entertainers, and broadcasters - from HubSpot and SportsBet, to 17Live and InvitePeople - providing them with reliable, scalable and low-latency chat. Ably Pub/Sub is already a fantastic fit for a variety of chat use cases. But we’ve been doing a lot of thinking about how we can better help developers to overcome the many challenges of delivering chat features to market quickly, at scale and in an economically viable way.

That’s why we're excited to kick off the private beta for our new chat product.

What is Ably Chat?

Ably chat is designed to meet a wide range of chat use cases, such as livestreams, in-game communication, customer support, or social interactions in SaaS products. Built on Ably's core service, it abstracts complex details to enable efficient chat architectures.

Ably Chat comes with purpose-built APIs for a host of chat features enabling you to create 1:1, 1:many, many:1 and many:many chat rooms for any scale; have users send and receive messages; and see the online status of the users in the room or the occupancy i.e. how many people are there in total. You can also build typing indicators, and room level reactions. We are actively working on other features like moderation and the ability to update and interact with chat messages.

Check out the documentation for a sneak peek of the functionality offered at this stage. If there are other chat features you'd like us to prioritize, please let us know.

What Ably Chat looks like

//Connect to a chat room with any number of participants
const room = chatClient.rooms.get('basketball-stream', { reactions: reactionsConfig });
await room.attach();

//Subscribe to chat messages
const {unsubscribe} = room.messages.subscribe(message) => {
  displayChatMessage(message);
}

//Subscribe to room reactions
const {unsubscribe} = room.reactions.subscribe((reaction) => {
  displayReactionAnimation(reaction);
});

//Send a chat message
await room.messages.send('That was such a cool shot!');
//Send a room reaction
await room.reactions.send({ type: 'like', metadata: { effect: 'fireworks' } });

Check out the live demo to play with it yourself!

Why Ably Chat?

When Ably started in 2016, our goal was to make realtime interactions a key part of digital experiences. Now, we have a mature, battle tested WebSockets platform that gives us the freedom to focus on creating user-centric features that meet your needs in the best way possible. This approach unlocks many benefits for users of Ably Chat, and other products:

Composable realtime

A truly great realtime experience often involves combining multiple features together - chat, live updates, collaboration, notifications and more! Unlike chat-specific products or building your own solution from scratch, Ably offers the best of both worlds - full flexibility to build what you want, rapidly.

Whilst we will keep adding new features into our SDKs we know we’ll never be able to satisfy every unique use-case possible. That’s why we are also evolving our product suite to give you the flexibility to mix and match APIs and create what you need, exactly the way you want to, quickly and efficiently.

Dependable by design

We’ve built a realtime experience platform that ensures predictability of latencies. It is designed to preserve continuity of service at a regional and global level, ensuring capacity and availability. This allows you to maintain varying levels of scale seamlessly. Finally, data integrity comes baked in with message guarantees for ordering and exactly once delivery. This enables you to focus on your application and the user experience, with no infrastructure to build or manage.

Cost optimizations

A realtime platform needs to be able to support reliable chat at extreme scale, but must do so cost effectively. Typically, as chat usage (especially concurrent usage) increases, so does the cost per user. Cost optimizations and affordability of technology have been top of mind for us when designing the roadmap. With the upcoming batching and aggregation features you can maintain a low and stable cost per user. Additionally, Ably’s pricing model has been built for operations at scale with customisations such as hourly billing, usage based pricing and volume discounts.

All these abstractions and optimizations ultimately mean one thing — you spend less time figuring out the design patterns for good efficiency and a great user experience.

Get started with Ably Chat

Stay tuned for more updates and features as we roll out this new initiative. Sign up for the chat private beta to access new features early and collaborate on upcoming functionality to shape our roadmap. We're just getting started, and there's plenty more to come!

Collaboration is futile without these UX best-practices

Srushtika Neelakantam — Mon, 16 Oct 2023 13:44:19 +0000

More aspects of our work and life are online than ever before, and the trend towards remote work and increasingly remote teams is set to continue. Whilst we enjoy the freedom and flexibility that remote work provides, remote teams also face a significant challenge: establishing (and desiring) a sense of connection despite the physical distance between individuals.

Many employees express a loss of team spirit and community when they work remotely, which inadvertently impacts productivity. This has led to a rise in online collaboration tools and services, which strive to solve this problem by bringing the qualities and benefits of working in person to the virtual workspace. Specifically, they are using collaborative features to deliver the unspoken cues that we have lost in today’s virtual office environments. Meta information in the form of online presence, movement and focus provides a sense of togetherness and helps users to decide what to do next. This enhances not only team-based work - but also team health.

This article explores the features required to create a great virtual collaborative environment within applications - explaining the purpose of each feature from an end user perspective and the key UX considerations.

The essential role collaborative features play in the user experience

So, what are the essential ingredients needed in an application to make users feel they are part of an ‘in-person’ experience? There are two needs to accommodate: the emotional sense of ‘togetherness’, and the sense of being productive.

From an application perspective this means you need to provide users with three pieces of realtime information: “who is here with me?”, “where are they?” and “what are they doing?”. In a virtual, collaborative setting this information can be provided through the following collaborative features:

Avatar stack - who is here with me
Member location - where are they?
Live cursors - where are they moving to?
Component locking - what are they doing?

Let’s look at each of these in turn, and how they enhance the virtual collaborative experience.

Avatar stack

Avatar stacks show all the users currently collaborating within a shared space. They provide the virtual equivalent of being able to see who is sitting in a room with you. They can help users make informed decisions by indicating the connection state of fellow collaborators, e.g. are they online, offline or have recently left. Avatars can take various forms from photos to initials or symbols.

Member location

Member location is a way to show where fellow collaborators are located within a shared space. Similar to being able to see where people are in a room. This can be useful for understanding what other members are working on, and how their work might affect yours.

Live cursors

Live cursors show the live pointer position of other collaborators. They can be useful to show what someone is currently looking at and their movement around a space. A bit like knowing if someone is looking at the whiteboard, their computer or out the window in a face-to-face session. This is what differentiates the purpose of live cursors from ‘member location’. A member location feature shows ‘what I’m working on’, whilst an animated live cursor shows ‘what I might be looking at’. They can take many forms from the conventional arrow to cursors showing names or avatars.

Component locking

Component locking can be a valuable tool for collaborative applications. A lock displayed over a particular component signals to other members that they are currently being edited. Based on that information they can choose to wait or move onto another task.

Component locking provides much more than a visual cue, however. It helps reduce the instances where users accidentally overwrite each other's work, which needs to be consolidated in the backend.

UX design considerations for collaborative features

Having looked at each of the features required to make virtual collaboration as close to in-person as possible, let’s consider the UX requirements for a great collaborative experience, and the features that contribute to it.

At the end of the day, UX is all about the user and helping them reach their goals. Therefore, any collaborative features should adhere to the following principles:

Keep it simple: Eliminate visual clutter wherever possible, and cut any unnecessary elements or steps that complicate the user experience.
Make it intuitive to use: Always consider function first over design. Any features you add should make it as effortless as possible for users to do what they came to do on your site or app.
Don’t distract: Collaborative features are not meant to be the star of the show, they are meant as subtle visual gestures.

In addition, there are a number of best practices to consider on a feature-by-feature basis. Doing so will help ensure you are delivering maximum value to the user.

Best practices when building avatar stacks

Think about whether it is practical or beneficial to show a full list of collaborators to the user. For example only showing the avatars for online or recently left members, maybe sufficient. Listing everyone could be overwhelming to users, even degrading the experience.
In instances where you must show the full list, make sure to order with the online members listed first. This will make it easier for users to see who is available to collaborate, without having to scroll through a long list of users.
The value of instant updates to online status changes is often underestimated. Avatar stacks are huge contributors to unsaid virtual cues in a shared workspace, so make sure the status updates are reflected immediately in the UI, enabling users to take appropriate action based on that information.

Best practices when building a member location feature

The level of detail that you provide about user location can vary depending on the needs of your users. For example, you might only show the page that a user is on, or you might show the specific component that they are working on.
There are a variety of ways to display user location, such as using a border matching the member's avatar, the avatar itself, or a textual name.
Member location is meant to provide a subtle, visual cue indicating who is working on what. The UI should not distract from the core application or components being worked on.
As with avatar stacks, don’t underestimate the importance of instant updates to member location. Make sure the location updates are reflected immediately in the UI so users take appropriate action based on that information.
Avatar stacks can be used in conjunction with member location to show where exactly someone is within a space - allowing them to jump to their location or follow their movements.

Best practices when building live cursors

Understanding the true value of live cursors is important. Make sure they are not a distraction. Consider how many users may need to work synchronously in the same space. Cursors for a few people are fine, but beyond that the UI becomes so busy that any benefits they would provide are gone.
The movement of cursors has to be smooth and performant. Jerky movements or having cursors jump around from place to place, will reflect poorly on the entire experience.
The beauty of live cursors is that it shows the movements of others around you. It is not about watching an animated view of your own cursor. The original cursor will suffice. Not only will this prevent confusion on “who am I again?” but it also means there are no issues like lag to deal with.
When multiple users are active in close proximity it can be difficult to understand who’s who, especially when things are fast moving. In such instances just matching the cursor colour to an avatar is not enough, consider instead the addition of a name and/or avatar to enhance the UX.
Make sure the cursor position is adjusted to the screen size. This will ensure that whatever a user is pointing to is accurately reflected to everyone on different devices - even if that means adjusting to different layouts.

Live cursors can extend way beyond pointing and showing movement around a space. Tracking the movement of the cursors also presents the opportunity to:

- Replay the ‘drag and drop’ animation for UI components.

- Show mouse interaction, such as clicks, mouse drag etc.

- Allow users to annotate and draw on the board.

- Add emotions to a cursor.

Best practices for building a component locking feature

There may be times where a user forgets they have locked a field or their connection drops. Ensuring the lock status is tied to a member's online status is useful in making sure that no components are locked indefinitely - the APIs in the Spaces SDK do this by default.
A user may have locked a component but might have stepped away from the system for a long time. It might be a good idea to trigger a dialogue after a certain period of inactivity to check if the user is still there and working on the locked component. Lacking any response, the lock may be automatically released.

Adding in-app collaboration

Adding collaborative features, can greatly enhance the end user experience. But there is much to consider, not just from the POV of the user but also the implementation and performance of the features e.g. optimizing for the payload structure, frequency of updates, etc. This often means developer focus is diverted onto figuring out architecting the lower-level detail rather than the application.

Creating collaborative environments in a few lines of code

For the reason mentioned above, we came up with a new product “Spaces” — to help developers build collaborative spaces in their apps in just a few lines of code. The Spaces SDK comes with a set of simple, intuitive realtime collaboration APIs for a range of features. With these you can set up a virtual environment, where members can see: a live view of other collaborators working with them; their location within the app, their movement around the space when working on different components, and components actively locked for editing. APIs can be used individually or in combination to create the perfect collaborative space for your users - with minimal effort on your part. We have worked through the architectural challenges of integrating these features, so you can focus on the UX.

See the Spaces product page and play with our interactive demo or head straight to docs to learn more on the APIs.

Introducing Spaces: Build collaborative environments in a few lines of code

Srushtika Neelakantam — Mon, 11 Sep 2023 13:39:52 +0000

Spaces comes with a purpose-built SDK to enable developers to add a collaborative environment around existing applications.

We are very excited to announce a new product — Spaces!

The Spaces SDK comes with an intuitive set of APIs that allow you to build realtime collaboration features such as avatar stacks, live cursors, member location and component locking, in days. Each API is optimized for their specific use-case, reducing integration effort.

Spaces at a glance

Feature specific APIs: The only product that comes with purpose-built, realtime collaboration APIs for adding avatar stacks, member location, live cursors, and component locking. All APIs are optimized for top performance. For example, the live cursors API automatically batches pointer position events to avoid unnecessary streaming of messages whilst ensuring negligible latency.
Easy to use: Spaces is a simple and intuitive SDK that can be used with any web application.
High performing: Powered by our market-proven, realtime infrastructure that’s built for reliability at scale, backed by five nines SLAs and messaging guarantees.
Realtime experiences unlocked: Spaces is built to work with Ably’s other complementary products like Pub/Sub Channels to unlock end-to-end realtime messaging throughout your app. While Spaces powers the features you need to enable synchronous collaboration for teams and manage their participant state, Pub/Sub channels allows you to flexibly broadcast and sync app state changes between members, your backend and any other pieces in your system design.

Enhancing your product with in-app collaboration & Spaces

With the Spaces SDK you can set up a collaborative environment on any part of your application (or your entire application) so that collaborators have contextual awareness of what everyone is up to. Members of a space can see which other members are online, what they are looking at, their location within the app (this could be a cell, a page, a slide or a folder) and any specific component they’ve locked for editing. All of this and more in just a few lines of code. No need to change your existing system design or app architecture. No realtime infrastructure to build and maintain to scale the collaboration to millions of virtual spaces and users.

SDK feature highlights

Here’s the full set of APIs available in the beta release with many more planned:

Space: A virtual collaborative space set up on your application where members can collaborate with each other in realtime.
Members: Online users connected to the virtual collaborative space. This powers an avatar stack for end users which is a visual representation of a member’s presence - showing them as online and connected.
Member location: The live location of a user within the app - page, cell, slide, block or anything else that makes sense for your application.
Live cursor: The pointer location of members in a virtual space showing what they are looking at. The live cursor API automatically batches messages and provides the message rate required for smooth performance of 100s of simultaneous cursors (though we don’t recommend going beyond 15 for a good user experience).
Component locking: Enables end-users to lock specific UI components while making their edits, so there’s no confusion with other collaborators or disruption to overall collaboration in the space.

Get started today

Spaces is currently available in beta. To get started simply

Sign-up for a free developer account
Take a closer look at Spaces
Dive into the Spaces docs
Sign-up for the webinar

The async/await post we promised

Srushtika Neelakantam — Wed, 08 Dec 2021 14:49:22 +0000

Great user experience on the web comes from being able to provide users with exactly what they want in the most seamless way possible. Behind the scenes, some user actions may take more time to process than others. For example, showing or hiding an HTML element is a quick operation whereas making an XHR request to get data from an external API is a slower operation. JavaScript provides us with a way to handle them all without giving up that instant delight users naturally expect.

In this article, we’ll describe how JavaScript executes asynchronous operations and review different ways to write asynchronous code from Callbacks to Promises and explain what’s best and why. Most importantly, we’ll use the modern and recommended way to write asynchronous JavaScript to instantiate and use Ably’s JavaScript SDKs.

Jump to "Using the promise-based version of the ably-js SDK"

If you are new to Ably, here's a quick summary - Ably provides APIs to add realtime messaging functionality to your applications. It is based on the Publish/Subscribe messaging pattern and operates mostly on the WebSockets protocol. You can plug in the Ably SDK and start publishing messages in realtime to millions of devices. Sign up for a free account to explore all the platform's features.

The inner workings of JavaScript

JavaScript is a single-threaded programming language. It is predominantly used on the web or in the form of NodeJS in the backend.

If we focus on the frontend, JavaScript-based applications run in a web browser. The actual execution of the JavaScript code is done by a JavaScript engine, which usually comes in-built with every browser. For example, Google Chrome comes with the popular V8 engine (which is also the engine running NodeJS), Mozilla Firefox comes with the SpiderMonkey engine, Microsoft Edge comes with the Chromium engine, and so on.

Being single-threaded means that JavaScript can only do one thing at a time and sequentially execute statements in a given piece of code. When the code takes longer to execute, like waiting for some data to return from an external API, the application would essentially halt at that point and the end-user would end up seeing an unresponsive screen.

But, this doesn’t usually happen, does it?

The everyday working of frontend JavaScript is made possible not just by the JavaScript engine provided by the web browser but with a supplement of three key things:

i) a myriad of Web APIs, we'll refer to these as Browser APIs to avoid confusing them with external APIs

ii) the Message Queue

iii) the Event Loop

Together these elements allow JavaScript to run asynchronous functions that can continue execution without needing to wait for things that take time.

Let’s look at how these language and browser features work together.

A whirlwind tour of asynchronous JavaScript

In a nutshell, the working of asynchronous JavaScript code can be visualized as shown in the diagram below.

The JavaScript engine has a memory heap and a call stack. The memory heap allocates memory for the data in the code and updates the values as per the logic. The call stack is a last in, first out (LIFO) data structure that keeps track of the statement to be executed next to run the code in sequential order.

What happens when things are slow? Let’s say the call stack encounters a setTimeout() function. Let's see how the execution of this statement proceeds in the above flow.

First, we can refer to the only thread that JavaScript has as the "main thread". In case of a setTimeout(), the main thread will kick off the execution of this statement by calling the function from the Browser APIs but not wait until the execution is complete.

When the Browser finishes executing the setTimeout() function, it returns the results. The tricky part, however, is getting these results back to the main thread and showing them in the application.

These results don't immediately get inserted into the call stack because that would disturb the flow of execution. Instead, it inserts the results at the end of the Message Queue. The event loop will then follow a process to decide the best time to pick this up and insert it into the call stack.

The best resource I’ve come across to understand the JavaScript event loop is this amazing talk by Philip Roberts - What the heck is the event loop anyway?. While I’ve summarized the explanation below I’d still recommend giving it a watch.

The Event loop is essentially an infinite while loop (hence the name) that continuously checks for two things:

i) if the call stack is empty

ii) if there are any elements in the Message Queue

When both these conditions become true, the event loop picks up the first element in the queue and puts it on the call stack for the main thread to execute it.

The interesting thing to consider here is how we can let the runtime know that a certain statement depends on an external resource (where the processing is being done somewhere else) and may take time to return. We want the execution to continue, not pause while waiting on that external function to complete. Let's explore this next.

From Callbacks to Promises to Async/Await

We can think of any asynchronous operation we perform in JavaScript as an API call. This call is done either to an in-built API provided by the browser, for example, setTimeout(), or to an API from a third-party provider, for example ably.channel.publish(). In fact, this call can also be done just to another function that's part of the same application but let's assume it is an external function for a better understanding. I've linked some code examples of native async functions in the 'Further reading' section at the end.

The implementation of an async function provided by the API needs to have a way to tell the main thread what needs to be done when it has finished executing the time taking task.

This can be done in one of the following three ways:

i) Callbacks

ii) Promises with .then syntax

iii) Promises with async/await syntax

Let’s explore them one by one.

Option 1 - Async JavaScript with callbacks

A callback is a function that is passed to another function as a parameter. When calling the async function initially, we provide it with a callback function as one of the parameters. When the async function finishes execution, it calls that callback function, along with the results of the execution as arguments. At this point, the callback function is placed on the Message Queue and will eventually be picked up by the event loop and dropped into the call stack for the main thread to execute it.

Let’s take a look at an example with the asynchronous channel publish function provided by Ably’s JavaScript SDK:

/* Code Snippet 1 */
import * as Ably from "ably";
const client = new Ably.Realtime({ authUrl: "/auth", clientId: "bob" });
const channel = client.channels.get("general-chat");

/* function reference:
publish(String name, Object data, callback(**ErrorInfo** err))
*/

channel.publish("new-chat-msg", "Hey there! What is up?", (error) => {
  if (error) throw error;
  console.log("Published successfully");
});

As you can see, the last (optional) parameter in the publish function expects a callback function.

From the Ably SDK (i.e. the async function provider) side of things, when the publish function is called, it executes that function logic. When it’s done, it calls the callback function and passes it some data if it's applicable. This would look something like so:

/* Code Snippet 2 */
class RealtimeChannel {
  publish(messages, callback) {
    /* do some stuff to execute the async operation */

    callback(error, result);
  }
}

As explained before, this callback function will be put at the end of the Message Queue. This will be picked up by the event loop and put onto the call stack which is when it’ll be executed by the main thread. At this point, it'll print the success message to the console depending on the value of the error parameter passed to it.

Ok, that's all well and good. We've understood a way to write asynchronous JavaScript functions, so why even consider other options?

Callbacks are a simple concept and work well for standalone asynchronous operations. However, they can quickly get tedious to write and manage if they have dependencies on each other. For example, consider a scenario where you need to do certain async things sequentially, using the data from one task in the other, say:

i) enter presence on a channel

ii) get some historical messages

iii) publish a new message on the channel with the first message retrieved from history

The callback-based implementation for this scenario will look as follows:

/* Code Snippet 3 */
import * as Ably from "ably";
const realtime = new Ably.Realtime({ authUrl: "/auth", clientId: "bob" });
const channel = realtime.channels.get("general-chat");

/* function references:
- enter(Object data, callback(ErrorInfo err))
- history(Object options, callback(ErrorInfo err, PaginatedResult<Message> resultPage))
- publish(String name, Object data, callback(**ErrorInfo** err))
*/

// step 1 - enter presence
channel.presence.enter("my status", (error) => {
  if (error) throw error;
  console.log("Client has successfully entered presence");

  // step 2 - get historical messages after presence enter
  channel.history((error, messagesPage) => {
    if (error) throw error;
    messagesPage.items.forEach((item) => console.log(item.data));
    let firstHistoryMessage = messagesPage.items[0].data;

    // step 3 - publish a new message after get history
    channel.publish("new-chat-msg", `Hey there! What is up?, my first history msg was ${firstHistoryMessage}`, (error) => {
      if (error) throw error;
      console.log("Published successfully");
    });
  });
});

While this is an accurate implementation and will work perfectly fine, it already looks messy and difficult to manage due to the multiple nested callbacks. This is commonly referred to as Callback Hell because debugging or maintaining anything which looks like this would be a daunting task. And, this is exactly the reason we have other, more modern ways of writing asynchronous JavaScript functions. Let's explore these next.

Option 2 - Async JavaScript with Promises (.then syntax)

The second option introduces a concept called ‘Promises’. Instead of calling a callback function, the API side implementation of the asynchronous function will create and return a "promise" to the requesting client that wants to execute the async function.

A Promise can have one of the following three states:

i) Pending - meaning we’ve started an async operation but its execution has not completed yet

ii) Resolved (or Fulfilled) - meaning we started an async task and it has finished successfully

iii) Rejected - meaning we started an async task but it finished unsuccessfully, in most cases with a specific error that will be returned to the client

Let's consider a Promise based async operation and again see both sides of the coin i.e. what happens on the API side implementation as well as the requesting client side. This time, let's first take a look at the API side of things:

/* Code Snippet 4 */
class RealtimeChannel {
  publish = (messages) => {
    return new Promise((resolve, reject) => {
      /*
      do some stuff to execute the async operation
      */
      error ? reject(error) : resolve(result);
    });
  };
}

The promise executor in the API calls the resolve() function if the async task was executed as expected, along with the results of the operation. However, if there was some issue with the execution it calls the reject() function.

A requesting client can consume such a Promise using a .then() function attached to the async function call. The .then() code block is similar to a callback code block and will be executed when the async task has finished executing. We can also attach a .catch() to the .then() block to catch any errors that may have occurred during the execution of the async task.

In terms of the explanation above, the .then() block will be executed when the promise executor in the API calls the resolve() function and the .catch() block will be executed when the API calls the reject() function.

At the time of writing this article, the Ably JS SDK doesn't provide promises by default. To be able to use the promise version of the SDK, we need to use new Ably.Realtime.Promise() constructor when instantiating the library.

Let's now see how our example will work on the client side

/* Code Snippet 5 */
import * as Ably from "ably";
const realtime = new Ably.Realtime.Promise({ authUrl: "/auth", clientId: "bob" });
const channel = realtime.channels.get("general-chat");
/* function reference:
publish(String name, Object data): Promise<void>
*/

channel
  .publish("new-chat-msg", "Hey there! What is up?")
  .then(() => {
    console.log("Published successfully");
  })
  .catch((error) => {
    console.log("There was an error while publishing: " + error);
  });

If you compare the above with the "Code Snippet 1", it seems more logical in the sense that we can understand that certain statements will execute after certain other statements due to the literal English meaning of the word 'then'.

The real advantage however can be seen if we need to perform multiple asynchronous tasks sequentially, in some cases using the data returned in the previous async task.

Let's consider the same scenario as we did in the callbacks version:

i) enter presence on a channel

ii) get some historical messages

iii) publish a new message on the channel with the first message retrieved from history

Let's see how this will look like using Promises with a .then syntax.

/* Code Snippet 6 */
import * as Ably from "ably";
const realtime = new Ably.Realtime.Promise({ authUrl: "/auth", clientId: "bob" });
const channel = realtime.channels.get("general-chat");

/* function references:
- enter(Object data): Promise<void>
- history(Object options): Promise<PaginatedResult<Message>>
- publish(String name, Object data): Promise<void>
*/

// step 1 - enter presence
channel.presence
  .enter("my status")
  .then(() => {
    // this block executes after the presence enter is done
    console.log("Client has successfully entered presence");
    //step 2 - get historical messages
    return channel.history();
  })
  .then((messagesPage) => {
    // this block executes after the channel history is retrieved
    messagesPage.items.forEach((item) => console.log(item.data));
    let firstHistoryMessage = messagesPage.items[0].data;
    //step 3 - publish a new message
    channel.publish("new-chat-msg", `Hey there! What is up?, my first history msg was ${firstHistoryMessage}`);
  })
  .then(() => {
    // this block executes after the message publish is done
    console.log("Published successfully");
  })
  .catch((error) => {
    // this block executes if there's an error in any of the blocks in this Promise chain
    console.log("We have an error:", error);
  });

As you can see, the Promise version with a .then() syntax reduces the complexity and the level of indentation when compared to the callbacks approach. This helps us understand and maintain the code much easily.

However, as you can see with this option, we need to wrap each execution step in a function call and return the results to the next .then(). Although a huge improvement from the callbacks syntax, it seems like it could still get verbose pretty quickly. This is what the async/await syntax helps us with. Let's understand that next.

Option 3 - Async JavaScript with Promises (async/await syntax)

This third option is just another version of the second option. There's no change on the API side of things. The API would still create a 'Promise' and either resolve() or reject() it after the async task is executed.

The way we consume it on the front end, however, is different (and better!). The async/await provides syntactic sugar to reduce the complexity in chained async tasks. Let's take a look at how the "Code Snippet 6" above would look like if we use async/await instead of .then().

/* Code Snippet 7 */
import * as Ably from "ably";
const realtime = new Ably.Realtime.Promise({ authUrl: "/auth", clientId: "bob" });
const channel = realtime.channels.get("general-chat");

/* function references:
- enter(Object data): Promise<void>
- history(Object options): Promise<PaginatedResult<Message>>
- publish(String name, Object data): Promise<void>
*/

async function main() {
  try {
    // step 1 - enter presence
    await channel.presence.enter("my status");
    console.log("Client has successfully entered presence");

    //step 2 - get historical messages
    let messagesPage = await channel.history();
    console.log("Retrieved history successfully");
    messagesPage.items.forEach((item) => console.log(item.data));
    let firstHistoryMessage = messagesPage.items[0].data;

    //step 3 - publish a new message
    await channel.publish("new-chat-msg", `Hey there! What is up?, my first history msg was ${firstHistoryMessage}`);
    console.log("Published successfully");
  } catch (error) {
    console.log("We have an error:", error);
  }
}

main();

As you may have observed, we've wrapped all our statements in a function this time. This is because the async/await syntax can only be used in functions starting with the async keyword. Such an async function can then contain zero or more await statements.

Statements that begin with the keyword await are asynchronous functions. Similar to the previous option with Promises using the .then() syntax, these statements get returned via the Message Queue when the underlying Promise provided by the API calls either a reject() or a resolve() function.

Concurrency of independent asynchronous statements

Given that the async/await approach looks a lot like writing synchronous statements, it is a common mistake to make independent code unnecessarily wait for the previous tasks to finish instead of having them execute concurrently (in parallel). For example, in the code examples we saw in the previous sections, if entering the client in the presence set, retrieving history and publishing a new message had no dependencies on each other, we can easily do these things in parallel instead of sequentially.

This can be done using the Promise.all() function as shown below:

/* Code Snippet 8 */
import * as Ably from "ably";
const realtime = new Ably.Realtime.Promise({ authUrl: "/auth", clientId: "bob" });
const channel = realtime.channels.get("general-chat");

/* function references:
- enter(Object data): Promise<void>
- history(Object options): Promise<PaginatedResult<Message>>
- publish(String name, Object data): Promise<void>
*/

async function main() {
  try {
    const enterPresence = channel.presence.enter("my status");
    const getHistoryMessages = channel.history();
    const publishMessage = channel.publish("new-chat-msg", "Hey there! What is up?");

    // perform all three async functions concurrently
    const values = await Promise.all([enterPresence, getHistoryMessages, publishMessage]);

    console.log("Client has successfully entered presence");
    console.log("Retrieved history successfully");
    console.log("Published successfully");
    let messagesPage = values[1];
    messagesPage.items.forEach((item) => console.log(`History message: ${item.data}`));
  } catch (error) {
    console.log("We have an error:", JSON.stringify(error));
  }
}

main();

/*
Note the publish function doesn't use any data returned 
by the History API in this case as we are considering the three functions
to be executed independently of each other.
*/

The case of asynchronous event listeners

By now, we have a good understanding that Promises with either the .then() or the async/await syntax are a big improvement over callbacks. But what happens in the case of asynchronous event listeners where you are constantly listening for some updates. For example, in case of a setInterval() from the inbuilt Browser APIs or ably.channel.subscribe() from the Ably API?

Promises are great for one off execution of an async task that either resolves or rejects based on some logic. However, in the case of a subscription, we'd need the resolution to happen multiple times i.e. whenever there's a new message to be pushed from the API to the listening client. Promises unfortunately cannot do that and can resolve only once. So, for active listeners that return data repeatedly, it's better to stick with callbacks.

Using the promise-based version of the `ably-js` SDK

As per the examples we've been seeing so far, it is clear that Ably’s JavaScript SDK provides a promisified version. This means we can consume the asynchronous functions (except for listeners) using the async/await syntax. In the devrel team, we've been using the async style API in our latest demo - the Fully Featured Scalable Chat app.

At the time of writing this article, the default way to consume async functions using the Ably JS SDK is using callbacks, but in this section, we’ll take a look at a few key functions where we consume the promisified API using the async/await syntax.

1. Importing and instantiating the Ably Realtime or Rest instances:

/* Code Snippet 9 */

import * as Ably from "ably";

//before - instantiating the Ably SDKs, callback version

const client = new Ably.Realtime(options);

const client = new Ably.Rest(options);

//now - instantiating the Ably SDKs, Promise version

const client = new Ably.Realtime.Promise(options);

const client = new Ably.Rest.Promise(options);

2. Attaching to a channel

/* Code Snippet 10 */

//before - attaching to a channel, callback version
client.channel.attach(() => {
  console.log("channel attached");
});

//now - attaching to a channel, promise with async/await version
async function attachChannel() {
  await client.channel.attach();
}
attachChannel();

3. Retrieving and updating presence status on a channel

/* Code Snippet 11 */

//before - presence functions, callback version
channel.presence.get((err, members) => {
  console.log("Presence members are: ", members);
});
channel.presence.enter("my status", () => {
  console.log("Client entered presence set");
});
channel.presence.update("new status", () => {
  console.log("Client presence status updated");
});
channel.presence.leave(() => {
  console.log("Client left presence set");
});

//now - presence functions, promise with async/await version
async function ablyPresenceStuff() {
  await channel.presence.enter("my status");
  await channel.presence.update("new status");
  await channel.presence.leave();
}

ablyPresenceStuff();

/*
Please note - the above code snippets are slightly 
different in terms of how they'd run.

The callback version concurrently executes all four functions,
whereas the async/await version executes all the statements 
sequentially.

Please scroll back up and read 
'**Concurrency of independent asynchronous statements'** 
if you are interested to learn more about this behaviour.
*/

3. Publishing messages

/* Code Snippet 12 */

//before - publishing messages, callback version
channel.publish("my event", "Hey, this is event data", () => {
  console.log("Publish done");
});

//now - publishing messages, Promise with async/await version
async function publishToAbly() {
  await channel.publish("my event", "Hey, this is event data");
  console.log("Publish done");
}
publishToAbly();

4. Subscribing to messages

/* Code Snippet 13 */

//before - subscribing to messages, callback version
channel.subscribe((msg) => {
  console.log("New message received", msg.data);
});

//now - subscribing to messages, Promise with async/await version
channel.subscribe((msg) => {
  console.log("New message received", msg.data);
});

/*
Please note, there's no change here. As described in the previous section 
Promises cannot be used with listeners which need be triggered multiple times.

Hence, in this case, we stick to callbacks.
*/

5. Retrieving historical messages

/* Code Snippet 14 */

//before - history API, callback version
channel.history({ limit: 25 }, (err, resultPage) => {
  resultPage.items.forEach((item) => console.log(item.data));
});

//now - history API, callback version
async function getAblyHistory() {
  const resultPage = await channel.history({ limit: 25 });
  resultPage.items.forEach((item) => console.log(item.data));
}
getAblyHistory();

We are moving to Promise by default

In the upcoming versions of the JS SDK, you won't need to instantiate the promise version explicitly with Ably.Realtime.Promise(options). Instead, if you do Ably.Realtime(options). It'll use the promisified API by default.

If you want to stick to using the callbacks version at that point, you can explicitly instantiate the Callbacks constructor with Ably.Realtime.Callbacks(options) and continue using callbacks as default.

References and further reading

Sample code snippets showing callbacks and promises using native functions.
Working examples of using the async style with the Ably APIs:
- Ably React Hooks NPM package
- A fully featured chat demo
The Ably JavaScript SDK repository
Realtime use-case demos on Ably Labs

Free DevRel mentorship

Srushtika Neelakantam — Fri, 19 Nov 2021 12:08:50 +0000

Srushtika Neelakantam

@srushtika

Hey folks! I'm offering some free mentorship sessions for anyone trying to get into DevRel or someone who recently started in this role and is trying to figure things out 🙌

(Details 🧵👇)

#devrel #developerrelations

11:21 AM - 19 Nov 2021

Srushtika Neelakantam

@srushtika

What is the timeline/schedule/bandwidth?

I have the bandwidth currently to mentor 3 people until Feb 2022 at least. After this period we'll review any time constraints on either side and decide the best ways to move forward. I'm in GMT and can do catchups between 9 - 5

11:21 AM - 19 Nov 2021

Srushtika Neelakantam

@srushtika

How will it work?

We'll establish a specific goal and try to work towards reaching that in the specified time. Along the way, we'll have regular calls or textual catchups to decide on the next steps or review previous ones.

11:21 AM - 19 Nov 2021

Srushtika Neelakantam

@srushtika

How can I reach out?

Please send me a DM on Twitter with some details around:
- why are you seeking this mentorship?
- very vaguely what would you like to achieve during this?
- anything else you'd like to say or questions you might have

11:21 AM - 19 Nov 2021

Database-driven realtime architectures: building a serverless and editable chat app - Part 2

Srushtika Neelakantam — Fri, 08 Oct 2021 12:30:49 +0000

Hello again!

Welcome to Part 2 of this article series where we go through database-driven architectures by understanding the nitty gritty of a chat app where you can edit messages.

If you missed the Part 1 of this article series, you can jump to it:

Database-driven realtime architectures: building a serverless and editable chat app - Part 1

Also, check out the editable chat app demo or explore the project on GitHub.

In the previous part, we explored the high-level architecture of the chat app and understood how we made it possible to implement database-driven realtime messaging using the Ably-Postgres connector.

In this part, we will explore each section of that architecture and focus on implementation details to get the app working. It is divided into the following sections and topics:

Section 1: Navigating the Nuxt app (even if you are not a Nuxt developer)
Section 2: PostgresDB setup for our chat app
Section 3: Lambda function setup on the Ably integrations dashboard
Section 4: Set up the Ably Postgres connector for the chat app
Recap of the app architecture
Deployment

Let's dive right in!

Section 1 - Navigating the Nuxt app (even if you are not a Nuxt developer)

Let's start with the frontend app written in NuxtJS. The chat-web-app folder in the GitHub repo contains the Nuxt app. The chat app is a static site that can be hosted on any CDN without needing an origin server. This is a classic example of Jamstack architecture.

If you have never worked with Nuxt, here are the basic things you need in order to understand the web application and, more importantly, the data flow within the app and external services. In this case, this is just Ably.

Folder structure

The assets folder contains any uncompiled assets we need in the app, such as various images for the tech stack logos and architecture diagrams.
The components folder contains all the Vue components needed in the web app. A component is a reusable Vue instance. Each component has its own HTML, CSS, and JavaScript. This makes it easy to plug the visual component into as many places as needed without needing to repeat the code. We are also using Tailwind CSS framework to easily add various CSS classes. You can find these classes in their documentation but the key thing to note here is that we use the @apply directive to apply these classes to various HTML elements.
The netlify/functions folder contains the logic for enabling an authentication endpoint for Ably. This is a format expected by Netlify to be able to automatically recognize each file in there as an executable serverless function.
The pages folder contains Vue components representing a specific route in the web app. Nuxt automatically creates routes for any files added within this folder. So index.vue would represent / and as an example products.vue would map to the /products route.
The static folder contains assets that don't need to be compiled and can be served as is. We just have a favicon here.
The store folder contains files related to the VueX store. VueX is Vue's state management library. It enables us to centrally manage the state of the app. This not only allows us to separate out visual components from the data flow within the app, but also allows us to easily extend or reuse any methods. VueX also provides a very structured way to manage state which allows for easy debugging and monitoring. Nuxt comes with VueX out of the box, so no need to install it separately. However, it is disabled until we add any files in the store folder. We'll look at the store extensively in the following sections.
The nuxt.config.js file contains configuration for various things in our Nuxt app. This is the place to include any HTML headers, metadata, etc. It also contains modules that are extensions to Nuxt's core functionality. We are mainly using tailwind (a CSS framework), fontawesome (library to add icons), and dotenv (library to allow using environment variables) as modules in our app.
The tailwind.config.js file is the default config file added by Tailwind which can be used to add any customizations. We don't have any at the moment so this file is mostly empty.

The Vue components

Here's a screenshot of the chat app with the visual components labelled as they appear in the repository.

There are two components missing:

UsernameInput.vue: a username input box that appears before you enter the chat
PresenceListMobile.vue: the mobile version of the presence list which is hidden by default.

The starting point for all these components is the index.vue file in the pages folder.

All Vue components have three sections:

HTML: everything between the <template></template> tags
Javascript: everything between the <script></script> tags
CSS: everything between the <style></style> tags

The JavaScript section exports a default object with various data variables, watchers, and methods. It also has imports at the top as needed. The following is a noteworthy import statement because it allows an easy way to work with the data in the store.

import { mapGetters, mapMutations, mapActions } from "vuex";

It is an easy way to access various elements in the global store of our app and manipulate the state of the web app. We'll learn more about what these various things mean in the next section.

VueX state management for the chat app

The index.js file in the store folder is the starting point of the state management in our application. It exports a new store after instantiating a VueX store with four objects: state, getters, mutations, and actions.

State: This single object contains the application-level state that represents the single source of truth and keeps components n sync with each other.

Getters: Getters are methods to compute derived states for use anywhere in the app. In the chat app, all the getters return the state object as is, except for getIsUsernameEntered which returns a boolean depending on whether the username was entered or not.

Mutations: Mutations are methods that change the value of a certain state object. They should always be synchronous to ensure that we have a good view of the state changes. To update the state based on an asynchronous operation, we use "actions" as described next.

Actions: We use actions to perform asynchronous operations and call a mutation when ready to change the state as a result of that async operation. This is the most important part of the store in our app as this is where we connect to Ably and publish and subscribe to messages on various channels.

Inside actions.js

Before adding any methods to be exported out of this file, I've imported the Ably npm library with import * as Ably from "ably";

Let's now take a look at each method:

1. The instantiateAbly() method

// init an Ably realtime instance using token auth
instantiateAbly(vueContext) {
  const ablyInstance = new Ably.Realtime({
    authUrl:
      "https://serverless-scalable-chat.netlify.app/.netlify/functions/ably-auth",
    echoMessages: false
  });
  // when ably is successfully connected, set state variables and call methods to attach to various channels and subscribe to the presence set
  ablyInstance.connection.once("connected", () => {
    vueContext.commit("setAblyClientId", ablyInstance.auth.clientId);
    vueContext.commit("setAblyConnectionStatus", true);
    vueContext.commit("setAblyRealtimeInstance", ablyInstance);
    vueContext.dispatch("initAblyChannels");
    vueContext.dispatch("subscribeToAblyPresence");
  });
}

In this method, the Ably Realtime library is instantiated with two parameters, called client options:

authUrl: this is the URL of the authentication endpoint. The app uses Token-based authentication and Netlify serverless functions to issue Token Requests to any legitimate frontend clients that'd like to authenticate with Ably and use its service.
echoMessages: By default, any messages published to Ably are also echoed back on the same connection. To override the default, set it to false.

Once Ably is successfully connected, some commit methods are called to save the data into variables. The commit method calls a mutation, which in turn updates the state of the app. The dispatch method is used to invoke other actions, in this case initAblyChannels and subscribeToAblyPresence. We'll see these next.

2. The initAblyChannels() method

// attach to the incoming and outgoing channels
initAblyChannels(vueContext) {
  const outgoingCh = this.state.ablyRealtimeInstance.channels.get(
    this.state.channelNames.outgoingChat
  );

  const incomingCh = this.state.ablyRealtimeInstance.channels.get(
    this.state.channelNames.incomingChat
  );

  vueContext.commit("setAblyChannelInstances", { outgoingCh, incomingCh });

  vueContext.dispatch("subscribeToChannels");
}

In this method, we instantiate both our Ably channels, one for publishing messages into, (outgoing chat) and the other for subscribing to messages ( incoming chat channel). We then commit the channel instances and call a new method to subscribe to channels.

3. The subscribeToChannels() method

// subscribe to the incoming and outgoing channel instances
subscribeToChannels({ commit, state }) {
  state.channelInstances.incomingChat.subscribe(msg => {
    let msgPayload = JSON.parse(msg.data);
    let operationPerformed = msgPayload.type;

    /* check if the update is about a new message being inserted or an existing message being edited */
    if (operationPerformed == "INSERT") {
      // set the update type to new, so we can scroll the message list to bottom
      commit("setChatMsgArrayUpdateType", "new");
      state.chatMessagesArray.push(msgPayload.row);
    } else if (operationPerformed == "UPDATE") {
      // set the update type to edit, find and update the array object with new data
      commit("setChatMsgArrayUpdateType", "edit");
      let msgObjToEdit = state.chatMessagesArray.find(
        msg => msg.msg_id == msgPayload.row.msg_id
      );
      msgObjToEdit.msg_data = msgPayload.row.msg_data;
      msgObjToEdit.is_edited = msgPayload.row.is_edited;
    }
  });
}

In this method, we subscribe to the incoming-chat channel instances from the previous method. When the subscription callback is triggered, we check if the message received is INSERT or UPDATE. The former would mean a new chat message was published (and thus inserted as a new row into the database). The latter would mean an existing chat message was edited (hence a row was updated in the database).

If it's a new message, we append it to the chat messages array. If it's an updated message, we find the message in the chat messages array using its unique msg_id property, and update the is_Edited property to true.

We also call a mutation to set the chatMsgArrayUpdateType. This ensures that the chat list scrolls to the bottom whenever a new message arrives but not when a message is edited.

4. The subscribeToAblyPresence() method

//subscribe to Ably presence updates on the outgoing channel
subscribeToAblyPresence(vueContext) {
  this.state.channelInstances.outgoingChat.presence.subscribe(
    "enter",
    msg => {
      console.log("Entered", msg);
      vueContext.dispatch("handleNewMemberEntered", msg);
    }
  );
  this.state.channelInstances.outgoingChat.presence.subscribe(
    "leave",
    msg => {
      console.log("Left", msg);
      vueContext.dispatch("handleExistingMemberLeft", msg);
    }
  );

  // get a list of members already present in the Ably presence list
  this.state.channelInstances.outgoingChat.presence.get(
    (err, presenceList) => {
      for (const member in presenceList) {
        vueContext.dispatch("handleNewMemberEntered", presenceList[member]);
      }
    }
  );
}

In this method, we subscribe to enter and leave updates on the outgoing chat channel and call new methods to handle entries and leaves as they happen. We also use presence.get to get a list of existing members on the channel. This is useful to retrieve a list of people who came online before the current client and are still connected. We update the onlineMembersArray via mutations in each case. We also keep a local count of the number of people present using the presenceCount state variable, and update it whenever someone leaves or joins also via mutations (invoked using the commit keyword from the methods in the "actions" object).

// handle a new member entering the Ably presence set
handleNewMemberEntered(vueContext, member) {
  vueContext.commit("setPresenceIncrement");
  vueContext.commit("setOnlineMembersArrayInsert", {
    id: member.clientId,
    username: member.data.username
  });
},

// handle an existing member entering the Ably presence set
handleExistingMemberLeft(vueContext, member) {
  vueContext.commit("setOnlineMembersArrayRemove", member.id);
  vueContext.commit("setPresenceDecrement");
}

5. The enterClientInAblyPresenceSet() method

// enter the current client in the Ably presence set
enterClientInAblyPresenceSet(vueContext) {
  this.state.channelInstances.outgoingChat.presence.enter({
    username: this.state.username
  });
}

In this method, we enter the current client into the presence set by calling the presence.enter() method on the outgoing chat instance. Doing this will invoke presence.subscribe('enter') for this and all the other clients. This is done when the user types in a username and clicks on the enter chat button.

6. The publishMyChatMsgToAbly() method

// global method to publish a chat message on the outgoing channel instance
publishMyChatMsgToAbly({ state }, chatMsg) {
    state.channelInstances.outgoingChat.publish("chatMsg", {
      username: state.username,
      content: chatMsg
    });
}

We use this method to publish a new chat message to Ably. This is invoked from the ChatInput component. This is included as part of the global state management methods to allow for reusability and publishing of messages to Ably from any component in our UI.

7. The publishMyEditedMsgToAbly() method

// global method to publish an edit update on the outgoing channel instance
publishMyEditedMsgToAbly({ state }, { editedMsg, msgIdToEdit }) {
    state.channelInstances.outgoingChat.publish("editedMsg", {
      username: state.username,
      content: editedMsg,
      msgIdToEdit: msgIdToEdit
    });
  }
}

Similar to the previous method, we use this one to publish a message to Ably indicating an edit. We include the following:

Username of the client editing it
New message content after edit
Unique ID of the message that was edited

Now that we have a good understanding of the methods and functionalities in the Nuxt app. Let's move on to the next step.

Section 2 - PostgresDB setup for our chat app

PostgreSQL is an open-source object-relational database. This comes from their website:

PostgreSQL comes with many features aimed to help developers build applications, administrators to protect data integrity and build fault-tolerant environments, and help you manage your data no matter how big or small the dataset. In addition to being free and open-source, PostgreSQL is highly extensible. For example, you can define your own data types, build out custom functions, even write code from different programming languages without recompiling your database!

I've used pg_admin to visualize the data and tables in the database and added a single table to store the data related to the chat app. I used the default postgres database and created a new table with the columns needed in the chat data table via the GUI, along with the constraints for each. Here's the SQL version of the same:

CREATE TABLE IF NOT EXISTS public.chat_data
(
    username text COLLATE pg_catalog."default",
    msg_id text COLLATE pg_catalog."default" NOT NULL,
    msg_data text COLLATE pg_catalog."default",
    client_id text COLLATE pg_catalog."default",
    incremental_record_id bigint NOT NULL DEFAULT nextval('chat_data_incremental_record_id_seq'::regclass),
    created_at_timestamp bigint,
    is_edited boolean,
    CONSTRAINT chat_data_pkey PRIMARY KEY (msg_id)
)

The msg_id is a unique identifier for each chat message and is therefore a primary key. The incremental record ID is useful to retrieve x number of messages starting from a particular point. The is_edited column indicates if the message has been edited.

Section 3 - Lambda function setup on the Ably integrations dashboard

We use a Lambda function to insert and update rows in the chat_data table dynamically. It is triggered when messages are published on the outgoing_chat channel and that is set up in the Ably dashboard.

External services to push or receive data in realtime, can be set up in the 'Integrations' dashboard of your Ably account.

Click on "New Reactor Rule", for the options to send or receive data from external systems. The chat app needs to push an event, i.e. trigger an AWS Lambda function, every time a new message is published on a channel. Select Reactor Event > AWS Lambda to set up the Lambda function along with the AWS credentials and the source of trigger for this endpoint. All of these fields are described further in the documentation.

Choose the source as "Message" to trigger the Lambda when messages are published on a channel. In the channel filter field, we can specify the channel we'd like to use, which is outgoing-chat in this instance. Note from the previous section about the front-end NuxtJS app, we use this channel to publish both new chat messages and edited messages. As you'll see later, the same Lambda function can handle both kinds of messages.

Ably assigns a unique msg_id to each message, which is available in its metadata, so we retain the default enveloped setting to allow for this data to be added in the database.

Use the Lambda function for database transactions

AWS Lambda is a serverless compute service that usually contains a single function's code to be triggered and executed as needed. To be able to use AWS Lambda functions, you need to have an account with AWS. You can then create a new function; ours uses the NodeJS environment.

From AWS documentation for Lambdas with NodeJS:

The Lambda function handler is the method in your function code that processes events. When your function is invoked, Lambda runs the handler method. When the handler exits or returns a response, it becomes available to handle another event.

The Lambda function for the chat app performs either an INSERT operation or an UPDATE operation on the Postgres database, depending on the event data that invoked the function.

It uses the pg module to perform various operations on the Postgres table. It connects to the table by passing various parameters at the time of instantiating a new client, and calls the connect() method to establish a connection to that table within the specified database.

const client = new Client({
    host: process.env.HOST,
    port: process.env.PORT,
    user: process.env.USER,
    password: process.env.PASSWORD,
    database: process.env.DATABASE,
});

client.connect();

Next, the incoming object (the event data that invoked the lambda) is formatted as required by the database.

const msgPayload = incomingObject.messages[0];
const msgData = JSON.parse(msgPayload.data);

Depending on the type of the message --- chatMsg or editedMsg --- the Lambda performs either an INSERT operation in the database or an UPDATE operation.

a. Insert a new row for a new chat message published in the chat app:

if (msgPayload.name == "chatMsg") {
  const queryText = "INSERT INTO chat_data(username, msg_id, msg_data, client_id, created_at_timestamp) VALUES($1, $2, $3, $4, $5)";
  const queryValues = [msgData.username, msgPayload.id, msgData.content, msgPayload.clientId, msgPayload.timestamp];

  client.query(queryText, queryValues, (err, res) => {
    console.log("Error", err);
    console.log("Result", res);
    client.end();
  });

  return `insert function done, ${queryValues} payload inserted`;
}

b. Update an existing row for an edit message published in the chat app. The specific row to be edited is identified using the unique identifier msg_id which is a part of the msgPayload:

else if (msgPayload.name == "editedMsg") {
  const queryText = "UPDATE chat_data SET msg_data = $1, is_edited = $2 WHERE msg_id = $3";
  const queryValues = [msgData.content, true, msgData.msgIdToEdit];
  client.query(queryText, queryValues, (err, res) => {
    console.log("Error", err);
    console.log("Result", res);
    client.end();
  });

  return `update function done, ${queryValues} payload updated`;
}

Parameterized queries allow us to pass any parameters to the SQL queries without needing to worry about parsing special characters or in our case even emojis.

With this setup, you can test out the working of the Lambda function and database updates, using the Nuxt chat app. Whenever you send a new message, it gets published on the 'outgoing-chat' channel, which in turn triggers the Lambda function, which in turn INSERTS or UPDATES the database table with the relevant data.

However, you can see that this change actually doesn't come back to the frontend chat app just yet. We are making changes to the database, but we've not yet seen how the Ably-Postgres connector listens to these changes and publishes them to Ably. Let's explore this next.

Section 4 - Set up the Ably Postgres connector for the chat app

The Ably Postgres connector can be used via NPM or directly from the GitHub repository.

Let's explore the second option. Clone the GitHub repository using

git clone <https://github.com/ably-labs/ably-postgres-connector.git>

There are a few ways to add the configuration details of our Postgres database. The config file option applied to the chat app is as follows:

{
  "dbConfig": {
      "host": "",
      "port": "",
      "user": "",
      "password": "",
      "database": ""
  },
  "connector": [
      {
          "tablename": "chat_data",
          "ablychannelname": "incoming-chat",
          "operation": "INSERT"
      },
      {
          "tablename": "chat_data",
          "ablychannelname": "incoming-chat",
          "operation": "UPDATE"
      },
  ],
  "ably": {
      "apiKey": "API_KEY"
  }
}

The table name is set to chat_data for INSERT and UPDATE operations. The ablychannelname object indicates which channel the connector should publish into, following any INSERT or UPDATE operations done on the database/ table that the connector is watching.

Run the connector on your local machine using the following commands:

cd examples
npm i
node with-json-config.js

Now if you publish any messages or edit already published messages in the Nuxt chat app, you should see these coming back in the UI as the Nuxt app is subscribed to the incoming-chat channel where the Postgres connector publishes the messages.

Recap of the app architecture

We have now followed all the steps to enable our chat app to publish messages and receive messages on two different channels with database-driven updates enabled via the Ably Postgres connector.

Deployment

Let's take a look at how various components of the chat app are deployed to work together in a serverless and scalable way.

Nuxt static app deployment on Netlify

Netlify provides a serverless, git-based workflow to deploy web applications. The chat app is a static site, meaning the HTML, CSS, and JS are pre-built and served to the users directly from CDN. In an article I wrote earlier, I explained the JAMstack architecture in detail and the many benefits it provides:

Jamstack proposes serving statically generated websites directly from the CDN, doing away with the need for an origin server to serve the site. This can be a bit confusing. To clarify, this doesn't mean that we can't have a server at all (we could have one for the app logic), but this server won't be responsible for serving our HTML page back to the user. In fact, the best option here would be to make use of the myriad serverless platform options available out there, to avoid having to manage any infrastructure in the backend.

With Nuxt, you have built-in tooling to generate a static site that can be deployed directly to Netlify.

What other options do I have?

Netlify is my personal favourite and the fact that it comes with serverless functions out of the box is a plus (as you'll see in the following section). There are other options to deploy a front-end site built in Nuxt (or any other framework for that matter).

Ably Token-based authentication using Netlify functions

When you instantiate Ably's libraries, you need to authenticate the client using Basic authentication or Token authentication.

While Basic authentication is mainly used for demos and quick prototypes, on a production level it is important to use Token authentication to ensure security. Implementing Token auth requires developers to set up an authentication endpoint that can check the legitimacy of the client and issue an Ably Token Request. The client can then use this Token Request to authenticate with Ably and use its services.

Because we deployed the app to Netlify, it makes sense to use Netlify serverless functions to authenticate our Ably clients.

const Ably = require("ably");
const rest = new Ably.Rest({ key: process.env.ABLY_API_KEY });

exports.handler = (_event, _context, callback) => {
  rest.auth.createTokenRequest(
    {
      clientId:
        "clientId-" +
        Math.random()
          .toString(36)
          .substr(2, 16)
    },
    (err, tokenRequest) => {
      if (err) {
        callback({
          statusCode: 500,
          body: JSON.stringify(err)
        });
      } else {
        callback(null, {
          statusCode: 200,
          headers: {
            "Content-Type": "application/json",
            "Access-Control-Allow-Origin": "*",
            "Access-Control-Allow-Headers": "Content-Type",
            "Access-Control-Allow-Methods": "GET, POST, PUT, DELETE"
          },
          body: JSON.stringify(tokenRequest)
        });
      }
    }
  );
};

In this case, we use the REST SDK as we don't need to set up a persistent connection with our clients. The communication is infrequent and REST requests are thus more efficient. Note that we are not actually verifying the clients against, say, a database, for the purposes of the demo. In a real-world scenario, the front-end client would pass some kind of identification, based on which the auth endpoint would decide whether or not it's a legitimate client.

We assign a randomly generated unique clientId to each client before returning it along with an Ably Token Request.

What other options do I have?

You can use either a dedicated authentication service such as Auth0, or other serverless function providers such as AWS Lambda Functions, Google Cloud Functions, and MS Azure Functions.

PostgresDB deployment on AWS RDS

Up to this point, we have assumed the Postgres database to be running on a local machine. For the chat app, PostgresDB is deployed on Amazon's Relational Database Service (RDS). Given that we are manipulating data using an AWS Lambda function, it's easier to host the database on the same service to ensure maximum security. We can make the database part of the same VPC (Virtual Private Cloud) so all our component services have access to each other and can scale automatically as needed.

AWS RDS is a scalable system where you can choose any one of the multiple database engine options available. One such option is PostgreSQL which is what we are working with. You can create a new Amazon RDS DB instance with the Postgres option for the database engine by following the documentation on AWS. You can create a new table for the chat_data as we did locally before. If you already have some data stored in the local instance, you can also import data into your Postgres table.

What other options do I have?

There are many services built to host and work with PostgreSQL. They are listed in the Postgres documentation. Some of the popular serverless options other than AWS RDS are Heroku, Aiven, and Digital Ocean.

Ably Postgres connector deployment on AWS Fargate (via ECS and ECR)

Lastly, we need to deploy the Ably Postgres connector so it can continuously listen for any changes on the database and relay them to Ably to stream to millions of clients.

For the chat app, I've used docker-compose to create an image and stored this on AWS ECR (Elastic Container Registry) which is used to store and share docker images. We can then host it using AWS ECS (Elastic Container Service) that allows us to easily deploy, manage and scale the container.

AWS Fargate is the serverless option to work with AWS ECS. Given that the PostgresDB is hosted on AWS (RDS), having the components that listen to this database (and need to have access to the database on AWS) also helps simplify the architecture.

What other options do I have?

There are other options supported by Docker such as Kubernetes and Azure ACI to create and deploy containers for use by your applications. You can find further information in their documentation.

Further improvements

With the architecture described in this article series, we completely depend on the database for all kinds of realtime messaging within the app. In a standard peer-to-peer Pub/Sub messaging scenario, there would be no dependence on the database for realtime messaging. Both of these have pros and cons and ideally, they'd need to be used in combination to achieve the best results. For example, in the editable chat app, we could publish regular messages directly to all the users on the same channel via Ably but only let the edited updates come via the database. This would massively reduce the latency and generally promote transient events without needing permanent storage.

In terms of specific things we could improve in the current architecture and implementation:

It is possible to reuse database connections from the Lambda function, thereby reducing the latency and improving the performance of frequent changes made to the database via the Lambda function.
We could also do away with the Lambda function having to directly interact with the database by using the RDS proxy which handles connection pooling among other things.
As you may have noticed, currently the presence updates are streamed p2p, which may make it difficult to scale. To get around this, you can consider aggregating the presence state from various participants into a single message, then fanning this to everyone. Check out the Ably Presence Collection project for more details on how to enable this.
We could also replace Postgres on RDS with AWS DynamoDB and use its in-built feature, DynamoDB streams, to stream updates to our end users.
The front-end stack is all plug and play! You can conveniently replace Nuxt with any other frontend web or native mobile framework.

Summary

I hope this article series was a useful description and implementation of the database-driven architectural patterns. If you'd like to check out the chat app, you can do so at https://serverless-scalable-chat.netlify.app/.

You can also dive into the GitHub repo, remix the code or raise a PR to improve it further :)

Here's a full index of both the articles in this series:

As always, please feel free to reach out to me if you have any questions :)

Database-driven realtime architectures: building a serverless and editable chat app - Part 1

Srushtika Neelakantam — Tue, 28 Sep 2021 15:12:13 +0000

Introduction

Database-driven realtime architectures are becoming more and more common as evidenced by key backers and widespread use of software like Firebase and Supabase.

The two key priorities for an app following database-driven realtime messaging are long-term storage and change data capture (CDC) updates from the database.

In this two part article series, we'll take a detailed look at a fully serverless chat app where you can edit previously published messages. The chat app uses the Ably Postgres connector to achieve this, the details of which you'll see throughout the article. You'll find the architecture and the example app useful if you are looking for ways to build database-driven applications that work seamlessly at scale.

We've previously written about decoupling databases from realtime messaging:

The main problem is lack of modularity: each tool should do its own job. Databases are bases of data. They should do one thing and one thing well, which is to be the one base source of truth and data. They should never be saddled with realtime concerns --- that is not their strength by design. They should do storage, and a different dedicated piece should do realtime comms. All the other problems stem from this.

While the idea of a realtime database sounds great and opens up a huge range of possible use-cases one could build with this architecture, a tight coupling of databases and realtime messaging might suffer from various issues described in the article linked above.

Moreover, not all event triggers constitute consequential payloads and thus don't need to go into storage. Perhaps some events are just transient to make a client aware of an event occurring, not necessarily even descriptive details about that event. For example, in a chat app, I'd be interested in storing messages, timestamps, etc. but not necessarily typing indicator events.

In a recent article, I introduced the Ably-Postgres connector built by one of our community experts. It uses Postgres DB's listen/notify feature to listen for changes on any DB tables and publish updates on specific Ably channels whenever a change occurs. This allows us to take advantage of database-driven architectures without worrying about the scalability of realtime messaging or the awkward relationship between the two.

Check out the editable chat app: https://serverless-scalable-chat.netlify.app/

The editable chat app architecture

Let me present to you a complex looking architecture which will make more sense by the time you've worked through this article.

From an end-user perspective, they will be publishing messages on the frontend app and expect to receive messages on it as well. The same goes with editing any messages: all participants will need a way to edit their own messages and also receive updates about any messages edited by others.

A common architectural set-up when using a pub/sub messaging service like Ably is to publish updates on a channel to which the client is also subscribed. Although this works perfectly well for regular chat messages or any other event triggers, it is more complex to edit previously published messages or to trigger updates about changes to previous messages because Ably messages are immutable by nature.

It is, however, possible to implement this functionality by using a slightly non-traditional approach. Instead of subscribing to a single chat channel to which users are publishing their messages, we can separate out the incoming and outgoing chat channels. Doing this allows us to perform various operations on the data before it comes back in a subscription callback. A common use case of this architecture is message filtering like applying a profanity filter.

In the case of the current chat app, we'll make use of a database to store all the published messages directly in a table. We'll also have a listener that can

i) observe the insert, update, and delete changes in the chat data table of our database, and

ii) publish a message on an Ably channel with the name of the operation as well as with the change data capture (CDC).

If we make our front-end clients subscribe to this channel into which the listener is publishing database updates, we'll not only receive new messages as a result of insert operations in the database, but also updates on previous messages resulting from update operations on the database. Each Ably message comes with a unique msgId assigned by Ably, so we can make use of this to uniquely identify each message in the table. The database will be the single source of truth in the app and also useful if we'd like to load previous messages in the chat like in the Ably-Airtable starter kit example.

Before proceeding, take another look at the architecture diagram above to put all the steps in perspective and tie it all together.

Tech stack

We have four main goals with the editable chat app:

Scalability
Serverless architecture
Editability of messages
Storage of messages

In view of the above, let me explain some of the reasoning behind various tech choices in this chat app, along with some alternative options.

NuxtJS for frontend web development

Nuxt aims to provide best-practice solutions to common web development problems like routing, state-management, code splitting, etc. It allows us to make use of various NPM utility libraries in a static site that can be deployed and used directly from a CDN, without needing a server, i.e. following the Jamstack architecture.

In the case of our chat app, it is useful in terms of separating out state management entirely from the visual components, so developers of all tech stacks can understand the communication and data exchange between the chat app and external services (mainly Ably in this case).

You can replace Nuxt with any other front-end web framework, vanilla JS or even use a native mobile programming language, depending on the needs and wants of your app.

Ably to enable the pub/sub pattern

Ably is a realtime messaging infrastructure as a service. It allows you to enable publish/subscribe-based messaging in your application with just a few lines of code. Ably provides highly-reliable low-latency messaging, and is able to work globally on any platform or device. It completely abstracts away the complex problem of scaling realtime communications across multiple regions around the planet, so developers can focus on their app logic.

AWS Lambda functions to insert and update rows in the database

We use PostgresDB to store messages from the chat app. In general, any database transactions which change table data shouldn't be done directly from the front-end to avoid potential security risks. Hence, we'll make use of AWS Lambda functions to make changes to the database on the users' behalf. Given that we are aiming to make this app fully serverless, Lambda functions fit right in with this theme.

PostgresDB hosted on AWS RDS to store the data

Postgres is an open-sourced SQL database. Its performance and reliability make it a good choice for complex production applications. There's another special reason to choose Postgres as you'll see in the next point.

Postgres doesn't come with hosting, we'll need to make use of another service to host the database. Again, in light of keeping everything serverless, I've made use of AWS RDS for Postgres. Using AWS also gives the advantage of the accessibility of the data between other AWS services, like the Lambda function in the previous point.

Ably Postgres connector to watch changes on the database tables and publish messages on every change

One of the key requirements of this chat app is being able to listen to changes on the database tables and publish these changes to Ably. The Ably Postgres connector is a community-built project which makes this possible. We use Postgres because the built-in listen/notify feature makes this connector work. We'll take a detailed look at it later.

AWS Fargate with AWS ECS and AWS ECR to deploy the Ably Postgres connector's dockerized image

AWS Fargate is a serverless compute engine that hosts containers. The Ably Postgres connector has a dockerized image that needs to be hosted somewhere. We'll use AWS Fargate to do this, because it makes it easy and secure to manage the backend deployment and hosting on a single service like AWS.

AWS Fargate works with AWS ECS which enables deployment and management of containerized applications. We use AWS ECR to upload the docker image so it can be stored in the registry to be used by ECS as needed.

Netlify to host the static Jamstack site

Netlify provides a serverless platform to deploy web applications. It also allows setting up git-based workflows to automate building and deploying new versions of a static site as changes are made to the repository. The Nuxt app is deployed using Netlify.

Netlify functions to enable a token auth endpoint to authenticate with Ably

Netlify's serverless platform also provides serverless functions which can be invoked to perform a piece of functionality. The Ably service requires clients to be authenticated in one of the two ways: basic authentication or token authentication. Basic authentication exposes the API Key directly in the frontend script, and thus shouldn't be used in production. You should almost always choose Token authentication. To enable this, we need to set up an authentication endpoint that can verify the credentials of the frontend client and issue Ably Token Requests. The frontend client can then use this Ably Token Request to authenticate with Ably and use its service.

Given that we use Netlify to host the chat app, it's only natural that we make use of Netlify functions to host our authentication endpoint. Even though AWS Lambda is already a part of the tech stack, it would require us to set up an IAM for our users before they can access AWS Lambda. Netlify, meanwhile, makes it very easy.

The Ably Postgres connector -- a key dependency

Before moving on to the details of the chat app, let's first understand the working of the Ably Postgres connector that makes this architecture possible.

I recently wrote an article explaining the Ably Postgres connector in detail:

The connector accepts a configuration file where you input the connection details for your database as well as for the tables you want to listen to for data changes. It also accepts an Ably API key to be able to instantiate and publish messages to your Ably app following any changes to the specified tables.

Using the config file, the connector creates in your database a special table called the "ablycontroltable". This table is used to maintain the Ably channel mapping for different changes to the tables in your database.

Next, the connector creates a procedure to listen to changes on the specified tables using the pg_notify function. This notify function then publishes the change data capture (CDC) payload on the relevant Ably channel, as specified in the config.

With this, I hope you have a better understanding of the high-level architecture of the serverless editable chat app.

In the next part of this two-part series, we'll take a closer look at various components of the chat app and dive into some code to better understand how each step is implemented.

Here are a few things we'll see in the next part:

Navigating the Nuxt app (even if you are not a Nuxt developer)
VueX state management
Postgres DB setup
Lambda functions setup on the Ably Integrations dashboard
Ably Postgres connector setup
Deployment of all the components

Stay tuned and watch this space for the next article. In the meantime, check out the demo of the editable chat app or dive into the GitHub repo.

Feel free to reach out to me if you have any questions.

Resources

Myth-busting: Jamstack can't handle dynamic content

Srushtika Neelakantam — Mon, 19 Jul 2021 16:20:43 +0000

Jamstack has brought forward a great way to rethink the infrastructure of modern-day websites. It shows us just how much we can abstract away in the process of serving websites and, as a result, gain tremendous benefits to User and Developer Experience.

However, much confusion exists around what kind of websites can actually fall under this classification. The whole premise of Jamstack apps is based on the fact that these sites can be served directly from a CDN (Content Delivery Network), without needing an origin server. You might ask: “So these are static sites then? That means only pre-rendered content with nothing dynamic?” Well, that’s untrue and is one of the biggest myths around Jamstack.

In this article, we’ll understand everything about Jamstack sites with dynamic content and specifically look at a realtime synced streaming application we built (dubbed as a live watch party app) to show off the wonderful benefits of Jamstack and the APIs around us, enabling us to elevate its possibilities. This app allows you and your friends to watch a video on your respective machines, synchronously, while chatting alongside - much like the Netflix watch party.

What is the Jamstack?

Before we take a stab at explaining anything here, we highly recommend watching this video where Phil Hawksworth of Netlify takes us through a beautiful explanation of the Jamstack and why it is great.

We’ve copied over one of his slides directly from the talk:

The JAM in Jamstack stands for JavaScript, APIs, and Markup - pretty much everything we’ve been using already in most of our web apps.

So, what’s different?

It’s the way these apps are architected and served to users across the globe.

As you see in the slide from Phil’s talk - for a traditional website that is dynamically served by a web server, the journey involves a few steps at the least. Your web browser goes to the CDN to get any static assets, then to the load balancer placed in front of the web servers capable of serving that particular site. The load balancer resolves which of the available web servers is best equipped to serve the page. The selected web server then serves the page back to the user following the same path. In some cases, the web server might request some data from the database before serving back the page to the client.

In contrast to having all these components between the user and the page they want to see, Jamstack proposes serving statically generated websites directly from the CDN, doing away with the need for an origin server to serve the site. This can be a little bit confusing. To clarify - this doesn’t mean that we can’t have a server at all, we could have one for the app logic, but this server won’t be responsible for serving our HTML page back to the user. In fact, the best option here would be to make use of the myriad of serverless platform options available out there, to avoid managing any infrastructure in the backend.

Although the initial site that’s loaded from the CDN is static, containing pre-rendered assets and data, we can immediately enhance the experience and functionality by retrieving dynamic data via API calls to our own server or any third-party endpoints.

This results in many benefits, most obvious of which are improved performance and better user and developer experience.

A bit more on static sites with dynamic/ realtime data

There is a huge assumption that static sites mean static data. The static assets served by Jamstack projects can contain JavaScript files; after all the “j” in Jamstack represents JavaScript. Just as JavaScript brought dynamic data to websites in the 90s, it can still do the same today. We can use the static JavaScript files in our Jamstack projects to interact with out webpages and provide dynamic experiences for our end users - hook that up with a pub/sub or real-time infrastructure service like Ably, and we have dynamic data on the Jamstack very easily.

Let’s understand with an example

For this project, we've been working closely with Daniel Phiri and the Strapi team. It all started a couple of weeks ago when we started to build a realtime Jamstack app in public for the dev community to follow along:

So excited about this project @malgamves and I are working on! A live watch party app to host dedicated sync stream parties with your friends. We'll update this thread as we make progress #BuildingInPublic pic.twitter.com/5LnnVNzcSR
— Srushtika Neelakantam (@srushtika) April 16, 2021

The watch party app allows a host to select a video from the library and watch it synchronously with their friends anywhere on the globe.

To give you a good idea, the host would follow these steps:

Enter their username, create a private watch party room, and share an invite link with friends.

Select a video from the library to watch along with friends.

Watch the synchronized video with friends, share live comments, and see who’s currently online.

The host gets to control the video playback - if they play, the video starts playing for everyone else, same for pause, seek, and so on. If the host leaves, that’s the end of the party.

Tech stack for our watch party app

To build out this application, we’ve leveraged four pieces of technology - Nuxt.js, Strapi, Ably and Netlify. Let’s get into what each does for us in this project.

What is Nuxt.js?

Nuxt.js is a web framework based on Vue.js - besides being one of the most convenient ways to build out Vue applications, it gives us the option to build server-side rendered or static websites. For our project, we’ve gone with the static option and hosted these files on Netlify.

What is Strapi?

Strapi is an open source Node.js-based headless CMS. Since it's headless, we can use Strapi to deliver content to almost any digital device via its API. We’ve used Strapi as a video content manager. We’ve managed the videos available in our Nuxt.js app with Strapi, as well as leveraged its extra customisation options to build out a service to handle token requests with Ably.

What is Ably?

Ably is a realtime messaging infrastructure as a service solution. It allows you to enable publish/subscribe-based messaging in your application with just a few lines of code. Ably provides low-latency messaging, high reliability, and is able to work globally on any platform or device. It completely abstracts away the complex problem of scaling realtime communication across multiple regions, from the developer space.

What is Netlify?

Netlify is a cloud hosting company that offers hosting and serverless backend services for web applications and static sites. It is particularly known for bringing the Jamstack approach of building websites to the attention of developers around the world.

App architecture - bringing all these technologies together

Jamstack sort of forces you to have a simplified architecture and infrastructure for your web app. For the watch party, the static site itself (that is just the initial page that allows hosts to create private watch party rooms) is hosted on Netlify’s CDN.

Both the admin version and non admin version of the site can be retrieved directly from the CDN (based on URL routing).
Ably’s Pub/Sub platform requires you to authenticate before you can use the service. There are two options for this - either embed the API key directly into the front-end web app (which would be a bad idea because anyone can steal it), or use Token authentication by requesting an auth server to help the front-end clients to authenticate securely. We’ll use Strapi as our auth server (in addition to its beautiful CMS capabilities which we’ll touch upon soon).
After we’ve received an Ably Token Request back from Strapi, we can send it to Ably to securely authenticate with the service and initialize the SDK. This sets up a persistent realtime connection with Ably, allowing any new updates to be pushed directly to our app and vice versa. We’ll use this to synchronize the video stream, as well as to share comments and live online status of participants.
After the host has authenticated with Ably (and transparently with Strapi via dummy user credentials), they’ll be able to share an invite link with any participants they’d like to invite to their private watch party.
Next, the host will be able to request the video library from the Strapi CMS. This will show them a grid of various videos to choose from. After they’ve chosen a video, the unique reference code for that video will be immediately published to all the participant apps via Ably. These non-admin participants can then (behind the scenes) request the particular video resource directly from the Strapi CMS.
On this final screen, everyone will be able to add live comments and it is up to the host to play the video, pause it, seek it to a certain timestamp etc - all of which would be synchronized with the rest of the viewers.

Code snippets and explanations

Let’s understand the main components of the app.

Creating a video collection and adding content to Strapi

After getting started with your Strapi app, a browser tab will open and take you to the Admin Panel. Create a user and log in. Once that's done, we can start building out the content schema to store our videos. Once you’re in, navigate to Content-Types Builder under Plugins in the left-hand menu.

Click the "+ Create new collection type" link.
Name it videos and click Continue.
Add a Text field (short text) and name it title.
Click the "+ Add another field" button.
Add another Text field (long text) and name it description.
Click the "+ Add another field" button.
Add a Media field and name it video.
Click the "+ Add another field" button.
Add another Media field and name it thumbnail.
Click the Save button and wait for Strapi to restart.

Everything should look like this once done:

These fields will store the video details for your pages. Now we can go on and add content to them by clicking Videos on the left menu. Adding content should look something like this:

Hosting Strapi

Strapi is a self-hosted headless CMS. With that in mind you have an array of deployment options, most of which are laid out nicely in Strapi’s Deployment Documentation.

For our project, we deployed our Strapi App to Heroku following the guide provided. We’d recommend this approach if you want a free hosting provider that lets you provision a Postgres database for your Strapi app with minimal effort.

Retrieving info from Strapi into our app

Before we can make requests to our Strapi server, we need to make sure that we have the right permissions set up to get them. To do this, we go to your User Permissions > Roles > Public Role > click the find and find one checkbox under videos, as well as the auth checkbox under ably-auth as shown below.

For the project we’re using the Strapi GraphQL API, so we’ll have to install it with yarn strapi install graphql - you have the option to use the REST API and the Strapi Nuxt module too.

Once the GraphQL plugin is installed, we can go to http://localhost:1337/graphql to access our GraphQL Playground and play around with different GraphQL operations.

Authenticating Ably via Strapi

Ably’s realtime messaging service expects client devices to be authenticated before they can start using the service. This can be done in two ways - either by using the API key directly in the front-end app (Basic Authentication strategy) or via tokens issued by an auth server (Token Authentication strategy).

As you might have guessed, embedding the API key wouldn’t be a wise choice because it can be easily misused. To implement Token Auth strategy, we’ll need to have a backend service use a direct API Key securely on our behalf and generate a valid token request via Ably. The frontend client can then use this token request to authenticate with Ably without leaking the API Key details. You can learn more about it in a short YouTube video.

For our app, since we are already using Strapi as a CMS for our data, we’ll also make use of it as a backend service generating our token request.

We took advantage of Strapi custom controllers and built out the logic to generate a token request for Ably. This is shown below:

'use strict';

const Ably = require('ably/promises');
const ABLY_API_KEY = process.env.ABLY_API_KEY;

const realtime = Ably.Realtime({
    key: ABLY_API_KEY,
    echoMessages: false
});


module.exports = {
    async auth(ctx) {
        const clientId =  'id-' + Math.random().toString(36).substr(2, 16)
        const tokenParams = { clientId };
        try {
            const ablyThing = await realtime.auth.createTokenRequest(tokenParams);
            console.log(ablyThing)
            return ablyThing
        }
        catch (err) {
            return ctx.badRequest("Daas not good!!")
        }
    }
};

Pub/Sub messaging with Ably

Ably’s core offering is a scalable realtime messaging that follows the Publish/Subscribe pattern.

Go to the dashboard, where we can divide realtime data into different categories, based on what the data represents and which participants are concerned with that data. These different categories of data can be published on different ‘channels’ within the app. Here’s an example:

const ably = new Ably.Realtime(<auth endpoint or api key>);
const channel = ably.channels.get(‘jamstack-news’);

// Publish a message to the jamstack-news channel
channel.publish('greeting', 'hello');

// Subscribe to messages on jamstack-news channel
channel.subscribe('greeting', function(message) {
  alert(message.data);
});

In the watch party app, we’ve used the following channels:

mainParty: used mainly to share presence data (this is explained below in this article).
video: used to share updates related to the video player, including play, pause and seek events, along with the current timestamp.
comments: used to share live comments between participants of the specific watch party.

Unique channels

Given that we use the same app to allow different groups of people to spin up their own breakout room, we also need to think about a way to separate out the realtime messages for each of those rooms. To do this, we assign a unique random code to each watch party room and use that to uniquely identify channels in the same Ably app. Given that different channels can have different participants and the data from one channel doesn’t go into another, unless explicitly published, this should be a good way for us to separate out concerns.

Another option is to use channel namespaces. These are useful when we want to apply certain features or restrictions to a set of channels as a whole. As we won’t be needing that for this app, we’ve just gone with the channel names to be watch-party-<random-room-code>, video-<random-room-code> and comments-<random-room-code>.

Utilizing Nuxt’s central store for efficient fan out of messages to required components

We’ve made use of the VueX store, which comes built into Nuxt. You can find this in store/index.js. This file works as a central store for most of the data in our static site. A typical VueX store contains four objects (possibly more depending on your specific app) - state, getters, mutations and actions.

State: This is a single object containing the application level state which represents the single source of truth and allows different components to be in sync with each other.

Getters: Getters are methods that allow us to compute derived states to be used anywhere in the app.

Mutations: Mutations are methods that change the value of a certain state object. Mutations should always be synchronous - this is to ensure that we have a good view of the state changes. If you need to update the state based on an asynchronous operation, you’d use actions described next.

Actions: You’d use actions to perform asynchronous operations and call a mutation when ready to change the state as a result of that async operation.

This central store is especially useful for the watch party app, because we have various channels, the async data from which is being used in different components. And because VueJS is reactive, our components can watch for changes on any of the variables and react to them immediately with UI updates.

The key things to notice in the store for our project are listed below:

The currentVideoStatus state object:

currentVideoStatus: {
   isVideoChosen: false,
   didStartPlayingVideo: false,
   chosenVideoRef: null,
   currentTime: null,
   isPlaying: false,
   isPaused: false
},

This is a single source of information about the video being played. For the host, this is always in sync with their video player. We publish this object whenever a new non host participant joins. This is also the object published when an existing participant clicks on the ‘force sync with admin’ button.

The instantiateAbly() method:

In this method, we instantiate Ably using Token authentication. As described previously, Token authentication is facilitated by a Strapi endpoint. So, in the init method, we pass in the url of that endpoint as a value to the authUrl object. We receive a client id when the connection is successful, which we then save in a local state object.

const ablyInstance = new Ably.Realtime({
  authUrl: this.$config.API_URL + "/auth-ably"
});

The attachToAblyChannels() method:

In this method, we attach to the three channels. Note that we add the unique room code to these channel names to make sure they are uniquely identified for this watch party room, across the app.

    attachToAblyChannels(vueContext, isAdmin) {
       //mainPartyChannel
       const mainParty = this.state.ablyRealtimeInstance.channels.get(
         this.state.channelNames.mainParty +
           "-" +
           this.state.watchPartyRoomCode
       );
    // similarly for the video and comments channels

The subscribeToChannels() method:

In this method, we subscribe to the channels we previously attached to. When a new update is published on that channel, the respective callback method will be triggered. We simply update the state variables to contain the latest message that has arrived.

state.channelInstances.comments.subscribe(msg => {
  state.channelMessages.commentsChMsg = msg;
});

The publishCurrentVideoStatus() method:

This method enables the admin to publish the currentVideoStatus object we described previously.

state.channelInstances.video.publish(
  updateEvent,
  this.state.currentVideoStatus
);

The requestInitialVideoStatus() method:

This method is used by non admin participants to request the latest video status. This is invoked once at the beginning when they’ve just joined, then again whenever they click on the force sync button.

requestInitialVideoStatus({ state }) {
   state.channelInstances.video.publish(
     "general-status-request",
     "request"
   );
},

The publishMyCommentToAbly() method:

This method publishes the user's comments. This will be displayed in the list of comments next to the video player.

publishMyCommentToAbly({ state }, commentMsg) {      state.channelInstances.comments.publish("comment", {
    username: state.username,
    content: commentMsg
   });
},

The utility methods are self explanatory but the rest of the methods are described in the next section.

Presence

Presence is an Ably feature that you can use to subscribe to realtime changes to a device or client’s online status (aka their connection status). Presence allows us to see who is currently online in the watch party room. This information is displayed in a tab next to the live comments. A live counter of the number of people online is also displayed above the video player for a quick look.

Here’s some explanation of the presence related methods in the store:

The getExistingAblyPresenceSet() method

Apart from a live subscription to ongoing presence updates, we also need a list of people who were already there when a user joins. In this method, we perform an API request to get the existing presence set.

this.state.channelInstances.mainParty.presence.get((err, members) => {....});

The subscribeToAblyPresence() method:

In this method we set up a subscription to presence on the main party channel and invoke various methods to handle new people joining or existing people leaving.

this.state.channelInstances.mainParty.presence.subscribe("enter", msg => {....});
       this.state.channelInstances.mainParty.presence.subscribe("leave", msg => {....));

The handleNewMemberEntered() and handleExistingMemberLeft() methods:

In these methods we update our local array with the latest presence set information and also update our local counters reflecting the aggregate number of people present in the watch party at any given time.

The enterClientInAblyPresenceSet() method:

In this method, we make the current client enter the presence set on the main party channel. This will publish an update to everyone else who is subscribed to the presence set and also include this user in the global presence set.

History and Rewind

Given that Ably is a pub/sub messaging service at its core, almost all of the messaging is transient. While Ably doesn’t store messages in the long term, it does provide storage options up to a certain extent. For example, you saw in the previous sections we were able to retrieve the presence set via an API call. Similar to that, even for regular messages on regular channels, Ably offers two ways to retrieve previously published messages:

History - an explicit API call to request previously published messages.
Rewind - allows you to get some previously published messages at the time of attaching and subscribing to a channel for realtime messages.

We can use rewind on the comments channel so that all the participants are able to see the comments published even before they join the watch party. With rewind, we can either specify a time period or number to indicate how many previously published messages we’d like to retrieve.

Hosting static sites with Netlify

As mentioned above when we introduced the service, we’ll deploy our watch-party app to Netlify!

To start out, create a Netlify account and make sure your project source code is hosted on GitHub. Click “Create new site from Git” and connect your GitHub to Netlify. Select your repo and fill in the details. Under Basic Build Settings, your build command should be yarn generate, and your publish directory should be dist. Select Advanced Settings and define your environment variables, add API_URL to key, and replace with the URL of your deployed Strapi app.

It’s worth noting that should you have both your Strapi app and watch-party apps in a monorepo configuration (both apps in the same Git repository) like our repository, then you need to add a base directory as shown below. These settings are available in Site Settings under Build & Deploy.

Should you have any trouble, you can reference the Nuxt documentation on deploying to Netlify.

Add-on options via Webhooks to further enhance our app

In terms of your product-specific custom architecture, you may want to add other components (such as a database), maybe trigger a cloud function to perform some computation, or even stream messages to a third-party service. Ably provides easy ways to integrate with external APIs and services via webhooks, serverless functions, message queues, or event streaming. You can also use incoming webhooks to trigger a message on an Ably channel from an external service. (Think of a scenario where you allow participants to answer your quiz via SMS messages!)

Conclusion

We’ve built a realtime Jamstack app and busted the myth. Jamstack CAN handle dynamic content. Jamstack is a great concept and works well if applied correctly.

I hope this article has given you a good view into realtime Jamstack apps, and got you quickly up and running with Strapi and Ably. It has been great to collaborate with Daniel on this project! We've also done a webinar together: Realtime data on the Jamstack with Ably and Strapi, where we've talked about the watch-party app and done some live Q&A.

You can check out the watch party yourself at: https://jamstack-watch-party.ably.dev/. Have feedback or want to exchange ideas? You can always find me on Twitter: @Srushtika. Happy to any questions too, my DMs are open!

Using Airtable as a database to store realtime messages

Srushtika Neelakantam — Wed, 16 Dec 2020 13:18:43 +0000

In this article, we'll see how to use Airtable to store realtime messages using a group chat app as an example. We'll use Ably's realtime infrastructure to power the chat app and make use of WebHooks to publish messages to Airtable directly in the correct order from Ably.

Check the full source code of the group chat app written in VueJS on GitHub and the live demo of the application at https://realtime-chat-storage.ably.dev/

What is Airtable?

Airtable describes itself as 'Part spreadsheet, part database, and entirely flexible' and that's exactly what it is to the word. It caters to the engineering and commercial departments in an organization alike with its robust REST API and very nice visual UI with custom fields to manage and represent the data. It combines a bunch of different tools like task managers, databases, CRMs, spreadsheets, etc, into a single product.

Airtable REST API

Airtable comes with a simple REST API to perform the basic CRUD operations on the data stored. You'll need to have a base i.e. a table/ sheet set up before you can check out the documentation. This is for a good reason - their whole documentation is dynamically displayed with real keys, ids, column names etc, along with sample responses, based off your data, making it super easy for you to just copy out the code and use as is. They provide this documentation in cURL and JavaScript. The JS code snippets require using the Airtable JavaScript Client SDK. Here's a look at the documentation for the chat app base.

How to use Airtable as a database

In this example, we'll look at two operations - to store and retrieve data from Airtable. We'll make use of WebHooks to send 'Create records' REST requests to the Airtable API each time a new chat message is published. We'll then make use of 'List records' to retrieve previously stored messages upon user request. Check out a subset of the database so you have an idea of the schema, or to simply put it, the column names in our database table/spreadsheet.

Each new message will have a unique (randomly created) msgId. This will be our primary key. The data is pre-ordered in ascending order by the ID column, which is an incremental number assigned to every new record automatically by Airtable.

Realtime updates with Ably and WebHooks

If you already use Ably, you can skip this section, if not, you can get started by creating an account. Ably provides a reliable realtime messaging infrastructure with high scalability. It primarily operates over WebSockets and provides Pub/Sub messaging infrastructure out of the box. It is protocol and platform agnostic in the sense that you can use it with WebSockets, MQTT or SSE, and with any language and platform that you are working with. You don't have to spend time understanding the hard distributed systems problems it solves but simply start publishing and subscribing to realtime data with a mere couple of lines of code.

We'll make use of Ably's JavaScript Realtime SDK to power the chat app and the WebHook Integration feature to integrate Airtable directly with an Ably app.

In terms of flexibility, Airtable and Ably are a perfect match as you can use both these platforms in exactly the way that suits your custom use-case.

Ably's Pub/Sub messaging is implemented using the concept of 'channels'. Each Ably app can have any number of channels where each channel carries a group of information. For example, in a logistics app, you would have one channel for location updates of the fleet and another for job updates to inform any changes to delivery conditions etc. Depending on the data, you can set permissions as to who can publish or subscribe to the data on that channel by attaching to it. You can learn more about channels in the official documentation.

What are WebHooks?

In simple terms, webhooks are user-defined HTTP callbacks (or small code snippets linked to a web application) that get triggered when specific events take place on an external website or service. They're especially useful when you're building notification functions and event-driven responses in applications. You can learn more about WebHooks in the conceptual deep-dive article.

WebHooks are a great fit for our use case - sending a message to an Airtable base as a result of an event i.e. a new chat message published on a specific channel. If you go to the Reactor tab on the Ably app dashboard after logging in/ signing up, you should be able to create a 'New Reactor Rule' and select the Reactor Event > WebHook option. In a reactor rule, you essentially configure an HTTP endpoint along with the relevant headers, format, etc. You then select the source of the event trigger. There are a few options here - 'Presence', 'Message' and 'Channel Lifecycle. All we need is a regular 'Message' in this case.

You'll also see options to batch the requests or envelop them with Ably metadata. You can choose the batch option if you expect the request to be triggered at high frequency. That'll prevent you from hitting the rate limit on Airtable, which at the time of this writing is 30 requests/ sec. We won't be enveloping the message with Ably metadata as Airtable expects the requests to be in a certain format exactly.

Bringing it all together in a group chat app built with VueJS

The group chat demo is written in VueJS. Here's an illustration to better understand how all the components fit together:

In terms of the folder structure you see in the GitHub project, the following are the main files of interest to us in this article.

ably-airtable-storage\
| __ src\
| __ | __ App.vue\
| ______ | __ components\
| __________ | __ infobox\
| __________ | __ chatbox\
| ______________ | __ ChatCard.vue\
| ______________ | __ UsernameInput.vue\
| ______________ | __ ChatMessage.vue\
| ______________ | __ ChatInput.vue\
server.js

The ably-airtable-storage folder holds the VueJS app whereas the server.js file in the root serves the VueJS app and issues auth tokens to the front-end app to authenticate with Ably. (More on this later)

As you saw in the live demo, we also have an information box on the side which shows the play-by-play of behind the scenes as you use the app. You can make use of that to understand what exactly is happening in each step and use the code snippets to try it out yourself. The code for this is in the infobox folder under the components folder. We won't be discussing much about the information box in this article.

Let's take a look at what's going on in the rest of the files.

server.js

This is a super simple Express server which serves the index.html page from the dist folder of the Vue app. The dist folder is generated when you run the build command after you are done working on the Vue app. You can learn more about this in VueJS docs.

You'll notice that we also have an /auth endpoint. As mentioned before, this is to issue tokens so the Vue app can authenticate securely with Ably's realtime service. Ably offers two ways of authenticating - Basic Auth and Token Auth. Basic auth uses the API Key directly whereas token auth expects auth tokens or JWT, making it a more secure way of authenticating the front-end applications. You can learn more about each of these types and trade-offs in Ably's documentation and best practice guide.

The VueJS chat app

App.vue

This is the main parent component for the entire app. So, a good place to instantiate and manage the connection with Ably.

We instantiate Ably in the created() lifecycle hook of this component and disconnect in the destroyed() lifecycle hook:

  created() {
    this.ablyRealtimeInstance = new Ably.Realtime({
      authUrl: "/auth",
    });
    this.ablyRealtimeInstance.connection.once("connected", () => {
      this.myClientId = this.ablyRealtimeInstance.auth.clientId;
      this.isAblyConnected = true;
      this.chatChannelInstance = this.ablyRealtimeInstance.channels.get(
        this.chatChannelId
      );
    });
  },
  destroyed() {
    this.ablyRealtimeInstance.connection.close();
  },

The authUrl object sent to the Ably.Realtime instance prompts Ably that we are looking to authenticate via token auth via the given URL to automatically renew tokens just before they expire.

After the connection status becomes connected, we get an instance of the channel to subscribe to later. If you remember from the previous step, we'd need to use the chat-airtable channel name for publishing and subscribing to the chat messages as that is the channel we are using to trigger messages sent to the Airtable database. If you notice, the full name we specify, however, is [?rewind=2m]chat-airtable. The channel name is preceded by some meta-information enclosed in the square brackets. The channel param used there is Rewind with a value set to 2 minutes. This allows you to get any previously published messages in the last 2 minutes before successfully establishing a connection to Ably and attaching to the channel. You can learn more about all the available channel params from Ably's docs.

2. ChatCard.vue

This is the parent component for the Group chat app, so we subscribe to updates on the chat channel here:

created() {
    this.isReadyToChat = false;
    this.chatChannelInstance.subscribe((msg) => {
      this.handleNewMessage(msg);
    });
},

We subscribe to the chat channel and call a new method to handle the new message every time the callback is invoked. More on this shortly.

This component has three child components:

UsernameInput.vue - accepts user's name before they join the chat
ChatInput.vue - accepts users' chat message if they'd like to send one
ChatMessage.vue - shows all chat messages in the group chat

The parent component has quite a few regular methods as well, here's a break down of each:

i) The saveUsernameAndJoin() method

saveUsernameAndJoin(username) {
  this.clientUsername = username;
  this.isReadyToChat = true;
  this.chatChannelInstance.presence.enter(username);
  backgroundEventBus.$emit("updateBackgroundEventStatus", "join-chat");
}

This method is invoked from the UsernameInput.vue component and saves the username entered by the user. Ably's Presence feature allows you to see the realtime connection status of any client. This is useful to see which users are currently online. We make this user enter the presence set with their username in this method. The backgroundEventBus is a VueJS state management mechanism to emit various events to the infobox component.

ii) The handleNewMessage() method:

async handleNewMessage(msg) {
  let messageContent = msg.data.records[0].fields;
  let msgTimestamp = msg.timestamp;
  await this.chatMsgsArray.push({
    messageContent,
    msgTimestamp,
    msgType: "live",
  });
  if (this.$refs.chatMsgsBox) {
    let divScrollHeight = this.$refs.chatMsgsBox.scrollHeight;
    this.$refs.chatMsgsBox.scrollTop = divScrollHeight;
  }
  if (messageContent.clientId != this.myClientId && this.isReadyToChat) {
    backgroundEventBus.$emit(
      "updateBackgroundEventStatus",
      "live-msgs-loaded"
    );
  }
}

Continuing from the channel subscription, this method is called for every new message pushed on chat the channel. We extract the required fields from the message and push it into the chatMsgsArray which is used to display messages in the chat screen. This is a live msg (vs one that is retrieved from a database).

iii) The loadPreviousMsgs() method:

loadPreviousMsgs() {
  if (this.chatMsgsArray[0]) {
    this.getMsgsFromDBWithMsgID();
  } else {
    this.getLatestMsgsFromDB();
  }
}

This method is called when the 'load previous messages' popup is clicked. It checks if there is a previous message present in the chat array or not. Accordingly, calls other methods to retrieve messages from the database.

iv) The getMsgsFromDBWithMsgID method:

getMsgsFromDBWithMsgID() {
  this.latestMsgId = this.chatMsgsArray[0].messageContent.msgId;
  this.showLoadMoreBtn = false;
  setTimeout(() => {
    this.showLoadMoreBtn = true;
  }, 500);
  this.base = new Airtable({
    apiKey: configVars.AIRTABLE_API_KEY,
  }).base(configVars.AIRTABLE_BASE_ID);
  let vueContext = this;

  this.base("Table 1")
    .select({
      view: "Grid view",
      filterByFormula: "SEARCH('" + vueContext.latestMsgId + "',{msgId})",
    })
    .eachPage(function page(records, fetchNextPage) {
      const latestRecordID = records[0].fields.ID;
      vueContext.dbAutoNumber = latestRecordID;
      if (latestRecordID) {
        vueContext.getMsgsFromDBWithAutoID();
      } else {
        fetchNextPage();
      }
    });
}

This method is invoked when there's a previous message present in the array. Note that all the records in the database are pre-ordered chronologically with an auto incrementing ID field. We use the msgId of the earliest message to find that record's ID in the Airtable database, then send another request to retrieve three records with ID less than the previously retrieved record's ID. This is done in the getMsgsFromDbWithAutoID() method added next:

getMsgsFromDBWithAutoID() {
  let vueContext = this;
  this.base("Table 1")
    .select({
      maxRecords: 3,
      view: "Grid view",
      filterByFormula: "({ID}<" + vueContext.dbAutoNumber + ")",
      sort: [{ field: "ID", direction: "desc" }],
    })
    .eachPage(
      function page(records, fetchNextPage) {
        records.forEach(async function(record) {
          await vueContext.chatMsgsArray.unshift({
            messageContent: record.fields,
            msgTimestamp: 123,
            msgType: "db",
          });
          backgroundEventBus.$emit(
            "updateBackgroundEventStatus",
            "db-msgs-loaded"
          );
          if (vueContext.$refs.chatMsgsBox) {
            vueContext.$refs.chatMsgsBox.scrollTop = 0;
          }
        });
        fetchNextPage();
      },
      function done(err) {
        if (err) {
          console.error(err);
          return;
        }
      }
    );
}

We add each of the retrieved records at the front of the chatsMsgsArray so they appear at the top of the chat list in the UI as the messages are ordered chronologically.

v) The getLatestMsgsFromDB() method:

getLatestMsgsFromDB() {
  this.base = new Airtable({
    apiKey: configVars.AIRTABLE_API_KEY,
  }).base(configVars.AIRTABLE_BASE_ID);
  let vueContext = this;
  this.base("Table 1")
    .select({
      maxRecords: 3,
      view: "Grid view",
      sort: [{ field: "ID", direction: "desc" }],
    })
    .eachPage(
      function page(records, fetchNextPage) {
        records.forEach(async function(record) {
          await vueContext.chatMsgsArray.unshift({
            messageContent: record.fields,
            msgTimestamp: 123,
            msgType: "db",
          });
          backgroundEventBus.$emit(
            "updateBackgroundEventStatus",
            "db-msgs-loaded"
          );
          if (vueContext.$refs.chatMsgsBox) {
            vueContext.$refs.chatMsgsBox.scrollTop = 0;
          }
        });
        fetchNextPage();
      },
      function done(err) {
        if (err) {
          console.error(err);
          return;
        }
      }
    );
}

This method is invoked if there were no messages in the chatMsgsArray, meaning there was no earliest record to reference. We simply need the last three messages available in the database. The previous option can be combined with this as the filterByFormula field is the only differentiator, but it's added in two separate methods to make the two cases evidently clear.

3. ChatInput.vue

As mentioned before, this method manages the input box to add a new chat message. It has a single method that is invoked when the send button is clicked:

publishMessage() {
  if (this.myMessageContent != "") {
    const uniqueMsgId =
      "id-" +
      Math.random()
        .toString(36)
        .substr(2, 16);

    this.msgPayload = [
      {
        fields: {
          clientId: this.myClientId,
          msgId: uniqueMsgId,
          username: this.clientUsername,
          "chat-message": this.myMessageContent,
        },
      },
    ];

    this.chatChannelInstance.publish("chat-msg", {
      records: this.msgPayload,
    });
    backgroundEventBus.$emit("updateBackgroundEventStatus", "publish-msg");
    this.myMessageContent = "";
  }
}

In this method, we compute a random (unique) id to assign to the message and publish it on the chat channel with the message copy and other information like the clientId and username. As the echoMessages Ably client option is turned off by default, the same client also receives this message as a subscription update on the channel, leading to that message being added to the array and ultimately appearing in the UI.

As the UsernameInput.vueand ChatMessage.vue components are pretty much self-explanatory with minor data transformation and display, we'll skip explanations on those.

With that, we've closed the full loop of data transfer from publisher to subscriber to the database and back to the subscriber. Here's the link again to the live demo so you can check it out again and piece the above information together: https://realtime-chat-storage.ably.dev/

Group chat app with Ably and Airtable

Exploring other ways to get data from Airtable into Ably

You might say it's all working fine, why explore of other ways? While we can publish messages directly into Airtable and retrieve those message again from the front-end app, we have a few gaps in this project stopping it from being production-ready.

What if for whatever reason, someone adds a message in Airtable? We won't be able to show those new messages in the chat app until the whole thing is refreshed and refreshing is not fun and a no-go when dealing with realtime data. While Airtable is not a realtime database i.e. it doesn't push any changes out, we have a work-around to this problem. Enter, Zapier!

Using Zapier and Ably to convert Airtable into a realtime database (well, kind of)

Zapier is a workflow management application which connects two or more SaaS platforms to share event-driven data. We can connect Airtable and Ably on Zapier and have it publish a message to a given Ably channel when a new record is added in the Airtable database. It would like something like this:

You can decide if you want these updates published on the same chat channel or a different one and manage those updates accordingly. A quick note here is that you can publish different events on the same Ably channel to differentiate different types of updates. You can learn more about event name object in the publish method in Ably docs.

Replacing REST requests with GraphQL

If you followed through the explanation for the chat app, you know that if we want to retrieve messages from a given point in the database, we'll need to send two subsequent requests to get the actual required data. Don't worry if you skipped through that entire section, I understand it was long :) You can just look for the phrase getMsgsFromDBWithMsgID and you'll land in the right section that I'm referring to here.

We can optimize that process by replacing the REST requests with the popular kid on the block - GraphQL! While it's not supported officially by Airtable, Thomas Cullen built a GraphQL plugin for Airtable as a community contributed project and it's perfect for this scenario. You can check it out on NPM.

Summing it up...

Airtable and Ably are great services with just the right level of flexibility in terms of usage. We saw how to publish and subscribe to realtime messages using Ably and automatically have those messages stored in Airtable in realtime using the WebHooks Integrations feature.

We also saw how to retrieve just the required messages from Airtable and display them for the user. We further explored other options of retrieving and publishing data to Airtable.

Live demo: https://realtime-chat-storage.ably.dev/

GitHub project: https://github.com/ably-labs/ably-airtable-starter-kit

Hope this post was useful. If you are building something with Airtable and Ably, I'd love to see your project and give it a shoutout. And of course happy to help you with any questions or concerns. You can raise them at devrel@ably.com or DM me on Twitter.

A scalable, realtime quiz framework to build EdTech apps

Srushtika Neelakantam — Fri, 23 Oct 2020 15:38:41 +0000

Hello developers!

TL;DR

I built a realtime quiz framework so developers building EdTech apps can quickly get started with scalable realtime messaging and focus on their app logic. It is a fully customizable framework built with NodeJS, VueJS, and powered by Ably's realtime infrastructure which primarily operates on WebSockets.

It also implements Node JS worker threads to simulate multiple 'quiz rooms' (aka dedicated servers spooled up on-demand) so different groups of people can simultaneously participate in different live quizzes organized by their dedicated hosts. You can check out the framework on GitHub at https://github.com/Srushtika/realtime-quiz-framework or try out the live demo at https://quiz.ably.dev/

What is EdTech?

One of the positive outcomes of this new world we are living in, is the rise of EdTech. Educational technology (EdTech) is the combined use of computer hardware, software, and educational theory and practice to facilitate learning. The origins of EdTech can be traced back to the late 90's and it has grown slowly but steady year after year, but it all changed in 2020 as 1.57 billion learners in 190 countries moved from classrooms to interactive, digital environments during Covid-19. This has accelerated EdTech’s existing growth, driving transformation that’s here to stay.

This incredible growth didn't come without new demands and requirements of innovation, and, a stepping stone to innovation is realtime communication: between students and teachers for synchronous learning, and between devices and the cloud for safety, security, and cheating prevention.

How a realtime quiz framework can help EdTech apps

Most of the EdTech tools and platforms have live collaboration features. If you think about, most of the collaborative realtime apps follow a common pattern and feature set, with customizations for their specific scenario.

For a live quiz, which can double up as a test-taking app for a class of high schoolers, or simply a movie trivia for a fun virtual Pub Quiz Friday with your workmates, there are a lot of common patterns without even needing much customization.

For example, they need to be able to create their own private quiz room and invite other people to that room. In most cases, one of the participants needs to have special admin controls (mostly the host). They need to be able to easily control the progression of the quiz. And so on...

Keeping in mind these commonalities and to avoid the need for developers working on EdTech platforms to build everything from scratch, I built an open-sourced realtime quiz framework to quickly get a working quiz/ test-taking platform up and running in no time. Developers can easily customize this extensible framework as per their specific use-cases and scenarios.

If you want to skip the details and take a look at the live demo, check it out at https://quiz.ably.dev/

Some background before I jump into the details

In the last few months, I worked on a few browser games and realized there were common patterns in all these realtime games that needed continuous high-speed streaming of data between the players and the game server.

The architecture was similar but the type of input from the players, and the game logic on the server changed according to the game in question. So I put all these observations together and built an arbitrarily scaling networking framework to build realtime multiplayer games. This proved to be useful to a bunch of people to set up realtime networking in no time and focus on customizing it per the game logic.

As I moved on from games, I did some research to see 'what's hot' currently and, as depressing are a lot of events around the world this year, a super positive and refreshing trend has been an increase in virtual live collaboration. This led me to think about e-learning in terms of various institutions conducting their classes and tests online. So I played around with a bunch of such EdTech platforms, and again, observed a lot of common patterns everywhere. I put these observations together to build this extensible realtime quiz framework. I hope this is a useful starting point if you are wanting to build an EdTech tool of any kind.

So, what's this framework all about?

The realtime quiz framework is a fully open-sourced full-stack project built with Node JS and Vue JS and the architecture consists of the Websockets protocol and the Publish/Subscribe messaging pattern. It's a working app with the basic functionality required to build a realtime quiz app.

To be honest, calling it a quiz app is a bit generic as the architecture would be the same for a test-taking app for an institution, a HQ style live trivia app, or a more one to one quiz like Quiz Up. As the framework is using Ably's Realtime infrastructure for realtime messaging between various components, it is readily scalable to an enterprise level.

The framework shows two ways of conducting a quiz.

You can have the questions stored in your server or have the end-users easily upload their own questions by sharing a link to their Google Sheet. You can easily extend this to attach a database for a more persisted data storage and allow a greater range of already available quizzes to select from.

The idea is that the host of the quiz and other players are already on a video call (on another platform). The host shares their screen invites other players to their quiz using a shareable link, and manages the progression of the quiz throughout. They'll be able to see the players with dummy avatars but real nicknames show up in a list as they join.

When the host feels that the expected players have joined, they can start the quiz. After a short timeout, the questions start showing up. The players can see the question, an image if available, and four options. The four options are buttons and the players can choose one of them to register their answer for that question. The interface is a bit different for the host in that they'll also see the question, optional image and four options but these are not clickable as the host is essentially not participating in the quiz themselves.

In addition to this, the host also sees a live stats panel which shows the number of players online and out of those the number of players who've already answered that question. You can easily extend this to show the names and avatars of those players or any other live stats like the high score etc.

Each question shows up for 30 seconds but if all the players have answered the question, the rest of the timer is skipped. One interesting thing to note about the timer is that every second update in the timer is a result of the data coming in from the server-side. This not only ensures that end-users have no way to tweak the timers and such in the client-side app, but also that all the participants in the quiz along with the host are fully in-sync. After a question's time has elapsed, the leaderboard info so far can be seen on the host app. The host then has an option to show the next question or end the quiz midway through. If you think of a pub quiz scenario where after each question, you have a random banter about how people thought their answers were correct, this gives the proper control to the host to allow time for that banter and show the next question only when everyone is ready.

In terms of the answer after each question, the host can see on their screen the correct answer to the previously displayed question. The players, in addition to being able to see the correct answer, will be prompted if the answer they chose was correct or wrong.

After all the questions in the chosen quiz have finished playing, a final leaderboard containing the scores of all the participants will be displayed on the host app.

As this is a base framework, each component can be easily customized! The framework is available on GitHub along with a detailed README.md on getting it working and customizing it. It also has a TUTORIAL.md if you'd like to understand in much more detail what's happening in every method of the app.

The live demo is also available at https://quiz.ably.dev/

I hope this framework is useful for your realtime EdTech adventures or anything else and I can't wait to see what you build with it! Please feel free to share it with me and Ably on Twitter and we'll be happy to give it a shoutout.

If you have any questions, feel free to DM me on Twitter or reach out to the support team at Ably.

Stay home, stay safe, and have fun virtually!

Header image illustration credits: School vector created by pch.vector - www.freepik.com

Presenting a realtime communication framework to build multiplayer games

Srushtika Neelakantam — Fri, 04 Sep 2020 07:26:25 +0000

Hello game developers!

TL;DR - I built a realtime communication framework so game developers can build multiplayer games without needing to worry about the networking side of it and instead focus on their game logic. It is powered by Ably's realtime infrastructure which primarily operates on WebSockets and is built in Node JS and Vanilla JS (use it with any JS framework you like 😉). It also implements Node JS worker threads to simulate multiple 'game rooms' (aka dedicated server spooled up on-demand) so different groups of players can simultaneously play the game. You should check it out: https://github.com/Srushtika/multiplayer-games-scalable-networking-framework

A few months back I got into the world of building high-frequency (realtime) multiplayer games to get my boss's attention as he's extremely passionate about game development (Challenge: Try to have a tech conversation with Ben Gamble, without game dev coming up!🌚🕹).

When I started, I had so many misconceptions about game dev. I underestimated certain aspects and had misconceptions about scalable game architectures. But I've learned so much about game dev since then and have built multiplayer versions of a few classics such as Multiplayer Space Invaders and Multiplayer Flappy birds. I've also written and spoken about game dev quite extensively on various platforms.

For each such presentation, I made improvements to the games by not only making them more efficient but also having them use the best and latest features of the underlying programming language. For example, I used Node JS worker threads (which was released as a stable version in January 2020) to simulate the idea of having multiple game rooms so different groups of people can play the game simultaneously. This is analogous to spooling up on-demand dedicated servers for each game. Ably's namespace feature made this strategy possible by ensuring none of the rooms had any access to the data from the other.

More games = repeating boilerplate? 🤯🥵

While I was continuing to think of more multiplayer game ideas that were cool enough to convince my team to let me work on them during Ably working hours 🌚, I realized some commonalities in all these projects:
All of my multiplayer games so far follow the Client/Server game strategy. Hence, basic networking architecture is exactly the same and has the following aspects:

The server maintains the game state and publishes it at high frequency to a channel that all the players are subscribed to. This ensures all players are in-sync.
Each player sends their current state (based on user input, etc) on a unique channel meant for that client. The server is subscribed to this and uses this info to maintain the up to date game state mentioned above.
The game has multiple game rooms.
The host player has options to start and end the game.
In general, there's awareness of every player's score, position, alive/dead status, join/ leave updates for the game, etc.

Come to think about it, it seems that this is pretty much what most real-time multiplayer games would need. So, I thought building a networking framework would benefit game developers to quickly add the multiplayer functionality to any game. All the more useful when it's built with Ably, as it means you can easily scale it and have access to so many other realtime communication features (like message ordering, availability, guaranteed message delivery, etc) So, here it is!

Multiplayer games scalable networking framework

This framework serves as a starter kit that allows you to add multiplayer functionality (that follows the Client/Server strategy to your game. It provides a communication framework so that your players can communicate with a central server, in realtime, for the entire duration of the gameplay.

It also allows you to implement a 'game rooms' feature using Node JS worker threads, which is analogous to spooling up dedicated servers on-demand, allowing you to spin up multiple instances of the game, each with a separate group of players. It comes with a skeleton demo app where this game functionality can be simulated. The idea is for game developers to take this framework and add their game logic to make it their own!

You can find the full project with a detailed guide on how to use it on GitHub: https://github.com/Srushtika/multiplayer-games-scalable-networking-framework

If you have any questions/ suggestions, etc. Feel free to shoot an email to devrel@ably.io or reach out to me directly on Twitter, I'll be so glad to talk about this 😃

If you like it, star it ⭐️ and don't forget to share any multiplayer games you build with this, I'd love to check them out and give a shoutout! 🥇🚀🔥

Building a realtime multiplayer browser game in less than a day - Part 4/4

Srushtika Neelakantam — Fri, 12 Jun 2020 10:04:20 +0000

Hello, and welcome to the final part of this article series where we are looking at the step-by-step implementation of a realtime multiplayer game of space invaders with Phaser3 and Ably Realtime. 🚀

Here's the full index of all the articles in this series for context:

In this article, we'll finish up the client-side code to render the game and also add the home and leaderboard screens for our game.

If you recall, in the first article we added the GameScene class and defined the preload() method in it. We also added the create() and update() methods but didn't define them fully.

Let's start by adding some variables that we'll use later. Add these at the top of script.js (which should be inside the public folder:

Make sure to update the BASE_SERVER_URL with your server's URL. If you have hosted the game locally, this URL would be your localhost with the port number.

Next, we'll have the client connect to Ably and subscribe to the channels. To do that, go to the Ably dashboard and add the following code, right below the variable declarations in script.js

One of the key things to note here is the gameRoom.presence.enter(myNickname); method. Ably uses a concept called Presence to determine connected clients in an app. It fires an event whenever a new client joins, or when an existing client leaves or updates their data.

Notice that this is where we instantiate a new game object with the GameScene that we started defining in the first part. So, let's resume that. The create() method of the class should now look as follows:

We had already defined the this.anims.create()method in the first article. Just above that, we add and initialize a few variables. We then subscribe to the game-state and game-over events on the gameRoom channels.

When we get a game-state update, we update the client-side variables as per the latest info from the server.

When we get a game-over update, we store the leaderboard info in local storage. We then unsubscribe the client from all channels and simply switch to a new webpage because either someone won or all players became dead.

Let's look at the update() method next:

In the update method, we move the existing game objects in accordance with the latest info. We also create new avatars for the newly joined players, and kill avatars of any player that has died by calling the explodeAndKill() method. We also update the score, and flash the join and leave updates in the <p> elements outside the game canvas.

If the server says a player just died, we call the explodeAndKill() method that will perform the explode animation and destroy that player's avatar.

A bullet gets fired once for every five game ticks. So the server either sends a blank or a bullet object, with a unique ID and a position that matches the ship's y-axis level. If it hasn't already been shot, its toLaunch flag will be true. So we check that and create a new bullet by calling the createBullet() method. Otherwise, we'll move an existing one.

We also check if the player has pressed the left of right keys via the publishMyInput() method.

Let's define these methods next. Please note that these methods are part of the GameScene class.

In the createBullet() method, we add a new bullet object according to the latest position of the ship and add this bullet to the visibleBullets associative array that's part of the GameScene class. We also add an overlap method for the current player's avatar and every bullet we add. This method will keep track of the overlap of the two game objects overlapping. When that occurs, Phaser will invoke a callback method, which in this case is publishMyDeathNews(). We'll define that later.

In the publishMyInput() method, we check if the left or right key was pressed, and if yes, publish that info to Ably. It's worth noting here that we never move the avatars directly as a result of user input. We publish this info to the server, which in turn fans it out to all the players, including the current player, resulting in a perfect state synchronisation. This communication happens so fast that it doesn't really feel any different to the user playing the game.

In the explodeAndKill() method, we create a new instance of the Explosion class. We haven't defined that yet, so let's take a brief detour from the script.js file that we've been working on, to add it. Create a new file in the public folder, call it explosion.js and paste the following code in it.

This class extends Phaser.GameObjects.Sprite and plays the explode animation that we defined in the create() method of our GameScene class in the script.js file.

Now let's get back to script.js and define one last method within the GameScene class, the publishMyDeathNews():

This method is invoked when a bullet object overlaps with the current player's avatar, meaning the player has been shot. When that happens, we simply publish this information to the server so it can update the game state accordingly and fan this information out to all the clients, including the current player, so they can update their respective game states accordingly.

We are all done with the game implementation. We just have to add the home and leaderboard pages to make the game more complete.

Adding the home and leaderboard pages

In the views folder, add four files:

gameRoomFull.html

intro.html

winner.html

gameover.html

In the public folder, add three files:

nickname.js

winner.js

gameover.js

The gameRoomFull.html is displayed when anyone tries to join the game after the preset maximum number of players have already joined.

The intro.html file gives the user a simple text box to enter their nickname. This info is used to flash join/leave updates and also show the info in the leaderboard.

The winner.html page is shown if the game ends due to a player winning the game. This page will then display their nickname as the winner and also show the first and second runners up.

The gameover.html page is shown if all the players in the game die. This page just shows the nicknames of the top two scorers.

The related JavaScript files simply retrieve the info from the local storage and set it in the relevant HTML elements.

That's it, we've now full implemented the game 🙌🏽🙌🏽🙌🏽

Let's go ahead and run it. We first need to run the server, so from your command line, navigate to the folder where the server file is and run node server.js. This will start the server. Now, open three browser windows and keep them side-by-side. Hit the base URL of your server from all the three windows. You should see the intro.html page being served asking for a nickname. Give each player a nickname and enter. After the third player enters, the ship starts with the bullets going off. Make sure to control each player to avoid getting killed.

If it's running as expected, you can host this game using a free hosting service such as Heroku or Glitch. This will allow you to access the game via a public URL letting you play the game for real with your friends on other computers.

A separate release relevant to this tutorial is available on GitHub if you'd like to check it out.

You can also follow the Github project for latest developments on this project.

As always, if you have any questions, please feel free to reach out to me on Twitter @Srushtika. My DMs are open :)

DEV Community: Srushtika Neelakantam

Say hello to Ably Chat: A new product optimized for large-scale chat interactions

What is Ably Chat?

What Ably Chat looks like

Why Ably Chat?

Composable realtime

Dependable by design

Cost optimizations

Get started with Ably Chat

Collaboration is futile without these UX best-practices

The essential role collaborative features play in the user experience

Avatar stack

Member location

Live cursors

Component locking

UX design considerations for collaborative features

Best practices when building avatar stacks

Best practices when building a member location feature

Best practices when building live cursors

Best practices for building a component locking feature

Adding in-app collaboration

Creating collaborative environments in a few lines of code

Introducing Spaces: Build collaborative environments in a few lines of code

Spaces at a glance

Enhancing your product with in-app collaboration & Spaces

SDK feature highlights

Get started today

The async/await post we promised

The inner workings of JavaScript

A whirlwind tour of asynchronous JavaScript

From Callbacks to Promises to Async/Await

Option 1 - Async JavaScript with callbacks

Option 2 - Async JavaScript with Promises (.then syntax)

Option 3 - Async JavaScript with Promises (async/await syntax)

Concurrency of independent asynchronous statements

The case of asynchronous event listeners

Using the promise-based version of the ably-js SDK

We are moving to Promise by default

References and further reading

Free DevRel mentorship

Database-driven realtime architectures: building a serverless and editable chat app - Part 2

Section 1 - Navigating the Nuxt app (even if you are not a Nuxt developer)

Folder structure

The Vue components

VueX state management for the chat app

Inside actions.js

Section 2 - PostgresDB setup for our chat app

Section 3 - Lambda function setup on the Ably integrations dashboard

Use the Lambda function for database transactions

Section 4 - Set up the Ably Postgres connector for the chat app

Recap of the app architecture

Deployment

Nuxt static app deployment on Netlify

Ably Token-based authentication using Netlify functions

PostgresDB deployment on AWS RDS

Ably Postgres connector deployment on AWS Fargate (via ECS and ECR)

Further improvements

Summary

Database-driven realtime architectures: building a serverless and editable chat app - Part 1

Introduction

The editable chat app architecture

Tech stack

NuxtJS for frontend web development

Ably to enable the pub/sub pattern

AWS Lambda functions to insert and update rows in the database

PostgresDB hosted on AWS RDS to store the data

Ably Postgres connector to watch changes on the database tables and publish messages on every change

AWS Fargate with AWS ECS and AWS ECR to deploy the Ably Postgres connector's dockerized image

Netlify to host the static Jamstack site

Netlify functions to enable a token auth endpoint to authenticate with Ably

The Ably Postgres connector -- a key dependency

Resources

Myth-busting: Jamstack can't handle dynamic content

What is the Jamstack?

A bit more on static sites with dynamic/ realtime data

Let’s understand with an example

Tech stack for our watch party app

What is Nuxt.js?

What is Strapi?

What is Ably?

Using the promise-based version of the `ably-js` SDK