DEV Community: Ably

Ably AI Transport is now available

Ably Blog — Tue, 20 Jan 2026 17:49:29 +0000

Today we’re launching Ably AI Transport: a drop-in realtime delivery and session layer that sits between agents and devices, so AI experiences stay continuous across refreshes, reconnects, and device switches — without an architecture rewrite.

The gap: HTTP streaming breaks down for stateful AI UX

AI has moved from “type and wait” requests to experiences that are long-running and stateful: responses stream, users steer mid-flight, and work needs to carry across tabs and devices. That shift changes what “working” means in production. It’s not just whether the model can generate tokens, it’s whether the experience stays continuous when real users behave like real users do.

Most AI apps still start with a connection-oriented setup: the client opens a streaming connection (SSE, fetch streaming, sometimes WebSockets), the agent generates tokens, and the UI renders them as they arrive. It’s low friction and demos well.

But HTTP streaming really solves only the first part of the problem, and it’s not a good place to end.

First: continuity. When output is tied to a specific connection, the experience becomes fragile by default. Refreshes, network changes, backgrounding, multiple tabs, device switches, agent handovers (even agent crashes) are normal behaviour. And they’re exactly where teams see partial output, missing tokens, duplicated messages, drifting state, and “start again” recovery paths. That’s where user trust gets lost.

Second: capability. A connection-first transport layer doesn’t just make UX fragile. It limits what you can build. Once you want true collaborative patterns like barge-in, live steering, copilot-style bidirectional exchange, multi-agent coordination, or a seamless human takeover with full context, you need more than “a stream.” You need a stateful conversation layer that can support multiple participants, resumable delivery, and shared session state.

So teams patch it: buffering, replay, offsets, reconnection logic, session IDs, routing rules for interrupts and tool results, multi-subscriber consistency, and observability once production incidents start. It’s critical work — but it’s not differentiation.

What Ably AI Transport does

AI Transport gives each AI conversation a durable bi-directional session that isn’t tied to one tab, connection or agent. Agents publish output into a session channel, clients subscribe from any device, and Ably handles the delivery guarantees you’d otherwise rebuild yourself: ordered delivery, recovery after reconnects, and fan-out to multiple subscribers.

It’s deliberately model and framework-agnostic. You keep your agent runtime and orchestration. AI Transport handles the delivery and session layer underneath.

The key shift: sessions become channels

In a connection-oriented setup, the “session” effectively lives inside the streaming pipe. When the pipe breaks, continuity becomes a headache.

With AI Transport, the session is created once and represented as a durable channel. Agents and clients can join independently. Refresh becomes reattach and hydrate. Device switching becomes another subscriber joining the same session. Multi-device behaviour becomes fan-out rather than custom routing. Agents and humans become truly connected over a transport designed for AI bi-directional low latency conversations.

How Ably AI Transport ensures a resilient, stateful AI UX

Resumable, ordered token streaming: A great AI UX depends on durable streaming. Output is treated as session data, so clients can catch up cleanly after refreshes, brief dropouts, and network handoffs.

Multi-device continuity: Conversations are user-scoped, not tab-scoped. Multiple clients can join the same session without split threads, duplication, or drifting state.

Live steering and interruption: Modern AI UX needs control, not just output. Interrupts, redirects, and approvals route through the same bi-directional session fabric as the response stream, so steering works even across reconnects and devices.

Presence-aware sessions: Once agents do real work, wasted compute becomes a serious cost problem. Presence provides a reliable signal for whether the user is currently connected (or fully offline across devices), so you can throttle, defer, or resume work accordingly.

Agents that collaborate and act with awareness: As soon as you have more than one agent (or an agent plus tools/workers), coordination becomes the product. Shared session state and routing prevent clashing replies, duplicated context, and “two brains answering at once,” so multiple agents can communicate directly with users coherently.

Seamless human takeover when it really matters: When an agent hits a boundary (risk, uncertainty, or policy) a human should be able to step in with full context and continue the session immediately. The handoff keeps the same session history and controls, so there’s no repeated questions, no “start again,” and no losing track of what happened mid-flight.

Identity and access control: Beyond toy demos, you need to know who can read, write, steer, or approve actions. Verified identity plus fine-grained permissions let multi-party sessions stay secure without inventing a bespoke access model.

Observability and governance: When AI UX breaks in production, it’s rarely obvious where. Built-in visibility into session delivery and continuity makes failures diagnosable and auditable instead of “black box streaming incidents.”

Concrete examples

Multi-device copilots: A user starts a long-running answer on desktop, switches to mobile mid-response, and the session continues without restarting. Steering and approvals apply to the same session regardless of device.

Long-running agents: A research agent runs multi-step tool work for minutes. If the user disconnects, the work continues; when the user returns, the client hydrates from session history instead of resetting.

Getting started (low friction)

You can get a basic session running in minutes:


js
import Ably from 'ably';

// Initialize Ably Realtime client
const realtime = new Ably.Realtime({ key: 'API_KEY' });

// Create a channel for publishing streamed AI responses
const channel = realtime.channels.get('my-channel');

// Publish initial message and capture the serial for appending tokens
const { serials: [msgSerial] } = await channel.publish('response', { data: '' });

// Example: stream returns events like { type: 'token', text: 'Hello' }
for await (const event of stream) {
  // Append each token as it arrives
  if (event.type === 'token') {
    channel.appendMessage(msgSerial, event.text);
  }
}

AWS us-east-1 outage: How Ably’s multi-region architecture held up

Ably Blog — Fri, 24 Oct 2025 12:24:17 +0000

Resilience in action: zero service disruption

During this week’s AWS us-east-1 outage, Ably maintained full service continuity with no customer impact. This was our multi-region architecture working exactly as designed; error rates were negligibly low and unchanged throughout. Any additional round trip latency was limited to 12ms, which is below the typical variance in any client-to-endpoint connection, and well below our 40–50ms global median; this is imperceptible to users and below monitoring thresholds. There were no user reports of issues. Taken together this means there was zero service disruption.

The technical sequence

Ably provides a globally-distributed system hosted on AWS with services provisioned in multiple regions globally. Each scales independently in response to the level of traffic in the regions, and us-east-1 is normally the busiest region.

From the onset of the AWS incident what we saw was that the infrastructure already existing in that region continued to provide error-free service. However, issues with various ancillary AWS services meant that our control plane in the region was disrupted, and it was clear that we would not be able to add capacity in the region as traffic levels increased during the day.

As a result, at around 1200 UTC we made DNS changes so that new connections were not routed to us-east-1; traffic that would have ordinarily been routed there (based on latency) were instead handled in us-east-2. This is a routine intervention that we make in response to disruption in a region. Pre-existing connections in us-east-1 remained untouched, continuing to serve traffic without errors and with normal latency throughout the incident. Our monitoring systems, via connections established before the failover, confirmed this directly.

Latency impact: negligible

We continuously test real-world performance in multiple ways. Monitors operated by Ably, in proximity to regional datacenter endpoints, indicated that the worst case impact on latency - which would have been clients directly adjacent to the us-east-1 datacenter, but which now have to connect to us-east-2 - was 12ms at p50. We also have real browser round-trip latency measurements using Uptrends, which more closely simulate real users, with actual browser instances publishing and receiving messages between various global monitoring locations.

These measurements taken during the incident are shown below; real-world clients experienced even lower latency impact, since from each of the cities tested, there is negligible difference in distance, and latency, between that location and us-east-2 versus us-east-1. Taken across all US cities that are monitoring locations, the measured latency difference averaged 3ms. That actual difference is substantially lower than normal variance in client connection latencies, and is therefore imperceptible to users and well below monitoring thresholds.

We restored us-east-1 routing on 21 October following validation from AWS and our own internal testing.

The architecture at work

This incident validated our multi-region architecture in production:

Each region operates independently, isolating failures
Latency-based DNS adapts routing to regional availability
Existing persistent connections are unaffected if the only change is to the routing of new connections
A further layer of defense, not used in this case, provides automatic client-side failover to up to five globally-distributed endpoints

That final layer matters. Even if us-east-1 infrastructure had failed entirely (it didn’t), client SDKs would have automatically failed over to alternative regions, maintaining connectivity at the cost of increased latency. It didn’t activate this time, since regional operations continued normally, but it’s a core part of our defense-in-depth strategy.

Lessons reinforced

The key takeaways for us from this incident:

A genuinely distributed system spanning multiple regions, not just availability zones, is essential for ultimate continuity of service
Planning for, and drilling, responses to this type of event is critical to ensuring that your resilience is real and not just theoretical
A multi-layered approach, with mitigations both in the infrastructure and SDKs, ensures redundancy and continuity even without active intervention. AWS continues to be an outstandingly good global service, but occasional regional failures must be expected. Well-architected systems on AWS infrastructure are capable of supporting the most critical business needs

Keep your realtime apps running smoothly, even when the internet breaks. Try Ably for free today!

How doxy.me turned realtime from a liability into a strategic asset

Ably Blog — Thu, 17 Jul 2025 16:30:49 +0000

When Realtime Breaks, Virtual Care Breaks.

Here’s the hard reality of telehealth: when infrastructure cracks, care collapses. A missed chat ping or delayed check-in notification isn’t just a glitch, it’s a broken line of communication between a provider and a patient. That can be the difference between timely diagnosis and uncertainty, between trust and frustration.

Doxy.me runs over 250,000 virtual visits every day. Their core product delivers browser-based telemedicine that works without downloads or installations. But behind the scenes, one part of the stack had become a chronic liability: realtime infrastructure.

It wasn’t just fragile - it was feared. VP of Engineering, Ben Anderson-Dukes recalled their engineering team viewing it as a “black box.” Over time, that black box became a bottleneck. Small issues like ghost check-in chimes and out-of-order messages became symptoms of deeper instability. Lag was unpredictable. Full-day outages were a looming threat. Worse still, this fragile infrastructure had become the second-largest operating expense after video delivery.

That wasn’t sustainable. But replacing their realtime provider wasn’t a simple procurement problem. It required a mindset shift from seeing realtime as a necessary evil to treating it like a core product surface that deserved strategic investment.

A Strategic Overhaul, Not Just a Quick Fix.

That partner was Ably. Unlike previous vendors, Ably took a collaborative, hands-on approach from day one, offering architectural guidance and flexible support throughout the migration.

“We needed a partner who could not only help us rebuild realtime into something reliable, scalable, and secure, but one that was developer-friendly and of a similar mindset to doxy.me.”

With support from external dev agency Walter Code, doxy.me and Ably planned a phased migration away from their existing realtime provider to Ably’s modern WebSocket-based infrastructure. The process was methodical: time it right, plan it right, implement it right. Nothing was rushed.

“Initially, we thought this would take a year. But with Ably, we went from design to 100% migrated in under six months.”

Despite handling more than 250,000 calls a day, the migration was completed with zero downtime. The transition not only modernized their infrastructure but also demystified it. Engineers gained new visibility into system behavior, and the team collectively regained confidence.

“They helped us not only get there, but raise the bar in internal education about realtime as well.”

Real Results, Not Just Promises.

Post-migration, the transformation was visible across both technical and business metrics:
• 65% reduction in realtime infrastructure costs
• 95% fewer patient queue issues
• 99% drop in app crashes caused by signaling issues
• 100% elimination of ghost check-in chimes

But beyond metrics, the real value came from stability. doxy.me’s CTO, initially skeptical, came to view realtime as a stable part of core infrastructure. The engineering team moved from firefighting to forward planning.

“Support tickets dropped, realtime errors disappeared, and our Datadog logs became clean and readable.”

The financial impact was just as compelling:

“We saw a full ROI in under six months, despite using an external team to handle the migration. That’s unheard of.”

“For me personally, it’s been a really great win. It’s bolstered Engineering’s reputation internally. There was much kudos served all-round.”

Looking Forward: Realtime as Innovation Engine.

With Ably in place, doxy.me isn’t just maintaining a stable stack, they’re building on top of it. New use cases are now in scope, including advanced presence, session orchestration, and collaboration tooling designed to make virtual care more intuitive and human.

“Even now, we’re exploring roadmap innovations together.”

One of doxy.me’s largest customers, accounting for nearly 30% of realtime traffic, now operates smoothly with no degradation in performance. Internally, engineers have better observability, faster diagnostics, and more freedom to innovate.

“Ably is helping us push realtime beyond the basics, into new opportunities that let providers be with their patients more reliably and securely.”

At its core, doxy.me is about connection - between patient and provider, between care and access.

“We believe that providers are the real heroes, and that doxy.me is their superpower.”

With Ably behind the scenes, that superpower now has a reliable, resilient realtime engine built for scale.

“Ably has helped us turn realtime from a liability into a strategic asset.”

Doxy.me built this on top of Ably Pub/Sub, the core messaging product in the Ably platform. Please see our docs if you're interested in the technical details.

Achieving low latency with pub/sub

Ably Blog — Wed, 22 Jan 2025 22:39:19 +0000

In pub/sub messaging systems, getting messages to flow quickly between publishers and users isn't just critical to its general performance, but central to its basic usability. Achieving this at scale introduces some extra challenges that require thoughtful architecture design and strategies for handling unexpected behavior (e.g. traffic spikes).

To better understand the best practices we can implement to our architecture to overcome these challenges, let's revisit how the pub/sub pattern works.

What is pub/sub?

Pub/sub (or publish/subscribe) is an architectural design pattern used in distributed systems for asynchronous communication between different components or services. Although publish/subscribe is based on earlier design patterns like message queuing and event brokers, it is more flexible and scalable. The key to this is the fact that pub/sub enables the movement of messages between different components of the system without the components being aware of each other’s identity (they are decoupled).

For a deeper dive into pub/sub, including examples and comparisons to other messaging patterns, see our guide: What is pub/sub?.

Why latency is crucial to pub/sub realtime systems

Latency is the time it takes for data to travel from the backend (like a datacenter) to the end-user’s device. Latency levels of <100ms are hard to achieve in general, but for pub/sub systems, it’s essential that those speeds remain not only consistent but also undetectable so that users remain engaged and don’t quit the app entirely. Applications that focus on, for example, crucial broadcasting updates, realtime chat, and live streaming services, need to be able to deliver seamless experiences with ultra-low latency to maintain their user bases.

This becomes especially important at scale for a global audience: If a pub/sub system can’t maintain these speeds as it scales up and reaches a global user base, message delays could render it unusable, even if your infrastructure has the raw capacity. Serving a single region is significantly simpler than achieving consistent low latency across a distributed global audience, where factors like inter-region data replication and network variability come into play. Global median latency is a good measure of average global latency if you’re operating at scale, and it’s the metric we use to measure our speeds at Ably.

Some architectural decisions you can make to achieve low latency are:

Global datacenter coverage: The physical proximity of datacenters or edge points of presence (PoPs) to end users significantly impacts round-trip times for messages. If you distribute datacenters and PoPs globally, you can drive down latency for your users.
Protocol efficiency: The choice of protocol affects how efficiently messages are transmitted. For example, WebSocket is highly efficient for realtime communication compared to HTTP long polling. (WebSockets are a particularly good protocol for achieving low latency in pub/sub systems since they maintain an open connection between the client and server without the need for frequent HTTP responses. For a deeper dive into how WebSockets compare to other protocols in pub/sub systems, check out our guide Pub/Sub vs WebSockets.)
Network robustness: A reliable, fault-tolerant underlying network infrastructure can ensure consistent low latency even under high traffic volumes.

Challenges to achieving low latency

The most straightforward obstacle to latency is network speeds - latency is inherently affected by the distance between clients and the server. The farther a client is from a datacenter, the longer it takes for messages to reach them.This is a critical consideration for global systems, where distances between users and datacenters can span continents. But there are other factors that can affect end-user latency:

Message routing: Poorly optimized routing can lead to bottlenecks, especially in use cases with high fanout where a single message is delivered to thousands or millions of subscribers.
Load balancing: If you don’t have a load balancer, or an improperly configured one, imbalances can cause overloading of certain nodes, resulting in delays for subscribers.
System resource contention: High message volumes can strain CPU, memory, and storage resources, leading to increased latency. This is particularly true during traffic spikes.
Encoding: Inefficient message encoding increases latency by slowing down the system’s ability to translate data into a transmittable format and back again.

Best practices for achieving low latency

Best practices for achieving low latency are, on paper, straightforward fixes to the points discussed above. However, making these changes to your architecture requires significant engineering effort and potentially a rehaul of your existing infrastructure. Here’s what we recommend you do:

Use a globally-distributed architecture

Deploying servers in multiple regions reduces the physical distance between clients and the server, minimizing network latency. Make sure that your infrastructure includes a combination of core datacenters and edge points of presence (PoPs). This secures fast round-trip and consistent round trip times for users anywhere in the world.

Optimize message routing

Efficient routing algorithms, such as consistent hashing, can ensure that messages are delivered to subscribers quickly and reliably. For systems with high fanout, prioritize techniques that minimize duplication and ensure messages are processed efficiently.

Have a load balancer

Dynamic load balancing distributes traffic evenly across servers, preventing overloading. For pub/sub systems, load balancers must account for both connection count and message throughput.

Use message delta compression

Compressing messages reduces their size, enabling faster transmission over the network. Use lightweight, efficient compression algorithms to minimize processing overhead.

Autoscale to reduce resource consumption

Optimize resource usage by scaling infrastructure elastically during traffic spikes. Use dynamic autoscaling to add capacity on demand and maintain a significant resource buffer.

Have redundancy and failover

Build redundancy into servers and have failover mechanisms that reroute traffic during outages. For global systems, failover strategies should account for regional redundancies to make sure that if one region experiences an outage, traffic can seamlessly shift to another without impacting users worldwide. This minimizes latency spikes during failover events and ensures uninterrupted service.

How Ably can help

For many people, building a system with all of these components from the ground up is impractical and is a huge time and skill investment. That investment also tends to be more expensive than initially expected because of maintenance costs and other challenges - like scalability and data integrity - that make maintaining a low enough latency even more difficult.

At Ably, our team is very familiar with the amount of work building a low-latency pub/sub system takes - and all the edge cases around optimum performance. We’ve made it our mission to provide the most reliable realtime service for you - and Ably Pub/Sub is devoted to pub/sub use cases.

Choosing a managed pub/sub service like Ably can save you and your team the headache of managing the architectural challenges of low latency at scale. Performance is one of Ably’s core pillars, and it’s built into what we do. Here’s how:

Predictable performance: A low-latency and high-throughput global edge network, with median latencies of <50ms.
Guaranteed ordering & delivery: Messages are delivered in order and exactly once, with automatic reconnections.
Fault-tolerant infrastructure: Redundancy at regional and global levels with 99.999% uptime SLAs. 99.999999% (8x9s) message availability and survivability, even with datacenter failures.
High scalability & availability: Built and battle-tested to handle millions of concurrent connections at scale.
Optimized build times and costs: Deployments typically see a 21x lower cost and upwards of $1M saved in the first year.

Low latency is non-negotiable for any pub/sub system that aims to deliver realtime experiences at scale. If you’re looking for a solution that scales up and ensures some of the lowest latencies in the business, Ably provides a robust and reliable platform to power your pub/sub needs. Sign up for a free account to try it for yourself.

How to use Ably LiveSync’s MongoDB Connector for realtime and offline data sync

Carolina Carriazo — Thu, 16 Jan 2025 13:01:19 +0000

In light of the recent deprecation of MongoDB Atlas Device Sync (ADS), developers are seeking alternative solutions to synchronize on-device data with cloud databases. Ably LiveSync offers a potential alternative and can replace some of ADS’s functionality, enabling realtime synchronization of database changes to devices at scale. LiveSync allows for a large number of changes to MongoDB to be propagated to end user devices in realtime and store the changes in any number of local storage options - from an embedded database to in-memory storage.

For instance, imagine an inventory app that needs to broadcast stock updates to multiple devices in realtime. Ably LiveSync allows you to automatically subscribe to inventory changes in your database and broadcast this data to millions of clients at scale, allowing them to remain synchronized with the state of your inventory in realtime.

This article explains why on-device storage is critical, explores existing solutions, and demonstrates how Ably LiveSync’s MongoDB connector can help with a brief code tutorial.

Why keep information on-device?

Local storage is a must for apps that need offline access or fast performance— like e-commerce inventory apps, or news apps downloading content for offline browsing. But not every app needs it. If your app is always online or just streams read-only data, you can skip the complexity of a local database. Thankfully, with Ably, you can adapt to your use case, whether you need offline support or just realtime updates. Some of the benefits of on-device storage are:

Offline access: Storing data directly on the device ensures users can seamlessly access and interact with information even when they have no internet connection or are in areas with poor connectivity. This is particularly crucial for users who frequently work in offline environments or travel to locations with unreliable network coverage.
Performance: Applications demonstrate significantly improved response times and reduced latency when accessing data stored locally, as opposed to making time-consuming server calls across the network. This local data access eliminates network-related delays and provides instantaneous data retrieval for critical operations. Cost efficiency: Users experience substantial savings on their data usage and associated costs since the application doesn't need to repeatedly download information from remote servers. This is especially beneficial for users with limited data plans or in regions where mobile data is expensive.
User experience: Users benefit from a consistently smooth and reliable application experience, maintaining uninterrupted access to their data regardless of their network status or connection quality. This reliability helps build user trust and satisfaction with the application.

Options for storing information on device

Modern mobile operating systems provide a variety of ways to store information on device:

iOS: Includes UserDefaults, CoreData, and SQLite, with flexibility for additional solutions based on specific needs.
Android: Provides shared preferences, Room database, and file storage.
Cross-platform frameworks: With React Native, react-native-async-storage is a popular starting library for simple needs. However, for advanced use cases requiring NoSQL-like abilities, some good choices here would be RealmDB (which, unfortunately, as we know, is being deprecated), UnQLite, LevelDB, or Couchbase.

Regardless of your choice of an on-device database and storage methodology, you can use Ably LiveSync to synchronize data from your managed or on-premises database to mobile devices in realtime. This includes MongoDB - as well as Atlas. While we currently support only MongoDB and PostgresSQL, we are working on adding support for other database engines.

What is Ably LiveSync?

Ably LiveSync lets you monitor database changes and reliably broadcast them to millions of frontend clients, keeping them up-to-date in realtime.

LiveSync works with any tech stack and prevents data inconsistencies from dual-writes while avoiding scaling issues from "thundering herds" — sudden surges of traffic that can overwhelm your database.

How to persist data locally with Ably

Let’s explore how to build a simple in-store management app that tracks product inventory using React Native and SQLite for local storage and a Mongo Atlas for our cloud database. Despite Mongo being a document storage and SQLite being a relational database, the two can be used in combination. We are going to use the Ably SDK callback methods to store documents and changes inside our local SQLite database.

Setting up Ably

For simplicity, we’ll stick to TypeScript. Before anything else, create a new React Native project using the CLI:

npx @react-native-community/cli@latest init AwesomeStore

Creating a MongoDB integration rule with Ably

Now we need to create a new channel that streams database changes to your clients. This ensures realtime updates whenever your MongoDB data changes. To create an integration rule that will sync your MongoDB database with Ably, you’ll first have to sign up for an Ably account.

Once that’s done, you should have access to your Ably dashboard. Create an app or select the app you wish to use. Navigate to the Integrations tab > Create a new integration rule > MongoDB. Fill out the Connection URL with your MongoDB connection URL; Database name with your db name (for this example, SQLiteDatabase); and Collection with your collection name (for this example, products). For more information on this process and the parameters involved, check out our docs.

This sets up a new channel, built on top of our core Ably Pub/Sub product, which streams changes (through MongoDB change streams) from your database to your clients.This essentially ensures that any change that occur in your database will be delivered to any device subscribed to a channel.

Creating the local datastore

We’ll create a new file in our project called datastore.js and initialize SQLite:

export const createTables = async (db: SQLiteDatabase) => {

  const query = `CREATE TABLE IF NOT EXISTS products(
        id INT32
        name TEXT NOT NULL
        description TEXT
        quantity INT32
    );`;

  await db.executeSql(query);
};

After the tables are created, we need a way to retrieve store products and update their stock:

export const getProducts = async (db: SQLiteDatabase): Promise<ToDoItem[]> => {
  try {
    const products: StoreProduct[] = [];
    const results = await db.executeSql(`SELECT id, name, description, quantity FROM ${tableName}`);
    results.forEach(result => {
      for (let index = 0; index < result.rows.length; index++) {
        products.push(result.rows.item(index))
      }
    });
    return products;
  } catch (error) {
    console.error(error);
    throw Error('Failed to get products!');
  }
};

export const saveOrUpdateProducts = async (db: SQLiteDatabase, products: StoreProduct[]) => {
  const insertQuery =
    `INSERT OR REPLACE INTO ${tableName}(id, name, description, quantity) values` +
    products.map(i => `(${i.id}, '${i.value}', '${i.description}', '${i.quantity}')`).join(',');

  return db.executeSql(insertQuery);
};

Receiving database changes from MongoDB over Ably

Let’s take a look at how we can receive changes from the configured Ably channel. More information can be found in our documentation, but this is the important snippet we need - setting up the Ably Realtime SDK:

import * as Ably from 'ably';

// Instantiate the Realtime SDK
const ably = new Ably.Realtime('[your API key]');

// Get the channel to subscribe to
const channel = ably.channels.get('store:1');

// Subscribe to messages on the 'store:1' channel
await channel.subscribe((message) => {
    // Print every change detected in the channel
  console.log('Received a change event in realtime: ' + message.data)
});

We need to write a new function that will take the payload of message.data and store it in our database:

const addOrUpdateProduct = async (data: any) => {
    try {
      let product: StoreProduct = {
            id: data.fullDocument.id,
            name: data.fullDocument.name,
            description: data.fullDocument.description,
            quantity: data.fullDocument.quantity,
      }
      const db = await getDBConnection();
      await saveOrUpdateProducts(db, product);
    } catch (error) {
      console.error(error);
    }
  };

We can call our new function in our message subscription:

// Subscribe to messages on the 'store:1' channel
await channel.subscribe((message) => {
      // Print every change detected in the channel
      console.log('Received a change event in realtime: ' + message.data)
      if (data.ns.coll === "products") {
            await addOrUpdateProduct(message.data)
      } else if (data.ns.coll === "store") {
            // Another function which updates changes for the `store` collection
      } else {
            console.warn("Unknown collection");
      }
});

The full workflow

With this setup, the app listens for realtime updates from your MongoDB collection and persists changes locally, ensuring an up-to-date inventory system even when offline.

Final thoughts: What makes Ably different

I hope that gives you a good overview of what Ably LiveSync’s MongoDB Connector can do! Besides providing a potential alternative to Atlas Device Sync, Ably, as a realtime communications platform, is built for scalability and reliability. Here are some features of Ably Pub/Sub, the backbone upon which LiveSync’s database connector is built:

Predictable performance: A low-latency and high-throughput global edge network, with median latencies of <50ms.
Guaranteed ordering & delivery: Messages are delivered in order and exactly once, even after disconnections.
Fault-tolerant infrastructure: Redundancy at regional and global levels with 99.999% uptime SLAs. 99.999999% (8x9s) message availability and survivability, even with datacenter failures.
High scalability & availability: Built and battle-tested to handle millions of concurrent connections at scale.
Optimized build times and costs: Deployments typically see a 21x lower cost and upwards of $1M saved in the first year.Try Ably today and explore our MongoDB connector.

Low latency at scale: Gaining the competitive edge in sports betting

Ably Blog — Mon, 06 Jan 2025 09:27:04 +0000

The sports betting industry has grown rapidly in recent years, fueled by changing regulations, advancements in technology, and a rising demand for realtime interactions from consumers. For many fans, in-play betting adds another dimension to how they can engage, making them feel closer to the action. When you then consider the increasing number of global followers many teams now have, it’s easy to understand why global revenues are projected to continue expanding at a compound annual growth rate (CAGR) exceeding 10%. To sustain this growth, data providers and betting companies face a key challenge: consistently ensuring fast, low latency delivery of data and services, even as they scale operations to meet growing demand – every fan needs to receive the same data at the same time.

Low latency - the rapid transmission of data with minimal delay—plays a crucial role in the sports betting ecosystem. A single delay in odds updates or live bet placement can impact user retention and create financial exposure for operators. This challenge becomes even more pressing as companies expand into new geographies, requiring infrastructure that can deliver consistent, low latency experiences worldwide.

Why low latency matters in sports betting

In sports betting, every second matters. For bettors, much of the appeal of sports betting lies in the immediacy of their interaction with live events. Odds must be delivered and displayed in realtime, and delays can result in missed opportunities for bettors and revenue loss for operators.

If odds are not updated instantly after a critical game event, operators risk users betting on outdated odds. For data providers, failure to deliver realtime data can strain relationships with clients, harm reputation and even have legal implications.

The importance of a consistent user experience

Scaling betting operations isn’t just about onboarding new users—it’s about ensuring these users have the same experience, regardless of location or time.

There are two key infrastructure requirements involved in making this possible: Consistent low latency, and the ability to handle disconnections.

When it comes to latency, users demand the same experience whether they’re betting from Europe, Asia, or the Americas. A lack of consistency can create an uneven playing field – or even cause users to abandon platforms as they develop a mistrust for their data. For both data providers and betting companies, this means achieving similar levels of low latency to clients in diverse geographies. From an infrastructure perspective, this means having a global network of points of presence which devices can connect to and achieve low latency, wherever your users are.

When it comes to handling disconnections and subsequent reconnections, this is particularly key when serving users in markets with unstable network conditions – but is also useful when a user changes data networks or is travelling. For betting operators, they need a strategy to define how a bet is treated if it is placed as a user loses connectivity, and the odds change while they are offline for example – should the original odds be ‘retained’, or should the odds be refreshed upon reconnection? Operators need to consider how to ensure that users with worse network connectivity don’t become ‘worse off’ as a result.

How Genius Sports delivers critical data at a global scale

Genius Sports serves betting companies that demand instantaneous data delivery for live sporting events. Before adopting their current realtime infrastructure, they relied on a traditional centralised system to deliver live data to its betting clients. Maintaining live data performance at low latency and on a global scale, meant locating ever larger and more costly on-premise facilities, close to customers. The system struggled with scalability and latency consistency, especially during high-demand events like global tournaments. As costs and latency demands increased, Genius Sports needed a new realtime solution.

To ensure a consistent user experience, the company switched to a cloud-native distributed infrastructure that could handle their need to reliably serve customers – regardless of their location. By leveraging a WebSocket-based realtime data streaming solution, Genius Sports is now able to deliver on the high expectations of both its B2B clients and the end-users who rely on their ultra-low latency data delivery without the headache of having to manage and scale the realtime infrastructure.

Responding to the growing competition for fan engagement

As the sports betting market becomes more competitive, betting companies are incorporating innovative ways to deliver realtime experiences to keep their customers ‘on-platform’. This means investing in infrastructure that can not only handle the fast-paced nature of betting and provide instant, low latency, updates but also enable the creation of interactive experiences.

Sportsbet faced a unique challenge with its innovative “Bet With Mates” feature, a product allowing friends to pool resources and bet together. A critical component was a chat feature that mirrored the functionality and speed of modern messaging applications like WhatsApp.

Sportsbet’s existing infrastructure could not deliver the rich user experience their customers expected, especially for features like realtime updates, reactions, and comments. As well as latency needs, Sportsbet also had very stringent security and data handling requirements. For Sportsbet, opting for a cloud-native realtime platform that handled both the underlying infrastructure and latency complexities but also made building their chat solution easy, was game changing. The solution saved Sportsbet valuable time and effort compared to them building a basic implementation from scratch.

Low latency, high stakes

It is clear that the sports betting industry is at an inflection point. As regulatory changes continue to unlock new opportunities, companies must rise to the challenge of delivering low latency experiences at scale. Whether it’s B2B providers like Genius Sports ensuring data consistency or B2C platforms like Sportsbet creating seamless user experiences, the ability to operate in realtime, no matter where and when, will define success.

How Sportsbet handles 4.5M daily chat messages on its 'Bet With Mates' platform

Ably Blog — Fri, 20 Dec 2024 12:35:55 +0000

Sportsbet is a leader in the Australian wagering market. Through their best-in-class platform, which includes ‘Bet With Mates’, they bring excitement to life for sports and racing enthusiasts - replicating the experience of punting with friends in the pub, no matter where they are.

But after launching 'Bet With Mates', Sportsbet customers were still off the platform (second-screening) to talk about their bets and banter with each other in WhatsApp and other chat apps. Sportsbet wanted to introduce that functionality into ‘Bet With Mates’ and provide everything their customers needed without ever having to leave the platform.

Sportsbet needed a chat feature that met their customers’ high expectations of messaging applications.

Selecting the right chat solution for 'Bet With Mates'

When deciding which solution to use for the 'Bet With Mates' chat, there were a few key criteria.

It had to be feature-rich including reaction and reply functionality and also update in realtime.

As an extremely event-driven business with huge traffic spikes during major events like the Australian Football League, National Rugby League finals, and the Melbourne Cup, the solution needed to be highly performant and scalable.

It had to demonstrate great frontend performance figures, integrate well into Sportsbet’s build pipeline, and be future-proofed for other realtime use cases that developers were planning.

As well as latency needs, Sportsbet also had very stringent security and data handling requirements, so the solution needed to be hosted within Australia on a dedicated cluster.

And finally, the team decided they needed to deliver the new product in less than four months so that it would be live and in customer hands before the next AFL season launch!

Why Sportsbet chose Ably

Based on their requirements, Sportsbet decided to move ahead with Ably after comparing them to other providers. Some key benefits for Sportsbet included:

Great customer support: Early assessments of Ably’s documentation encouraged Sportsbet to quickly move to build a proof of concept. This involved dedicated support from the Ably team to consult on requirements, providing access to SDKs for their chosen tech stack, and creating a sandbox environment to conduct some integration testing and analysis.
Easy to get started: Sportsbet started out building a prototype using Ably React Hooks with their existing react client and were impressed with how quickly they could get a basic chat feature going without having to build services. They then added some components that published events including bet placements and group activity as well as features like reactions and comments in the same message stream.
Support for data security: Ably rapidly spun up a new dedicated cluster within Australia specifically for Sportsbet, which removed another potential barrier to being ready for the AFL season. Ably’s SAML integration also enabled Sportsbet to plug into their existing SSO system in record time.

Sportsbet + Ably: The results

The ‘Bet With Mates’ chat feature has proven a hit with fans, contributing to the organic growth of the overall ‘Bet with Mates’ platform. It has also proven sticky – customers who use ‘Bet With Mates’ Chat use it regularly.

Sportsbet put this success down to Ably’s unwavering reliability when it comes to performance and message delivery. They also reported that autoscaling has performed flawlessly without any incidents of concern in over a year, even on high traffic days. Peak figures for these high traffic periods have reached around 4.5 million published messages a day.

Reflecting on the success of the project and relationship with Ably, Andy commented:

“By choosing to partner with Ably, we were able to deliver a high quality outcome in a frankly impressive timeframe, and free up our delivery teams earlier to focus on other initiatives. It’s a testament to the strength of Ably’s offering how much of our time with them is spent discussing other potential use cases rather than the current implementation.”

Ably: The definitive realtime experience platform. Built for scale.

Sportsbet is one of the thousands of companies that depend on Ably to power realtime experiences for billions of people - including live updates, chat, collaboration, notifications and fan engagement. Reliably, securely and at serious scale.

Why choose Ably?

99.999% uptime SLA: We guarantee 5x9s of uptime, but consistently do better. We've had 100% uptime for 5+ years.
No scale ceiling: Ably handles massive amounts of data throughput and concurrent connections without SREs breaking into sweat.
Strong data integrity: Guaranteed data ordering, delivery, and exactly-once semantics. Even under unreliable network conditions.
Almost-infinite elasticity: Bursty connection traffic? Ably seamlessly and automatically absorbs millions of concurrent connections arriving at once.
Composable realtime: Our range of application building blocks and integrations enable developers to create the live experiences users and businesses demand. From live chat to data broadcast, and collaborative UXs to notifications, our SDKs unlock innovation - with no infrastructure to build.
Customer-first pricing, affordable at scale: Ably's pricing offers per-minute billing, consumption-based pricing, and volume-based discounts to keep you ROI positive, as you scale.

For more information, read our docs, or sign up for free!

How Mentimeter deliver reliable live experiences at scale

Ably Blog — Thu, 19 Dec 2024 16:51:00 +0000

There are no second chances when it comes to live events: Mentimeter's solutions have to work perfectly every time. Their audience engagement features must be accessible via mobile devices without user sign-in. They must also be fast. And they need to scale effortlessly to cope with huge spikes in demand; a single event can drive connections from zero to 70,000+ participants in a matter of seconds.

Mentimeter's engineers designed their systems to cope with those exacting demands, but as the business experienced rapid growth, its realtime infrastructure provider struggled to keep pace. The platform's performance at scale started to suffer.

Eventually, a tipping point came when a spike of a relatively small number of concurrent connections – ~35,000 – caused part of their realtime system provider's network to crash. Luckily, Mentimeter had fallback solutions in place for realtime communication so their services continued to operate, but with a degraded user experience for several hours. They knew it was time to find an alternative solution.

Selecting the right solution for growth-ready realtime

Johan Bengtsson, Mentimeter CTO, swiftly dismissed the possibility of building realtime infrastructure in-house due to the time and cost to build, along with significant ongoing operating costs and engineering burden. His team set out to identify a new partner that could meet Mentimeter's exacting demands while supporting all Mentimeter's current and future use cases.

Any new realtime provider needed seamless integration with Mentimeter's React front end, Ruby and Node.js backend, and AWS Kinesis for analytics. Performance, scalability, and reliability were essential requirements, particularly as Mentimeter had ambitions to raise its concurrent connections limit and cater for events of up to 150,000 participants or more. What's more, a new provider needed to provide Bengtsson with scope to innovate in bringing new features to the platform.

Why Mentimeter chose Ably
Mentimeter identified Ably as the ideal solution based on a market comparison exercise and a recommendation from existing Ably customer Split.io.

According to Bengtsson: "Ably is very transparent about engineering for reliability and scalability with its Four Pillars of Dependability. Along with a solid five-nines SLA and clear pricing, that gave me a lot of confidence. I also felt that Ably wanted to be more than a supplier. There was a real sense that it would be a true partner that would work closely with us, listening and responding to our needs and supporting our innovation roadmap."

Bengtsson was able to get a proof of concept up and running within two hours. Even though Mentimeter's developer team was stretched at the time, migration to Ably was complete inside a month, thanks to expert and fast support and a wealth of documentation.

Mentimeter + Ably: The results

The decision to move to Ably has been crucial for Mentimeter to deliver a consistent, high-quality customer experience. Bengtsson and his team are confident they can rely on its services to help them deliver business value through improved communication and audience engagement.

First and foremost, Ably's meticulous design for elastic scale and high availability has solved the scalability and reliability issues that had become an issue previously.

Bengtsson explains: "Ably has been super reliable, a part of our stack I know I can depend on. It copes easily with big loads generated by multiple presentations happening simultaneously across the globe. We put the reliability of presentation experience front and center. If Ably had issues, we'd know about them, but now we don't have to worry about stability, even when we get huge traffic spikes. Ably is a key partner for us, one that we refer to as 'Enably' because it allows us to innovate at pace to elevate our core proposition. In fact, we're so confident now, we're looking to triple our concurrent connections limit to 150,000."

Mentimeter has used Ably to innovate as it seeks to enhance and extend the value its platform delivers to customers. For instance, the Mentimeter team builds upon the Ably presence feature to ensure customers collaborating upon interactive presentations can see who is online and carrying out edits. Before adopting Ably, this was functionality the team was going to build in-house.

Bengtsson said: "We're delighted with the range of capabilities Ably gives us, while the support and documentation have helped to spark ideas at our end. The limitations with leaderboard scaling were hurting us in terms of our ability to innovate the core product. With Ably, those limits are a thing of the past."

Finally, moving to Ably has helped Mentimeter find efficiencies, spend less time on maintenance and realtime incidents, and allow its engineers to develop the core platform.

"It's part of our ethos that if there is a service providing the capabilities we need, we'll use it rather than trying to build and maintain ourselves," Bengtsson explained. "With Ably in place, our engineers can focus on what they are good at. They don't have to worry about realtime any more, which makes for a happier, more productive team."

Ably: The definitive realtime experience platform. Built for scale.

Mentimeter is one of the thousands of companies that depend on Ably to power realtime experiences for billions of people - including live updates, chat, collaboration, notifications and fan engagement. Reliably, securely and at serious scale.

Why choose Ably?

99.999% uptime SLA: We guarantee 5x9s of uptime, but consistently do better. We've had 100% uptime for 5+ years.
No scale ceiling: Ably handles massive amounts of data throughput and concurrent connections without SREs breaking into sweat.
Strong data integrity: Guaranteed data ordering, delivery, and exactly-once semantics. Even under unreliable network conditions.
Almost-infinite elasticity: Bursty connection traffic? Ably seamlessly and automatically absorbs millions of concurrent connections arriving at once.
Composable realtime: Our range of application building blocks and integrations enable developers to create the live experiences users and businesses demand. From live chat to data broadcast, and collaborative UXs to notifications, our SDKs unlock innovation - with no infrastructure to build.
Customer-first pricing, affordable at scale: Ably's pricing offers per-minute billing, consumption-based pricing, and volume-based discounts to keep you ROI positive, as you scale.

For more information, read our docs, or sign up for free!

Scaling PubSub with WebSockets and Redis

Steven Lindsay — Thu, 19 Dec 2024 14:24:24 +0000

There is increasing demand for realtime data delivery as users expect faster experiences and instantaneous transactions. This means not only is lower latency required per message, but providers must also handle far greater capacity in a more globally-distributed way.
When building realtime applications, the right tools for the job address these requirements at scale. WebSocket has become the dominant protocol for realtime web applications, and has widespread browser support. It offers a persistent, fully-duplexed connection to achieve low latency requirements. Publish/Subscribe (pub/sub) has been around even longer, and enables providers to scale data transmission systems dynamically, something that is crucial in providing these new capacity needs. While Pub/Sub itself is just an architectural pattern, tools like Redis have built out pub/sub functionality to make it easier to develop and deploy a scalable pub/sub system.

This article details how to build a simple pub/sub service with these components (WebSockets and Redis). It also discusses the particular technical challenges of ensuring a smooth realtime service at scale, the architectural and maintenance considerations needed to keep it running, and how a realtime service provider could simplify these problems and your code.

First, let’s briefly define the fundamental components at play.

What is pub/sub?

The core of pub/sub is messaging. Messages are discrete packets of data sent between systems. On top of this, channels (or topics) allows you to apply filtering semantics to messages; publishers (clients that create messages); and subscribers (clients that consume messages). A client can publish and consume messages from the same channel or many channels.

Because of this, pub/sub can be used to support a number of different patterns:

One-to-one: Two clients that both need to subscribe and publish messages to the same channel. This is common in support chat use cases.
One-to-many: Many clients may receive information from a centralized source, as with dashboard notifications.
Many-to-one: Many clients publish messages to one channel; for example, in a centralized logging system, all logs of a certain tag are routed to one repository.
Many-to-many: Multiple clients send messages to all in the members of the group - Such as a group chat or presence feature.

The pub/sub pattern allows clients to send and receive messages directly, without a direct connection to recipients with a message broker. It promotes an async and decoupled design, which lends itself to building scalable, distributed systems that can handle concurrent traffic for millions of connected clients. However, in practice, building a pub/sub system like this requires a reliable and performant data store.

What is Redis?

Redis is a highly performant in-memory key-value store, though it can also persist data to non-volatile storage (disk). It is capable of processing millions of operations per second, so makes it ideal for systems with high throughput. Its feature set includes pub/sub, allowing clients to publish to a channel which is then broadcast out to all subscribers of that channel in realtime.

What are WebSockets?

WebSockets are a protocol for bidirectional communication, and are a great choice where low latency and persistent connectivity are required. In contrast to something like HTTP long polling, which repeatedly queries the server for updates, WebSockets make a continuous connection between the client and server, so that messages flow in both directions as and when they occur. Because of this, WebsSockets can be ideal for applications like chat systems, live notifications, or collaborative tools. Having a persistent connection removes the need to poll a server for updates, this in turn reduces latency, a key metric in realtime applications.

They are commonly used for communication between some web client (browser) and some backend server. The persistent connection is formed over TCP.

A tutorial: A simple pub/sub service

Now, let’s see how a simple realtime pub/sub service comes together with Redis and WebSockets, where multiple clients can subscribe to a channel and receive channel messages.

A typical architecture consists of a WebSocket server for handling client connections, backed by Redis as the Pub/Sub layer for distributing new messages. A load balancer like NGINX, or AWS ALB is used to handle incoming WebSocket connections and route them across multiple server instances; this is key to distributing load on our service.

Autoscaling can be used to dynamically adjust our servers to match demand, maintaining performance while reducing cost during quiet periods. To do this, we would need our WebSocket servers to remain stateless.

Breakdown of components

WebSocket server - Clients connect to this server to receive updates from a channel. The server subscribes to Redis channels and forwards messages it receives through the pub/sub subscription. We can build this with something like Gorilla-WebSocket or Express-ws.
Redis as a pub/sub layer - The distribution service between publishers (backend services) and subscribers (our WebSocket servers). For simplicity, we will run a single instance of Redis. In production this will likely need to be clustered to avoid becoming a bottleneck.
Load balancer (NGINX) - Distributes incoming WebSocket connections across multiple WebSocket server instances. It could do this naively in a round-robin fashion, or use some other method like IP hashing, though care should be taken to avoid creating hotspots.
Autoscaler - It’s highly likely we will experience changing traffic, so an autoscaler will adjust our WebSocket servers according to available resources. We can implement this with Kubernetes, which can monitor metrics and respond accordingly.
Monitoring and logging - Monitoring tools like Prometheus and Grafana track system performance, latency, and message delivery success rates. This will allow us to see how our system is operating and catch any errors that occur. It’s a good idea to configure some alerting on these metrics too.

We will write the server in TypeScript (TS) using Node as our runtime environment. We won’t cover setup and usage of auto scaling, but more information on this topic can be found here.

1. Set up NGINX

Install NGINX and get it running. On Mac or Linux operating systems, we could do this with a package manager like brew.

brew install nginx && brew services start nginx

We also need to configure it to route connections, forward requests to the server, and ensure NGINX handles the WebSocket requests correctly, something like this:

# Define the upstream WebSocket server
   upstream websocket_backend {
       server 127.0.0.1:8080; # Your WebSocket server address
   }
location /ws {
           # Proxy WebSocket traffic to the backend
           proxy_pass http://websocket_backend;
           # Required for WebSocket
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection "Upgrade";
           # Pass the original Host header
           proxy_set_header Host $host;
           # Timeouts for WebSocket
           proxy_read_timeout 300s;
           proxy_send_timeout 300s;
       }

2. Set up Redis

Install Redis and get it running.


brew install redis && brew services start redis

We should now have a local instance available at localhost. Note: this creates an instance of Redis with no authentication, and is not an advisable setup for production.

3. Set up the server

Now, we will need to write the server code to handle client requests, integrate with our Redis instance, and allow WebSocket connections. TypeScript has a vast range of packages that we can use, such as WS, which provides WebSocket client and server capabilities. WS implements the WebSocket protocol and abstracts away a lot of the details like handshakes, framing and connection management, so we get a nice high-level API.


import WebSocket, { WebSocketServer } from 'ws';

We will also need the ioredis package, which provides a fully-featured client to manage our connections to Redis.


import Redis from 'ioredis';

Let’s create a basic application that will do the following;

Accept client requests and upgrade them to WebSockets
Provide simple handling of the WebSocket and Channel lifecycle
Store the new connections so we can send messages as needed
Connect to Redis and subscribe to a specified channel
When a new message is received on the channel, iterate through all subscribed clients and send the message along.
Remove unused redis connections

This looks like:


// Maps to track channel subscriptions
const redisClients: Map<string, Redis> = new Map(); // Redis client per channel


// Maps to track WebSocket connections
const channelSubscribers: Map<string, Set<WebSocket>> = new Map(); // WebSocket clients per channel


// WebSocket Server
const wss = new WebSocketServer({ port: 3000 });
console.log('WebSocket server is listening on ws://localhost:3000');


wss.on('connection', (ws: WebSocket) => {
   console.log('New WebSocket connection established');


   ws.on('message', (message: string) => {
       try {
           const data = JSON.parse(message);


           // Validate the message format
           if (data.action === 'subscribe' && typeof data.channel === 'string') {
               const channel = data.channel;


               // Ensure a Redis client exists for this channel
               if (!redisClients.has(channel)) {
                   const redisClient = new Redis();
                   redisClients.set(channel, redisClient);


                   // Attempt to subscribe to the Redis channel
                   redisClient.subscribe(channel, (err) => {
                       if (err) {
                           console.error(`Failed to subscribe to Redis channel ${channel}:`, err);
                           ws.send(JSON.stringify({ error: `Failed to subscribe to channel ${channel}` }));
                       } else {
                           console.log(`Subscribed to Redis channel: ${channel}`);
                       }
                   });


                   // Handle incoming messages from the Redis channel
                   redisClient.on('message', (chan, message) => {
                       if (channelSubscribers.has(chan)) {
                           channelSubscribers.get(chan)?.forEach((client) => {
                               if (client.readyState === WebSocket.OPEN) {
                                   client.send(JSON.stringify({ channel: chan, message }));
                               }
                           });
                       }
                   });
               }


               // Add this WebSocket client to the channel's subscribers
               if (!channelSubscribers.has(channel)) {
                   channelSubscribers.set(channel, new Set());
               }
               channelSubscribers.get(channel)?.add(ws);
               console.log(`WebSocket subscribed to channel: ${channel}`);
           } else {
               ws.send(JSON.stringify({ error: 'Invalid action or channel name' }));
           }
       } catch (err) {
           console.error('Error processing WebSocket message:', err);
           ws.send(JSON.stringify({ error: 'Invalid message format' }));
       }
   });


   ws.on('close', () => {
       console.log('WebSocket connection closed');


       // Remove this WebSocket from all subscribed channels
       channelSubscribers.forEach((subscribers, channel) => {
           subscribers.delete(ws);


           // If no more subscribers for the channel, clean up Redis client
           if (subscribers.size === 0) {
               redisClients.get(channel)?.quit();
               redisClients.delete(channel);
               channelSubscribers.delete(channel);
               console.log(`No more subscribers; unsubscribed and cleaned up Redis client for channel: ${channel}`);
           }
       });
   });


   ws.on('error', (err) => {
       console.error('WebSocket error:', err);
   });
});


}

4. Send some messages

Produce some messages on the channel so that the server can begin sending the data on to connected clients. Here, as an example, we'll query for telemetry data of the International Space Station (ISS) each second, then publish this to our channel.

import axios from 'axios';
import Redis from 'ioredis';

const telemetryUrl = 'http://api.open-notify.org/iss-now.json'; 

const redisChannel = 'iss.telemetry'; // Redis channel name
const publishInterval = 1000; // Interval in milliseconds

// Create a Redis client
const redis = new Redis();

// Function to query URL and publish to Redis
async function queryAndPublish() {
   try {
       console.log(`Fetching data from ${telemetryUrl}...`);

       // Query the URL
       const response = await axios.get(telemetryUrl);

       // Get the response payload
       const payload = response.data;

       // Publish the payload to the Redis channel
       const payloadString = JSON.stringify(payload);
       await redis.publish(redisChannel, payloadString);

       console.log(`Published data to Redis channel "${redisChannel}":`, payloadString);
   } catch (error) {
       console.error('Error publishing data:', error);
   } finally {
       // Schedule the next query and publish task
       setTimeout(queryAndPublish, publishInterval);
   }
}

// Start the query and publish loop
queryAndPublish()

// Handle graceful shutdown
process.on('SIGINT', () => {
   console.log('Shutting down...');
   redis.disconnect();
   process.exit();
});

5. Create the client

Construct a simple client that will point to our NGINX server and establish a WebSocket connection. It should begin receiving messages published to the channel. We have kept things simple here, but a production solution would need to handle things like heartbeats, retry mechanics, failover, authentication and more.

import WebSocket from 'ws';

const serverUrl = 'ws://localhost/ws';
const channelName = 'iss.telemetry';

// Create a new WebSocket connection
const ws = new WebSocket(serverUrl);

// Handle the connection open event
ws.on('open', () => {
   console.log('Connected to WebSocket server');

   // Send a subscription request for the specified channel
   const subscriptionMessage = JSON.stringify({ action: 'subscribe', channel: channelName });
   ws.send(subscriptionMessage);
   console.log(`Subscribed to channel: ${channelName}`);
});

// Handle incoming messages
ws.on('message', (message) => {
   console.log(`Message received on channel ${channelName}:`, message.toString());
});

// Handle connection close event
ws.on('close', () => {
   console.log('WebSocket connection closed');
});

// Handle errors
ws.on('error', (error) => {
   console.error('WebSocket error:', error);
});

After starting up the server, publisher, and client, we should see data being printed out to the standard output (stdout).

Challenges of a Redis + WebSockets build at scale

The code we’ve written is a basic implementation, but at scale in production, we will have to account for additional complexities. There are some issues with using Redis at scale, including:

Message persistence

In Redis pub/sub, messages are not persisted and are lost if no subscriber is listening (fire-and-forget). For some realtime use cases, like chat, this is likely not acceptable.

Horizontal scaling

A single instance of Redis is insufficient at scale, and as there is a limit to how far it can be vertically scaled, so a horizontally-scaled infrastructure is going to be your best option.

Redis can be deployed in a clustered mode for high availability and scalability. This essentially turns your single Redis instances into a network of interconnected redis nodes, then data can be distributed across these nodes. It allows for potentially much higher throughput and greater numbers of subscribers. Each node would take a portion of the data, this can be managed automatically, to distribute the workload (through sharding). But this comes with its own complexities, including:

Needing active management to ensure even data distribution and cluster health - not specific to Redis, but general distributed systems issues, including:
- Failover mechanisms when a node goes down
- Manual node provisioning in case there are no replica sets
- Robust monitoring to catch network issues
- Increased networking overheads for cross-node communication
Node discovery, leader election and split brain (where multiple leader/primary nodes become disconnected and we need to work out which nodes we will consider to still be active). Redis cluster uses a gossip protocol to share cluster state and quorum based voting for leader election and to help prevent split brain.
- There should be an odd number of nodes in a redis cluster setup, so if a network partition occurs, a quorum can still be reached.
Lack of strong consistency guarantees - redis cluster prioritises throughput and works on eventual consistency, this means you can face data loss in some cases. You could use something like redisraft (experimental) that implements the RAFT consensus algorithm, but this writes to only a single leader at a time, creating a bottle neck for write-heavy workloads. (And RAFT does not account for byzantine failures, though these are rare in practice.)

Disaster recovery and geo distribution

For realtime use cases at scale, a globally-distributed system is non-negotiable. For example, what happens if our single datacenter fails? Or what if customers on the other side of the world require the lowest latency possible? Forcing a request between geographic regions would incur unacceptable latency penalties. Both problems would require us to geolocate our cluster in multiple regions.

A Redis cluster can be globally distributed - in the case of AWS, this would mean running in multiple availability zones. But this introduces more management and maintenance complexities on top of those we’ve already accumulated for horizontal scaling, like the need to ensure correct sharding, manage reconciliation of concurrent writes across regions, route traffic in the case of a net split, and rebalance to avoid hot nodes.

Challenges of WebSockets as a protocol

WebSockets on their own have potential issues to consider in the context of building out a realtime service:

Implementation

WebSocket libraries provide the foundations to build a WebSocket service, but they aren’t a solution we can use straight out of the box. For example, how do you handle routing requests when your server is becoming overloaded? What monitoring do you put in place? What scaling policies do you have in place, and what capacity do we have to deal with large traffic spikes? Home brewed implementations often fall short for large-scale, realtime apps, a gap that usually requires a huge amount of additional development to fill.

Authentication

WebSockets also complicate authentication, as they lack the standardized mechanisms found in HTTP protocols, like HTTP headers. Also, many client-side WebSocket libraries don’t provide much support for authentication, so developers have to implement custom solutions themselves. These gaps can complicate the process of securing WebSocket connections, so it’s paramount to select an approach here that gives a good balance of security with usability.

Basic Authentication, passing API keys as URL parameters, is a simple but insecure method and should not be used for client-side applications. A more secure and flexible alternative is Token-Based Authentication like JSON Web Tokens (JWT).

How do we manage these keys though, and how do we implement token revocation? How do we ensure a token is refreshed before a client loses access to resources?

Handling failover

WebSocket connections can encounter issues that often require fallback mechanisms. Secure ports like 443 are preferred for reliability and security, but fallback transports such as XHR polling or SockJS are needed for when networks have stricter restrictions. It’s likely we would need to implement this too if you’re looking to implement your own WebSocket solution.

Power consumption and optimization

WebSocket connections are persistent, so thought must be taken when considering devices like mobiles, as the drain on battery life must be well managed. Heartbeat mechanisms like Ping/Pong frames (aka heartbeats) can help maintain connection status but may increase power consumption. We could use push notifications, waking up apps without maintaining an active connection, but they don’t provide the same kind of reliability or ordering guarantees of WebSocket streams.

Handling discontinuity

A simpler way: dedicated services

As we can see, developing and maintaining WebSockets means managing complexities like authentication, reconnections, and fallback mechanisms if we want to provide a robust realtime experience.

The considerations needed across Redis and WebSockets for an at-scale realtime service need an entire team dedicated to them. If this sounds like a headache, that’s because it is! We should avoid doing this unless we absolutely have to - whole companies have been set up to solve these problems for us.

Ably provides a globally distributed pub/sub service, with high availability and a scalable system able to meet any demand while maintaining low latencies that are crucial for realtime experiences.

There is a way we can achieve the same results with Ably while alleviating all of the issues discussed in previous sections:

WebSocket management - Ably provides pre-built WebSocket support in multiple SDKs, with SLAs that guarantee availability and reliability, and infrastructure currently handling over 1.4 billion connections daily.
Connection reliability - Ably’s SDKs are designed to maintain consistent connectivity under all circumstances. They have automatic failover, selecting the best transport mechanism and ensuring connections remain stable even during network issues. If a temporary disconnection occurs, the SDKs automatically resume connections and retrieve any missed messages. Additionally, message persistence is supported through REST APIs, allowing retrieval of messages missed due to connection failures or extended offline periods.
Scalability and capacity - Ably uses AWS load balancers to distribute WebSocket traffic efficiently across stateless servers, enabling limitless scaling to handle connections. With autoscaling policies in place, the infrastructure dynamically adjusts to meet changes in demand while maintaining a 50% capacity buffer to handle sudden spikes. Ably’s high-availability SLA guarantees 99.999% uptime, relying on redundant architecture and globally distributed routing centers.
Global distribution - Ably operates core routing datacenters in seven regions, with data persistence across two availability zones per region. This setup ensures message survivability, achieving a 99.999999% guarantee by accounting for the probability of availability zone failures and enabling quick replication. Connections are routed to the closest datacenter to minimize latency, achieving a P99 latency of less than 50ms globally.
Data distribution - Ably uses consistent hashing for channel processing, but also has dynamically scaling resources and will distribute the load as needed. This includes things like connection shedding.
Operations and maintenance - Since Ably is a fully-managed service, it has dedicated monitoring, alerting and support engineers with expertise in operating a distributed system at scale.
SDKs - 25+ SDKs for different environments.

In essence, Ably gives a comprehensive, managed Pub/Sub solution that handles the difficulties of building a distributed system with high availability, reliability, and survivability guarantees. This means developers can focus on their core application without the overhead of managing complex infrastructure like this.

Ably in practice: a code example

Let’s try using Ably’s ably-js SDK.

We can condense all of the code in the previous example into a few lines, for subscribing to our data channel:

import * as Ably from 'ably';

let client: Ably.Realtime;

async function subscribeClient() {
 client = new Ably.Realtime('xxx.yyy');

 const channel = client.channels.get('iss.data');
 try {
   await channel.subscribe((message) => {
     console.log(`Received message: ${message.data}`);
   });
 } catch (error) {
   console.error('Error subscribing:', error);
 }
}

// Handler for SIGINT
function exitHandler() {
 if (client) {
   client.close();
   console.log('The client connection with the Ably server is closed');
 }
 console.log('Process interrupted');
 process.exit();
}

// Handle SIGINT event
process.on('SIGINT', exitHandler);


(async () => {
 await subscribeClient();
})();

And publishing to our data channel:

import * as Ably from 'ably';
import axios from 'axios';

let client: Ably.Realtime;

async function publishData() {
 client = new Ably.Realtime('xxx.yyy');
 const channel = client.channels.get('iss.data');

 try {
   // Query the endpoint
   const response = await axios.get('http://api.open-notify.org/iss-now.json');

   // Get the response
   const payload = response.data;

   // Publish the payload to the Ably channel
   const payloadString = JSON.stringify(payload);
   await channel.publish({ data: payloadString });
 } catch (error) {
   console.error('Error publishing:', error);
 }
}

// Handler for SIGINT
function exitHandler() {
 if (client) {
   client.close();
   console.log('The client connection with the Ably server is closed');
 }
 console.log('Process interrupted');
 process.exit();
}

// Handle SIGINT event
process.on('SIGINT', exitHandler);

(async () => {
 await publishData();
})();

And in just a few lines of code, we have removed the need to handle the infrastructural complexities we mentioned earlier. If you’re interested in trying Ably, sign up for a free account today.

How Genius Sports slashed costs and lowered latencies for last-mile data delivery

Ably Blog — Wed, 18 Dec 2024 10:18:46 +0000

Genius Sports provides its customers with live sports and betting data across 240,000 events worldwide – each generating 10,000+ realtime messages.

Its value proposition is built on reliable, live data and on-premises infrastructure, but rapid growth brought challenges when it came to maintaining live data performance at low latency and global scale.

Genius Sports had been forced to locate ever larger, on-premises RabbitMQ clusters physically close to customers to deliver the reliable, predictable realtime performance at low latency that in-play betting demands. If data arrives late, players can bet on events that have already happened, costing Genius Sports’ customers money and damaging its reputation.

As a result, hardware capital and support costs were increasing exponentially, while infrastructure teams were constantly firefighting hardware issues.

Wondering how they overcame this challenge?

Genius Sports chose to migrate to Ably to benefit from our elastically scalable realtime infrastructure for the edge. Using Ably enabled Genius Sports to essentially hand off the heavy-lifting associated with live data processing and delivery, and gave developers the insight and flexibility they needed to innovate the core service and to offer customers more granular control of the data they subscribe to.

We spoke to Gary Williams, IT Infrastructure Team Lead at Genius Sports in our webinar - and you can now watch it on demand!

The key takeaways were:

Shifting to the cloud was key: Genius Sports chose Ably to provide realtime messaging infrastructure as a service in an early implementation of an ongoing shift to a cloud-first strategy – safe in the knowledge that the migration from RabbitMQ would require no significant refactoring of its wider architecture, coding or developer work.
The migration was straightforward: According to Gary Williams, "Overall, even with a super-cautious approach we had Ably live on our production system in less than two months. For a system handling business critical data, that is quite an achievement."
The move has freed up time and budget: The migration to Ably has eliminated annual hardware costs and freed up around 30 hours per month in RabbitMQ maintenance that Genius Sports’ developers and engineers can now refocus on service optimisation and innovation.

Chat API pricing: Comparing MAU and per-minute consumption models

Carolina Carriazo — Tue, 10 Dec 2024 15:48:47 +0000

Pricing is critical to deciding which chat API you will use - however, it can often feel like there are limited options. Whether you are looking to gradually scale a chat app or anticipate large and sudden spikes in traffic, pricing models can make or break the bank depending on your usage - and most vendors will expect you to accept the one or two industry standards.

Chat API providers predictably fall into a handful of pricing model categories. We’ll explore them in this article by explaining them, comparing them, and ultimately concluding which is best for each particular use case.

Chat APIs: Common pricing models

Chat API pricing models are designed to align with different usage patterns - like a steady user base and usage, or periodic spikes - but they also introduce trade-offs depending on an application’s scale and messaging demands. These models are generally categorized as forms of consumption-based pricing, where costs are tied to how the service is used. Let’s look at the most common pricing models in use today:

Monthly active users (MAU)

The Monthly Active Users (MAU) model is one of the most widely used pricing models in the industry. Providers like CometChat, Sendbird, Twilio, and Stream charge based on the number of unique active users per month.

You pay for each user who interacts with the chat API within a given month, regardless of the number of messages they send or receive. While this can simplify billing, it comes with the tradeoff of assuming the “typical usage” of a monthly active user. For example, an individual MAU may actually use much less active connection time or send much fewer messages than what is assumed for an average MAU. Simply put, this method is not granular.

This model is predictable for applications with small and steady user bases, since, if you’re not expecting much user volatility, it’s easy to estimate costs. But any volatility in workloads, like experiencing a brief viral period and dipping back down, can result in overpaying for peak costs (peak MAUs) in a monthly period.

For chat services operating at scale, the monthly amount spent on peak MAUs often grossly exceeds the bill for actual usage; it wastes allocated resources and money.

There is a pricing model designed to tackle these pricing issues at scale, however - and we use it at Ably.

Per-minute consumption

A per-minute consumption model goes beyond traditional consumption-based pricing by billing customers based on their actual usage of service resources—connection time, channels, and messages. This approach directly addresses the inefficiencies inherent in MAU pricing models. This isn’t a common model in the industry, but we’ve adopted it here at Ably to meet the usage needs of our customers at scale.

Per-minute consumption measures actual usage in fine-grained units, such as:

Connection minutes: The total time devices are connected.
Channel minutes: The time channels remain active.
Message events: Each message sent or received by users.

By tracking usage at this granular level, it ensures customers only pay for what they consume, without overpaying for resources they don’t use. Traffic spikes don’t necessarily lead to hugely increased costs either - the pricing is distributed across these dimensions, smoothing the overall impact. For example, livestreaming events, which may have a huge number of messages at their peak but a low number of channels, would see a more modest increase in cost than if they were billed by user count. Instead of penalizing a single metric, this approach provides greater predictability and reflects resource utilization more holistically.

Per-minute consumption also incentivizes resource optimization, such as reducing idle connections or batching messages, which can further mitigate cost surges during spikes. (Batching comes in handy when many:many chat interactions lead to an exponential increase in delivered messages, which we're implementing soon at Ably on the server side).

Popular pricing models compared

Deciding whether an MAU, message throughput, or per-minute consumption pricing model works for you depends on your use case - but if you are looking to scale a chat application to any considerable degree, as a general rule, per-minute consumption will be the best option.

MAU pricing assumes a “typical user” for billing purposes. This involves bundling resources such as connection time, message throughput, and storage into a fixed monthly fee per active user, which doesn’t accurately reflect the actual usage of the user.

Now imagine a customer operating a live event platform. They’re running a live event for two hours in the month that peaks at 50,000 users. What would the monthly prices look like between an MAU model and a per-minute consumption model?

Let’s say that the MAU model assumes that each “average” user will send 1k messages per month. While the bill comes in based on user count only, built into the cost per user is an assumption on how much one would use (in this case, a total of 50 million messages for 50k users). The MAU model then bills the whole month based on the peak of 50,000 users.

With per-minute consumption, costs reflect the actual connection time and messages used - we’ll estimate generously:

Connection time: 50,000 users × 240 minutes (accounting for pre- and post-event activity) = 12 million connection minutes.
Message volume: 50,000 users sending an average of 200 messages = 10 million messages.
Channels and channel time: let’s say 5 channels x 240 minutes = 1,200 channel minutes.

Even without specific prices to hand, we can see that billing for a typical user is inefficient in this scenario. Per-minute billing focuses on ensuring fairness and transparency for highly volatile traffic situations like these (for more information on this, Matt O’Riordan, Ably’s CEO, talks about pricing model issues in his blog post).

What does this mean in practice?

This table breaks it down:

Model	Examples	Best suited to	Challenges
Monthly Active Users (MAU)	Stream, Sendbird, Twilio	Apps with steady or low user activity	Paying for peak costs during volatile usage periods
Per-minute consumption	Ably	Apps with scalable, high-volume messaging	Requires tracking of usage metrics

Ably’s per-minute consumption model

If the per-minute consumption model we discussed above sounds promising to you, here’s some more information on how this works specifically with Ably.

At Ably, we’ve developed a pricing model designed to align more closely with the needs of realtime chat applications. Unlike traditional MAU or throughput-based models, Ably offers per-minute pricing that scales predictably and transparently with your application.

Here’s how Ably stands out:

Flexibility: Pay only for what you use, with no penalties for growing user bases or unexpected spikes in message throughput.
Scalability: Ably’s infrastructure supports billions of messages daily, with costs optimized for applications of any scale.
Transparency: Ably’s pricing eliminates the hidden costs often associated with rigid MAU or throughput models, giving you full visibility into your expenses.

Ably’s platform is built on a globally-distributed infrastructure designed for high-performance, scalable, and dependable messaging. With support for exactly-once delivery, message ordering, and <50ms global average latency, Ably ensures a seamless chat experience for users anywhere in the world.

Our Chat SDK in private beta offers fully-fledged chat features, like chat rooms at any scale; typing indicators; read receipts; presence tracking; and more. And of course, our per-minute pricing means that your consumption is as cost-effective as possible.

Data integrity in Ably Pub/Sub

Carolina Carriazo — Thu, 21 Nov 2024 14:40:26 +0000

When you publish a message to Ably Pub/Sub, you can be confident that the message will be delivered to subscribing clients, wherever they are in the world.

Ably is fast: we have a 99th percentile transmit latency of <50ms from any of our 635 global PoPs, that receive at least 1% of our global traffic. But being fast isn’t enough; Ably is also dependable and scalable. Ably doesn’t sacrifice data integrity for speed or scale; it’s fast and safe.

This post describes the Ably Pub/Sub architecture and features that guarantee your Pub/Sub message is delivered, in order, exactly once, to clients globally; while protecting against regional data center failures, individual instance failures, and cross-region network partitions.

How Ably regional Pub/Sub architecture and persistence works

Each region in Ably is capable of operating entirely independently, but regions also coordinate with each other to share and replicate messages globally.

In each region, a single Pub/Sub channel has exactly one primary location across a fleet of Ably servers. When a message is published by a client attached to that region, the message is processed and stored by that single Pub/Sub channel location before the message is ACKed. Once the publishing client receives the ACK, they can be confident that the message will not be lost, and will be delivered to all subscribing clients.

As well as exactly one primary location, an Ably Pub/Sub channel also has exactly one secondary location. Both the primary and secondary location durably store a copy of the message before the ACK is sent to the client.

If the primary location of a Pub/Sub channel fails, the secondary location is ready to take over. The secondary location already has a copy of all the required message data and becomes the primary. A new secondary location is created and the message data is replicated to that new location. This means that clients are isolated from individual instance failure.

The message is immediately replicated to the primary and secondary locations in other regions globally who have subscribing clients. Ably will store up to 14 copies of each message, globally, to mitigate against the failure of entire regions of the Ably service.

All messages are persisted durably for two minutes, but Pub/Sub channels can be configured to persist messages for longer periods of time using the persisted messages feature. Persisted messages are additionally written to Cassandra. Multiple copies of the message are stored in a quorum of globally-distributed Cassandra nodes.

How Ably SDK clients interact with the Pub/Sub architecture

Each Ably region is capable of operating independently, so Ably SDK clients can connect to any region. By default clients connect to the region providing the lowest latency, but if an issue with that region is detected (perhaps the region is erroring or is slow to respond), the SDKs will connect to another fallback region, and continue operating as normal.

Clients are isolated from region failures. There is no single point of failure in the regional Ably architecture. Regional failures will not affect the global availability of Ably, because clients will fallback to another region and continue operating.

Exactly-once delivery

Ably Pub/Sub messages support exactly-once delivery, which is achieved through two mechanisms: idempotent publishing, and message delivery on the SDK.

Idempotent publishing: Pub/Sub messages have a unique ID field which is used to deduplicate messages. When a message arrives in a region – either because it was published by an Ably SDK connected to that region, or replicated from another region – the primary Pub/Sub channel location verifies the message’s uniqueness by checking its ID. This idempotency check is performed against 2 minutes of message history, and ensures against a client accidentally publishing a message twice. The message is persisted at the primary location and checked for uniqueness in a single atomic operation, which guards against a race between checking uniqueness and durably storing the message.

The primary location in each region performs the idempotency check both for messages that are published to it directly, and for messages replicated from another region, so that the SDKs can retry the publish against another region, and that message will still be checked for uniqueness and not delivered to subscribing clients twice.

Message delivery on the SDK: On the subscribing client, the messages are delivered with a series ID. If the client disconnects, they can provide the last-seen series ID when reconnecting with the resume operation. This resume operation allows the client to pick back up from the exact point it was processed in the stream of Pub/Sub messages, ensuring no duplicates or gaps in the message stream. By default, the SDKs will retry failed message publishes.

Message ordering

The order of messages published by a Pub/Sub WebSocket client on a single WebSocket will remain fully ordered. The regional location of the Pub/Sub channel will share those messages in order with all the other regional locations. Those Pub/Sub messages will be delivered in the same relative order to all subscribing clients, regardless of the region that client is connected to.

To make sure that regions in Ably Pub/Sub can operate independently, there’s no guaranteed order between clients connected and publishing to different regions. This is to allow the two clients, connected to different regions, to be able to publish concurrently at high throughputs and low latency without needing to coordinate globally with each other. The messages published by each client in their respective WebSocket connections will always retain the same order relative to other messages on that same WebSocket connection.

In classic distributed system parlance, each client retains their own causal consistency so that Hello -> World never becomes World -> Hello.

So, based on what we know about regional Ably architecture, the message ordering that each region sees is defined by when those messages arrive in that region; either by being replicated from another region, or by a client connected to that region performing a message publish. Once the order of messages is established in a single region, all the clients connected to that region will see those messages in the exact same order, but clients in a different region might have a slightly different regional ordering.

Ably's data integrity: a summary

When using a Pub/Sub product, you want it to “just work”. You don’t want to have to worry about if your subscribing clients are going to receive the message you published, or if the messages are going to keep their published order, or if there’s failure in some data center that Ably is running in. You don’t want to have to care. We’ve spent a long time thinking about failure cases, and designing a system that’s both super fast, but also retains the integrity of the data published on Pub/Sub channels.

This post describes the internals for some of the features we use to ensure that Ably Pub/Sub “just works”.