I was a bit ill last week and took a few days off work. I ended up getting some rest but I also started hacking around on my laptop on a little experiment I'd been meaning to try out for a while. It escalated pretty quickly into an epic nerd-snipe and consumed my entire weekend as well, but it was fun.
Frequent Flyer Miles
I've been playing with Fly.io for a while and trying to get my head around it's implications for the general Web Dev landscape. If you haven't heard of it yet, it's a platform that allows deploying apps to multiple datacenters around the world, with Anycast DNS to automatically route incoming requests to the closest datacenter. Pretty edgy. Essentially it provides devs with a set of superpowers that were previously only available to large tech companies.
The thing is; as soon as you step out of the safe space of developing SRAs (Single-Region Applications) and into the twilight zone of MRAs (Multi-Region Applications) you very quickly find yourself in a wildly different problem space. We're not in Kansas any more.
The Nerd Snipe
So the idea I wanted to explore was: "what does it look like to hook up a system of live-DOM-updates-over-Websockets to a caching layer that's distributed across multiple regions?". Essentially; a multi-region app where a client-side interaction in one region could be propagated out to all regions and translated into live DOM updates to all users.
I knew it would involve a distributed KeyBD setup with multi-active replication, it would be hosted on Fly, and CableReady would be handling broadcasting DOM updates from the server side. Also I'm a Rails dev so; Ruby on Rails, baby.
I had the vague idea that if I hooked up all the different parts and actually made it work, it wouldn't make any sense without some kind of simple demo in front of it that people could click on and play with; a complex multi-region events system in the backend isn't really anything if you can't actually see it.
As this was primarily an exploration aimed at learning new things and seeing what was possible (and having fun), I decided to eschew TDD in favour of a more free-form approach, based on the traditional principles of Fuck-Around-And-Find-Out-Driven-Development.
The resulting demo is here: https://global-presence.fly.dev/
Wait, what's a KeyDB?
Alright, I'd better do some introductions.
KeyDB is a fork of (everyone's favourite cache store) Redis, and it's messaging protocol and API is 100% compatible with Redis. What that means is you can just point any Redis client (like Hiredis or redis-rb) at a KeyDB instance, and it'll Just Work™️, with no changes required. The KeyDB selling points are: 1) multi-threading by default, and a lot of work was ploughed in to high performance around multi-threading in KeyDB, 2) compatible with all the features of regular Redis, 3) some advanced features which Redis only offers in it's paid/enterprise version are included for free in KeyDB, and the big one for me is multi-active replication, which is what I'm playing with here.
CableReady is a powerful library for doing live DOM updates in Rails apps, and amongst many other things it enables pushing client-side DOM updates from the server-side. It's part of the StimulusReflex stack.
Hasty Build
I dived right in with no real plan and an urgent sense that I had to knock it out before the weekend finished. I threw together the simplest possible MVP first and iterated a few times. After getting the distributed multi-active cache system working (which was shockingly easy) and getting some StimulusReflex/CableReady bits hooked up I had most of the components in place: the client-server bits were easy and writing entries to the cache in one region was automatically replicating those changes outwards to all regions. Yay! The bit that was missing was a Pub-Sub mechanism on top of KeyDB so that when an entry in the cache was replicated from one region to another, that change could be picked up and acted on in the receiving region. Tricky...
The Redis Firehose
Luckily I'd seen this handy article showing how to stream Redis keychange events into Rails a couple of months before and it had stuck in my head. I made a quick detour to RTFM on Redis notifications and did some direct experimenting starting with the Redis CLI, using:
$ redis-cli config set notify-keyspace-events KEA
$ redis-cli --csv psubscribe '__key*__:*'
I then threw together a stripped-down and simplified version of a Redis-keychange-event-subscriber and kicked it into a simple Thread in an initializer, hooked it up to the KeyDB setup and some CableReady broadcasts and the backend bits were basically complete.
The majority of the total build time was then spent on the remaining task: creating some simple contrived mini-features for the demo (mostly faffing with CSS and trying to make it look pretty TBH).
Details
The source code is here. It not exactly battle-tested, production-grade finely-tuned stuff, but it does some interesting things and it works. The relevant bits are mostly under app/lib, app/channels, and app/reflexes. Feel free to plagiarise it ruthlessly!
The KeyDB setup I used was a customised version of this example helpfully provided by Fly.io, deployed in multiple regions alongside the app, with some modest storage volumes attached.
Gotta Go Fast
The details of how the bits fit together is what interests me the most. I'll explain a bit here.
When a client-side event (like a click) happens, it's sent to the server over a Websocket, a persistent two-way connection that avoids the need to create a new request, perform a HTTP handshake, renegotiate TLS and all that jazz, and also avoids hitting a big part of the stack that's involved in handling those requests. The upshot is it's really fast. At that point a custom key with the event data is written to the cache (in that region) and as it's Redis/KeyDB: it's really fast. That data then gets replicated out to the cache instances in all other regions and at that point things get a bit murky; it's passing between geographic regions but going via Fly's internal Wireguard network which connects their datacenters through some kind of mesh of encrypted tunnels (which I'm guessing involves some persistent connections?). Only the wizards at Fly understand the fine details there. Anyway, turns out it's fast! The instant that new bit of data arrives in the cache instance(s) in each other region, it's picked up by the KeySubscriber process which fires off a CableReady broadcast to each client connected to that region. That's also going out through a persistent Websocket connection from the server to it's clients, so: it's also really fast. I haven't tried implementing pingtime checks for this yet (maybe that's an experiment for another day), but each each step in that path is potentially a few milliseconds.
So... it's kind of a low-latency geographically-distributed event bus system hooked up to Websockets on both ends? It's pretty easy to replicate, too. No pun intended. If you're into crunchy replication details though; it has eventual-consistency and a last-write-wins strategy.
Rails is Exciting!
Whilst the primary motivation for this mini-app was pure curiosity, it was also a nice chance to demonstrate some things that are currently possible with Rails. There was no Javascript written for this app, and the live bits use server-side-rendered HTML. With the advent of Rails 7 and Hotwire, and the availability of libraries like StimulusReflex and CableReady filling out the HTML-Over-The-Wire space, there's a world of new options for building reactive real-time apps without the need for creating a vast, sprawling SPA. Or to put it another way: the hottest new frontend framework is... the backend.
There's a sweet spot where StimulusJS, Hotwire/Turbo, CableReady, StimulusReflex can all weave together into a beautiful tapestry (I haven't even scratched the surface of it here, I'll try to find time write about it soon) and what it ultimately looks like is: insane productivity, joyful developer experience, conceptual compression and unbounded potential for creativity.
If you're a full-stack Rails dev and you don't think this is the most mind-blowingly exciting time to be alive, you might want to take a closer look at what options are currently on the table!
Top comments (0)