Update as of Dec 2020: now that the Hey support code has been extracted and officially released under the name Hotwire, we know that this article actually speaks about the Turbo Streams. Basecamp renamed and polished a few things but the principles stay the same as described here.
In parts III and IV we’ve seen how partial page updates can be done via “Turbolinks frames“ (the <turbolinks-frame> element), either automatically or upon request. That was nice and cool but that’s not the only way to do partial updates that Hey supports. In fact, I think that it’s even not the nicest and coolest way to do them! At least when compared to today’s topic - the <template> element…
This element (and associated JavaScript code) handles page updates in many places in Hey. Let’s now study a use case with just about the perfect level of complexity − non-trivial but not too complex: the “Mark seen” function. But first of all, we need to talk about how the main email listing is organized in the Imbox HTML code.
Email listing in Imbox
We all know that the Imbox page has two lists of emails: the new ones and the previously seen ones below. But how do they look in the HTML source? Let’s point our dev tool picker:
Oh, it’s just a single list! A single flat list of <article> elements. How do the new and previously seen emails differ then? As is apparent from the image, the already seen emails have a data-seen="true" attribute, whereas the new ones don’t.
OK, but how is the “PREVIOUSLY SEEN” header made then? Turns out it’s pure CSS and it’s a neat trick (see the image below): the CSS rule with the header in a :before pseudo-class targets the first element in the .postings (emails) list that has the data-seen attribute set. (Either the very first such element in the list when there are no new emails, or the first element after another that has not this attribute set.) So, simply adding this attribute to the email element prepends it with the header!
I think that this is a nice example of a presentation that is handled by a simple HTML structure with a few “sprinkles” of specific data attributes, CSS rules or a bit of JavaScript instead of e.g. handling all this logic in a more complex JS code. It’s an approach which uses the combined strengths of the “classic” technologies that, overall, have been here since many many years ago and are thus very well supported, tested and familiar to devs. No need to learn a new big framework every one or two years! This “composition pattern” can be seen …just about everywhere in Hey and I find it very sensible and appealing. And, most importantly, it plays very well with partial page updates via template elements…
The "Mark seen" action analysis
So what goes on when you select a new email and click on the “Mark seen” popup menu item? Let’s have a look!
Not surprisingly, the Mark seen item is a standard HTML form. It triggers a POST action to the server and the only data it sends along is the IDs of the email(s) selected. As Hey employs Turbolinks, the form is submitted asynchronously via AJAX by default.
BTW, the
data-remote="true"attribute used to do a similar thing in Rails UJS, included by default in ActionView. Rails UJS, or the corresponding ActionView JS code, is not included among Hey JS scripts so I assume, the new Turbolinks handles the async form posting in a different way and this attribute is actually unused.
Anyway, this is not just a standard async form posting, there is one important detail going on: when Turbolinks code intercepts the form submit event, it first dispatches its own custom event called turbolinks:before-fetch-request. (This can be seen in the turbolinks/dist/fetch_request module, which is probably transpiled from TypeScript or similar language and is unfortunately not properly source-mapped, so it’s harder to grasp…) Still before actually fetching the AJAX request, this custom event is handled by JS code in initializers/page_updates.js and it modifies the Accept header in the request:
What does it mean? The Accept header tells the server what Media types (i.e. “types of data”, often called “MIME types”) the browser expects to receive back in the response. Turbolinks adds the "text/html; page-update" media type. Standards-wise, this is a common text/html media type with a custom parameter denoting that the browser will accept a “page update” response from the server and that the response should generally be a HTML response, of course!
So, to sum it up, the AJAX form submit request adjusted by Turbolinks looks like this in the Network tab:
And the server indeed responds with the “page update” media type as can be seen in the content-type response header:
So what does such response look like? Let’s take a look at its body:
Oh at last, here they are, the <template> elements!
Processing the page update response
Right after Turbolinks receives the response, it again triggers a custom event, turbolinks:before-fetch-response (in fetch_request.js), which allows some special treatment of the response. This event is again handled in the same initializer as was the request event − page_updates.js − in the handlePageUpdates method. This method first checks for the custom media type in the response content-type header and if it is present, it calls the processPageUpdates method from a tiny custom library called page-updater:
I assume the reason why this JS code is not part of Turbolinks but is a separate library is that it is, indeed, independent − you can employ such a thing anywhere, you just need to render proper response content on your server.
The page-updater library is pleasantly small and concise. All it does is it extracts all <template> elements that have the data-page-update attribute set, parses them into individual page update commands and executes them, one by one.
The commands are encoded in the <template> elements in a simple way:
<template data-page-update="command#html-id">
...
</template>
where command is the operation that is about to be run and html-id is the… you guessed it… HTML ID of the element the operation should be run against. Optionally, the <template> element can also have its own content, which is needed for some commands. There are five distinct page update commands defined: append, prepend, replace, update and remove. They are quite self-explaining, I think, maybe I’ll just add that the update command leaves the target element intact and only updates its content whereas replace takes away the content as well as the target element itself. Perhaps, it’ll be best to show a picture instead of a ”thousand words“:
Unfortunately, there is no
morphcommand, i.e. you cannot make only the smallest possible changes to the element (such as changing only its class or attribute), you have to resort to updating the whole thing. I see this as a disadvantage when compared to StimulusReflex which leverages the awesome morphdom library to effectively morph elements on the page.
And by the way, as it turns out, <template> element is defined in the HTML standard and denotes “an element for holding HTML that is not to be rendered immediately when a page is loaded but may be instantiated subsequently during runtime using JavaScript”. It seems like a perfect fit for what this element actually does in Hey!
The “Mark seen” response processing
So, let’ get back to the “Mark seen” action. In the response body image above we can see that the response contains two page update commands:
- remove the email element from the page,
- prepend the new version of the email element (given in the response) at the beginning of emails list.
Have you spotted anything weird here? How come the returned mail element is put at the beginning of the emails listing? We already know that we need this email element somewhere in the middle of the list, as it’s a single flat list and we still have some unseen emails at the top!
Eh, you know what? This is getting long and I’m going to cowardly cut this post here. I have a lot more thoughts about this topic! While I generally like this approach very much, I can see some possible caveats, too. Next time, I’ll finish the partial page updates analysis as well as try to discuss this pattern overall, and compare it to the “Turbolinks frames” pattern. Stay tuned and in the mean time you may try to solve the puzzle with mails ordering…! 😉
Top comments (6)
Matouš, you are really knocking these out of the park. Please keep them coming.
This one in particular was enlightening because we (the team behind StimulusReflex) kept asking DHH and co how what they are doing compares to our techniques, which comes down to using the morphdom JS library via CableReady. Finally, we can see that this is the answer to the question that we've been curious about so long.
Have you played much with morphdom? I feel like the next edition would benefit from some kind of practical comparison between what they've cooked up and morphdom. To me, the question is whether there is some distinct advantage to building something new from scratch or whether it's a particularly loud example of "not invented here" syndrome.
Eventually, I'd love for you to break down a comparison of what we can glean from the HEY stack and what we've created in the meantime with StimulusReflex and CableReady. While it's true that both of these projects are websockets specific today, we actually have a functioning message_bus implementation working in the lab, and we aspire to be transport agnostic sooner than later.
If you have any questions about what we're up to, you're always welcome in our Discord: discord.gg/XveN625
Thank you very much for your comment!
Frankly, while I like StimulusReflex a lot as an idea and library, I myself haven’t played with it yet (nor morphdom - just looked at it and it seems lovely) because I just was never that sure about websockets... There's something reassuring for me in pure HTTP that every request has its start and finish and message transfers are nicely guaranteed in a transparent way by TCP itself :). I can understand that websockets are great for async server-generated updates but as a main means of client-server communication channel, I don't know, perhaps I still need to grow into this...
From what I understood about Hey so far, the elements are truly agnostic about the transfer protocol and I've seen them used both in HTTP (as analyzed in this post) and in a websocket channel (try watching the channel while receiving a new email in Hey). This is the biggest difference for me between the two right now but if you say that StimulusReflex will be protocol-agnostic, too, then there's not much left as a difference, I guess! Apart from the feeling that StimReflex is much more advanced (and probably somewhat more complex, too) than these basic template page updates in Turbolinks...
Thanks again!
We figured that getting to a proof-of-concept state with message_bus would take weeks, but our comrade @julianrubisch knocked it out in a casual day of tinkering. It's not ready for prime time, but it's real.
I hear what you're saying about websockets as a transfer medium. We all read Sam Saffron's post a few years back; it was easy to come away with jitters, to say the least. However, in practice... I'm not sure I've heard anyone using ActionCable talk about losing packets.
I definitely don't want to "mansplain" - you clearly know your shit - but it sounds like I have worked more with ActionCable than you at this point. (And I'd worked a lot with Faye before that; then later, Pusher.) What I've come to appreciate is that the Rails team (I assume that it was mostly Sam?) really did an impressive job of borrowing TCP-esque service guarantees for the library. So, while it's a statement of fact that the ActionCable documentation is embarrassingly bad, the framework itself doesn't get nearly enough credit. For example, if the connection gets dropped, it has retry logic built in. It might not be bank/military grade but you can pretty much assume that the packets arrive in the right order. There's no ACK, but you could build a transaction concept with receipt confirmation very quickly if that was a motivation.
The other thing people tend to roast ActionCable for is scalability. What I can say is that when people outgrow ~4000 connections with AC, there's a drop-in socket server replacement written in Go called AnyCable that boasts ~10,000 connections per server, and is horizontally scalable.
As for relative complexity, it's genuinely hard to say. We have really high quality documentation and a growing number of training assets, like the now-infamous Twitter clone video: youtube.com/watch?v=F5hA79vKE_E
We'd be thrilled if you gave it a shot. You might be pleasantly surprised.
Dear @leastbad , thank you, I really hear you. I can say that your comments alone alleviated my anxiety about making an app dependent on websockets tech tremendously and I'm definitely about to give it a try!
Yes, I read the Sam's article and in effect it materialized my vague concerns at that time. I noticed the article when I was doing some initial exploration of the technology and I can say it jumps out of Google really fast... I would definitely love to read a follow up response that would address the issues mentioned in the article (or elsewhere) from today's point of view! Just as you did here!
Thank you for that and I'm looking forward to trying the tech out.
Thank you for these! Appreciate taking the time.
Thanks, Andrei!