In the last few years, I have spent an insane amount of time thinking through reactivity on the frontend! In all that we've achieved in the space, the State of Reactivity today still leaves much to be desired:
reactive primitives - in the various mechanisms for intercepting and reflecting change, there is still no easy way around the concept of change detection! From language-level primitives - proxies and accessors - to custom primitives, this has continued to be a tricky game!
syntax and mental model - in the overall language and programming paradigm around the idea, there's still no ergonomic and mentally efficient way to express reactive logic! (Much rigor here often as we try to model normal logic using functions; or try to wield a compiler!)
These challenges greatly impact our work, sometimes in less obvious ways, given how much of the frontend paradigm space this useful magic now occupies, and thus, how much of it accounts for the code we write and our overall tooling budget! For a paradigm so intrinsic to frontend, it couldn't be more compelling to revisit the various painpoints!
To this end, I took a stab:
- at re-exploring the fundamental idea of change detection in JavaScript (i.e. reactive primitives) and the overall language and programming paradigm around reactivity (i.e. syntax and mental model) for some "cleaner" magic!
- at re-exploring full language support for all of it in each case, given how the idea seems to suffer from meager language support and thus, remains, for the most part, externalized to tooling!
This is a pretty long article - with 3 main sections:
- Re-Exploring Change Detection (The design discussion around the Observer API)
- Re-Exploring the Language of Reactivity (The design discussion around Reflex Functions)
- The Duo and the Prospect for Reactivity
Show Full Outline
Update Jan, '24
Reflex Functions has had a major update since this initial announcement! This includes:
- A name change: being now Quantum JS!
- A major API revamp: majorly in the return values of Quantum Functions - from being a tuple
[ value, reflect ] = sum()to being a "State" objectstate = sum()! - An entire compiler overhaul towards a more runtime-driven form of reactivity!
Generally, while the ideas shared here under the term "Reflex Functions" are largely the same with its Quantum JS successor, there are important differences in the implementation!
SECTION 1/3
Re-Exploring Change Detection
Change detection comes as the first challenge to reactivity in JavaScript!
The general idea deals with detecting and responding to changes made to JavaScript objects to drive synchronization - sometimes data binding, but generally, any functionality that is dependent on object state. This in JavaScript has been approached in many different ways past and present.
This wouldn't be about Signals and Hooks, because - paradigm mismatch:
Those "functional" primitives are designed with the Functional Programming paradigm in mind, where immutable data and pure functions are emphasized. Object observability, on the other hand, is more closely associated with the Object-Oriented paradigm, where mutable state and object interactions are the idea. And whereas the former basically represents an intermediary event system, the latter deals with object-level change detection via native mechanisms.
Meanwhile, later in the second half of this article, we will be touching on Signals and Hooks.
Immutability is a programming principle that treats data as unchangeable. Here, the idea of change is modelled by functions that repesent a pipeline of transformations for whole data wherein each transformation creates a new instance, and observability happens at the event level, well, instead of at the fine-grained level. Mutability, on the other hand, embraces the concept of change, wherein objects are modified in place.Compare also: Mutability vs. Immutability
Historically, certain techniques have had their day here! For example, dirty-checking was the technique in Angular; and custom pub/sub mechanisms were the idea in Backbone Models and Ember Models. These and probably other approaches at the time have now falling out of favour for newer, more straightforward solutions!
Accessors and Proxies; the Tricky Parts
Today, object accessors (since ES5) and proxies (since ES6) have greatly revolutionized the problem space! Being integral to the language, these primitives constitute the only way to detect change at the program level:
Accessors
const person = {
// name accessors
get name() { ... },
set name(value) { ... },
// age accessors
get age() { ... },
set age(value) { ... },
};
Proxies
// Original object
const $person = {
name: 'John',
age: 30,
};
// Proxy wrapper
const person = new Proxy($person, {
get(target, key) { ... },
set(target, key, value) { ... }
});
Such that we're able to hide a ton behind a dot syntax:
person.name = 'James';
person.age = 40;
Giving a clean interface where you'd normally have needed intermediary get()/set() mechanisms is perhaps what makes these so nifty that you definitely want to use them! But then comes additional details to walk through, where you end up with an unexpected amount of boilerplate and a tricky implementation!
Let's add here that there's something on the horizon: Decorators! (On which Dr. Axel Rauschmayer takes a deep dive in their latest, stage 3, syntax.) But it turns out, while decorators might be solving something in the broader scope of meta programming in JavaScript, they aren't adding anything to change detection other than syntax sugars over existing possibilities! So, if the question is object observabilty, I'm the least excited about decorators!
Accessors Got a Flexibility Problem There
It's probably the biggest deterrent to using them: the inflexibility of working per-property and not supporting on-the-fly properties! This requires you to make upfront design decisions around all the possible ways an object could change at runtime - an assumption that's too unrealistic for many dynamic applications; one that fails quickly if you tried!
Accessors were the primary reactive mechanism in Vue 2, and that's probably the best place to see their reallife limitations - the biggest of which is perhaps their inability to detect new property additions! This became a big enough issue for the framework that they didn't hesitate to move away when they had the chance to!
Proxies Got an Identity Problem There
It turns out that for every "proxy" instance, you practically have to account for two different object identities, and that comes very problematic for code that relies on type checks (x instanceof y) and equality checks (x === y). Heck, even two proxy instances with the same object wouldn't equate, due to their different identities!
As David Bruant would have me do here, let's add that with some hack, you could get "instanceof" working correctly: via the
getPrototypeOftrap. (But I guess that doesn't make the whole idea any less tricky but more!)And maybe let's reference Tom van Cutsem on the concept of membranes and Salesforce's implementation of the idea - wherein you're able to have all references to the same target return the same proxy instance, and thus have references pass "equality checks"! (But I guess membranes aren't what you're really out here to do!)
All of this is probably another way to see the problem for reactivity with proxies: tricky, no easy way out!
The Whole Idea of Internal Traps
It comes as a gap between the communication model that those internal methods enable and the typical model around reactivity: these primitives don't bring with them the regular subscription model wherein anyone can subscribe to changes on a subject! They just give you a way to intercept - in terms of internal traps - but not a way to observe from the outside - in something like a .subscribe() method! And in however you go from here, this becomes a tricky game all the way:
-
Tricky if you were to take those internal methods for generic "observers"! In the case of accessors, you couldn't have another "observer" without re-defining a property, and thus inadvertently displacing an existing one! And in the case of proxies, while you wouldn't be able to displace traps anyway as those are finalized at instance time (i.e. closed-in by the constructor), you also couldn't have another "observer" without, this time, creating another proxy instance over same object, and thus inadvertently ending up with multiple redundant proxies that are each tracking very different interactions - giving you a leaky, bypassable tracking mechanism each time!
Accessors
Object.defineProperty(person, 'age', { set() { /* observer 1 */ } }); // Singleton Error: A new observer inadvertently breaks the one before Object.defineProperty(person, 'age', { set() { /* observer 2 */ } });Proxies
let $person1 = new Proxy(person, { /* observer 1 */ }); goEast($person1); // Singleton Error: A new observer inadvertently creates redundancy let $person2 = new Proxy(person, { /* observer 2 */ }); goWest($person2);Turns out, those internal methods don't at all add up to a reactive system! Until you actually build one, you don't have one!
-
Tricky in going from those internal methods to a "reactive system"! You'd need to build an event dispatching mechanism that those internal methods get to talk to, and that couldn't happen without having to walk through new technical details! And there lies many pitfalls! For example, it's quick and easy to inadvertently introduce a breaking change to public object interfaces - e.g.
window.document- by polluting/patching their namespace with setters and getters, or with something like a.subscribe()method!
Accessors
Object.defineProperty(window.document, 'custom', { set() { /* observer 1 */ } });
const person = { // Namespace pollution _callbacks: [], subscribe(callback) { this._callbacks.push(callback), }, // Accessors get name() { ... }, set name(value) { ... }, };Proxies
const _callbacks: [], function subscribe(callback) { _callbacks.push(callback); } const $person = new Proxy(person, { get(target, key) { // Namespace pollution return key === 'subscribe' ? subscribe : target[key]; }, set(target, key, value) { ... } });Turns out, there's no direct/neat path to go from "internal methods" to "reactive system"!
The Whole Idea of "Magic Objects"
That comes as the narrative for the form of reactivity that these techniques enable, and the implication is in terms of the final model that these magic objects create: a system where observability is an object-level feature, or in other words, an object-level concern! This comes with much complexity and multiple points of failure at the granular level!
Being a concern at such a granular level of your architecture, "reactivity" now tends to govern every detail of your work: objects have to be purpose-built for observability, and existing ones retrofitted; and in all you do, you must be explicit about reactivity! And because it is hard to gain and easy to lose, you also have to be conscious about how each object flows through your application; and there lies the footguns and bottlenecks:
-
Tricky to account for across the various points of interaction! Given a system where functionality relies on data and reactivity being bundled together and passed around together, it happens so easily that the latter is lost on transit without notice! That becomes a big problem in any fairly-sized codebase!
Accessors
// Using the accessor-based ref() function in Vue.js import { ref, watchEffect } from 'vue'; const count = ref(0); // Reactivity gained watchEffect(() => { console.log('Count value changed:', count.value); }); function carelessHandling(count) { delete count.value; count.value = 2; } carelessHandling(count); // Reactivity lostIt turns out, any reactive system based on "magic objects" is susceptible to failure at the granular level!
-
Tricky in dealing with object trees! You couldn't gain reactivity beyond the surface without some form of recursive transformations, and yet all of that would still be subject to the assumption that your object tree is a static structure! Things get extra tricky this time because, in reality, that assumption isn't always the case!
Proxies
// Using the depth-capable reactive() function in Vue.js import { reactive } from 'vue'; // Creating a reactive object tree const reactiveTree = fetch(resourceUrl).then(res => res.json()).then(json => { return reactive(json); }); // Problem reactiveTree.entries.push({ ... });Turns out, any reactivity gained from stitching proxies or accessors together is most often than not unpredictable and rickety!
A Way to Think About the Problem
Proxies and accessors constitute the only native way today to do mutation-based reactive programming - not particularly because they perfectly reflect the desired approach, but because they're the only native way! However, this isn't a case of poorly designed language features, this is rather about usecase mismatch! Whereas proxies and accessors have their perfect usecases across the broader subject of meta programming, reactive programming represents a usecase not well captured; or in other words: something that isn't very much their usecase!
Proxies, for example, are mind blowing in what they can do across the broader subject of meta programming, having had a massive technical and academic work go into their design (as can been seen from ES-lab's original design document), and it is on this backdrop they seem to be the silver bullet to reactivity! But it turns out that it is on just a subset of that - typically something around three internal traps:
get,set,deleteProperty- that we're building our reactive world, and well, the rest of which isn't fully captured in the design of proxies!
While object observability as a technique may have an overlap with the usecase for proxies and accessors, it isn't the main thing these were designed for, and these primitives conversely don't come ideal for the problem case! If I were to represent that mathemaically, that would be 25% overlap, 75% mismatch:
This leaves us with a desire to find something made for the problem! And it turns out, this isn't the first time we're hitting this wall!
Rediscovering Prior Art and Nailing the Problem
This is where history helps us!
Amidst the remnants lies a design precedent to object observability waiting to be re-discovered: the Object.observe() API, which was depreciated in November 2015! It probably was ahead of its time, but in its 8 years in the mud, there's something time has taught us: the platform needs native object observability, and it's okay to have things coexist for different usecases!
Consider the approach in this API:
Object.observe(object, callback);
Notice that, now, the tracking mechanism is separate from the objects themselves, and what's more, universal for every object! (It's much like the upgrade from .__defineSetter__() and .__defineGetter__() to Object.defineProperty()!) And this basically solves all of our problems today with primitives!
- Now, we can put all of that "magic object" idea behind us: observability shouldn't have to be an object-level feature (or object-level concern)!
- We no more have to be explicit about reactivity; or be governed by that concern!
- It shouldn't have to be gained or lost; or be passed around together with data!
- Now everything just works on the fly; no more upfront work nor dealing with internal traps!
- No more patching objects, nor transforming object trees!
- Now we can interact with real objects across our codebase: things no more rely on intermediary functions or wrappers!
- No more accounting for two different object identities or hacking around that!
- No more dealing with leaky, bypassable tracking mechanism!
And doesn't that nail it? Why shouldn't we explore this further?
Having struggled around object observability myself over the years, I find it necessary to re-explore this prior art today as a new project: the Observer API!
Introducing the Observer API
This is a new effort that re-explores object observability along the lines of Object.observe(), but this time, with a more wholistic approach! It takes a leap at what could be a unifying API over related but disparate APIs like Object.observe(), the "Proxy traps" API, and the Reflect API (which is a complementary API to proxies)!
This is an upcoming proposal!
An Overview
As an improvement over its original form - Object.observe(), the object observability API is being re-imagined!
Whereas the previous API lived off the global "Object" object, the new idea is to have it on the global "Reflect" object as if being one of the missing pieces in a puzzle! (But this is subject to whether everything considered here still falls within the scope of the Reflect API.) But that's just one of two possibilitis considered! The other is to have it on a new Reflect-like object called Observer, which is, this time, a fully-featured reactivity API!
While that unfolds, a prototype of the Observer API exists today as Observer, featuring a superset of each Reflect method in addition to other specialized APIs. Much of the code here is based on this Observer namespace.
| Observer API | Reflect API |
|---|---|
apply() |
apply() |
construct() |
construct() |
observe() |
- |
set() |
set() |
setPrototypeOf() |
setPrototypeOf() |
Being fully compatible with the Reflect API, the Observer API can be used today as a drop-in replacement for Reflect.
To begin, here's the observe() method:
└ Signatures
// Observe all properties
Observer.observe(object, callback);
// Observe a list of properties
Observer.observe(object, [ propertyName, ... ], callback);
// Observe a value
Observer.observe(object, propertyName, inspect);
└ Handler
function callback(mutations) {
mutations.forEach(inspect);
}
function inspect(m) {
console.log(m.type, m.key, m.value, m.oldValue, m.isUpdate);
}
From here, additional features become necessary! We discuss some of these here and link to the rest.
Featuring Path Observability
A much needed feature for Object.observe() was path observability! The void that initially having it left out created required much boilerplate to fill. A few polyfills at the time (e.g. Polymer's observe-js) took that to heart. And that certainly has a place in the new API!
This time, instead of following a string-based "path" approach - level1.level2 - a path is represented by an array - a Path array instance:
const path = Observer.path('level1', 'level2');
An array allows us to support property names that themselves have a dot in them. And by using Observer's Path array instance, we are able to distinguish between normal "property list" array - as seen earlier - and actual "path" array.
The idea here is to observe "a value" at a path in a given tree:
// A tree structure that satisfies the path above
const object = {
level1: {
level2: 'level2-value',
},
};
Observer.observe(object, path, (m) => {
console.log(m.type, m.path, m.value, m.isUpdate);
});
object.level1.level2 = 'level2-new-value';
Console
type
path
value
isUpdate
set[
level1, level2, ]level2-new-valuetrue
And well, the initial tree structure can be whatever:
// A tree structure that is yet to be built
const object = {};
const path = Observer.path('level1', 'level2', 'level3', 'level4');
Observer.observe(object, path, (m) => {
console.log(m.type, m.path, m.value, m.isUpdate);
});
Now, any operation that changes what "the value" at the path resolves to - either by tree extension or tree truncation - will fire our listener:
object.level1 = { level2: {}, };
Console
type
path
value
isUpdate
set[
level1, level2, level3, level4, ]undefinedfalse
Meanwhile, this next one completes the tree, and the listener reports a value at its observed path:
object.level1.level2 = { level3: { level4: 'level4-value', }, };
Console
type
path
value
isUpdate
set[
level1, level2, level3, level4, ]level4-valuefalse
If you were to find the exact point at which mutation happened in the path in an audit trail, use the event's context property to inspect the parent event:
let context = m.context;
console.log(context);
And up again one level until the root event:
let parentContext = context.context;
console.log(parentContext);
And you can observe trees that are built asynchronously! Where a promise is encountered along the path, further access is paused until promise resolves:
object.level1.level2 = Promise.resolve({ level3: { level4: 'level4-new-value', }, });
Documentation
Visit the docs for full details - including Timing and Batching, full API Reference, and more.
The Project
The Observer API is being developed as something to be used today - via a polyfill. The polyfill features all of what's documented - with a few limitations!
webqit
/
observer
A simple set of functions for intercepting and observing JavaScript objects and arrays.
The Observer API
Motivation • Overview • Documentation • Polyfill • Getting Involved • License
Observe and intercept operations on arbitrary JavaScript objects and arrays using a utility-first, general-purpose reactivity API! This API re-explores the unique design of the Object.observe() API and takes a stab at what could be a unifying API over related but disparate things like Object.observe(), Reflect APIs, and the "traps" API (proxy traps)!
Observer API is an upcoming proposal!
Motivation
Tracking mutations on JavaScript objects has historically relied on "object wrapping" techniques with ES6 Proxies, and on "property mangling" techniques with getters and setters. Besides how the first poses an object identity problem and the second, an interoperability problem, there is also much inflexibility in the programming model that each enables!
This is discussed extensively in the introductory blog post
We find a design precedent to object observability in the Object.observe() API, which…
Show support with a star on github! Also, all forms of contributions are welcome at this time.
SECTION 2/3
Re-Exploring the Language of Reactivity
While we may have fixed object observability, we still need to find a way to actually write reactive logic - in how we normally would write JavaScript programs! There's a big difference here, and it's like the difference between playing individual musical notes and composing a song!
But while this constitutes a different problem from what object observability solves, they've got be related, and that's definitely where this journey culminates!
Come to writing application logic...
Here is how we learned to write applicatons (notice: in normal, imperative JavaScript):
let count = 5;
let doubleCount = count * 2;
console.log(doubleCount); // 10
and the challenge for reactive programming is to find a way to express the same such that the logic...
count -> doubleCount -> console.log(doubleCount)
...continues to hold even when a part of the dependency chain is updated!
This magic by itself has had no formal language, yet it has managed to become the most spoken language on the frontend today! But unfortunately, reactivity is a difficult language to speak:
This has involved compiler-driven syntaxes and whole new DSLs, and on a more general note, making the shift to writing applications under an entirely different programming paradigm: Functional Programming! And there lies the big problem for the average developer: that paradigm shift from "normal" JavaScript!
It just happens that there's no easy way around the idea! And when you look: what always goes for reactivity is syntax and mental model! Only, some approaches can be considered better off than others on either of those factors.
The Toll On Syntax and Mental Model
Enter the typical language of reactivity: Functional Programming...
Whereas imperative programs are written in literal terms and are based on a linear execution flow (a sequential, line-by-line evaluation of statements and control flow structures), functional programming requires a series of change detection primitives to model the given application logic!
Given our sample program in context, here's what the "functional" equivalent could look like across frameworks, using a pretty identical set of primitives:
React
import { useState, useMemo, useEffect } from 'react';
// count
const [count, setCount] = useState(5);
// doubleCount
const doubleCount = useMemo(() => count * 2, [count]);
// console.log()
useEffect(() => {
console.log(doubleCount);
}, [doubleCount]);
// Update
setCount(10);
SolidJS
import { createSignal, createMemo, createEffect } from "solid-js";
// count
let [count, setCount] = createSignal(5);
// doubleCount
let doubleCount = createMemo(() => count() * 2);
// console.log()
createEffect(() => {
console.log(doubleCount());
});
// Update
setCount(10);
Vue.js
import { ref, computed, watchEffect } from 'vue';
// count
const count = ref(5);
// doubleCount
const doubleCount = computed(() => count.value * 2);
// console.log()
watchEffect(() => {
console.log(doubleCount.value);
});
// Update
count.value = 10;
Svelte
import { writable, derived } from 'svelte/store';
// count
const count = writable(5);
// doubleCount
const doubleCount = derived(count, $count => $count * 2);
// console.log()
doubleCount.subscribe(value => {
console.log(value);
});
// Update
count.set(10);
MobX
import { observable, computed, autorun } from 'mobx';
// count
const count = observable.box(5);
// doubleCount
const doubleCount = computed(() => count.get() * 2);
// console.log()
autorun(() => {
console.log(doubleCount.get());
});
// Update
count.set(10);
What do we have here? A common reactive language, and, well, the problem case: a programming paradigm shift!
What is in traditional JavaScript a literal variable declaration is in the functional approach a very different type of declaration:
// Variable
let items = ['one', 'two'];
// Read/write segregation
const [items, setItems] = createSignal(['one', 'two']);
Also, what is to traditional JavaScript a "mutable" world is to the functional approach an "immutable" world (or you end up defying the reactivity system):
// Mutation world
items.push('three');
// Immutable world
setItems([...items(), 'three']);
And the story goes the same for conditional constructs, loops, etc.!
The focus entirely shifts now from writing idiomatic JavaScript to following paradigm-specific constraints, and in addition, other implementation-specific details:
- In the Signals family: that technique around fine-grained reactivity - wherein you have to carefully craft, or "group", logic "around data flow" to actually stay fine-grained. Also, that extra invokation syntax -
count(),count.get(),count.value- that goes with what was supposed to be a literal variable. - In the Hooks family (React, Preact): the underestimated dependency array, etc.
All of this on top of the required diligence needed to manually model the control flow and dependency graph of your application logic - by piecing together multiple primitives! Anyone who's been there can tell how the many moving parts and high amount of abstraction and indirection significantly impact the authoring experience and cognitive load!
It turns out that this is a problem with any "runtime" magic as there must always be a thing to put up with: syntax noise and other ergonomic overheads, a difficult-to-grok execution model and other cognitive overheads, etc.!
This has been a strong enough case across frameworks that everyone has had to support their "functional" magic with some other type of magic that tries to solve either for syntax or for mental model!
Solving for Syntax
Vue, for example, will help you automatically unwrap certain refs used in .vue templates, and in a few other contexts. (The idea even made it as a more extensive syntax transform project!) Similarly, in React, work is underway to put memoization and more behind a compiler!
Svelte stands out here as it goes full-blown with the idea of a compiler to let you write whole logic in literal, imperative JavaScript in .svelte templates, on top of its "functional" core:
<script>
let count = 0;
$: doubleCount = count * 2;
</script>
<main>
<button on:click={() => count += 1}>Double Count: {doubleCount}</button>
</main>
Only, it suffers from being unintuitive by the "twist" - the magic - that went into it: compiling from literal, linear syntax to a functional, non-linear execution model, thus defying the general expectation of the "imperative" paradigm it appears to go by!
For example, as documented:
<script>
export let person;
// this will update `name` when 'person' changes
$: ({ name } = person);
// don't do this. it will run before the previous line
let name2 = name;
</script>
And you can see that again here in how the program seems to go bottom-up:
<script>
let count = 10;
$: console.log(count); // 20
// Update
$: count = 20;
</script>
There goes the non-linear execution model behind the imperative syntax! So, it turns out, what you get isn't what you see when you look in the face of your code, but what's documented! Indeed, Svelte scripts "are nothing like 'plain JavaScript' yet people seem to be accepting of those and some even advertising them as such." - Ryan Carniato
Nonetheless, Svelte's vision underscores the strong case for less ergonomic overheads in plain, literal syntax - the bar that every "syntax" quest actually has in front of them! Only, this cannot be rational with a functional, non-linear core - as seen from its current realities!
Solving for Mental Model
React has long stood on what they consider a "simpler" rendering model:
Whereas in normal functional programming, changes propagate from function to function as explicitly modelled, React deviates from the idea in its "hooks" approach for a model where changes don't propagate from function to function as explicitly modelled, but propagate directly to the "component" and trigger a full "top-down" re-render of the component - giving us a "top-down" rendering model!
The pitch here is:
"You don't have to think about how updates flow through your UI anymore! Just rerender the whole thing and we'll make it fast enough." - Andrew Clark
And in relation to Signals:
"That's why I don't buy the argument that signals provide a better DX/mental model." - Andrew Clark
Which from this standpoint should remain highly-protected:
"I think you guys should be way more aggressive in pushing back against this stuff. Throwing out the react programming model and adopting something like signals (or worse, signals plus a dsl) is a huge step backwards and most people haven’t internalized that." - Pete Hunt
And when you look, you see a vision here for a "linear" execution model, much like the linear "top-down" flow of regular programs!
Only, it suffers from being counterintuitive by itself with the "twist" - the magic - that went into it: defying the general expectation of the "functional" paradigm it appears to go by! So, it turns out, what you get isn't what you see when you look in the face of your code, but what's documented, which for many devs makes the Signals approach more grokable, wherein "functional" is "functional":
"its funny, the first time i ever used signals wasn't for perf, it was for understanding where change happens easier. plus you get LSP references to change.
just seems.... most rational" - ThePrimeagen
Being as it may:
"The React thought leadership had long cultivated the narrative of 'our model is more correct', almost becoming a dogma. In reality, many engineers are simply more productive with the signals mental model - arguing on a philosophical level is pointless when that happens." - Evan You
And the whole idea seems to fall short again with the extra useMemo/useCallback performance hack needed to control the "re-render" mdoel!
Nonetheless, React's vision underscores the strong case for less mental overheads in terms of a linear execution model - the very thing we hope to regain someday! Only, this doesn't seem to be what function primitives can provide rationally - as seen from its current realities!
Solving for Both
Given the ongoing quest for reactivity without the ergonomic and mental overheads, the new bar for reactivity is to not solve for one and fall short on the other! We may not be hitting the mark in a single implmenentation now, but not until more and more people discover the power of compilers and make the full shift!
"Smart compilers will be the next big shift. It needs to happen. Look at the optimization and wizardry of modern C compilers. The web needs similar sophistication so we can maintain simpler mental models but get optimized performance." - Matt Kruse
In their ability to take a piece of code and generate a different piece altogether, compilers give us a blank check to write the code we want and get back the equivalent code for a particular problem, becoming the hugest prospect for reactive programming! Soon on the Language of Reactivity, we may never again talk about "functional" primitives or "functional" core! And there's only one place this leads: reactivity in just plain JavaScript; this time, in the literal form and linear flow of the language!
"It’s building a compiler that understands the data flow of arbitrary JS code in components.
Is that hard? Yeah!
Is it possible? Definitely.And then we can use it for so many things…" - sophie alpert
There goes the final bar for reactivity! It is what you see given half the vision in Svelte: "syntax" (assuming a path fully explored to the end) and half the vision in React: "top-down flow" (assuming a path fully explored to the end); the point where all quests on the Language of Reactivity culminate!
But what if everything in this "reactivity of the future" thesis was possible today, and what's more, without the compile step?
Bringing Reactivity to Imperative JavaScript and Nailing the Problem
You'd realize that the idea of "building a compiler that understands arbitrary JS code" is synonymous to bringing reactivity to "arbitrary" JavaScript, and if we could achieve that, we could literaly have reactivity as a native language feature! Now this changes everything because that wouldn't be a compiler thing anymore!
Can we skip now to that fun part?
We explore this by revisiting where we've fallen short of bringing reactivity to "arbitrary" JavaScript and fixing that! Svelte comes as a good starting point:
In Svelte today, reactivity is based on the custom .svelte file extension! (Being what carries the idea of a reactive programming context in the application!)
calculate.svelte
<script>
// Code here
</script>
But what if we could bring that reactive programming context to a function...
...for a handy building block that can be easily composed into other things?
function calculate() {
// Code here
}
And next is how you express reactivity!
That, in Svelte today, relies on the dollar sign $ label - being another type of manual dependency plumbing, albeit subtle:
<script>
let count = 10;
// Reactive expressions
$: doubleCount = count * 2;
$: console.log(doubleCount); // 20
</script>
But what if we simply treated every expression as potentially reactive...
...having already moved into a reactive programming context?
function calculate() {
let count = 10;
// Reactive expressions
let doubleCount = count * 2;
console.log(doubleCount); // 20
}
And lastly, the execution model!
Updates propagate in Svelte today "functionally", and thus, in any direction, to any part of the program:
<script>
let count = 10;
// Reactive expressions
$: doubleCount = count * 2;
$: console.log(doubleCount); // 40
// Update
$: count = 20;
</script>
But what if we could get updates to propagate "top-down" the program...
...and actually regain the "top-down" linear flow of actual imperative programs?
function calculate() {
let count = 10;
// Reactive expressions
let doubleCount = count * 2;
console.log(doubleCount); // 20
// Update
count = 20;
}
Well, notice now that the update to count in the last line wouldn't be reactive! And that brings us to the question: does anything ever react to anything in this model? The answer lies in what "dependencies" mean in normal programs!
Conventionally, programs run in sequential, linear flow, along which references to "prior" identifiers in scope create "dependencies"! This means that statements should really only be responding to changes happening up the scope, not down the scope (as the case may be in Svelte today)! And that for a function scope would be: changes happening "outside" the scope - up to the global scope!
function calculate(count) { // External dependency
// Reactive expressions
let doubleCount = count * 2;
console.log(doubleCount); // 20
}
let value = 10;
calculate(value);
count is now a dependency from "up"/"outside" the function scope - by which reactivity inside the function scope is achieved! But what would that even look like?
Imagine where the function has the ability to just "statically" reflect updates to its external dependencies:
// An update
value = 20;
// A hypothetical function
reflect('count'); // "count" being what the function sees
not particularly in their being a parameter, but particularly in their being a dependency:
let count = 10; // External dependency
function calculate() {
// Reactive expressions
let doubleCount = count * 2;
console.log(doubleCount); // 20
}
calculate();
// An update
count = 20;
// A hypothetical function
reflect('count'); // "count" being what the function sees
And that brings us to our destination - this point where we've checked all the boxes; where:
- The "reactive programming context" isn't some file, but a function - a handy new "reactivity primitive"!
- Reactivity isn't some special language or special convention, but just JavaScript - the only real place you'd arrive if you tried!
- Execution model isn't anything "functional" by which we defy the semantics of literal syntax, but "linear" - just in how imperative JavaScript works!
All of this in real life is Reflex Functions!
Introducing Reflex Functions
Reflex Functions are a new type of JavaScript function that enables fine-grained Reactive Programming in the imperative form of the language - wherein reactivity is drawn entirely on the dependency graph of your own code!
This is an upcoming proposal! (Introducing Imperative Reactive Programming (IRP) in JavaScript!)
An Overview
Reflex Functions have a distinguishing syntax: a double star notation.
function** calculate() {
// Function body
}
See Formal Syntax for details.
Function body is any regular piece of code that should statically reflect changes to its external dependencies:
let count = 10; // External dependency
function** calculate(factor) {
// Reactive expressions
let doubled = count * factor;
console.log(doubled);
}
Return value is a two-part array that contains both the function's actual return value and a special reflect function for getting the function to reflect updates:
let [ returnValue, reflect ] = calculate(2);
console.log(returnValue); // undefined
Console
doubled
returnValue
20undefined
The reflect() function takes just the string representation of the external dependencies that have changed:
count = 20;
reflect('count');
Console
doubled
40
Path dependencies are expressed in array notation. And multiple dependencies can be reflected at once, if they changed at once:
count++;
this.property = value;
reflect('count', [ 'this', 'property' ]);
Change Propagation
Reactivity exists with Reflex Functions where there are dependencies "up" the scope to respond to! And here's the mental model for that:
┌─ a change happens outside of function scope
└─ is propagated into function, then self-propagates down ─┐
Changes within the function body itself self-propagate down the scope, but re-running only those expressions that depend on the specific change, and rippling down the dependency graph!
Below is a good way to see that: a Reflex Function having score as an external dependency, with "reflex lines" having been drawn to show the dependency graph for that variable, or, in other words, the deterministic update path for that dependency:
It turns out to be the very mental model you would have drawn if you set out to think about your own code! Everything works in just how anyone would predict it!
Plus, there's a hunble brag: that "pixel-perfect" level of fine-grained reactivity that the same algorithm translates to - which you could never model manually; that precision that means no more, no less performance - which you could never achieve with manual optimization; yet, all without working for it!
Documentation
Visit the docs for details around Formal Syntax, Heuristics, Flow Control and Functions, API, and more.
The Project
Reflex Functions is being developed as something to be used today - via a polyfill. The polyfill features a specialized compiler and a small runtime that work together to enable all of Reflex Functions as documented, with quite a few exceptions.
webqit
/
quantum-js
A runtime extension to JavaScript that enables us do Imperative Reactive Programming (IRP) in the very language!
Quantum JS
Overview • Creating Quantum Programs • Implementation • Examples • License
Quantum JS is a runtime extension to JavaScript that enables us do Imperative Reactive Programming (IRP) in the very language! This project pursues a futuristic, more efficient way to build reactive applocations today!
Quantum JS occupies a new category in the reactivity landscape!
Overview
Whereas you currently need a couple primitives to express reactive logic...
import { createSignal, createMemo, createEffect } from 'solid-js';
// count
const [ count, setCount ] = createSignal(5);
// doubleCount
const doubleCount = createMemo(() => count() * 2);
// console.log()
createEffect(() => {
console.log(doubleCount());
});
// An update
setTimeout(() => setCount(10), 500);
Quantum JS lets…
Show support with a star on github! Also, all forms of contributions are welcome at this time.
SECTION 3/3
The Duo and the Prospect for Reactivity
The magic that drives Frontend has got all the more powerful and completely intuitive!
From where I am sitting, I can see the amount of heavy-lifting that the Observer API will do for large swaths of today's usecases for "internal traps" and the whole "magic objects" idea! (It's probably time for that "revolution" that never took off!)
And we've certainly needed language support for reactive logic! Think of the different proposals out there around JavaScript-XML-like DSLs - which of course don't translate well to a native ECMAScript feature! And consider how much we've had to rely on compilers today for the idea!
Now that we can get "arbitrary" JavaScript to be reactive with Reflex Functions, a huge amount of that can be met natively! I am more than excited at how much we can regain using just JavaScript in each of those areas where we've relied on tooling!
I presume that experimenting around these new ideas, using the polyfill in each case, will furnish us new learnings around the problem! For now, I am happy to have started the conversation!
Now, did you know that these two reactive primitives could individually, or together, solve very unlikely problems? Here's where you can find early examples:
If you'd find this an interesting idea, feel free to share!
Acknowledgements
- Much thanks to David Bruant, Andrea Giammarchi, and others for the various feedbacks and insights on early drafts of this article.
References
- What is reactivity? - a Vue.js definition
- A Brief History of Reactivity - a Miško Hevery narrative
-
Object.observe()- web.dev - On the design of the ECMAScript Reflection API - ES-Lab
- What is a membrane? - Tom van Cutsem
- Observable Membrane - Salesforce
- A Hands-on Introduction to Fine-Grained Reactivity - Ryan Carniato
- The Quest for ReactiveScript - Ryan Carniato
- Rethinking Reactivity - Rich Harris
- React Forget - React
- Reactivity Transform - Vue.js
Top comments (14)
Hi Oxford, this is a crazy thought. Thoroughly enjoyed the analysis behind each of the APIs! Like, there's much I never really figured out before now.
At the end of the day, you realize that much of the sophistication around reactivity isn't really for the application, **nor for the developer. They just cater to "the approach"! For example, who really needs "functional" primitives? Is it particularly the runtime, the application, or the developer?
Not that these don't have a use case, but we seem to have made what's supposed to be the exception the default!
I think the Observer API and Reflex Functions will be a huge leap for reactivity!
Hi Godswill, I'm excited to hear that, and to have you try things out with the polyfill!
We have seen many new approaches over the last few years, that are intendeded to make life easier. Many solve one problem by creating two new ones. And - any new approach brings a new layer of complexity to the game - which is not desireable at all.
Let me give an example from Svelte. I really like the Svelte-approach, but - for me - it has a conceptual weakness. Here is an example, that displays an array
With their approach, they had to implement new control elements like #each, #if, #else, that introduce their own syntax and new rules. But we already have JS working on this site, so why can´t we use this?
In my oppinion, it would be far better if they could use JS directly like this:
There are good reasons why they did it differently, but here we get a bunch of new concepts, that need to be explained and which are a potential source of confusion and errors. Using the conventional syntax of Javascript would have removed this whole layer (possibly bringing other problems).
I still have no clear opinion about the benefit of Reflex Functions, but we always should ask:
I couldn't agree more that along with what seems like a solution often comes some more complexity! Template languages and custom DSLs have never seemed like a good-enough answer to me! They just double our syntax space!
Wow. I think you just led us to one thing that Reflex Functions addresses! (How come that wasn't immediately obvious?)
This is what you're looking for, which is possible with Reflex Functions today:
It doesn't look like there's Reflex Functions present at all. That's right; this is plain JavaScript that would work as is! Reflex Functions comes in when you need "reactivity" on top of this plain logic, in which case, you could imagine flipping a "reactivity" feature for your script in something like a Boolean attribute (which I'll talk about next):
That would signify that the contents of the script should be evaluated within a Reflex Function! So under the hood, that would translate to:
from which point, you have the
reflect()function for reflecting updates!Point is, Reflex Functions can be your compile target - in this case, for your
<script>elements; "reflex" scripts!In fact, something like "reflex" scripts might be by far the most common way people will experience Reflex Functions! (Or may be not. But point is, this is a primitive that can be under the hood of anything.)
Now, lest you're imagining having Svelte work this way today, "reflex" scripts are here already (and hopes to be a companion proposal to Reflex Functions) at OOHTML (This might be really worth your time!)
With "reflex" scripts, you don't need a compile step at all as the case may be in Svelte today; everything hits the browser running! And here's a good way to see that:
a list element that receives an array to render from a running application, such that when the list is updated by the application, the loop runs again to re-render:
You may want to see how reactivity works with Loops, and other control flow constructs:
Hy Oxford,
maybe I missed a point, but I have great problems understanding the concept of Reflex Functions.
As far as I understood, they allow to rerun parts of the code, depending on the change of external data, right? This gives me a lot of questions:
Any hint´s are welcome
Hi Eckehard,
Would it help to link you to the "Usecases" section of the README?
Could at least give us some context. If you pick one of the usecase examples, I could clarify anything about it!
Does that help? Or have a specific case in mind?
Meanwhile, to answer ques #1 off the top of my head...
If you have a global dependency in your Reflex Function:
or in a script element that compiles to the same
then the idea is that you should observe that property for change and get the function to reflect it:
and in the script example
So, in other words, the observer API makes it automatic!
And from your question, if a global change happens and isn't part of the "Reflex System
" (by which I think you mean: a global thing that isn't referenced at all in a Reflex Function), then it rightly has no effect within Reflex Functions! Changes that aren't a dependency don't need to be observed and reflected, and even if
reflect()is called - reflect(['globalThis', 'property2']), it won't have an effect!As a tip, if you were to know the exact external dependencies that needs to be observed, the
ReflexFunction.inspect()method shows you just that!Meanwhile, the fact that Reflex Functions don't concern themselves with change detection on the outside world is by design! It allows them to fit with different ways things change on the UI; e.g. via events, etc. They just want to do one thing well: accept a textual representation of what has changed on the outside scope and scan their own scope top-down to re-run dependent statements!
Reactivity with Reflex Functions is all based on static source text analysis, and that's fundamentally different from the "callbacks-and-closures network" nature of the "functional" approach to reactivity! This text-synthesis approach is cheaper - because it happens as part of the source code parsing process within the engine - and saves much runtime overheads on your code. It is also the secret to its "literal-syntax and linear-flow" advantage as discussed in the main post!
To answer #2: Reflex Functions don't rely on being pure! It's normally a mutable world in programming and Reflex Functions are designed to embrace that! But if you'd rather follow an immutable principle, they work as perfectly too!
Assume, there is a global variable, that is incremented on every render cycle. Assume, this has an effect "upstream" in your code, e.g. a part of your code is only executed, until the counter is below 10. If you only rerun the downstream part of your code, how should the change be detected?
I do not really understand, what you mean by "up" and "down" the scope? As long, as an application runs, we can say: "After" execution is "before" execution. If the execution of code has and effect on a global variable, this changes the "state" of the whole application. So, it is unsure, if this has an effect on the result.
Data drive vs event driven design
Today, application design if often driven by data, this makes change detection tricky. You only see, that data has changed, but you do not know why this change happend.
If you follow an event driven approach, there is always a "reason", why data changed. If a user hits a button, or if he/she inputs data, this is an event. Even a change in the database can cause an event. This makes it very easy to detect changes, as you only need to check a small amount of data, that are in the scope of the event.
Svelte has a much more serious problem - any exception breaks the entire reactivity system completely. This is the fundamental difference between tasks and invariants: the task can and should fall as early as possible, but the invariant should continue working despite the errors in the middle. Semantically, functions in JS are tasks. An invariant can call tasks within itself, but it is not a task itself.
Unfortunately, what you are proposing is not a revolution, but a dead-end branch of development. Here are just a few problems that I see right off the bat:
I told more about all this here: dev.to/ninjin/main-aspects-of-reac...
It's very sad that you didn't mention this material, but you didn't even hear about it.
Regarding your statement that there is no way for JS to fully use reactive programming without a significant change in semantics - this is not true. Familiarize yourself with the concept of channels and reactive memoization, which are the basis of object-based reactive programming. This is a much more flexible approach in terms of (de)composition.
Read more about it in this article: dev.to/ninjin/designing-the-ideal-...
As a spoiler, look at this, for 9 years now, the code on the $moll framework has been working optimally without glitches without any reactive compilers:
Hi Jin, thanks ofcourse for your response.
I could attempt addressing some things you raised, but I still feel I'm unable to make sense of everything! Maybe some context and specificity would help. (Especially if what you're getting at is a counter proposal, which is itself welcomed.)
If it's what I think I understand,
I think point #1 might be misplaced. I don't think anyone evaluates proposals this way unless what is being proposed is something that is already significantly addressed by existing features.
Point #2 exposes a significant gap between what you might have thought this was and what it really is! We don't rely on compilers, and in fact the whole point is to no more rely on compilers for reactivity!
I'd need more context for point #3. Also the links and the sample code aren't particularly helpful! (For example, is that
decoratorsyntax in your code? And if so, have you found a way to use decorators without a compiler? Also, what is your code doing?)It appears that Jin consistently steers conversations towards promoting his $mol framework. However, I remain unconvinced by his approach.
Please check out this post on medium: Why State Management is All Wrong. Reactivity often is bound tightly to the View-layer of an app. This is ok, if Javascript is just used to create some "reactivity" in the UI. But modern applications often do much more than just creating some nice visual effects.
In traditional programming, the View-layer was a realtively thin layer ontop of a massive application logic. State changes had to be reflected in the UI very fast, so there have been concepts to achieve this efficiently. Most languages featured an event driven approach and a class based system wide communication.
The Web is not that different. There are more bottlenecks, but we should be more specific about that. What precisely causes performance issues? Waiting to fetch some data from a server may be slow, but this is - in the first line - not depending on the framework or technology you use. It is more a result of a bad system design and an overwhelming size of the boilerplate.
On the other hand: using reactivity in an app, that runs completely in the browser, will hardly be very slow, if the UI is not forced to redraw the same content again and again.
It often seems to me, that some of the tools created to make the web faster and our life easier achieve just the opposite...
Thanks for the last sentence; and I can tell you that much of our underlying equations on the web the past decade will eventually be revisited! It's just a matter of time!
And just as Brad Lemley puts it there, State Management is Wrong! Thanks for that link! I might not be fully convinced how the proffered solution answers the question, but I couldn't agree more that:
I wonder why we've treated State as a separate component in our frontend architectures, when that should really be a bi-product of functionality! I look at any piece of UI code today and everything is talking about "data flow" instead of "control flow" (the latter being the way we naturally write programs).
That's why you enter Svelte scripts and it feels different because it's more about the control flow this time and less about the data flow! Then you enter Reflex Functions and its full blown "control flow" on top of which reactivity is based: Imperative Reactive Programming! ("Imperative" being just as you would write your code if there wasn't anything called reactivity!)
People will gain clarity on how a state-first thinking makes life very complicated, but not so fast, I'm sure!
State first thinking is a natural result of stateless programming. Instead of managing state changes, you just rebuild the whole view tree and let your VDOM handle the state changes. This is a completely view centric approach, that dominates the whole application.
Brad Lemley just asks the right questions.
Reactive programming is an approach to make the life of a web designer easier. But I´m pretty sure, it is not viable as a general programming concept for large scale applications.
Managing complexity was the holy grail of computer science for a long time. Putting all variables in a central value store was surely never a solution for that problem.