Designing the ideal reactivity system

Hello, my name is Dmitry Karlovsky and I am... extremely bad at building social connections, but slightly less bad at building programmatic ones. I recently summarized my eight years of reactive programming experience with a thorough analysis of various approaches to solving typical childhood pain points:

Main Aspects of Reactivity

I highly recommend that you read that article first, to better understand the narrative that follows, where we will develop from scratch an entirely new TypeScript implementation that incorporates all the coolest ideas to achieve unprecedented expressiveness, compactness, speed, reliability, simplicity, flexibility, frugality...

This article is splitted up into chapters, linked with relevant aspects from the above analysis. So if you happen to get lost, you can quickly reconstruct the context.

The narrative will be long, but if you make it to the end, you can safely go to your boss for a promotion. Even if you are your own boss.

Unfortunately, I don't have enough resources to translate it into English, so I offer you original in Russian and automated translation into English.

Next, I have prepared a brief table of contents for you to understand how much is waiting for you there.

Origin

• Different abstractions of state work are examined: fields, hooks, and a new type is introduced - channels, allowing both pushing values and pulling, fully controlling both processes, through a single function.
• Examples are given of working through a channel with a local variable, handling events, delegating work to another channel, and forming chains across different layers of abstraction.

let _title = ''
const title = ( text = _title )=> _title = text

title()                  // ''
title( 'Buy some milk' ) // 'Buy some milk'
title()                  // 'Buy some milk'

Property

• The use of channels as object methods is considered.
• The $mol_wire_solo decorator is introduced, memorializing their operation to save computation and ensure idempotency.

class Task extends Object {

    @ $mol_wire_solo
    title( title = '' ) {
        return title
    }

    details( details?: string ) {
        return this.title( details )
    }

}

Recomposition

• The composition of several simple channels into one composite channel is considered.
• And vice versa - working with a composite channel through several simple ones.

class Task extends Object {

    @ $mol_wire_solo
    title( title = '' ) { return title }

    @ $mol_wire_solo
    duration( dur = 0 ) { return dur }

    @ $mol_wire_solo
    data( data?: {
        readonly title?: string
        readonly dur?: number
    } ) {
        return {
            title: this.title( data?.title ),
            dur: this.duration( data?.dur ),
        } as const
    }

}

Multiplexing

• We consider channels multiplexed in a single method that takes a channel identifier as the first argument.
• A new decorator $mol_wire_plex for such channels is introduced.
• Demonstrates the approach of taking copypaste from multiple solo channels into one multiplexed channel in a base class without changing the API.
• Demonstrated by moving state storage of multiple objects to local storage via multiplexed singleton and obtaining automatic tab synchronization.

class Task_persist extends Task {

    @ $mol_wire_solo
    data( data?: {
        readonly title: string
        readonly dur: number
    } ) {
        return $mol_state_local.value( `task=${ this.id() }`, data )
            ?? { title: '', cost: 0, dur: 0 }
    }

}

// At first tab
const task = new Task_persist( 777 )
task.title( 'Buy some milk' ) // 'Buy some milk'

// At second tab
const task = new Task_persist( 777 )
task.title()                  // 'Buy some milk'

Keys

• A library is implemented that gives a unique string key for equivalent complex structures.
• The universal principle of support for user-defined data types is explained.
• Its application for identification of multiplexed channels is demonstrated.

@ $mol_wire_plex
task_search( params: {
    query?: string
    author?: Person[],
    assignee?: Person[],
    created?: { from?: Date, to?: Date }
    updated?: { from?: Date, to?: Date }
    order?: { field: string, asc: boolean }[]
} ) {
    return this.api().search( 'task', params )
}

Factory

• We introduce the notion of reactive factory method controlling the lifecycle of the created object.
• The lazy creation of a chain of objects followed by its automatic destruction is considered.
• The principle of capturing the ownership of an object and the predictability of the moment of its destruction is explained.
• The importance of lazy object creation for the speed of component testing is emphasized.

class Account extends Entity {

    @ $mol_wire_plex
    project( id: number ) {
        return new Project( id )
    }

}

class User extends Entity {

    @ $mol_wire_solo
    account() {
        return new Account
    }

}

Hacking

• The technique of tuning an object by redefining its channels is discussed.
• Demonstrates how to raise the stack using hacking.
• The advantages of hacking for linking objects that know nothing about each other are emphasized.

Binding

• Object bindings are classified by direction: one-way and two-way.
• As well as by method: delegation and hacking.
• The disadvantages of linking by the synchronization method are emphasized.

class Project extends Object {

    @ $mol_wire_plex
    task( id: number ) {
        const task = new Task( id )

        // Hacking one-way
        // duration <= task_duration*
        task.duration = ()=> this.task_duration( id )

        // Hacking two-way
        // cost <=> task_cost*
        task.cost = next => this.task_cost( id, next )

        return task
    }

    // Delegation one-way
    // status => task_status*
    task_status( id: number ) {
        return this.task( id ).status()
    }

    // Delegation two-way
    // title = task_title*
    task_title( id: number, next?: string ) {
        return this.task( id ).title( next )
    }

}

Debug

• The possibility of factories to form globally unique semantic object identifiers is disclosed.
• It demonstrates the display of identifiers in the debugger and stacktrays.
• Demonstrates the use of custom formatters to make objects even more informative in the debugger.
• Demonstrated is the logging of state changes with their identifiers displayed.

Fiber

• We introduce the notion of a fiber - suspendable function.
• We estimate memory consumption of naive implementation of fiber on hash tables.
• The most economical implementation on a regular array is proposed.
• The technique of bilateral links with overheads of only 16 bytes and constant algorithmic complexity of operations is disclosed.
• It is substantiated the limitation of memory sprawl occupied by an array during dynamic graph rearrangement.

Publisher

• Introduces the notion of publisher as a minimal observable object.
• The memory consumption of the publisher is evaluated.
• The application of publisher for reactivation of usual variable and page address is demonstrated.
• A micro library that provides a minimal publisher for embedding in other libraries is proposed for use.
• The creation of a reactive set from a native set is demonstrated.

const pub = new $mol_wire_pub

window.addEventListener( 'popstate', ()=> pub.emit() )
window.addEventListener( 'hashchange', ()=> pub.emit() )

const href = ( next?: string )=> {

    if( next === undefined ) {
        pub.promote()
    } else if( document.location.href !== next ) {
        document.location.href = next
        pub.emit()
    }

    return document.location.href
}

Dupes

• A structural comparison of arbitrary objects is discussed.
• Heuristics to support custom data types are introduced.
• The importance of caching is justified and how to avoid memory leaks is explained.
• Application of caching for correct comparison of cyclic references is disclosed.
• It is proposed to use independent micro-library.
• The results of performance comparison of different deep object comparison libraries are given.

Subscriber

• Introduces the concept of a subscriber as an observer capable of automatically subscribing to and unsubscribing from publishers.
• The memory consumption of subscriber and subscriber combined with publisher is evaluated.
• An algorithm for automatic subscription to publishers is disclosed.
• Manual low-level work with the subscriber is considered.

const susi = new $mol_wire_pub_sub
const pepe = new $mol_wire_pub
const lola = new $mol_wire_pub

const backup = susi.track_on() // Begin auto wire
try {
    touch() // Auto subscribe Susi to Pepe and sometimes to Lola
} finally {
    susi.track_cut() // Unsubscribe Susi from unpromoted pubs
    susi.track_off( backup ) // Stop auto wire
}

function touch() {

    // Dynamic subscriber
    if( Math.random() < .5 ) lola.promote()

    // Static subscriber
    pepe.promote()

}

Task

• Introduces the notion of a task as a one-time fiber, which is finalized on completion, freeing up resources.
• The main types of tasks are compared: from native generators and asynchronous functions, to NodeJS extensions and SuspenseAPI with function restarts.
• Introduces the $mol_wire_task decorator, which automatically wraps the method in the task.
• It is explained how to fight with non-dempotency when using tasks.
• A mechanism for ensuring reliability when restarting a function with dynamically changing execution flow is disclosed.

// Auto wrap method call to task
@ $mol_wire_method
main() {

    // Convert async api to sync
    const syncFetch = $mol_wire_sync( fetch )

    this.log( 'Request' ) // 3 calls, 1 log
    const response = syncFetch( 'https://example.org' ) // Sync but non-blocking

    // Synchronize response too
    const syncResponse = $mol_wire_sync( response )

    this.log( 'Parse' ) // 2 calls, 1 log
    const response = syncResponse.json() // Sync but non-blocking

    this.log( 'Done' ) // 1 call, 1 log
}

// Auto wrap method call to sub-task
@ $mol_wire_method
log( ... args: any[] ) {

    console.log( ... args )
    // No restarts because console api isn't idempotent

}

Atom

• The concept of an atom as a reusable fiber that automatically updates the cache when the dependencies change is introduced.
• The mechanism of interaction of different types of fibers with each other is disclosed.
• The example of the use of problems to combat the non-dempotence of references to atoms that change their state dynamically is given.

@ $mol_wire_method
toggle() {
    this.completed( !this.completed() ) // read then write
}

@ $mol_wire_solo
completed( next = false ) {
    $mol_wait_timeout( 1000 ) // 1s debounce
    return next
}

Abstraction Leakage

• The weak point of channel abstraction - the possible violation of invariants during nudging - is emphasized.
• Different strategies of behavior when the result of pushing contradicts the invariant are considered: auto-pretensioning, auto-post-pretensioning, manual tightening.
• Alternative more rigorous abstractions are considered.
• The choice of the simplest strategy that minimizes overhead and maximizes control by the application programmer is justified.

@ $mol_wire_solo
left( next = false ) {
    return next
}

@ $mol_wire_solo
right( next = false ) {
    return next
}

@ $mol_wire_solo
res( next?: boolean ) {
    return this.left( next ) && this.right()
}

Tonus

• We present 5 states in which a fiber can be: calculated, obsolete, doubtful, actual, finalized.
• The purpose of the cursor for representing the fiber lifecycle states is disclosed.
• Transitions of states of nodes in the reactive graph when values change and when they are accessed are illustrated.
• The permanent relevance of the value received from the atom is substantiated.

Order

• The mechanism of automatic update from the entry point, which guarantees the correct order of calculations, is disclosed.
• It substantiates the delayed recalculation of invariants exactly at the next animation frame, which saves resources without visible artifacts.

Depth

• The main scenarios for working with atoms, which may depend on the depth of dependencies, are considered.
• Two main approaches to realization of these scenarios are considered: cycle and recursion.
• The choice of the recursive approach is justified in spite of its limitation in the depth of dependencies.
• The example of stacktrace analysis is given and the importance of its informativeness is emphasized.
• Transparent behavior of reactive system for popping exceptions is explained.

Error

• The possible meanings of fiber are classified: promise, error, correct result.
• The possible ways of passing a new value to a fiber are classified: return, throw, put.
• The normalization of fiber behavior regardless of the way of passing a value to it is substantiated.

Extern

• The features of working with asynchronous and synchronous interfaces are discussed.
• The mechanism of SuspenseAPI, based on promises popping, is explained.
• The possibilities of tracking dependencies in synchronous functions, asynchronous functions and generators are discussed.
• The results of measuring the speed of different approaches are given.
• The problem of colored functions and the necessity of their discoloration are emphasized.
• The choice of the synchronous approach is justified.

something(): string {

    try {

        // returns allways string
        return do_something()

    } catch( cause: unknown ) {

        if( cause instanceof Error ) {
            // Usual error handling
        }

        if( cause instanceof Promise ) {
            // Suspense API
        }

        // Something wrong
    }

}

Recoloring

• Introduces proxies $mol_wire_sync and $mol_wire_async allowing to transform asynchronous code into synchronous and vice versa.
• An example of synchronous, but not blocking data loading from the server is given.

function getData( uri: string ): { lucky: number } {
    const request = $mol_wire_sync( fetch )
    const response = $mol_wire_sync( request( uri ) )
    return response.json().data
}

Concurrency

• The scenario where the same action is started before the previous one is finished is discussed.
• The $mol_wire_async feature is disclosed, which allows to control whether the previous task will be cancelled automatically.
• An example of using this feature to implement debounce is given.

button.onclick = $mol_wire_async( function() {
    $mol_wait_timeout( 1000 )
    // no last-second calls if we're here
    counter.sendIncrement()
} )

Abort

• The existing JS mechanisms for cancelling asynchronous tasks are discussed.
• Explains how to use the lifetime control mechanism for promises as well.
• An example of a simple HTTP loader, capable of canceling requests automatically, is given.

const fetchJSON = $mol_wire_sync( function fetch_abortable(
    input: RequestInfo,
    init: RequestInit = {}
) {

    const controller = new AbortController
    init.signal ||= controller.signal

    const promise = fetch( input, init )
        .then( response => response.json() )

    const destructor = ()=> controller.abort()
    return Object.assign( promise, { destructor } )

} )

Cycle

• A naive implementation of a temperature converter with cyclic dependence is disassembled.
• The correct temperature converter without cyclic dependence is implemented by moving the truth source to a separate atom.
• The technique of algorithmic complexity reduction through reactive memoization on the example of Fibonacci numbers calculation is disclosed.

Atomic

• The problems of transactional consistency with external states that do not support isolation are considered, using personal notes and local storage as examples.
• The importance of not only internal consistency, but also consistency with external states is emphasized.
• The problems of user deception, which only exacerbate the situation with which they are supposed to fight, are disclosed.
• The futility of rollback of changes already adopted and the inevitability of inconsistency of external states are substantiated.
• A decision is made not to mess with the application programmer's head, but to concentrate on giving him/her a better understanding of what is going on.
• It is proposed to write application logic that normalizes the inconsistency of input data.

Economy

• The results of speed and memory consumption measurements of $mol_wire in comparison with its nearest competitor MobX are given.
• The decisive factors allowing $mol_wire to show more than twofold advantage in all parameters in spite of the head start because of improved debug experience are disclosed.
• Given measurements showing the competitiveness of $mol_wire even in someone else's field, where the possibilities of partial recalculation of states are not involved.
• The importance of maximum optimization and economy of the reactive system is justified.

Reactive ReactJS

• The main architectural problems of ReactJS are given.
• Introduces architectural improvements from $mol such as controlled but stateful, update without recomposition, lazy pull, auto props and others.
• Most of the problems are solved by implementing a basic ReactJS component with $mol_wire bolted on.
• A component that automatically displays the status of asynchronous processes within itself is implemented.
• We implement a reactive GitHub API, which does not depend on ReactJS.
• We're implementing a button that indicates the status of an action.
• We implement a text input field and a number input field that uses it.
• We implement the application allowing to enter the number of the article and downloading its title from GitHub.
• Demonstrates a partial lifting of the component's stack.
• The work logs in different scenarios are given, showing the absence of unnecessary renders.

Reactive JSX

• ReactJS is not useful in a reactive environment.
• The mol_jsx_lib library, which renders JSX directly to the real DOM, is introduced.
• Discovered improvements in hydration, non-rendered component moves, DOM node access, attribute naming, etc.
• Demonstrated the possibility of cascading styling by automatically generated names of classes.
• Given measurements showing the reduction of the bandl in 5 times at a comparable speed of operation.

Reactive DOM

• The main architectural problems of the DOM are presented.
• Suggests a proposal for adding reactivity to the JS Runtime.
• The `mol_wire_dom' library is introduced, allowing you to try reactive DOM now.

Lazy DOM

• The need for lazy DOM construction to fill only the visible part of the page is justified.
• The complexity of virtualizing DOM rendering at both framework and application level is emphasized.
• Strategies to promote reactivity to standards are suggested.

Reactive Framework

• It reduces the size of the application code by several times by abandoning JSX in favor of all the features of $mol.
• It also expands the functionality of the application without any additional moves.

Results

In summary, by introducing a simple but flexible abstraction of channels, we have worked out many patterns of using them to achieve a variety of purposes. Once we've figured it out, we can build applications of any complexity, and have fun integrating with a wide variety of APIs.

Adding reactive memoization channels with automatic revalidation, resource release and asynchrony support has given us both a radical simplification of application code and increased efficiency in CPU and memory resource consumption.

And for those who, for whatever reason, are not yet ready to completely switch to $mol framework, we have prepared several independent microlibraries:

• $mol_key (1 KB) - unique key for structures
• $mol_compare_deep (1 KB) - fast deep object comparison
• $mol_wire_pub (1.5 KB) - minimum publisher to integrate into reactive runtime
• $mol_wire_lib (7 KB) - a complete set of tools for reactive programming
• $mol_wire_dom (7.5 KB) - the magic of turning a regular DOM into ReactiveDOM.
• $mol_jsx_view (8 KB) - a truly Reactive ReactJS.

Grab their hands and let's rock out together!

Growth

• Real cases are given, where $mol has shown itself well in speed of learning, development, launching, responsiveness, and even in reducing team size while maintaining competitiveness.
• The main advantages of the new-generation oupensor web platform we are developing on its basis are disclosed.
• The rosy prospects of import substitution of many web services at a new level of quality are highlighted.
• The projects we've already started, science-intensive articles we've written and hardcore reports we've recorded are discussed in detail.
• It is suggested that give us money to continue this banquet or start making your own appetizers.