DEV Community: Vitaliy Akimov

Productivity-boosting JavaScript debugger

Vitaliy Akimov — Wed, 02 Sep 2020 10:59:55 +0000

Effectful Debugger is a VSCode plugin for JavaScript/TypeScript debugging. Besides the typical debugger's features, it offers:

Time-traveling
Persistent state
Platform independence (one debugger for node, all browsers, embedded engines etc)
Programmable API
Hot mocking of functions or even parts of a function
Hot code swapping
Data breakpoints

If you debug a lot, you often encounter a situation when you need to go a few steps backward to see your program's state before. These include variable values, object properties value, DOM, and so on.

If your debugger supports time-traveling, you can stop execution at any time and step backward. The feature helps a lot, especially if this particular application state is hard to reach. For example, you may need to re-press the necessary combination of keys, reset the DB state to some specific values, etc.

All operations changing state are stored in a trace. The trace is used for restoring some past states. This looks like running an application in a reverse direction, including breakpoints.

The whole application state can be saved and loaded back. And the application can even continue execution from the point where it was saved.

EDBG.capture API function return JSON.stringify serializable object. The same object passed to EDBG.restore resets the application to the captured state.

We can even duplicate the state, opening another clone of the application, while the old is still running. The two applications are displayed as threads in VSCode.

All the variables, including the closure captured ones, are stored and restored too.

The time-traveling trace is stored/restored with EDBG.capture/EDBG.restore functions too. We can use this for post-mortem analysis, and we can even resurrect the program.

The code can be changed while the application runs.

Hot swapping will work better in some next version. Now, only small changes will be updated without a problem. A better solution is just re-run a chunk of code from some point. It is also possible to set up some more complex strategies using APIs.

Here I develop while the debugger runs. I get results immediately after any file is changed in case of any problem, I just set a breakpoint and run backward and fix a problem, getting results immediately again. In this demo, it was missed case in switch. It took a few seconds to go to the exact execution location, with correct variable values. This increases productivity a lot.

If you ever wondered what changed your program state (including variables of objects fields, arrays elements, etc.). If your debugger has data breakpoints and time-traveling, it is enough to add a breakpoint and execute it backward.

How it works

EffectfulJS is a JavaScript transpiler for adding any computational effect into the language without changing its syntax. We can use this for debugging applications only if they don't use EffectfulJS. It transpiles all the sources (including dependencies) and injects debugging instructions. They implement VSCode debugging protocol and also collect traces with current variable values, DOM and external function calls.

Since the debugger transpiling sources, this adds extra code. It is slower than executing without the debugger. However, using debuggers speed ups development a lot. There is no need to restart anything. Programs can be debugged even after it returned some results. With fast-restart, only a necessary part of the program will be re-executed and so on.

I already use this debugger in my React, NextJS and Node projects and it helps a lot, but it is in an early stage. Some JavaScript standard edge cases don't work correctly, and it could have better performance.

Please let me know if you are willing to contribute. These might be anything from its issues list with debugger tag or anything else you think is more useful.

Async Generators as an alternative to State Management

Vitaliy Akimov — Tue, 01 Sep 2020 07:31:35 +0000

Async Generators is a simple but powerful feature that is now a part of JavaScript. Applying Transducers approach from Functional Programming with Async Generators is a simple and powerful tool for improving software architecture, making it more agile, simplifying extension and composition.

TL;DR

With Async Generators there is no longer need for components state, state management tools, component lifecycle methods, and even the latest React Context and Suspense APIs. It is much simpler to develop, maintain, and test.
Unlike a state management approach, async generators tame asynchronicity leaving mutations harmless (if visible only in the generator’s scope).
This approach has a functional programming background too. State persistence for things like time traveling, universal apps is also available.
The article uses React and JavaScript, but the technique is applicable in any other framework or programming language with generators (coroutines).
I’m advertising my tool only at the end and very briefly. Most of the article is about async generators without any dependency.

Let’s start with a statement from Redux motivation page:

This complexity is difficult to handle as we’re mixing two concepts that are very hard for the human mind to reason about: mutation and asynchronicity. I call them Mentos and Coke. Both can be great in separation, but together they create a mess.

Redux and other state management tools are mostly focusing on restricting or controlling data mutations. Async generators can handle asynchronicity. This makes mutation safe if it is visible only within a particular generator scope.

All the common state management techniques can be split into two big classes.

The first class maintaining data dependencies graph to propagate changes through handlers — React Component State, MobX, RxJS. Maintaining these dependencies is a complex task. The underlying libraries are taking charge of part of this complexity by managing subscriptions, optimizing the order of handlers execution, batching them, but it is still confusing to use sometimes, often requires hard fine-tuning, e.g., with shouldComponentUpdatemethod.

Another approach limits mutation to only a single cell (storage) (e.g., Redux). This needs much smaller libraries, with less magic in them. It is more a pattern than a library. Unfortunately, the programs are more verbose, and this breaks data encapsulation. There are many patterns, wrappers to solve this though, but they make a single cell approach to be more similar to the graph based one.

The technique in this story and Redux are both based on Event Sourcing pattern, and they have many similarities. It also offers encapsulated data and synchronous deterministic order of executions for operations with side effects.

This approach can be abstractly viewed as a dependency graph as well, but the changes are propagated in reverse direction, from its root to towards leaves of its spanning tree. In each node we check should the propagation proceed to children or not. This makes the scheduling algorithm very lightweight and easy to control. It doesn’t require any library, basing only on JavaScript built-in features.

Let’s first port Redux VanillaJS counters example to illustrate the idea.

The original reducer is replaced with async generator function. The function calculates and stores its state in a local variable. It also yields the calculated value, the new value is stored in the singleton storage, and it is visible from event handlers. I’ll remove that singleton storage in the next steps.

This version doesn’t look much different from Redux. The async generator there could be Redux storage middleware. This violates one of Redux principles though, namely storing all application state only in the storage. Even if the generator doesn’t have any local variables, it still has its execution state — the position in the code where the execution is suspended in yield or await.

Turning components inside-out

Generator functions are functions returning iterators. We can do with them everything we can do with plain functions. For example, by composing generator functions, we can split computation into a few independent stages. Each stage has own encapsulated state. Each stage receives messages which were yielded on the previous stage, handles them yielding another messaging and passing them to the next stage.

The payload of the messages can contain VDOM elements. Instead of having a monolithic components tree we emit parts of it and send them to the next stage, where they can be assembled or transformed. Here is the same Counters example with React.

There pipe function is a function composition. The functions take two arguments. The first is async iterable for messages from the former stage. And the second is to send a message into the start of the pipe. It should be called only from event handlers.

The example above briefly shows extensibility by decoupling a few menu buttons from the root component into a separate stage. Instead of abstracting menu buttons into a separate component it maintains a placeholder where it injects components it receives in messages with “MENU_ITEM” type. It is an Inversion of Control for components. Both techniques React Components and these Inverted Components can be used together of course.

Extension

An exciting point of this technique is nothing should be preliminary designed to make the program reusable and decoupled. Nowadays premature abstraction is probably a bigger evil than premature optimization. It almost definitely leads to an overdesigned mess impossible to use. Using abstract generators, it is easy to keep calm and implement the required features, splitting when needed, without thinking about future extensions, easy to refactor or abstract some common parts after more details are available.

Redux is famous for making programs simpler to extend and reuse. The approach in this story is also based on Event Sourcing, but it is much simpler to run async operations and it doesn’t have a single store bottleneck, nothing should be designed prematurely.

Many developers like single storage because it is easy to control. The control is not a free thing though. One of the widely accepted advantages of Event Sourcing pattern is an absence of a central DB. It is simpler to change one part without danger of breaking something else. There is another problem of single storage discussed in Persistence section below.

There is Decouple Business Logic article with more detailed cases study. At some step there, I added a multi-select feature to drag and drop without changing anything in single element handling. With a single store, this would mean changing its model from storing a single currently dragging element to a list.

There are similar solutions in Redux, namely applying higher order reducer. It could take a reducer working with a single element and translate into reducer working for a list. The generators solution uses higher order async generators instead, taking a function for a single element and generating the one for a list. It is similar but much less verbose, as the generator encapsulates data and implicit control state.

As an illustration, let’s make a list of counters. This step is covered in “Decouple Business Logic” article, I’m not giving many details here. The forkfunction is the async iterators transforming function, running its argument in threads per item. It is a not simple, but it is a generic one, works in many contexts as is. In the next section, for example, I apply it recursively to get a tree view.

Performance

Async generators overhead is much smaller than for state management libraries. But there are a lot of ways to get performance problems here too, e.g. over flooding with messages. But there are also a lot of almost effortless ways to improve performance.

In the former example, there are useless calls to ReactDom.render. This is obviously a performance problem, and there is a simple solution. Solving it quickly by sending another message with type “FLUSH” after each dispatched event. React render runs only after it receives this message. The intermediate steps can yield whatever they need in between.

Another awesome side of this approach is you may not worry about performance until it is a problem. Everything is structured in small autonomous stages. They are easy to refactor, or even without refactoring — many performance problems can be solved by adding another generic state in the pipe of steps, e.g., batching, prioritizing, saving intermediate data, etc.

For example, in the demo constructed React elements are saved in local variables and React can re-use them. Changes are propagated from the root towards leaves, so optimizations like overridingshouldComponentUpdate aren’t needed.

Testing

Comparing to Redux reducer tests, generators fit a bit darker box testing strategy. The tests don’t have access to the current state. Though still, they are very simple to write. With Jest snapshots, the test can be a list of input messages with comparing output using snapshots.

test("counterControl", async () => {
  expect.assertions(3)
  for await(const i of Counter.mainControl([
         {type:"MENU", value:<span>Menu</span>},
         {type:"VALUE", value:10},
         {type:"CONTROL", value:<span>Control</span>},
         {type:"FLUSH"},
         {type:"VALUE", value: 11},
         {type:"FLUSH"}]))
    if (i.type === "CONTROL")
      expect(renderer.create(i.value).toJSON()).toMatchSnapshot()
})

If you prefer unit-tests as documentation policy, there are many ways to make a self-documenting API for testing. Say, a function eventually/until as an addition to traditional BDD expressions.

Persistent state

There is another motivation for Redux described in You Might Not Need Redux article by Dan Abramov — namely providing access to the state and it can be serialized, cloned, diffed, patched, etc. This can be used for time travel, hot reloading, universal applications and more.

For this to work, the whole application state should be kept in Redux storage. Many Redux applications(even Redux samples) have some part of state stored outside of their store. These are components state, closures, generators or async functions state. Redux based tools can not persist this state.

Having a single source of truth as a single storage Redux, of course, makes programs simpler. Unfortunately, it is often impossible. Consider for example distributed application, e.g., data are shared between frontend and backend.

Lindsey Kuper

@lindsey

"Oh, you wanted to *increment a counter*?! Good luck with that!" -- the distributed systems literature

18:55 PM - 09 Mar 2015

376 583

Event Sourcing is very successful for distributed applications. With generators, we can write a proxy sending all incoming messages to the remote side and yielding all received messages. There can be separate pipelines on each peer, or it can be the same application but a few running processes. Many configurations are easy to set up, use and re-use.

For example pipe(task1, remoteTask2, task3). Here remoteTask2 may be either proxy or, it may be defined here, say, for debugging purposes.

Each part maintains its own state, it doesn’t need to be persistent. Say if each task is implemented by a separate team they are free to use any model for the state, change it any time without worrying about the other team’s work is broken.

This fits well for Server Side Rendering too. Say, there can be a particular higher order function to cache resulting values depending on inputs on the back-end.

const backend = pipe(
    commonTask1,
    memo(pipe(
         renderTask1
         renderTask2)),
    commonTask2)

Here the memo higher order function examines incoming messages and may find out some calculation may be reused. This may be a server-side rendered string, and some next stage builds HTTP response with it.

The render tasks can run async operations, requesting something remote. For better user experience we want pages to load fast. To increase initial page load time applications can load components lazily displaying some loading placeholder instead of the component until it is ready. Having a few such components on a page with a bit different loading time causes page re-layouts worsening user experience.

React team recently announced Suspense API to solve this problem. It is an extension of React embedded into its renderer. With the Inverted Components like in this article, Suspense API isn’t needed, the solution is much simpler and not a part of UI framework.

Say the application uses dynamic imports to load lazy controls, this can be done with:

yield {type:"LAZY_CONTROL"}
yield {type:"CONTROL", value: await import("./lazy_component")}

There is another generic next stage. It collects all "LAZY_CONTROL" messages, and awaits either all "CONTROL" messages are received after or a threshold time interval. After, it emits "CONTROL" messages either with the loaded control or with loading indicator placeholder. All the next updates can be batched as well using some specific timeout to minimize re-layouts.

Some generator can also reorder messages to give animation a bigger priority than server data updates. I’m not even sure there are needs for a server-side framework. A tiny generator could transform initial HTTP request into messages or threads depending on URL, auth session, etc.

Functional Programming

Commonly used state management tools have FP background. The code from the article doesn’t look like FP in JavaScript because of imperative for-of/switch/break statements. It has a corresponding concept in FP too. It is so called Monads do-notation. For example one of their use in Haskell is to resolve problems like React components property drilling.

To keep this story practical I don’t digress from the main subject here, there is another article — Using Generators as syntax sugar for side effects.

Effectful.js

Effectful.js is a babel preset implementing do-notation working for any monad without any JavaScript syntax extension. It also supports state persistence with a reference implementation in es-persist library. For example, this may be used to convert all async generators example above into pure functions.

State persistence is not the primary goal of the tool. It is for higher level business logic description. Nevertheless, the tool is abstract and has many purposes. I’ll write about them more soon.

Here is the summary sample on GitHub with all the feature above plus automatic Undo/Redo and storing its full state in localStorage. And here is running transpiled version (it writes to your browsers local storage but no information is sent to the server side). I’m not giving many details in this article, it is about async generators without dependency, but I suppose the code is straightforward to read. Check for example undoredo.js for easy time traveling implementation details.

The original sample requires almost no changes, I only replaced not serializable Promises, with corresponding functions from “es-persist” and replaced closures with invocations of R.bindfunction from the same library. EffectfulJS toolchain has another transpiler to make all the functions, including closures serializable, but not used in this example to keep it simpler.

The story is just a brief description of the technique. I’m using it for a couple of years already, and happy because of improvements it provides. Try it, and I’m sure you’ll enjoy it too. There are many things to describe in depth. Stay tuned!

React Suspense is to a Monad as Hooks are to Applicative Notation

Vitaliy Akimov — Tue, 01 Sep 2020 07:29:14 +0000

Monads and Applicative Functors are extensively used in functional programming. There is a relationship between them and React Suspense for Data Fetching and React Hooks APIs. This is a quick and simple introduction to Monads and Applicatives along with a description of their similarities.

The post is about experimental React Suspense for Data Fetching, not about recently released React Suspense for Code Splitting (React.Suspense and React.lazy).

No prior knowledge of Monads, Applicatives and the experimental Suspense is required to read this article. It explains the essence of them.

Monad do-notation

The React framework approach encourages developers to use functional programming techniques. At least component render functions should not have observable side effects. JavaScript has no way to ensure this, but there are programming languages which can. For example, Haskell doesn’t accept side effects at all.

Pure functions make the code modular, predictable and easier to verify. But they also significantly increase verbosity. Here is a statement from Phil Walder’s Monads for functional programming(1995) tutorial:

It is with regard to modularity that explicit data flow becomes both a blessing and a curse. On the one hand, it is the ultimate in modularity. All data in and all data out are rendered manifest and accessible, providing a maximum of flexibility. On the other hand, it is the nadir of modularity. The essence of an algorithm can become buried under the plumbing required to carry data from its point of creation to its point of use.

Monads solve this problem for Haskell. And Suspense/Hooks solve the same problem in React.

So what is a Monad? It is a simple abstract interface which has two functions, let’s call them of and chain.

of — takes any value and returns some monadic (effectful) value
chain — takes an effectful value and a function from any value to an effectful one and returns another effectful value

The effectful values there may encapsulate any concrete implementation-specific information. There are no requirements what exactly it should be, it is some opaque data. The interface’s concrete implementations should follow a set of laws, and this is it.

There is nothing to say more about monads since they are abstract. They don’t necessarily store anything, wrap or unwrap anything or even chain anything.

But why do we need this if it is so abstract and defines almost nothing? The interface provides an abstract mean to compose computations with side effects.

If you write code in JavaScript, you may wonder now. You’ve already composed a lot of computations with side effects without seeing any Monad. But in fact, you can consider you’ve already used them there.

In computer science, Monads first appeared for studying side effects in imperative languages. They are a tool to embed imperative worlds into a pure math world for further studying.

This way if you want to convert your imperative program into math formulas representing it, doing this with Monad expressions would be the simplest and the most straightforward way. It is so straightforward what you don’t even need to do it manually, there are tools that do it for you.

Haskell has a syntax sugar called do-notation exactly for this. This makes writing imperative programs in Haskell possible. There is a special tool in its compiler. It converts such imperative programs into a Monadic pure Haskell expressions. The expressions are close to math you see in textbooks.

JavaScript is an imperative language. We can consider any imperative code to be a do-notation already. But unlike the one in Haskell’s, it is not abstract. It works only for built-in side effects. There is no way to add support of any new one except extending the language.

There are such extensions, namely generators, async and async generator functions. JavaScript JIT compiler converts async and generator functions into concrete built-in API calls. Haskell doesn’t need such extensions. Its compiler converts do-notation into abstract Monads interface function calls.

For this post, we need only two JavaScript built-in effects. Let’s call them Mutation and Exception. They have clear meanings. Mutations allow changing values of some references. JavaScript has the Exceptions effect embedded using throw/ try-catch statements.

We can convert some effects into others. This way we can write async code using Generators.

This conversion trick can be applied to other effects too. And apparently, just Mutation and Exception are enough to get any other effect. This means we can turn any plain function into an abstract do-notation already. And this is exactly what Suspense does.

When the code encounters some effectful operation and requires suspension it throws an exception. It contains some details (for example a Promise object). One of its callers catches the exception, waits while the promise in the argument is settled, stores the resulting value in a cache, and re-runs the effectful function from the beginning.

After the Promise is resolved the engine calls the function again. The execution goes from its start, and when it encounters the same operations it returns its value from the cache. It doesn’t throw an exception and continues execution until the next suspension request or the function’s exit. If the function doesn’t have any other side effects its execution should go the same paths and all pure expressions are recalculated producing the same values.

Let’s re-implement Suspense. Unlike React, this one works with the abstract Monads interface. For simplicity, my implementation also hides a resource cache. Instead, the runner function counts invoked effects and uses the current counter value as a key for the internal cache. Here is the runner for the abstract interface:

/** effectful expression throws this object if it requires suspension */
const token = {};

/** Pointer to mutable data used to record effectful computations */
let context;

/** Runs `thunk()` as an effectful expression with `of` and `chain` as Monad's definition */
const run = (of, chain) => thunk => {
  /** here it caches effects requests */
  const trace = [];
  const ctx = {trace};
  return step();
  function step() {
    const savedContext = context;
    ctx.pos = 0;
    try {
      context = ctx;
      return of(thunk());
    } catch(e) {
      /** re-throwing other exceptions */
      if (e !== token)
        throw e;
      const {pos} = ctx;
      return chain(ctx.effect,
                   (value) => {
                     trace.length = pos;
                     /* recording the resolved value */
                     trace[pos] = value;
                     ctx.pos = pos + 1;
                     /** replay */
                     return step(value);
                   })
    } finally {
      context = savedContext;
    }
  }
}

/** marks effectful expression */
const M = eff => {
  /* if the execution is in a replay stage the value will be cached */
  if (context.pos < context.trace.length)
    return context.trace[context.pos++];
  /* saving the expression to resolve in `run` */
  context.effect = eff;
  throw token;
}

Now let’s add a concrete Async effects implementation. Promises, unfortunately, aren’t exactly monads since one Monad law doesn’t hold for them, and it is a source of subtle problems, but they are still fine for our do-notation to work.

Here is concrete Async effect implementation:

const runPromise = run(
  v => Promise.resolve(v), 
  (arg, f) => arg.then(f));

And here’s a simple example, it waits for delayed values before rendering proceeds:

The sandbox also contains Component wrapper. It turns an effectful functional component into a React component. It simply adds chain callback and updates the state accordingly. This version doesn’t have a fallback on threshold feature yet, but the last example here does have it.

The runner is abstract, so we can apply it for something else. Let’s try this for the useState hook. It is a Continuation monad, not a State monad as its name may suggest.

Effectful value here is a function which takes a callback as an argument. This callback is called when the runner has some value to pass further. For example when the callback returned from useState is called.

Here, for simplicity, I use single callback continuations. Promises have one more continuation for failure propagation.

const runCont = run(
  value => cont => cont(value),
  (arg, next) => cont => arg(value => next(value)(cont)));

const useState = initial =>
  M(cont => 
    cont([initial, function next(value) { cont([value,next]); }]));

And here is a working usage example, with most of “kit.js” copy-pasted, except the monad’s definition.

Unfortunately, this is not exactly the useState hook from React yet, and the next section shows why.

Applicative do-notation

There is another extension for do-notation in Haskell. It targets not only Monad abstract interface calls but also calls of Applicative Functors abstract interface.

Applicative interfaces shares the of function with Monads and there is another function, let’s call it join. It takes an array of effectful values and returns a single effectful value resolving to an array. The resulting array contains all the values to which each element of the argument array was resolved.

I use a different one from Haskell’s interface. Both are equal though — it is simple to convert Haskell’s interface into the one used here and back. I do this because this basis is much simpler to use in JavaScript, it doesn’t need any higher-order functions, and there is already its instance in the standard runtime.

In Haskell and in JavaScript any Monad is immediately an Applicative Functor. This means we don’t need to write a concrete implementation of Applicative interface, we can generate it automatically.

If there is a default implementation, why do we need Applicative Functors? There are two reasons. The first one is not all Applicative Functors are Monads, so there is no chain method from which we can generate join. Another reason is, even if there is chain, custom join implementation can do the same thing in a different way, probably more efficiently. For example, fetching resources in parallel rather than sequentially.

There is an instance of this interface for Promises in the standard runtime. It is Promise.all(ignoring some details here for simplicity again).

Let’s now return to the state example. What if we add another counter in the component?

The second counter now resets its value when the first one is incremented. It is not how Hooks are supposed to work. Both counters should keep their values and work in parallel.

This happens because each continuation invocation erases everything after it in the code. When the first counter changes its value the whole next continuation is re-started from the beginning. And there, the second counter value is 0 again.

In the run function implementation, the invalidation happens at line 26 — trace.length = pos — this removes all the memorized values after the current one (at pos). Instead, we could try to diff/patch the trace instead. It would be an instance of Adaptive Monad used for incremental computations. MobX and similar libraries are very similar to this.

If we invoke effectful operations only from a function’s top level, there are no branches or loops. Everything will be merged well overwriting the values on the corresponding positions, and this is exactly what Hooks do. Try to remove the line in the code sandbox for two counters above.

Transpiler alternative

Using Hooks already makes programs more succinct, reusable and readable. Imagine what you could do if there were no limitations (Rules of Hooks). The limitations are due to runtime-only embedding. We can remove these limitations by means of a transpiler.

Effectful.JS is a transpiler for embedding effectful into JavaScipt. It supports both Monadic and Applicative targets. It greatly simplifies programs in the designing, implementing, testing, and maintaining stages.

Unlike React Hooks and Suspense, the transpiler doesn’t need to follow any rules. It works for any JavaScript statement (branches, loops, exceptions etc). It never re-plays functions from the beginning. This is faster. Plus the functions can use any JavaScript built-in side effect.

Effectful.JS is not exactly a transpiler but rather a tool to create transpilers. There are also a few predefined ones and a lot of options for tuning. It supports double-level syntax, with special markers for effectful values (like await expressions in async functions, or Haskell’s do). And it also supports a single level syntax where this information is implicit (like Suspense, Hooks or languages with Algebraic Effects).

I’ve quickly built a Hooks-like transpiler for demo-purposes — @effectful/react-do. Calling a function with names starting with “use” is considered effectful. Functions are transpiled only if their name starts with “use” or they have “component” or “effectful” block directive (a string at the beginning of the function).

There are also “par” and “seq” block-level directives to switch between applicative and monadic targets. With “par” mode enabled the compiler analyzes variable dependencies and injects join instead of chain if possible.

Here is the example with two counters, but now adapted with the transpiler:

For demo purposes, it also implements Suspense for Code Splitting. The whole function is six lines long. Check it out in the runtime implementation @effectful/react-do/main.js. In the next example, I’ve added another counter which rendering is artificially delayed for demo purposes.

Algebraic Effects

Algebraic Effects are often mentioned along with Suspense and Hooks. These may be internals details or a modeling tool, but React doesn’t ship Algebraic Effects to its userland anyway.

With access to Algebraic Effects, users could override operations behavior by using own Effect Handler. This works like exceptions with an ability to resume a computation after throw. Say, some library function throws an exception if some file doesn’t exist. Any caller function can override how it can handle it, either ignore or exit process, etc.

EffectfulJS doesn’t have built-in Algebraic Effects. But their implementation is a tiny runtime library on top of continuations or free monads.

Invoking a continuation also erases everything after the corresponding throw. There is also special syntax and typing rules to get Applicative (and Arrows) API — Algebraic Effects and Effect Handlers for Idioms and Arrows. Unline Applicative-do this prohibits using any anything which requires Monad operations.

Generators

For monadic notation, we can use Generators, but with some limitations. There are more details in Using Generators as syntax sugar for side effects article.

Wrapping up

The transpiler is a burden, and it has its own usage cost. Like for any other tool, use it only if this cost is smaller than the value you get.

And you can achieve a lot with EffectfulJS. It is a new way to write JavaScript programs. It is useful for projects with complex business logic. Any complex workflow can be a simple maintainable script.

As an example, Effectful.JS can replace Suspense, Hooks, Context, and Components State with tiny functions. Error Boundaries are the usual try-catch statements. Async rendering is an async scheduler. But we can use it for any computations, not only for rendering.

There are a lot of other awesome application-specific uses, and I’m going to write more about them soon. Stay tuned!

Using Generators as syntax sugar for side effects

Vitaliy Akimov — Mon, 31 Aug 2020 11:17:45 +0000

This story shows Generators as an explicit but seamless syntax for programs with asynchronous operations, shared mutable state and other side effects. The conversion is based on so-called Monads.

In spite of the scary name, Monads are a very simple concept. You already use them when you change some variable value or output anything or throw/catch an exception. Monads first appeared in Computer Science to support mathematical reasoning about side-effects in imperative languages such as JavaScript.

Other researchers were designing practical languages with only pure functions. Using only pure functions makes programs more verbose and harder to read. Monads were applied as a practical tool for converting programs with effects into pure ones. Here is a citation from one of the best monads tutorial — Monads for functional programming by Philip Wadler (1995):

It is with regard to modularity that explicit data flow becomes both a blessing and a curse. On the one hand, it is the ultimate in modularity. All data in and all data out are rendered manifest and accessible, providing a maximum of flexibility. On the other hand, it is the nadir of modularity. The essence of an algorithm can become buried under the plumbing required to carry data from its point of creation to its point of use.

Doesn’t it sound familiar? E.g., property drilling for React Components and one of the reasons for state management to solve it.

Original abstract do-notation is a macro converting programs looking like imperative one into abstract API calls. The concrete implementations of that APIs can construct an Iterable object, or a Promise or many other things. This way the same syntax (and the same programs) can be reused in different contexts.

JavaScript has syntax extensions similar to do-notation. They are async and/or generator functions. Unlike original do-notation they are converted to concrete API calls ( Promise#then, and Iterable constructors) by JavaScript compiler or some transpiler (e.g. regenerator). These concrete functions are almost instances of abstract monad API.

Async and generator functions are based on Coroutine monad. It can be converted into many other monads, but not all. There is a well-known fact in JavaScript community — generator functions can replace async functions. Asynchronous functions written using generators syntax can be canceled, unlike standard async functions. The only overhead is the need to call a wrapper function, which converts Iterable into a Promise.

There are many examples of using generators for async code so I’ll illustrate the idea on another case instead. It is maybe not so practical, but we can turn a generator function with a mutable state into a pure function.

function* incr() {
  return (yield set((yield get) + 1))
}

function* incrX2() {
  return (yield* incr()) + (yield* incr())
}

const main = state(incrX2)

// framework
function state(iter) {
  const i = iter()[Symbol.iterator]()
  return walk()

  function walk(arg) {
    const step = i.next(arg)
    return step.done ?
      state => [state, step.value] :
      state => {
        const [next, value] = step.value(state)
        return walk(value)(next)
      }
  }
}

function set(s) { return _ => [s, s] }
function get(s) { return [s, s] }

Here both functions incr and incrX2 have side effects. They change and read shared data. But the resulting function state(incrX2) is pure. The function state does actual conversion between Iterable into a State monad.
This is how it looks with inlined framework functions:

function incr(s) {
  const ns = s + 1
  return [ns, ns]
}

function incrX2(s) {
  const [s1, r1] = incr(s)
  const [s2, r2] = incr(s1)
  return [s2, r1 + r2]
}

The example skips abstract API layer. There are quite a few options to chose its basis, but the most simple one is two functions: of and chain. They both return monadic(effectful) value. It is an abstract thing and can be anything depending on the concrete API implementation. For the abstract interface, the values of this type are entirely opaque.

of — takes any value and returns effectful value, what it does exactly with the argument is defined by interface implementation
chain — takes some effectful value and a function mapping anything to another effectful value, returns some other effectful value

The concrete implementations of the function can do anything if this conforms to so-called monadic laws. In fact, my choice of chainname is typical for JavaScript libraries but a bit misleading. It suggests some concrete implementation, chaining of something. But, again, it is some abstract thing, and the only requirement is monad laws conformance.

Here are the laws:

(f, x) => chain(of(x), f) should be equal to (f, x) => f(x)
m => chain(m, x => of(x)) should be equal to m
(m, f, g) => chain(chain(m, f), g) should be equal to (m, f, g) => chain(m, x => chain(f(x), g))

If the laws hold we can use the API in do-notation, or in some abstract functions working for any monad.

For example, the first law means the value of x should be stored somewhere by of until processing by chain. This is why monads are often explained as something wrapping some value (burrito). However, in the general case, monad values aren't required to wrap anything (if they are constructed by something but not of).

Let’s convert Iterable into this abstract interface. It is almost the same like for State except the functions are replaced with abstracted calls.

const run = (of, chain) => fun => {
  const i = fun()[Symbol.iterator]()
  return walk()
  function walk(arg) {
    const step = i.next(arg)
    return step.done ? of(step.value) : chain(step.value, walk)
  }
}

For State, effectful value is a function taking some original state and returning a pair of a resulting value and a new state value. Here is State monad definition using the abstract intermediate layer:

const state = run(v => s => [v, s],
                  (arg, fun) => s => {
                     const [nextArg, nextState] = arg(s)
                     return fun(nextArg)(nextState)
                   })
function set(s) { return _ => [s, s] }
function get(s) { return [s, s] }

And Promises:

const promise = run(v => Promise.resolve(v),
                    (arg, fun) => arg.then(fun))

The first monad law doesn’t work for Promises if x is another object with then method (Thenable). This then method will be called, but the law requires it to be returned as is.

It is okay for practical purposes. However, sometimes this leads to unwanted consequences. For example, dynamic importing a module that exports any then function will call it and do something unpredictable.

Considering generators are enough for Promises and Iterators one may wonder why we need Async Generators. Indeed it is pretty easy to convert monad combining the two effects from plain generators. However, there won’t be any replacement for await-of statement. It is yet another syntax sugar for traversal.

Coroutines/Iterables cannot define be converted to any monad. For example, Observable is a monad, but generators cannot be used as a do-notation for them.

Another useful example is non-determinism. I.e. a computation returning several values. This can be used for embedding logical programming into JavaScript. It is simple to make an implementation for the abstract interface:

const nonDet = run(
  function*(value) { yield value },
  function*(arg, fun) { 
    for(const i of arg)
       yield* fun(i) 
  })

There are libraries defining abstract APIs with a few concrete implementations in JavaScript, e.g. fantasy-land. There are also a few do notation as generators implementations trying to simulate multiple resumes by restarting and replaying iterators for the beginning example - burrido. The approach is not safe and not efficient.

There is an alternative single-layered way. React uses it for Suspense for data fetching and Hooks. I described this in more details in React Suspense is to a Monad as Hooks are to Applicative Notation.

I now work on a babel plugin based implementation for do-syntax working for any Monad — EffectfulJS. It offers many options, like persistent state, concrete implementation inlining, implicit parallelization. This may significantly simplify JavaScript code. I’m going to write more about it soon, stay tuned!

Why abstract interface for effects matters

Vitaliy Akimov — Mon, 31 Aug 2020 10:26:13 +0000

JavaScript recently received three new computational effects kinds. There are:

generator functions
async functions
async generator functions

The latest is a combination of the first two. If at some point JavaScript gets some other effect extension in syntax it will have to have seven language extensions for each combination, and so on. This may be, for example, recently added in runtime Observable effects for reactive programming. It may get own syntax extension sooner or later.

Async generators extension, isn’t without problems. There is no way now to cancel async computation in JS now. This means it is impossible to combine async iterators as easily as Observables without introducing leaks. There is a proposal to add the cancellation, but it is still on too early stage and will probably take years. Alternatively, you may use Effectful.js immediately without waiting.

Libraries should be ready for abstract interface too. For example, higher order functions, like Array traversal functions (Array#map, Array#forEach, etc.). Say, we have a function duplicating elements implemented as a simple generator function:

function* dup(arr) {
  for(const i of arr) {
     yield i;
     yield i;
  }
}

We cannot just copy-paste the body of the loop into, say, forEach body like this:

arr.forEach(function* i() {
     yield i;
     yield i;
})

To make a fully general library we need four versions of Array#forEach, and each other higher order function: Array#map, Array#filter, etc. Say, filter predicate may need to consult some remote service and so it should be async.
It is even worse if the task like AST transforms with a significant number of nodes and Visitors written for traversal. To make a generic library there should be four versions of visitor types again.

There is an abstract interfaces for effects, for example Monads. Only one implementation for the functions above is needed if it uses such abstract interface to build result.

The Monad interface came from math, namely category theory.
Its only purpose was an abstraction. Theorems proved using abstract monads interface may be immediately applied to likely also abstract but more concrete interface instances, like some structures from universal algebra or topology. Mathematicians refactored math frameworks by abstracting common things into Category Theory and Monad as a part of it.

Later the interface was utilized to describe and reason about programming languages with effects in domains theory. The same purpose again, simplifying reasoning about programming languages and programs.

After, Monads reached practical programming languages. Other researchers worked on programming languages with only pure functions to simplify reasoning about programs. Unfortunately having only pure function requires threading functions parameters and results, result statuses, etc. This makes programs very verbose and hard to read. The problem was solved by applying Monads. Imperative code with a mutable state, exceptions can be converted into a pure function.

Applying Curry–Howard correspondence, where programming language types are theorems, and programs are their proofs, Monads are abstract API. So like in math, proved theorems with some general math structure may be applied to any concrete realization of that structure. In programming languages, a function using abstract type for arguments objects may be invoked with any concrete implementation of that object.

There are many different options in choosing names for interface functions, or a set of basis functions, or split it into a hierarchy of other abstract notions somehow. There are a few libraries in JavaScript already defining the interface and providing a few of its implementations. Like fantasy-land.
Effectful.js compiler uses own interface, but it may be adapted to any other either by implementing wrappers or inlining calls to other methods. It doesn’t introduce any syntax extensions, but can overload JavaScript ones, e.g., generators or await/async syntax to generalize them.

Many custom effects or changes/fixes for ECMAScript can be applied immediately without awaiting committee for years. For example:

cancelation to async functions, thus making async iterators compasable
improve performance (drop some not-efficient and useless standard feature, remove JavaScript engine scheduler from async iterator)
persistent state (time traveling, distributed apps, workflows)
non-deterministic programming (bind form’s data as logical formulas, reactive programming)

Decouple Business Logic using Async Generators

Vitaliy Akimov — Mon, 31 Aug 2020 10:10:46 +0000

Async generators are new in JavaScript, and I believe it is a very remarkable extension. It provides a simple but powerful tool for splitting programs into smaller parts, making sources easier to write, read, maintain and test.

The article shows this using an example. It implements a typical front-end component, namely drag and drop operations. The same technique is not limited to front-ends. It is hard to find where it cannot be applied. I use the same in two big compiler projects, and I’m very excited how much it simplifies there.

You can drag boxes from a palette in the top and drop into any of gray areas. Each drop area has its specific actions. A few items can be selected. Yellow ones have inertial movement.
All the features are independent there. They are split into stages. Some stages compute information shared by a few features. This does introduce some dependency, but can be surely avoided or controlled. All the features are simple to enable, disable, develop, test and debug separately. A few developers or teams could work on it in parallel very efficiently.

I assume some basic knowledge of async generators (or at least of async functions and generators separately) and some fundamentals of HTML DOM (at least knowing what it is). There are no dependencies on third-party JavaScript libraries
For the demo, let’s pretend we don’t know full requirements set and add new a feature only after we finish something and it works. Playing with already working software on intermediate stages typically boosts creativity. It is one of the main components of the agile software development core. I’d better write something not perfectly designed but working first. We can improve it using refactoring any time. Async generators will help.

Typically, at the beginning of any project, I don’t want to spend time on choosing the right framework, library or even an architecture. I don’t want to overdesign. With the help of async iterators, I can delay the hard decisions to a point where I have enough knowledge to make a choice. The earlier I take some option, the more chances there are for mistakes. Maybe I won’t need anything at all.

I’ll describe a couple of steps only here. The other steps are small and can be read directly from code effortlessly. They are just a matter of working with DOM, not a subject of the article. Unlike the transpiled final demo above, the demos in fiddles below work only in a browser supporting async generators. These are, for example, Chrome 63, Firefox 57. First examples also use pointer events polyfill, replaced in the last example.

Async generators

All the samples share nano-framework sources. It is developed once, at the beginning and copy-pasted without any change. In the real project, these are separate modules, imported into other modules if needed. The framework does one thing. It converts DOM events into async iterator elements.
Async iterator has the same next method like ECMAScript plain iterator, but it returns a Promise resolving to Objects with value, done fields.

Async generator functions is an extended function returning an async iterator. Like original non-async generator is a function returning a non-async iterator.

Async generators combine async functions and generator functionality. In the bodies of such functions, we can use await together with yield expressions, and they do exactly what these expressions do in async functions and generators respectively. Namely suspends execution control until Promise in await argument is resolved and for yield outputs value and suspends until caller requests next value.

Here’s preliminary framework implementation, with the first version of business logic:

It is a working sample, press Result there to see it in action. There are four elements you can drag within the page. The main components are send, produce and consume transducers. The application subscribes to DOM events and redirects them into the framework using send function. The function converts the arguments into elements of async iterator returned by produce call. The iterator never ends and called at a module’s top level.

There is for(;;) loop in produce. I know it looks suspicious, you may even have it denied in your team code review checklist or event by some lint rule, since for code readability we want exit condition for loops to be obvious. This loop should never exit, it is supposed to be infinite. But it doesn’t consume CPU cycles since most of the time it will sleep in awaitand yield expressions there.

There is also consume function. It reads any async iterator in its argument, doing nothing with the elements, never returning. We need it to keep our framework running.

async function consume(input) {
  for await(const i of input) {}
}

It is an async function (not generator), but it uses new for-await-of statement, an extension of for-of statement. It reads async iterators, rather than original ECMAScript iterator, and awaits each element. Its simplified implementation could transpile the original consume code into something like this:

async function consume(input) {  
    const iter = input[Symbol.asyncIterator]()
    for(let i;(i = await iter.next()).done;) {}
}

The main function is an entry point of the application’s business logic. The function is called between produce and consume in the module’s top level.

consume(main(produce()))

There is also a small share function. We need it to use the same iterator in a few for-await-of statements.

The first monolithic version of business logic is fully defined in main. With the example, you can already see the power of async generators. The application state (where we started dragging — x, y variables) are local variables, encapsulated inside the function. Besides data state, there is also execution control state. It is a kind of implicit local variable storing position where the generator is suspended (either on await or yield).

The same function could be rewritten without generators, for example into something like this:

function main(state) {
  for(;;) {
    switch(state.control) {
    case "init":
      state.action = "read"
      state.control = "loop1"
      return
    case "loop1":
      const i = state.value 
      if (i.type === "pointerdown") {
        const element = state.element = i.target.closest(".draggable")
        if (element) {
          const box = element.getBoundingClientRect()
          state.x = box.x + window.pageXOffset - i.x
          state.y = box.y + + window.pageYOffset - i.y
          state.control = "loop2"
          state.action = "read"
          return
        }
      }
      state.control = "loop1"
      state.action = "yield"
      state.value = i
      return
    case "loop2":
      const j = state.value
      if (j.type === "pointerup") {
        state.control = "loop1"
        break
      }
      if (j.type === "pointermove") {
        state.element.style.left = `${j.x + state.x}px`
        state.element.style.top = `${j.y + state.y}px`
      }
      state.action = "yield"
      state.control = "loop1"
      state.value = j
      return
    }
  }
}

It is much more verbose comparing to main function in the original version, isn’t it? It is also less readable, the execution control isn’t clear. It is not immediately seen how execution control reaches some state.

There are quite a few other implementation options. For example, instead of switch statement we could use callbacks for the control state, we also could use closures to store the state, but that won’t change much. To run the function, we also need a framework. It interprets the action the function demands to execute ("read", "yield" in the example), compose the stages, etc.

Splitting

The size of the function and no framework requirements are not the only advantages of async generators. The real magic begins when we combine them.

The most often used function combination is their composition, say for function f and g this is a => f(g(a)). The composition doesn’t need any framework. It is a JavaScript expression.

If we compose two plain functions, the second function starts doing its job only after the first one exists. If they are generators, both functions run simultaneously.

A few composed generator functions make a pipeline. Like in any manufacture, say cars, splitting jobs into a few steps using an assembly line significantly increases productivity. Similarly, in the pipeline based on async generators, some function may send messages to the next using values its result iterator yields. The following function may do something application specific depending on a content of the message or pass it to the next stage.

These functions are the component of business logic. More formally it is any JavaScript function, taking async iterator as its parameter and returning another async iterator as a result. In most cases, this will be async generator function, but not necessary. Someone may create some combinator functions building resulting object with async iterator interface manually.

There are many names commonly in use for such functions now. For example Middleware, Epic, etc., I like name Transducer more and will use it in the article.

Transducers are free to do whatever they want with the input stream. Here are examples of what transducers can do on some message arrival:

pass through to the next step (with yield i)
change something in it and pass next (yield {…i,one:1})
generate a new message (yield {type:”two”,two:2})
don’t yield anything at all thus filtering the message out
update encapsulated state (local variables) based on the message field values
buffer the messages in some array and output on some condition (yield* buf), e.g., delaying drag start to avoid false response
do some async operations (await query())

Transducers mostly listen for incoming messages on for-await-of loops. There may be a few of such loops in a single transducer body. This utilizes execution control state to implement some business logic requirements.

Let’s see how it works. Let’s split the monolithic main function from the above sample into two stages. One convert DOM events into drag and drop messages — makeDragMessages (types "dragstart", "dragging", "drop") and other updates DOM positions — setPositions. The main function is just a composition of them two.

I split the program here because I want to insert some new message handlers between them. The same way when writing new software I wouldn’t focus too much on how to split the code correctly before I understand why I need this. It should satisfy some reasonable size constraint. They also must be separated on logically different features.

The main function there is actually a transducer too (takes async iterators returns async iterator). It is an example of a transducer which is not an async generator itself. Some larger application may inject main from this module into other pipelines.

This is the final version of the nano-framework. Nothing is to be changed there regardless what new features we add. The new features are function specified somewhere in the chain in main.

First features

Now back to the new features. We want to do something else. Not just dragging something on a page. We have special message names for dragging ("dragstart", "dragging", "drop"). Next transducers can use them instead of mouse/touch events. For example, anytime next we can add a keyboard support, changing nothing for this.

Let’s make some mean to create new draggable items, some area where we can drag them from, and something to remove them. We’ll also flavor it with animation on dropping an item on trash area or outside of any area.

First, everything starts with palette transducer. It detects drag start on one of its element, clones it into a new element and replaces all the original dragging event after with the clone. It is absolutely transparent for all the next transducers. They know nothing about the palette. For them, this is like another drag operations of existing element.
Next assignOver transducer does nothing visible for end-user, but it helps next transducers. It detects HTML element a user drags an item over adds it to all messages using over property. The information is used in trash and in validateOver transducers to decide if we need to remove the element or cancel drag. The transducers don’t do that themselves but rather send "remove" or "dragcancel" messages to be handled by something next. Cancel message is converted to "remove" by removeCancelled. And "remove" messages are finally handled in applyRemove by removing them from DOM.

By introducing another message types, we can inject new features implementations in the middle without replacing anything in the original code. In this example it is animation. On "dragcancel" the item moves back to original position, and on "remove" its size is reduced to zero. Disabling/enabling animation is just a matter of removing/inserting transducers at some specific position.
The animation will continue to work if something else generates "dragcancel" or "remove". We may stop thinking about where to apply it. Our business logic becomes more and higher level.

The animation implementation also utilizes async generators but not in the form of transducers. This is a function returning values from zero to one in animation frames with specified delay, default to 200ms. And the caller function uses it in whatever way it likes. Check for the demo animRemove function in the fiddle above.

Many other animation options are simple to add. The values may be not linear but output with some spline function. Or it may be based not on delay but on velocity. This is not significant for functions invoking anim.

Multi-select

Now let’s add incrementally another feature. We start from scratch, from the nano-framework. We will merge all the steps in the end effortlessly. This way the code from the previous step will not interfere with the new development. It is much easier to debug and write tests for it. There are no unwanted dependencies as well.

The next feature is a multi-select. I highlight it here because it requires another higher order function combination. But at first, it is apparently straightforward to implement. The idea is to simulate drag messages for all selected elements when a user drags one of it.

Implementation is very simple but it breaks the next steps in the pipeline. Some transducers (like setPosition) expect exact messages sequence. For a single item, there should be "dragstart" followed by a few "dragging" and a "drop" in the end. This is no longer true.

A user drags a few elements at the same time. So there’ll be messages now for several elements simultaneously. There is only one start coordinate in setPosition x and y local variables. And its control flow is defined only for one element. After "dragstart" it is in the nested loop. It doesn’t recognize any next "dragstart" until exiting that loop on "drop".

The problem can be solved by resorting to storing state, including a control state, in some map for each element currently dragging. This would obviously break all async generator advantages. I have also promised there are no changes to the nano-framework. So it is not the solution.

What we need here is to run transducers expecting to work with a single element in a kind of a separate thread. There is a byElement function for this. It multiplexes input into a few instances of a transducer passed as its argument. The instances are created by calling the transducer in argument supplying it filtered source iterator. Each source for each instance outputs only messages with the same element field. The outputs of all the instances are merged back into one stream. All we need to do is to wrap transducers with byElement.

First, it converts DOM events into application-specific messages in makeSelectMessages. The second step adds selection indicator and highlight selected items after selection ended in selectMark. Nothing is new in the first two. The third transducer checks if a user drags a highlighted item, it gets all other highlighted items and generates drag and drop messages for each of them in propagateSelection. Next setPosition runs in a thread per each element.

Final result

After the multi-selection feature is implemented it is implemented once and for everything. All we need to change is to add it to main and correctly wrap other transducers with byElement if needed. This may be done either in main or in a module where the transducers are imported from.

Here is the fiddle with the final demo with all the features merged:

All the transducers are in fact very lightweight thread. Unlike real threads they are deterministic but they use non-deterministic DOM events as a source. So they must be considered non-deterministic as well.

This makes all the typical problems of multi-threaded environments possible, unfortunately. These are racings, deadlocks, serializations, etc. Fortunately, they are simple to avoid. Just don’t use mutable shared data.

I violate this constraint in the demo by querying and updating DOM tree. It doesn’t lead to problems here, but in the real application, it is something to care about. For fixing this, some initial stage may read everything needed from a DOM and pack into messages. The final step may perform some DOM updates based on messages received. This may be some virtual DOM render, for example.

Communicating with the messages only allows isolating the thread even more. This may be Web Worker, or even a remote server.

But again, I wouldn’t worry before it became a problem. Thanks to async iterators, the program is a set of small, isolated and self-contained components. It is straightforward to change anything when (if) there is any problem.

The technique is compatible with other design techniques. It will work for OOP or FP. Any classic design pattern applies. When main function grows big, we can add some dependency injection to manage the pipeline, for example.

In the example byElement calls abstract threadBy. In practice, you’ll have more and more such abstract utilities. I wrote a concrete implementation for grouping streams by element, and only after abstracted it. It was very simple, as the concrete implementation was very small.

The technique reduces worrying about application’s architectures. Only write a specific transducer for each feature you need to implement. Abstract common parts into stand-alone transducers. Split it into a few if something else is to be done in the middle. Generalize some parts into abstract reusable combinators only when(if) you have enough knowledge for this.

Relation to other libraries

If you are familiar with node-streams or functional reactive libraries such as RxJS, you could already spot many similarities. They use different stream interfaces.

Transducers don’t require to be async generators as well. It is just a function taking a stream and returning another stream regardless of what interface the stream has. Same technique to split business logic may be applied to any other stream interfaces. Async generators just provide excellent syntax extension for them.

Someone familiar with Redux may notice messages handlers are very similar to middlewares or reducers composition. Async iterator can be converted into Redux middleware as well. Something like, for example, is done in redux-observable library but for different stream interface.

Though, this violates Redux principles. There is no longer a single storage now. Each async generator has its own encapsulated state. Even if it doesn’t use local variables the state is still there, it is the current control state, position in the code where the generator was suspended. The state is also not serializable.

The framework fits nicely with the Redux underlying patterns though, say, Event Sourcing. We can have a specific kind of messages propagating some global state diffs. And transducers can react accordingly, probably updating their local variables if needed.

The name, transducer, is typically associated with Clojure style transducers in JavaScript world. Both are the same things on a higher level. They are again just transformers of stream objects with different interfaces. Though Clojure transducers transform stream consumers, async iterator transducers from this article transform stream producers. A bit more details are in Simpler Transducers for JavaScript article.

We could transform consumer in async iterators as well, by transforming arguments arrived in next/throw/return methods of iterators. In this case, we won’t be able to utilize for-await-of though, and there are no evident benefits.

Extensions

I now work on a transpiler for embedding effects in JavaScript. It can handle ECMAScript async, generators and async generators function syntax extensions to overload default behavior.

In fact, the transpiled demo above was built with it. Unlike similar tools like regenerator, it is abstract. Any other effect can be seamlessly embedded in the language using a library implementing its abstract interface. This can significantly simplify JavaScript programs.

At the moment there are libraries only for standard effects implementation. There’ll be more soon.

For example, possible applications are faster standard effects, saving current execution to a file or DB and restore on a different server or recover after hardware failure, move control between front-end and back-end, on changing input data re-execute only relevant part of the program, use transactions, apply logical programming techniques, even Redux principles for async generators may be recovered.

The compiler implementation itself uses the technique described in the article. It uses non-async generators since it doesn’t have any async messages source. The approach significantly simplified the previous compiler version done with Visitors. It now has almost a hundred options. Their implementation is almost independent, it is still simple to read and extend.

Simpler Transducers for JavaScript

Vitaliy Akimov — Mon, 31 Aug 2020 10:10:30 +0000

Developers often want to split computations into several separate stages. The smaller the stage, the easier it is to reason, develop and maintain. For example we split some computation into 3 stages in functions f, g, k with resulting one is input => f(g(k(input))) or using Ramda R.comp(f,g,k) or any other library with function composition operator.

The problem with this approach is intermediate data passed between functions, and each sub-stage should finish its computation completely before passing the result to next stage. The data size they operate with may be large or even infinite if it is some server requests stream. In a case of unlimited data k will never return control. As it is often occurring task, there are many solutions, like nodejs streams with their .pipe() operation adding stream transformer to the chain.

The worse approach would be to pass a single value between the stages and mutate it. It is very difficult to reason about shared data mutation especially if it is some recursive structure, like programming language abstract syntax tree.
Transducers described in this post may be seen as a simpler solution to the problem, working simultaneously, no intermediate data, no data mutations.

Transducers are easy to compose. In fact they are just functions and function composition is just enough, the expressions above (input => f(g(k(input))) and R.comp(f,g,k)) are the same for transducers. The resulting transducer is a pipeline of computations receiving data from producer and passing it to consumer. Producer and consumer may do many things, read/write network data, file, DB, or just in-memory array.

The term transducers became popular after introducing them in Clojure in Transducers are coming blog post and ported to JavaScript by a few libraries including Ramda. Clojure style transducers are different to the ones described in this post. They transform consumers, which are called reducers in Clojure. Unlike these transducers which transform producers. This distinction makes them much simpler to define in use in ES6 because of generator functions.

Clojure transducers type from the original blog post is:

;;reducing function signature
whatever, input -> whatever

;;transducer signature
(whatever, input -> whatever) -> (whatever, input -> whatever)

There is an earlier paper with the example of transducers transforming producers instead of consumers: Lazy v. Yield: Incremental, Linear Pretty-printing in Haskell. And data types there are:

type GenT e m = ReaderT (e -> m()) m
type Producer m e = GenT e m ()
type Consumer m e = e -> m ()
type Transducer m1 m2 e1 e2 = Producer m1 e1 -> Producer m2 e2

To see Consumer there is a reducer from Clojure substitute State e a = s -> m (a, s) into Consumer definition:

Consumer (State whatever) input
= input -> State whatever ()
= input -> whatever -> ((), whatever)
= whatever, input -> whatever

Producer in the paper has a more complex type. Haskell doesn’t have embedded generators.

Fortunately, JavaScript now has. Producers are just any Iterable value. It may be some in-memory array or any generator function. Consumer is a function taking Iterable value and interpreting it somehow, for example by saving results to file, or JavaScript standard Array.from function stores result in in-memory Array. The approach will work even if the sequence is infinite.

Transducers take input Producer (Iterator) along with other optional parameters and return another Producer-iterator with another computation stacked on top of it.

A typical pattern is:

function* myFun(parameter, input) {
  // local variable storing this transducer's internal state
  // state with initial values
  let sum = 0;
  for(const i of input) {
    // use the current value of `i` along with the current
    // state `sum` to compute its next output value `o`
    sum += i;
    const o = sum * 10;
    yield o;      
  }
}

For example map function applying a function to each element is:

function* map*(fun, input) {
  for(const i of input) {
     yield fun(i); 
  }
}

Or filter, passing further only elements satisfying some predicate:

function* filter(pred, input) {
  for(const i of input) {
    if (pred(i))
      yield i;  
  }
}

Taking first num element is:

const take = num => function*(input) {
  let count = 0
  for(const i of input) {
    yield i
    if (++count === num)
      return
  }
}

Next is a more complex chunk function. It receives stream of arrays of arbitrary length, and partition them into arrays of fixed size:

const chunk = size => function*(input) {
  const buf = []
  for(const i of input) {
    buf.push(...i)
    while (buf.length > num)
      yield buf.splice(0, num)
  }
  if (buf.length)
    yield buf
}

Hierarchical data may be handled as well; an example is @effectful/transducer project for JavaScript syntax transformation.

But everything is not this simple if there is an asynchronous code at some stage. Likely this is required in Producer because IO in JavaScript is typically asynchronous. It is possible to call next of an iterator in an asynchronous callback, but not yield.

Recently EMCAScript has got async generators and for await-of syntax extension for this. Everything in this story works for async generators too, except for-of is replaced by for await-of. There is a more detailed case study for async generators as transducers in “Decouple Business Logic Using Async Generators” article.