RxJS Transducer - Harness the Power of RxJS Operators

Use the well known RxJS operators to manipulate arrays or iterables in a blazing fast way using the new tiny library rxjs-transducer

Most JavaScript developers by now have learned to use Array’s builtin methods like filter, map, reduce, some, and every to manipulate arrays of data in a functional programming style. This style of programming has the advantage of being easier to reason about than imperative loop style programming. It does have a number of drawbacks, however:

• The operations only work for arrays, not iterables.

• There is a fairly limited set of operations. Notable omissions are operations like take, skip, first, last, single.

• Bad performance: When chaining multiple operations, they will each create an intermediate array and thus iterate the array as many times as there are operators. E.g:

Will create 3 intermediate arrays and then iterate the last array to reduce it to a string, a total of 4 array iterations. Not exactly efficient. Of course this is not an issue when the source contains 10 or 100 elements. But if there are millions of elements, it could be a problem.

Using RxJS operators instead

When thinking of RxJS, you usually think asynchronous stream processing, but RxJS will in fact process streams synchronously when possible. This means that we can use RxJS to create a synchronous stream of values from an array or other iterable using the from function:

This snippet will only iterate the array once, transforming and filtering values as it goes along. It is however a bit clunky to have to use the from, pipe, and subscribe keywords, so I have written a small transducer library that reduces the snippet above to:

The cool thing about this transducer is that it supports iterables such as infinite sequences so you can do stuff like:

Furthermore it is written in TypeScript, so it will give you full TypeScript support in your IDE:

Performance

So, how does it perform you say?

Let us make a test using console.time where we map, filter and reduce an array of 10,000,000 random numbers:

310ms vs 47ms! So in this case the rxjs-transducer is more than 6 times as fast as standard array chaining. Your mileage may vary, but in almost all cases it will be quite a lot faster than array chaining.

How to get it

npm install rxjs-transducer (< 1KB GZipped)

Check out the code on GitHub:

rasmusvhansen / rxjs-transducer

A transducer implementation using the excellent operators from RxJs

rxjs-transducer

A transducer implementation using the excellent and well known operators from RxJS The benefits are:

• Performance: Doing a array.map().filter().reduce() causes the array to be iterated 3 times. Using rxjs-transducers, the array is only iterated once. Doing a filter().map().Math.max() on an array with 1,000,000 items is roughly three times as fast with the transducer as with normal array operations.
• Ability to work with lazy and infinite collections (generators)
• Access to a huge library of well tested operators from RxJS such as map, filter, reduce, skip, take, takeWhile, and many others
• Full TypeScript support

Installation

npm install rxjs-transducer --save

Usage

TypeScript / ES6

import { transducer } from 'rxjs-transducer';
import { map, filter, reduce, skip, take } from 'rxjs/operators';
const source = ['a', 'ab', 'abc', 'abcd', 'abcde'];
// Works as standard array

…

View on GitHub

Playground

I have created a StackBlitz playground for you to try it out in your browser:

Let me know what you think.

https://twitter.com/rasmusvhansen

Comments

[Alex Lohr]

Arguably, take is present in the form of slice, while the lack of first with an array is a non-issue. The chaining performance was already improved here: dev.to/danielescoz/improving-javas.... Using RxJS for that task is an interesting approach, though.

[Mihail Malo]

Nice one, is this the best implementation of Transducers in TS(or even all of JS) right now, or only popular because of it's part of RxJS?

In any case, two questions:

1. How do I drain/collect the output into a collection? Especially ES6 Map is of interest.
2. How do I compose the little transducers into one Big Chungus transducer, so that I can assign it to a variable and have it allocated ahead of time instead of constructing it on every invocation of the function that involves the piping?

[rasmusvhansen]

Hi, thanks for replying. I only just created the library a few days ago, so I wouldn't say it is popular (yet) :-)
But if you have a project where you are already using RxJS and are already familiar with the operators, I think it has a low entry barrier.

1. So far it always drains to an array, so the only way to do that is new Map(transducer(xxx)(yyy)) which has a performance penalty of first allocating the array.
2. You can't at the moment. But it is a good suggestion. I think I can solve both issues by allowing the transducer to produce a lazy iterable instead. I will look into that.

Thanks for the feedback!

[Mihail Malo]

My apologies.
I assumed it was a not-even-recently-added part of the ancient RxJS behemoth.
Guess that's what I get for not clicking the I have written a transducer function link :)
I see this post was more of a camouflaged release announcement than a tutorial for an established package. 😉
I think that the line that sent me on this path the most was

RxJS will in fact process streams synchronously when possible

[rasmusvhansen]

Oh. I will make it more clear that this is a new library, and make it less camouflaged :-)

[Mihail Malo]

The first two code examples are mixed up.
It shows the transducer pipe first, then the native ES5 Array methods.

[rasmusvhansen]

Yes, I noticed that too, but when checking the Markdown, the links to the gists are correct. Seems like a glitch in dev.to?

I created an issue here: github.com/thepracticaldev/dev.to/...

[Mihail Malo]

😱

[Gil]

This is great!