Observables, Reactive Programming, and Regret

As of this writing, I've been working on the RxJS project for almost 6 years, I think. When I started out, I really had no idea what I was getting into (And I wouldn't have been able to ship those first versions without Paul Taylor and others, for sure). I can remember looking at the number of weekly downloads on npm, and being able to figure out how exactly how many of them were mine. Fast forward to today, RxJS and observables have become extremely popular... Loved by many, hated by a few, and my fear is, misunderstood by most.

Observables vs Reactive Programming vs RxJS

A big problem I see nowadays is how observables have now been intimately linked, for better or worse, to RxJS. Looking back, I sort of wish we had published the Observable primitive as a separate package, and the operators in another package.

When this effort started, I was naively optimistic that Observable would land in the ECMAScript standard, and RxJS would just "become a collection of helper functions" as I think I put it. But years passed, and the TC39 proposal stalled. And the way the world came to know Observable was through RxJS.

Observables are not RxJS. Observables do not require "operators". They are a primitive. The "dual" of the Iterable. A simple push-based type. Nothing more.

Reactive programming isn't necessarily observables. Reactive programming is a paradigm or a practice. It can be done with functions, Promises, etc. In essence, if you can compartmentalize your code into functions that will "react" to incoming events without knowing anything about the source, congrats, you're "reactive".

RxJS is a library of functions built around observables, not the other way around. Observables can, and do, exist in the wild without RxJS. They show up in other libraries, often times in slightly different shapes, but the overall concept is the same. Facebook's Relay has an internal Observable implementation that is eerily similar to RxJS's implementation. In fact, I've lost count of the number of times I've seen an abstraction that amounts to an interface that accepts a callback to handle multiple values, an error, or a completion, and returns or otherwise uses some sort of cancellation semantic.

Regrets

1. The huge API

RxJS 5 inherited its HUGE API surface area from RxJS 4 and under. RxJS 4 and under, in turn, inherited it's API from RxNET, many, many years ago. So much of the API that some might deem "unnecessary" exists because "it always has been, and always must be". RxJS 5 might have been our only chance in the history of the library to truly ween that down. Which we did a bit, but probably not enough. The large API surface leads to confusion and loathing in the community. All of which is understandable, IMO.

2. RxJS out-shined Observable

Observables never had a chance to shine on their own. The real win, IMO, to RxJS is the Observable type itself. Not the operators. Those are just fluff that allow you to do some cool things. Having a lazy type with guarantees like Observable is actually a bigger deal.

With Observable you're guaranteed:

1. Once it's complete, errored, or unsubscribed, you will get no more messages
2. Registered teardown WILL occur. If you complete, error, or unsubscribe, you are guaranteed to clean up resources.
3. A unified API that can represent a wide variety of things: Events, multiple values, single values, user interactions, streaming data, synchronous values, asynchronous values, etc. etc.

There are other great advantages to its design. But IMO, those are the biggest.

RxJS and all of its operators are inseparable in some people's heads from observables. And that's a real shame. Observable is a simple thing. A very simple type. RxJS is complicated with it's huge API and odd names.

3. We never really outlined where RxJS would best serve people

Disclaimer: These are MY opinions on RxJS/Observable use, and not really the RxJS core team's. Feel free to use RxJS or whatever library in whatever way you see fit. If it works, you can maintain it, and you can test it, IMO it's good code. The end.

To put it simply, once people get into RxJS, it's an exciting technology. It suddenly gets used for everything. It's fair to say this mentality exists in tech for a lot of libraries and frameworks. But I think with RxJS it becomes insidious to the detriment of the RxJS community.

Examples:

• You have a button that, when clicked, fetches the latest data and displays it. Do you need full-on RxJS? No, probably not. "But what about cancellation???" .. You wanted an observable. Not operators. You can use RxJS here for the Observable implementation, but I would caution against jumping into concatMap et al. Especially if your team isn't used to RxJS. But that doesn't mean you shouldn't use Observable. In fact, you probably should.

• You have streaming data over a web socket, and you need to split it into a couple of different streams and update two parts of your UI. Yes! This is what RxJS is for. You're a filter operator away from a solid use case.

• You have complex async coordination and/or race conditions, even with APIs that return promises? Honestly, you might want to use RxJS here as well, because of guarantees provided by Observable, and useful operators like concatMap that can guarantee ordering, etc, and have complete interop with async/await and Promise.

4. We never taught people how to write readable code with RxJS

We handed people powerful tools and let them go at it. No guidance or experienced wisdom provided with how to effectively use the library so that you didn't drive your coworkers crazy. This is sort of like getting a power tool set with no manuals. How do you maintain it? How do you resolve issues? Where do you store the tools? etc.

The result of this is people write code they don't understand when they revisit it. Most amazingly, some engineers, who are usually a rational bunch, then declare RxJS to be "unreadable", as in, no matter what they did, they could never make the code readable. Seems defeatist to me. Like anything else, good practices and strategies around reading and organizing rxjs code can be learned and taught. But I know that I personally haven't done enough to spread this know-how.

Consequences

For the most part, I think the response to RxJS has been overwhelmingly positive. The community has organized a conference. I've seen a lot of discussion about it across many communities (beyond just Angular). And usage has been steadily growing.

But on the back swing, there is a trail of destruction to the reputation of RxJS and Observable that has been wrought by misunderstandings about Observable and RxJS, and misuse of the library in general, IMO. There have been well-known tech personalities who have called out "wishing RxJS didn't exist". And my fear is that sort of thinking, if it spreads, will spell doom for the Observable type itself. That would be the biggest shame to this, honestly.

The Observable itself is huge win. It's a primitive that, like I said above, shows up in many forms in many places, and I think it deserves a spot in the language as much as Iterable and Promise. People having a distaste for RxJS's API and/or abuse and misuse is completely understandable, IMO.

There are parts of RxJS I don't like, and here I am unable to pivot the library quickly because it's so popular we'd simply break too many people. But the parts I like the most, the Observable itself, and the guarantees it provides, are in jeopardy of being thrown out with the bath water by some folks. And that's tragic, IMO.

The road forward

For my part, I plan on trying to continue to champion promoting understanding of the when/where/why of RxJS and Observable. And I want to do better to disambiguate Observable from RxJS. I also want to work very hard to simplify the RxJS API: Tighten the API, remove what does not need to be there, improve documentation and readability, add more guidance for folks on how to make their code more maintainable, etc.

Don't get me wrong, I have other regrets with regards to RxJS as it stands, but I'm confident we can remedy all of those things over time. My deepest concern is that there are a huge numbers of people that still don't understand the Observable primitive and its benefits, because they associate it with RxJS and are standoffish about getting involved there because of the learning curve.

Comments

[swyx]

we all have regrets but not enough of us are brave enough to publicly admit them and listen to our detractors. this is one of the best personal reflections I have ever read. the rxjs community is so lucky to have you as it's leader!

[Eric Smith]

3 is the inspiration for a talk and workshop I give on ReactiveX. I had a similar experience when I first found RxJS - I knew something was super cool there, but I couldn't stick a use case on it.

For what it's worth I think this is true of the entire ReactiveX ecosystem. The ReactiveX homepage starts with a marble diagram of debounce. i just completed a nine month Kotlin project that used RxKotlin extensively and we used debounce...once...I think, yet were very effective using Rx tech all over the place. I think it's easy to fall into the trap, after a while, of saying "here's the cool features!" instead of saying what those features are good for.

This is a really nice article, and RxJS is great.

[Richard Tong]

Thank you for taking the time to reflect on RxJS. The truth is, I probably wouldn't be creating a library right now if RxJS didn't exist because RxJS set such a huge precedent for the functional (reactive) programming paradigm. Sometimes you just gotta try new stuff to see what works. I respect that mentality.

RxJS is a library of functions built around observables, not the other way around.

This is my biggest gripe with RxJS. If in a separate universe this were not the case, I'm sure I would be an active RxJS advocate right now. I started rubico because I saw the value of the functional programming paradigm but could not find any existing solutions I was happy with. The primary reason I was not satisfied with RxJS was in fact the Observable; if RxJS had been built around Promises instead of Observables, I would have reached for it in a heartbeat. That is really the only difference between our two libraries; RxJS is to Observables as rubico is to Promises.

Why do I oppose Observables so much?

It's because they, by themselves, make life hard. You can't just assert.deepEqual the value of an Observable. Instead you have to switch your mind to virtual time and use mock Observables. Promises have way better interop with the regular types; it's a one-to-one mapping. Promise of string? Cool. Promise of Array? Also cool. Observables are inherently many-to-one. This is why it's pretty common in practice to use RxJS operators mergeMap or concatMap; you have to deal with Observables of Observables. flatMap is a powerful concept, but in the case of Observable, it's a necessity.

The Observable is the dual of the Iterable, but that by itself should not give it any more of a case to make it into the spec. Shouldn't we as library authors create things that make developers' lives easy? All I see is Observables making life hard, and I can't get behind that.

[Yufan Lou]

You can't just assert.deepEqual on Observable not because it is many-to-one, but because it is lazy, while Promise is strict. When you compare two Promises you are comparing their state and result cache, whereas when you compare two Observables you are comparing all their callbacks, which makes less sense.

Laziness is what gives the cancellation control. Without it you need the cancellation token workaround. C# uses the same workaround too. Between the guarantees and strictness, we each pick our trade-off.

The Observable is the dual of the Iterable, but that by itself should not give it any more of a case to make it into the spec.

The dual of an existing tool is almost always a equally powerful tool, as borne out by mathematics many times over. I agree that Observable needs more polish to improve developer experience. But I am more disappointed by a spec which chooses to break duality without informed ground. It was a chance to future-proof the Promise API and the spec threw it.

flatMap is a powerful concept, but in the case of Observable, it's a necessity.

FlatMap (monadic bind) is a necessity for programming at all. It is a mathematical model for sequential context. Historically programs focus on one such context: the single-thread procedure. The source code line sequence represents that. In Haskell, any such context can use the do notation and avoid calling flatMap directly. In JS, async/await does that for Promise (of which then is flatMap anyway), and there is no reason it can't do so for Observable apart from that Promise is in the spec but Observable is not. Arguing that flatMap is necessary for Observable but not for Promise is misunderstanding Promise. Arguing Promise is more worthy to be in the spec for that reason is circular logic.

[Richard Tong]

You can't just assert.deepEqual on Observable not because it is many-to-one, but because it is lazy, while Promise is strict.

Of course you wouldn't just assert.deepEqual two Observables or two Promises, it's the step between getting something assert.deepEqual-able from Observables that is so much harder than from Promises. With Observables, you have to mock them in virtual time or fully consume them to some arbitrary type (losing precision) to test their behavior. With Promises, there is no time parameter; it's a one-to-one mapping from the Promise to its associated value.

Laziness is what gives the cancellation control.

The Observable's implicit higher order as a type is what gives both laziness and cancellation control. Observables live at a level of abstraction that makes cancellation convenient. I would know because my library lives at a similar level of abstraction. It even implements Promise cancellation on async iterable transformations. The proposal for promise cancellation was withdrawn because

TC39 has decided to investigate a cancellation mechanism in the core library

Maybe they were talking about cancel tokens here? Either way, proponents for Observables rejoiced when this proposal was withdrawn, yet happily continue to cancel Observables. Cancellation in the observable proposal is pretty up in the air, by the way, and is far from standardized. I don't understand why you guys keep waving it around.

The dual of an existing tool is almost always a equally powerful tool, as borne out by mathematics many times over.

Are you talking about Observables here? And implying it should thusly make it into the spec? Does that mean all things dual by mathematics should be standardized? Also, could you clarify what you mean when you say you are "more disappointed by a spec which chooses to break duality without informed ground?" Right now I'm interpreting it as you think Observables should make it into the spec by mathematical duality.

FlatMap (monadic bind) is a necessity for programming at all.

It's really not. Have you tried working with built-in types? I understand if you're working with effectful types all the time that you might see the world this way, but in practice I hardly need flatMap at all.

In JS, async/await does that for Promise (of which then is flatMap anyway)

All you're telling me here is you haven't grokked Promises. .then has its own semantics and is absolutely not a flatMap for Promises. This example with a .flatMap polyfill on Promise illustrates their difference. The following two expressions are equivalent.

Promise.resolve(1).flatMap(x => Promise.resolve(x + 1)) // > Promise { 2 }
// flatMap expects a Promise in the return

Promise.resolve(1).then(x => Promise.resolve(
  Promise.resolve(Promise.resolve(x + 1)),
)) // > Promise { 2 }
// with .then, the Promises are flattened automatically to the nested value

Arguing that flatMap is necessary for Observable but not for Promise is misunderstanding Promise.

Arguing flatMap is necessary for Promise is misunderstanding Promise.

Arguing Promise is more worthy to be in the spec for that reason is circular logic.

Promise is already in the spec.

[Yufan Lou]

could you clarify what you mean when you say you are "more disappointed by a spec which chooses to break duality without informed ground?"

I am no proponent of Observable, nor have I interacted with them. I feel the same as you about its verbosity, and I appreciate your rubico library. In the end everyone mostly have moved on to async functions and async iterators, which are at where I wished the level of abstraction Promise spec was. As references, Rust Future is conceptually Future { poll :: () -> Pending | val }, and Haskell Async is conceptually Async { _asyncWait :: () -> SomeException | a }, both of which have their own async/await. Promise is essentially the Pending | val part.

.then has its own semantics and is absolutely not a flatMap for Promises

.then has its own semantics including a flatMap for Promises. The minor mismatches are just a few joins or a pure away. The important part is the concept of sequence.

I understand if you're working with effectful types all the time that you might see the world this way, but in practice I hardly need flatMap at all.

Who are not? I guess we differ about what is effectful. State and error handling are central to programming and are effectful in my vernacular. You just don't need to explicitly write out the flatMap because the language takes care of it.

[Richard Tong]

State and error handling are central to programming and are effectful in my vernacular. You just don't need to explicitly write out the flatMap because the language takes care of it.

Actually I see your point here. I'm becoming more aware that basically everything is an effect, especially when it comes to designing languages. I see how the language can take care of stuff like this; in fact rubico provides a language that takes care of Promises in this way.

The minor mismatches are just a few joins or a pure away. The important part is the concept of sequence.

This thought crossed my mind when I was writing out that example. Basically Promise .then does a whole bunch of flatMaps arbitrarily, if required. I'll chalk it up to a difference of what is effectful.

[George Bullock]

As I said via Twitter, owning your mistakes and making them public takes high-levels of both courage and maturity. Mad props to you for doing that and thank you for elaborating on your Tweet in this write-up 👍🏾 .

The best way to follow through on the first sentence of "The Road Forward" is to write an open-source book. Instead of targeting mid-level and senior developers, assume your readers are juniors that know nothing about Observable.

Before reading this article, I heard of Observable, but I had no clue what it was. Since RxJS is so popular, I would pay $30 for a book that teaches it from scratch. I'm imagining YDKJS, but for RxJS and Observable. That said, if you want to spread the word to as many people as possible, an open-source book is a better option.

Thanks again for the write-up. I wish you and the RxJS team the best going forward.

[Randall Koutnik]

Not sure about the rules of self-promotion here, but I wrote a book on learning Observables/RxJS with practical demos: pragprog.com/titles/rkrxjs/ Happy to answer any questions you might have (though Ben's the true expert)

[George Bullock]

Thanks, Randall!

[Dmitry A. Efimenko]

Really a great read! I've been a user of RxJS for a long time now and consider myself pretty comfortable with it. However, even though that's the case, it really is helpful to hear again how important it is to mentally separate the Observable primitive type from the operators.

[brucou]

I find it interesting, as you suggest, to separate the primitive from the operators. However, I don't see Rxjs size, at least in its latest versions, as being a problem. As operators can be imported separately, there is minimal impact bundle-wise. The impact seems to be more of a cognitive nature, with some folks maybe assuming that because there are 100 operators, it must be complicated, probably twice as complicated as if there were 50? This is like saying that npm is unusable because there are million of packages there.

I think at core there is both a teaching and learning problem. Beating up yourself, and reducing the number of operators to make the library artificially more palatable will not automatically solve those problems.

Teaching problem:

• While you were busy developing, testing and using the library, folks have filled a void that existed with all sorts of explanations about what/why/where to use Rxjs. Those explanations have extremely varying quality, and are sometimes downright incorrect, At some point people produce observables tutorial that are like monad tutorials, i.e. most of the time confusing. By not having your own take at those questions, you did loose the control of the Rxjs narrative.
• I was perplex when I saw that there were going to be a Rxjs conference. I am glad it was a success, but I did not understand why you need a conference around what is barely more a specific implementation of a pattern. Like, you would not do a tech conference just about lodash, or would you? The signal I received was that Rxjs was so big in the world that it became its own thing.
• lodash, ramda and others have more methods than Rxjs ever will. Nobody is dissing lodash as being unncessarily complex. That is because they understand each function. If they give some input, they can mentalize what will the output be. This is harder for Rxjs operators because the functional relationship is between stateful objects (explained in the next paragraph).
• The available documentation is plentiful (that is good) but for a long time (I don't know if that has been solved now, haven't looked at it in a year) it was incomplete. I explain:
  • most operators are well described in the 95% of their operation (which is taking some data, computing something and producing some other data).
  • what I found (back in the days) is not often or well described is the 5% left: i.e. errors and completion. That is fine when you deal with one observable, but quickly gets tricky when you deal with several. If f is your operator, and i_n the input observables for f, with o the output: o = f(i_1, ..., i_n). Because any i_n can be in three possible states, to describe (document) f accurately you have to explain what f does in 3*n cases (worse case). I remember blocking on withLatestFrom of old because one parameter observable had not emitted yet
  • it is actually worse if the f computation depends on timing or ordering of each source input... This is real complexity cropping in, and that is best done by drawings that by natural language (marble diagram are great but should describe more edge cases).
• subjects are seriously under explained, and yet fairly important. So same problem with confusing, contradictory, poorly argumented advices from the community.
• schedulers and concurrency should get more love too. Arguably you may remove schedulers from Rxjs and keep 90% of the use cases, but concurrency is not going to disappear from our applications. The problem that schedulers solve if taken out of Rxjs will have to be solved in some other ways.

Learning problem:

• what I observed is that programmers have trouble understanding the operators when they do not understand the observable type in the first place. Writing an operator from scratch with learners from the primitive may help dispel the magic.
• developers are pragmatic beasts and not everybody wants to be lectured about concepts. Some folks rather open a playground and play with stuff. Streams in that sense are hard to play with. You can't inspect them easily, debugging is unclear, and time consuming. My worse experience with debugging is not when an observable does not emit the expected, it is when it does not emit at all... So good tracing and debugging are mandatory here. They help debug but they help learn, and acquire.

Last thing, the readability issues that you mention comes from the fact that behaviors and events are not well introduced, yet they are a key part of understanding how to use Rxjs. That is more important and very linked to the hot/cold dichotomy (another confusing metaphor IMO). If you had separate types. you could document some operators as accepting only behaviors, like withLatestFrom (is is sample/audit now?), and save some programmers the debugging.

Alright there are so many more things (structure readable code and so on), but this is getting too long. In any case, while I appreciate the humility and capacity of self-reflection that you show , I would not want you to make the wrong diagnostic here. Observables are an abstraction. Some programmers will always balk at abstraction that they cannot get in 5mn. For the rest of them, I mostly see documentation as an issue. Rxjs design is quite good. But if you could make a better tracing and debugging story, you may not even need to do too much documentation-wise. You could just let people play with it and build their own intuition.

In summary, keep up the good work and don't get demoralized by the critics.

[K]

Thanks for that article!

I found observables intriguing, and Cycle.js got me to look into libs like most and xstream.

I wish JS had observables instead of promises. But in the end, I guess promises were easier to grasp 🙃

[Florin Cosmin]

thanks for your work on rxjs. I have just begun working with rxjs through angular. I think is such a powerful primitive, but it has links into functional programming which is another thing people (including me) seem to strugle with. That doesn't say that functional programming is bad or Observables are bad. In fact, I would love to master both. I believe I would be able to build much better software if I would master these concepts.

Again, thanks for your awesome work!

[Peter]

We handed people powerful tools and let them go at it

I'm quiet new to RxJS, but I feel this is my biggest issue with it.
I especially think pushing RxJS into Angular was a big mistake, as it ends up everywhere and it seems to me that people don't really try to understand it, and blindly apply some operators until it works.

Now that isn't really RxJSs' fault, but it would help if the library would provide a little less foodguns, and a bit more guidance / best practices.

[Dennis O'Keeffe]

Takes a lot of humility and level headedness to write something of this caliber! I’ve been a fan of the project since hearing the talk on how you implemented it at Netflix years ago, but I will also be the first to admit it is a project that I misunderstood at first. Don’t let those Tweets get to you! Twitter is no place for constructive criticism.