DEV Community

Observables, Reactive Programming, and Regret

Ben Lesh on June 29, 2020

As of this writing, I've been working on the RxJS project for almost 6 years, I think. When I started out, I really had no idea what I was getting ...

Read full post

swyx • Jun 29 '20

we all have regrets but not enough of us are brave enough to publicly admit them and listen to our detractors. this is one of the best personal reflections I have ever read. the rxjs community is so lucky to have you as it's leader!

Eric Smith • Jun 30 '20 • Edited

3 is the inspiration for a talk and workshop I give on ReactiveX. I had a similar experience when I first found RxJS - I knew something was super cool there, but I couldn't stick a use case on it.

For what it's worth I think this is true of the entire ReactiveX ecosystem. The ReactiveX homepage starts with a marble diagram of debounce. i just completed a nine month Kotlin project that used RxKotlin extensively and we used debounce...once...I think, yet were very effective using Rx tech all over the place. I think it's easy to fall into the trap, after a while, of saying "here's the cool features!" instead of saying what those features are good for.

This is a really nice article, and RxJS is great.

Richard Tong • Jun 30 '20 • Edited

Thank you for taking the time to reflect on RxJS. The truth is, I probably wouldn't be creating a library right now if RxJS didn't exist because RxJS set such a huge precedent for the functional (reactive) programming paradigm. Sometimes you just gotta try new stuff to see what works. I respect that mentality.

RxJS is a library of functions built around observables, not the other way around.

This is my biggest gripe with RxJS. If in a separate universe this were not the case, I'm sure I would be an active RxJS advocate right now. I started rubico because I saw the value of the functional programming paradigm but could not find any existing solutions I was happy with. The primary reason I was not satisfied with RxJS was in fact the Observable; if RxJS had been built around Promises instead of Observables, I would have reached for it in a heartbeat. That is really the only difference between our two libraries; RxJS is to Observables as rubico is to Promises.

Why do I oppose Observables so much?

It's because they, by themselves, make life hard. You can't just assert.deepEqual the value of an Observable. Instead you have to switch your mind to virtual time and use mock Observables. Promises have way better interop with the regular types; it's a one-to-one mapping. Promise of string? Cool. Promise of Array? Also cool. Observables are inherently many-to-one. This is why it's pretty common in practice to use RxJS operators mergeMap or concatMap; you have to deal with Observables of Observables. flatMap is a powerful concept, but in the case of Observable, it's a necessity.

The Observable is the dual of the Iterable, but that by itself should not give it any more of a case to make it into the spec. Shouldn't we as library authors create things that make developers' lives easy? All I see is Observables making life hard, and I can't get behind that.

Yufan Lou • Jul 1 '20 • Edited

You can't just assert.deepEqual on Observable not because it is many-to-one, but because it is lazy, while Promise is strict. When you compare two Promises you are comparing their state and result cache, whereas when you compare two Observables you are comparing all their callbacks, which makes less sense.

Laziness is what gives the cancellation control. Without it you need the cancellation token workaround. C# uses the same workaround too. Between the guarantees and strictness, we each pick our trade-off.

The Observable is the dual of the Iterable, but that by itself should not give it any more of a case to make it into the spec.

The dual of an existing tool is almost always a equally powerful tool, as borne out by mathematics many times over. I agree that Observable needs more polish to improve developer experience. But I am more disappointed by a spec which chooses to break duality without informed ground. It was a chance to future-proof the Promise API and the spec threw it.

flatMap is a powerful concept, but in the case of Observable, it's a necessity.

FlatMap (monadic bind) is a necessity for programming at all. It is a mathematical model for sequential context. Historically programs focus on one such context: the single-thread procedure. The source code line sequence represents that. In Haskell, any such context can use the do notation and avoid calling flatMap directly. In JS, async/await does that for Promise (of which then is flatMap anyway), and there is no reason it can't do so for Observable apart from that Promise is in the spec but Observable is not. Arguing that flatMap is necessary for Observable but not for Promise is misunderstanding Promise. Arguing Promise is more worthy to be in the spec for that reason is circular logic.

Richard Tong • Jul 2 '20 • Edited

You can't just assert.deepEqual on Observable not because it is many-to-one, but because it is lazy, while Promise is strict.

Of course you wouldn't just assert.deepEqual two Observables or two Promises, it's the step between getting something assert.deepEqual-able from Observables that is so much harder than from Promises. With Observables, you have to mock them in virtual time or fully consume them to some arbitrary type (losing precision) to test their behavior. With Promises, there is no time parameter; it's a one-to-one mapping from the Promise to its associated value.

Laziness is what gives the cancellation control.

The Observable's implicit higher order as a type is what gives both laziness and cancellation control. Observables live at a level of abstraction that makes cancellation convenient. I would know because my library lives at a similar level of abstraction. It even implements Promise cancellation on async iterable transformations. The proposal for promise cancellation was withdrawn because

TC39 has decided to investigate a cancellation mechanism in the core library

Maybe they were talking about cancel tokens here? Either way, proponents for Observables rejoiced when this proposal was withdrawn, yet happily continue to cancel Observables. Cancellation in the observable proposal is pretty up in the air, by the way, and is far from standardized. I don't understand why you guys keep waving it around.

The dual of an existing tool is almost always a equally powerful tool, as borne out by mathematics many times over.

Are you talking about Observables here? And implying it should thusly make it into the spec? Does that mean all things dual by mathematics should be standardized? Also, could you clarify what you mean when you say you are "more disappointed by a spec which chooses to break duality without informed ground?" Right now I'm interpreting it as you think Observables should make it into the spec by mathematical duality.

FlatMap (monadic bind) is a necessity for programming at all.

It's really not. Have you tried working with built-in types? I understand if you're working with effectful types all the time that you might see the world this way, but in practice I hardly need flatMap at all.

In JS, async/await does that for Promise (of which then is flatMap anyway)

All you're telling me here is you haven't grokked Promises. .then has its own semantics and is absolutely not a flatMap for Promises. This example with a .flatMap polyfill on Promise illustrates their difference. The following two expressions are equivalent.

Promise.resolve(1).flatMap(x => Promise.resolve(x + 1)) // > Promise { 2 }
// flatMap expects a Promise in the return

Promise.resolve(1).then(x => Promise.resolve(
  Promise.resolve(Promise.resolve(x + 1)),
)) // > Promise { 2 }
// with .then, the Promises are flattened automatically to the nested value

Arguing that flatMap is necessary for Observable but not for Promise is misunderstanding Promise.

Arguing flatMap is necessary for Promise is misunderstanding Promise.

Arguing Promise is more worthy to be in the spec for that reason is circular logic.

Promise is already in the spec.

Yufan Lou • Jul 2 '20

could you clarify what you mean when you say you are "more disappointed by a spec which chooses to break duality without informed ground?"

I am no proponent of Observable, nor have I interacted with them. I feel the same as you about its verbosity, and I appreciate your rubico library. In the end everyone mostly have moved on to async functions and async iterators, which are at where I wished the level of abstraction Promise spec was. As references, Rust Future is conceptually Future { poll :: () -> Pending | val }, and Haskell Async is conceptually Async { _asyncWait :: () -> SomeException | a }, both of which have their own async/await. Promise is essentially the Pending | val part.

.then has its own semantics and is absolutely not a flatMap for Promises

.then has its own semantics including a flatMap for Promises. The minor mismatches are just a few joins or a pure away. The important part is the concept of sequence.

I understand if you're working with effectful types all the time that you might see the world this way, but in practice I hardly need flatMap at all.

Who are not? I guess we differ about what is effectful. State and error handling are central to programming and are effectful in my vernacular. You just don't need to explicitly write out the flatMap because the language takes care of it.

Richard Tong • Jul 2 '20 • Edited

State and error handling are central to programming and are effectful in my vernacular. You just don't need to explicitly write out the flatMap because the language takes care of it.

Actually I see your point here. I'm becoming more aware that basically everything is an effect, especially when it comes to designing languages. I see how the language can take care of stuff like this; in fact rubico provides a language that takes care of Promises in this way.

The minor mismatches are just a few joins or a pure away. The important part is the concept of sequence.

This thought crossed my mind when I was writing out that example. Basically Promise .then does a whole bunch of flatMaps arbitrarily, if required. I'll chalk it up to a difference of what is effectful.

George Bullock • Jun 30 '20 • Edited

As I said via Twitter, owning your mistakes and making them public takes high-levels of both courage and maturity. Mad props to you for doing that and thank you for elaborating on your Tweet in this write-up 👍🏾 .

The best way to follow through on the first sentence of "The Road Forward" is to write an open-source book. Instead of targeting mid-level and senior developers, assume your readers are juniors that know nothing about Observable.

Before reading this article, I heard of Observable, but I had no clue what it was. Since RxJS is so popular, I would pay $30 for a book that teaches it from scratch. I'm imagining YDKJS, but for RxJS and Observable. That said, if you want to spread the word to as many people as possible, an open-source book is a better option.

Thanks again for the write-up. I wish you and the RxJS team the best going forward.

Randall Koutnik • Jul 1 '20

Not sure about the rules of self-promotion here, but I wrote a book on learning Observables/RxJS with practical demos: pragprog.com/titles/rkrxjs/ Happy to answer any questions you might have (though Ben's the true expert)

George Bullock • Jul 2 '20

Thanks, Randall!

Dmitry A. Efimenko • Jun 29 '20

Really a great read! I've been a user of RxJS for a long time now and consider myself pretty comfortable with it. However, even though that's the case, it really is helpful to hear again how important it is to mentally separate the Observable primitive type from the operators.

brucou • Jul 2 '20 • Edited

I find it interesting, as you suggest, to separate the primitive from the operators. However, I don't see Rxjs size, at least in its latest versions, as being a problem. As operators can be imported separately, there is minimal impact bundle-wise. The impact seems to be more of a cognitive nature, with some folks maybe assuming that because there are 100 operators, it must be complicated, probably twice as complicated as if there were 50? This is like saying that npm is unusable because there are million of packages there.

I think at core there is both a teaching and learning problem. Beating up yourself, and reducing the number of operators to make the library artificially more palatable will not automatically solve those problems.

Teaching problem:

While you were busy developing, testing and using the library, folks have filled a void that existed with all sorts of explanations about what/why/where to use Rxjs. Those explanations have extremely varying quality, and are sometimes downright incorrect, At some point people produce observables tutorial that are like monad tutorials, i.e. most of the time confusing. By not having your own take at those questions, you did loose the control of the Rxjs narrative.
I was perplex when I saw that there were going to be a Rxjs conference. I am glad it was a success, but I did not understand why you need a conference around what is barely more a specific implementation of a pattern. Like, you would not do a tech conference just about lodash, or would you? The signal I received was that Rxjs was so big in the world that it became its own thing.
lodash, ramda and others have more methods than Rxjs ever will. Nobody is dissing lodash as being unncessarily complex. That is because they understand each function. If they give some input, they can mentalize what will the output be. This is harder for Rxjs operators because the functional relationship is between stateful objects (explained in the next paragraph).
The available documentation is plentiful (that is good) but for a long time (I don't know if that has been solved now, haven't looked at it in a year) it was incomplete. I explain:
- most operators are well described in the 95% of their operation (which is taking some data, computing something and producing some other data).
- what I found (back in the days) is not often or well described is the 5% left: i.e. errors and completion. That is fine when you deal with one observable, but quickly gets tricky when you deal with several. If f is your operator, and i_n the input observables for f, with o the output: o = f(i_1, ..., i_n). Because any i_n can be in three possible states, to describe (document) f accurately you have to explain what f does in 3*n cases (worse case). I remember blocking on withLatestFrom of old because one parameter observable had not emitted yet
- it is actually worse if the f computation depends on timing or ordering of each source input... This is real complexity cropping in, and that is best done by drawings that by natural language (marble diagram are great but should describe more edge cases).
subjects are seriously under explained, and yet fairly important. So same problem with confusing, contradictory, poorly argumented advices from the community.
schedulers and concurrency should get more love too. Arguably you may remove schedulers from Rxjs and keep 90% of the use cases, but concurrency is not going to disappear from our applications. The problem that schedulers solve if taken out of Rxjs will have to be solved in some other ways.

Learning problem:

what I observed is that programmers have trouble understanding the operators when they do not understand the observable type in the first place. Writing an operator from scratch with learners from the primitive may help dispel the magic.
developers are pragmatic beasts and not everybody wants to be lectured about concepts. Some folks rather open a playground and play with stuff. Streams in that sense are hard to play with. You can't inspect them easily, debugging is unclear, and time consuming. My worse experience with debugging is not when an observable does not emit the expected, it is when it does not emit at all... So good tracing and debugging are mandatory here. They help debug but they help learn, and acquire.

Last thing, the readability issues that you mention comes from the fact that behaviors and events are not well introduced, yet they are a key part of understanding how to use Rxjs. That is more important and very linked to the hot/cold dichotomy (another confusing metaphor IMO). If you had separate types. you could document some operators as accepting only behaviors, like withLatestFrom (is is sample/audit now?), and save some programmers the debugging.

Alright there are so many more things (structure readable code and so on), but this is getting too long. In any case, while I appreciate the humility and capacity of self-reflection that you show , I would not want you to make the wrong diagnostic here. Observables are an abstraction. Some programmers will always balk at abstraction that they cannot get in 5mn. For the rest of them, I mostly see documentation as an issue. Rxjs design is quite good. But if you could make a better tracing and debugging story, you may not even need to do too much documentation-wise. You could just let people play with it and build their own intuition.

In summary, keep up the good work and don't get demoralized by the critics.

K • Jun 29 '20

Thanks for that article!

I found observables intriguing, and Cycle.js got me to look into libs like most and xstream.

I wish JS had observables instead of promises. But in the end, I guess promises were easier to grasp 🙃

Florin Cosmin • Jul 2 '20

thanks for your work on rxjs. I have just begun working with rxjs through angular. I think is such a powerful primitive, but it has links into functional programming which is another thing people (including me) seem to strugle with. That doesn't say that functional programming is bad or Observables are bad. In fact, I would love to master both. I believe I would be able to build much better software if I would master these concepts.

Again, thanks for your awesome work!

Peter • Jun 30 '20 • Edited

We handed people powerful tools and let them go at it

I'm quiet new to RxJS, but I feel this is my biggest issue with it.
I especially think pushing RxJS into Angular was a big mistake, as it ends up everywhere and it seems to me that people don't really try to understand it, and blindly apply some operators until it works.

Now that isn't really RxJSs' fault, but it would help if the library would provide a little less foodguns, and a bit more guidance / best practices.

Dennis O'Keeffe • Jun 29 '20

Takes a lot of humility and level headedness to write something of this caliber! I’ve been a fan of the project since hearing the talk on how you implemented it at Netflix years ago, but I will also be the first to admit it is a project that I misunderstood at first. Don’t let those Tweets get to you! Twitter is no place for constructive criticism.

Arran Fletcher • Jul 2 '20

For me, RxJS had the biggest learning curve.

I didn't understand observables, I didn't know how RxJS tied in, and the code examples I did find, looking back they're terrible code.

But RxJS and observables have also had the biggest payoff once I understood how they work, their use, their seperation, and the use cases that make performance and handling of data so easy. I don't want to go back to not using observables and I hope they get their own native implementation soon!

Robert Franklin • Jun 30 '20

Thanks for sharing. It took me a while to wrap my head around the concept of RxJS as a whole as there was a bit of a learning curve. But after using it consistently, I was able to understand what it was actually doing and how to better implement it. The community appreciates your work.

Manu Sodhi • Jun 29 '20

Really good read. I've used rxjs and Observables before, but brushing up on Observables tonight. This post has me intrigued about Observables!
Maybe adding a quick example (in the post) of Observables usage could help folks better understand how it's used vs what they'd normally do.

Mátyás Fodor • Jun 30 '20

I think this is gonna be a really good direction for RxJS. It's such a powerful library and I was really looking forward to use it when migrated an angularjs app to Angular8. With no internal knowledge of RxJS in the team, at some point we got stuck, so I think more documentation and a smaller API will definitely help. And reactivex.io/learnrx/ is probably still the best functional tutorial out there (not sure who maintains it)

adam-ashored • Apr 13 '21

I love this post (1 year later). RxJS is amazingly powerful and terrible. The complexity that can get introduced with pipes, the callback hell (nested switchMaps that need catchError) that that community spent so long getting rid of all of a sudden became ok again in RxJS. Nobody can write good RxJS code unless they've been doing it for a year. The amount of time seniors have spent walking juniors through an RxJS pipe (and then the amount of time juniors have spent trying to make a change in that pipe and understand what they've done) is massive.

But, as you said, Observable itself is golden.

Go for it - split out the Observable into a separate package and make RxJS 7 depend on it!

Rares Portan • Jul 6 '20

Thank you for writing this.
It must be hard to admit things could have been better. But this is the right way to move forward. I wish other community leaders would follow you, and talk openly about their community problems and how they plan to fix them.

Jan Jaworski • Nov 17 '20

Thanks. This sums up a lot of my problems when working on Angular app.
I often felt that what I am writing is not how it should be, but finding examples of what are good practises was really hard. When things clicked I usually was really happy with how powerful and expressive RxJS is, although as you mentioned, size of the API was daunting and knowing what and when to use was a herculean task.

Si Houlton • Jul 4 '20

Knockoutjs is a great library that extracts the observable without the operators. I find it also lacks the ceremony of Angular which makes it easier to understand and Steve Sanderson has created a Visual Studio template that uses typescript and has a hot reload.

I still find that RXJS (the combination of observances and operators) can do things that as of yet the detractors can not provide alternative solutions to. This means I’ve had a project that I started without RXJS fail but then solve it by using RXJS.

Sarsaparilla • Jun 29 '20 • Edited

Precisely how I feel as a user. I've said RxJS is a baby in a tub of dirty bathwater. I just don't know where the baby ends and the bathwater begins. Well, thanks for clearing that up.

Ochuko Ekrresa • Jun 29 '20

Thanks for the clarification about Observables and RxJS, Ben. As someone who is interested in learning RxJS for quite some time, this has been a huge help.

Andy Edwards • Jul 3 '20 • Edited

Is it worth mentioning zen-observable here? It's pretty bare-bones and Apollo uses it, so it's the observable API I became familiar with and started using in my own projects.

Seanmclem • Jul 1 '20

Honestly the purpose of Rxjs is less clear to me than ever, but still a very interesting read. Thanks

Tom Byrer • Jul 8 '20

Thanks for your insights!

We never taught people how to write readable code

I've noticed the libs that have better documentation & real world examples people can copy/paste do the best.

leob • Jul 8 '20

Good and honest article ... I think the two things that you mention (large API surface and opaque use case) are exactly what kept me from ever getting "into" RxJS.

SteveALee • Jul 8 '20 • Edited

what Shawn said!!!