This series is about sharing some of the challenges and lessons I learned during the development of Prism and how some functional concepts taken from Haskell lead to a better product.
Note: As of January 2021, I no longer work at Stoplight and I have no control over the current status of the code. There is a fork on my GitHub account that represents the state of the project when I left the company.
While explaining the journey of refactoring Prism around and writing this article, I've received a set of FAQ. I've grouped them here.
What’s the state of the functional rewrite?
I am generally satisfied with the shape of Prism; but this journey is not over yet.
At first, not all the parts have been converted/refactored to use fp-ts. Although this series is only talking about that, I want to emphasize it has never been our primary focus. We never halted the regular development of Prism to rewrite its parts; on the contrary we have continued to fix bugs and ship new features. We never broke the user space.
There are still good opportunities to refactor and make the codebase even better. To give you a brief idea of what we’re working on at the moment:
- We recently introduced the
Donotation (lend from Haskell) to make the code even more readable, with everybody's quick approval on that: https://github.com/stoplightio/prism/pull/1143 - We will hopefully start to work on the third validation refactor, where I hope to introduce a new data structure (
These) that will allow us to have an accumulating (and not halting) validation in case of warnings
On the other hand, there are some parts that will probably never be refactored to be functional, because the value that it would bring is less than the cost of making the transformation. About this, one good example is logging: as I mentioned in part 3, logging is deterministic but it has side effects; therefore, it should be wrapped in a IO monad.
I do not see this happening. This is one of the tradeoffs and I think it is important to stay pragmatic; I am not one of these programmers that are feticist about functional concepts.
Sometimes I had to intervene to stop the FP discussion from going too far: https://github.com/stoplightio/prism/pull/649#discussion_r329107225
What do your coworkers think about it
Believe it or not, so far all the people that have been working on Prism — after some time — ended up loving it. What I have observing with all the people that had to work on Prism (and some other software, since I’ve been in meantime expanding the usage of this into the internal Stoplight’s codebase — is that there are essentially 4 phases the people go into:
- What the heck is this
- I understand it, but I do not like it at all
- Oh now I get why this is useful
- I think I am in love and I want to write all the software with it
I went into the same exact steps listed here but more importantly I do remember going through the same phases also when I had to use React the first time — and recently when I started to use TypeScript professionally.
Hopefully this is a good evidence that functional programming and its related tooling has no difference with any new methodology/piece of technology: people are just scared about new stuff and with the good amount of education they will go over it.
At the time of writing, I have people in phase 4 as well as in phase 1 and 2.
There is a difference with adopting React or TypeScript though. The first two have a very wide audience and online you can consult. It is easy to find people familiar with the technologies.
Although functional programming has been around for way more years than React or TypeScript, we have to face the fact that’s not that spread around as some of us want.
If we pair this with JavaScript, then the niche is already significantly narrowing and if we add TypeScript on top of it, we start to run out of options. For this reason, I was essentially the education for my comrades. Here are some examples of conversations we had:
https://github.com/stoplightio/prism/pull/648#discussion_r328157183
https://github.com/stoplightio/prism/pull/648#discussion_r328160675
https://github.com/stoplightio/prism/pull/648#discussion_r328162406
https://github.com/stoplightio/prism/pull/648#discussion_r328165087
You can see that these comments, more than code reviews, were more like live tutorials on the code. From my point of view, they have helped my fellow coworker speed up the onboarding significantly. It also made him excited about it
How much time did it take to get this far? Was it worth it at all?
Giving a precise timeline is hard, since we never stopped completely working on Prism to refactor the codebase. This has always been a parallel and opportunistic work. Just looking at the dates though, we started in June last year and we still haven’t finished it yet.
On the worthiness of the whole operation your mileage will of course vary. I still have people in the company that did not even dare read the codebase and just claimed it’s bad, but I firmly believe the quality maintainability of the code outweighs pushing contributors away and I’ve stayed away from these kinds of conversations. There are some points that we gained with the switch that alone, were worth the efforts.
- I have never seen Prism crashing on my computer. I’ve never seen Prism crashing on our servers in the hosted version. I’ve never seen a bug report about a crash. Sure, it will respond incorrectly from time to time — but that is a completely separate issue. Since all our errors are modeled as
Either, there is no way you can forget to handle an error making the software crash - We as a team are always automatically on the same page. There’s no more debate about throwing an exception vs return
nullvs returnundefinedand then try to handle all the use cases somehow. There are a lot of areas where the appliance of functional concepts just makes everybody agree. We have only one rule: if it composes, then 99% it’s good. If it does not, then something is wrong.
During this journey and while telling the people about it, I received some questions multiple times. I’m going to try to answer all of them here.
Why not using Haskell/F#/Clojure/WhatEverFancyLanguage?
This is a non-question for me. I am familiar with Haskell and for sure, I would love to have Prism in Haskell. It would probably be an even better product.
On the other hand we have to stay pragmatic and Stoplight made the initial investment in TypeScript — and such language is here to stay. This does NOT mean that I can’t write good software though.
In particular, I got the feeling that TypeScript is mature enough to give a non optimal, but still a pretty decent experience when writing functional code. With Prism, I finally have a tangible example when people are pointing out that this is not possible and we’re condemned to write shitty code forever and ever:
Do you think FP makes always sense?
That’s also kind of a non-question too. I have seen what I call feticist that are like “everything is functional or you’re out of the game” — or something along these lines. I do think it is possible to stay pragmatic and grab the abstractions you need for your application.
For instance, in Prism the functions emitting logs are considered pure, even though they clearly are not (if you remember from the previous article, console.log is deterministic but it has the side effect to write on the screen). This is theoretically wrong, but for the sake of my application, I really do not care.
I will say though that it is always going to make sense to model a significant class of errors not as an exception, but as a real entity of your domain model. Making your application error-aware is only going to give you benefits.
For instance, when looking for an user via email in a database — the fact that such a user does not exist is very possible. There is no reason to throw an exception for that instead of returning an Error object that people will have to handle accordingly. The choice of the errors to treat in such a way is ultimately up to you.
In the case of Prism, we are kind of lucky since it has almost no interactions with the outside impure world (file system, network) and when they happen, most of them are confined in the CLI. The core of Prism is pretty much pure and functional and so almost all errors are properly modelled. Functional core, imperative shell. In a line of business applications though, things might be different.
Are you seriously betting all this on an unknown library called fp-ts?
The short answer is yes. In general I have never been hype or GitHub star driven, and for this case I do not care about the library itself at all.
The reason is simple: fp-ts and any other alternative that you can find on the web (another notable one is funfix) are simply formalising a mathematical structure. It’s a set of proven laws that, unless some mathematician wakes up one day and claims “We did it all wrong for the last 200 years” — ain’t going to change.
What about the performances?
There’s always going to be somebody in the audience asking this question, and my feeling is that somebody asking this is probably missing the point of the whole presentation.
In any case, since I was getting this so much I decided to collect some random data and see what the results would look like.
I am not going to go too much into details, but essentially by looking at the flamegraph of Prism responding to 10k, it turns out the bottleneck is mostly in the validation and the example generation. I was barely able to find any overhead driven by the monads used in Prism. I have never ran any memory benchmarking, and I am not planning to run one at the moment.
Thanks!
If you've arrived here, it means that you've probably enjoyed the whole series and I hope it brought you some value.
I wanted also to thank everybody that's been proofreading all the parts and did some constructive observations and comments. I would have to put so many names that it's probably better to just say thanks to all.
Top comments (2)
Thanks for sharing your story! the fp-ts learning curve includes learning quite much functional concepts as well as the lib API itself so it's a challenge, reading your post kind of inspired me to start converting one of our side projects while trying to educate my team to level up their functional knowledge..
Hey! It’s great to see people going on the same journey. I’d love to help and know more!