You Either Have It Or You Don't

Reading Material

Yet Another Static-vs-Dynamic Shindig

This week I came across a fantastic article by Eric Normand on the whole "static vs dynamic" debate, written by someone with the unique perspective of having worked professionally in both Clojure and Haskell.

The article is worth a read, but I want to focus one one line in particular that stuck with me. I don't fully understand it, and am hoping DEV may be able to shed some extra light.

The author was quoting a Rich Hickey talk. For the uninitiated, Rich Hickey is the creator of Clojure, a dynamic functional Lisp dialect for the JVM, and current CTO of Cognitect. Part of his cult of personality has stemmed from the rather excellent and rather opinionated talks he's given discussing both Clojure specifically and his personal take on the current state of the software ecosystem and how it applies to the sorts of problems he solves. (The rest stems from his phenomenal hairdo.)

That's a huge takeaway, something Normand takes extra care to identify: Rich Hickey isn't talking about the merits of static vs. dynamic typing in the general case. He's concerned with how it applies to his problem space. This is a detail that is all too often lost when these debates flare up, but is a factor when making absolutely any technical decision at all, and is important to keep in mind when engaging in this sort of discourse.

The Efficacy of Maybe

The line in particular that caught my eye was when he's discussing the use of Maybe in languages like Haskell to handle the concept of optional values. Instead of a String, you'd type your field Maybe String. Now if can either contain a string or not and still be properly typed. You used pattern matching and destructuring to get at that actual value when your code needs to do something with it. You use it when you don't know for sure whether or not this value will exist at all times in a given record.

Rich Hickey doesn't like this. He disagrees with this approach, stating "you either have it or you don’t". It makes sense, too. For the sorts of programs Clojure is designed to address, everything would be wrapped in a Maybe. These are long-running systems with high fault tolerance and constantly changing business needs. These systems do not lend themselves to hyper-rigid beautiful strongly-typed structures. If the spec changes, you don't want to have to make structural updates across every file in your project. You want to be able to just deal with objects of different shapes.

I can understand why it may be cumbersome to explicitly include a Maybe on every single value, but I'm less clear on why it actually represents an inaccurate model of your system. In fact, not knowing Kotlin it actually kinds seems like that's what they're approaching - reducing the overhead for a nullable type to a single character.

That sounds like it works to me, but it also feels like there's truth to Hickey's perspective. When your system is running and you have a record, he's right. You don't "optionally" have a string here at runtime. You've either got it or you don't. That's just true. It seems like the Maybe-based model is indeed a workaround for the convenience of the programmer, as opposed to an accurate model of a real-world phenomenon. At the same time, though, it does seem to model the expected behavior, and at runtime you'll still need to dispatch logic based on whether or not a given field is present.

Do you agree? Disagree? I've always felt Maybe to be an effective solution to this problem, and this read has made me want to step back, consider the issue more carefully, and collect some perspectives.

Photo by marc liu on Unsplash

Comments

[Curtis Fenner]

I don't understand what the phrase "you either have it or you don't" means.

Not having something can cause significant departures in logic (or make some operations altogether impossible or meaningless). The ability to insist that a value is present is extremely valuable because it dramatically simplifies the specification of a unit and dramatically simplifies its implementation. More importantly, it dramatically simplifies the review and understanding of code, since types are the best source of trustworthy information about your code (Every day I curse the inability for me to ensure fields aren't null in Java, because I have to spend so much time manually determining whether or not they are, and most of the time I still don't end up very sure)

Further, as specifications change, types are a help, not a hindrance. If you change a business requirement which leads to many files needing to be changed because their types are now in conflict with the new business requirement... Then those files are wrong under the new business requirement. They would be wrong in a dynamically typed language too! It's just that in a dynamically typed language, you would never know! And, in the statically typed language, you have accurate refactoring tools which can safely manage large, multi file changes which have to be done manual in a dynamically typed language.

[Ben Lovy]

Cool, agreed on all points. I suppose I'm thrown because, as Michael said, I can see the logic behind most other Hickey quotes. This one just doesn't seem like a useful or relevant statement to me, neither a point for or against dynamic typing. It's just kinda nonsensical.

[Kasey Speakman]

So, I have a round-about answer. I write in F# and in a simplified manner which mostly avoids the type puzzles created by Haskell. Instead the types are used more as validation (Option is very valuable to identify which fields are usable but not required when deserializing from JSON, for example) or to have assistance from tooling (exhaustive pattern match, autocompletion, etc). For writing business code, the clarity that types bring and the questions they force me to ask is very valuable.

However when writing infrastructure code, I sometimes think "this would be easier in Clojure". Because infrastructure code is often more general, so it needs more flexibility. I might not know exactly which settings I can pull from the environment vs a config file vs the command line. And then I may be able to run with modified functionality (or defaults) depending on which config values are present. The kind of typed code you have to write for this is very tedious. One of his statements to the effect of "if you used Maybes, then everything is a Maybe" fits really well in describing this scenario.

Another infrastructure example is the "context" object for APIs. It gives you the particulars about the request you are handling. It can be hard to create a single type that covers all the different resources or special situations you might have in your API.

And in both of these cases, things may get added or removed over time to support new features. And since they are infrastructure, structural type changes are breaking to everyone using it. In fact, many libraries for typed languages include a less-typed data structure for this kind of information. For example ASP.NET has a Configuration feature which keeps the data as a (string) key-value collection. Also in ASP.NET the HttpRequest type (a context object) is capable of storing arbitrary dynamic object values in its internal key-value collection. Both of which are to cover unforeseen cases that you may have in a non-breaking way. The difference here is that Clojure is designed to work with this kind of data. But in .NET it is not encouraged nor as well-supported. It can be a downright pain.

For us, using F# is well worth it since we mainly want to focus on the business use cases. And the types are, on average, a huge help there. But I definitely feel the other side of the trade-off in certain kinds of code.

[Ben Lovy]

Wow, what a complete answer! It makes a lot of sense to think about "business" vs "infrastructure" code - and I think I've only ever personally written the former. I don't think I've had that experience you're talking about - I've always either been grateful for my type system or wished I had one. But reading this anecdote it makes perfect sense where that type of flexibility fits.

Thanks for your response!

[Hamish Dickson]

Wow. Interesting!

I've never used clojure so correct me if I'm wrong, but clojure is a lisp and everything is a list in lisp? (hence the name) - so the "you either have it or you don't" just translates to either a list with one or more elements or an empty list?

Not sure if it helps, but this is how I use/think about types: when you're coding your program you are actually defining a computation graph. Using types (which like you say are only a compile time thing) are simply there to help you get the graph as correct as possible - it's a tool for the dev/compiler. And you can take that to the extreme in a language like Idris

It's unfortunate in some ways that Maybe is linked so closely with null/not null, because you can think of it another way - sometimes when we're building programs we need to do things which can fail or are outside of our control, like reading from a database or calling some http endpoint. How do you represent that in a computation graph? How do you represent it can fail as well as succeed? How do you say "don't do anything until you get this result"? Monads let us represent computations like this (interview tip: if someone asks you what a monad is, just say "it's a way of representing sequential computations") and Maybe has a monad.

For example, in our code we can say "go to the db and get this record. put the result into a Maybe". Now our compiler knows that there's a section of code which will either return Some blah or None and if it's Some blah everything went well, otherwise there was an error (and we threw away the error message)

Anyway, I can see what he's trying to say and if I'm right about the list thing it even makes some sense for clojure... but I don't think I agree overall :/

[Ben Lovy]

Thanks for the detailed response!

just translates to either a list with one or more elements or an empty list?

Clojure actually provides a rich set of persistent data structures in addition to lists like sets and maps. I think he's referring most commonly to map keys here, but the concept applies to lists as well.

fail as well as succeed

Great point. You're absolutely right, Maybe is being shoehorned here into something it isn't, but happens to be applicable towards.

it's a way of representing sequential computations

This is definitely more concise than my current word salad answer. I still have never managed to "English" the idea as succinctly as I think it should be able to be expressed but I remember specifically they day it "clicked" - truly not a complicated idea.

I think his point is that this idea of the potential for failure or gaps in your data will inevitably (in certain systems) apply to all data, so then why are we dealing with it in the domain level at all? Clojure kinds offloads that to how it manages data in general with it's STM engine. However, moving the complexity doesn't avoid that complexity.

[Simon Massey]

My take is that he is making the point that in many systems your data is much like documents that don’t have a schema and that is well modelled by nested associated arrays (aka maps) where a not present key is different from a present key with a null value.

I don't think what he is saying is particularly controversial its simply part of a larger point he is making. He is saying that Clojure works well where there is a lot of unstructured, incomplete or varying data. He is asserting that that is a common case where in a typed language you use Maybe<Any> for every document attribute.

His says that in Clojure you pass through what you were not looking for. I am not a Clojure programmer but what I think he is implying is that you can use the presence or absence of keys that are not erased at compile time as a way to pattern match whether a function operates on data. So I read “you have it or you don't” to mean ”run or don't run” without messing around with boilerplate type code that adds no real value in this case. Hopefully someone here can enlighten me.

If thats how Clojure works then I can believe that that is better than trying to deserialise something like a JSON document into a nested Maybe<Any> type to then pattern match and check every maybe is a particular something before doing an action.

So far so uncontroversial. Parsing JSON into types is easy if your JSON was created from structured types. It is very messy when the data structure isn't known. In typed languages we tend to side step that with libraries that reflect upon your types and then do the messy parsing without hand writing all the null checking code. When we want higher performance we use a code generator to create the parser and we don’t mind how ugly the generated code is. We think of it as a small price to pay to extract our typed objects.

What seems far more controversial are the statements that he makes that types are an anti-pattern to maintainability. That is the long running “types v. untyped” where he makes some very clear points. Here on dev.to there are many JavaScript vs TypeScript articles where people have experience of large code bases in both arguing on different sides. It is certainly food for thought...

[Ben Lovy]

This is a great answer, thank you. You're right, Clojure is well-suited to processing maps with varying keys, which by extension fits well with use cases like arbitrary unknown JSON.

It seems like the "antipattern to maintainablity" stance also only makes sense when taken in the context of this specific domain. Still controversial, but less so than the blanket statement.

[Simon Massey]

What is interesting is that Rich made so many systems in C++ and one in C# and after 18 years came to the conclusion that types don't help at all and that the examples of where they help are contrived.

Back when I started OO was the big new thing over procedural. It is still the orthodoxy. Yet more and more people are coming to the the conclusion that class hierarchy polymorphism only helps in very narrow places and hurts maintainability. I worked on several large Java systems with dozens of developers and now agree that classes are over used and abused on the typical business apps I worked on. The last large scale multiple dev team system I worked on was written in Scala and found that algebraic types, pattern matching, lexical scoping and functional programming was much better for maintainability.

I have come to the conclusion that the examples where class hierarchy helps are contrived. Yet as a grad student I taught OO as the orthodoxy and I completely bought into it. The fact that Rust and Go doesn't have that class hierarchy based polymorphism and JavaScript only recently introduced classes fits with my experience that people have quietly moved away from the orthodoxy and more great software is being written as a result.

The talk by Rich suggests to me that I should be more open minded about types in general being over-applied much like I have come to appreciate classes were over-applied.

[albert lai]

when you focus on pure data.

what is the different between

{:name "mr. b" :age 12}
{:name "mr. b" :age nil}
{:name "mr. b"}
{:name nil :age nil}
{}
nil

what's the problem maybe could solve?

[michelemauro]

It is different if you have different use cases on when :name has been specified, or not.

In a static type system, those use cases are encoded in a type, which becomes integral part of the information carried by the data. In dynamic type systems these are (hopefully) encoded in documentation and tests.

There is place and use for both systems.

[Ben Lovy]

True, but how do you encode business requirements? What does the defrecord look like? Is this what spec is for?

[Erik Ostrom]

Looks like he followed up that brief remark with a full hourlong talk about Maybe, in case you want to really get into it.

[Gerard Klijs]

It's a great one, to bad the new spec still isn't stable.

[Ben Lovy]

Ah, I do! Thanks for the link, I completely missed this one.

[Ben Lovy]

That makes a lot of sense - from what I've read form him, Clojure seems like it was almost a knee-jerk reaction from a career Java engineer who was losing his mind, and wanted something that addresses his own perceived shortcomings.

I have not yet written enough of anything to lose my mind about it. Context is important.

[Andy Fingerhut]

Maybe this isn't what you meant, but to check, I looked up the dictionary of "knee-jerk" to find it defined as "automatic and unthinking". I think Rich Hickey's approach to Clojure has been anything but that. Deliberate, careful, nuanced, yes. Always fully explained in an instant to everyone who goes looking for answer to why Clojure is the way it is? Definitely not.

I see a theme in all of his work as how to avoid complexity, especially unnecessary complexity that many aspects of software development create.

[Ben Lovy]

Absolutely, you're completely right. I more meant as a reaction borne from desperation to make what he felt was an unworkable environment into a workable one, but you're right, it was a poorly chosen phrase for the resultant product.

[Gerard Klijs]

I'll both like the Java Optional which you can use when at runtime there might not be anything. It has a nice API to work with it. But the Clojure way of returning nil when the argument is nil works also well in most cases, but can sometimes cause weird errors as it's not always fine to pass nil.

[Hamish Dickson]

ah cool - thanks for clarifying. Then I guess I didn't really get his point!

about Either - yes, exactly - I was kind of deliberately over simplifying to make the point that there's other, sometimes nicer ways to think about Maybe

[michelemauro]

Maybe, IIRC, comes from Haskell, which doesn't have nullability so it has nothing to do with nullability checking.

It has everything to do with encoding the fact that having a value or not having one are two different, equally informative, and perfecly legal values. And your code must be able to distinguish those and act accordingly.

Lisp, and then Clojure probably have different idioms (i.e. the empty list) that do not require a union type, and can be treated as equally informative. That's probably what Hickey has in mind.

[Ben Lovy]

Right, I'm with you. Call it whatever you like, encode it at whatever level of your stack you like, reality will come knocking one way or another.

[Gerard Klijs]

Mostly agree, but in Java especially it can be very confusing. Something typed as Optional could still hold a null value. And there is also some diversity in @NonNull annotations, some working compile time, and some runtime. In that case it's much easier if anytime might actually be nil, and that's the only thing you have to take into account. Especially with functions like 'if-let' and how 'and' works in clojure, it's much easier and cleaner to write null safe code in Clojure is my experience.

[Ben Lovy]

That's interesting. I definitely have not spent nearly enough time with Clojure, but always had a better time around null using optional types. My Clojure I never felt as confident about - but I think this is a familiarity issue.