DEV Community: Taylor Fausak

2020 State of Haskell Survey

Taylor Fausak — Sun, 01 Nov 2020 13:26:03 +0000

Hi friends! I'm very excited to announce the fourth annual state of Haskell survey. Please take a few minutes to fill it out. Thanks! https://haskellweekly.news/survey/2020.html

Golfing language extensions

Taylor Fausak — Wed, 29 Jul 2020 11:37:01 +0000

Recently Baldur Blöndal posted an interesting challenge on Twitter:

// Detect dark theme var iframe = document.getElementById('tweet-1287682449863041024-476'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1287682449863041024&theme=dark" }

When I first saw this there were already a few responses. I played with them a little to produce this monstrosity:

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,'(),)→[0b_0^0e0| !_←[]|_←[],then\case]

Those 73 characters require 21 language extensions to compile correctly! This blog post aims to quickly explain what that code does by showing how to require various language extensions with the least amount of code.

Before we get started, a couple notes:

The challenge was to require the most language extensions with the fewest characters. I would be happy to be proven wrong, but I think you can't do much better than about 10 extensions with 23 characters. Most things have so much syntactic overhead that it screws up the ratio.
Because of that, I decided to change the challenge into roughly: "How many extensions can you require from a single 80-character line of code?" This allows showing off more extensions even though they require more than about two characters.
I'm doing all of this with GHC 8.10 on Ubuntu. My goal is to produce a program that compiles. I don't care about warnings, runtime errors, or portability. The code should fail to compile without each extension. Duplicate extensions like RankNTypes and Rank2Types don't count.

Alright. Let's get started.

Baseline

Since we're playing some variant of code golf here, the idea is to use the fewest number of characters. We should set a baseline for the smallest program that will compile. Since GHC will compile an empty program, that's a total of zero characters. But that's not very interesting.

The original challenge looked for a high ratio of extensions to characters, so I thought it made sense to use a regular value declaration, like this:

-- shortest non-empty program ghc compiles
x=0

There's almost no overhead with that. You have to come up with a name and a value, but those can probably be used later. The equals sign is true overhead, but it's less than other approaches like types or classes.

-- these ones are a little longer
type T=()
class C a
instance Show()

MagicHash

The first thing we can do is add a hash to the end of our value's name to require the MagicHash extension. Like many of the extensions we'll encounter, this is purely syntactic. It simply allows you to name things with a hash after them.

x#=0

ScopedTypeVariables

Typically you would give a top-level declaration a standalone type signature. Or perhaps you would annotate its value. But did you know that the ScopedTypeVariables extension allows you to provide the signature inline? It's true!

x#::Int=0

Normally this extension is used to add type signatures to polymorphic local bindings, but it works just as well for our purposes. Here's an example of its more typical usage:

-- typical scoped type variable usage
main = do
  let zero :: Int = 0
  line :: String <- getLine
  print (zero, line)

UnicodeSyntax

That pesky type signature is taking up two whole characters! We can do better. The UnicodeSyntax language extension allows :: to be replaced by ∷, which is only a single character.

x#∷Int=0

Note that code golf usually counts the number of bytes to discourage exploiting Unicode like this.

PartialTypeSignatures

It's kind of annoying that we have Int in there. It takes up three characters without really requiring any language extensions itself. Fortunately PartialTypeSignatures lets us save two characters and require another extension.

x# ∷_=0

Note that we have to put a space after the hash otherwise this would compile without the MagicHash extension because #∷ would be interpreted as an operator.

This gets GHC to infer the type even though we have an explicit type signature. In normal usage this is used to specify a small part of a larger type.

TypeApplications

We've been focused on the left side of the equals sign. Let's turn to the right side. First let's introduce an unnecessary call to id. It doesn't do anything by itself, but we'll expand on it in a second.

x# ∷_=id 0

Since id is polymorphic, we can use TypeApplications to pick a concrete type for it. But just like before we don't want to actually pick a type because that will take too many characters to write out. Instead we'll let GHC infer it.

x# ∷_=id @_ 0

Although the type application looks like it would require PartialTypeSignatures, it doesn't. Note that there must be a space before the type application otherwise it won't compile.

KindSignatures

We have one last type-level trick up our sleeve. In addition to giving type signatures to values, we can give kind signatures to types. Since we have two type signatures, we have to pick which one to decorate with a kind signature. Each case would require parentheses, so we'll add it to the type application since it will allow us to remove the space before the 0.

x# ∷_=id @(_∷_)0

Curiously this kind signature doesn't require PartialTypeSignatures either. Also note that we're still using the Unicode ∷ instead of ::.

BlockArguments

Normally if you want to pass a do block to a function you either have to use parentheses (f (do ...)) or some operator (f $ do ...). The recent BlockArguments extension allows you to avoid any decoration around your argument.

x# ∷_=id @(_∷_)do 0

RecursiveDo

I've never used this extension, so I don't have much to say about it. By changing our do to an mdo, we can require another extension. The mechanics of regular versus recursive do binds don't matter since we're not using them anyway.

x# ∷_=id @(_∷_)mdo 0

BinaryLiterals

At long last we come to that 0 at the far right side. It's not requiring any extensions. Let's fix that!

By default Haskell lets you write octal (0o0) and hexadecimal (0x0) literals. With the BinaryLiterals extension you can also write numbers in binary.

x# ∷_=id @(_∷_)mdo 0b0

NumericUnderscores

The NumericUnderscores extension lets you put underscores in numeric literals, as you might have guessed from the name. We can take advantage of this by adding another underscore that will require another extension.

x# ∷_=id @(_∷_)mdo 0b_0

NumDecimals

There's another extension to the numeric literal syntax that allows you to write integral values using scientific notation. This is useful for writing really big numbers without adding a bunch of zeroes.

Unfortunately this one is a little tricky to actually require. Simply writing a number like 1e2 will infer a Fractional constraint and compile just fine without the extension. We need a way to write a number like 1e2 but also forbid it from being Fractional. And we need to do all that without eating up too many characters.

Fortunately there's a handy operator in the Prelude that puts the right constraints on its arguments:

(^) :: (Num a, Integral b) => a -> b -> a

So we can put a number that requires the NumDecimals extension as the second argument to (^) and we should be good.

x# ∷_=id @(_∷_)mdo 0b_0^0e0

ImplicitParams

At this point it starts to feel like we've run out of easy wins. Time to reach for something that adds a bunch of characters but also adds a bunch of flexibility: ImplicitParams.

This is going to add a bunch of characters because we're forced to also add a let ... in ... expression in order to bind the implicit parameter. That's alright though, we'll make up for it later.

x# ∷_=id @(_∷_)mdo let?_=()in 0b_0^0e0

MultiWayIf

It would be nice if we could replace that let ... in ... with something more exotic. Fortunately multi-way ifs don't have a lot of overhead themselves and allow us to exploit a quirk of Haskell's syntax. This will take a little explaining.

When you use guard syntax, the expression must return a Bool. You're allowed to put any expression in there, including something with let ... in ... as part of it. For example:

unit = case "example" of
  string
    | let size = 7 in length string == size
      -> ()

The weird thing about this is that you can have let without in when you're in a guard clause. The following example is completely valid and compiles without warnings:

unit = case "example" of
  string
    | let size = 7
      -> ()

The guard clause is effectively empty and always evaluates to True even though we never produced a Bool from it. We can use this trick to introduce a let binding without paying for the corresponding in.

However that would hardly be worth it if we had to use case because that introduces so many other characters. Thankfully multi-way ifs are more compact. Bringing it all together we can replace our boring let?_=()in with the novel if|let?_=()→.

x# ∷_=id @(_∷_)mdo if|let?_=()→0b_0^0e0

Note that we're using the Unicode → arrow instead of the ASCII -> to save a character.

HexFloatLiterals

Once again we are going to abuse numeric literals for our game of code golf. This time we'll be writing a floating point literal using hexadecimal syntax, which requires an extension.

x# ∷_=id @(_∷_)mdo if|let?_=0x0.0→0b_0^0e0

TupleSections

Why stop there? Now that we have an unused binding just sitting there, we can cram a bunch of interesting values into it. We'll start by converting it into a tuple section, which is a concise way of writing a lambda that produces a tuple.

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,)→0b_0^0e0

TemplateHaskellQuotes

Somehow we've made it this far without introducing any Template Haskell or quasi-quotes. As far as I can tell the easiest way to require some form of TH is to use the quoted name syntax. We'll be grabbing the name of () because it's the shortest constructor name available.

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,'(),)→0b_0^0e0

BangPatterns

This one requires a bit of a walk. It feels like we should have been able to require a bang pattern by now, but I don't think they're allowed on any of the binders we have. We need to introduce a new binder that can have a bang pattern on it. We're not going to do this in the most efficient way, but hopefully it'll be worth it later.

We're going to use a list comprehension to introduce a new binding. As mentioned, this is a bunch extra characters.

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,'(),)→[0b_0^0e0| !_<-[]]

Note that we need a space before the exclamation point otherwise |! would be interpreted as an operator.

ParallelListComp

Fortunately for us, list comprehensions have a few associated language extensions. By adding five more characters we can introduce a parallel list comprehension.

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,'(),)→[0b_0^0e0| !_←[]|_←[]]

TransformListComp

I've got to say, this is one of the weirder extensions. It augments list comprehensions with a SQL-like syntax for querying data. But that doesn't really matter for us. All we care about is how many characters it takes to require another extension. Answer: eight.

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,'(),)→[0b_0^0e0| !_←[]|_←[],then id]

LambdaCase

We're nearly running out of room on our 80-character line. How can we add more language extensions? There's another lightweight syntactic extension to reach for: LambdaCase. It lets us write a short lambda and case together without naming the lambda's variable.

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,'(),)→[0b_0^0e0| !_←[]|_←[],then\case x→x]

EmptyCase

The actual body of the case is kind of annoying. It's taking up five characters without contributing anything! Wouldn't you know it, the EmptyCase extension lets us get rid of that.

x# ∷_=id @(_∷_)mdo if|let?_=(0x0.0,'(),)→[0b_0^0e0| !_←[]|_←[],then\case]

So there you have it. That's as many language extensions as I can figure out how to require on a single 80-character line of Haskell code. Can you do better? Are there tricks that I'm missing? Let me know!

Here's the whole thing written with slightly more normal formatting:

{-# language BangPatterns #-}
{-# language BinaryLiterals #-}
{-# language BlockArguments #-}
{-# language EmptyCase #-}
{-# language HexFloatLiterals #-}
{-# language ImplicitParams #-}
{-# language KindSignatures #-}
{-# language LambdaCase #-}
{-# language MagicHash #-}
{-# language MultiWayIf #-}
{-# language NumDecimals #-}
{-# language NumericUnderscores #-}
{-# language ParallelListComp #-}
{-# language PartialTypeSignatures #-}
{-# language RecursiveDo #-}
{-# language ScopedTypeVariables #-}
{-# language TemplateHaskellQuotes #-}
{-# language TransformListComp #-}
{-# language TupleSections #-}
{-# language TypeApplications #-}
{-# language UnicodeSyntax #-}

x# ∷ _ = id @(_ ∷ _) mdo
  if | let ?_ = ( 0x0.0, '(), ) →
      [ 0b_0 ^ 0e0
      | !_ ← []
      | _ ← []
      , then\case
      ]

Running HLint as a GHC source plugin

Taylor Fausak — Mon, 25 May 2020 11:51:28 +0000

HLint is a static analyzer for Haskell that suggests how to improve your code and make it more idiomatic. Normally it's run as a standalone executable or an extra test suite. Thanks to GHC 8.6 and HLint 3 it's now possible to run HLint as part of GHC by using a source plugin. This post introduces Splint, which does exactly that.

Motivation

Running HLint by itself is a fine workflow, and you may still prefer doing that even if it's available as a source plugin. But why would you want a linter plugin in the first place?

The main motivation is parsing. It used to be that HLint used its own custom parser called haskell-src-exts rather than the one GHC uses to actually compile your code. This can lead to annoying bugs where GHC parses your code just fine but HLint has trouble with it.

However since version 3 HLint uses GHC for parsing instead of haskell-src-exts. It seems like this should fix the problem, right? Well, yes and no. You might call GHC with some options, like -XTemplateHaskell to enable the Template Haskell extension. Even though HLint is using GHC's parser, you still have to make sure it sets up GHC in the same way, otherwise your module may fail to parse. This is possible to do, but it can be tedious to keep things in sync.

This is why source plugins are nice. You can hook into the compilation process right after the module has been parsed. Then you can pass that parsed module directly to HLint, without parsing it again or doing any serialization. This way you can be sure the module parsed correctly, and you can avoid doing any extra work.

Inspiration

Hopefully you're convinced that running HLint as a GHC source plugin is at least not a bad idea. In fact it was a good idea even before HLint switched over to using GHC's parser. Two years ago Ollie Charles made hlint-source-plugin as a proof of concept.

Wait, what? If hlint-source-plugin already exists, why did I make my own project and write this blog post? As mentioned, Ollie created his project back when HLint used its own parser. That means the plugin has to re-parse modules after GHC has parsed them. It could be upgraded to avoid re-parsing, but the plugin itself is so small that it seemed easier to make a new one from scratch.

At any rate, thanks to Ollie for showing that something like this was possible!

Usage

To use Splint, pass -fplugin=Splint to GHC. Any ideas suggested by HLint will be reported as warnings by GHC. For example, if you define Main.hs as:

main = print . concat $ map pure [ 'a' .. 'z' ]

You would expect HLint to tell you to use concatMap. Normally you would need to both compile your module with GHC and lint it with HLint. However with Splint you can compile it and get suggestions from HLint all at once by running:

ghc -fplugin=Splint Main.hs

Among all the usual output from GHC, you should see this new warning:

Main.hs:1:8: warning:
    Use concatMap
    Perhaps: print (concatMap pure ['a' .. 'z'])
  |
1 | main = print . concat $ map pure [ 'a' .. 'z' ]
  |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

And that's all there is to it! HLint suggestions as part of your normal build process. What a time to be alive.

If you want to pass arguments through to HLint, you can use -fplugin-opt=Splint:arg. For example you can ignore the warning above with -fplugin-opt=Splint:'--ignore=Use concatMap'. Usually this won’t be necessary because Splint will use your .hlint.yaml configuration file.

Trade offs

Running HLint as a GHC source plugin has some upsides:

Modules are only linted when they're compiled, so if they haven't changed they won't be linted again.
HLint's suggestions are reported just like GHC's warnings. They're formatted the same way and they're reported at the same time.
Each module is only parsed once.
Parsing is done by GHC instead of something like haskell-src-exts. HLint already works like this, but by using a plugin you can be sure that all of the versions and options line up correctly.

However it's also got some downsides:

Using Splint means adding it as a dependency to the targets you want to lint. Normally HLint is either a test dependency or just installed on the system.

You may be able to lessen the impact of this by providing a flag to control linting. That way you can enable it locally and in CI, but not require everything downstream of you to depend on HLint.
```
flag lint
  default: False
  manual: True
library
  if flag(lint)
    build-depends: splint
    ghc-options: -fplugin=Splint
```
It's slower. I've found that it adds about a tenth of a second per module.
You can't use the automated refactorings that HLint provides.
Using plugins marks every module as unsafe.

Conclusion

Hopefully this post has explained why you might want to integration HLint into your build process as a GHC source plugin. Splint makes it a cinch, so please check it out!

How to define JSON instances quickly

Taylor Fausak — Tue, 19 May 2020 11:23:29 +0000

Working on a large Haskell project means grappling with GHC's long compile times. This can be especially frustrating when doing iterative development. To help with that, Matt Parsons recently posted some tips for keeping compilation fast. He suggests ways to organize types, instances, and modules in order to speed up the build. In a similar vein, a few years ago I posted about how deriving type class instances takes longer than you might think.

Today I'm going to explore various methods for defining type class instances and their relative performance. Is it faster to write instances by hand or to derive them? Should you put everything in one module or put each type in its own? Is it always faster to use more threads? How long does optimization take? Let's find out.

Data types

If we're going to talk about the performance of deriving type class instances, we'll need some types to derive the instances for. Fortunately I maintain Rattletrap, a command-line program that parses Rocket League replays and outputs JSON. As you might expect, it defines a bunch of types. There's nothing too surprising about them. They fall into three main categories: simple type wrappers, records with fields, and enums with constructors. For example:

-- simple type wrapper
newtype Word32le
  = Word32le Word32

-- record with fields
data Version = Version
  { major :: Word32le
  , minor :: Word32le
  , patch :: Maybe Word32le }

-- enum with constructors
data RemoteId
  = PlayStation Text [Word8]
  | Steam Word64le
  | Xbox Word64le

Type classes

There are many type classes that you can define instances for. We're going to focus on Aeson's ToJSON and FromJSON type classes. Since Aeson is the most popular JSON library on Hackage, you're probably going to need these type classes at some point. Plus they're easy to implement but still interesting enough to avoid being trivial. If you're not already familiar with them, here they are:

class ToJSON a where
  toJSON :: a -> Value
  toEncoding :: a -> Encoding

class FromJSON a where
  parseJSON :: Value -> Parser a

The extra toEncoding method on the ToJSON type class is there for performance reasons. It encodes directly without creating an intermediate value.

Instances

I won't get into what the instances do too much since that's not really the point of this post. But when it comes to implementing the instances, you have at least three options:

Write the instances by hand. This is tedious and error prone, but also straightforward and easy to understand.

instance ToJSON Version where
  toJSON x = object
    [ "major" .= major x
    , "minor" .= minor x
    , "patch" .= patch x ]
  toEncoding x = pairs
    ( "major" .= major x
    <> "minor" .= minor x
    <> "patch" .= patch x )

instance FromJSON Version where
  parseJSON = withObject "Version" $ \ x -> Version
    <$> x .: "major"
    <*> x .: "minor"
    <*> x .: "patch"

Derive the instances using generics. This requires the DeriveGeneric and DeriveAnyClass extensions, along with the GHC.Generics module. It doesn't require any special syntax and behaves the same as deriving more typical type classes like Eq and Show.
```
{-# LANGUAGE DeriveGeneric, DeriveAnyClass #-}
import GHC.Generics
data Version = Version
  { -- omitted, but same as before
  } deriving (Generic, ToJSON, FromJSON)
```
Derive the instances using Template Haskell. Unsurprisingly this requires the TemplateHaskell extension. It uses a different syntax and runs custom code at compile time.
```
{-# LANGUAGE TemplateHaskell #-}
$( deriveJSON defaultOptions ''Version )
```

Details

Since this is a benchmark, I'm going to be pedantic about how I'm running things. Feel free to skip this section if you don't care.

I extracted 77 data types from the Rattletrap project. I put them into a Haskell script for generating code in various configurations. I uploaded the code to this Gist so you can run it for yourself.

I ran the code using Stack 2.3.1 with the LTS-15.13 snapshot, which includes Aeson 1.4.7.1. Instead of using the provided version of GHC, I upgraded to 8.10.1.

I ran the code on two desktops: a Windows machine with an Intel Core i5-8400 (6 cores, 6 threads) and an Ubuntu machine with an AMD Ryzen 5 2400G (4 cores, 8 thread). Both machines use SSD storage.

All measurements are given in seconds. Times are taken from all the configurations listed in this post. Initially I ran each configuration three times, but the times didn't change much between runs. So in the end I ran each configuration only once.

All compilation was done from scratch using -fforce-recomp along with -v0 and -w.

Optimization

GHC offers three optimization levels, from least optimized to most: -O0, -O1, and -O2. Usually development builds are done with -O0 and production builds are either done with -O1 or -O2.

`-On`	Minimum	Average	Maximum
0	1.75	4.74	8.03
1	15.93	30.09	48.57
2	19.32	32.79	52.25

Unsurprisingly -O0 is by far the fastest. For quick feedback from the compiler, make sure you're disabling optimizations by setting -O0. For production builds, this workload didn't show a big difference between -O1 and -O2. (Note that I only analyzed compile time performance, not run time.)

Layout

When writing code, you can choose to put everything in one module or to put each thing in its own module. (You can, and probably do, do something in between, but I didn't benchmark that.) Working in a single module can be easier to reason about, since you don't have to jump around to different files.

Layout	Minimum	Average	Maximum
Single module	2.34	25.28	52.25
Multiple modules	1.75	19.81	48.55

Splitting up your code into multiple modules will almost always make your code faster to compile. Even if it didn't, it would still be a good idea because after the first compile you'd only have to recompile the things that changed.

The only time that a single module is faster than multiple modules is when you only have one core/thread at your disposal. And even then it's not too much faster.

Jobs

GHC can use multiple threads when compiling. This seems like it should help a lot, since compiling each module can be done more or less in isolation.

`-jn`	Minimum	Average	Maximum
1	2.34	26.75	49.56
2	2.33	22.50	48.92
4	1.75	20.17	50.23
8	1.81	20.75	52.25

As expected, using more cores is faster, up to a point. On my machine, going from 4 to 8 threads slowed things down rather than speeding them up.

Averaging these measurements downplay how significant of an impact they can have on your compile times. For example when compiling multiple modules with hand written instances at -O1, going from -j1 to -j4 makes the build nearly twice as fast (from 31 seconds to 17).

Implementation

As explained in the intro, I looked at three different implementation strategies: writing instances by hand, deriving them with Template Haskell, and deriving them with generics.

Implementation	Minimum	Average	Maximum
Manual	1.75	17.37	35.72
TH	3.79	21.27	40.79
Generics	4.32	29.00	52.25

In spite of its reputation as being a heavyweight, Template Haskell easily beats out generics when it comes to speed of deriving instances. However both of them are absolutely crushed by hand written instances. If you want compilation to be as fast as possible, you should write your own instances. Otherwise you should probably use TH.

That being said, Template Haskell does have one major downside when compared to generics: It recompiles more often. Let's say module A depends on module B. If you change module B, module A will be compiled if it contains TH. If it doesn't contain TH, it may not be recompiled. (It depends on what changes you made to B.) So if you frequently change modules below ones with TH in the hierarchy, you may be better off using generics.

Conclusion

When it comes to type class instances in Haskell, if you want compilation to be fast you should:

Compile with -O0 for development.
Put each type in its own module.
Use lots of threads, but not too many. (-j4 is probably a good default.)
Prefer Template Haskell over generics. (Write them by hand if you don't mind the tedium.)

Future work

Are there other build configurations I should have benchmarked? Let me know and I'll try them out!

I had a hard time slicing and dicing these numbers. If you want to take a deeper look at the data, I encourage you to open the spreadsheet and poke around.

I think it would be interesting to explore using GHC source plugins to derive instances quickly without forcing recompilation as often as Template Haskell.

Match URI templates with Burrito

Taylor Fausak — Sat, 09 May 2020 22:43:16 +0000

URI templates are a powerful way to succinctly describe URLs. For example the URI template https://hackage.haskell.org/package/{name} shows you how to find a Haskell package on Hackage. By expanding that template with name := "parsec" you can get the full URL of a package: https://hackage.haskell.org/package/parsec.

>>> let Just template = parse "https://hackage.haskell.org/package/{name}"
>>> expand [("name", stringValue "parsec")] template
"https://hackage.haskell.org/package/parsec"

RFC 6570 specifies how URI templates work. Unfortunately it doesn't describe how to go in the other direction. That is, how to take a full URL and parse it according to a template. For example you may want to take a package URL and match it against the template to get the package name. This is kind of like doing template expansion in reverse.

>>> match "https://hackage.haskell.org/package/parsec" template
[[("name", String "parsec")]]

I'm happy to announce that version 1.1.0.0 of Burrito, my URI template library for Haskell, can now perform this matching operation! So if you find yourself needing to expand or match URI templates, please check it out.