DEV Community: Nate May

CircleCI and Haskell

Nate May — Fri, 29 May 2020 21:25:17 +0000

I've tried several different approaches to using CircleCI for Haskell projects and it's taken me a while to find a solution that works well. This post documents the solution I like and why I chose it over the alternatives.

CircleCI Language Guide - Stack

The language guide for Haskell provides a working configuration file for stack projects that uses a simple docker image instead of a CircleCI Orb. At the time of writing the docker image is actively being maintained. If you're already happy with using stack with your project this is a fine way to go. The biggest downside I saw to using this approach is that the Spin Up Environment stage where CircleCI downloads the docker image takes about two minutes. Compare this with the standard Haskell docker images which take about 30 seconds, and the cached JVM environments CircleCI officially supports which take about three seconds.

Haskell-works Orb - Cabal

If your project doesn't use stack there's a CircleCI orb for cabal projects that at the time of writing is being actively maintained. This orb tries to do as much magic for you behind the scenes as possible and documents the many orb-specific configurations. This orb lets users choose where they want their cache to be maintained: either in CircleCI, or an AWS bucket. This orb works, but I found it more difficult to use than just using cabal commands in a config file myself. The Spin Up Environment phase only takes about 30 seconds which I found perfectly acceptable.

Custom Config

After trying both of the above solutions I decided to stick with using my own configuration file. I found it a little difficult to get cabal to do exactly what I want so I'm going to document what commands ended up working for me.

This walkthrough goes through all the files in a small example GitHub project. Use this to see all the code snippets in context.

The `.cabal` File

The first step in setting up your Haskell project to work with any CI is to make sure the test stanzas in your .cabal file are set to use exit codes so that your CI can detect when tests pass or fail.

test-suite unit
  type:               exitcode-stdio-1.0
  hs-source-dirs:     src/test
  main-is:            UnitTests.hs
  build-depends:      circleci-cabal ==0.1.0.0
                    , base           ==4.13.0.0
                    , HUnit          ==1.6.0.0
  ghc-options:        -Wall
  default-language:   Haskell2010

The line type: exitcode-stdio-1.0 is the one that enables tests to emit proper exit codes.

Inside your tests you may need to manually emit the exit codes yourself like this example with HUnit:

import System.Exit  (ExitCode(ExitFailure), exitWith, exitSuccess)

...

main :: IO ()
main = do
  results <- runTestTT $ TestList allTests
  if errors results + failures results == 0
    then exitSuccess
    else exitWith (ExitFailure 1)

Other libraries such as test-framework and tasty provide a function defaultMain that do this exit code mapping for you.

CircleCI Config

Now that our tests are set up let's walk though the steps in the .circleci/config.yml file:

version: 2.1

jobs:
  build:
    docker:
      - image: haskell:8.8.3
    steps:
      - checkout
      - restore_cache:
          name: Restore Cached Artifacts
          key: haskell-artifacts-{{ checksum "circleci-cabal.cabal" }}
      - run:
          name: Update Dependencies
          command: cabal new-update && cabal new-install --lib
      - run:
          name: Build
          command: cabal new-build
      - run:
          name: Build Tests
          command: cabal new-build --enable-tests
      - save_cache:
          name: Cache Artifacts
          key: haskell-artifacts-{{ checksum "circleci-cabal.cabal" }}
          paths:
            - "/root/.cabal"
            - "dist-newstyle"
      - run:
          name: Run Tests
          command: cabal new-test --enable-tests --test-show-details=streaming

Docker

CircleCI lets you can use any hosted docker image, but I chose to use one of the official Haskell images.

docker:
  - image: haskell:8.8.3

checkout

Checkout out your code base from version control

steps:
  - checkout
...

Restore Cached Artifacts

Restores cached artifacts from a previous run that had the exact same cabal file.

steps:
  ...
  - restore_cache:
      name: Restore Cached Artifacts
      key: haskell-artifacts-{{ checksum "circleci-cabal.cabal" }}
  ...

Update Dependencies

cabal new-update: fetches the latest package list from hackage
cabal new-install --lib: installs the library and dependencies without looking for an executable. --lib is an undocumented flag but is referenced in many GitHub Issues.

steps:
  ...
  - run:
      name: Update Dependencies
      command: cabal new-update && cabal new-install --lib
  ...

Build

Compiles all non-test artifacts

steps:
  ...
  - run:
      name: Compile
      command: cabal new-build
  ...

Build Tests

Compiles all artifacts including tests. Existing non-test artifacts are already built so they won't be built again.

This outputs an executable that can be called directly as a standalone application to produce an exit code.

steps:
  ...
  - run:
      name: Build Tests
      command: cabal new-build --enable-tests
  ...

Cache Artifacts

At this point everything successfully built so we can save the artifacts for future runs.

/root/.cabal: cabal settings
dist-newstyle: artifacts

steps:
  ...
  - save_cache:
      name: Cache Artifacts
      key: haskell-artifacts-{{ checksum "circleci-cabal.cabal" }}
      paths:
        - "/root/.cabal"
        - "dist-newstyle"
  ...

Run Tests

Uses cabal to run all tests. Could instead use the executable built in the "Build Tests" step by referencing it with a CircleCI environment variable. This is useful when debugging your pipeline to decide if cabal is the issue.

cabal new-test --enable-tests: runs tests
--test-show-details=streaming: this flag redirects all print statements inside tests to stdout instead of a log file.

steps:
  ...
  - run:
      name: Run Tests
      command: cabal new-test --enable-tests --test-show-details=streaming

Additional Gotchas

I have a project that has two separate test stanzas:

A test that forks a Haskell thread
A test that forks an operating system thread

Both use the ghc flag -threaded in their stanzas in the .cabal file

The first test passes and emits the exit code properly.

The second test passes but does not emit the exit code properly. On CircleCI the process hangs indefinitely. To resolve this I used environment variables to call the built test binary directly.

Further reading: Haskell threads vs operating system threads

Conclusion

I honestly had a hard time coming up with this so I wanted to share it both to help anyone else who runs into the same issues, but also to help future me when I inevitably forget some of these details.

Why Learning Haskell Sucks

Nate May — Mon, 24 Feb 2020 05:31:10 +0000

image: Marion Bolognesi watercolor face

Even for brilliant minds, Haskell often takes multiple tries to learn. There are some significant technical hurdles you encounter early on, but that's not the problem. There are endless resources for learning about Haskell syntax, the IO monad, type class hierarchies and essentially every other unfamiliar aspect of Haskell. If you ever post a question to Stack Overflow with the Haskell tag, you'll quickly get thoughtful comments and clear answers from brilliant people. Once you discover Hoogle you immediately wonder how you ever coded without it. There are even talks within the community helping current Haskellers empathize with new learners. Yet even with all this community support and passion for the language, at the peak of my frustration during my first attempt I remember saying out loud,

"It feels like no one actually wants me to learn this."

Now, after my second attempt at learning Haskell I feel confident that I can solve problems with Haskell and learn the parts I don't know yet, but I also know why I felt this way in the beginning.

I recently started listening to the Corecursive podcast by Adam Gordon Bell and in one conversation with Gabriel Gonzales they talk about why different tech products become mainstream and others do not. This resonated with me as a way to frame my experience with learning Haskell.

"Early adopters" are the people who are willing to put up with the warts of a new technology to build something they care about. Successful technologies that move into the mainstream must win over "mainstream users" who want proven and polished products.

If you got a strong adverse reaction to the title of this post, you might be an early adopter. And early adopters are great! They're visionaries who imagine the way the world could be and care about doing things the right way even if it's inconvenient. They also help people creating new technologies discover different directions their product could go.

However, early adopters are never going to convince mainstream developers to put in the work and potentially put their projects at risk for tech that's unpolished and completely new to their social circles. The best way to convert "mainstream users" is to offer an easy solution to a very painful problem.

And here is where we discover the core of the issue. I am, like most developers, a mainstream developer. The only reason Haskell entered my circle was because I started dating a PL Ph.D. student. I actually thought Haskell was really cool when I learned about it, but nothing stopped me from quitting the learning process when it got hard.

Haskell doesn't solve any problems for me that I can't already solve in Scala.

That doesn't mean that Haskell isn't a good solution, and it doesn't mean that Haskell doesn't solve other problems Scala can't solve. It just means it doesn't solve my problems in a way that makes the switch worth it. It also means that it's not the best solution to the kinds of problems my friends are having while they write CRUD apps in Javascript, Python, Java, and C# either.

For people creating new programming languages, Haskell does offer a proven solution in the form of an extensive and mature ecosystem when it comes to making compilers. However, this success hasn't exactly broken Haskell into the mainstream market. You can tell Haskell isn't mainstream because there's no first-class support for it on any platforms. No cloud providers, no official database drivers, nothing. No other technologies have succumbed to the pressure of mainstream developers demanding Haskell support, because they're simply not demanding it. Even newer languages that are arguably worse languages, like Go, have successfully made it into the mainstream.

And this is why it felt like "no one actually wanted me to learn this." The problems I cared about were actively harder to solve in Haskell. And when experienced Haskell developers tried to persuade me with their early adopter rationale, it just made everything worse.

A sentiment I often heard from experienced devs is that Haskell is actually much "simpler" than other imperative programming languages and thus it's easier to understand. Now that I've learned enough Haskell I can absolutely see why this is the case. However, this is a much higher level conclusion that I believe is actively harmful to mainstream developers trying to learn Haskell for the first time. I know it made me feel really stupid for not being able to understand something that everyone else sees as "simple."

When I learned Scala it legitimately felt so much simpler than Java. In Java I would have to write getters, setters, and constructors for every class I created. In Scala I just make a one line case class and all that work is done for me in a pretty frictionless way.

When Haskell enforces purity and immutability this does make reasoning about functions much simpler. However, for your average mainstream developer it makes writing them so much harder. This "simplicity" doesn't actually benefit the developer until they've retrained themselves to write code with these new restrictions which takes time.

Think about what it takes to copy a "hello world" program, and modify it so that it asks for your name and prints "hello <name>". Doing simple things in Haskell is not simple. You can't write that program without running into immutability and the IO Monad.

And I'm not suggesting we change that! But I am suggesting that we change the way we talk about it and just admit that doing simple things isn't simple in Haskell. Haskell developers might argue that an app that interacts with users directly isn't actually simple, and they're not wrong. But Haskell developers don't get to frame the world everyone else is coming from. A learner is going to have their own pre-conceptions of what should be simple, and we can't pretend it's going to be easy to learn the Haskell way of doing those things.

Even though there were lots of hurdles to get to this point, I haven't given up on Haskell. I know it will likely never be the kind of polished experience I want from my programming languages, but I'm going to keep learning it and using it to expand the way I think about software.

There are lots of compassionate people in the Haskell world trying to make the path to learning Haskell easier, and these are amazing efforts that I am truly grateful for. However, Haskell isn't accidentally going to become a mainstream language. If that ever happens it's going to take an enormous coordinated effort to make it worth it for mainstream developers like me to make the switch. And if that does happen, you better believe that stuff like monads and type classes won't stop us.

Don't Read This Monad Tutorial

Nate May — Thu, 19 Sep 2019 20:49:52 +0000

In an attempt to break the cycle where every monad tutorial claims to be better than the others, I seriously don't think this tutorial will make you understand monads better than any other one. That's because reading tutorials can only get you so far. Instead, you should write your own with the intent to publish it. Dig up all the details, try everything yourself, and answer all the questions your readers will inevitably have. I found inspiration in Dan Piponi's idea that I could have invented monads myself and Brent Yorgey's assertion that monads are not, in fact, burritos.

Below is the tutorial I wrote to help me understand Monads back in September 2018 with some minor modifications.

The Issue of Ugliness

In scala we really like stringing together our method calls like this. It's easy to read and easy to debug.

List(1, 2, 3, 4, 5)
  .map(_ + 10)
  .filter(_ % 2 == 0)
  .take(2)

Instead of a regular list, the hypothetical app we're working on really needs a List where these functions return both the list result and a debug string from each of these functions.

object DebugList {
  //variable argument syntax so it works like List.apply
  def apply[A](a: A*):DebugList[A] = DebugList(List(a: _*))
}

case class DebugList[A](l: List[A]){
  def map[B](f: A => B): (DebugList[B], String) =
    (DebugList(l map f), "mapped")

  def filter(f: A => Boolean): (DebugList[A], String) =
    (DebugList(l filter f), "filtered")

  def take(n: Int): (DebugList[A], String) =
    (DebugList(l take n), s"took$n")
}

DebugList is used like this ...which is so ugly.

val (dl1, str1) = DebugList(1, 2, 3, 4, 5).map(_ + 10)
val (dl2, str2) = dl1.filter(_ % 2 == 0)
val (dl3, str3) = dl2.take(2)

But this is what our app needs, so we're gonna try and make it work.

The problem is that we have to do all this clunky matching on the tuples. Plus if we want to see our debug string in order, we need to do something like

s"$str1 $str2 $str3"

to read, print, or write them. Since the tuple is the issue let's try putting the tuple in another class so we can write functions like flatMap to do the gross stuff for us.

flatMap and map

When we fill in the definition of flatMap and map for this case class we have to make sure the resulting debug object has a string with both the contents from this debug object and from the result of f.

case class Debug[A](a: A, s: String) {
  def flatMap[B](f: A => Debug[B]): Debug[B] = {
    val Debug(b, s1) = f(a)
    Debug(b, s + s1)
  }

  def map[B](f: A => B): Debug[B] =
    Debug(f(a), s)
}

Next we'll need a constructor in the companion object that gives us a way to turn regular old A objects into Debug[A] objects.

A way inside

When we fill in the unit function, the empty string is a reasonable choice when one isn't already available.

case object Debug {
  def apply[A](x: A): Debug[A] =
    unit(x)

  def unit[A](x: A): Debug[A] =
    Debug(x, "")
}

Customizing List

Now that we've refactored that tuple, we can make a BetterDebugList with functions that return our newly refactored Debug type instead of tuples.

object BetterDebugList {
  //variable argument syntax so it works like List.apply
  def apply[A](a: A*): BetterDebugList[A] = BetterDebugList(List(a: _*))
}

case class BetterDebugList[A](l: List[A]){
  def map[B](f: A => B): Debug[BetterDebugList[B]] =
    Debug(BetterDebugList(l map f), "mapped")

  def filter(f: A => Boolean): Debug[BetterDebugList[A]] =
    Debug(BetterDebugList(l filter f), "filtered")

  def take(n: Int): Debug[BetterDebugList[A]] =
    Debug(BetterDebugList(l take n), s"took $n")
}

BetterDebugList can now be used like this:

val debug = Debug(BetterDebugList(1, 2, 3, 4, 5))
  .flatMap(_.map(_ + 10))
  .flatMap(_.filter(_ % 2 == 0))
  .flatMap(_.take(2))

Whoa! that's looks pretty similar to how we originally used List. No more tuple matching!

In order to hide those explicit flatMap calls we can use for comprehensions because they're prettier but do exactly the same thing.

val debug = for {
  w <- Debug(BetterDebugList(1, 2, 3, 4, 5))
  x <- w.map(_ + 10)
  y <- x.filter(_ % 2 == 0)
  z <- y.take(2)
} yield z

And if you're annoyed with assigning names for each of your intermediate states, for comprehensions let you call them all the same thing. It almost makes our immutable code read a bit like it's mutable.

val debug = for {
  x <- Debug(BetterDebugList(1, 2, 3, 4, 5))
  x <- x.map(_ + 10)
  x <- x.filter(_ % 2 == 0)
  x <- x.take(2)
} yield x

Now you can get that debug string out like this:

println(debug.s)

Surprise! You made a monad.

Just like many other functional programming tools, a monad takes legitimately useful code that might otherwise be very awkward to use and makes it feel more natural. Notice how we can use this same Debug class to make a debuggable version of any other class we want.

In Scala we use monads all the time because they are so natural. List and Option are both monads that we see in nearly every beginner Scala tutorial.

photo source: my twitter

Ok but what is a monad?

A monad needs...

1 - flatMap (sometimes called bind)

2 - unit (usually implemented with apply in Scala)

3 - follow the three monad laws

Here are some examples of monads you're already familiar with. They all use the apply method instead of a function named "unit" and they all have flatMap.

List(1)                    == List.unit(1)
List(1,2,3)                == List.unit(1,2,3)
Option(5)                  == Option.unit(5)
Try(throw new Exception()) == Try.unit(throw new Exception())

In order to be a monad it has to follow the three monad laws too. These laws just make sure we can refactor our code in the way we expect and have predictable results.

The Monad Laws

f and g are functions

m is an instance of a monad which is also called a "monadic action"

1 - Right Identity

unit(z).flatMap(f) == f(z)

2 - Left Identity

m.flatMap(unit) == m

3 - Associativity

m.flatMap(f).flatMap(g) == m.flatMap(x => f(x).flatMap(g))

Let's look at examples of these law definitions using the Monad List. Imagine how weird using List would be if these statements were not always true:

1 - Right Identity

List(2).flatMap(x => List(x * 5)) == List(2 * 5)

2 - Left Identity

List(2).flatMap(List(_)) == List(2)

3 - Associativity

List(2).flatMap(w => List(w, w)).flatMap(y => List(y * 2)) == 
List(2).flatMap(x => List(x, x).flatMap(z => List(z * 2)))

Break the Monad Laws

The FMCounter class counts how many times flatMap has been called on it. It looks like a monad, but it breaks some of the 3 laws.

Here's its definition. Let's find out which laws it breaks.

case object FMCounter {
  def unit[A](a: A): FMCounter[A] =
    FMCounter(a, 0)

  def append(str: String, end: String): FMCounter[String] =
    unit(str + end)
}

case class FMCounter[A](a: A, counter: Int) {
  def flatMap[B](f: A => FMCounter[B]): FMCounter[B] = {
    val FMCounter(b, k) = f(a)
    FMCounter(b, counter + k + 1)
  }

  def map[B](f: A => B): FMCounter[B] =
    FMCounter(f(a), counter)
}

FMCounter breaks right identity. Here's a counter example:

FMCounter.unit("My").flatMap(x => FMCounter.unit(x + "Counter")) = FMCounter(MyCounter,1)
FMCounter.unit("My" + "Counter") = FMCounter(MyCounter,0) 
// not the same!

FMCounter breaks left identity. Here's a Counter example:

FMCounter.unit("MyCounter").flatMap(FMCounter.unit) = FMCounter(MyCounter,1)
FMCounter.unit("MyCounter") = FMCounter(MyCounter,0) 
// not the same!

But FMCounter is actually associative.
If you look at the definition of associativity, you call flatMap the same number of times on each side which is a pretty good indication it passes.

But in case you were looking for something more formal, here's an unconventional proof that uses scala-ish syntax. Feel free to just roll on past if this if it's not your jam.

let {FM} be the set of all monadic actions of type FMCounter
let f : A => FMCounter[B]
let g : B => FMCounter[C]

Theorem:  ∀ x ∈ {FM} x.flatMap(f).flatMap(g) == x.flatMap(a => f(a).flatMap(g))
          x                               = FMCounter[A](a: A, i:         Int)
          f(a)                            = FMCounter[B](b: B, k0:        Int)
          x.flatMap(f)                    = FMCounter[B](b: B, i+k0+1:    Int)
          g(b)                            = FMCounter[C](c: C, k1:        Int)
          x.flatMap(f).flatMap(g)         = FMCounter[C](c: C, k0+k1+2:   Int)

          h: A => FMCounter[C]            = (a: A) => f(a).flatMap(g)        
          h                               = (a: A) => {
                                                f(a) = FMCounter[B](b: B, k0:      Int)
                                                g(b) = FMCounter[C](b: C, k1:      Int)
                                                       FMCounter[C](b: C, k0+k1+1: Int)
                                            } 

          h(a)                            = FMCounter[C](b: C, k0+k1+1: Int)
          x.flatMap(a => f(a).flatMap(g)) = x.flatMap(a => h(a))
          x.flatMap(a => h(a))            = FMCounter[C](b: C, k0+k1+2: Int)

          substitution: FMCounter[C](c: C, k0+k1+2: Int) == FMCounter[C](c: C, k0+k1+2: Int)
          TRUE

Since FMCounter counts the number of times flatMap has been called, it breaks the properties which require expressions to be equal that have different numbers of flatMaps.

Because it breaks two laws, there are multiple ways to correctly write the same code that result in different flatMap counts. All that means is that it's probably not the solution we're looking for. But also that it's not a monad.

Conclusion

If you were faced with a specific problem like stringing together functions that return tuples, you really might have invented Monads yourself. Monads are simply a tool to make otherwise clunky solutions feel more natural. We use Monads all the time already so it's worth understanding why they're so good at what they do.

DEV Community: Nate May

CircleCI and Haskell

CircleCI Language Guide - Stack

Haskell-works Orb - Cabal

Custom Config

The .cabal File

CircleCI Config

Docker

checkout

Restore Cached Artifacts

Update Dependencies

Build

Build Tests

Cache Artifacts

Run Tests

Additional Gotchas

Conclusion

Why Learning Haskell Sucks

Don't Read This Monad Tutorial

The Issue of Ugliness

flatMap and map

A way inside

Customizing List

Surprise! You made a monad.

Ok but what is a monad?

The Monad Laws

Break the Monad Laws

Conclusion

The `.cabal` File