As a project grows it accumulates more and more supporting tooling. At some point this toolset can encounter growing pains familiar to any growing software projects, related to UX, maintainability, and performance.
A build system can help address these pains. A build system allows us to organize our tooling code into small build rules with explicit dependencies, improving maintainability. Aware of these dependencies the build system can choose to skip parts of the build whose dependencies didn't change, improving performance. And if our full build command is quick then developers don't need to know tons of smaller scripts, improving UX.
In this post I would like to look at one build system in particular. Shake is build system written in Haskell, and defines an API for writing build rules in Haskell too. Shake has a great API for calling external commands and shell scripts, so we can add it to an existing toolset bit by bit. Shake doesn't make assumptions about technologies used, making it a great fit for projects containing a mix of languages that no language-specific build system covers well. Lastly, Shake supports a 'shared cache' which allows a build to skip steps if they have already ran before on the same machine or another machine! Note that this cache option is off by default.
If you've always wanted to try out Haskell, Shake is a great opportunity to learn while building something useful. Its Manual/Tutorial assumes no prior Haskell knowledge and serves as a great introduction. In the rest of this post I'm going to assume familiarity with the material that tutorial covers.
It's important we're able to trust our build system. If our build system skips a part of the build it shouldn't, and as a result passes a build that should have failed, the ultimate consequence of that could be that we deploy broken code into production. That's obviously not something we want and so we will need to understand our build system code well.
One way to read Shake code is to walk through the code in the same way that Shake will, following along with its evaluation model. Shake is a complicated bit of software, and when I try to run its evaluation model in my mind I get so confused I end up with less trust of my Shake code rather than more. That might sound pretty damning but I do believe it's possible to write and read Shake code with confidence, if we keep a particular mental model in mind.
The mental model I use likens a shake codebase to a recipe book. Our shake rules are the recipes that make up the book. Each rule contains steps to create a particular file, which in our analogy is the dish we're preparing. Recipes contain ingredients and so do our shake rules, they are the dependencies of a rule.
I claim that if the recipe analogy applies to all the rules in our build system, that build system is correct. The recipes need to be honest though: They cannot leave out ingredients, and they cannot produce different dishes then the ones they advertise. To understand why this honesty is important let's look at a couple of examples of rules that aren't honest and see what breaks.
Imagine we have the following small Elm application.
ai-banana/ elm.json src/ Main.elm Banana.elm
We define the following rule to build this application.
rules :: Rules () rules = "index.js" %> \out -> do need ["elm.json", "src/Main.elm"] cmd_ "elm make --output" [out]
This rule contains a bug. It doesn't define a dependency on the
The bug won't surface the first time we run
shake index.js. If we then make a change to
Banana.elm and run
shake index.js again we'd want Shake to run the rule again and rebuild
index.js, but instead Shake will skip doing this work. Shake thinks there's nothing to do because none of the dependencies of
index.js it knows about have changed.
Imagine we have the following rules to produce a changelog for the fictitious
rules :: Rules () rules = do "node_modules/.bin/*" %> \out -> do need ["package.json"] cmd_ (Stdout out) "npm install" "CHANGELOG" %> \out -> do need ["node_modules/.bin/changelog"] cmd_ "node_modules/.bin/changelog ai-banana"
We have one rule that installs the
changelog application from
npm and another that uses it to produce a
The rule for
node_modules/.bin/* isn't honest though. It doesn't just produce executable files in
node_modules/.bin/, but also essential backing modules in other
node_modules/ subdirectories without which the executables won't run.
If we enable Shake's shared cache (which is off by default) the rule above will be buggy. The first time we run
shake CHANGELOG everything will be fine. If afterwards we remove
node_modules/ and run
shake CHANGELOG we will see a failure. Shake will avoid doing unnecessary work and retrieve
node_modules/.bin/changelog from its cache rather than running the build rule for it again.
node_modules/.bin/changelog cannot work on its own though, it will attempt to
require a bunch of other files that aren't around and fail.
Don't define a rule that produces a tiger in a cage and call it "tiger". When requesting the tiger Shake might just give you what you asked for.
We've seen there's two primary ways of withholding information from our build system: we can omit dependencies to build rules and we can define build rules that produce more files than they advertise. In both cases we can end up with bugs.
Sometimes though a small lie told to our build system can seem a necessary, or at least awfully tempting part of addressing a tricky design challenge. In other cases the way we write our Shake rules might have made it easier to be dishonest by accident. Let's take a look at a couple of these traps and see how we can avoid them.
Shake has two functions
resultHasChanged. They allow you to query whether a particular file has changed since the last time you ran a rule. These functions are neither dependencies nor build rules and so don't fit the "rules are recipes" mental model. To asses whether rules using
resultHasChanged are correct it's not enough to check the rule is complete in defining its dependencies and outputs. Instead we're back at stepping through the rule in our mind and attempting to follow Shake's evaluation model. I recommend you avoid these two functions.
Often we can redesign our rule to use
addOracleCache instead of
addOracleCache comes with boilerplate and type magic but fits the "rules as recipes" mental model well.
Consider two ways in which a Shake rule can read a file.
parseConfiguration1 :: FilePath -> Action Config parseConfiguration1 file = do contents <- liftIO $ readFile file parse contents parseConfiguration2 :: FilePath -> Action Config parseConfiguration2 file = do contents <- readFile' file parse contents
readFile comes from Haskell's
Prelude. It returns an
IO String which we can turn into an
Action String using
liftIO. Shake provides a
readFile' that directly returns an
Action String. The two
parseConfiguration functions behave differently. Shake's
readFile' will add the file read as a dependency to the rule calling this function, causing that rule to be rebuilt if the contents of the file changed. Haskell's default
readFile function will do no such thing.
readFile' is one example of a broader pattern in the functions Shake makes available to us. Some return an
Action, and these will automatically track any files they use. Others will return an
IO and won't track dependencies, putting the onus on us to check we're not overlooking a dependency.
The pattern is worth extending to helper functions we define ourselves. In the example above
parseConfiguration1 is not following the pattern because it returns an
Action but doesn't add the files it tracks as dependencies. Tracked and untracked versions of that function that do follow the pattern look like this.
parseConfiguration :: FilePath -> Action Config parseConfiguration file = do -- Easier would be to use `readFile'` here, but for the -- purposes of this example we'll pretend it doesn't exist. need [file] contents <- liftIO $ readFile file parse contents parseConfigurationIO :: FilePath -> IO Config parseConfigurationIO file = do contents <- liftIO $ readFile file parse contents
We can't entirely avoid using
liftIO because Shake doesn't come with helper functions for all possible kinds of
IO, and because there's legit usages of untracked functions that don't break our honesty principles. For example, a rule might produce and then consume a helper file. Such a file is not a dependency of the rule that created it, and adding it as a dependency could lead to weird behavior.
IO we need to take extra care to ensure we're not missing a dependency. How convenient for review then that we signpost every use of
IO with a call to
liftIO. We can help review further by adding a comment to every use of
liftIO, explaining why it's necessary and safe.
Lastly, the fewer lines of code we have running in IO, the fewer lines we have to check for missing dependencies. We can reduce the amount of lines running in
IO by pushing our
liftIO's as far down as we can. Consider the helper function below.
largeHelper :: IO () largeHelper = do ... file <- readFile "all-pianos.txt" ...
By putting the
liftIO in front of the
readFile call we could make
largeHelper return an
Action. Now a single line in
largeHelper is suspect, rather than every line in the function.
Sometimes we need to sacrifice the readability of our code improve its performance. When writing Shake code too we might see opportunities to improve its performance by compromising on "rules as recipes" honesty. Should we take these opportunities? Performance is important in a build system, it's the reason we're using a builds system in the first place!
Shake provides two great helpers that allow us to address performance concerns without compromising on "rules as recipes" honesty. They are
newCache prevents duplicate work within a single build, and
addOracleCache across several builds.
Consider the following rule, which produces a localized
index.html file based on some
rules :: Locale -> Rules () rules locale = "index.html" %> \out -> do need ["template.html"] template <- readFile' "template.html" let localized = localize locale template writeFile out localized localize :: Locale -> String -> String localize = ...
This rule has a dependency that Shake doesn't know about: the
locale argument. Should that argument change Shake won't know the rule needs to be rebuilt. One way to fix the problem would be to write a rule which produces a different output file for each locale.
rules :: Rules () rules = "*/index.html" %> \out -> do let locale = localeFromString (takeDirectory out) need ["template.html"] template <- readFile' "template.html" let localized = localize locale template writeFile out localized localeFromString :: String -> Locale localeFromString = ... localize :: Locale -> String -> String localize = ...
A build system like Shake is good at creating files. It can grow to become a central tool in our work because a lot of development tasks are to create files. For example: compiling source files into a binary or running tests to generate a report. If we make Shake responsible for all these tasks a logical next step is to try make it responsible for all other development tasks as well.
Some tasks are more about doing something than about building something, and they aren't a great fit for a build system. Examples are deploying, running code formatters, starting services like a database, or installing a development environment. These tasks don't fit the honesty criteria of our "rules as recipes" mental model.
You can embed such tasks in Shake using helpers like
historyDisable, but you're probably better of building them as separate scripts. We can still write those scripts in Haskell and use Shake library functions like
cmd_, but we stop pretending these scripts are recipes.
We can put our Shake build tasks and Haskell scripts back under one umbrella by creating a custom cli tool that provides a UI to both. Shake's functionality is available as a library, so we can totally use it as the engine for some build related tasks in a larger tool.