Navigatable breadcrumbs and technical debts

TL-DR

If you want to see my current progress, check out here:

https://hydra.dhall-lang.org/build/64022/download/1/docs/

dhall-docs

Hi there again! In this new post, I'm going to explain about two new features on dhall-docs:

• Cleaner Haskell API;
• Navigatable breadcrumbs;
• Test-setup

Cleaner Haskell API

Previously, all the functions returned some IO wrapped value, and this makes tests complicated to setup.

The first thing I did before setup tests was to split all core logic behind docs generation from the IO related tasks.

Posting the whole code here is a non-sense (~436 LOC), but the API for dhall-docs package ended up like this in summary:

-- | The result of the doc-generator pure component
data GeneratedDocs a = GeneratedDocs [DocsGenWarning] a

{- `GeneratedDocs` is an instance of MonadWriter to make warning logging easier,
   omitted the instance declaration since it makes some noise in the post -}

{-| Generate all of the docs for a package. This function does all the `IO ()`
    related tasks to call `generateDocsPure`
-}
generateDocs
    :: Path Abs Dir -- ^ Input directory
    -> Path Abs Dir -- ^ Link to be created to the generated documentation
    -> Text         -- ^ Package name, used in some HTML titles
    -> IO ()

{-| Generates all the documentation of dhall package in a pure way i.e.
    without an `IO` context. This lets you generate documentation from a list of
    dhall-files without saving them to the filesystem.

    If you want the `IO` version of this function, check `generateDocs`
-}
generateDocsPure
    :: Text                    -- ^ Package name
    -> [(Path Rel File, Text)] -- ^ (Input file, contents)
    -> GeneratedDocs [(Path Rel File, Text)]

This introduced some cool benefits:

• Pure code is easier to test.
• From my little Haskell experience, the less you are coupled with an IO context the better.
• The code is cleaner than before.

Thanks to my mentors for pushing me into this direction!

Breadcrumbs generation

The challenging part on breadcrumbs generation is to wire up the links to other super-folders correctly since they depend on the type of the HTML file you're generating.

Let's take the following breadcrumb:

my / super / breadcrumb

The last breadcrumb component is never a link since it always refers to the current page you're looking at. If you're on an index page, the page where you'll be redirected will be something like this:

# "breadcrumb" is your current page
my                 / super           / breadcrumb
"../../index.html"   "../index.html"   no link 

whereas if you were on a Dhall file page, it will render the link like this:

# "breadcrumb" is your current page
my                 / super           / breadcrumb     / file.dhall
"../../index.html"   "../index.html"   "./index.html"   no-link

Get this right was cumbersome by using only lists, so I created my custom list-like ADT:

-- | ADT for handling bread crumbs. This is essentially a backward list
data Breadcrumb
    = Crumb Breadcrumb String
    | EmptyCrumb
    deriving Show

{-| Convert a relative path to a `Breadcrumb`.

>>> relPathToBreadcrumb [reldir|a/b/c|]
Crumb (Crumb (Crumb EmptyCrumb "a") "b") "c"
>>> relPathToBreadcrumb [reldir|.|]
Crumb EmptyCrumb ""
>>> relPathToBreadcrumb [relfile|c/foo.baz|]
Crumb (Crumb EmptyCrumb "c") "foo.baz"
-}
relPathToBreadcrumb :: Path Rel a -> Breadcrumb

The reason why it's a backward list is to make it easier to generate the relative links backward while mapping these breadcrumbs to Html ().

After adding this, we started noticing that the code was growing a lot and things start to complicate. We needed a way to ensure that adding a new feature doesn't break anything else. Oh yeah, we needed tests!

(before digging into the test-setup section, I'll show you some screenshot of how the breadcrumbs look)

On a Dhall's file documentation

On an index:

Test-setup

Currently, our Haskell API changes frequently on each PR, and testing each function in isolation will complicate things a little.

Remember that the input of dhall-docs is a list of dhall files with their relative paths, and the output is a list of generated HTML documents that represents the generated documentation.

We want out test-setup to be easy to maintain and that adding new test cases is as easy as adding an input file and an expected output file

A golden-tests setup is really good for this! Golden tests are structured like this

• There are two types of files:
  • Input files: The input of the function
  • Output files: Are written by the function given an input file
  • Golden files: The expected output of the function. The test-runner should compare them with the output files after execution.

You may be wondering that this is no different as the usual test-setup like:

actual <- function(input)
shouldBeEqual(actual, expected)

but when we are dealing with tests that involve accessing the file-system, this is a better approach.

Almost all tests-setup in dhall-haskell are written in tasty, which has several sub-packages that can be seen as plugins. For golden-tests, we decided to use tasty-silver.

The core of our test-setup is in the goldenVsAction function from the tasty-silver function:

goldenVsAction
  :: TestName -- ^ test name
  -> FilePath -- ^ path to the «golden» file (the file that contains correct output)
  -> IO a -- ^ action that returns a text-like value.
  -> (a -> T.Text) -- ^ Converts a value to it's textual representation.
  -> TestTree -- ^ the test verifies that the returned textual representation

Look that we don't use output files because they mess a little with the VCS. In this function, we mimic writing a file via actions.

The test-runner includes a --accept flag that updates or creates golden files in case of test-failures. This is handy to be used when we add a new feature that changes all outputs, and we want to update them in a single run; or when we add a new test-file to generate its corresponding golden file.

The whole test-setup fits under 66 lines of code, and we probably won't have the need to edit it again in the future: If we want to write a test for a new feature or bug, we add the file in the test package directory.

Before leaving

Thanks for reading! Keep in tune for my next posts!