Season finale: Dhall documentation generator

This is my last post and final report as part of GSoC 2020, working on the Dhall documentation generator

Initial problem

Dhall is a relatively new language. The only way to see how to use a Dhall package was to manually inspect the code and reading the comments. The Dhall Prelude is a good example of this approach by adding a comment to its headers as shown in this example. Some package doesn't even have comments, so you'd need to actually read the code to see how to use the package.

Main contribution

As part of my GSoC project, I built dhall-docs that takes a Dhall package and generates HTML documentation by running static analysis over each file on the package. The tool has the following features:

• Takes a folder with Dhall files, extracts its headers and outputs HTML that will render that documentation (#1845)
• Renders the HTML with some simple CSS and JS that improves the UI/UX (#1848, #1895)
• The index's dhall files section includes the type of each file (extracted from source code) (#1898)
• Assert expressions are extracted and rendered in a dedicated section in the generated HTML (#1899)
• A comment's syntax specification was created to correctly handle indentation before using a Markdown parser (#1929)
• Renders the source code with jump-to-definition (JTD) capabilities on the following places:
  • Import expressions (#1959)
  • Let binding names (#1966)
  • Lambda argument names (#1982)
  • Record (literals and types) field names (#1991)

Besides the work outlined in the PRs of the above list, I made the following technical improvements to the dhall-docs package at the dhall-haskell repository:

• Add a golden tests setup to ensure that the generated documentation is preserved through code changes (#1899)
• Generate documentation for the Prelude, dhall-kubernetes, and the test demo package that the golden-test setup of dhall-docs uses.

The list of all PRs I authored on the dhall-haskell repository can be found here

...and I made some upstream changes to the Dhall Language Standard:

• Add more tests for record field labels (#1013)
• And changes related to how does dhall-docs manages packages (#1026, #1053)

I openly and actively discussed things related to dhall-docs behavior not only with my mentors but with the wide community via Discourse posts. The most relevant posts were:

• The first discussion about the dhall-docs goals and my introduction to the community
• The announcement of the first release of dhall-docs

If you want to see my messages on discourse.dhall-lang.org, go here

...and to end this section, although it's not directly related to my project, I familiarized myself with the existing codebase by taking small issues on the community bonding period. I already talked about that in this post.

All the source code can be found here. We already have a release of dhall-docs (on a GitHub release and on Hackage) but it doesn't contain all features. We're waiting for this release PR on the standard to be merged to make another release of the dhall-haskell packages, including dhall-docs.

Pending work

If we compare the work done vs. my proposal, the things that are missing are:

• Syntax-highlighting on rendered source code. We had it at the beginning, but adding jump-to-definition required changing how we render source code. It is still possible to add this with the current fragments function used for JTD
• Type-on-hover. We have a WIP PR that I couldn't finish, although it's in good shape.
• Read comments from other places in the source-code. In this moment, dhall-docs only supports the Header comment of the file, and it would be great to add support for other places such as record fields, lambdas and function-type arguments. The most difficult part for this work is to properly preserve whitespace, but we're making progress.
• Jump-to-definition on expressions that involve imports. We currently jump to another URL when the clicked expression is a relative and remote import. But things like (./Import.dhall).field.deep are not handled yet and it would be really useful. A common pattern seen on dhall packages is that they use a package.dhall file that contains every file in the package, and using the above example with it can improve navigability a lot

I'm not sad that these features weren't done. I made a couple of improvements on the Dhall parser to preserve more whitespace that dhall-docs needed for JTD and that took more weeks as expected. The best part is that the work done there can also be used to fix a long-standing issue on the dhall format command. It was a good trade-off, after all 😄

Screenshots & Demo package

The following are some screenshots of how the generated documentation looks. These were taken from the dhall-haskell CI/CD results for the dhall-docs execution on sample packages, and since the job is not updated with the latest dhall-lang commit, almost all links on relative imports are broken (dhall-docs doesn't report broken links)

The Dhall Prelude package index:

Prelude/Bool/package.dhall is essentially a sub-package inside Prelude and it re-exports all files inside Prelude/Bool

The Prelude/Bool/fold is really complete as it shows the extracted header as markdown, assertions, and jump-to-definition capabilities:

On the test demo package a good example of jump-to-definition on record fields is the "jump-to-definition with nested labels":

We can use relative imports links to navigate to another expression in the same package:

Final thoughts

I still remember that I literally yelled at my room after receiving the email from the GSoC team that my proposal was accepted, and a lot of things have changed since then. The German from that moment wouldn't believe all the things that He learned during these last 4 months, not only about Haskell but about Open Source projects.

The most important part of an Open Source project is its community: without it, a project is just a bunch of code. And it's difficult to be proactive (quoting from Gabriel, my mentor) if you're not really passionate or if you don't get any funds from it. Google's initiative on this program helps a lot of projects finish their ongoing issues, and to offer new issues for arising tools (such as Dhall, for instance).

Contributing to Open Source for the first time may be complicated, but once you get used to it it's really simple and you get a lot of benefits:

• Be a member of a community
• Help your favorite ecosystem grows
• Improve your career in technical and non-technical aspects

Before contributing to dhall-haskell, my Haskell experiences came from university projects. I could say that those projects used ok-ish Haskell: I'm really proud of those. I didn't have somebody reviewing my code, and I felt that a lot of things could be improved and I didn't have any point of comparison.

When I cloned dhall-haskell and opened any file I knew that this was completely new to me. Things such as LANGUAGE pragmas were completely unknown. Even things like:

{-# LANGUAGE OverloadedStrings #-}

l :: Data.Text.Text
l = "string literal"

confused me a lot at the beginning (how a String is compatible with Data.Text.Text???????), though right now it's really simple!

Working properly with Functors, Applicatives, and Monads (even implementing my own!) were things that I wanted to do so many time ago, and after a couple of PRs on the repository started to feel really natural (not a pro, yet).

I started creating more data-types to make illegal states unrepresentable, a thing that I felt afraid (I over-abused Maybe and [] on my old projects) and lenses, although I didn't use them a lot for dhall-docs, stopped to scaring me.

There are even more things that I could write, but I don't want this post longer than it currently is.

Although this is the end of my work on the GSoC program, I'm pretty sure that I'll keep contributing in the short-coming future

Acknowledgments

To my mentors Gabriel, Simon and Philip: Thanks for the opportunity, for always being there and for your patience over all the project. Thanks for the guidance on the development of dhall-docs and for telling me about your experiences on Open Source.

To the Dhall community, to be proactive on their responses to my open-questions.

To Haskell.org for hosting the project and help me decide what project to choose 😉

And last but not least: To Google for providing the GSoC program. It's a really good way to improve your career and help the Open Source community grow bigger and stronger.

Thanks ❤️