DEV Community: German Robayo

Season finale: Dhall documentation generator

German Robayo — Wed, 26 Aug 2020 12:45:59 +0000

This is my last post and final report as part of GSoC 2020, working on the Dhall documentation generator

Initial problem

Dhall is a relatively new language. The only way to see how to use a Dhall package was to manually inspect the code and reading the comments. The Dhall Prelude is a good example of this approach by adding a comment to its headers as shown in this example. Some package doesn't even have comments, so you'd need to actually read the code to see how to use the package.

Main contribution

As part of my GSoC project, I built dhall-docs that takes a Dhall package and generates HTML documentation by running static analysis over each file on the package. The tool has the following features:

Takes a folder with Dhall files, extracts its headers and outputs HTML that will render that documentation (#1845)
Renders the HTML with some simple CSS and JS that improves the UI/UX (#1848, #1895)
The index's dhall files section includes the type of each file (extracted from source code) (#1898)
Assert expressions are extracted and rendered in a dedicated section in the generated HTML (#1899)
A comment's syntax specification was created to correctly handle indentation before using a Markdown parser (#1929)
Renders the source code with jump-to-definition (JTD) capabilities on the following places:
- Import expressions (#1959)
- Let binding names (#1966)
- Lambda argument names (#1982)
- Record (literals and types) field names (#1991)

Besides the work outlined in the PRs of the above list, I made the following technical improvements to the dhall-docs package at the dhall-haskell repository:

Add a golden tests setup to ensure that the generated documentation is preserved through code changes (#1899)
Generate documentation for the Prelude, dhall-kubernetes, and the test demo package that the golden-test setup of dhall-docs uses.

The list of all PRs I authored on the dhall-haskell repository can be found here

...and I made some upstream changes to the Dhall Language Standard:

Add more tests for record field labels (#1013)
And changes related to how does dhall-docs manages packages (#1026, #1053)

I openly and actively discussed things related to dhall-docs behavior not only with my mentors but with the wide community via Discourse posts. The most relevant posts were:

If you want to see my messages on discourse.dhall-lang.org, go here

...and to end this section, although it's not directly related to my project, I familiarized myself with the existing codebase by taking small issues on the community bonding period. I already talked about that in this post.

All the source code can be found here. We already have a release of dhall-docs (on a GitHub release and on Hackage) but it doesn't contain all features. We're waiting for this release PR on the standard to be merged to make another release of the dhall-haskell packages, including dhall-docs.

Pending work

If we compare the work done vs. my proposal, the things that are missing are:

Syntax-highlighting on rendered source code. We had it at the beginning, but adding jump-to-definition required changing how we render source code. It is still possible to add this with the current fragments function used for JTD
Type-on-hover. We have a WIP PR that I couldn't finish, although it's in good shape.
Read comments from other places in the source-code. In this moment, dhall-docs only supports the Header comment of the file, and it would be great to add support for other places such as record fields, lambdas and function-type arguments. The most difficult part for this work is to properly preserve whitespace, but we're making progress.
Jump-to-definition on expressions that involve imports. We currently jump to another URL when the clicked expression is a relative and remote import. But things like (./Import.dhall).field.deep are not handled yet and it would be really useful. A common pattern seen on dhall packages is that they use a package.dhall file that contains every file in the package, and using the above example with it can improve navigability a lot

I'm not sad that these features weren't done. I made a couple of improvements on the Dhall parser to preserve more whitespace that dhall-docs needed for JTD and that took more weeks as expected. The best part is that the work done there can also be used to fix a long-standing issue on the dhall format command. It was a good trade-off, after all 😄

Screenshots & Demo package

The following are some screenshots of how the generated documentation looks. These were taken from the dhall-haskell CI/CD results for the dhall-docs execution on sample packages, and since the job is not updated with the latest dhall-lang commit, almost all links on relative imports are broken (dhall-docs doesn't report broken links)

The Dhall Prelude package index:

Prelude/Bool/package.dhall is essentially a sub-package inside Prelude and it re-exports all files inside Prelude/Bool

The Prelude/Bool/fold is really complete as it shows the extracted header as markdown, assertions, and jump-to-definition capabilities:

On the test demo package a good example of jump-to-definition on record fields is the "jump-to-definition with nested labels":

We can use relative imports links to navigate to another expression in the same package:

Final thoughts

I still remember that I literally yelled at my room after receiving the email from the GSoC team that my proposal was accepted, and a lot of things have changed since then. The German from that moment wouldn't believe all the things that He learned during these last 4 months, not only about Haskell but about Open Source projects.

The most important part of an Open Source project is its community: without it, a project is just a bunch of code. And it's difficult to be proactive (quoting from Gabriel, my mentor) if you're not really passionate or if you don't get any funds from it. Google's initiative on this program helps a lot of projects finish their ongoing issues, and to offer new issues for arising tools (such as Dhall, for instance).

Contributing to Open Source for the first time may be complicated, but once you get used to it it's really simple and you get a lot of benefits:

Be a member of a community
Help your favorite ecosystem grows
Improve your career in technical and non-technical aspects

Before contributing to dhall-haskell, my Haskell experiences came from university projects. I could say that those projects used ok-ish Haskell: I'm really proud of those. I didn't have somebody reviewing my code, and I felt that a lot of things could be improved and I didn't have any point of comparison.

When I cloned dhall-haskell and opened any file I knew that this was completely new to me. Things such as LANGUAGE pragmas were completely unknown. Even things like:

{-# LANGUAGE OverloadedStrings #-}

l :: Data.Text.Text
l = "string literal"

confused me a lot at the beginning (how a String is compatible with Data.Text.Text???????), though right now it's really simple!

Working properly with Functors, Applicatives, and Monads (even implementing my own!) were things that I wanted to do so many time ago, and after a couple of PRs on the repository started to feel really natural (not a pro, yet).

I started creating more data-types to make illegal states unrepresentable, a thing that I felt afraid (I over-abused Maybe and [] on my old projects) and lenses, although I didn't use them a lot for dhall-docs, stopped to scaring me.

There are even more things that I could write, but I don't want this post longer than it currently is.

Although this is the end of my work on the GSoC program, I'm pretty sure that I'll keep contributing in the short-coming future

Acknowledgments

To my mentors Gabriel, Simon and Philip: Thanks for the opportunity, for always being there and for your patience over all the project. Thanks for the guidance on the development of dhall-docs and for telling me about your experiences on Open Source.

To the Dhall community, to be proactive on their responses to my open-questions.

To Haskell.org for hosting the project and help me decide what project to choose 😉

And last but not least: To Google for providing the GSoC program. It's a really good way to improve your career and help the Open Source community grow bigger and stronger.

Thanks ❤️

Jump to definition on source code

German Robayo — Mon, 24 Aug 2020 15:35:14 +0000

Introduction

Hi there! It's been a while since my last post and a cool feature that I've been working on recently it's roughly done, and it's really cool!

The feature is jump-to-definition (I'll abbreviate that to JTD in the rest of the post), and it allows rendered source code to be more navigatable.

TL;DR

To see some examples of this feature, you can visit the following links:

Prelude.JSON.renderAs https://hydra.dhall-lang.org/build/70743/download/1/docs/JSON/renderAs.dhall.html
Every file with the JumpTo prefix on this list https://hydra.dhall-lang.org/build/70746/download/1/docs/

Jump to definition

The main purpose of using a documentation generator for your projects (in any programming language) is to make it easy for your users to learn how to use your library/framework/package/whatsoever. Simple things such as analyzing source-code comments (as dhall-docs already does) are good enough, but we can always add more features that make code discoverable and easier to read.

Jump to definition is a feature that most documentation generators tools have. Let's use haddock as an example: when haddock generates the documentation of a Haskell packages it generates HTML for:

The source-code comments that the user marked as documentation.
The actual source code of the package.

If we use the dhall package, for example:

This is the generated documentation for Dhall.Core (which re-exports Dhall.Syntax) https://hackage.haskell.org/package/dhall-1.34.0/docs/Dhall-Core.html;
This is how haddock renders the source code for the Dhall.Syntax module https://hackage.haskell.org/package/dhall-1.34.0/docs/src/Dhall.Syntax.html

If you open the rendered source code for Dhall.Syntax, you will see that:

It is rendered using syntax-highlighting
It's full of links on the data-types and function names, which suggests you that it's clickable and it allows you to share the implementation of anything inside that module. For instance, you can see here the definition of the Expr data type: the Dhall AST.

We want to add the same for dhall-docs: take a file and do static analysis to see where each name was defined, highlight it correctly when the user hovers it and jump to the definition of that name when the user clicks it.

Implementation

The Dhall AST

The Expr data-type from the dhall package contains a special constructor useful to get the source-span of a parsed expression:

data Expr s a =
    ...
    | Note s (Expr s a)
    ...

(you can see its haddock here)

The purpose of Note is to create useful parse and type-errors by underlining the problematic expression. The Dhall parser outputs an Expr Src Import (explanations of Src and Import can be found on Expr haddock) where each meaningful expression is wrapped on a Note constructor. Srcs contains metadata information about the parsed expression, such as source text and line and column numbers. For example, if you have the following expression:

1 + 2

it will be parsed to something (roughly) like this:

Note (source code info of the "1 + 2" expression)
  (NaturalPlus -- for the `+` operator
    (Note (source code info for the "1") (NaturalLit 1))
    (Note (source code info for the "1") (NaturalLit 2))
  )

Besides Note, there are other constructors that preserve extra-information about the source code. It is mostly used by the dhall format and dhall lint commands.

Code renderer

Now the question is, how can I manipulate/traverse the Expr in a way that lets me generates HTML for the following?

The declaration & use of names (variables)
Relative imports
The rest of the code, that should not be affected at all.

I first started making this simple data-type:

-- we will come to the ??? later
data NameDecl = NameDecl Src Text ???

data SourceCodeType
    = ImportExpr Dhall.Import
    | NameUse NameDecl
    | NameDeclaration NameDecl

data SourceCodeFragment = SourceCodeFragment Src SourceCodeType

You can think of each SourceCodeFragment as a subsequent section of the source code. Similar to a token (but not exactly that, though). Each one is discriminated by its SourceCodeType, which carries the information of how the code should be rendered in the HTML.

Now we need to transform an Expr Src Import to a [SourceCodeFragment] and that's where the fragments function comes:

fragments :: Expr Src Import -> [SourceCodeFragment]

The implementation of the fragments function is quite large to fit in this post (110 lines, to be exact) but what it does is run a simple context-analysis on the AST to generate SourceCodeFragment whenever a name was declared and used. It is clever enough to know when a name was not declared, and when a name was unused and showing no highlight on its HTML (i.e. not generating a SourceCodeFragment for it).

All the work is done in the infer function, declared inside fragments:

-- explanation for ??? will come up later
infer :: Dhall.Context NameDecl -> Expr Src Import -> Writer [SourceCodeFragment] ???

Dhall.Context is a context table that allows us to insert and query for names when traversing an AST. We insert names when it makes sense to (i.e. let-bindings, record fields, and lambda parameters.

All of this work allowed dhall-docs to generate jump-to-definition for the following:

Let-bindings
Lambda argument names
Record (literals and types) field names
Imports (relative and remote imports)

You can see here for a demo. The test cases are named with the feature that is being shown there.

What about the `???` on the code?

let-binding and lambda argument names share the same JTD logic:

Calculate the source span position of the binding name
Traverse the sibling expressions on both Let and Lam adding the name to the context accordingly.

However, adding support for record field is slightly different. We want to highlight the names where fields in selector-expressions were defined. In the following example we want to highlight foo when the user hovers the foo in the selector-expression a.foo:

let a = { foo = 2 }
in  a.foo

Pushing the foo label to the context isn't the right solution because handling the exact expression to remote the foo label would be complicated.

What we instead do, is to use another data-type:

data JtdInfo
    = RecordFields (Set.Set NameDecl)
    | NoInfo

??? corresponds to JtdInfo. You can think of JtdInfo as extra information recovered from the static-analysis that could aid in more precise generated JTD. Replacing ??? on the above snippets, we got:

data NameDecl = NameDecl Src Text JtdInfo

infer :: Dhall.Context NameDecl -> Expr Src Import -> Writer [SourceCodeFragment] JtdInfo

Now each NameDecl has an associated JtdInfo, and every expression returns a JtdInfo from infer. Now we can pattern match against the Field constructor to correctly setup JTD:

Field e (FieldSelection (Just Src{srcEnd=posStart}) label (Just Src{srcStart=posEnd})) -> do
  fields <- do
    dhallType <- infer context e
    case dhallType of
      NoInfo -> return mempty
      RecordFields s -> return $ Set.toList s

  let src = makeSrcForLabel posStart posEnd label
  let match (NameDecl _ l _) = l == label
  case filter match fields of
    x@(NameDecl _ _ t) : _ -> do
      Writer.tell [SourceCodeFragment src (NameUse x)]
      return t
    _ -> return NoInfo

We setup JTD only if it makes sense. Thankfully pattern matching helps us elegantly handle all cases.

To end

This was the last feature that I developed as part of my GSoC project. Keep in tune to read my last report about a summary of what the whole project and a story about all my experience through.

Thanks for reading!

Comment's format specification

German Robayo — Sun, 26 Jul 2020 06:27:02 +0000

Introduction

Hi everyone! In this post, I'll write a summary of what I did this week on my GSoC project: Define and implement a specification for dhall-docs comment's format.

Why do we need a spec?

Because of two particular reasons.

Distinguish between internal and documentation comments

Is really common on several programming languages to write comments that you may not want to render in the language documentation generator. A simple way of avoiding so is by having "markers" on comments. Haddock, for instance, requires that you use | as a marker in both single-line and block comments:

-- | Valid haddock comment

-- ignored haddock comment

{-| Valid haddock comment -}

{- Ignored haddock comment -}

Javadoc also does it, by requiring that Javadoc comments must start with /**.

For the same reason above, we need to be able to distinguish between internal and documentation comments on dhall-docs.

Indentation issues

We are going to use Markdown as a markup language for the documentation generator, specifically, We will use a particular flavor of Markdown named CommonMark. Markdown, unlike Dhall, is sensitive to indentation. The following markdown document:


first column
    4 columns later

will render "first column" in a normal paragraph, and "4 columns later" as an Indented Code Block. I invite you to give it a try.

In the beginning, since dhall-docs only supports Header comments, we used the first column of the line as a base of indentation. That means that something like this:

{-
foo
bar
baz
-}

will be rendered as 3 different paragraphs, in different lines, whereas:

{-
     foo
     bar
     baz
-}

will render all three lines in an indented-code-block. But to support other documentation comments (such as record fields), this doesn't scale well. Take this example:

        {
            {-
            should this be indented???
            -}
            foo = bar
        }

Comment's content indentation is now clear, and forcing users to write this case like this:

        {
            {-
should this be indented???
            -}
            foo = bar
        }

is awful and was never an option.

Final specification

Rewriting here the final specification is a non-sense, but you can read it here. Block-comments were heavily based on Dhall's multiline strings to make it really familiar to something already implemented on the language. You can read here the specification for multiline strings here. Single-line comments, on the other hand, was something completely new, or at least couldn't find anything similar from the Dhall's language design that could help, but I think that it will be comfortable for users.

To give you a summary:

Block-comments starts with {-| and a newline e.g.
```
{-|
foo
bar -}
```
Single-line comments can span several lines, but the first one should init with --| (note the final whitespace) and every other line should start with -- (note the two whitespaces). Also, they need to be vertically aligned e.g.
```
--| foo
--  bar
```

Implementation

I have to admit: Text-manipulation is really hard and messy for me. My first implementation of the specification was really awful and cumbersome. One of my mentors gave me this post about some tips on type-driven design and I applied them to the implementation. I heavily recommend reading that post.

I defined the following data-type:

{-# LANGUAGE DataKinds         #-}
{-# LANGUAGE KindSignatures    #-}

data CommentType = DhallDocsComment | MarkedComment | RawComment

type ListOfSingleLineComments = NonEmpty (SourcePos, Text)

data DhallComment (a :: CommentType)
    = BlockComment Text
    | SingleLineComments ListOfSingleLineComments
    deriving Show

newtype DhallDocsText = DhallDocsText Text

Note the Language extensions I'm using (disclaimer: I'm still learning Haskell and specifically the use of language extensions and some of them require some math background. I apologize if I say some nonsense. If you want to give me a term fix, please leave a comment on the post):

DataKinds allows us to extend Haskell's kind system by promoting our data-types to kinds.
KindSignatures allow the a :: CommentType syntax. In that example it means "I accept any valid value of kind CommentType i.e. DhallDocsComment, MarkedComment, RawComment.

That allows us to have these type of comments:

DhallComment RawComment
DhallComment MarkedComment
DhallComment DhallDocsComment

and you know for sure what kind of text stores each possible type. This is something I love about Haskell: the type-system itself helps you write correct programs.

After that, all left was doing some mappers between each possible DhallComment:

-- checks if a DhallComment has the `|` marker
parseMarkedComment
    :: DhallComment 'RawComment
    -> Maybe (DhallComment 'MarkedComment)

-- check that a MarkedComment is valid against the `dhall-docs` spec
parseDhallDocsComment
    :: DhallComment 'MarkedComment
    -> Either CommentParseError (DhallComment 'DhallDocsComment)

-- Manipulates the comment's text. Since the Comment
-- is a 'DhallDocsComment it should never fail
parseDhallDocsText
    :: DhallComment 'DhallDocsComment
    -> DhallDocsText

and to ensure that the implementation worked properly, the dhall-docs test setup was enhanced to add several unit test cases for this which you can see here

Examples of using this new spec

Recently a PR was opened to modify the Dhall's Prelude header files. You can see the PR here:

Use `dhall-docs` comment format for Prelude #1045

Gabriel439 posted on Jul 24, 2020

... so that the comment headers are included in the generated documentation

View on GitHub

To end

I have 1 month left to finish the project and there are some core and cool features that are missing. As soon as I finish them I'll post here so you can check out. I have to say: I'm really excited to finish this project :)

Thanks for reading!

Record keys prefix whitespace recovering

German Robayo — Sun, 26 Jul 2020 01:46:49 +0000

Introduction

Hi folks! I haven't posted here since two weeks ago, so I'd like to give you an update on the progress of my #GSoC project.

The main subject of this post is the recovering of prefix whitespace on Record keys, which will allow users to document record types and literals' keys.

First problem

Haskell's Dhall's parser treats both block and single-line comments as whitespace, which ignores since Dhall's is whitespace insensitive.

A limitation of this implementation is that we cannot fully reconstruct a Dhall file with the consumed whitespace and therefore the comments. This has been reported in some old issues on the repository. In particular, dhall format removes comments after formatting the files

dhall-format removes comments #145

psibi posted on Sep 29, 2017

Example file:

--- This is dhall file
{ foo = "jax" }

The Output it gives:

{ foo = "jax" }

View on GitHub

Due to the size of the project, some simple solutions for this have been made. The parser preserves:

The Header whitespace before the first non-whitespace character of the file. This is what dhall-docs supports at this moment.
Whitespaces in-between let-bindings components, as highlighted by this haddock comment

... but no more expressions have been handled yet because it's a little onerous to code and it introduces a lot of backward-incompatible changes.

Two weeks ago, I started to add support to Record keys prefix on both record literals and types. The whole work is on the following PR

feat(comments): support prefix comments on Record's key-value pairs. #1908

german1608 posted on Jul 10, 2020

This PR tries to add support to record field comments both on formatting and dhall-docs documentation extension. Specifically the following ones:

{
  -- before each record field
  a : Text
  -- after each record field
, ...
}

I updated the Record's Expr constructor from:

Record (Map Text (Expr s a))

Record (Map Text (RecordField s a))

where RecordField stores the last of the expression on the right side of the : and the prefix and suffix comments. Although the prefix comment is annotating the label field, the best place to save that comment is there.

However, I'm having a problem with unsafeSubExpressions, because it can't traverse the Maybe s I put on RecordField. A quick solution for that is to change those RecordField's Maybe s' to Nothing, although that means we loose that information when traversing.

I'll keep this PR as a draft to get your opinion in that matter.

View on GitHub

I made this because:

It will be another element that dhall-docs will document for
It will help to fix the issue with dhall format removing comments
To take more knowledge of the whole dhall codebase.

In the following sections, I'll describe how the task was made.

New data constructors

The Dhall's AST constructors for Record literals and types are the following:

data Expr s a
    = ...
    -- | > Record       [(k1, t1), (k2, t2)]        ~  { k1 : t1, k2 : t1 }
    | Record    (Map Text (Expr s a))
    -- | > RecordLit    [(k1, v1), (k2, v2)]        ~  { k1 = v1, k2 = v2 }
    | RecordLit (Map Text (Expr s a))
    ...

They don't contain any property that will allow us to recover whitespace. So the first thing we need to do is to modify the constructors to include them. I ended up writing the following data-type:

data RecordField s a = RecordField
    { recordFieldSrc  :: Maybe s
    , recordFieldValue :: Expr s a
    } deriving (Data, Eq, Foldable, Functor, Generic, Lift, NFData, Ord, Show, Traversable)

And modify the Expr s a constructors like this:

data Expr s a
    =
    -- | > Record [ (k1, RecordField _ t1)          ~  { k1 : t1, k2 : t1 }
    --   >        , (k2, RecordField _ t2)
    --   >        ]
    | Record    (Map Text (RecordField s a))
    -- | > RecordLit [ (k1, RecordField _ v1)       ~  { k1 = v1, k2 = v2 }
    --   >           , (k2, RecordField _ v2)
    --   >           ]
    | RecordLit (Map Text (RecordField s a))
    ...

The captured whitespace is the prefix of the label. Although the Map on the constructor maps labels to values, the less cumbersome place to save that was on the RecordField data-type. This will allow us to map the following:

{ -- my single-line before `a`
    a = 1
}

To something (roughly) like this:

RecordLit [("a", RecordField "-- my single-line before `a`" (NaturalLit 1)]

Note that we don't save suffix whitespace right after each record value because the Dhall's parser consumes them and there is no pretty way that doesn't involve modifying a lot the parser to fix that. Thankfully, places like the following:

a {- A -} : {- B -} B

will be easy to recover in the future and add value in the context of #145. I didn't add support for those yet since I feel that people won't write a lot of useful documentation on that place, and I wanted to finish this as quickly as possible to start other tasks more directly related to my project.

After modifying those records, the cumbersome part of the task came in: tracking the type-checking errors on the whole project was exhausting. The good thing is that it wasn't difficult, and it helped me learn more about the whole project, which is good :)

Parser modifications

Dhall's parser is located at the Dhall.Parser.Expressions module. It is built using megaparsec parser combinators. The relevant sections of the code are here on alternative04:

alternative04 = (do
    _openBrace

    src0 <- src whitespace
    mComma <- optional _comma

    -- `src1` corresponds to the prefix whitespace of the first key-value
    -- pair. This is done to avoid using `try` to recover the consumed
    -- whitespace when the comma is not consumed
    src1 <- case mComma of
        Nothing -> return src0
        Just _ -> src whitespace

    a <- recordTypeOrLiteral src1

    _closeBrace

    return a ) <?> "literal"

and here:

recordTypeOrLiteral firstSrc0 =
        choice
            [ emptyRecordLiteral
            , nonEmptyRecordTypeOrLiteral firstSrc0
            , emptyRecordType
            ]

On the first snippet note that we parse the prefix whitespace of the first key-value pair of the record, and pass it to the recordTypeOrLiteral combinators. That allows us to share that whitespace over all possible choices on recordTypeOrLiteral efficiently. If we didn't do that, we would recur to the try function from megaparsec that backtracks on parse failures, which increases the benchmark numbers.

By doing all this work, the prefix whitespace of the record was preserved. You can check out that by using the following example:

{ -- I'm recovered
    a = 1,

    {- me too! -}
    b = 2,
}

...on dhall haskell-syntax-tree --noted

To end

This was a hard job. I feel great that I'm contributing not only to my particular project but the whole Dhall's Haskell's implementation.

I'd like to thank my mentors Simon and Gabriel for their help with this PR not only by giving me suggestions and guidelines but on the exhausting job of reviewing that huge PR.

Thanks for reading!

Announcing the first experimental release of dhall-docs 🎉

German Robayo — Fri, 10 Jul 2020 14:15:25 +0000

Hi there! This short post is to let you know that dhall-docs was released 🎉.

The tool is quite experimental yet, but you can start to generate useful documentation.

For more information read this discourse post:
https://discourse.dhall-lang.org/t/announcing-the-first-experimental-release-of-dhall-docs/300?u=german1608

If you want to try dhall-docs on your Dhall package and you'd like to give me feedback, please participate in the discourse post.

Cheers 🥳

Navigatable breadcrumbs and technical debts

German Robayo — Thu, 09 Jul 2020 00:25:04 +0000

TL-DR

If you want to see my current progress, check out here:

https://hydra.dhall-lang.org/build/64022/download/1/docs/

dhall-docs

Hi there again! In this new post, I'm going to explain about two new features on dhall-docs:

Cleaner Haskell API;
Navigatable breadcrumbs;
Test-setup

Cleaner Haskell API

Previously, all the functions returned some IO wrapped value, and this makes tests complicated to setup.

The first thing I did before setup tests was to split all core logic behind docs generation from the IO related tasks.

Posting the whole code here is a non-sense (~436 LOC), but the API for dhall-docs package ended up like this in summary:

-- | The result of the doc-generator pure component
data GeneratedDocs a = GeneratedDocs [DocsGenWarning] a

{- `GeneratedDocs` is an instance of MonadWriter to make warning logging easier,
   omitted the instance declaration since it makes some noise in the post -}

{-| Generate all of the docs for a package. This function does all the `IO ()`
    related tasks to call `generateDocsPure`
-}
generateDocs
    :: Path Abs Dir -- ^ Input directory
    -> Path Abs Dir -- ^ Link to be created to the generated documentation
    -> Text         -- ^ Package name, used in some HTML titles
    -> IO ()

{-| Generates all the documentation of dhall package in a pure way i.e.
    without an `IO` context. This lets you generate documentation from a list of
    dhall-files without saving them to the filesystem.

    If you want the `IO` version of this function, check `generateDocs`
-}
generateDocsPure
    :: Text                    -- ^ Package name
    -> [(Path Rel File, Text)] -- ^ (Input file, contents)
    -> GeneratedDocs [(Path Rel File, Text)]

This introduced some cool benefits:

Pure code is easier to test.
From my little Haskell experience, the less you are coupled with an IO context the better.
The code is cleaner than before.

Thanks to my mentors for pushing me into this direction!

Breadcrumbs generation

The challenging part on breadcrumbs generation is to wire up the links to other super-folders correctly since they depend on the type of the HTML file you're generating.

Let's take the following breadcrumb:

my / super / breadcrumb

The last breadcrumb component is never a link since it always refers to the current page you're looking at. If you're on an index page, the page where you'll be redirected will be something like this:

# "breadcrumb" is your current page
my                 / super           / breadcrumb
"../../index.html"   "../index.html"   no link

whereas if you were on a Dhall file page, it will render the link like this:

# "breadcrumb" is your current page
my                 / super           / breadcrumb     / file.dhall
"../../index.html"   "../index.html"   "./index.html"   no-link

Get this right was cumbersome by using only lists, so I created my custom list-like ADT:

-- | ADT for handling bread crumbs. This is essentially a backward list
data Breadcrumb
    = Crumb Breadcrumb String
    | EmptyCrumb
    deriving Show

{-| Convert a relative path to a `Breadcrumb`.

>>> relPathToBreadcrumb [reldir|a/b/c|]
Crumb (Crumb (Crumb EmptyCrumb "a") "b") "c"
>>> relPathToBreadcrumb [reldir|.|]
Crumb EmptyCrumb ""
>>> relPathToBreadcrumb [relfile|c/foo.baz|]
Crumb (Crumb EmptyCrumb "c") "foo.baz"
-}
relPathToBreadcrumb :: Path Rel a -> Breadcrumb

The reason why it's a backward list is to make it easier to generate the relative links backward while mapping these breadcrumbs to Html ().

After adding this, we started noticing that the code was growing a lot and things start to complicate. We needed a way to ensure that adding a new feature doesn't break anything else. Oh yeah, we needed tests!

(before digging into the test-setup section, I'll show you some screenshot of how the breadcrumbs look)

On a Dhall's file documentation

On an index:

Test-setup

Currently, our Haskell API changes frequently on each PR, and testing each function in isolation will complicate things a little.

Remember that the input of dhall-docs is a list of dhall files with their relative paths, and the output is a list of generated HTML documents that represents the generated documentation.

We want out test-setup to be easy to maintain and that adding new test cases is as easy as adding an input file and an expected output file

A golden-tests setup is really good for this! Golden tests are structured like this

There are two types of files:
- Input files: The input of the function
- Output files: Are written by the function given an input file
- Golden files: The expected output of the function. The test-runner should compare them with the output files after execution.

You may be wondering that this is no different as the usual test-setup like:

actual <- function(input)
shouldBeEqual(actual, expected)

but when we are dealing with tests that involve accessing the file-system, this is a better approach.

Almost all tests-setup in dhall-haskell are written in tasty, which has several sub-packages that can be seen as plugins. For golden-tests, we decided to use tasty-silver.

The core of our test-setup is in the goldenVsAction function from the tasty-silver function:

goldenVsAction
  :: TestName -- ^ test name
  -> FilePath -- ^ path to the «golden» file (the file that contains correct output)
  -> IO a -- ^ action that returns a text-like value.
  -> (a -> T.Text) -- ^ Converts a value to it's textual representation.
  -> TestTree -- ^ the test verifies that the returned textual representation

Look that we don't use output files because they mess a little with the VCS. In this function, we mimic writing a file via actions.

The test-runner includes a --accept flag that updates or creates golden files in case of test-failures. This is handy to be used when we add a new feature that changes all outputs, and we want to update them in a single run; or when we add a new test-file to generate its corresponding golden file.

The whole test-setup fits under 66 lines of code, and we probably won't have the need to edit it again in the future: If we want to write a test for a new feature or bug, we add the file in the test package directory.

Before leaving

Thanks for reading! Keep in tune for my next posts!

Type on index & examples from assertions

German Robayo — Wed, 08 Jul 2020 19:05:17 +0000

Introduction

Hi there! Today I'm going to tell you some progress made in the project so far, specifically the following features:

Type of files on each index
Examples from assertions

TL-DR

If you just want to see the progress, check out here

https://hydra.dhall-lang.org/build/64022/download/1/docs/

Let's get started!

Type of a dhall file

Usually in typed languages like Dhall or Haskell, the type of any value is more useful than its name. You can get an insight into what a function does with just its type.

In the case of Haskell, this fact is exploited a lot. There is a website similar to "google" that let you search functions that match a type that you defined: http://hoogle.haskell.org/

In another project (firelink) that I was developing, there was a circumstance where I wanted to get the Just values from a list of Maybes, something like this:

f :: [Maybe a] -> [a]

I "hoogled" the type and it showed me catMaybes, which does exactly what I wanted to. Without looking up for the name, just the type.

On Dhall, we have a similar urge: types are more meaningful than the name. So a cool feature that I add was showing the type of the expression on the index page.

Type-checking a dhall expression is something that involves several steps. This can get cumbersome in this context and fact, we (my mentors and I) didn't want to do so (the whole algorithm is described here). The way we extract the type from a dhall file is by analyzing the source code.

The only kind of expressions that this feature supports are let-bindings, which in the internal AST representation are constructed using the following constructor:

-- Let (Binding _ x _  Nothing  _ r) e ~ let x     = r in e
-- Let (Binding _ x _ (Just t ) _ r) e ~ let x : t = r in e
Let (Binding s a) (Expr s a)

Dhall also supports multiple-let bindings, to write code like this:

let a = 1
let b = 2
in a + b

So what we do is check if the last expression in the binding is a variable name, and traverse backward the list of bindings to get the desired one and finally its associated type (which may be absent).

There is a slight modification to support referencing variables with their indexes, for instance:

let x : Bool = True
let x : Natural = 1
in x

... will return Natural as its type because it is the closes lexical binding, but if we change the last line to the following:

in x@1

it will return Bool. The final code for that was the following:

-- The `V` constructor is used to handle name references on
-- Dhall expressions, whereas
extractTypeIfInSource :: Expr Void Import -> Maybe (Expr Void Import)
extractTypeIfInSource expr = do
    V name index <- maybeNameInLet expr
    (Binding _ _ _ (Just (_, exprType)) _ _) <-
        getLetBindingWithIndex index $ getLetBindingsWithName name
    return exprType
    where
    -- | For an expression of the form @let x0 = y0 let x1 = y1 ... in e@
    --   where @e@ is a variable, maybeNameInLet returns the variable name.
    maybeNameInLet :: Expr Void Import -> Maybe Var
    maybeNameInLet (Var v@(V _ _)) = Just v
    maybeNameInLet (Let _ e) = maybeNameInLet e
    maybeNameInLet _ = Nothing

    getLetBindingsWithName :: Text -> [Binding Void Import]
    getLetBindingsWithName name = filter bindName $ reverse $ bindings expr
        where
        bindName (Binding _ x _ _ _ _) = x == name


    getLetBindingWithIndex :: Int -> [Binding Void Import] -> Maybe (Binding Void Import)
    getLetBindingWithIndex i bs =
        case drop i bs of
            [] -> Nothing
            binding : _ -> Just binding

When I get this right I was kind of excited because it was the first time in my life where I used Maybe as a Monad. The code works (thanks tests!) and takes into account the cases with variable indexing.

Here you can see a preview of how it looks like:

Cool right? 😎

Let's get to the next feature.

Examples from assertions

Dhall has a nice feature named assertions. This lets you add unit tests to your files and the type-checking phase will fail if any of these assertions fail.

We wanted a way to get examples of using a Dhall file expression, and the best way to do so was extracting them from the assertions body.

For instance, if we have something like this in a Dhall file:

let example0 = assert : e0
let example1 = assert : e1
...

on the generated documentation you will look e0 and e1 as code examples.

The code manipulates the syntax-tree as the extractTypeFromSource does, but it's more compact and easy:

examplesFromAssertions :: Expr Void Import -> [Expr Void Import]
examplesFromAssertions expr = Maybe.mapMaybe fromAssertion values
    where
    values :: [Expr Void Import]
    values = map value $ bindings expr

    fromAssertion :: Expr Void Import -> Maybe (Expr Void Import)
    fromAssertion (Assert e) =  Just e
    fromAssertion _ = Nothing

You can see a preview of it here:

Neat 💥

If you read this...

Thanks for reading! Keep in tune for future posts 😊.

If you liked this, please comment or react in the post ❤️

Document generation & Rendered Source Code

German Robayo — Sun, 28 Jun 2020 22:51:39 +0000

TL;DR

If you only want to see my current progress, you can visit this website to see it https://hydra.dhall-lang.org/build/63538/download/1/docs/

Motivation

What is a document generator? If you're familiar with other programming languages such as Java or Haskell, for example, you may know of tools that analyze your source code, extract useful information in your comments and display them in a a markup language such as HTML, for instance.

In haskell, that tool is haddock. It analyzes your source code, searching for comment annotations on your data-types and functions, to report them in a nice way using HTML.

For instance, the following function declaration:

-- | Function description
haddockExample
  :: a -- ^ input description
  -> b -- ^ output description

it will render something like this on the generated HTML:

The generated HTML can be then uploaded to any host to serve them. This is useful if you're a package maintainer to let your consumers know how to use your packages.

The main goal of this #GSoC project is to build a similar tool for the Dhall configuration language. I set some milestones for the project, and this post will focus on the following:

Generate some readable with a nice UI/UX from a dhall package. A Dhall package is essentially (at this moment) a folder with several Dhall files, ending with the .dhall extension.
Add rendered source code (first iteration). Haddock does it. For each haskell module, it will create:
- The HTML documentation
- A HTML rendered source code, similar to this

Documentation generator

At this moment, I developed the dhall-docs executable. That takes the following flags:

--input, which is a relative or absolute path to the Dhall package.
--output-link, which is a symlink (defaulting to ./docs) to the generated documentation.
--package-name, which is the actual package-name used in HTML titles and in the place where generated documentation are actually saved.

The tool will traverse the whole --input directory in a recursive way, searching for all the files that ends in .dhall and parse them. If a dhall file fails to parse as a Dhall expression, it won't be included.

Structure of the generated documentation

On each directory on the generated documentation, an index.html file is generated listing the subpackages (the directories in it) and the exported dhall files in that level.

If we visit a .dhall file, we can see something like this:

The following is the actual source code that the tool took as its part of its input:

{-
`subtract m n` computes `n - m`, truncating to `0` if `m > n`
-}
let subtract
    : Natural → Natural → Natural
    = Natural/subtract

let example0 = assert : subtract 1 2 ≡ 1

let example1 = assert : subtract 1 1 ≡ 0

let example2 = assert : subtract 2 1 ≡ 0

let property0 = λ(n : Natural) → assert : subtract 0 n ≡ n

let property1 = λ(n : Natural) → assert : subtract n 0 ≡ 0

let property2 = λ(n : Natural) → assert : subtract n n ≡ 0

in  subtract

As you would notice, the Documentation header on the generated HTML corresponds to the Header comment, and the actual source code is the rest of the file. The header is written in Markdown, dhall-docs uses mmark as markdown parser and preprocessor.

This is the first iteration of the work, I have plans on expanding the places where annotation comments can go,
like record type labels, for example.

Here is the list of PRs involved on this task:

dhall-haskell#1833 introduced the repository skeleton
dhall-haskell#1845 first attempt to generate this documentation without any css, parsing the header without using a markdown pre-processor.
dhall-haskell#1848 improved the css rules
dhall-haskell#1863 parsed the header markdown contents and rendered them on the html page
dhall-haskell#1871 added a small ci/cd configuration to generate sample documentation. you can see it here. This was my hardest task since I didn't knew any of nix, feels good to actually have accomplish it.
dhall-haskell#1876 stored the generated documentation at $XDG_DATA_HOME/dhall-docs following the XDG specification. Documentation is stored at $XDG_DATA_HOME/dhall-docs/${SHA256_OF_DOCS}-${PACKAGE-NAME} this makes it content-addressable.

All of the work was really ad-hoc, so I won't add any implementation details: you can see them on the PRs. Next section was way more interesting to implement, so please keep reading :)

Rendered Source Code (first iteration)

On the previous section I showed up a first iteration on rendered source code. There were several ways of doing this task, but the thing that I was almost about to start to do was to traverse the Dhall AST, generating Html (). In FP terms, I should create a catamorphism. In non-FP terms, I should create a mapper.

But this was going to involve a lot of lines of code, and actually some repetition of what the Dhall.Pretty module of the dhall package does i.e. define formatting rules for the AST elements and tokens.

The Dhall.Pretty module used under the hood the prettyprinter, its core consists of the following functions and ADT:

data Ann
  = Keyword     -- ^ Used for syntactic keywords
  | Syntax      -- ^ Syntax punctuation such as commas, parenthesis, and braces
  | Label       -- ^ Record labels
  | Literal     -- ^ Literals such as integers and strings
  | Builtin     -- ^ Builtin types and values
  | Operator    -- ^ Operators
  deriving Show

-- Create a `Doc Ann` from a dhall expression
-- annotating elements using our syntatic rules
prettyExpr :: Pretty a => Expr s a -> Doc Ann

-- SimpleDocStream can be later rendered as `Text` on
-- a terminal
layout :: Doc ann -> Pretty.SimpleDocStream ann

This module contained basically all of what I have to do, and repeating code is bad! The prettyprinter says on its package description:

A prettyprinter/text rendering engine. Easy to use, well-documented, ANSI terminal backend exists, HTML backend is trivial to implement, no name clashes, Text-based, extensible.

so I thought: "man, there should be a way to generate Html () from a dhall expression using this module".

And there was a way. On the package documentation, they recommend using SimpleDocTree instead of SimpleDocStream to render HTML-like output, and the package itself exports an utility to do the conversion: treeForm. Traversing the SimpleDocTree ADT made all this work possible in the following function:

import Lucid
import Dhall.Pretty (Ann (..))

import qualified Data.Text.Prettyprint.Doc.Render.Util.SimpleDocTree as Pretty
import qualified Dhall.Pretty

exprToHtml :: Expr Src Import -> Html ()
exprToHtml expr = renderTree prettyTree
  where
    prettyTree = Pretty.treeForm
        $ Dhall.Pretty.layout
        $ Dhall.Pretty.prettyExpr expr

    textSpaces :: Int -> Text
    textSpaces n = Data.Text.replicate n (Data.Text.singleton ' ')

    renderTree :: Pretty.SimpleDocTree Ann -> Html ()
    renderTree sds = case sds of
        Pretty.STEmpty -> return ()
        Pretty.STChar c -> toHtml $ Data.Text.singleton c
        Pretty.STText _ t -> toHtml t
        Pretty.STLine i -> br_ [] >> toHtml (textSpaces i)
        Pretty.STAnn ann content -> encloseInTagFor ann (renderTree content)
        Pretty.STConcat contents -> foldMap renderTree contents

    encloseInTagFor :: Ann -> Html () -> Html ()
    encloseInTagFor ann = span_ [class_ classForAnn]
      where
        classForAnn = "dhall-" <> case ann of
            Keyword -> "keyword"
            -- ommited for brevity

This is similar to the first option: transform the Dhall AST (Expr s e) to Html (), the difference is that we don't have to worry about the types of syntactical elements: that logic can be kept on the Dhall.Pretty module, and this function only creates the Html () from it.

The PR that introduced that change is dhall-haskell#1892, and we can see how many additions/deletions that change involved:

Neat! A lot of value in less than 160 lines of code.

The things I learnt along the way

Of course, I've improved my haskell, specifically the package ecosystem. One of the things that overwhelmed (and sometimes annoyed me) is how package versions are resolved, and since we have to ensure our project works on several GHC versions with several package versions, we have to be really sure about the version of a package that we are adding.

This was difficult on the first two weeks, since I had to do some little research about packages to render HTML and parse markdown, and fight against the ci/cd pipeline when a version error occurred.

Thankfully, now I understand better how it works, and in the future of the project I don't think I'll add more packages, but now I'm sure how to tackle that kind of issues.

Another thing that I've learned a little on the project was Nix. In short words, its a functional package manager. Fun fact: I see that a lot of people that enters the functional programming world tends to use only tools that uses that paradigm. Everytime I searched something about Nix, it was using a haskell project.

If you made it this far

Thanks for reading!

Prelude

German Robayo — Wed, 10 Jun 2020 23:33:58 +0000

Prelude

GSoC application

First things first: I really like Haskell. I started learning the language a year ago~ on university courses about compiler design and I wanted more real-life experience with the language.

I heard of GSoC from a friend last year and decided to apply. Of course, I wanted to go with a haskell project. The list of project ideas provided by the haskell commitee looked really challenging at the begginning:

The hard ideas were really hard not only because it required an advanced understanding on the language but also deeper understanding of hard CS subjects and contributing on some of the biggest projects of the whole haskell ecosystem like QuickCheck, hasktorch and even the compiler itself ghc
There were some ideas on project that I didn't knew and didn't understand what its purpose was

After reading digesting the whole list, I decided to look after the Dhall Documentation Generator project. I didn't knew anything about dhall before reading it on that list, but I took a look at the language and decided to give it a try and write my proposal. After contacting the mentors, who were (and are) really friendly and brainstorming a little about the goal of the project, I submited my proposal to GSoC.

I got selected!

On May 4 news arrived to my email: my proposal was selected! Immediately, my mentors contacted me congratulating me and they give me an onboard with the project details.

I cloned the repository where I'm going to be developing my project, and try to read and understand the existing code.

The repository is really huge and well documented, the good thing about haskell is that by reading a function's type definition you get an idea of what the function actually does.

But that wasn't enough to get a deep understanding of the project. The first thing I like to do before starting working on a new project either on an academic or a work context is to take some small existing issues and solve them. I looked at the issue tracker of the repository to see what I was able to do without a deep understangind of the project and submitted (and successfully merged 🎉) the following PRs:

What's next?

After investing enough time in understanding not just the haskell code but also how the project is structured, I decided to start earlier my project. I hope that project goes well in its development.

Thanks for reading!

GSoC 2020: Dhall Documentation Generator

German Robayo — Wed, 10 Jun 2020 23:30:49 +0000

Hi everyone!

This is my very first post here and I'd like to start with something really great that happened recently.

I've submitted a proposal for a GSoC and got accepted! My concrete proposal summary can be found here. In summary, I'm going to develop a Dhall Documentation Generator as this post title suggests. The milestones of this project are:

Generate a simple HTML folder with the documentation of a dhall package;
Render the dhall source code as haddock (the documentation generator for haskell) does;
Jump to definition on both documentation and source code;
Type on hover on both documentation and source code

I'll try to write several posts each time I accomplish a milestone of a project, so I might end up writing at least 4 posts. This post will contain a "table of contents" for all the development of the project.

If you don't know what the dhall configuration language is, a good place to learn is its website

DEV Community: German Robayo

Season finale: Dhall documentation generator

Initial problem

Main contribution

Pending work

Screenshots & Demo package

Final thoughts

Acknowledgments

Jump to definition on source code

Introduction

TL;DR

Jump to definition

Implementation

The Dhall AST

Code renderer

What about the ??? on the code?

To end

Comment's format specification

Introduction

Why do we need a spec?

Distinguish between internal and documentation comments

Indentation issues

Final specification

Implementation

Examples of using this new spec

Use `dhall-docs` comment format for Prelude #1045

To end

Record keys prefix whitespace recovering

Introduction

First problem

dhall-format removes comments #145

feat(comments): support prefix comments on Record's key-value pairs. #1908

New data constructors

Parser modifications

To end

Announcing the first experimental release of dhall-docs 🎉

Navigatable breadcrumbs and technical debts

TL-DR

dhall-docs

Cleaner Haskell API

Breadcrumbs generation

Test-setup

Before leaving

Type on index & examples from assertions

Introduction

TL-DR

Type of a dhall file

Examples from assertions

If you read this...

Document generation & Rendered Source Code

TL;DR

Motivation

Documentation generator

Structure of the generated documentation

Rendered Source Code (first iteration)

The things I learnt along the way

If you made it this far

Prelude

Prelude

GSoC application

I got selected!

What's next?

GSoC 2020: Dhall Documentation Generator

What about the `???` on the code?