DEV Community: Drew Knab

Writing Custom Loaders and Generators

Drew Knab — Sun, 23 Aug 2020 03:35:42 +0000

What happened last time?

Last time we had an introduction to Fornax loaders and generators. Today we're going to build a custom loader and generator using an external REST API.

Before We Begin

To facilitate working with data from APIs or sources outside of markdown we're going to bring in a library. Because Fornax is, at heart, a small collection of F# scripts consumed by a library, we won't be dealing with a package manager. I went to FSharp.Data and downloaded the precompiled binaries and dropped FSharp.Data.dll and FSharp.Data.DesignTime.dll into the _lib folder.

FSharp.Data is a collection of parsers and type providers that simplify common data routines and allows a more F# centric, functional first, way than if we were to use built in .NET Framework or C# libraries.

The API we're going to use is Cat Facts because it's free, open, and is a collection of facts about cats.

Custom Cat Facts Loader

We'll make a new file in the loaders folder called catfacts.fsx.

At the top of the file we'll bring in our assembly references:

    #r "../_lib/Fornax.Core.dll"
    #r "../_lib/FSharp.Data.dll"
    open FSharp.Data

That's it, now we can use FSharp.Data to handle our HTTP request and JSON parsing.

We'll declare a JsonProvider with the basic shape of the response from the Cat Facts API and the CatFacts type that we'll be using internally.

type CatFactsTemplate = JsonProvider<"""
{
    "all":
    [
        {
            "_id": "string",
            "text": "string"
        }
    ]
}""">

type CatFacts =
    { Id : string
      Text : string }

We could make both of these type definitions more elaborate. We get a lot more information out of the Cat Facts API, but this is enough for now.

To explain, CatFactsTemplate uses JsonProvider out of FSharp.Data. When we go to parse the JSON from the HTTP request and translates it into an F# object we can use.

Now that our data types are out of the way, let's dig into the loader function.

let loader (projectRoot: string) (siteContent: SiteContents) =
    let httpRequest = Http.RequestString("https://cat-fact.herokuapp.com/facts")
    let jsonResponse = CatFactsTemplate.Parse(httpRequest).All

    let facts = seq {
        for record in jsonResponse do
            { Id = record.Id ; Text = record.Text }
    }

    facts
    |> Seq.take 10
    |> Seq.iter (fun f -> siteContent.Add f)

    siteContent

Taken in steps we can see that we make an HTTP request to the API and pull back a list of facts. FSharp.Data handwaves away a lot of the details that we might have had to do if we used WebClient out of .NET Framework.

After that, we call CatFactsTemplate.Parse to take the JSON body and convert it into an F# object. We tack All on the end so we have only the Array of objects with Id and Text.

Under the hood, the Array we get out of the parse doesn't actually implement IEnumberable<'T> so we can't iterate over it by piping it into Seq.xyz. Instead we'll create an IEnumerable<'T> ourselves with:

let facts = seq {
    for record in jsonResponse do
        { Id = record.Id ; Text = record.Text }
}

This creates a sequence, which does implement IEnumberable<'T>, by looping through jsonResponse and returning our CatFacts type with the data defined in CatFactsTemplate.

facts
|> Seq.take 10
|> Seq.iter (fun f -> siteContent.Add f)

siteContent

Now that we have our sequence we can pipe that into Seq.take to cut down on the objects we're going to add into siteContent. Then we just Seq.iter to loop through our ten objects and add them to site content so we can generate HTML from them.

That's it, we now have a custom loader that consumes an external API.

Custom Cat Facts Generator

Now that we have all of our Cat Facts data loaded into SiteContents, let's bring it out into a generator. We'll make a new file called catfacts.fsx in the generators folder.

Then we'll whip up something that looks like this:

#r "../_lib/Fornax.Core.dll"
#load "layout.fsx"

open Html

let catfactCard (fact : Catfactsloader.CatFacts) =
    div [Class "card article"] [
        div [Class "card-content"] [
            div [Class "media-content has-text-centered"] [
                !! fact.Text
            ]
        ]
    ]

let generate' (ctx : SiteContents) =
    let facts =
        ctx.TryGetValues<Catfactsloader.CatFacts>()
        |> Option.defaultValue Seq.empty
        |> Seq.toList
        |> List.map (catfactCard)

    Layout.layout ctx "Home" [
        div [Class "container"] [
            section [Class "articles"] [
                div [Class "column is-8 is-offset-2"] facts
            ]
        ]
    ]

let generate (ctx : SiteContents) (projectRoot: string) (page: string) =
    generate' ctx
    |> Layout.render ctx

We pull in the files we need at the top and open the module we'll be using, in this case HTML, to handle the Domain Specific Language (DSL) that we talked about in the last article.

let generate (ctx : SiteContents) (projectRoot: string) (page: string) =
    generate' ctx
    |> Layout.render ctx

Like we also talked about last time, generate gets called first by Fornax at build time. It calls another function called generate' and pipes the resulting HtmlElement content into the Layout.render function. Render then parses the template into a string for the resulting HTML document.

let generate' (ctx : SiteContents) =
    let facts =
        ctx.TryGetValues<Catfactsloader.CatFacts>()
        |> Option.defaultValue Seq.empty
        |> Seq.toList
        |> List.map (catfactCard)

    Layout.layout ctx "Home" [
        div [Class "container"] [
            section [Class "articles"] [
                div [Class "column is-8 is-offset-2"] facts
            ]
        ]
    ]

First, we try get to our CatFacts out of SiteContent with ctx.TryGetValues(). Then we set a default value if we wind up not having records in SiteContents. Next, we turn the Sequence into a List. Finally, we map over the List with the catfactCard function to return a List.

let catfactCard (fact : Catfactsloader.CatFacts) =
    div [Class "card article"] [
        div [Class "card-content"] [
            div [Class "media-content has-text-centered"] [
                !! fact.Text
            ]
        ]
    ]

Here we see catfactCard is just a function that takes a CatFact and returns an HtmlElement with CatFact.Text in a div. With that, we should have a working generator ready to build some content.

Final Steps

The last thing we have to do is register the generator in config.fsx in the root of our project.

let config = {
    Generators = [
        {Script = "less.fsx"; Trigger = OnFileExt ".less"; OutputFile = ChangeExtension "css" }
        {Script = "sass.fsx"; Trigger = OnFileExt ".scss"; OutputFile = ChangeExtension "css" }
        {Script = "post.fsx"; Trigger = OnFilePredicate postPredicate; OutputFile = ChangeExtension "html" }
        {Script = "staticfile.fsx"; Trigger = OnFilePredicate staticPredicate; OutputFile = SameFileName }
        {Script = "index.fsx"; Trigger = Once; OutputFile = NewFileName "index.html" }
        {Script = "about.fsx"; Trigger = Once; OutputFile = NewFileName "about.html" }
        {Script = "contact.fsx"; Trigger = Once; OutputFile = NewFileName "contact.html" }
    ]
}

This array of Generators determines how Fornax treats the generators we make. In this case we want to generate the file once by adding the following entry to the collection.

{Script = "catfacts.fsx"; Trigger = Once; OutputFile = NewFileName "catfacts.html" }

Script is the generator script we're using and Trigger defines how to handle the generation of the HTML file. We are choosing the built in Fornax value of Once but we could define our own logic for how to handle file generation. Then we have the OutputFile and we're going to call it catfacts.html.

With that, we can fornax watch and visit localhost:8080/catfacts.html to see our result.

Now that we've created a very simple loader and generator to consume an API we could expand that out to any arbitrary data source. We can see that Fornax gives us a very simple method of pulling in and transforming that data. Moving forward we could, for instance, bring in Tweets, dev.to interactions, CSV data, or arbitrary database connections. Any data we can manipulate in F# can be turned into static content.

Next time I think we'll look into some of the features of Fornax and what it's lacking from other static site generators.

First Look at Loaders and Generators in Fornax

Drew Knab — Tue, 07 Jul 2020 03:07:37 +0000

What happened last time?

Last time we had a brief introduction to Fornax (a static site generator written in F#). We went over a brief introduction of getting up and running as well as an overview of the files in a Fornax project. This time we're digging into a nuts and bolts by covering loaders and generators.

Loaders

In order to talk about loaders we have to take a brief detour to talk about SiteContents. In Fornax, SiteContents is a type defined in the Fornax.Core.dll that can hold any arbitrary number of arbitrary lists of arbitrary objects. Normally, this is a collection of markdown files that are parsed into a collection of Post objects, but that's just the beginning.

Now that we have that out of the way, lets have a look at a loader from the project we made last time. Go ahead and take a peek at postloader.fsx. There's quite a lot to take in all at once, so let's break it down.

type Post = {
    file: string
    link: string
    title: string option
    author: string option
    published: System.DateTime option
    tags: string list
    content: string
    summary: string
}

We declare the post type. So that further down in our program we can add a collection of that type to SiteContents. F# has a fairly robust type system and there's a number of improvements we could make to ensure data integrity, but we'll leave it as it is. We'll jump down to the bottom of the file and take a look at the loader function.

let loader (projectRoot: string) (siteContent: SiteContents) =
    let postsPath = Path.Combine(projectRoot, contentDir)

    Directory.GetFiles postsPath
    |> Array.filter (fun n -> n.EndsWith ".md")
    |> Array.map loadFile
    |> Array.iter siteContent.Add

    siteContent

We can see here that we harness .NET Framework to use Directory.GetFiles from the postsPath that we declare above it. If you're new to F#, the syntax may be a little confusing, but conceptually it's very similar to chaining array methods in JavaScript.

With the collection of files we return from Directory.GetFiles, we Array.filter the collection and return a new collection of only .md files. Then we iterate over that collection with Array.Map and loadFile function which we'll look at in more detail later. After maping the collection we finally Array.iter and add each object in the collection to siteContent and return siteContent after our additions.

let loadFile n =
    let text = File.ReadAllText n

    let config = getConfig text
    let summary, content = getContent text
    let fileName = buildFileName n
    let file = Path.Combine(contentDir, fileName ".md").Replace("\\", "/")
    let link = "/" + Path.Combine(contentDir, fileName ".html").Replace("\\", "/")

    let filter = filterConfig config
    let title = filter "title" |> Option.map trimString
    let author = filter "author" |> Option.map trimString
    let published =
        filter "published" 
        |> Option.map (trimString >> System.DateTime.Parse)

    let tags =
        let tagsOpt =
            filter "tags"
            |> Option.map (trimString >> fun n -> n.Split ',' |> Array.toList)
            defaultArg tagsOpt []

    { file = file
      link = link
      title = title
      author = author
      published = published
      tags = tags
      content = content
      summary = summary }

The function loadFile does a lot of things I'll leave it to you to figure out the process of reading and parsing a given markdown file as passed in from the loader function. The thing I'd like to note, however, is that each time we iterate through this function we return an object that matches the shape of the Post type that exists at the top of the file.

Now that we've loaded our posts into SiteContents we can take a look at what generators do with that content.

Generators

Once we've churned through that loader and have our contents in memory we can generate some front end content. Let's have a look inside the post.fsx inside the generators folder.

This file is a lot shorter than postloader. Let's start at the bottom.

let generate (ctx : SiteContents) (projectRoot: string) (page: string) =
    generate' ctx page
    |> Layout.render ctx

Immediately we call generate' and pass in SiteContents and page then we pipe the value returned from that function to Layout.render. Layout.render is outside the scope of this article, but it lives in layout.fsx.

For now though, let's look at generate'. we'll look at it block by block.

let generate' (ctx : SiteContents) (page: string) =
    let post =
        ctx.TryGetValues<Postloader.Post> ()
        |> Option.defaultValue Seq.empty
        |> Seq.find (fun n -> n.file = page)

First we try to find the post. We start by using TryGetValues method on SiteContents to pull back all posts that match the shape of the object Post that exists in the loader we looked at previously. Then we iterate through the sequence until we find one that matches the page we're currently iterating through.

let siteInfo = ctx.TryGetValue<Globalloader.SiteInfo> ()
let (desc, title) =
    siteInfo
    |> Option.map (fun si -> (si.description, si.title))
    |> Option.defaultValue ("", "")

We do the same thing to grab SiteInfo (held in globalloader.fsx) and map any description and title we have stored.

Layout.layout ctx (defaultArg post.title "") [
    section [Class "hero is-info is-medium is-bold"] [
        div [Class "hero-body"] [
            div [Class "container has-text-centered"] [
                h1 [Class "title"] [!!title]
                span [] [!!desc]
            ]
        ]
    ]
    div [Class "container"] [
        section [Class "articles"] [
            div [Class "column is-8 is-offset-2"] [
                Layout.postLayout false post
            ]
        ]
    ]
]

This part is pretty different from what we've looked at previously. This is a feature of Fornax that harnesses an interesting feature of F# called Domain Specific Language. Instead of writing HTML in Fornax we use this DSL to simplify the templating process. If you've done work with HTML markup. I'll admit it feels a little awkward to use DSL to write a template, but once I got the hang of it I've found it pretty compelling.

From there generate' returns the DSL HtmlElement we constructed where it can be parsed and turned into a string in Layout.render when we finally output the file into the _public folder.

That's the basic overview of loaders and generators. We only scratched the surface of what we can do with loaders. Because we're working with F# scripts we can get our data from any source we might want.

RESTful API? For sure.

SQL DB? No problem.

CSV? Absolutely.

Similarly, we can output data in a generator in any number of formats. RSS, CSV, JSON, plain text, and so on.

That's enough for today I think. Next time, we'll take a look at creating our own loader using an external API.

Getting Started with Fornax

Drew Knab — Thu, 02 Jul 2020 15:00:23 +0000

What is Fornax?

Fornax is a static site generator written in F#. I think it's pretty neat. Fornax generates static sites through a pipeline of F# scripts called loaders that inject content into a global context that is then consumed by more F# scripts called generators that mold the content its final form. The workflow is decidedly hands-on with a distinct code first approach. This gives Fornax incredible flexibility.

Getting Started

Fornax is installed as a global .NET Core tool on the command line with:

dotnet tool install fornax -g

We can then create a new folder for our new Fornax site, move into it, then:

fornax new

This will create a new Fornax site using the default blog template.

Now that we have a default Fornax site set up we can run fornax watch which will spin up a little server that watches for changes to our project.

Let's take a look at the folder structure now that we have a working Fornax site.

_lib
- Any external libraries we might like to use go in here and can be referenced from the script files.
- By default this contains:
  - Fornax.Core.dll
  - Markdig.dll
_public
- This is where the final production built files are created after a build.
loaders
- Contains all scripts that load data into the global context.
generators
- Contains all scripts that generate files in _public based on the global context created by loaders.
images
- Images used by our app
js
- JS used by our app
posts
- This is a domain specific folder used to hold blog posts written in markdown.
style
- CSS used by the app
config.fsx
- This script runs first, we use it to configure the behavior of our generators and add any additional data like secret keys or SQL connection strings.

Next time we'll take a look an in depth look at loaders and generators and what they can do.