DEV Community: Jonathan Higgs

A Complete Cake Script for Wyam

Jonathan Higgs — Sun, 22 Sep 2019 00:00:00 +0000

Introduction

The previous post gave instructions for creating setting up a blog website that included a simple cake build script. This post will take that script, extract some of the implicit parameters and add some extra steps for clarity and utility

Jump to the end of the post to see the complete script

Original Script

First things first, here is the simple script from the previous post. Currently there are implicit parameters for the output directory and the branch that will be deployed

#addin nuget:?package=Cake.Wyam

var target = Argument("target", "Deploy");

Task("Build")
    .Does(() => Wyam());

Task("Preview")
    .Does(() => Wyam(new WyamSettings {
        Preview = true,
        Watch = true,
    }));

Task("Deploy")
    .IsDependentOn("Build")
    .Does(() => {
        StartProcess("git", "add .");
        StartProcess("git", "commit -m \"Output files generated for subtree\"");
        StartProcess("git", "subtree split --prefix output -b master");
        StartProcess("git", "push -f origin master");
    });

RunTarget(target);

Like any function, it is best to have all of the parameters that might vary specified at the start of the script. While it might initially seem strange that a 'parameter' is hard coded to a specific value in this script it is useful to extract them so that the script can be copy-pasted and reused in another project with minimal fuss and a single place in the script where parameters are set. Currently the input and output directories, and the branch that gets deployed are implicitly used in all the steps

var deployBranch = "master";
var output = "output";

Now the parameters are explicitly set to variables, all instances of them can be replaced in the rest of the script using a bit of string interpolation

Task("Deploy")
    .IsDependentOn("Build")
    .Does(() => {
        StartProcess("git", "add .");
        StartProcess("git", "commit -m \"Output files generated for subtree\"");
        StartProcess("git", $"subtree split --prefix {output} -b {deployBranch}");
        StartProcess("git", $"push -f origin {deployBranch}");
    });

Similarly to the parameters, the locations of directories and files can be extracted and set once. This script doesn't make use of these paths yet, but will do shortly. Again, the paths can be built on-top of each other so each part of the path is only specified in one place

var root = MakeAbsolute(Directory("."));
var inputDirectory = Directory($"{root}/input");
var outputDirectory = Directory($"{root}/{output}");

Continuing the theme of moving all configuration to the start of the script, the settings object passed into wyam are created at the point of usage and contain implicit parameters. They are moved to the start of the script and the paths can be passed in

var build = new WyamSettings {
    InputPaths = new [] { inputDirectory.Path },
    OutputPath = outputDirectory
};

var preview = new WyamSettings {
    InputPaths = new [] { inputDirectory.Path },
    OutputPath = outputDirectory,
    Preview = true,
    Watch = true,
};

Task("Preview")
    .Does(() => Wyam(preview));

Task("Build")
    .Does(() => Wyam(build));

Cake.Git Add-In

The script currently uses out-of-process calls to interact with git but there is a Cake.Git add-in which has methods for lots of the common git commands. Specifically the GitCommit method takes additional parameters for setting the username and email that will be used for the commit. This can be quite useful for scripts that create commits as part of the build process. Here the parameters are extracted to the start of the script, and another gitMessage parameter is used for the message

var gitMessage = Argument("message", "Output files generated for subtree");

var gitUser = "My Name";
var gitEmail = "myname@site.com";

Task("Deploy")
    .IsDependentOn("Build")
    .Does(() => {
        GitAddAll(root);
        GitCommit(root, gitUser, gitEmail, gitMessage);

        StartProcess("git", $"subtree split --prefix {output} -b {deployBranch}");
        StartProcess("git", $"push -f origin {deployBranch}");
    });

Unfortunately the git add-in does not contain any cake aliases for subtree so that step will still need to call out-of-process. Also, there is a long-running issue with the ssh library that Cake.Git uses which means that push and pull won't work if the repository is setup to an ssh url. I hope both of these get fixed one day

The git message will still default to the same value as the original script, but it can also be overwritten by passing it in to the script

$ powershell ".\build.ps1" --message="I did a thing"

Publish Workflow

The main functional change to this script is to create a workflow for working on posts but not publishing them until they are ready. This will be achieved by creating a new posts directory for the WIP posts, including it in the preview but not the main build, and adding a task that will move a post over when a valid post name is supplied

The first thing to do is create a work-in-progress directory that will be visible in preview but not publish. The wyam settings allow for multiple paths to be passed into the InputPaths parameter, and the Blog recipe will look in a posts sub-directory when creating the static pages

/input
    /posts
/output
/wip
    /posts
...

Path variables for both these directories will be needed in a moment, so they're added to the start of the script with the other path declarations. The build settings do not need to change, but the wip directory are passed into the preview settings to include the wip posts in preview content generation

var postsDirectory = Directory($"{root}/input/posts");
var wipDirectory = Directory($"{root}/wip/posts");

var build = new WyamSettings {
    InputPaths = new [] { inputDirectory.Path },
    ...
};

var preview = new WyamSettings {
    InputPaths = new [] { inputDirectory.Path, wipDirectory.Path },
    ...
};

Publish is a new task that will move posts from the wip directory into the input/posts directory. The first thing required will be a script argument into which the name of the post will be passed. Here the posts are kept organized by year so the script can ensure that the posts subdirectory with the current year exists before performing the actual move. File paths are built up from the path declarations which aid in making the script readable. Also note that the .md extension is being omitted from the argument and added by the task for brevity

var post = Argument("post", string.Empty);

Task("Publish")
    .Does(() => {
        EnsureDirectoryExists($"{postsDirectory}/{DateTime.Now.Year}");

        MoveFile(
          File($"{wipDirectory}/{post}.md"),
          File($"{postsDirectory}/{DateTime.Now.Year}/{post}.md")
        );
    });

Wyam blog posts have a published document metadata field that can be automatically set during this move. Cake.FileHelpers is another add-in that has the ReplaceRegexInFiles method to do this

#addin nuget:?package=Cake.FileHelpers

Task("Publish")
    .Does(() => {
        ReplaceRegexInFiles(
            $"{wipDirectory}/{post}.md",
            "Published:\\s*\\d{4}-\\d{2}-\\d{2}",
            $"Published: {DateTime.Now.Date.ToString("yyyy-MM-dd")}"
        );
        ...
    });

The only issue with this task so far is that it requires a post to be moved on every build, which will not always be the case. Cake allows conditional execution of tasks without blocking subsequent tasks from executing, and all that is required is a check on whether a post has been passed into the script to achieve this

Task("Publish")
    .WithCriteria(!string.IsNullOrEmpty(post))
    .Does(() => {
        ...
    });

Finally, the message passed to git can be modified with the name of post

Task("Publish")
    .Does(() => {
        ...
        gitMessage = $"Published: {post}";
    });

Assuming that there is a file in the wip/posts directory called my-post.md then the following command would make use of the Publish task to move it from wip and into the inputs directory, and hence it would be published to the main site. One thing to note is that there seems to be a bug in the command line argument parsing that will assume a second script name is being passed in when a dash appears in a argument value, so the value must be placed in double quotes

$ powershell ".\build.ps1" --post="my-post"

Info Task

The final step to add does not make a functional change, but simply dumps out parameters and variables. Generally, this can be very useful when porting the script to a new project and debugging when it doesn't quite work the first time, or if the script is being executed by a CI process rather than locally

Task("Info")
    .Does(() => {
        Information("Parameters:");
        Information($"-target {target}");
        Information($"-post {post}");
        Information("\nPaths:");
        Information($"Root Directory {root}");
        Information($"Output Directory {outputDirectory}");
        Information($"Post Directory {postsDirectory}");
        Information($"WIP Directory {wipDirectory}");
        Information($"\nGit Settings:");
        Information($"Git User {gitUser}");
        Information($"Git Email {gitEmail}");
    });

Final Script

Putting it all together, here is the final script

////////////////////////
// Add-ins & Tools
//////////////////////

#addin nuget:?package=Cake.FileHelpers
#addin nuget:?package=Cake.Git
#addin nuget:?package=Cake.Wyam

////////////////////////
// Parameters
//////////////////////

var target = Argument("target", "Deploy");
var post = Argument("post", string.Empty);
var gitMessage = Argument("message", "Output files generated for subtree");

var deployBranch = "master";
var gitUser = "My Name";
var gitEmail = "my.name@site.com";

var output = "output";

var root = MakeAbsolute(Directory("."));
var inputDirectory = Directory($"{root}/input");
var postsDirectory = Directory($"{root}/input/posts");
var outputDirectory = Directory($"{root}/{output}");
var wipDirectory = Directory($"{root}/wip/posts");

////////////////////////
// Settings
//////////////////////

var build = new WyamSettings {
    InputPaths = new [] { inputDirectory.Path },
    OutputPath = outputDirectory
};

var preview = new WyamSettings {
    InputPaths = new [] { inputDirectory.Path, wipDirectory.Path },
    OutputPath = outputDirectory,
    Preview = true,
    Watch = true,
};

////////////////////////
// Tasks
//////////////////////

Task("Info")
    .Does(() => {
        Information("Parameters:");
        Information($"-target {target}");
        Information($"-post {post}");
        Information("\nPaths:");
        Information($"Root Directory {root}");
        Information($"Output Directory {outputDirectory}");
        Information($"Post Directory {postsDirectory}");
        Information($"WIP Directory {wipDirectory}");
        Information($"\nGit Settings:");
        Information($"Git User {gitUser}");
        Information($"Git Email {gitEmail}");
    });

Task("Preview")
    .IsDependentOn("Info")
    .Does(() => Wyam(preview));

Task("Publish")
    .WithCriteria(!string.IsNullOrEmpty(post))
    .IsDependentOn("Info")
    .Does(() => {
        ReplaceRegexInFiles(
            $"{wipDirectory}/{post}.md",
            "Published:\\s*\\d{4}-\\d{2}-\\d{2}",
            $"Published: {DateTime.Now.Date.ToString("yyyy-MM-dd")}"
        );

        EnsureDirectoryExists($"{postsDirectory}/{DateTime.Now.Year}");

        MoveFile(
          File($"{wipDirectory}/{post}.md"),
          File($"{postsDirectory}/{DateTime.Now.Year}/{post}.md")
        );

        gitMessage = $"Published: {post}";
    });

Task("Build")
    .IsDependentOn("Publish")
    .Does(() => Wyam(build));

Task("Deploy")
    .IsDependentOn("Build")
    .Does(() => {
        GitAddAll(root);
        GitCommit(root, gitUser, gitEmail, gitMessage);

        StartProcess("git", $"subtree split --prefix {output} -b {deployBranch}");
        StartProcess("git", $"push -f origin {deployBranch}");
    });

////////////////////////
// Invoke
//////////////////////

RunTarget(target);

Create GitHub Pages Blog with Wyam

Jonathan Higgs — Sat, 21 Sep 2019 00:00:00 +0000

Introduction

This is a guide to setting up a blog website that is hosted by github pages and uses wyam to generate the static content. It is based off an older guide which had some issues when following through the steps. This guide fixes those steps and streamlines the process

The overall strategy that will be used it to make a github repository called <username>.github.io that will get published to that same url. Only html content in the master branch will get published, so there will be a second orphaned branch that will contain all the source markdown and configuration files. Then the master branch will get populated by using the git subtree split command from the wyam output

First thing to do is create a folder and initialize the git repository

$ cd /path/to/dev-stuff
$ mkdir <username>.github.io
$ cd <username>.github.io
$ git init

This will initialize the master branch which should only contain the wyam output, so immediately switch to some other branch, which will be called dev here

$ git checkout --orphan dev

It is also a good idea to add the .gitignore at this stage to ensure that only the correct files are added to the repository. Note that the output folder is not added to the ignores and will be kept under source control

# Visual Studio
.vs/

# Cake build
tools/
!tools/packages.config

# Wyam
wwwroot/
config.wyam.hash
config.wyam.dll

Just a quick note, I use VisualStudio and an extension for rendering markdown content called MarkdownEditor which is why .vs/ is excluded in the ignores. It isn't required at all, but worth adding the ignore for any editor specific files that might be created

Setting up the Blog

Wyam is a excellent tool for generating static web content. It is all written in dotnet and highly modular, so you can extend it fairly easily, but also has pre-baked recipes. We'll be using the blog recipe which will give us a pipeline for transforming markdown files into a full blog site

The easiest way install wyam is through the dotnet cli as below, and other methods can be found on the wyam site

$ dotnet tool install -g Wyam.Tool

Wyam comes with a blog recipe that contains a pipeline of wyam modules. The recipe is initialized with the following command which will pull down additional tools it needs from nuget, create some configuration files, and a folder called input which is where all the source markdown pages are stored

$ wyam new -r Blog

From the input files, the wyam build command will execute the static generation pipeline and create the html pages in the output folder. This can take a few moments for a mostly blank project and up to a few minutes when there are a lot of pages to generate

$ wyam build

A development server with the standard file watch and hot reload is included to allow previewing the site on a localhost server

$ wyam preview

Cake Build

Cake is a wonderful build automation tool that uses C# script files to define the build process and some Roslyn magic to compile and run them. It is very easy to extend and has an extensive set of add-ins that cover all sorts of tools that are useful for build, including wyam

First thing to do is download the powershell bootstrapper which, when run, will download the cake executable and execute the cake script

Next is the cake build script itself, which is split into a couple of subsections. The first section does two things, first it uses preprocessor directives to inform cake that the Cake.Wyam nuget package should be downloaded and loaded for use in the script. Second it gets the command line target argument which will be used to set which task is going to be invoked

#addin nuget:?package=Cake.Wyam

var target = Argument("target", "Deploy");

Next is the most interesting section. It defines but does not execute the tasks (yet). The first two tasks are very simple, either run the Build or run the Preview server (which includes a build). These call the Wyam() method, which is a cake extension imported from the Cake.Wyam nuget package in the first section and used the settings objects created in the second section

The final Deploy task is where the git subtree split magic happens. Deploy has a dependency on Build, so the output directory will have updated with the latest set of generated files. These files get committed to git and then subtree split will filter out all changes and files other than those in the output folder and then put those new commits into the master branch (omitting the ./output directory in the path as well). The result of this will be that the master branch has a complete versioned history of the generated (and only the generated) output, which is exactly what github pages wants

Task("Build")
    .Does(() => Wyam());

Task("Preview")
    .Does(() => Wyam(new WyamSettings {
        Preview = true,
        Watch = true,
    }));

Task("Deploy")
    .IsDependentOn("Build")
    .Does(() => {
        StartProcess("git", "add .");
        StartProcess("git", "commit -m \"Output files generated for subtree\"");
        StartProcess("git", "subtree split --prefix output -b master");
        StartProcess("git", "push -f origin master");
    });

The final step is a method call to tell cake which task to run. In this case target will default to Deploy unless the parameter is set when calling the script. Cake will find the specified task and create a tree of tasks that need to be run using the dependency information to execute all the required tasks

RunTarget(target);

Running the Build

Execution of the script is fairly easy. From a command prompt and assuming powershell is on the path

$ powershell ".\build.ps1"

Running the preview server is done by setting the target parameter and passing in the preview task name

$ powershell ".\build.ps1" -target Preview

Note: the versions of the cake tool and add-ins don't always match up, so it is likely you will want to add an extra argument to silence the version mismatch warning. Ignoring the warning could cause issues running the script, but most often it is fine

$ powershell ".\build.ps1" --settings_skippackageversioncheck=true

GitHub Repository

The final step is to create a repository on github and push the initial contents. The repository must be named <username>.github.io for github to understand that it is for pages, and you might also need to enable it in the 'GitHub Pages' section of the repository settings page

All that is required now is set this as the origin remote in the local repository with an https or ssh uri (both shown)

$ git remote add origin https://github.com/<username>/<username>.github.io.git
$ git remote add origin git@github.com:<username>/<username>.github.io.git

Now the cake build will push the output to github and a few seconds later the content will be available on the world-wide-web

See the next post for a complete cake build script for working with wyam

The Developers Creed

Jonathan Higgs — Thu, 08 Nov 2018 12:43:08 +0000

This is my debugger. There are many like it, but this one is mine. My debugger is my best friend. It is my life. I must master it as I must master my life. My debugger, without me, is useless. Without my debugger, I am useless. I must use my debugger constantly. I must set the breakpoints to find the bug that is trying to evade me. I must find it before it hits production. I will... My debugger and I know that what counts in this sprint is not the total lines of code we write, the frameworks we use, nor the fancy language features we use. We know it is this fix that counts. We will fix... My debugger is human, even as I, because it is my life. Thus, I will learn it as a brother. I will learn its weaknesses, its strengths, its features and its plugins. I will ever guard it against the ravages of the environment as I will ever guard my hands, my eyes and my stack overflow reputation against damage. I will keep my debugger close and ready. We will become part of each other. We will... Before the compiler, I swear this creed. My debugger and myself are the defenders of the repository. We are the masters of understanding the codebase. We are the saviours of the users. So be it, until victory is ours and there are no bugs, but peace!