DEV Community: Precious Chicken

Githubhack23: Vegaration or Visualising continuous integration using Github actions and vega-lite

Precious Chicken — Mon, 22 May 2023 23:14:15 +0000

vegaration (noun) [vay-gah-grey-shuhn]: The use of vega-lite to visualise continuous integ-ration.

What I built

A tool to aid development teams by visualising Continuous Improvement. I used GitHub actions to log information about every commit made on a repo (e.g. user, sha, passing tests, failing tests) into a json file stored within the repo. This file was then used as the source for a number of visualisations implemented with the vega-lite specification. The same GitHub action workflow then published these visualisations to GitHub Pages so they could be viewed by all stakeholders.

This demonstration proves the possibility of logging and visualising a whole suite of Continuous Improvement metrics which could help teams identify patterns, address problems and increase efficiency. It differs from other CI visualisations (e.g. Jenkins) in the flexibility it offers: vega-lite is a data science tool meaning it offers a very wide range of visualisation possibilities; likewise logging via GitHub actions allows you the freedom to capture almost anything about the CI/CD process that your team is interested in.

Category Submission:

DIY Deployments

App Link

Continuous Integration Visualisation on GitHub Pages:
https://preciouschicken.github.io/vegaration/

Screenshots

Description

Visualising continuous integration using GitHub actions and vega-lite, natively hosted on GitHub Pages.

Link to Source Code

GitHub repository:
https://github.com/PreciousChicken/vegaration

Permissive License

MIT

Background (What made you decide to build this particular app? What inspired you?)

The integration of GitHub Actions and Vega-Lite to visualise continuous integration for software development offers a highly useful and beneficial approach for several reasons.

Firstly, it allows developers to gain a comprehensive and real-time understanding of the continuous integration process by providing visual representations of various metrics. These visualisations make it easier to identify patterns, trends, and potential issues at a glance, enabling faster and more efficient decision-making. This then enhances the ability of a team to track progress, evaluate the impact of changes, and identify areas for improvement.

Secondly, by integrating the visualisation directly into the software development workflow, teams can foster a culture of transparency and collaboration. Developers, project managers, and stakeholders can all access and interpret the visualisations, enabling a shared understanding and facilitating communication.

Ultimately, the combination of GitHub Actions and vega-Lite empowers software development teams to make data-driven decisions, streamline processes, and ensure the delivery of high-quality software.

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

I utilised GitHub actions, GitHub Pages and the vega-lite visualisation specification. I learnt lots about all three in the process! I did consider using vega rather than vega-lite, but after trying both out decided for the higher-level grammar.

Additional Resources/Info

Worked example / Step-by-step guide:
https://dev.to/preciouschicken/vegaration-visualising-continuous-integration-using-github-actions-and-vega-lite-a97

Vegaration: Visualising continuous integration using Github actions and vega-lite

Precious Chicken — Mon, 22 May 2023 17:50:14 +0000

Introduction

vegaration (noun) [vay-gah-grey-shuhn]: The use of vega-lite to visualise continuous integ-ration.

This is a worked example for Linux which aims to visualise the software development practice of continuous integration (CI). It uses GitHub actions to manage the CI workflow and vega-lite as the visualisation specification. Or to put it simply: every time someone commits to a repository on GitHub we can log elements associated with their commit (e.g. what tests passed, what tests failed) and then present this data visually using a wide suite of open-source metrics.

This demonstration proves the possibility of logging and visualising a whole suite of Continuous Improvement metrics which could help teams identify patterns, address problems and increase efficiency. The example created here, which can be viewed on the Vegaration GitHub Page, offers a very small slice of the total possibilities on offer. It differs from other CI visualisations (e.g. Jenkins) in the flexibility it offers: vega-lite is a data science tool meaning it offers a very wide range of visualisation possibilities; likewise logging via GitHub actions allows you the freedom to capture almost anything about the CI/CD process that your team is interested in.

All code is contained in the repo PreciousChicken/vegaration.

This was written using Manjaro Linux 22.1.1, node v16.13.2, vega-lite 5.9.1 and npm v8.3.2.

Create the GitHub repo

The first step is creating a new repo on GitHub as demonstrated:

Ensure as you do so you set set the .gitignore option to Node. Readme is of course optional.

Clone into the repo

At the terminal clone into the repo using the following command - replacing the repo with the one created at the step above.

git clone git@github.com:PreciousChicken/vegaration.git

And then change directory into the repo you have created, e.g.:

cd vegaration

Testing...

Installing jest

To demonstrate the continuous integration it is necessary to create some mock tests. The Jest framework will be our testing weapon of choice, so at the command line enter:

npm i jest --save-dev

This will install the jest package, but we now need to add the test script to our package.json file which should be amended to look like this (the jest version will naturally be different depending on whatever is current):

{
  "devDependencies": {
    "jest": "^29.5.0"
  },
  "scripts": {
    "test": "jest"
  }
}

The tests

Now to create some simple functions and associated tests, nothing too complex. Create a file called add.js and paste the following:

function sum(a, b) {
  return a + b;
}
module.exports = sum;

And then create the associated test file add.test.js and paste:

const sum = require('./add');

test('adds 1 + 1 to equal 2', () => {
  expect(sum(1, 1)).toBe(2);
});

We may as well do subtraction too, so create a file called minus.js and paste:

function subtract(a, b) {
  return a - b;
}
module.exports = subtract;

and the associated test file minus.test.js with the following:

const subtract = require('./minus');

test('substract 2 - 1 to equal 1', () => {
  expect(subtract(2, 1)).toBe(1);
});

That now gives us two functions and two tests. Try them out by entering npm test at the terminal, and assuming copy and paste worked out, then jest should recognise the test files and both tests should pass with flying colours.

Hosting the visualisation on GitHub pages

Our visualisation has to be hosted somewhere - and it makes sense to use GitHub pages. To make things not so bare-bones we are going to ornament these with the Dinky theme, but it makes little difference.

Base

Create a folder to base the GitHub pages and then switch to it:

mkdir visualisation && cd visualisation

Typically this folder gets called Docs, but being descriptive is never a bad thing.

Create a configuration file for the GitHub pages named _config.yml (yes, it does start with an underscore) and paste:

theme: jekyll-theme-dinky
title: Vegaration
description: Visualising continuous integration using Github actions and vega-lite

Vega-liteify

To allow vega-lite visualisations to be embedded the GitHub pages need to know where to look to understand the visualisation, so create a new folder (again starting with an underscore) within the visualisations folder and switch to it:

mkdir _includes && cd _includes

Then create the file head-custom.html and paste:

<script src="https://cdn.jsdelivr.net/npm/vega@5"></script>
<script src="https://cdn.jsdelivr.net/npm/vega-lite@5"></script>
<script src="https://cdn.jsdelivr.net/npm/vega-embed@6"></script>

This snippet correspondingly gets picked up by the theme and entered within the <head> section of the finalised HTML (if we weren't using a theme we would place it directly in the <head>, but given we are using Dinky this is much neater).

The visualisation

Returning to our visualisation folder:

cd ..

we are going to create the file which will actually craft our visualisation, which is a mix of markdown and HTML as allowed by Jekyll (the engine which runs GitHub pages). This is where we will place the vega-lite specification which will create the embedded charts. This worked example is not going to dive into the vega-lite specification itself, for that you are probably best starting at the vega-lite tutorial. That said create the file index.md and paste:

---
title: Continuous Integration Visualisation
---

## Commits per user

Bar chart showing commits per user:

<div id="commitsPerUser"></div>

<script type="text/javascript">
    var commitsPerUser = {
        "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
        "description": "Bar chart showing commits per user.",
        "config": {
        "background": "#FBFAF7"
        },
        "width": 300,
        "data": {"url": "./log.json"},
        "transform": [
        {
        "aggregate": [{
        "op": "count",
        "field": "sha",
        "as": "Commits"
        }],
        "groupby": ["user"]
        }
        ],
        "mark": "bar",
        "encoding": {
        "x": {"field": "Commits", "type": "quantitative",
        "axis": {"tickMinStep": 1}
        },
        "y": {"field": "user", "type": "nominal", "sort": "descending"}
        }
        };
    vegaEmbed('#commitsPerUser',commitsPerUser);
</script>

## Passing vs Failing Tests

### add.test.js

Pie chart showing fails vs passes for add.test.js (hover over segment for tooltip display):

<div id="addPie"></div>

<script type="text/javascript">
    var addPie = {
        "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
        "data": {"url": "./log.json"},
        "description": "Pie chart showing pass vs fail for add_test_js",
        "config": {
        "background": "#FBFAF7"
        },
        "transform": [
        {"filter": {"field": "add_test_js", "oneOf": ["FAIL", "PASS"]}},
        {"aggregate": [{"op": "count", "as": "count"}], "groupby": ["add_test_js"]}
        ],
        "mark": {
        "type": "arc",
        "tooltip": true
        },
        "encoding": {
        "theta": {"field": "count", "type": "quantitative"},
        "color": {
        "field": "add_test_js",
        "type": "nominal",
        "scale": {
        "domain": ["FAIL", "PASS"],
        "range": ["red", "green"]
        },
        "legend": {
        "title": "Result",
        "labelExpr": "datum.value === 'FAIL' ? 'Fail' : 'Pass'"
        }
        },
        "tooltip": [
        {"field": "count", "type": "quantitative"},
        {"field": "add_test_js", "type": "nominal"}
        ],
        "text": {"field": "count", "type": "quantitative"}
        },
        "width": 300,
        "height": 300
        };
    vegaEmbed('#addPie', addPie);
</script>


### minus.test.js

Pie chart showing fails vs passes for minus.test.js (hover over segment for tooltip display):

<div id="minusPie"></div>

<script type="text/javascript">
    var minusPie = {
        "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
        "data": {"url": "./log.json"},
        "description": "Pie chart showing pass vs fail for minus_test_js",
        "config": {
        "background": "#FBFAF7"
        },
        "transform": [
        {"filter": {"field": "minus_test_js", "oneOf": ["FAIL", "PASS"]}},
        {"aggregate": [{"op": "count", "as": "count"}], "groupby": ["minus_test_js"]}
        ],
        "mark": {
        "type": "arc",
        "tooltip": true
        },
        "encoding": {
        "theta": {"field": "count", "type": "quantitative"},
        "color": {
        "field": "minus_test_js",
        "type": "nominal",
        "scale": {
        "domain": ["FAIL", "PASS"],
        "range": ["red", "green"]
        },
        "legend": {
        "title": "Result",
        "labelExpr": "datum.value === 'FAIL' ? 'Fail' : 'Pass'"
        }
        },
        "tooltip": [
        {"field": "count", "type": "quantitative"},
        {"field": "minus_test_js", "type": "nominal"}
        ],
        "text": {"field": "count", "type": "quantitative"}
        },
        "width": 300,
        "height": 300
    };
    vegaEmbed('#minusPie', minusPie);
</script>

Lastly we need to create a placeholder file where as commits are made data is placed so that it can be visualised; therefore in the same visualisation folder create the file log.json and paste:

[
]

Committed

The rest of the steps that need to take place happen on the GitHub site, so we should commit and push the code we have written so far:

cd ..
git add .
git commit -m "Initial commit"
git push

Action, man

So now we have lined up our tests and the visualisation that shows us information every time they run. The next step is to link these up with a GitHub action. This action will run every time there is a commit to the repo and will:

Run the tests ¹
Log the results of the test in log.json
Publish an updated version of the visualisation on GitHub pages

This workflow however is best created from the GitHub website itself. Therefore navigate to the Settings on your online repo and then select Pages from Code and automation. Under the heading Build and deployment select the GitHub Actions from the Source dropdown menu and then click on the create your own link as shown:

You will then be taken to the code editor online, in the box entitled Name your file... paste the filename visualisation.yml and then in the main Edit body of the code editor itself delete the default contents and paste:

name: Visualising continuous integration

on:
  # Runs on pushes targeting the default branch
  push:
    branches: ["main"]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

env:
  # set this to the node version to use if not 16
  NODE_VERSION: '16.x'                

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: write
  pages: write
  id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  # Saves results of tests
  output_test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      - name: npm install
        run: npm ci
      - name: Run tests and add to log file
        run: |
          # Create an shell variable called `tests` and add git commit sha
          tests="{\"sha\": \"${{ toJSON(github.sha) }}\", "
          # Run npm test and append test results to `tests` variable 
          # Uses grep to remove all test results apart from FAIL / PASS and test
          # Awk used to write to file and replace filename periods with underscores
          # Vega-lite does not play well with periods, so add.test.js -> add_test_js
          tests+=$(npm test 2>&1 >/dev/null | grep 'FAIL\|PASS' | awk '
            {
            js = gensub(/^\.\/(\w*)\.(\w*)\.(\w*)/, "\\1_\\2_\\3" ,"g", $2) 
            printf("\"" js"\": \""$1"\", ")
          }')
          # Appends github user name to `tests` shell variable
          tests+="\"user\": \"${{ toJSON(github.actor) }}\"}"
          # Shell variable `tests` appended to end of log.json
          awk -i inplace -v env_var="$tests" -v RS='^$' -v ORS= '{
            print gensub(/\[/,"[\n  "env_var",","g")
            }' visualisation/log.json
          # Ensure no trailing commas at end of log.json
          awk -i inplace -v RS='^$' -v ORS= '{
            print gensub(/,\n\]/,"\n]","g")
          }' visualisation/log.json
      - name: Push updated log
        run: |
          # Should be changed to name of user listed in repo
          git config user.name PreciousChicken
          # Should be changed to email of user listed in repo
          git config user.email hello@preciouschicken.com
          git add .
          git commit -m "Auto-generated by github actions"
          git push
  # Build job
  build:
    runs-on: ubuntu-latest
    needs: output_test
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup Pages
        uses: actions/configure-pages@v3
      - name: Build with Jekyll
        uses: actions/jekyll-build-pages@v1
        with:
          source: ./visualisation/
          destination: ./_site
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v1

  # Deployment job
  deploy:
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v2

There are two lines in the above that will need to be changed dependent on the repo you are using. These are: git config user.name PreciousChicken and git config user.email hello@preciouschicken.com, change to whatever is appropriate for your repo.

Once you have pasted this into the online code editor and made the changes, click the Commit changes... button and commit directly to the main branch. At time of writing there is a bug which will trigger two workflow runs, of which the second will fail. However future pushes onto the branch should work correctly.

You'll notice a fair amount of RegEx used and old school tools such as grep and awk², which serve to capture the test results and append this data to the log.json as a new line each time a commit is run. The appended line looks something like this:

{"sha": "a0d7b73dfb0c5273e6dceb380e7568dfe735d65b", "minus_test_js": "FAIL", "add_test_js": "PASS", "user": "PreciousChicken"},

Of course this capture could have been achieved with something like Python, but seen as the GitHub workflow runs on an Ubuntu virtual machine, GNU tools work just fine. They also have the advantage that you are implementing all the logic in the workflow itself, which is arguably more transparent and portable.

NB - If you have sharp eyes you might notice that the filename periods have been changed to underscores in the json e.g. minus.test.js to minus_test_js. This is because vega-lite does not play well with field names that include periods, so to avoid complications these have been swapped out.

Results and Conclusion

From this point on any further commits will cause the GitHub action to run, the tests to be activated and log.json to get a new entry, which will be visualised by the vega-lite embedded in the GitHub pages. This will result in a visualisation that looks similar to this:

It is worth noting it can take some time for this page to be updated; indeed possibly about ten minutes on first creation according to this StackOverflow answer.

Clearly this particular visualisation may not have any particular benefit for your project (although seeing which tests are failing more than others and who is the most active participant, might be of use). However given the flexibility of vega-lite and the option to log just about anything you would want with GitHub actions, there are literally limitless use cases.

Feedback? Please leave a message below. If you like what I've done here please take a moment to star the repo and let me know.

Normally failing tests would halt the progress of GitHub actions - for good reason - a failed test probably means you do not want to deploy. As this was a visualisation however which was intended to capture both passing and failing tests, although the tests were run their results were captured in a shell environment variable, rather than being outputted to standard output / standard error which would have halted the workflow. In a production system therefore you would want a seperate workflow which is actually running the tests in their normal manner allowing a return to standard output / standard error. ↩
Grep was probably redundant and I could have used awk for everything, but it seemed simpler to use both in this instance. ↩

A no-tears guide to adding references in Groff

Precious Chicken — Tue, 17 Jan 2023 13:10:15 +0000

Introduction

Groff is a decades-old typesetting system present on most Linux distributions which uses an application called refer to add citations. I found adding citations using this mechanism a challenge, therefore as an aide-memoire this is a minimal worked example of adding references to a Groff document with the refer program. I'm using GNU Groff version 1.22.4, GNU refer (groff) version 1.22.4 on Manjaro Linux 22.0.0. I'm using the ms macro, but the basic principles should apply to most macros. All text examples can be found in the associated github repo.

Creating documents

First create the document we are placing the references into, so create a file named notears.ms and paste the following:

.R1 \" Citation commands start
database notears.bib # Path to bibliography file
accumulate # Collates References at end of documentation
move-punctuation # Ensures that citation appears before full-stop.
label "(A.n|Q) ', ' (D.+yD.y%a*D.-y)" # Actual format of citation (e.g. (Author, Date)
bracket-label " (" ) "; " # Bracket style
no-label-in-reference # Does not display full citation (e.g. Author, Date) within References
.R2 \" Citation commands end
.ds FAM H \" Sets font family
.TL
A no-tears guide to adding references in Groff
.AU
PreciousChicken
.AB
The abstract of a no-tears guide to adding references in Groff, see https://preciouschicken.com/blog/posts/no-tears-references-groff/ for more detail.
.AE
.PP
Here is a reference to a story about brain cells playing ping-pong
.[
shepherd22
.]
in my first paragraph.
.PP
And another reference to an AI story
.[
callaway20
.]
in my second paragraph.
.PP
Now we are on the final paragraph and I have my third
.[
nao22tax
.]
and then last reference,
.[
nao22bbc
.]
both from the NAO and both published in 2022.  I've also added a footnote for comparison purposes\**.
.FS \" Footnote start
This is the only footnote.
.FE \" Footnote end.

The section starting .R1 and ending .R2 is where commands are added as to how the citations will displayed. These commands could also be entered on the command line rather than in the document text - but I prefer the latter as that keeps it all together if you are trying to recreate the document later. For a full list of the applicable commands see man(1) refer. The point where the citations are introduced into the text are marked by the opening .[ and closing .], with the keyword of the citation inbetween (e.g. callaway20).

Keen readers may note the comments format for the text between .R1 and .R2 follows a different format (comments begin with # not \") as this section is intended for the preprocessor refer and not groff itself.

Now we need to create the bibliography file that this document points to, so in the same folder¹ create another file named notears.bib and paste:

%K shepherd22
%A Shepherd, T.
%B London
%D 2022
%T Scientists teach brain cells to play video game Pong
%J The Guardian
%O Available at: https://www.theguardian.com/australia-news/2022/oct/13/scientists-teach-brain-cells-to-play-virtual-pong [Accessed 14 Oct 22]

%K callaway20 
%A Callaway, E.
%B London
%D 2022
%T 'It will change everything': Deepmind's AI makes gigantic leap in solving protein structures.
%J Nature News
%O Available at: https://www.nature.com/articles/d41586-020-03348-4 [Accessed 8 Sep 22]

%K nao22tax
%Q National Audit Office
%C London
%D 2022
%T Managing tax compliance following the pandemic
%O Available at: https://www.nao.org.uk/reports/managing-tax-compliance-following-the-pandemic/ [Accessed 17 Jan 23]

%K nao22bbc
%Q National Audit Office
%C London
%D 2022
%T A digital BBC
%O Available at: https://www.nao.org.uk/reports/a-digital-bbc/ [Accessed 17 Jan 23]

Each reference is separated by a blank line, with each field of the reference on a new line. Each reference line starts with the code for the field, so for instance %C is city of publication. The keyword we referred to in notears.ms is given on the %K line. Again for the guide see man(1) refer. The mom macro has also introduced some fields which were not in the original (for instance %l as translator).

Producing the PDF

We now have enough to produce a document with references. There are a number of commands² you can use to produce the final file, but we'll go for:

groff -R -m ms -T pdf notears.ms > notears.pdf

The -R option here being critical as it informs groff that there are references and to use the refer preprocesser before producing the final document.

If you now open the resulting notears.pdf with your pdf viewer of choice you should see:

Of note is that the National Audit Office citations are disambiguated - i.e. 2022a and 2022b. This is as a result of the rather cryptic D.+yD.y%a*D.-y code in the label field of the document - and is copied straight from man(1) refer.

Conclusion & further references

If you've found this useful please leave a comment below or star the github repo. You might also be interested in a previous post I wrote on producing correspondence letters in groff or if you are a vim user my vim plugin: vim-groff-viewer.

Apart from the man(1) refer page there is not a huge amount of concise online guidance on including references within groff, however a useful video is Luke Smith's Your Brain Using REFER to do your bibliographies automatically in groff/troff.

The bibliography can be in another folder, in which case include the full path in the database command. ↩
You could also use refer notears.ms | groff -m ms -T pdf > notears.pdf or grog -T pdf --run notears.ms > notears.pdf or pdfroff -R -m ms notears.ms --pdf-output=notears.pdf all of which would produce the same result. Though interestingly in my testing I noticed that pdfroff used a more up to date version of the PDF format: v1.7 compared to the others using v1.4. ↩

Sprechen sie GraphSQLite? Or querying data in a SQLite database using GraphQL and Apollo Server

Precious Chicken — Thu, 02 Jun 2022 11:32:38 +0000

Introduction

The website sqlitetutorial.net has some good pointers on using the SQLite database. This worked example builds on their Querying Data in SQLite Database from Node.js Applications tutorial. The differences being I'm using the better-sqlite3 rather than the sqlite3 npm package¹ and I'm using a GraphQL API (powered by Apollo Server) to present the data (as opposed to console.log). As per the sqlitetutorial.net site I'm using the sample database chinook.db. This worked example is designed to mimic the tutorial as closely as possible - the queries and responses are almost exactly the same. It probably isn't a great place to start though if you don't understand the GraphQL fundamentals, in that case you might try Oh-so minimal GraphQL API example with Apollo Server first.

All code can be found at the repo PreciousChicken/sqlite-graphql-apollo-server.

I'm using Manjaro Linux 21.2.6, node v16.13.2, npm v8.3.2, better-sqlite3 v7.5.3 and apollo-server v3.8.1.

Initialisation

At the terminal create a directory, and install the relevant packages:

mkdir sqlite-graphql
cd sqlite-graphql
npm init -y
npm install apollo-server better-sqlite3

Download the database

Download and unzip the sample Chinook.db database from sqlitetutorial.net into the folder you just created, ensuring it is named chinook.db. If you have curl and 7z installed you could do this at the command line:

curl https://www.sqlitetutorial.net/wp-content/uploads/2018/03/chinook.zip --output chinook.zip
7z x -tzip chinook.zip chinook.db && rm chinook.zip

Create the server

Copy and paste the following into a new file called index.js:

const { ApolloServer } = require('apollo-server');
const { typeDefs, resolvers  } = require('./schema');

const server = new ApolloServer({ typeDefs, resolvers });

// The `listen` method launches a web server.
server.listen().then(({ url }) => {
  console.log(`Server ready at ${url}`);
});

Create the schema

Create a file named schema.js and paste the following:

const db = require('better-sqlite3')('chinook.db', { readonly: true, fileMustExist: true});
const { gql } = require('apollo-server');

const typeDefs = gql`
    type Playlist {
        "PlaylistId"
        id: ID
        "Name"
        name: String
    }

    type Customer {
        "Customer ID"
        id: ID
        "First name"
        firstName: String
        "Last name"
        lastName: String
        "Country"
        country: String
        "Email"
        email: String
    }

    type Query {
        playlists: [Playlist]
        playlist(id: ID!): Playlist
        customerLocation(country: String!): [Customer]
    }
`

const resolvers = {
    Query: {
        playlists: () => db.prepare(
            'SELECT DISTINCT Name name ' + 
            'FROM playlists ' + 
            'ORDER BY name'
        ).all(),

        playlist: (_, args) => db.prepare(
            'SELECT PlaylistId id, Name name ' +
            'FROM playlists ' + 
            'WHERE PlaylistId  = ?'
        ).get(args.id),

        customerLocation: (_, args) => db.prepare(
            'SELECT FirstName firstName, LastName lastName, Email email ' + 
            'FROM customers ' + 
            'WHERE Country = ? ' + 
            'ORDER BY FirstName'
        ).all(args.country),
    },
}

exports.typeDefs = typeDefs;
exports.resolvers = resolvers;

A close examination will notice that the SELECT in each statement renames the elements - so the column FirstName in chinook.db is renamed firstName. This is done so that the data returned by the SQL matches the GraphQL naming convention and which is used for the definitions in the typeDefs. Not strictly necessary, but good practice.

Start the server

Having written the code, let's start the server. At the terminal enter:

node index.js

Assuming success you should see a message similar to the following:

Server ready at http://localhost:4000/

Apollo Sandbox

Pointing our browser at the URL above we should now be invited to Query your server. Below are listed the examples given in the sqlitetutorial.net tutorial, broken down into operation (i.e. query), optional variables and expected response. Give it a go in the Apollo Sandbox ².

a. Return all distinct playlists

Corresponds with the Querying all rows with all() method example.

Operation

query Playlists {
  playlists {
    name
  }
}

Response

{
  "data": {
    "playlists": [
      {
        "name": "90’s Music"
      },
      {
        "name": "Audiobooks"
      },
      [...]
    ]
  }
}

b. Find a playlist given an ID

Corresponds with the Query the first row in the result set example.

Operation

query Playlist($playlistId: ID!) {
  playlist(id: $playlistId) {
    id
    name
  }
}

Variables

{
  "playlistId": 1
}

Response

{
  "data": {
    "playlist": {
      "id": "1",
      "name": "Music"
    }
  }
}

c. Find all customers in a given country

Corresponds with the Query rows with each() method example - although we are not using the each() method here, as we are returning the entire set of results with GraphQL and not iterating over the results.

Operation

query CustomerLocation($country: String!) {
  customerLocation(country: $country) {
    firstName
    lastName
    email
  }
}

Variables

{
  "country": "USA"
}

Response

{
  "data": {
    "customerLocation": [
      {
        "firstName": "Dan",
        "lastName": "Miller",
        "email": "dmiller@comcast.com"
      },
      {
        "firstName": "Frank",
        "lastName": "Harris",
        "email": "fharris@google.com"
      },
      [...]
    ]
  }
}

Conclusion

Comments, feedback? Post below.

Related work

Some other resources I have produced on GraphQL are:

Tie down scheme for an Apollo GraphQL server in a Node Docker container - Worked example deploying a GraphQL server in a docker container.
Oh-so minimal GraphQL API example with Apollo Server - A more comprehensive tutorial on GraphQL APIs.
A no jokes guide to testing a GraphQL API with Jest - Worked example as to testing a GraphQL API.
Stacking Vercel, a GraphQL Apollo Server and React - Deploying a GraphQL API on Vercel.

I preferred the documentation of better-sqlite3, plus it claims to be faster. ↩
Sandbox has replaced Playground, which has been retired. ↩

Introducing vim-groff-viewer: A vim plugin for displaying Groff files in a document viewer

Precious Chicken — Wed, 18 May 2022 19:58:44 +0000

Introduction

I like (Neo)vim, I like Groff. So much so that I’ve blogged about letter writing in Groff before. What I don’t like is writing markup in Neovim, exiting Neovim, compiling Groff, opening a document viewer and then going back into Neovim to edit again. And repeat.

To remedy this my first vim plugin - vim-groff-viewer. It is probably easier to demonstrate, than explain, so…

Video demonstration

Download, usage, configuration, etc

Interested? Find more at preciouschicken.com/software/vim-groff-viewer/.

Look Ma, no Neovim plugin manager!

Precious Chicken — Sat, 14 May 2022 06:47:32 +0000

Handsfree

To manually install a Neovim plugin, for example Tim Pope's Commentary, without using a plug-in manager then at the terminal :

mkdir -p ~/.local/share/nvim/site/pack/tpope/start/commentary
git clone https://tpope.io/vim/commentary.git ~/.local/share/nvim/site/pack/tpope/start/commentary

The install path always follows the pattern of ~/.local/share/nvim/site/pack/foo/start/bar, where the variables foo and bar can be whatever makes sense to you. Commonly foo is the name of the author (e.g. tpope) and bar is the name of the plugin (e.g. commentary).

Automation

Which is great, but from time to time your plugins will get updated and you will need to pull fresh versions. This can easily be achieved on Linux systems with a small script. Create a file named ~/.local/bin/vimpluginupdate and paste the following:

#!/bin/sh
NVMPTH="$HOME/.local/share/nvim/site/pack"
git -C $NVMPTH/tpope/start/commentary pull || git clone --depth 1 https://tpope.io/vim/commentary.git $NVMPTH/tpope/start/commentary

The rather long git line pulls the repository if it already exists, and if it does not exist (i.e. you haven't done the first step) then it clones a fresh one (using the depth option to just get the latest version and not previous changes).

Now make the file executable:

chmod u+x ~/.local/bin/vimpluginupdate

And whenever you want to update your plugins run¹:

vimpluginupdate

As you install more updates simply add new lines.

Helptags

To load any help documentation associated with the plugin then run :helptags ALL within Neovim when the plugin is active. So for instance the vim-commentary plugin will be active when Neovim has open any file recognised as a programming language (i.e. any file with an extension .js, .py, etc). In vim-commentary's case the help docs can then be accessed with :help commentary.

But I use vim?

All I've really done here is amended the vim-commentary install section, so check that out for standard vim (NB - the main difference is the directory used).

I demand an authoritative source

See :help packages and :help add-package.

Version control

This was tested corrected on Ubuntu Linux 22.04 LTS and Neovim v0.6.1.

Interestingly the ~/.local/bin/ directory is not added to your PATH on a fresh install, so if the command does not work at first, log out then in again. ↩

Étoile du jour: Deploying NASA's Astronomy Picture of the Day on Azure (Microsoft Azure Trial Hackathon)

Precious Chicken — Sun, 20 Feb 2022 20:37:04 +0000

Overview of My Submission

A React Single Page Application deployed as an Azure Web App allowing users to select dates from NASA's Astronomy Photo of the Day archive (accessed via a NASA API).

The app is live at: nasa-apod-picker.azurewebsites.net

Submission Category

Wacky Wildcards

Link to Code on GitHub

PreciousChicken / azure-react-apod

A demonstration of NASA's Astronomy Picture of the Day (APOD) using the React framework, hosted as an Azure web app.

NASA's APOD Picker

A demonstration of NASA's Astronomy Picture of the Day (APOD) using the React framework, hosted as an Azure web app; designed to accompany the Étoile du jour: Deploying a React Web App on the Microsoft Azure Cloud tutorial.

The deployed final version can be seen at: nasa-apod-picker.azurewebsites.net/.

View on GitHub

Additional Resources / Info

Screenshot of app:

Accompanying worked example tutorials:

Video -

Blog -

Étoile du jour: Deploying a React Web App on the Microsoft Azure Cloud

Linking to a heading in the same page in Hugo

Precious Chicken — Sun, 20 Feb 2022 14:58:24 +0000

Linking to another page in markdown in Hugo is easy: i.e. [Auntie Beeb](https://www.bbc.co.uk/) renders as Auntie Beeb - but when you want a link to a heading in the same page I can never remember, so this is a quick aide-memoire:

HTML

In HTML a relative link to an element in the same page looks like:

<!-- <a> element links to the section below -->
<p><a href="#Section_further_down">
  Jump to the heading below
</a></p>

<!-- Heading to link to -->
<h2 id="Section_further_down">Section further down</h2>

Hugo markdown

In Hugo to achieve the same effect:

<!-- <a> element links to the section below -->
[Jump to the heading below](#section-further-down)

<!-- Heading to link to -->
## Section further down

Étoile du jour: Deploying a React Web App on the Microsoft Azure Cloud

Precious Chicken — Fri, 21 Jan 2022 22:16:06 +0000

Introduction

Hosting a React framework Single Page Application as a Web App on the Microsoft Azure can be an extremely frustrating process, for a number of reasons. To start with despite React being (rightly or wrongly) the most popular web framework, there are no specific instructions within the Microsoft documentation for how one might deploy a React Single Page Application (SPA) as an Azure Web App. Far worse however is that the choice of operating system (Windows or Linux) when creating a React web app on Azure is key - in short it is very difficult to use create-react-app to deploy to a base Linux O/S - yet this is both unintuitive and seemingly undocumented (see the What's up with Linux? section).

And continuous integration / continuous development via Github workflow is painful on Azure too; I couldn't find a single explanation which worked from start to finish without inexplicably failing half way through. As an aside the access the Azure App Service, which I'm not using in this demo, wants of your Github account is onerous too (you can't restrict access to private repos - which, er, are meant to be private).

What follows therefore is a fully worked example of how to deploy a React Single Page Application to an Azure web app. GitHub will be used for continuous integration / continuous development. It assumes you have an Azure account, a GitHub account and node / npm installed. Some common command line tools are also used: sed and curl. Development is being done on Manjaro Linux 21.2.1, Node 16.13.2, npm 8.3.2. To give the demo more flavour, I am going to use the NASA Astronomy Picture of the Day API to make a fully featured app - you don't have to do this though, Section 3 can be omitted if you just want to deploy the create-react-app template to Azure.

The steps below are also demonstrated in an accompanying video:

All code is available on the PreciousChicken/azure-react-apod GitHub repository.

1. Set up an Azure Web App

First we need to set up a Web App on Azure. From the Azure portal select Create a Resource and then Web App. Go through the various options until you are faced with a confirmation screen that looks like:

The key point here is that the Operating System is given as Windows, if Linux is selected it will not work (see What's up with Linux?). You might want to check that you are on the free plan too (F1) as opposed to the paid for (B1) which is the default. We've named the app nasa-apod-picker but you will need to give it a different name to prevent conflict.

2. Use create-react-app to generate the React SPA

On our local machine from the terminal enter:

npx create-react-app azure-react-apod

A React SPA will now be created using create-react-app tool.

Once installed change directory into the App:

cd azure-react-apod

3. Make some changes to the React App

If you are just interested in uploading any old React Single Page Application to Azure then you can safely skip this section and leave the default create-react-app settings in place.

However that does seems a bit dull. The (somewhat detailed) instructions below will let us create a rather lovely app where we can pick NASA's Astronomy Photo of the Day by date.

3a. Change app title and description

Change the App title and description to something more meaningful (we could do this by editing the public/index.html file, but using sed from the terminal is quicker:

sed -i 's/React App/NASA APOD Picker/g' public/index.html
sed -i 's/Web site created using create-react-app/Date picker for NASAs Astronomy Picture of the Day/g' public/index.html

3b. Replace logos

The favicon and logos are currently the default create-react-app one; so we'll replace that by using curl to overwrite them with suitably space-themed ones¹ (feel free to use whatever files you want, or simply leave the c-r-a one in place):

curl https://www.preciouschicken.com/blog/images/azure-react-apod_favicon.ico > public/favicon.ico
curl https://www.preciouschicken.com/blog/images/azure-react-apod_logo192.png > public/logo192.png
curl https://www.preciouschicken.com/blog/images/azure-react-apod_logo512.png > public/logo512.png

3c. Replace App.js file

The meat of the change, erase the contents of the src/App.js file and replace with:

import './App.css';
import 'isomorphic-fetch';
import { useState } from "react";
import TextField from '@mui/material/TextField';
import AdapterDateFns from '@mui/lab/AdapterDateFns';
import LocalizationProvider from '@mui/lab/LocalizationProvider';
import enLocale from 'date-fns/locale/en-GB';
import DatePicker from '@mui/lab/DatePicker';
import { format, isWithinInterval } from 'date-fns';

function App() {
  const [apod, setApod] = useState([]);
  const [userDate, setUserdate] = useState(null);
  const [isLoading, setIsLoading] = useState(false);
  const [hasError, setHasError] = useState(false);
  const [errorState, setErrorstate] = useState(null);

  // Earliest photo in APOD
  const apodEarliest = new Date(1995,5,16); 
  // Latest photo in APOD, ie today
  // Hours set to midnight to ensure user selection of date+time always before this date
  const apodLatest = new Date().setHours(23,59,59); 

  //NASA API URL
  const url = "https://api.nasa.gov/planetary/apod?api_key=" + process.env.REACT_APP_NASA_API_KEY + "&date=";

  // Fetches photo from NASA API
  const fetchPost = async (uDate) => {
    setIsLoading(true);
    setHasError(false);
    try {
      const response = await fetch(
        url + uDate
      );
      if (response.status >= 400) {
        setErrorstate(response.status);
        hasError(true);
      }
      const data = await response.json();
      setApod(data);
    } catch (error) {
      setHasError(true);
    }
    setIsLoading(false);
  };

  // Sets userDate as today if null
  // e.g. loads today's photo on first render
  if (!userDate) {
    setUserdate(new Date());
    fetchPost(format(new Date(), 'yyyy-MM-dd'));
  }

  // Adjusts HTML for image vs video
  function MediaType(media) {
    var mediaLink;
    // Determines whether APOD picture is image or video
    if(media === "image") {
      mediaLink = <img src={apod.url} alt={apod.title}/>;
    } else {
      mediaLink = <iframe title={apod.title} src={apod.url} allowFullScreen> </iframe>;
    }
    return (
      <>
        {mediaLink}
        <p><strong>{apod.title}</strong></p>
        <p>{apod.explanation}</p>
        <p>Copyright: { apod.copyright ? apod.copyright : "NASA (Public Domain)" }</p>
      </>
    );
  }

  // Component that serves photo, with error handling
  function Picture() {
    // Return if server has error
    if (hasError) {
      return (
        <p className="userMsg">Error connecting to server. 
          <br/>
          Server response: {errorState}</p>
      );
    }
    // Return when waiting for API
    else if(isLoading) {
      return <p className="userMsg">Loading...</p>;
    }
    // Return once user has selected date and no longer loading and no error
    else if(userDate) {
      return MediaType(apod.media_type);
    }
    // Default return if date not chosen or no error
    // This should never return: userDate is set as today if null on load
    return <p className="userMsg">Select a date between 16-Jun-1995 and today</p>;
  }

  return (
    <>
      <h1>NASA Astronomy Picture of the Day Picker</h1>
      <div className="datepicker">
        <LocalizationProvider dateAdapter={AdapterDateFns} locale={enLocale}>
          <DatePicker 
            minDate={apodEarliest}
            maxDate={apodLatest}
            value={userDate}
            onChange={(newDate) => {
              if (isWithinInterval(
                newDate,
                {start: apodEarliest, end: apodLatest })) {
                setUserdate(newDate);
                fetchPost(format(new Date(newDate), 'yyyy-MM-dd'));
              };
            }}
            renderInput={(params) => <TextField {...params} />}
          />
        </LocalizationProvider>
      </div>
      <Picture />
      <p className="footer"><a href="https://apod.nasa.gov/apod/lib/about_apod.html">About NASA's Astronomy Picture of the Day</a></p>
      <p className="footer">This page is part of the web development tutorial: <br/> <a href="https://www.preciouschicken.com/blog/posts/azure-react-apod/">Étoile du jour: Deploying a React Web App on the Microsoft Azure Cloud</a></p>
    </>
  );
}

export default App;

3d. Replace App.css file

Erase the contents of the src/App.css file and replace with:

body {
  background-color: white;
  color: black;
  display: flex;
  justify-content: center;
}

h1 {
  text-align: center;
}

div.datepicker {
  text-align: center;
  margin-bottom: 1em;
}

p {
  max-width: 75ch;
  margin-right: auto;
  margin-left: auto;
}

iframe {
  width: 100%;
  height: 500px;
}

img {
  width: 100%;
}

p.userMsg {
  text-align: center;
  font-size: x-large;
}

p.footer {
  text-align: center;
  font-size: small;
}

3e. Replace App.test.js

When we continuously deploy using GitHub it automatically runs through any test files prior to deploying. Create-react-app automatically creates ones of these, src/App.test.js, for you and this will fail if you make any changes to the src/App.js file. You can either therefore delete src/App.test.js or amend it to include updated tests; as tests are good I've done the latter:

import { render, screen } from '@testing-library/react';
import App from './App';

test('renders H1 correctly', () => {
  render(<App />);
  const nasaHeader = screen.getByRole('heading', { name: /nasa astronomy picture of the day picker/i });
  expect(nasaHeader).toBeInTheDocument();
});


test('renders link to APOD about', () => {
  render(<App />);
  const nasaAbout = screen.getByRole('link', { name: /about nasa's astronomy picture of the day/i });
  expect(nasaAbout).toBeInTheDocument();
});

3f. Install additional node dependencies

At the terminal:

npm i date-fns isomorphic-fetch @mui/material @emotion/react @emotion/styled @mui/lab

4. Add a workflow to Github

We now need to tell GitHub how to deploy the web app to Azure by adding a workflow.

Therefore create a new file, named azure-webapps-node.yml, located in a new GitHub workflow directory on your machine:

mkdir -p  ./.github/workflows/ && touch ./.github/workflows/azure-webapps-node.yml

Then copy and paste the following into this .github/workflows/azure-webapps-node.yml file (changing the name of the env variables to suit your Azure configuration):

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  # change to your application's name in Azure
  AZURE_WEBAPP_NAME: nasa-apod-picker    
  # change both instances of azure-react-apod to the name of your repo
  AZURE_WEBAPP_PACKAGE_PATH: '/home/runner/work/azure-react-apod/azure-react-apod'   
  # set this to the node version to use if not 16
  NODE_VERSION: '16.x'                
  # NASA API key recorded as GitHub secret, delete if you haven't followed APOD section
  REACT_APP_NASA_API_KEY: ${{ secrets.REACT_APP_NASA_API_KEY }} 

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - name: Set up Node.js
      uses: actions/setup-node@v2
      with:
        node-version: ${{ env.NODE_VERSION }}
        cache: 'npm'

    - name: npm install, build, and test
      run: |
        npm install
        npm run build --if-present
        npm run test --if-present        

    - name: Upload artifact for deployment job
      uses: actions/upload-artifact@v2
      with:
        name: node-app
        path: ./build

  deploy:
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: 'Development'
      url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}

    steps:
    - name: Download artifact from build job
      uses: actions/download-artifact@v2
      with:
        name: node-app


    - name: 'Deploy to Azure WebApp'
      id: deploy-to-webapp 
      uses: azure/webapps-deploy@v2
      with:
        app-name: ${{ env.AZURE_WEBAPP_NAME }}
        publish-profile: ${{ secrets.AZURE_WEBAPP_PUBLISH_PROFILE }}
        package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}

The majority of the above yaml code came from websitebeaver.com's Deploy Create React App to Azure App Services tutorial; however a number of changes were made. The addition of the process.json code step was removed as it was redundant ². Also the AZURE_WEBAPP_PACKAGE_PATH was changed from build to /home/runner/work/azure-react-apod/azure-react-apod, the latter being the location that the build apparently stores the zipped Web App in ready for upload to Azure ³.

5. Create an empty Github repo

We are going to enable continuous deployment (CD) on this project via Github. Therefore point your browser at GitHub and follow the instructions to create a new repo entitled azure-react-apod (though don't add a README, .gitignore or licence). We don't actually need to push the project at this stage however.

6. Download publish profile from Azure

Now navigate to your Web App within the Azure portal and select the Get publish profile option, as shown below, to download a file named your_app_name.PublishSettings (e.g. nasa-apod-picker.PublishSettings) to your local machine:

7. Add secrets to GitHub repo

Following the GitHub guide Creating encrypted secrets for a repository create a new secret with the name AZURE_WEBAPP_PUBLISH_PROFILE and copy and paste the contents of the publish profile you downloaded at the last step. To make this easier on Linux you can copy a file into the clipboard from the terminal (assuming you have xsel installed) like so:

xsel -b < ~/Downloads/nasa-apod-picker.PublishSettings

This secret will allow GitHub to access Azure portal in order to deploy your web app on your behalf.

If you have been following the Astronomy Picture of the Day instructions in Section 3, then you will also need to generate a NASA API key too. Go through the steps again to add this key to GitHub as a repository secret, but this time with the name REACT_APP_NASA_API_KEY ⁴.

8. Git push

All that is left to do is the initial push of the project to our repo and GitHub and Azure will take care of the rest. Digital Ocean features a good tutorial on pushing an existing project to GitHub.

As a reminder the basic terminal commands, changing the github PreciousChicken url to your own, are:

git add .
git commit -m "Initial commit"
git remote add origin git@github.com:PreciousChicken/azure-react-apod.git
git branch -M main
git push -u origin main

The final product

If all has gone to plan your app should be deployed, depending on your Azure set-up configuration, at an address following the format your_app_name.azurewebsites.net, e.g. nasa-apod-picker.azurewebsites.net; as demonstrated:

What's up with Linux?

My default is Linux in any form of virtual machine environment. As you will note however this example uses a Windows base O/S; Linux and React did not play well within Azure.

The nub of the problem is that once you've created a React app the actual part that gets shown to the public is produced by running npm run build and then serving the resulting web pages found in the build folder to the web. Everything outside the build folder is for development and not public view. Often this detail is taken care of behind the scenes, so for instance the Vercel platform knows what a React SPA is and does the donkey work for you (see Stacking Vercel, a GraphQL Apollo Server and React for more). To replicate this in Azure you manually have to upload the contents of the build folder to Azure's /site/wwwroot directory. However this only works if you have chosen Windows as your O/S when creating the Web App, if you have chosen Linux it simply does not work. No explanation given, and a couple of hours consigned to trying to figure out where you have gone wrong.

There are answers by Lutti Coelho and bach vo on StackOverflow which fix this problem by amending the configurations in your Azure dashboard. I can't vouch for this solution, but clearly it adds one more point of failure if you haven't defaulted to Windows in your original set up.

A concrete Python 'Hello World' with the Neovim API

Precious Chicken — Mon, 27 Dec 2021 21:20:42 +0000

Neovim's API is typically called through a Remote Procedure Call (RPC) using the MessagePack-RPC specification. The neovim documentation provides a 'Hello World' example of how to interface with the Neovim API; this post is a worked example of how to implement this - more of a note to self than anything else. I'm running Neovim v 0.6.0 on Manjaro 21.2.0 Qonos linux.

First we need to determine the servername that Neovim has set on startup. Fire up an instance of nvim from the terminal with the command:

nvim -c "echo v:servername"

The -c command option used above causes neovim to run a command on opening, in this case to print out the servername of the instance. Resultantly we should see the servername of the instance in the statusline; it should look a little like: /tmp/nvimt7lIuM/0.

We can now use this address to command the neovim instance remotely. Create a file named nvim-hw-api.py and paste the following - altering the path to match the servername we found above:

from pynvim import attach
nvim = attach('socket', path='/tmp/nvimt7lIuM/0')
nvim.command('echo "hello world!"')

At the terminal run (assuming Python3 is installed):

python3 nvim-hw-api.py

And if all works as forecast we should see hello world! appear in the statusline of the neovim instance!

Groff and the art of letter writing

Precious Chicken — Sun, 11 Jul 2021 15:55:45 +0000

Introduction

Although I've blogged previously about using LaTeX for academic writing I wanted to try groff out for the less complicated task of letter writing - groff being similar to LaTeX but far less popular. Using a typesetting system such as groff instead of word processing software has several advantages: saving documents in plain text is memory efficient, easily searchable and scores highly for digital preservation (proprietary word processing file formats tending to eventually obsolesce).

Below is a worked example on how to use groff to produce a formal letter on Linux. I'm using groff version 1.22.4, Manjaro Linux 21.0.7 and Zathura 0.4.7.

The letter

Open a file titled complaint.me and copy / paste:

.wh 2c hd \" Top margin (header) set to 2cm
.wh -2c fo \" Bottom margin (footer) set to 2cm
.po 2c \" Right margin set to 2cm
.ll 17c \" Line length set to 17 cm
.in 13c \" Address block indented 13 cm
12 High Street 
.br \" Line break
Leicester
.br
LS1 1AB
.br
01234 567890
.br
another@example.org
.sp \" Spaces one line
8th July 2021
.sp
.in 0 \" Address block indent removed
SuperFibre Broadband
.br
Basildon
.br
PO Box 200
.sp
Dear Sir / Madam,
.pp \" Paragraph start
I am outraged and upset with your response\** \" Places footnote
.(f \" Footnote start
\** Reference C/A/O123 dated 1st July 2021.
.)f \" Footnote end
to my recent complaint.  To suggest that 
.i "my chickens" \" Italics
pecking through 
.i "your broadband wire"
was an event outwith my contract with you is ludicrous.  
Any reputable communications company would ensure that their equipment had been through a testing process which ensures 
.b "fair wear and tear"  \" Bold
caused by farmyard animals.
.pp
I insist that you release me from the remaining twelve months of my broadband contract immediately, and furthermore compensate me for the distress caused to my poultry.
.sp 2 \" Spaces two lines
Yours faithfully,
.sp 2 
A N Other

View on screen with Zathura

Assuming you have the Zathura document viewer installed at the terminal type:

groff -me -dpaper=a4 complaint.me | zathura -

This command pipes postscript into Zathura's standard input with the following result:

Output directly to print

Assuming a default printer has been set, then we can also send the postcript directly to the printer:

groff -me -dpaper=a4 complaint.me | lp

As with the previous command this instructs groff to format for A4 sized paper.

Output pdf file

The previous commands avoided the use of PDF all together, but if you would rather save in that format, then adding the -T flag indicates you want the output something other than default postscript:

groff -me -dpaper=a4 -T pdf complaint.me > complaint.pdf

Me, me, me?

As groff is very low-level then macro facilities are grouped together into packages to allow routine operations (such as footnotes) to be done efficiently. Me is one of these packages, although far from the only option; in fact a new macro package, -mk, was released this very month. The -me flag on the command line informs groff this notation is being used, likewise the recommended file extension to use is .me.

vim-groff-viewer

If you are a Vim / Neovim user then the vim-groff-viewer plugin wraps these commands into a couple of key mappings so you don't have to leave the comfort of Vim for the command line.

Conclusion

I guess the one obvious question is why use groff at all, as opposed to LaTeX? Possibly because there is something satisfying about learning modern uses for an application whose lineage dates back to 1964. Or maybe just, to paraphrase George Mallory, because it is there? Thoughts, observations? Feel free to add below.

Further References

How to format academic papers on Linux with groff -me. Gentle introduction to groff and the -me package.
Writing Papers with NROFF using −me. Introductory paper written by Eric P. Allman, author of the -me package.
-me Reference Manual. Full reference for -me, again written by Allman.
groff_me man page. Linux groff_me(7) Miscellaneous Information Manual.
groff/troff: MUH MINIMALIST Documents Luke Smith's introductory video to groff, though uses the -ms package rather than -me.
The GNU Troff Manual. The definitive groff manual. However only one sentence on the -me package.

TiddlyWiki5, Raspberry Pi and Vim: A guide for the command line aficionado

Precious Chicken — Sat, 15 May 2021 21:03:28 +0000

Introduction

The practice of personal information management has always left me unsatisfied; a square hole in the puzzle of life that you just don't have a square peg for. After looking into options for square pegs I've opted for a zettelkasten method implemented via a TiddlyWiki.

I wanted to host this on a Raspberry Pi and access this on all the computers on my local network (e.g. tablets, phones, etc). However I also wanted the ability to directly edit tiddlers (the basic unit of information in TiddlyWiki) using neovim, so I don't always have to go through a browser. Seen as I was using neovim I also wanted to be able to initiate a tiddler from the command line - copying a title in from whatever I'm reading and having it CamelCase create a tiddler in Tiddlywiki as per the above video.

This proved problematic, which I explain in the long-winded and unnecessary background, so I thought worth recording the steps I took on both the Raspberry Pi and my Linux box below.

Long-winded and unnecessary background

This turned out to be more difficult than I thought. The vanilla node.js installation of TiddlyWiki doesn't play well with direct editing of tiddlers, i.e. editing the tiddlers requires a manual node restart before it appears in the browser. OokTech has an executable that does allow direct editing - but this doesn't work on a Raspberry Pi. Not to fear however as OokTech also have a plugin which gives the same functionality.

So I used this for a number of months - however there was a snag. Every so often a tiddler would appear to save correctly after a browser edit, but on return the changes would have dissapeared. Not knowing exactly what the problem was my first thought was to upgrade to the latest version, in case this solved the problem¹. So I updated both TiddlyWiki5 and the OokTech plugin; at which point it stopped working altogether.

Rather than starting over again I wondered if there might be another way.

1. Raspberry Pi

First the steps you need to take with the Pi. This assumes you've installed the standard Pi OS (see Version control for the software used).

1a. Enable Port 8080

As blogged about previously, if you are using the default Raspberry Pi OS ports are shut down by default. These therefore need to be opened by entering the following into the Pi terminal:

sudo apt install ufw
sudo ufw allow ssh
sudo ufw enable
sudo ufw allow 8080

This installs Uncomplicated Firewall onto your Pi, enables it and then allows the port that your TiddlyWiki will be listening on.

1b. Install TiddlyWiki5 and Nodemon

At the terminal:

npm install tiddlywiki nodemon -g

The -g flag installs the software globally, rather than in a particular folder.

1c. Initiate a wiki

We now use the TiddlyWiki software just installed to initiate a fresh wiki. At the terminal:

tiddlywiki wiki --init server

This creates a directory called wiki which includes server-related components. You can change the name of the directory to whatever you wish, but the remainder of the instructions assume the directory is called wiki.

1d. Find your IP

Now we have to find the IP address of your Raspberry Pi on your local network and make a note of it. For the purposes of this tutorial we are going to say it is 192.168.0.12.

1e. Starting TiddlyWiki with nodemon

Ok, so now the fun bit, starting. If you weren't bothered about editing the tiddlers in a text editor you could just simply go for:

tiddlywiki wiki --listen host=192.168.0.12

And if you go to http://192.168.0.12:8080/ in your browser you should see a new wiki.

All well and good, however this makes editing tiddlers in a text editor awkward. Using the above method means edits do not get reflected in the browser - or not without manually restarting the node instance of TiddlyWiki.

This is where nodemon comes in. Nodemon watches a directory and restarts a node process any time there is a change in that directory. It is commonly used for development purposes - if you are working on a JavaScript file you want node to be refreshed and reloaded every time you make a save - and not have to manually restart. However it can be used to look for changes to tiddlers, rather than code.

To do so enter at the terminal this intimidatingly long command:

nodemon --delay 30 -e tid --ignore 'wiki/tiddlers/$*.tid' --watch wiki/tiddlers/ $NVM_BIN/tiddlywiki wiki --listen host=192.168.0.19

At this point you can simply walk away leaving the terminal open and running if you have been directly inputting commands into your Raspberry Pi. If you are accessing your Pi remotely via SSH then close the terminal window itself without exiting the running program (on Manjaro i3 this is done via the mod-Shift-q keypress, but your distro will likely be different). This method of exit is required because closing the terminal via standard methods (e.g. ctrl-c followed by exit) will likely also kill nodemon.

So what does this do? The flags / arguments to nodemon are as follows:

--delay 30 ensures that after a change nodemon waits 30 seconds before restarting. If this did not occur then the server would be constantly restarting.
e tid tells nodemon that it needs to look for changes in files ending in tid, that is in tiddlers.
--ignore 'wiki/tiddlers/$*.tid' ignores tiddlers that are generated by the system (which by convention start with $) and not the user.
--watch wiki/tiddlers/ tells nodemon the folder to watch where changes will happen.
$NVM_BIN/tiddlywiki wiki --listen host=192.168.0.19 lastly this is the previous command as at the start of the section, but as we aren't running the command via node we have to explicitly tell nodemon where to find the tiddlywiki executable (in $NVM_BIN).

2. Desktop / Local machine

Switching back from the Raspberry Pi to your Linux machine, the following will allow (relatively) seamless editing in vim / neovim.

These steps assume you have generated SSH keys allowing you to access your Pi from your Linux machine - if not follow DigitalOcean's excellent How To Set Up SSH Keys tutorial before anything else.

2a. Mapping a drive

So that we have the tiddlers accessible on our local machine, the wiki directory on the Pi needs to be mounted. Although the actual mount command is given in the tw shell script below, some initial steps need to be taken for this to work.

First create the mountpoint (e.g. where on your local system you want the tiddlers to appear):

mkdir ~/wiki

Now we need to install sshfs which allows us to mount a drive over SSH. I'm using Manjaro so it is:

pamac install sshfs

If you are using Ubuntu or derivative then it will be sudo apt install sshfs.

To make things a bit easier once we've decided where our tiddlers are mounted we will set this path as an environment variable so we aren't having to retype the path again. We do this by editing our ~/.bash_profile (or whatever shell we use - I use zsh so it is ~/.zshenv)² to include:

export TIDDLYWIKIPATH=$HOME/wiki/tiddlers/

2b. Vim plugin

The vim-tiddlywiki plugin is absolutely outstanding for managing tiddlers in Vim. If you don't use a plugin manager, install it into Neovim as follows:

 git clone --depth 1 https://github.com/sukima/vim-tiddlywiki ~/.config/nvim/pack/sukima/start/vim-tiddlywiki

If you are using original Vim rather than Neovim, then the path above will need amending as Vim does not use the .config directory. Or simply use a plugin manager.

This plugin not only means that the tiddler format is recognised, but also allows you to create new tiddlers with the right metadata in place and jump to the other tiddlers using CamelCase links.

For full use you will also need to let the plugin know where you keep your tiddlers by adding the following line to your ~/.config/nvim/init.vim or ~/.vimrc, i.e.:

let g:tiddlywiki_dir=$TIDDLYWIKIPATH

2c. Command line editing

To make the process buttery smooth I want to automate my workflow. This being: read an article on the net, copy/paste the title of the article and have neovim open a tiddler already named using CamelCase. So for example, let's say I want to take some notes on Hayek's paper on open markets: The use of knowledge in society. I want to open a terminal, type tw The use of knowledge in society and start editing a tiddler named TheUseOfKnowledgeInSociety.tid in neovim.

Create a file named tw wherever you keep user-specific executable files (so for me this is ~/.local/bin/tw, but if you are using Ubuntu it will likely be ~/bin/tw) and copy / paste the following code. Alternatively you can download from the github repo. This code assumes that you've created your mount point at ~/wiki as above, if not then you will need to change the sshfs line.

#!/bin/bash

# Opens a new tiddlywiki tiddler named after arguments that follow commands
# Uses vim-tiddlywiki plugin to create metadata
# https://www.preciouschicken.com/blog/posts/tiddlywiki5-raspberry-pi-guide/

# Checks if pi is mounted, if not mounts in folder created earlier
if (( $(mount | grep -e 'pi@'| wc -l) == 0 ));
then     
    sshfs pi@192.168.0.19:/home/pi/wiki "$(dirname $TIDDLYWIKIPATH)" -o reconnect
fi

# Replaces non-alphanumeric characters in arguments
arguments=$(echo $@ | sed "s/[’]//g")
arguments=$(echo $arguments | sed 's/[^a-zA-Z0-9]/ /g')

# CamelCase converts all arguments
tid_title=''
for word in $arguments
do
    # Changes argument to lowercase
    lowercasevar=${word,,}
    # Capitalises first character of argument
    sentencevar=${lowercasevar^}
    tid_title+="$sentencevar"
done

# Opens neovim with correct tiddler name
if [[ -f $TIDDLYWIKIPATH$tid_title'.tid' ]]
then
    # Updates metadata if tiddler exists
    nvim -c TiddlyWikiUpdateMetadata $TIDDLYWIKIPATH$tid_title'.tid'
else
    # Creates tiddler if not found
    nvim -c TiddlyWikiInitializeTemplate $TIDDLYWIKIPATH$tid_title'.tid'
fi

Then make this executable:

chmod a+x ~/.local/bin/tw

On next restart of terminal, typing tw The use of knowledge in society should create a new tiddler ready for us to type. If the tiddler already exists then the shell script will recognise that and update the metadata instead.

The shell script does its best to cope with non-alphanumeric characters. So apostrophes are deleted (We're not really strangers gets changed to WereNotReallyStrangers) while it substitutes other non-alphanumerics characters for a space (so Knee-deep in the Big Muddy ends up as KneeDeepInTheBigMuddy). I'm sure there will be edge cases which will not work out - feel free to comment below.

A note on spawning

It is worth noting that nodemon can be a bit difficult to stop once it is started. One option is to find the process running and stop them individually, to quote one particular solution:

ps aux | grep -i nodemon, find out which process number nodemon is, then issue a kill -9 [process ID]

Or alternatively reboot the Pi. That can be much easier...

Version control

If this doesn't work for you, it might be due to version conflicts. At the time of writing I was using:

Pi: Raspberry Pi 3 Model B Rev 1.2 running Raspbian GNU/Linux 10 (buster) armv7l. Node v14.11.0, npm v6.14.8. TiddlyWiki v5.1.23.
Desktop: Manjaro Linux 21.0.7 Omara. zsh v5.8. neovim v0.4.4.

Conclusion

This is not perfect - for instance there is a 30 second delay between creating a tiddler on the command line and it being reflected in the browser. And there are probably all sorts of circumstances where the bash script will not work out. Square peg in a square hole or Linux kludge? Comments, feedback, etc below.

Addendum: Other options

A thread on the TiddlyWiki mailing list, subsequent to this post, pointed out a number of alternatives to using nodemon:

watch-fs plugin

The watch-fs TiddlyWiki plugin states that it "enables TiddlyWiki to watch the change in your disk, and if you edit one of your tiddler using editor likes VSCode and save it on the disk, the change will immediately reflected in the browser."

TiddlyWiki API

An option suggested by Saq Imtiaz is instead of saving newly created tiddler files directly into the wiki directory, one could save the tiddler with a PUT request via cURL using the TiddlyWiki API; for example:

curl -X PUT -i 'http://192.168.0.12:8080/recipes/default/tiddlers/NewTiddlerTitle' --data '{
 "tags": "firstTag anotherTag",
 "creator": "gene",
 "modifier": "gene",
 "text": "The use of knowledge in society"
}' -H "X-Requested-With: TiddlyWiki"

Using this example as a basis a Vim plugin or a shell script called by Vim could be written.

TW5-Bob plugin

Although as previously covered this did not quite work out for me, the TW5-Bob plugin is also an alternative given that it offers "two-way real-time syncing between the browser and file system".

For clarity I suspect the problem was something to do with how I had installed or configured Bob, rather than an error in the system itself. But a fresh install is never a bad idea anyway. ↩
You could also use your ~/.bashrc or ~/.zshrc, but I think the profile / env files are preferable. ↩