DEV Community: Henrik Sommerfeld

Usability Learnings from Building a CLI

Henrik Sommerfeld — Tue, 17 Nov 2020 21:36:00 +0000

From the past months of iterating on a CLI for managing micro services in our company, I've drawn some conclusions regarding usability.

The problem

As the number of commands, arguments and flags have grown, so has the risk of confusion. Naming things is not the easiest thing in our industry – should the command for creating a service be called create, init or add? Since it's idempotent (can be run multiple times), should it instead be called ensure?

After changing this back and forth, to ensure/create/add confusion, we sort of gave up and used aliases.

The solution

We use Oclif (The Open CLI Framework), which provides a few features we can use.

Aliases

By using aliases, you don't have to think too hard on whether it's supposed to be get/set, show/edit or list/add/remove.

export default class UpstreamsShow extends Command {
  static aliases = ['upstreams:get', 'upstreams:list']
  static description = 'Show service upstreams'
  static usage = 'upstreams:show'
  static examples = [
    `$ aw upstreams:show -s hello-cats -e dev
Lists the upstream services for service 'hello-cats' in environment 'dev'.`,
    `$ aw upstreams:show
Lists the upstream services for current service (from repo). You will be prompted for environment.`,
  ]
  ...
}

Help

By making sure all commands have a description, usage and possibly examples, --help becomes better. Examples are really good for commands with several arguments or flags. When using topics in Oclif, it's also helpful to add command descriptions in package.json as described on the topics documentation page

Autocomplete

I also find the autocomplete plugin helpful, since you can just tab your way to the full command.

Search for Static Website Without External Service

Henrik Sommerfeld — Thu, 23 Jul 2020 21:28:40 +0000

When you have a static website, there are a few things that you usually don't have out-of-the-box. One such thing is search. You can argue that you don't need it, but if you want it and your site isn't that large, I'll describe how I've set it up without an external service.

This post is part 5 of my Hugo Pipeline Series, so I'll use Hugo as the example here, but I've done a similar setup with this Gatsby plugin as well.

The steps I use are the following:

Create a json file with everything I want in my search index (Hugo)
Create a search index from the json file (NodeJS)
Download and load the index (Web Browser)
Perform search and present results (Web Browser)

1. Create file to index

I have a Hugo layout for the indexable content where I output all pages of the types I want. The type article is what all blog posts use and shortcuts-windows7 is a special layout I want to include in search (see it here, if you're curious). My About page is not included, since I figure you can find that anyway if you can find the search feature.

Title, relative permalink, tags, the full content as plain text, the summary (excerpt) and the date (formatted and raw), are the fields I picked as searchable + available for search result presentation.

I also exclude the list page named Articles (that I don't know how to get rid of, please create a PR if you know how and want to help).

layouts/search-index/single.html

{{- $.Scratch.Add "index" slice -}}
{{- range where .Site.Pages "Type" "in" (slice "article" "shortcuts-windows7") -}}
    {{- if ne .Title "Articles" -}}
        {{- $.Scratch.Add "index" (dict "title" .Title "ref" .RelPermalink "tags" .Params.tags "content" .Plain "summary" (partial "summary.html" .) "dateformatted" (dateFormat "2, January 2006" .Date) "dateiso" (time .Date)) -}}
    {{- end -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

This layout needs to be referenced and for that I have search-index.md which is empty, except for the frontmatter.

---
date: "2017-06-21T06:51:27+02:00"
title: "search index"
type: "search-index"
url: "data-to-index.json"
---

2. Create index

Now that we have something to index, it's time to switch to NodeJS land and install Lunr, yarn add lunr. I have a script that reads the file created in the previous step (data-to-index.json) and creates a new file, search-index.json in the output directory (public). This is also the place to configure Lunr with boosting and such. I'm not good att tweaking search, so these settings are pretty basic. This was written before I got more heavily into NodeJS development, but it has worked without problems for a few years now.

'use strict'

const lunr = require('lunr');
const fs = require('fs');
const path = require('path');

const outputFilePathParameter = process.argv && process.argv.slice(2)[0];
const inputFilePath = path.resolve(__dirname, '../public/data-to-index.json');
const outputFilePath =  outputFilePathParameter || path.resolve(__dirname, '../public/search-index.json');
console.log('Reading ' + inputFilePath);
const documentsToIndex = require(inputFilePath);
const store = {};

console.log('Indexing ' + inputFilePath);
const searchIndex = lunr(function () {
  this.ref('ref')
  this.field('title', {boost:10}),
  this.field('tags', {boost:5}),
  this.field('content')

  documentsToIndex.forEach(function (doc) {
    store[doc.ref] = {
        'title': doc.title,
        'summary': doc.summary,
        'dateiso': doc.dateiso,
        'dateformatted': doc.dateformatted
    };

    this.add(doc)
  }, this)
})

console.log('Saving index at ' + outputFilePath);

const dataToSave = JSON.stringify({
    index: searchIndex,
    store: store
});

fs.unlink(outputFilePath, function(err){

    if (err && err.code !== 'ENOENT')
        throw err;

    const options = { flag : 'w' };
    fs.writeFile(outputFilePath, dataToSave, options, function(err) {
        if (err) 
            console.error(err);
        else
            console.log('Saved index at ' + outputFilePath);
    });
});

This is run with an npm script after Hugo has produced its output.

> node build/index-search.js public/search-index.json

Reading /Users/henrik/Code/blog-hugo/public/data-to-index.json
Indexing /Users/henrik/Code/blog-hugo/public/data-to-index.json
Saving index at public/search-index.json
Saved index at public/search-index.json
✨ Done in 0.52s.

To have the search index available during development, I run the Hugo command twice. This isn't perfect, but since hugo server (like most dev servers) doesn't save the files on disk, this is necessary and not really a problem. The npm script looks like this: hugo && npm run index && npm run hugo:watch (see full package.json here).

3. Loading index

Most of my visitors come straight to a post from a Google search, so I'm probably the biggest user of the site search myself (maybe the only one 😳). Therefor I don't want the search index to be downloaded before the user has shown an intention to use the search feature. The index is currently a download of 134 kB (compressed), which I think is fine considering that people are watching video on web pages and that the alternative of using an external service has several other drawbacks (complexity, cost, etc). Still, the index size is worth keeping an eye on and this setup requires error handling (if the download fails or the user has started to type before the download is complete).

The index are downloaded through a regular fetch call when the search dialog is opened (the open function).

const search = {
  isOpen: false,
  textInSearchBox: '',
  index: null,
  store: null,
  indexLoadFailed: false,
  indexLoading: false,
  hits: [],
  open: function () {
    blog.isModalOpen = true;
    this.isOpen = true;
    this.textInSearchBox = '';
    this.indexLoadFailed = false;
    this.downloadIndex();
  },
  downloadIndex: function () {
    if (this.index) return;

    this.indexLoading = true;
    this.fetchIndex().then(({ index, store }) => {
      this.index = window.lunr.Index.load(index);
      this.store = store;
      this.indexLoading = false;
      this.searchBoxChanged(this.textInSearchBox);
      console.log("🔍 Search index downloaded")
    });
  },
  fetchIndex: function () {
    return fetch('/search-index.json')
      .then(this.handleFetchResponse)
      .catch(this.handleFetchResponse);
  },
  handleFetchResponse: function (response) {
    this.indexLoadFailed = !response.ok;
    return response.ok && response.json ? response.json() : this.index;
  },
  ...
}

4. Searching and presenting results

I have touched on this in my previous post about Alpine.js, so go there for more code, but this is simply about calling the search function on the Lunr index. Since everything is in memory, I call the search function on every keypress in the searchbox.

Good luck in implementing your own site search!

Alpine.js – Benefits and Limitations

Henrik Sommerfeld — Mon, 29 Jun 2020 20:14:54 +0000

This post is part of my Hugo Pipeline Series, but the benefits and limitations I discuss are not specific to using Alpine.js together with Hugo.

What is Alpine.js and why?

Alpine.js is meant to be used with an existing HTML document (server-side rendered HTML that isn't produced by a JavaScript framework), just like plain JavaScript or jQuery. This is the way it was done when JavaScript was introduced and how it's done today when using Hugo.

I use JavaScript for the following features on my blog:

Opening and closing a hamburger menu
Lazy loading of images
Lightbox for images and code
Providing site search
Easter eggs
Comments and analytics (but that's not my code)

I started out using jQuery when migrating the blog from WordPress to Hugo, which I think was the obvious choice at the time. Later I migrated to plain JavaScript. That was fairly straight forward and the code looked quite similar after the migration, although a bit lengthier. This worked fine and I didn't need a library at all, so why add one again?

Looking at what the JavaScript I have is doing, we can see where I can benefit from using a library:

Changing CSS classes on an element, mostly body
Adding event listeners to handle interactions
Rendering search results

In all these areas I benefit from using declarative code, it's just less code and easier to read. This is where Alpine.js comes in. Alpine.js has borrowed a lot from Vue.js when it comes to syntax, but work with an existing DOM. I haven't used Vue.js and that can make you feed a bit excluded when the documentation explains something by saying that "it works just like in Vue.js". It is however a small API, so I found it easy to get started with.

An example with keyboard navigation

This is the relevant code I use for showing/hiding outline for the element that has focus, based on whether the user is navigating by mouse or keyboard.

HTML

<body x-data="window.blog" :class="{ 'keyboard-navigation' : keyboardNavigation }"
  @mouseup="keyboardNavigation = false" @keydown.tab="keyboardNavigation = true">
…
</body>

JavaScript

window.blog = {
  keyboardNavigation: false
}

CSS

body.keyboard-navigation a:focus {
  outline: 2px solid var(--accent-color);
}

Doing this with imperative code is simply messier, so this is one example where Alpine.js helps.

An example with search results

Another example is the search results I present. This is a situation where I could just add any of the popular JavaScript frameworks, since this part of the page creates the HTML in JavaScript. This is also a situation where pure JavaScript quickly gets messy, like concatenating strings and setting innerHTML, especially if you need event listeners on those new elements.

<div class="search-results-container">
  <div id="search-output" x-show="search.textInSearchBox">
    <div id="no-results-message" x-show="search.store && search.textInSearchBox && !search.hits.length">
      No matching posts found. You can use wildcards and search only in titles, e.g. <code>title:iot</code>
    </div>
    <div id="index-loading-message" x-show="!search.indexLoadFailed && search.indexLoading && search.textInSearchBox">
      <span class="icon-spinner" aria-hidden="true"></span> Loading search index, please wait...
    </div>
    <div id="index-failed-message" x-show="search.indexLoadFailed && search.textInSearchBox">
      Search index failed to download 😢
    </div>
    <div id="number-of-hits-message" x-text="search.getHitsText()" x-show="search.hits.length"></div>
    <ol class="result-list" x-show="search.hits.length" x-ref="hits">
      <template x-for="hit in search.hits" :key="hit.ref">
        <li>
          <h2><a :href='hit.ref' x-text="search.fromStore(hit).title"></a></h2>
          <div class="entry-meta">
            <time class="published" :datetime="search.fromStore(hit).dateiso">
              <svg class="icon icon-calendar"><use xlink:href="#icon-calendar"></use></svg>
              <span x-text="search.fromStore(hit).dateformatted"></span>
            </time>
          </div>
          <p x-text="search.fromStore(hit).summary"></p>
        </li>
      </template>
    </ol>
  </div>
</div>

search is the object that contains the functions and properties being referenced in this mark-up. It's in a separate JavaScript file not included here, but hopefully you get the point of the declarative approach instead of doing this in imperative JavaScript.

Benefits

Hopefully I've been able to highlight some of the benefits in the examples above, but to conclude:

Easy to get started with
Same kind of declarative data binding that we love with other JavaScript frameworks

Limitations

Now to the interesting stuff, things that Alpine.js not so good for – the stuff you generally don't find in documentation or tutorials.

You cannot have nested components or have them communicate easily.
The page (DOM) isn't updated when updates are triggered by non-interactive events.
Doesn't work with Turbolinks

In the case of my blog, I made the body tag the Alpine component, which works just fine as I'm mostly setting different CSS classes on the body tag anyway. For a more complex use, A guide to Alpine.js component communication describes how you can have sibling components talk to each other and have the DOM react to non-interactive events, see answer on GitHub. A non-interactive event is when the user hasn't clicked or typed anything, such as when data is fetched (a promise is resolved) and you set that data to a property.

The theme switcher I have reacts to theme (light/dark) changes in the operating system and also when the theme setting in localStorage is changed. The code I have to listen for those events can update a property bound to the Alpine component, but it won't update the DOM. Rather than implementing some involved dispatch mechanism, I prefer to use pure DOM manipulation for these situation, starting with document.getElementById() and setting the element's properties.

Another thing to note is that if you're using Turbolinks (which gives navigation without full page reloads), it doesn't seem to work with Alpine.js.

Conclusion

Overall, I think the migration from plain JavaScript to Alpine.js was worth it for me. The code is easier to read now and that's what I was aiming for. I just wish I understood the limitations earlier, that would have saved some time.

There are also features of Alpine.js that I don't use, namely animations and x-ref when using a .js file. Maybe this is because I came from a world of plain JavaScript and animations in CSS. It seems that the convention when using Alpine.js is to include all JavaScript in <script> tags rather than separate .js files. I didn't go that route and found that document.getElementById() works just as well as passing x-refs around (they don't seem to work in .js files otherwise). Using a <script> tag is probably better as long as the code is as simple as in my keyboard navigation example above, but as it grows, I find it better to use a separate file 🤷‍♂️

Hugo Pipeline Series – Developing and Deploying

Henrik Sommerfeld — Mon, 22 Jun 2020 13:53:06 +0000

In terms of making code changes to my Hugo site, I'll focus on the JavaScript parts, since Hugo templates and CSS isn't much to talk about. I use a few libraries that I've installed with npm and those need to be processed before they are sent to the browser. The JavaScript code I have written myself, does not have that requirement. In that case it's just a matter of how old browsers I want to support.

So, by splitting libraries (installed through npm) from my own code, I'm able to rely solely on Hugo's file watcher with live reload. For development (using hugo server), the libraries are built once, and the file with custom code is served to the browser as is. I use Browserify for the libraries and that's good enough for my needs. If you need something more powerful, you might consider Victor Hugo that comes with Webpack preconfigured. I could also have referenced the libraries directly from a public CDN, but that would make it harder to see when there's a new version of a library and more importantly – my site wouldn't work on localhost without an Internet connection.

{{ if .Site.Params.MinifyBundles }}
  {{ $opts := (dict "minified" true "compact" true "noComments" true) }}
  {{ $main := resources.Get "main.js" | babel $opts }}
  {{ $libs := resources.Get "libraries.js" }}
  {{ $bundle := slice $libs $main | resources.Concat "bundle.js" | resources.Fingerprint "sha512" }}
  <script src="{{ $bundle.RelPermalink }}" integrity="{{ $bundle.Data.Integrity }}"></script>
{{ else }}
  {{ $libs := resources.Get "libraries.js" | resources.Fingerprint "sha512" }}
  <script src="{{ $libs.RelPermalink }}" integrity="{{ $libs.Data.Integrity }}"></script>
  {{ $mainDev := resources.Get "main.js" | resources.Fingerprint "sha512" }}
  <script src="{{ $mainDev.RelPermalink }}" integrity="{{ $mainDev.Data.Integrity }}"></script>
{{ end }}

For the "production build", I just minify the code and concatenate the libraries with my custom code into a single file. I use Babel from within the Hugo template (available since Hugo 0.70) which removes the need for yet another npm script.

When doing code changes I always create a new branch in Git. As I described in the previous post, anything pushed to master is automatically built and deployed (if build is successful) to the live site without any tests. By creating a pull request and using GitHub Actions, I can feel confident that I haven't broken anything when the checks are green.

These are the steps:

Create a feature branch in Git
Develop the new feature in the new branch
Create a pull request from the feature branch to master branch (through GitHub)
Netlify builds & deploys a canary release on a separate URL (this is reported back to the PR in GitHub
When the canary release URL responds with a 200 status code, Cypress tests are run in GitHub Actions
Test results are shown in the PR in GitHub. If green, I merge the feature branch into master
Netlify deploys the latest code from the master branch to the live site (Prod)
GitHub Actions workflow waits 2 min and then performs a Lighthouse audit against the live site

When I create a pull request, either through GitHub's web interface or preferably by using their CLI (gh pr create), two things happen. (1) Netlify deploys a preview of the site (canary release) and (2) GitHub Actions runs my Cypress tests against that deployed site.

Running the tests against a deployed site (as opposed to a dev server on localhost) transform the tests from function tests to end-to-end tests. Before I had this setup I once broke the comments feature (using Disqus) by fiddling with Content Security Policy headers, something I would have caught today when running the tests against a deployed site.

Since both building + deploying a site and installing Cypress to run the tests take some time, it's great that it can run in parallel. The yarn install run in GitHub Actions is typically done before the canary release is deployed by Netlify, so I use an action that waits for a 200 response from Netlify and then run the tests. Super happy with this setup!

master-pull-request.yml

name: Tests

on:
  pull_request:
    branches:
      - master

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - name: Install
        run: |
          yarn install
      - name: Waiting for 200 from the Netlify Preview
        uses: jakepartusch/wait-for-netlify-action@v1.1
        id: waitFor200
        with:
          site_name: "henriksommerfeld"
          max_timeout: 300
      - name: Integration tests
        run: |
          CYPRESS_BASE_URL=${{ steps.waitFor200.outputs.url }} npm run cypress:cli     

      - name: Run Lighthouse
        uses: foo-software/lighthouse-check-action@master
        id: lighthouseCheck
        with:
          accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
          author: ${{ github.actor }}          
          branch: ${{ github.ref }}
          urls: ${{ steps.waitFor200.outputs.url }}
          sha: ${{ github.sha }}
          prCommentEnabled: true
          slackWebhookUrl: ${{ secrets.LIGHTHOUSE_CHECK_WEBHOOK_URL }}

As you can see from my workflow file above, I get a notification in Slack (I have a personal workspace) when the canary release is deployed and when a Lighthouse audit is completed (after the Cypress tests). If I've made changes to the look and feel (like a CSS change), I naturally take a look at the deployed canary release, but if not, I'll just go ahead and merge (which triggers a deploy to the live site).

One thing to say about the Lighthouse audits is that they are of questionable value for the canary releases. The performance score is always lower on the first page load, so you need to load at least two pages to get a fair result. Secondly, the SEO score won't say much since the canary release is purpously blocked from indexing and has a different URL than the canonical URL set for the site. For this reason, I run a Lighthouse audit on the live site as well. In this case I can't wait for a 200 response, so I'll just wait a bit longer than a deploy normally takes and run the audit after that.

master-push.yml

name: Tests

on:
  push:
    branches:
      - master

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - name: Wait 2 min for Netlify Deploy to complete
        uses: jakejarvis/wait-action@master
        with:
          time: '2m'

      - name: Hit once
        uses: wei/curl@master
        with:
          args: https://www.henriksommerfeld.se

      - name: Run Lighthouse
        uses: foo-software/lighthouse-check-action@master
        id: lighthouseCheck
        with:
          accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
          author: ${{ github.actor }}          
          branch: ${{ github.ref }}
          urls: 'https://www.henriksommerfeld.se/about,https://www.henriksommerfeld.se'
          sha: ${{ github.sha }}
          prCommentEnabled: true
          slackWebhookUrl: ${{ secrets.LIGHTHOUSE_CHECK_WEBHOOK_URL }}

The Slack notifications from a "production deploy" looks something like this:

Hugo Pipeline Series – Editing and Deploying

Henrik Sommerfeld — Mon, 15 Jun 2020 07:09:47 +0000

In the first post in this series I said that I don't use a proper Content Management System (CMS). To manage my content I use a code editor (VS Code) and Git (GitHub). The advantage of having my content in text files in the same repository as the code is huge. No database to backup or sync between environments.

See Scott Hanselman looking at all his blog posts from 2005 and onwards. It's all XML files, which might not be trendy today, but is still human readable and easily convertible to another text format. I use Markdown now, but the important thing is that it's text files that can be converted if I need to. I have already changed the code highlighting once, which was an easy search-and-replace operation. When I did the same in WordPress some years ago, it was…harder.

Different workflows for content and code changes

Since correcting a simple spelling mistake using a static site generator requires a new build, it's beneficial to reduce the amount of stuff that needs to happen between a push and a deploy. Therefore I have split content changes and code changes (see next post) into two different workflows.

Reduce the stuff to build

I have some npm packages and have split dependencies from devDependencies to reduce the amount of packages that need to be installed for a content change. Installing dependencies with yarn install --production installs 44 MB of node_modules, while yarn install installs 110 MB of node_modules.

My netlify.toml file:

[build.environment]
  HUGO_VERSION = "0.72.0"
  YARN_VERSION = "1.22.4"
  YARN_FLAGS = "--production"
  NODE_ENV = "production"

My content deployment workflow consists of pushing directly to the master branch. That triggers a web hook that does a build and deploy by Netlify. This requires the discipline not to push code changes directly to the master branch, but since I'm the only developer, that's a policy easy to enforce.

Images

The biggest speed difference, both in building on my own laptop and in deploying, is how images are handled. This is strikingly obvious when I compare to the two Gatsby sites I have. When writing or editing a blog post, I have the Hugo development server running. When I add a new image, Hugo creates the different versions (sizes) of the image that my templates specify – once. Then I commit the Markdown file and the images (original and generated) to Git. That's it, the image processing for that specific image is done, it won't ever have to be processed again unless I want to. No CPU cycles are spent on any server to generate images and that saves time in deployment. It also saves time against the free build minutes quota I have, currently 300 minutes per month on Netlify.

The local dev server never have to re-generate images it has already processed, so it starts quicker as well. Just as a test when I created a new shortcode with new image sizes, I ran Hugo with the option to re-generate everything from scratch (Markdown, SCSS and images), hugo --ignoreCache:

                   | EN   
-------------------+------
  Pages            | 197  
  Paginator pages  |  42  
  Non-page files   |  97  
  Static files     |  27  
  Processed images | 459  
  Aliases          |  63  
  Sitemaps         |   1  
  Cleaned          |   0 

Total in 39741 ms

All of the 8 cores on my laptop where working here and it took 40 seconds. This is something I won't most likely ever have to do again, so 40 minutes would have been fine in this extreme situation as well.

Total build & deploy time

As I mentioned, I use Netlify to both build and deploy. That process takes about 2 minutes, which regardless of everything else, is better than the 25 minutes Wes Bos is experiencing.

Hugo Pipeline Series – Intro

Henrik Sommerfeld — Mon, 15 Jun 2020 07:01:26 +0000

Who is this for?

This is the first part in a series of posts where I'll describe how I created my personal blog with a fair amount of work to achieve simplicity. The intended audience is:

Myself (as documentation)
Someone with their own Hugo blog, looking to improve automation
Someone interesting in learning one way of working with a Hugo blog

This will be an overview of the different pieces I use, it won't be a detailed description or tutorial, but I'll try to link to libraries and concepts.

TL;DR

Repo is here. README shows how to run it locally
A build & deploy log is here

Site characteristics

The site I'll be describing has the following characteristics:

Annual running cost of nothing, except for the domain name
Static site (built with Hugo)
400+ images
< 2 minutes combined build & deploy time
Custom domain with HTTPS
Lazy loading of images (with blurry preview when JS is available)
Search with Lunr
Lightbox for images and code
Dark/light mode
Alpine.js for custom JavaScript code
Cypress tests that run automatically on pull requests
Lighthouse audit results automatically run after deploy
All code is in a public repo on GitHub
Runs locally without Internet access

What it doesn't have:

Dynamic data loading (except for comments and analytics)
SPA characteristics (it has full page reloads)
Offline availability (it's not a PWA)
A proper CMS
Configurable UI, like themes intended for re-use tend to have

A fashion blogger (or other non-technical friend) might view this setup as horribly complicated, but that's because the complexity is hidden to the user in a typical blogging platform.

Admittedly, I've spent some time on this over the years, with small adjustment every now and then, to achieve this level of sophistication and simplicity. It might sound arrogant with "sophistication" and "simplicity" but storing the content as text files instead of in a database and having a static site that can run from a CDN is simple.

It's sophisticated in that the pipeline is automated and has tests in the same way as our professional applications we develop at work. I'll create a new branch, do code or design change and make a pull request (PR) to myself. If the tests that run automatically when a PR is created are green, I merge without hesitation. More details coming in future posts…

8 JavaScript Recommendations to a Struggling Student

Henrik Sommerfeld — Sun, 31 May 2020 09:48:16 +0000

I'm sure there are thousands of posts on this topic already, but since this turned out to be mostly general advice, I might as well publish it to save a few keystrokes if I get a similar request in the future.

Context

I was recently asked for advice by a student struggling with his web development assignment from school. This was an exercise from the level below university – high school, upper secondary school, or whatever it's called in your country.

I got a zip file with the assignment as a PDF file and the code in its current state where the student was stuck. Unsurprisingly, it wasn't one specific thing that wasn't working with a clear question on how to solve that specific problem. There were errors in the web browser console and long functions with wrong indentation that made it all hard to understand.

I've never been able to "take a quick look" at a problem like this and give valuable feedback. To be able to give some helpful advice, I need to understand the assignment and see how far from a solution the current state is. In this case I ended up making my own implementation and then writing down my advice. We then had a screen sharing session walking through the problem step by step, until only some minor stuff remained to implement.

Photo by NESA by Makers

My general advice

There was some specific advice I could give based on the assignment, but the general stuff was as follows.

1. Variable declarations

A variable should be declared, either with var, let or const, but avoid

var – see https://hackernoon.com/why-you-shouldnt-use-var-anymore-f109a58b9b70.

2. Naming

Giving functions and variables good names is one of the most difficult and most important things to get understandable code. Generally, I recommend using English names throughout.

If it makes it easier for you to reason about the problem domain in your own language, then you might be better off using your native language, but if you do – use the whole alphabet (like umlauts) and not some crippled version of your language. Most importantly: be consistent.

Here is a clear walkthrough of conventions you benefit from following: https://www.robinwieruch.de/javascript-naming-conventions

3. Indenting

Indenting is, just like naming, something that helps or hinders the brain when reading code. It might feel trivial, but it does make a difference, maybe to your grade on this exercise as well.

Code doesn't have to be pretty before it works, but wrong indentation can make you put something inside instead of outside a block ({}) – and just like that, 10 minutes is wasted on troubleshooting.

4. Short functions

Make sure your functions fit on the screen without scrolling. Especially if you have many levels of indentation, it's a sign you might be able to extract some of it into a new function, like the contents of an if statement or a for loop.

What I'm reaching for here isn't code aesthetics that will give bonus points. I assume you're only interested in getting it to work at this point, and that's when these general things help me proceed in the right direction.

5. Limit state manipulation

Use as few global variables (declared outside of any function) as possible and keep them in one place. Try to set them in as few places as possible.

6. Baby steps

Keep the web browser developer tools (F12) open to spot errors in the console. If you have an error, stop what you're doing and fix the error. Save and test often.

Make sure you address one issue at a time. Even if you're not "done" with the feature you're working on, you can watch your progress by adding console.log(variable) statements, inspecting the HTML in the browser or setting breakpoints and stepping through the code (debugging).

7. Backup when something works

Ideally you should use a version control system (like Git), but if you haven't been taught how to use that, you can always copy the code folder and give it a sensible name, whenever you've managed something to run like it should. It's just as easy to break something that worked before, as it's frustrating when it happens.

8. Google (verb)

If you know what you want to achieve, but not how to type it out – google it, we all do.

Conclusion

Having gone through the assignment, solving it together with the student in a two-hour screen sharing session, I conclude that methodical problem-solving skills is the most important. Of course you have to know the basics of the programming language you're using and have an understanding of the assignment to be solved, but there are no shortcuts.

The other obvious insight is that apart from my first JavaScript specific advice on variables, this applies to most of coding – not only most programming languages, but most of programming you will ever do in your career.

The Grass is Greener on the Other Side

Henrik Sommerfeld — Mon, 04 May 2020 19:53:14 +0000

Approaching 40 years old and two years since I changed direction as a software developer, I conclude that the grass really is greener on the other side. Perhaps I should have jumped earlier.

Where I'm coming from

Most of my career I've been a consultant working with .Net development and specialised in SharePoint (described in Getting a Divorce From SharePoint). When I read The Phoenix Project I strongly identified with the pre-transformation description and the different characters described in that book. At one time, me and a colleague had a great laugh comparing our work situation to the film The Martian (where the main character is stuck alone on Mars, click the link and see the trailer to get a feeling for what I mean). Our job felt just as complex and challenging – and we were only maintaining a bloody website! To be fair, it was a website selling products for a couple of hundred million EUR annually, but still.

Not all problems should be solved, some should be avoided. When things are complicated due to accidental technical debt rather than complex due to supporting a complex world, walk away.

Another aspect of your career is how much you learn over time. Do I have x years of experience or 1 year of experience x times?

Sample tech:

Visual Studio (Not responding)™
.Net Framework 4.x
Team Foundation Server (using TFVC)
SharePoint Server
MS SQL Server
Windows Server
Virtual machines weighing 100+ GB
Lenovo P50
README for monthly "deploy weekend"

Where I'm now

Now I'm in ops, part of a team named Tooling that provides the infrastructure and tools for other development teams to run their applications on. I'm the least experienced in the team when it comes to infrastructure and the tools we use, so I learn a ton of new stuff all the time. Having taken a web interface for granted for almost all of my career, the primary user interface our team is building right now is a CLI.

As a developer in our organisation, you create a new service (API, web app, etc.) by using our CLI. It will ask you a few questions: name and description for the service, what team owns it, select the upstream services it depends on etc. That takes care of all the boring stuff you need, but don't want to deal with as an application developer, like logging, monitoring, TLS certificates etc.

Naturally, the idea of infrastructure as code isn't something you have to sell to someone like me with a coding background, but seeing it in action is still fantastic. Even if this isn't feasible for a small company, we're certainly not as big as Google, Amazon, Microsoft or Facebook.

I find it delightful to primarily work with open source products. We buy commercial software when we find that's the best option, but we don't buy a product that claims to do everything. There is a big difference in choosing the pieces that best fits the puzzle you're trying to assemble, vs buying a complete puzzle that you have to retrofit your existing pieces into.

Sample tech:

Visual Studio Code (still avoid Vim)
JavaScript/TypeScript
Go
Node JS with koa or Apollo Server
Docker, Terraform, Nomad, Consul, Jenkins, Neo4j, Grafana, Graylog, Prometheus...
Mob station with large TV and Ubuntu Linux NUC + personal MacBook Pro

Next actions

I will keep evaluating my learning and how fun I think my job is. In hindsight I should probably have made a change earlier, but I wasn't unhappy before either and I guess that's the tricky part when it isn't as clear as black and white.

I hope I have contributed to better ways of working earlier in my career as well, but some things are hard to change within a specific role or organisation. Some people prefer stability and familiarity, but "git hooks" are better than "deployment weekends"!

Read (or listen to the audio books) The Phoenix Project and The Unicorn Project and listen to Rich Hickey's talk Simple Made Easy if you haven't, that's my next actions for you 😉

Asuswrt-Merlin Firmware Update Checker

Henrik Sommerfeld — Thu, 09 Apr 2020 13:31:26 +0000

I have an Asus RT-AC68U router at home. I've previously used the build-in update checker together with a notification script that ran on the router itself. Recently I noticed that I hadn't got any update notifications in a long time, a downside of silent failures.

When I ran a manual check using the router's web interface, it just said: "Temporarily unable to get the latest firmware information. Please try again later." It doesn't seem to be that temporary though.

TLDR

The code is in this GitHub repo and the scheduling piece with cron is described a the end of this post.

Building my own update checker

Since the project website of Asuswrt-Merlin presents the latest version in an easily parsable way, I decided to write my own checker using screen scraping in NodeJS.

Version checker

To find the latest version, I just looked at the website, inspected the HTML, installed the packages request-promise and cheerio, and finally extracted the version number of interest.

import rp from 'request-promise';
import $ from 'cheerio';

export async function getLatestStableVersion() {
  try {
    const html = await rp('https://www.asuswrt-merlin.net/');
    const text = $('#block-currentrelease', html).text();
    const rows = text.split('\n');
    const stableOthers = rows.find(r => r.startsWith('Others:'));
    const stableVersion = stableOthers.split('Others:')[1].trim();
    return stableVersion;
  }
  catch (error) {
    Promise.reject(error);
  }
}

Saving last checked version

In order to know if there is a new version since my last check, I of course need to keep track of what the version was the last time I checked. I did this with a simple text file on disk.

import { existsSync, readFileSync, writeFileSync } from 'fs';
import * as path from 'path';

const savedVersionFilePath = path.resolve(path.resolve(''), './last-checked-version.txt');

export function getLastCheckedVersion() {
  if (!existsSync(savedVersionFilePath)) {
    return "0.0"
  }

  const lastCheckedVersion = readFileSync(savedVersionFilePath, 'utf8');
  return lastCheckedVersion;
}

export function saveLastCheckedVersion(version) {
  writeFileSync(savedVersionFilePath, version);
}

Notifier

I already had a working notification script using the service Pushover that I ported from Bash to NodeJS.

import dotenv from 'dotenv';
import rp from 'request-promise';

export function sendPushoverNotification(message) {
  dotenv.config();
  var options = {
    method: 'POST',
    uri: 'https://api.pushover.net/1/messages.json',
    body: {
      token: `${process.env.PUSHOVER_TOKEN}`,
      user: `${process.env.PUSHOVER_USER}`,
      message: message
    },
    json: true
  };

  rp(options)
    .then(function (parsedBody) {
      console.log(`Pushover notification sent: ${message}`);
    })
    .catch(function (err) {
      console.error(err);
    });
  }

Gluing it together

By sending a notification both when there is no update and when an error occurs, I won't have any silent failures unless I made a mistake here somewhere.

import { getLatestStableVersion } from './latest-version-checker.js';
import { sendPushoverNotification } from './notify.js';
import { getLastCheckedVersion, saveLastCheckedVersion } from './localFile.js';

async function main() {
  try {
    const lastCheckedVersion = getLastCheckedVersion();
    console.log("main -> lastCheckedVersion", lastCheckedVersion)
    const latestVersion = await getLatestStableVersion();

    if (latestVersion !== lastCheckedVersion) {
      const message = `🔔 New firmware version ${latestVersion} is now available at 

      https://www.asuswrt-merlin.net/`;
      sendPushoverNotification(message);
      saveLastCheckedVersion(latestVersion);
    }
    else {
      const message = `🤷‍♂️ No firmware released. ${latestVersion} is the latest.`;
      sendPushoverNotification(message);
    }
  } catch (error) {
    console.log("main -> error", error)
    const message = `⚠️ Router firmware update check failed`;
    sendPushoverNotification(message);
  }
}

main();

Scheduling the update checker

I'm running this on a RaspberryPi and it's scheduled to run once a week, 18:10 on Wednesdays. I found https://crontab.guru to be helpful for not mixing up the time settings.

crontab -e

10 18 * * 3 /home/pi/router-update-check.sh >> /home/pi/router-update-check.log

The trickiest thing for me as a terrible Linux admin, was to get the cron scheduling working. Adding the output of echo $PATH at the top of the script did the trick. Logging the output (to router-update-check.log in this case), also helped.

router-update-check.sh script contains the following:

#!/bin/bash
PATH=/home/pi/.nvm/versions/node/v13.12.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/snap/bin

cd /home/pi/Code/asuswrt-merlin-update-check/
node ./main.js

The result of running the script once every minute (while troubleshooting) showed up in my phone like this. I now have an update checker that I can only blame myself if it doesn't work. Great success!

Code Quality and Automated Tests for Legacy Systems

Henrik Sommerfeld — Thu, 12 Mar 2020 19:01:38 +0000

My team recently had a discussion trying to agree on a few rules of thumb for our ambition level regarding code quality and automated tests for our legacy systems. These systems are business-critical, but planned to be replaced in a Big Bang fashion.

Photo by Alex Motoc

One of my colleagues challenged our usual code of conduct of continuous improvement of code quality in regards to these systems. For a code base that has a set timeline for decommissioning, I agreed that we shouldn't strive to improve the code quality. This is especially true when we're talking about a code base that's fragile and where any change is more or less risky.

Our mission is first and foremost to keep these systems running, second to fix high priority bugs and where possible, make highly requested minor improvements.

Code quality

The code of conduct we came up with for these systems, is something along these lines:

Strive for a code quality that won't bite us later

Meaning, if we believe we will have to come back to a particular piece of code and make another change later – apply good practices so that we are nice to our future selves, but don't do any refactoring of surrounding code. If adding another if statement solves the problem – do it and move on.

Automated tests

For automated tests, we said the following:

Write a test when we're fixing a bug, if it's reasonable

Creating an automated test or two as part of fixing a bug is great to avoid regressions later. But we also realise that for some systems it isn't worth the effort. This is when there are no existing tests, meaning no testing infrastructure in place. Some of our old systems have a lot of logic in stored procedures in the database, and to add tests for that just doesn't make sense in our situation.

Write tests where it's making development easier

The most obvious case for writing tests are when it helps development directly. This could be a validation function or some other non-trivial pure function where unit tests make the function easier to write. So even if the tests are never run again, it would still be worth the effort.

Write tests that have as good pay off as possible

An example of this is UI tests that can catch many possible errors in relation to the amount of test code. This is of course given that we already have a stable set of tests, we don't want to spend a lot of time troubleshooting flaky UI tests.

Conclusion 🤷‍♂️

Is this just common sense or too vague to be useful? Maybe it's vague, but given the diversity of the systems we're talking about here, we couldn't find a way to make it more concrete without making it system specific. Doesn't this apply to all systems? Well, sort of. I think the only relevant difference is the amount of time we're willing to spend on improving existing code – and that amount of time is more of a gut feeling than a discrete number. But in any case, I think it's good we had the discussion and hopefully we can waste less time trying to improve code that will die fairly soon anyway.

How I Tackled Parental Leave Boredom With Code

Henrik Sommerfeld — Wed, 12 Feb 2020 10:07:32 +0000

Approaching the end of my parental leave, I'll take a moment reflecting on how I tackled the occasional boredom of a longer absence from work.

Photo by Picsea

Background

I live in Sweden where we have good possibilities for taking long parental leaves. In my case it has been 7 months – 8 if I count the summer vacation. While I'm absolutely thankful for the possibility to stay home with my daughter and getting to know her, I'll never get a second chance etc etc 🙄, it's still boring as hell from time to time. Having an ocean of time, but all the time being consumed by baby care was a new and frustrating experience for me.

What saved me from becoming chairman of the bored was the small technology projects I could work on during the short periods of focus time (like when the child was asleep). Problem solving during stroller walks was also a great way to make progress even in pure AFK days (Away From Keyboard).

What worked for me

Looking back at these months, I'm quite satisfied with my accomplishments. In short:

Rebuilt personal website for my wife from WordPress to Gatsby, saving us the annual fee for WordPress.com and ended up with a more personal site.
Added dark mode to my own blog
Added lazy loading of images on my blog
Built a family recipe collection site with a CMS
Kept the daughter alive (happy and well) 👶🏻

Small projects

I think a key success factor was to start small, keep the scope smaller than what you think you'll manage to complete (because you'll be wrong, you'll always be able to do less than what you initially think).

MVP

Define a Minimal Viable Product (MVP) and keep track of nice to have stuff separately. When you have other stakeholders, like with my wife's personal website, it can be a good idea to explain the concept of an MVP – what's simplest version I can build that you can use and we still can kill the old site?

Once the new site was live, I could continue to do small improvements without time pressure (when the WordPress.com subscription was already cancelled). These were things that was easy to do without longer periods of focus.

Have your laptop with you

Since you'll need a bag with diapers, change of clothes, food, water and so on, you might as well throw in your laptop when leaving home. That way you can make use of the random chunks of time that appears when the kid falls asleep.

Ensure you can take notes on the go

When you get a great idea during a walk, it's nice to be able to easily catch that so that you can process it later. I carry the David Allen Notetaker wallet (not sold any more) since many years, but I guess most people take notes on their phone.

Final words

I'm not saying that everyone should do as much work related stuff as possible during their parental leave. For you the best decision might be to disconnect from work as much as possible, but for me this was a way of staying sane. Thanks for reading!

Lazy Loading Images in Hugo

Henrik Sommerfeld — Wed, 05 Feb 2020 11:47:01 +0000

When writing another post, I realised that I hadn't documented/described my image lazy loading implementation in Hugo anywhere, so here it comes. Full source code is in this repo.

The first thing we need are responsive images, not the thing you get when setting the width to 100% in CSS, but different versions of the same image in different resolutions so that the web browser can pick the best one (using srcset).

Changing folder structure

The structure I had before implementing this was all the Markdown files in the same folder with the images in a common static/images folder. To use image processing in Hugo we need to use Page Resources, meaning we have to create a folder for each post where we put both the Markdown file and the images related to that post.

Before:

content
 - articles
   - 2019-11-15-dark-mode.md
   - 2019-11-17-netlify-cms.md

After:

content
 - articles
   - 2019-11-15-dark-mode
     - index.md
     - jay-wennington-loAgTdeDcIU-unsplash.jpg
   - 2019-11-17-netlify-cms
     - hero-image.png
     - index.md
     - insert-image.png

Writing a shortcode

To be able to reference the images in the Markdown files, we need a shortcode. This is the trickiest part to explain since the code is a bit lengthy. This is because I have the possibility to pass parameters for whether or not to have a border, a custom width, a lightbox and caption. It might also be because my Go template skills are questionable.

I use the shortcode in my index.md file to reference the image like this:

{{<post-image image="hero-image.png" />}}

What happens in the shortcode implementation below is that I create a blurred version of the image resized to 48px wide (keeping the aspect ratio). This is the image you will see before a better version is fetched. This small image is embedded into the HTML as a Base64 encoded string, so that there is no extra request required to get the initial image.

A few different versions are then generated, but only if the original image is larger than the version I'm trying to generate. I don't want to upscale an image (it will be blurry). The variable $src_set is appended with each version and will contain all versions (sizes) when the HTML part begins.

Shortcode post-image.html:

{{ $image := (.Page.Resources.GetMatch  (index .Params.image)) }}
{{ $alt := .Get "alt" }}
{{ $width := .Get "width" }}
{{ $borderless := .Get "borderless" }}
{{ $placeholder := ($image.Resize "48x q20") | images.Filter (images.GaussianBlur 6) }}
{{ $src := $image }}
{{ $src_set := ""}}

{{ $src_set = (print $image.RelPermalink " " $image.Width "w") }}
{{ $src := $image }}

{{ if ge $image.Width "500"}}
{{ $x_small := $image.Resize "500x" }}
{{ $src_set = (print $src_set ", "  $x_small.RelPermalink " 500w") }}
{{ end }}

{{ if ge $image.Width "800"}}
{{ $small := $image.Resize "800x" }}
{{ $src_set = (print $src_set ", " $small.RelPermalink " 800w") }}
{{ end }}

{{ if ge $image.Width "1200"}}
{{ $medium := $image.Resize "1200x" }}
{{ $src_set = (print $src_set ", " $medium.RelPermalink " 1200w") }}
{{ end }}

{{ if gt $image.Width "1500"}}
{{ $large := $image.Resize "1500x" }}
{{ $src_set = (print $src_set ", " $large.RelPermalink " 1500w") }}
{{ end }}

{{ $border_class := "image-border" }}
{{ if $borderless}}
{{ $border_class = "" }}
{{ end }}


<noscript>
  <style>
    figure.lazy {
      display: none;
    }
  </style>
  <figure class="{{ $border_class }}">
    {{ if .Get "lightbox" }}
    <a href='{{ $image.RelPermalink }}'>
      {{ end }}
      <img src="{{ $src.RelPermalink }}" {{ if $width }}width="{{$width}}"{{ end }} />
      {{ if .Get "lightbox" }}
    </a>
    {{ end }}
    <figcaption>
      <em>{{ .Inner }}</em>
    </figcaption>
  </figure>
</noscript>

<figure class="{{ $border_class }} lazy">
  {{ if .Get "lightbox" }}
  <a href='{{ $image.RelPermalink }}'>
    {{ end }}
    <img class="lazyload" data-sizes="auto" src="{{ $src.RelPermalink }}" {{ if $width }}width="{{$width}}"{{ end }}
      srcset="data:image/jpeg;base64,{{ $placeholder.Content | base64Encode }}" data-src="{{ $src.RelPermalink }}"
      data-srcset="{{ $src_set }}" width="{{ $image.Width }}" height="{{ $image.Height }}" alt="{{ $alt }}" />
    {{ if .Get "lightbox" }}
  </a>
  {{ end }}
  {{ if .Inner }}
  <figcaption>
    <em>{{ .Inner }}</em>
  </figcaption>
  {{ end }}
</figure>

Adding Javascript

There are a few CSS classes set here that acts as a signal to different Javascript features. lightbox is such a thing, but the interesting one here is lazyload. I use a Javascript library called lazysizes that is included in my Javascript bundle and I have created the HTML to work with that library.

You might have noticed that the code above looks somewhat duplicated, that's because I have a noscript tag for those with Javascript disabled. In that case, a srcset tag is still used to provide the best image, but without the lazy loading.

There is no "blur up effect" here. That's a conscious decision, since this is a bit simpler to implement and feels faster, in my opinion.

Performance consideration

In Hugo's documentation for image processing it's clearly stated that it's recommended to include the generated images in source control. This is to avoid generating the same images over and over again. On a large site with lots of images, this can make a big difference.