DEV Community: Austin S. Hemmelgarn

The magic of `git worktree` — How I multitask despite tests taking forever to run.

Austin S. Hemmelgarn — Tue, 13 Dec 2022 02:45:57 +0000

As a programmer, sometimes there are situations where you need to jump to a different task when you’re right in the middle of something. Most decent VCS software provides a selection of ways to approach this, ranging from things like patch queues, to branches, to any number of other mechanisms.

However, one of the big limitations of these mechanisms is that they require you to change out the entire working tree, so you can’t use them if your existing task happens to require the working tree, such as running a build or tests, or if you need to preserve the state of an unclean working tree.

You can kind of work around this by creating another clone of the repository, but this has a couple of major limitations:

Special effort needs to be made if you need to keep the two working trees in sync.
Special effort needs to be made to keep track of exactly which copy of the repo you’re working with.
Twice as much disk space is required.

With git, you can avoid most of the extra disk space by using git clone --shared to clone the local repository, but this is still a wholly separate repo, so it still runs into the other two issues.

Thankfully git has a wonderful solution to this...

Enter `git worktree`

One of the lesser known features of git is that it supports having multiple working trees attached to the same repository. Each git repository (except for bare repositories) has a ‘main’ working tree, which is the one that actually has the .git directory for the repository, plus zero or more ‘linked’ working trees, which use a file called .git to link back to the .git directory.

To manage these linked working trees, git provides the git worktree subcommand. Creating a new linked working tree that has a specific branch checked out is simple as running:

git worktree add /path/to/worktree branch

Instead of a branch, you can specify any tag, commit, or other ref to check out that specific ref. If you don’t specify a ref to checkout, then a new branch will be created automatically from the currently checked-out commit with a name based on the basename of the new worktree.

Once created, you can change to a linked working tree, and then use it (mostly) just like a regular git repository. The key difference is that it shares a .git directory with the main working tree. This means that if you create a new branch in the linked working tree and add commits to it, that same branch and commits will be immediately accessible in the main working tree and any other linked working trees.

Worktrees can be listed:

git worktree list

Moved:

git worktree move /old /new

Or removed:

git worktree remove /path/to/worktree

But how to use it?

Say as an example that you’ve got a build running in your main copy of the repository, but need to hop to a different branch to work on something else in a different branch while waiting for the build to finish. You can easily do this using git worktree from a different shell as follows:

WORKTREE_PATH="/path/to/tree"
git worktree add "${WORKTREE_PATH}" branch
pushd "${WORKTREE_PATH}"
# make needed changes and commit them...
popd
git worktree remove "${WORKTREE_PATH}"

The same approach can be used to create temporary work trees for running tests (I do this on a regular basis, especially when dealing with cross-architecture builds where things need to run under QEMU userspace emulation for testing), or as a way to provide two full copies of different versions of a source tree to compare using some tool that doesn’t integrate with git.

OK, what’s the catch?

As amazingly useful as git worktree is, there are still a few major limitations:

The .git file in a linked worktree embeds the path of the main worktree. Because of this, if you move the main worktree, any linked worktrees will become unusable until you run git worktree repair.
Similarly, the main worktree’s .git directory embeds the paths to any linked worktrees. This means that if you want to move a linked worktree, you either need to use git worktree move, or you need to run git worktree repair in the linked worktree after you move it.
If for some reason both the main and the linked worktrees get moved at the same time, you have to run git worktree repair in the main worktree, and pass it the new paths to each linked worktree.
While git technically supports per-worktree configuration, it’s tricky to use, and there are a number of gotchas.
Even if you have multiple worktrees, you cannot safely check out the same branch more than once.
Support for submodules is a bit lacking. It mostly works, but you have to force removal when removing a worktree that contains checked-out submodules, and have to manually move worktrees that contain submodules.

More details can be found in the git documentation.

Skip CI stages in Travis based on what files changed.

Austin S. Hemmelgarn — Tue, 24 Dec 2019 00:59:17 +0000

At my new job, I've ended up partially responsible for the CI pipeline for our flagship product. One of the big complaints from both our internal devs and our FOSS contributors is how long CI builds (and thus the CI checks on pull requests on GitHub) take to complete.

Digging into this, I pretty quickly realized that the main culprit was our unit testing, which includes stress tests for one of the components which take almost 15 minutes to run, even if the build is for a PR or commit that doesn't even touch that code.

So, I set about looking into how to skip those tests when the changes being tested had nothing to do with that component.

A First Look

We currently use Travis CI to do our CI/CD builds. Travis actually provides you a rather significant amount of control over what jobs run as part of a build based on what you're doing. You can match on quite a few things, including stuff like whether the build is for a PR or not, what distribution the build is being done on, and even whether the repo the build is for is a fork or not.

However, there's no way from the conditionals to check anything about the contents of the branch being built (you can match on branch name or commit message, though those aren't that useful for our purposes). Given this, we need to do the checks inside the scripts for the unit testing using data from the build environment.

A Matter of Depth

Before going further into figuring out how to do this, there's a rather important detail about Travis that needs to be taken into account. By default, when Travis clones a Git repository, it does a shallow clone with only the most recent commit. This provides a significant performance improvement to the cloning process when dealing with large repos, but it means that you can't find any information about anything except the most recent commit, which is an issue when trying to determine what files have changed.

Thankfully, disabling this and getting a deep clone of the source repo with the full history is rather easy, you just need to add the following to any job that you need to inspect the history with:

git:
  depth: false

You could also specify a number between 1 and 50 (Travis imposes this 50 commit limit, not git) instead of false to get a clone with that number of commits. For most use cases, just specifying 50 there should be more than enough. However, the performance difference between fetching the 50 most recent commits and the whole repo is rather minimal unless you have an absolutely huge history on the branch you're checking against, so it's safer to just disable the shallow cloning altogether.

The Build Environment

Back to the task at hand, Travis provides a huge amount of info in the build environment in the form of environment variables (see the full list here). This includes a lot of the same stuff that you can match in conditionals, but also includes quite a bit more than that, including such things as what compiler is being used, what CPU architecture is being used, and even the URL for the build logs.

Rather helpfully, there is an environment variable provided by Travis that tells you what range of commits you're testing with this build, called TRAVIS_COMMIT_RANGE. This provides the commit range being tested as two shortened commit hashes separated by .... Based on this, if you're using git for version control, the following snippet of bash code will give you a list of all the files that changed for this build, one per line, in the variable CHANGED_FILES:

# `git` expects a `..` delimiter between commits, not `...`
COMMIT_RANGE="$(echo ${TRAVIS_COMMIT_RANGE} | cut -d '.' -f 1,4 --output-delimiter '..')"
CHANGED_FILES="$(git diff --name-only ${COMMIT_RANGE} --)"

Once you have that, you can easily use any of a number of tools to check the list of changed files against a set of matching conditions to decide on what to run.

A Few Small Issues

The above code works great most of the time. However, there are three caveats to using TRAVIS_COMMIT_RANGE that can cause the above code to fail:

If the build is for a newly pushed branch, TRAVIS_COMMIT_RANGE will be empty.
If the build is for a branch that just had it's history rewritten (for example, due to a rebase and force push), the starting commit may not actually be accessible in the repo anymore.
TRAVIS_COMMIT_RANGE only lists the range of commits for the most recent push. This means that if a user incrementally updates a PR by pushing to the branch it's based on, the above won't give you the full list of files for the PR, just the most recent push.

The third is actually the easiest to solve, so we'll cover that first.

Looking at the whole PR

In the case of a PR build, the environment variable TRAVIS_PULL_REQUEST will be set equal to the PR number on GitHub. For other builds, it's set to the exact string false. Thus, checking if a build is a PR or not is as easy as checking if TRAVIS_PULL_REQUEST is not equal to the string false.

Once we know we're dealing with a PR build, there are two additional facts we can rely on to determine what commits are part of the PR:

The currently checked out revision (HEAD in git terms) will be the most recent commit in the PR.
The the name of the branch the PR is against (that is, the one it will be merged into) can be found in the TRAVIS_BRANCH environment variable.

Based on this, we can update the above code like so to inspect the entirety of a PR when dealing with a PR build:

CHANGED_FILES=

if [ "${TRAVIS_PULL_REQUEST}" = "false" ] ; then
    # This isn't a PR build.
    COMMIT_RANGE="$(echo ${TRAVIS_COMMIT_RANGE} | cut -d '.' -f 1,4 --output-delimiter '..')"
    CHANGED_FILES="$(git diff --name-only ${COMMIT_RANGE} --)"
else
    # This is a PR build.
    CHANGED_FILES="$(git diff --name-only ${TRAVIS_BRANCH}..HEAD --)"
fi

The Challenging Cases

The other two issues are a bit more complicated. Both of them also functionally require looking at the 'whole' branch instead of the most recent commits, but unlike a PR we don't have a well defined base branch to check against, and on top of that we don't know exactly where the branch diverged.

However, detecting these cases is actually pretty easy.

As mentioned above, if the build is for a new branch (one Travis has never seen before), TRAVIS_COMMIT_RANGE will be empty. That's trivial to check with a simple [ -z "${TRAVIS_COMMIT_RANGE}" ].

The case of a history rewrite is a bit trickier, because we need to figure out if git recognizes both the starting and ending commit as actual commits. Luckily, this can be done with a single git command: git cat-file -t $foo, where $foo is the commit hash we want to check. If the hash represents an actual commit, that command will return the exact string 'commit', which we can also easy check for.

Putting that all together, we get:

CHANGED_FILES=

if [ -z "${TRAVIS_COMMIT_RANGE}" ] ; then
    # This is a new branch.
else
    # This isn't a new branch.
    if [ "${TRAVIS_PULL_REQUEST}" = "false" ] ; then
        # This isn't a PR build.

        # We need the individual commits to detect force pushes.
        COMMIT1="$(echo ${TRAVIS_COMMIT_RANGE} | cut -f 1 -d '.')"
        COMMIT2="$(echo ${TRAVIS_COMMIT_RANGE} | cut -f 4 -d '.')"

        if [ "$(git cat-file -t ${COMMIT1} 2>/dev/null)" = commit -a "$(git cat-file -t ${COMMIT2} 2>/dev/null)" = commit ] ; then
            # This was a history rewrite.
        else
            # This is a 'normal' build.
            CHANGED_FILES="$(git diff --name-only ${COMMIT_RANGE} --)"
        fi
    else
        # This is a PR build.
        CHANGED_FILES="$(git diff --name-only ${TRAVIS_BRANCH}..HEAD --)"
    fi
fi

Once you've got the detection done, there are three approaches that can be taken here:

Just assume that everything you might do in this job needs to be done and move on with things. This is the simplest and most reliable option overall. You can find an example of a full script doing this here.
Assume a specific 'base' branch to compare against and use git merge-base to find the 'starting' commit to work from. This has some reliability issues because git merge-base can be unpredictable under certain circumstances, and won't account reliably for changes that happened in the base branch relative to your current branch.
Assume a specific 'base' branch to compare against, and then use the same logic as was used for PR builds. This is more reliable and predictable than using git merge-base, but it can still miss things on occasion.

Which approach you take really depends on what you're trying to do conditionally. If it's just something like unit tests, then the first approach (just assume you need to do everything) is generally fine. However, if you're using this for conditional deployment, you're probably going to want one of the other two approaches so that you don't redeploy things you don't need to.

Any tips for someone just starting out learning Elixir?

Austin S. Hemmelgarn — Fri, 06 Sep 2019 18:55:46 +0000

I recently decided to start learning Elixir. I've already got a project in mind (I'm going to be converting some existing tools I use regularly on my systems from Python (a language I already have solid experience with) to Elixir since the concurrency model should help them be significantly more efficient), and I've worked through the whole 'Getting Started' guide on the website without any significant issues.

However, it's drastically different from most of what I'm used to (I'm coming from a mostly Python background, with some JS and SH experience), and I'm wondering if anybody might have some advice for people new to the language? Any particular gotcha's to watch out for that aren't well documented? Possibly things that need to be handled significantly differently in Elixir compared to other languages for efficiency reasons?

The HTML5 template element.

Austin S. Hemmelgarn — Mon, 19 Aug 2019 02:00:02 +0000

Templated content is huge on the web. Most sites do at least some form of templating either on the server side or the client side, and it's essentially mandatory on the client side if you're writing a single-page application (and wish to retain your sanity).

Outside of sites using Web Components though, any templating on the client side is almost always done using either a dedicated templating library, or an application framework that supports templating.

There is, however, another way: The HTML5 <template> element.

What exactly is the `<template>` element?

Put simply, it provides an easy way to define a reusable fragment of HTML that can be manipulated just like you would the contents of the document itself, but without the overhead of actually updating the DOM or having to compile and parse strings of HTML.

Anything inside of a <template> tag gets parsed just like regular HTML, except:

It doesn't get rendered.
<script> tags inside of it don't get run.
<style> tags inside of it don't get evaluated.
It doesn't load any external resources (so you won't see any requests for the contents of <img> or <embed> tags).
It can be accessed as a DocumentFragment instance via the special content property of the <template> element.

That content property is the powerful bit here. DocumentFragment instances provide an API for manipulating their contents that's largely the same as the global document object, so you can manipulate them like their own separate document. On top of that, inserting a DocumentFragment instance into the DOM is really fast compared to manipulating an element's innerHTML property (or using insertAdjacentHTML()), because the DOM structure already exists in memory, so it just needs to get linked into the DOM tree.

So what can you do with it?

A simple example might look like this:

<template id='#closeTemplate'>
    <button type="button" class="close" data-dismiss='modal' aria-label="Close">
        <span aria-hidden="true">&times;</span>
    </button>
</template>

This should be pretty recognizable to anybody who's used Bootstrap 4 before. It just defines a template for a close icon for a Bootstrap 4 modal.

const closeTemplate = document.querySelector('#closeTemplate').content

for (const modalHeader of document.querySelectorAll('.modal-header')) {
    modalHeader.append(closeTemplate.cloneNode(true))
}

This then takes that template, and inserts it's contents at the end of each element with the modal-header class (right where it should be if you're using regular Bootstrap 4 modals).

The most important thing here is the call to the cloneNode method of the template content. This method, when called with true as an argument, creates a completely new DocumentFragment instance that's an exact copy of the original (this sounds like an expensive operation, but it's actually not that bad, especially for a small template like this). This is important because a given DocumentFragment can only exist in the DOM in one place, so subsequent calls to any method that would insert it into the DOM will fail (without raising an exception for some reason that I cannot fathom). By creating a copy on each loop iteration, and inserting that into the DOM instead of the original, we avoid this issue.

OK, but what about using it for actual templates?

Of course, most templating isn't as trivially simple as that. In real life, you usually have to put something into the template for it to be useful, and you usually need to make sure certain attributes are set correctly before it gets put in the DOM.

This is where the DocumentFragment class having an interface almost the same as the global document object comes in. With this, you can call querySelector on the fragment just like you would to find elements in the page, and get a real Element or NodeList back that you can then manipulate just like if you had requested elements in the page itself.

Consider the following example code:

<template id='modalTemplate'>
    <div class='modal fade' tabindex='-1' role='dialog' aria-hidden='true'>
        <div class='modal-dialog' role='document'>
            <div class='modal-content'>
                <div class='modal-header'>
                    <h5 class='modal-title'></h5>
                    <button type='button' class='close' data-dismiss='modal' aria-label='close'>
                        <span aria-hidden='true'>&times;</span>
                    </button>
                </div>
                <div class='modal-body'></div>
                <div class='modal-footer'>
                    <button type='button' class='btn btn-secondary' data-dismiss='modal'>Close</button>
                    <button type='button' class='btn btn-primary'>Save</button>
                </div>
            </div>
        </div>
    </div>
</template>

Now our template is a basic skeleton for a Bootstrap 4 modal. Note that there are no ID attributes (or anything that references them) here. Template contents still need to meet the uniqueness requirement for ID attributes across the whole page, so it's safest to just avoid them in the template itself and populate them from your code as you're using the template.

const modalTemplate = document.querySelector('#modalTemplate')

function createModal(id, title, body) {
    const node = modalTemplate.cloneNode(true)
    const modal = node.querySelector('.modal')
    const modalTitle = node.querySelector('.modal-title')
    const modalBody = node.querySelector('.modal-body')

    modal.id = id
    modal.setAttribute('aria-labelledby', `${id}Title`)

    modalTitle.id = `${id}Title`
    modalTitle.textContent = title

    modalBody.innerHTML = body

    document.body.append(node)

    return document.querySelector(`#${id}`)
}

And here's a function to turn that template into an actual modal in the document. This adds appropriate id and aria-labelledby attributes to the root element of the modal, an appropriate id and text contents to the title bar, and then adds whatever needs to be in the body of the modal before adding the modal itself to the end of the document's <body> element.

But wait, there's more!

Because you're doing all of this template construction from JavaScript, you also have the full power of JavaScript available for flow control logic. You'll notice in the example above that we used JavaScript template strings to compute the correct values for the title's ID and the modal's aria-labelledby attribute, but you're not even limited to that. You can easily do complex flow-control using loops, conditionals, and even try/catch statements.

On top of that, because DocumentFragment works almost the same as the DOM, you can inject HTML5 templates into other HTML5 templates. For example, the function above could easily be extended to instead accept a DocumentFragment instance for the modal body, which could itself be created from another HTML template.

By leveraging these tow facts, you can create complex layouts composed of multiple HTML5 templates with little effort.

Additionally, the templates (including the associated JavaScript) tend to be smaller than the equivalent pre-compiled template code for a lot of JavaScript templating libraries (given my own experience, it's about 5-25% smaller than an equivalent pre-compiled lodash or underscore template), they render faster when inserted into the DOM, and you need no special tools to deal with building or checking them, because they're just plain HTML and plain JavaScript.

But surely it's not widely supported?

The HTML5 <template> element is actually very well supported. Every major browser released in the last 3 years fully supports it, and most released in the past 5 years do too. You can check the exact support data on Can I Use?. As of writing this, it's just short of 95% market share for availability.

The caveat here is that Internet Explorer lacks support (and Opera Mini and the BlackBerry browser, but neither of those has enough market share to matter in many cases). However, there are a number of polyfills out there that will get you proper support in IE 11, and rudimentary support in IE 9 and IE 10, as well as smoothing over some of the issues with the older implementations (the Can I Use? link above includes links to a couple of good polyfills, and there's also one included with the Web Components project).

Now, obviously, this is only really any good if you're not using a full application framework. Most of them do the templating themselves because it's just easier, and thus don't do all that well when you mix them with other templating techniques. On the other hand, if you don't need a full application framework for anything other than templating, this may be an easy alternative that would let you eliminate one of your dependencies (and probably speed up your app too).