DEV Community: Tim Abell

📹 Video: How I implement David Allen's GTD® system in Obsidian

Tim Abell — Tue, 15 Sep 2026 00:00:00 +0000

Lately I've been using Obsidian to organise my tasks. I am a fan of David Allen's excellent "Getting Things Done (GTD®)" methodology for orgnising life's infinite demands on one's time, and is it is tool agnostic I have been able to move my setup from Trello to Obsidian (via Logseq where I never got anywhere with GTD).

I've taken the time to publish an 11 minute video showing how I'm using it in a way that is already working pretty well for me.

In the vid I show some basic use of task checkboxes, and then expand on that with queries across markdown files to pull together a persistent homepage in the style of GTD that gives me view of all my projects, actions and inbox.

I've been practicing my jump-cut editing in kdenlive to avoid making anyone endure pondering moments and second attempts, let me know what you think.

You can find the demo vault used in the video on github, including the homepage file source code.

[Boost]

Tim Abell — Fri, 13 Mar 2026 22:29:56 +0000

Existential Dread and the End of Programming

David Whitney ・ Feb 18

#ai #architecture #programming

Automatic release notes with semantic commits and git-cliff

Tim Abell — Thu, 12 Mar 2026 00:00:00 +0000

"What's going out in this release?" - A question that always seems to be more hassle than it should be. I've wanted to automate release notes for years, but have just never quite got there till now. It's taken a while to put together all the necessary parts, and has taken quite a bit of [ongoing] learning about how to effectively operate the system which I'll share with you here.

This is 110% a "standing on the shoulders of giants" post as you will see.

Backstory

Thanks to the combined creation of "semantic/conventional commits" and git-cliff I have finally been able to prove out the concept of generating release notes completely automatically from nothing but git commit logs on some of my open source projects, and even been able experiment with it in a commercial setting on teams of developers which has certainly caused some interest.

I started experimenting with "conventional commits" — i.e. prefixing commit message subject lines with "feat:" or "fix:" etc. — at least as early as 2023. Hunting around I can find a commit from one of my open source projects from Dec 2023: "feat: Render sections in order they were parsed" · github/timabell/SlnEditor@d0d6e5c, and I think I may have dabbled with the idea in private projects before then.

Even though the idea of neatly categorized commits seemed theoretically appealing, I struggled to stick to doing it as it seemed to lack any deep value. It was hard to figure out what "good" looked like without some concrete goal or output. The list of commits looked maybe a bit tidier but it didn't seem worth all the time agonizing over what prefixes to use for every commit.

All that changed recently when I finally managed to connect that with the automatic release note generation. Suddenly they had a purpose, and more importantly I could look at the release notes that were generated and assess whether the git logs had led to a good quality release note that would actually be meaningful to the end user of the software or library.

A real example

So that you don't doze of while we go through all the detail, let's cut to the chase of what we get out of this semantic commit + cliff setup.

Here's a real example of a release note from one of my open source tools.

https://github.com/timabell/gitopolis/releases/tag/v1.13.0

[1.13.0] - 2025-11-28

Features

Respect remote names and default to origin when cloning (#222, #246)

Bug Fixes

Skip adding remotes when repo already exists during clone (#247)

Note that it focus only on the things that an end user would actually care about - what was fixed, and what new features are available. It is not a complete log of everything that has been done to get there, that is what the git logs are for.

This release note for gitopolis release "v1.13.0" was generated automatically by git-cliff from by parsing all the commits since the previous release tag up to the current release tag.

The full graph of git commits since the previous release is as follows:

╰─$ git log --graph --oneline v1.12.4..v1.13.0
* 373123a (tag: v1.13.0) Improve multi-remote handling
|\
| * 323a66b Strip \\?\ prefix from Windows extended-length paths in tests
| * c3bfe0f Normalize backslashes in temp path for Windows compatibility
| * 201873c Use canonicalize for temp path in clone tests for macOS compatibility
| * 8797e75 fix: Skip adding remotes when repo already exists during clone (#247)
| * 305123b Strip temp path prefix in clone tests for git's absolute path behavior
| * 20edf05 feat: Respect remote names and default to origin when cloning (#222, #246)
|/
* a6e6e8c Script for testing cliff release notes
* 0961469 ci: Build all branches
* ee82e95 Add Arch install instructions
* ecb6d40 Add crates.io install to readme

You'll note that only two of them show up in release notes:

* 8797e75 fix: Skip adding remotes when repo already exists during clone (#247)
* 20edf05 feat: Respect remote names and default to origin when cloning (#222, #246)

and all the rest are ignored, including the ci: commit.

This is intentional, and gives the best of both worlds - the ability to do the behind-the-scenes work that users don't directly care about that enables a good quality product — including github actions / ci improvements — without cluttering up the release notes available to the end user while simultaneously being able to surgically craft the release notes that we want as we do the work of coding, committing and merging.

You'll also note that that those two commits are on a branch that was subsequently merged to main, just like in the normal pull-request flow, which means we can build up release notes during our normal flow of working on feature branches, including previewing what release note would be generated if this was released. Even better for teams, this means the release notes can be part of what is reviewed during pull request review.

Setting up & configuring

In the days of ai-coding, you can pretty much just ask your favourite LLM (claude code in my case) to write a cliff.toml file for you to meet your specific needs. I've written a few variations lately for different purposes. How you set it up depends on what kind of project you are working on, and how mature the release process is or can be.

Open source library projects

I've created a demonstration project github-nuget-demo that demonstrates automatic release notes, plus some other advanced release processes:

Generation of release notes from git commit logs
Automatic versioning of the library based on putting bump: major/minor in the commit messages - this allows managing semantic versioning in the very commit that introduces the breaking/non-breaking changes.
Automatic build/test & publish to
- github releases
- nuget.org package hosting (would work for npm, crates.io (rust) etc etc)
Automatic git tagging

Take a look at the github actions to see how it all hangs together.

In there you'll find the cliff.toml that defines how to parse the commits.

You'll also find a detect-bump.sh shell script because although cliff has some version management capabilities built in, I fell of the edge of its built in capabilities when I wanted to find bump: in the commit footer in order to decide how much to bump the version. git-cliff does the actual version bump calculation for us, including finding the previous version number.

Commercial projects with Jira

For non-library projects we don't want the automatic versioning, and might not want the automated releasing. And where the automated release/deploy is less advanced me want to be able to arbitrarily answer the question "what's in the next release if we cut from main now". While still using the manual tagging of released.

For that you can find an example of a bash script that can be manually run at any point here.

This example — in addition to looking for semantic commit prefixes like feat: — also hunts for jira URLs in the commit bodies, and then is able to say which jira tickets are completed in this release, and which are merely mentioned or worked towards.

cliff config example

Here's the cliff.toml from the nuget example

[changelog]
header = ""
body = """
{% if version %}## Release {{ version }}
{% endif %}\
{% for group, commits in commits | group_by(attribute="group") %}
### {{ group | striptags | trim | upper_first }}

{% for commit in commits %}\
- {{ commit.message | split(pat="\n") | first | split(pat=": ") | slice(start=1) | join(sep=": ") | upper_first }} ({{ commit.id }})
{% endfor %}
{% endfor %}\
"""
footer = ""
trim = true

[git]
conventional_commits = false
filter_unconventional = false
split_commits = false
commit_parsers = [
    { message = "^feat", group = "<!-- 1 -->✨ Features" },
    { message = "^fix", group = "<!-- 2 -->🐛 Bug Fixes" },
    { message = "^perf", group = "<!-- 3 -->⚡ Performance" },
    { message = "^doc", group = "<!-- 4 -->📚 Documentation" },
    { message = "^style", group = "<!-- 5 -->🎨 Style" },
]
filter_commits = false
tag_pattern = "v[0-9].*"

You can see how it gives a templated release note, and then a bunch config showing how to parse commit messages and turn them into headings for release notes.

The numbered html comments are a hack to make it put the headings in the right order in the output.
The pattern matching for release tags has been customized to match semantic-version tags.
The booleans are set to ignore commits that don't have the prefix

Crafting commit messages

Once you (or someone else on your project) as set up the automation/scripting for release notes (and optionally automated releases, semantic versioning and tagging), then you not only have to write your git commits for future readers of git logs — something I hope you and your team already do — you now have a new audience to write for: the end users of your software. Plus if you are in a software delivery organisation you are also writing for all the other people involved that need to understand what is being released (product managers/owners, sales, engineering leaders, C-suite, marketing etc etc). That's a lot to consider if you are used to just writing "wip", "misc bugfixes" or "Fix ticket 123" as your entire commit message.

My first attempts at writing commit messages to feed into automated releases produced pretty dismal results. When I looked at the generated release notes I ended up manually rewriting most of what had been created - it had too much detail, too little detail, technical details irrelevant to the end user, duplicated entries etc etc.

Over time — by nothing more than the simple feedback loop of writing commits, previewing release notes, publishing releases and reviewing the resultant notes — I slowly got better and better at writing the commit messages the first time round that would make for nice release notes, making some tweaks to the cliff config along the way (in the same branch) when it didn't quite produce the desired results.

I also learned to shift-left and get faster feedback by more regularly running the preview of the release notes on my feature branch as I went along. Once you have merged a branch it's a lot more painful to tweak messages, and if your team has the main branch protected it may just be too late by the time the branch is merged to fix the generated release notes.

There isn't really any magic here. Write some commits, preview the release notes, and think about whether they would read well for your end users. Repeat until you get good results and happy users. The main thing is just having convenient infrastructure and scripting set up to make that easy.

Mainline & pull requests

There is no shortage of bike-shedding about git branching structures (trunk based, github flow, git flow, straight-line only).

When I started this journey I was a bit confused about where to put these magical release-note-generating commit messages. It originally seemed to me that it would make sense to have every commit on main have one of these prefixes to describe what type of commit it is - feat: Some feature, refactor: Reformat things, fix: Fix bug in X, but having attempted that for a bit I found it was sometimes really hard to work out what category to put everything in; and when I looked at the final list from a user perspective of "what changed" I just couldn't see the wood for the trees. Then there is the issue of the standard "github flow" of creating a feature branch, adding a set of [clear and well described] commits, and then creating a merge commit back into main (with or without github's help) - where should these prefixes go? In just the merge commits? In the branch commits? Both?

It was only when I actually started generating real release notes that I realized it's actually very simple:

There should be one "conventional commit" in the branch for each thing you want in the release notes
Everything else leading to that, including refactors and groundwork towards that thing should not have the conventional commit prefix.
git-cliff should be configured to ignore anything lacking the prefix

That way all you have to do is make sure you have one commit with the right prefix somewhere in your branch for each thing you want to show in the release notes. Everything else can carry on being committed as you always have done without affecting the release notes.

Ideally you are careful with your commits, and they all lead up to a commit that makes the feature or fix "done" which is the one with the special prefix. But even if you are still in the land of "wip" and "fixup" and "oopsie" commit messages, so long as you throw one well written "feat: Something users care about" you'll still get good release notes.

Better still, if you are doing pull requests and reviews - the generation of release notes from your not-yet-merged feature branch can be checked as part of the review, so you can even peer-review your release notes.

Not every commit is worthy of inclusion

I originally thought every commit had to have some kind of prefix. Including ci:, refactor: etc.

Having actually practiced with generation of real notes I've realised:

It's okay to put prefixes on commits for ci: and refactor: - but they should be skipped when building release notes. End users of a library or web app don't generally care about your cleanup and your build/deploy pipeline
All the other normal development work can just not have the prefix - and that is okay. Just because you are using the fancy-sounding "conventional commits" to generate release notes doesn't mean you have to dogmatically prefix every commit. I mean, if you want to prefix everything with prefixes or even emoji then knock yourself out, but you only need to scan for a few key entries for release note generation (feat, fix, perf).

Who are you writing release notes for

You presumably have a reason you are even bothering with release notes. And you presumably have someone who wants to know what's in a release. Otherwise why would you bother with any of this.

If you are continuously-deploying an e-commerce site, then maybe no-one cares and you should just focus on shipping whatever as fast as possible. Your product owner probably already knows what they asked you to do, and a kanban board of todo/doing/done might be enough.

If you are writing a library for other developers then it's pretty standard to use semantic versioning to indicate whether changes are likely to break api compatibility, and it's pretty normal for users of a library to expect to have some explanation of what changed between published versions (and no "security and performance updates" is not good enough). If this is the case, then all you have to do when writing your semantic commits for your release notes is consider what the library user would want to know and write your commits in those terms. Then check the preview of the release notes to make sure it came out as you planned.

If you are writing some kind of deployed app, say a microservice or web-app, then you have to consider your versioning strategy, why you even need release notes, and who your audience is. This could be an internal audience - say customer services or marketing, or it could be something you publish online for your end users. Whoever it is, make sure you preview your release notes as you go to ensure they are written with those people in mind, and write it the same way you would explain it to them on a call.

Better still get feedback from actual end users on how well the release notes are helping them, and then use that to drive even better semantic commit writing.

Ticket URLs

It's worth adding full ticket or github issue URLs to the end of your commit message(s) regardless of release notes - it makes it much easier for future readers of git logs to find the relevant ticket/issue, and by using full URLs instead of just numbers it prevents confusion later on over which system they refer to (teams do sometimes change their ticketing systems).

You can configure git-cliff to hunt for those ticket/issue URLs and include them in the release notes. This is a nice touch and make it easier to quickly jump from release notes back to the ticket.

For github issues you only need to include the number with a hash and it'll autolink those if it finds them in release notes (e.g. Some fix (#123)). I got cliff to find the "Fixes " in the commit footer (which also automagically closes the issue when merged to main) and suffix it on the line in the release notes.

Having run this for a while I ran into a complication with ticket links: sometimes you'll do some work that is related to or works towards a ticket, but doesn't actually finish the feature as far as an end user is concerned. If you were to then include that ticket in the release notes the user (or your wider team) would wonder why you have said it's done when it's not. To get around this I recommend using the "Fixes " for when you have completed a feature/fix, and the bare "" with no prefix in the footer of your commit messages to make it clear whether it is complete or just related; and then configure cliff so that anything that doesn't have the "Fixes" prefix ends up in a "Related work" or "Work towards" section.

Get started then iterate

It's hard to get this reliably right first time, but get that feedback cycle going by getting the tooling in place, starting the practice of writing commits in this style and looking at the generated release notes.

If you are sharing release notes manually you can always review and edit them if you can't get them exactly right, but the closer you can get to complete automation the better.

It's better to get started imperfectly now and improve than to put it off because the results aren't perfect.

You'll find it takes time to get the automation and parsing just right, and take time for the team to get the hang of getting the messages just right to generated high quality release notes. Help each other out, practice and iterate.

Over to you

This is the ultimate shift-left on release notes. You are now crafting your release notes at the same time as you are crafting the code for the feature for your dear users.

It seems to be a fairly uncommon practice currently, but it's now pretty easily achievable with a bit of time on the tooling and a bit of practice from the devs in writing commits in this style. I'm keen to help with adoption of this improvement, and I believe a key piece of the puzzle is knowing it's possible and how to do it effectively, as such I want to make this post a good reference guide to jump-start teams who'd like to get this going as a practice. Please do give me a shout if you'd like to talk it over, and let me know if you think I've missed anything, anything isn't clear, or if you disagree with any of it.

And please do give it a go, and report back with how you get on.

Addendum — em-dashes

Before anyone complains that AI wrote this post, I put the em-dashes in manually, I wrote this entire blog post by hand — unless you count vim — and did not even have the LLM do editing or proof reading.

Thanks to the world of LLM output I actually now know what an em-dash is and how to use it when writing. So with that, I can totally own all the em-dashes, typos mistakes and errors herein. Enjoy!

Using Gitopolis to Manage Multiple Git Repositories

Tim Abell — Sat, 25 Oct 2025 00:00:00 +0000

At some point, every microservices project turns into a small city of Git repos.

You start with three or four, and before long you’ve got twenty.

Gitopolis makes that manageable.

auth/
billing/
users/
orders/
payments/
inventory/
notifications/
web-ui/
admin-ui/
mobile/
infra/
ci/
monitoring/
alerts/
gateway/
docs/
devtools/
scripts/
design-system/
shared/

Alice is a backend developer, but she needs to keep an eye on frontend and infra changes too.

All of it matters. All of it lives in Git.

Day One: Setup

She installs Gitopolis and points it at her working directory:

cd ~/work
gitopolis add *

Fifteen existing repos are added instantly.

Gitopolis writes a .gitopolis.toml file listing their paths, remotes, and metadata.

It’s plain text — no magic, no database.

Then she tags them by team:

gitopolis tag backend auth billing users orders payments inventory notifications gateway
gitopolis tag frontend web-ui admin-ui mobile design-system
gitopolis tag infra infra ci monitoring alerts
gitopolis tag shared shared devtools scripts docs

Now she can focus on only what matters that day.

Day Two: Payoff

Instead of jumping between folders, she runs everything from the top level.

Pull all backend repos:

gitopolis exec -t backend -- git pull

Check their current status:

gitopolis exec -t backend -- git status

See what’s changed in the last week:

gitopolis exec -t backend -- git log -n 5 --since="One Week"

She creates a few shell aliases:

alias gpl='gitopolis exec -t backend -- git pull'
alias gst='gitopolis exec -t backend -- git status'
alias gls='gitopolis exec -t backend -- git log -n 5 --since="One Week"'

Now she runs gpl with her morning coffee, gst before lunch, gls before stand-up.

Her workflow stops feeling scattered.

Day Three: A New Starter

The company’s onboarding process mentions “Install Gitopolis”.

Alice drops a copy of .gitopolis.toml into the shared Google Drive folder and links it in the new-starter docs.

When Dana joins the team, she runs:

mkdir ~/work && cd ~/work
cp ~/GoogleDrive/dev-setup/.gitopolis.toml .
gitopolis clone

All the right repos appear in the right places.

Alice shows her how to explore the setup:

$ gitopolis tags --long

backend
        auth
        billing
        users
        orders
        payments
        inventory
        notifications
        gateway
        shipping

frontend
        web-ui
        admin-ui
        mobile
        design-system
        returns

infra
        infra
        ci
        monitoring
        alerts

shared
        shared
        devtools
        scripts
        docs

Dana can instantly see which repos belong to which teams — backend, frontend, infra, shared.

She starts working without having to ask where anything lives.

Day Four: Evolving

As the project grows, Alice adds and syncs new repos as needed.

Adding a local repo she’s just created:

gitopolis add shipping
gitopolis tag backend shipping

Cloning a new remote one that Chloe (frontend) just pushed:

gitopolis clone git@github.com:org/returns.git
gitopolis tag frontend returns

When remotes move or change names:

gitopolis sync --read-remotes

The .gitopolis.toml stays current. Everyone stays in sync.

Immediate payoff for one developer before anyone else uses it.
Tag-scoped focus to limit noise across teams.
Simple onboarding with a shared config file.
No daemons, no central service — just text and Git.

Hope that helps you see how Gitopolis can be useful in a real project.

I’ve done everything above with names changed to avoid mentioning clients and details simplified for the sake of explanation.

Let me know how you get on with it or if you have any problems.

More info: github.com/rustworkshop/gitopolis

Should you create tickets for tech tasks?

Tim Abell — Mon, 07 Oct 2024 00:00:00 +0000

In the manner of choosing the colour to paint the bikeshed, the decision of whether to create a ticket for every single tiny commit, no matter whether it’s a giant feature or the tiniest whitespace cleanup in the readme file continues to consume countless hours. This question is not really that important when it comes to delivering software, yet it still comes up from time to time and there is a right and wrong answer, which should be modified only with consideration for the current situation.

In short

Correct: create tickets only for features and larger technical work. Allow engineers to commit smaller improvements ad-hoc.
Incorrect: create an atmosphere of stifling bureaucracy where ll attempts to keep the work area tidy will require sign-off from the CFO and approval from the product owner who has never written a line of code (some have, but that’s a rare wonder).

Who cares?

If anyone wonders why engineers get so bent out of shape about this seemingly harmless request, imagine if someone asked you to fill out a form and submit it for every single piece of your work that you currently don’t have to. Want to pick up the phone?… create a ticket. Want to send an email?…

create a ticket. Can you imagine that this might make you a) miserable, b) less productive, and c) give up trying to do more than the bare minimum. By requesting tickets for everything you are introducing friction directly into the path of a programmer’s most important output - the codebase. We instinctively know that fast iteration and short cycle times lead to better delivery of value to users, and instinctively dislike anything that gets in the way. Note that the “auditing” and “why it changed” are already covered by source control (git) history and “pull requests” so adding tickets is redundant toil.

Turning to GPT for explanations

And now, given that there can be reasons for adding a ticket link to everything (for example generating change logs from git), and in the modern tradition, we turn to ChatGPT for a properly argued discussion of both sides of the argument.

The following is an edited result of my discussion with ChatGPT on the subject. Should you find yourself in the situation where someone has decided that every brush stroke during the painting of the bikeshed should require a JIRA ticket then this article will provide a handy reference to all the arguments that can be made around this oh-so-trivial subject.

I particularly like the fact you can ask GPT to write in the styles of different personas, so first we have the curt “lead dev” version that gets to the point and is devoid of superfluous wording, empathy, and any other affordances to anyone who thinks in any other way than the tech lead; and then we have the more wordy default style of GPT that explains the points a bit more fully for the benefit of those who need a bit more explanation around the subject. Both are useful depending on the audience, so take your pick.

Tech-lead-GPT Says…

Following the usual wordy explanation I fed GPT this follow-up prompt:

“write it in the style of a busy lead dev engineer - still too fancy and formatted. plain ascii, missing capitalisation in places, nothing more than the bare facts”

And got this straight-to-the-point explanation, lazy capitalization and all:

requiring a ticket for every tech improvement pr has upsides but will slow us down in key areas. here’s why:

pros:

prioritization: a ticket makes sure we focus on impactful work. it forces us to ask if this improvement is really needed now or if it can wait.
consistency: everyone follows the same process. good for cross-team work or when other stakeholders need visibility.
accountability: tickets clarify why a change is being made, what the risks are, and who owns it.

cons:

overhead: creating tickets for minor improvements (e.g. small refactors or cleanup) adds unnecessary steps. this slows devs down and eats up time for actual coding.
discourages quick fixes: devs spot opportunities to fix things as they go. forcing them to stop and make a ticket breaks their flow and makes it less likely they’ll bother with small but important changes.
ticket bloat: if every tiny improvement needs a ticket, we’ll end up with a cluttered backlog full of minor stuff, making it harder to see what’s actually important.
developer autonomy: devs should be trusted to make small improvements without red tape. too many rules can kill morale and make them feel micromanaged.

suggestion:

use tickets for significant changes (major refactors, cross-team impact, etc.), not small things like code cleanup or minor improvements.
use pr descriptions to document and discuss small technical changes. keeps things lightweight without losing track of what’s been done.
let devs decide when a ticket is needed vs when a quick pr will do. they’ll know best.

bottom line: tickets are useful for big changes but requiring them for every single tech improvement will slow us down more than it helps. let’s keep the process flexible where it makes sense.

Default friendly GPT explains fully

This part is in GPT’s default style, with some editing of what points to include. It’s pretty much as it was generated, and I think does a good job of explaining the trade-offs for discussion:

Requiring a ticket for every technical improvement pull request is a policy with both advantages and drawbacks, and it’s important to weigh these carefully to ensure the team’s workflow is as efficient and effective as possible. Pros of Requiring Tickets for Technical Improvements

Improved Prioritization and Planning Tickets create a formal process for prioritizing work. When technical improvements require a ticket, it forces a discussion on whether the improvement is truly necessary or how urgent it is. This helps teams focus on the most impactful changes and avoid unnecessary or lower-priority adjustments, allowing better alignment with the overall project goals.
Consistency Across Teams In larger teams or organizations, requiring tickets for all work (including technical improvements) helps ensure consistency. Everyone follows the same process, which makes it easier to manage cross-team dependencies, ensure the necessary approvals, and communicate changes to stakeholders.
Clearer Communication and Accountability Tickets provide a clear space for discussing the reasoning behind technical improvements, including potential risks, trade-offs, or alternatives. This promotes better communication among developers, product managers, and other stakeholders, and helps ensure that everyone is on the same page. It also adds accountability, as each improvement is documented and has a clear owner.

Cons of Requiring Tickets for Technical Improvements

Increased Administrative Overhead One of the biggest downsides is the added overhead. Many technical improvements are small, straightforward fixes or optimizations that don’t warrant the time and effort needed to create, manage, and track a ticket. Forcing developers to do this extra work can slow down progress and take time away from actual coding. Over time, this could lead to frustration or a slowdown in the pace of incremental improvements.
Discourages Quick, Organic Improvements Developers often spot opportunities for small technical improvements while they’re working on other tasks. Having to create a ticket for each of these can disrupt their workflow, making them less likely to act on these improvements when they notice them. This can lead to missed opportunities to clean up code, reduce technical debt, or optimize performance in small but meaningful ways.
Potential for Ticket Bloat In a fast-moving development environment, the requirement for tickets on small improvements could create a flood of minor issues in the ticketing system. This can make it harder to manage more critical work and obscure visibility into the progress of higher-priority features or bugs. The sheer volume of tickets can become difficult to sort through, adding cognitive load to project management.
Less Developer Autonomy Developers often thrive in environments where they can take initiative to improve the codebase. Requiring tickets for every small change might reduce their sense of ownership, as it introduces a layer of bureaucracy between identifying an issue and resolving it. This could dampen their motivation to take care of small, yet important, aspects of code quality.

A Balanced Approach

Given the pros and cons, there may be a middle ground that allows the team to capture the benefits of ticketing while avoiding its downsides:

Ticketing for Significant Changes: Tickets could be required for larger technical improvements that have broader implications, such as major refactors, changes that affect multiple teams, or improvements that touch critical parts of the system. For smaller changes, like simple code cleanup, minor refactoring, or bug fixes, developers could be allowed to open a pull request without the overhead of a ticket.
Use Pull Requests for Tracking: The pull request itself can serve as a lightweight way to document and discuss technical improvements. Detailed PR descriptions, along with the review process, can provide enough visibility and communication for smaller improvements without the need for a formal ticket.
Developer Discretion: Allow developers to decide whether a ticket is needed based on the scope of the improvement. They are often best positioned to know whether a change is minor enough to be handled via a simple PR or whether it requires more discussion and prioritization through a ticket.

Conclusion

Requiring tickets for every technical improvement has clear advantages in terms of prioritization, consistency, and accountability, particularly in larger teams or complex projects. However, the added overhead, potential for slowing down minor improvements, and impact on developer autonomy are significant downsides to consider. A balanced approach that applies ticketing requirements more selectively, while still ensuring visibility and discussion through PRs, could be a better fit for fostering both productivity and quality in the development process.

Why user stories should be shippable increments

Tim Abell — Thu, 12 Sep 2024 00:00:00 +0000

Sometimes I see people create work items / jira tickets / user- stories that if they were implemented and merged in to the codebase would leave the system in a half-finished state, either broken or inconsistent.

The problem with non-shippable tickets

This to me is instinctively a “bad thing”, however as not everyone has grokked this important fundamental of team software delivery, I turned to chat gpt for an explanation of why I think this is important, and I have to say it did a stellar job of clearly explaining the things that I have absorbed over my decades of commercial software development. So here you go, here’s why non-shippable tickets are a problem:

Incomplete Features Impact User Experience

A half-finished feature can confuse users or negatively impact their experience. If the feature is exposed to users before it’s fully functional, it may appear broken, resulting in dissatisfaction or mistrust in the system’s reliability.

Difficult to Test and Validate

Testing half-finished work is challenging, as it’s not in a stable or coherent state. Quality Assurance (QA) teams need a fully functional piece of work to validate it against the requirements. When a ticket is only partially completed, it’s hard to determine if it meets the intended functionality or introduces bugs.

Complicates Future Development

Leaving an unfinished task can create dependencies and conflicts with future work. If another developer later works on a related feature, they might have to deal with the incomplete state, which could introduce confusion or lead to mistakes in integrating related functionality.

Increases Technical Debt

Incomplete features often accumulate as technical debt, which means extra work will be needed later to finish, refactor, or fix them. This can slow down future progress, as more time will need to be spent revisiting and cleaning up the half-finished work.

Blocks Related Development

A half-finished ticket can block progress on other features that depend on it. For example, if a partially completed feature introduces bugs or changes a shared system in an untested way, it could delay the development of other tickets or cause cascading issues across the system.

Reduces Team Efficiency

When a ticket is left in a half-finished state, developers may need to pick it up again later or pass it to another developer, leading to context-switching. This results in inefficiency, as time is wasted on re-clarifying the problem, re-understanding the code, and figuring out where the previous developer left off.

Unclear Ownership and Accountability

If a ticket is left unfinished, ownership and accountability become unclear. Other team members may not know who is responsible for completing it or what still needs to be done, which can cause confusion and reduce productivity.

Inconsistent System Behavior

Partial implementation can cause the system to behave inconsistently. For example, a feature that’s half-completed might trigger errors, create discrepancies in data, or cause parts of the system to function incorrectly or unpredictably.

Hinders Deployment and Releases

Half-finished tickets often disrupt deployment cycles, as incomplete features cannot be included in production. This results in unnecessary delays or the need to work around unfinished tickets, making releases more complex and error-prone.

Wasted Effort and Resources

When a ticket is not fully completed, there’s a risk that the work may be scrapped or need significant changes by the time it’s revisited. This can result in wasted development effort, resources, and time.

Unmet Stakeholder Expectations

Leaving a ticket incomplete prevents stakeholders from seeing tangible progress, undermining confidence in the team’s ability to deliver. Stakeholders expect finished, usable deliverables that they can review and assess, not half-completed tasks that require further work.

Difficult to Manage in Agile Processes

In Agile and Scrum methodologies, the goal is to deliver shippable increments of work at the end of each sprint. Leaving a ticket half-finished breaks this cycle, making it hard to close sprints successfully and creating spillover tasks that carry into future iterations.

In short

Avoiding half-finished tickets ensures that the system remains stable, progress is clear, and each ticket represents a functional, complete deliverable. This promotes better workflow, higher quality, and more reliable software.

How to create small shippable batches

That’s all very well you say, but I don’t want to end up with giant, big-bang stories where engineering vanish for a month or six until every last thing works (if it ever does), so what to do?

Again I’ve turned to gpt as a starting point, and it’s not done a bad job, I’ve done a bit more editing to this set and added some of my own thoughts:

Use Vertical Slicing

Vertical slicing means breaking down features into small, independent pieces that cut through all layers of the stack (UI, backend, database, etc.) and provide end-to-end functionality. Each slice delivers a fully functional piece of the feature, even if it’s a smaller or simpler version of the final product. Example: Instead of building a complete “user profile” page, deliver a ticket for “adding a profile picture” with all the components needed (UI, API, storage).

Define Clear Acceptance Criteria

Each ticket should have clear, well-defined acceptance criteria that detail exactly what constitutes a “done” state. These criteria ensure that the ticket can be completed independently and is deliverable as a unit.

Focus on the minimum viable product

Like the Minimal Viable Product (MVP) concept, you can apply the same thinking to individual product increments.

When designing features, focus on delivering the smallest possible version that is still valuable to users. This forces the team to identify the core functionality that is immediately useful, allowing for iterative improvement later.

Example: Instead of delivering a fully-featured analytics dashboard, start with a basic version that shows just the key metrics, and expand later.

Implement Feature Flags

Use feature flags to allow partially completed features to be merged into the main codebase without affecting the user experience. This ensures that work can be released incrementally but hidden or disabled until it is complete.

Example: A half-built “comments section” can be developed over several tickets but hidden from users until all parts are ready.

Break Down Complex Features

Complex features should be split into smaller sub-features that can each be delivered independently. Ensure that each sub-feature is fully functional, even if it doesn’t have all the final functionality.

Avoid Over-Engineering

Start with the simplest solution to a problem, and iterate.

Avoid adding extra complexity upfront by sticking to what’s necessary to fulfill the ticket’s requirements for delivery.

This includes “gold plating” and “future proofing” when implementing the ticket. Just do the minimum high-quality code that you know you need now.

Make Use of Stubs and Mocks

To avoid leaving a ticket half-finished when other dependencies aren’t ready, use stubs or mocks for parts of the system that are still under development. This allows you to complete and deliver your ticket while waiting for those dependencies.

Example: If the backend API isn’t ready, a front-end ticket can use mock data to simulate responses, allowing the UI to be built and delivered without waiting.

This is more appropriate for very early product development rather than for modifying an established system.

Communicate Dependencies Clearly

If a feature is dependent on other parts of the system, make sure the dependencies are identified and managed.

Sometimes it makes sense to coordinate related tickets so they’re developed together or in the right sequence.

Example: If a new API endpoint is needed for a UI feature, ensure the API ticket is prioritized so the front-end team isn’t blocked, or vice versa.

Collaborate in Cross-Functional Teams

Ensure cross-functional teams collaborate on the entire stack of a feature (UI, backend, database, etc.) within the same sprint or ticket. This ensures no part of the feature is half-done or stuck waiting for another team to finish their part.

Example: A ticket to create a new form should include front-end developers, back-end developers, and testers working together to ensure it’s functional from end to end.

Cross-functional brainstorming of story creation

There’s nothing like all members of a team, including product, engineering, UX etc all sitting round a table and throwing around ideas to see if they can come up with a creative way to break down a problem into smaller but still shippable pieces.

I’ve repeated this exercise with many clients and the flashes of inspiration that come out of it cannot be replicated by thinking harder on one’s own while starting at Jira.

Summary

By structuring work this way, teams can maintain a balance between small, manageable batches of work and ensuring that each piece is fully deliverable and functional on its own.

This approach maximizes delivery velocity while maintaining system integrity and delivering value incrementally.

Running a beefy virtualbox dev server

Tim Abell — Fri, 07 Jun 2024 00:00:00 +0000

Why

Sometimes I have to work on hairy old sprawling legacy code bases in ye-olde c#, which means, unfortunately, no shiny new dotnet-core with Rider on linux and all the speed gains that brings.

So having fallen over another codebase that takes unbearably long to build and work on when combined with the need to run visual studio in a windows virtualbox vm on top of my linux box I’ve shelled out for a desktop pc with a top-end cpu to use as a dev box for those trickier builds. But who wants to give up the joy of a portable laptop, so here I am with the challenge of connecting to virtualbox from another machine. Turns out there’s a few non-obvious things and options that may or may not work for you, so here’s what worked for me.

GNU’s Not Unix and neither is Windows

Aside - windows will never again have ownership of my computer, I relegated it to a VM for emergencies (i.e. clients paying money), where it can stay within those steely confines unable to bother me the rest of the time. But that’s another blog post really.

Basic setup

This is pretty simple:

Install Linux Mint because home servers are a lot easier to deal with when they’re actually just normal desktop operating systems that are switched on more.
1. Enable full disk encryption during setup (advanced > lvm > encrypt) (this is a bit of a nuisance for remote management but important for security if the machine is stolen)
Turn on automatic updates
Install virtualbox and openssh-server
Copy the “Virtualbox VMs” folder across by opening sftp://ur-server-here/ in nemo (the file browser) on the source machine. It’s not the quickest but it did 200gb in 5 hours, and I have other things to do in my life so that’s fine.
Fire it up on the desktop with the virtualbox gui just to check it’s fine

Remote access - linux

To get some remote control of the machine simple ssh and ssh -X are enough for cli actions and some gui actions respectively (-X tunnels x-windows over ssh, meaning you can launch graphical programs on the remote machine and they pop up on the local display. Magic.)

Remote access windows vm

This proved to be less obvious.

There are three possible layers to remote in:

Remote access to the host machine UI
Remote access to the virtualbox process
Remote access to the windows vm

Only number 3, directly connecting to the guest windows vm with RDP worked for me.

1. Host connection

Remote desktop in linux is still a bit of a mess.

VNC is klunky old, and I don’t even know if it’s secure any more, it’s also not available out of the box. I didn’t even try.

There’s some more or less proprietary ones like nx (nomachine?), rustdesk, go-to-my-pc etc, which I either haven’t had any luck with previously or haven’t tried / don’t trust.

I tried running the virtualbox command gui over ssh, but it had two problems:

It was too laggy / slow updating the screen to be useable for intense visual studio coding work
Disconnecting killed the virtualbox process and terminated the vm without proper shutdown

I quickly gave up on this and tried #2…

2. Remote access to virtualbox

Tantalizingly virtualbox has RDP support available…. but the punchline is that it’s not part of the open source project, and is a proprietary “extension” that requires agreement to a personal/evaluation license that explicitly forbids commercial use, so that was out. Darn. If you’re only doing personal things you could try this, but I’m not so can’t.

3. Remote access to windows vm

This was the winner in the end but requires a few steps to get set up:

Change the virtualbox vm network settings to “bridged” instead of the default “NAT” so that the vm is available directly on the network.
Grab the generated ipv6 address for the machine (or give it a static ip4 lease)
In the windows guest:
1. Enable remote desktop
2. Give the user account a password if it doesn’t already have one (you can’t connect remotely with a passwordless user).
Start the vm with the virtualbox cli instead of the gui (avoids it being terminated when the shell session ends):
1. vboxmanage startvm WinDev2404Eval --type headless
On the client laptop install the remmina rdp client with apt
Connect to the vm in remmina with the ip6 address of the guest machine (not the host desktop)
Enter the windows user’s username & password when prompted

Remote management tips

Here’s some more useful commands for managing the server machine remotely over ssh:

List virtual machines (useful for getting the name):

vboxmanage list vms

Start a virtual machines

vboxmanage startvm WinDev2404Eval --type headless

Shut down the host

sudo shutdown -h now

(see https://explainshell.com/explain?cmd=shutdown+-h+now)

Suspend the host:

systemctl suspend

(ref https://askubuntu.com/questions/1792/how-can-i-suspend-hibernate-from-command-line/1795#1795

Was it worth it?

Build time of a gnarly project on laptop:

========== Rebuild completed at 02:44 and took 20:08.230 minutes ==========

And the same vm running the same full rebuild remoted into the new desktop:

========== Rebuild completed at 02:28 and took 04:15.142 minutes ==========

You can even copy paste text like this straight out of the RDP’d VM which will be immensely useful.

The end

That’s all for this one. An obscure technical thing, that was useful to me, and just tricky enough to be worth documenting for future-me and the rest of the interwebs in case anyone finds it useful. Hurrah for blogs and zero cost of replication.

Till next time. 👋

Trunk based (mainline) development is (mostly) wrong

Tim Abell — Thu, 18 Apr 2024 00:00:00 +0000

Mainlining

Some very experienced developers, some of whom I’ve heard it from in-person, strongly advocate what is often called “mainline” or “trunk-based” development, meaning that the git history is a series of commits directly to the main branch, with no pull requests or merge commits in sight. This is often held up as the panacea for achieving fast, high-quality delivery.

I. Do. Not. Agree. The absolute rule of “no merge commits on main” (aka straight-line history), sometimes enforced by github configuration, is absolute balderdash. This assertion is typical of the black-and-white presentations of topics that seem oh-so-appealing when presented in the conference talk circuit where there is no room for nuance and actual trade-offs made in the trenches, and where big extreme statements win the game of attention.

There is a place for thoughtful, individual mainline commits, so don’t take this as saying the opposite extreme of “only merge commits / PRs on main”. A small-to-medium internally consistent patch (say 1 to 200 lines of diff at most), can be quite a reasonable thing to just push to main.

Does pairing remove the need for PRs?

For anything more than trivial changes in your production code and test code you probably want two pairs of eyes on it, and pairing is a great way to achieve that in real-time. But does pairing remove the need for a PR entirely? Some say yes, but I say it depends on the actual patch …

The question of pairing and the question of “mainline vs pull request” are almost entirely orthogonal. Pairing fixes the async-review problem, and certainly reduces the need for PRs to achieve peer-review, but that doesn’t mean we should reflexively throw out feature branches, merge commits and PRs with the proverbial bath water.

Why branches and merge-commits matter

Unconditionally mainlining everything loses some important capabilities:

The ability to group a series of related commits (e.g. reformat, refactor, add feature), into one coherent group by way of merging them to main in one hit. (With or without github and its PRs.) A future reader can use this to look at history at two levels of detail - using git log --first-parent to look first at the merges and commits to main, and then only if it’s interesting look at the series of patches in a feature branch that was merged in. And conversely it allows breaking down a patch that needs to go into main in one hit (to avoid breaking things by shipping half-finished work) into a series of easier to understand patches with meaningful commit messages.
The ability to check your changes with CI in github before merging to main to be sure you don’t break main for everyone else (plus the permanent record of another “Fix main build, oopsie” commit).
The ability to offload regression checks to github actions instead of having to always run all the tests locally.
The ability to create a series of commits, discover they are wrong several hours later, and throw them out or rewrite them before they ever hit main.

If you think mainlining generates a clean history, how often have you made a series of commits before realising that the shape of code isn’t right and now you have another series of commits to get it into a different shape. With mainline development that completely irrelevant first attempt is now in history to confuse future readers for ever more. With branch based development you can throw it out and pretend it never happened.

When to mainline, branch or PR

It’s a bit of an art, so here’s some rules of thumb to guide you through the mainline / branch / PR decisions as you write code:

Pair on everything you can, eliminating the need for a PR for peer-review. Only peel off to solo work when it’s painfully obvious that there’s zero value / knowledge-transfer / alignment to be gained by pairing (e.g. library upgrades).
Trivial, uncontroversial changes can be mainlined solo (if your org allows any solo pushes to main), especially if you run the tests locally before pushing. Watch out for linters and formatters in CI. Pre-commit / pre-push checks can help you avoid breaking the build with silly mistakes.
Slightly bigger changes can be pushed directly to main if you are pairing, but they should still be restricted to smaller, simple to understand patches; not entire large features.
When working on a larger feature, try and peel off as many unrelated pieces as you can and mainline or PR them separately as you go instead of lumping them in to your feature patch/branch, rebasing your feature patch/branch onto the updated main.
If a piece of work requires a few logically coherent steps to make the change, that are related or build on each other, then group them together in a feature branch. If you are pairing then you can run your tests & lint locally, create a merge commit on main that merges your branch in, and push that to main. No PR needed as it’s already been reviewed.
You may opt to open a PR in any of the above circumstances anyway in order to:
1. Get a run of CI
2. Get some async input from the broader dev team or across team boundaries.
3. To put a novel idea/technique into public and give it time to be considered by yourself and others. The draft PR feature of github can be useful for this. Also useful for “spikes” (experiments in code to learn something or try something out that aren’t necessarily production quality or ready to merge).
Beware of long running branches with many commits, even if they are lots of well crafted commits. It’s a sign your deliverable increment is too large (at feature branch and/or story level), and you should look to take a step back and break it down into smaller increments; maybe even ditching the branch and starting again from main.

Note that you need to be constantly considering these rules of thumb as you make every change to the code because in the fluid development flow it’s normal to range across all sorts of types of changes in one coding session (e.g. cleanup, refactoring, bug fixes, feature changes, quality improvements, ci fixes, editor config changes, massive file/folder renames, and the actual features, etc etc), and each of these might require a separate approach to generating a high-quality history.

Split up commits before you write more code

Flipping to main and making a separate commit when you realise you want/need a change that is unrelated to your feature branch is a great habit/skill to build. It does however take discipline and practice.

I see many developers who default to branch-based development and just pile any old thing they need into their current branch, ending up with mammoth PRs that change far too much in one go. They often then complain that splitting it back up is too hard so they should just squash it. As the old saying goes “if I wanted to get to there I wouldn’t have started from here”; it’s far better to split into commits as you go rather than struggling with git’s tricky-but-powerful git rebase --interactive.

I’ve had devs say to me that “it’s not worth the time” to create a series of atomic coherent and well described commits. And for evidence they often point to their massive commit that does too many things, or their pile of incoherent “wip” commits to a branch, combined with the effort it takes to split and recombine commits into something better. In commercial projects I often relent because at that point the horse has bolted, and they are right that it isn’t worth spending that much client/employer time rescuing their history aside from the opportunity to demonstrate rebase-interactive to them. (Besides, it’s their name on that commit forever, not mine). That doesn’t however make it good enough, or the correct conclusion. The correct answer is that you should work on getting the series of patches right as you go, and the cost of interactive rebases largely goes away.

Feature flags DO NOT replace feature branches

Some claim that feature flags (the ability to toggle capabilities on and off at runtime) are the reason you no longer need branches.

To be sure, they are a useful thing, and an answer to long-lived feature branches that can’t be merged because the feature isn’t ready to ship yet.

But just because you know how to use feature flags, still doesn’t mean you should throw out the richness and tools available to you with merge commits to main and PRs. The fact that some teams have painted themselves into a corner of PR-hell doesn’t mean you should throw out PRs entirely. You can have feature flags and the option to use branches, merge commits and PRs when it is the right tool for the job.

“But no-one cares about the history”

Someone said this to me today. They aren’t the first person to say it to me either. This is empirically wrong. A friend of mine calls people with this mindset pejoratively “BSBITS devs - Big Save Button In The Sky Developers” meaning that they think that the only thing that matters is the current production code, and that source control is a fancy save/share system.

Here are some of the ways that I have seen first-hand many many developers including myself “care” about history:

Understanding why the current code is how it is in order to know how/whether to change it.
Understanding the changes another developer made in order to replicate those changes in a different microservice in a different team.
Reviewing a large PR with 1000+ lines of code changed, where some of it is refactoring and reformatting and some is functional changes - looking at the commits of the PR/branch allows you to see the intended functional change (sometimes just a few lines) separately from large rote refactors etc.
To judge the capabilities and productivity of a developer with a view to hiring/firing. Yes this actually happens, I’ve seen it. And guess what, that developer doesn’t get to explain their poor quality history, and they can be judged harshly for it. Are you really sure no-one cares?
To get a feel for the capabilities of a co-worker.
To keep up to date with changes to a project that you are either involved in directly or indirectly, perhaps a repo belonging to another team.
Principle developers that operate cross-team looking for best/worst practices that they need to take action on. (They often have significant HR influence, do you want them judging your poor history badly because “no-one cares”?

Ironically I’ve even seen some of the people who’ve said to me “no-one cares about history” then go on at a later date to look at the git history in order to figure something out while I’m on a screenshare with them. It’s all I can do not to point out the hypocrisy.

Build that habit

In the end, a lot of the arguments I see come from people who haven’t mastered the skill of creating high-quality history with commits, branches and thoughtful merges.

The answer in my eyes is not to declare half of the tooling we have off-limits, and to claim that the generated history is unimportant, it is instead to get really good at creating good history as you go so that it become effortless.

Treat generation of history just like generation of code. It’s the most visible and permanent record of what you do as a developer of software, and you should treat it with the same pride and diligence as you do the code that runs in production.

Fin

Coding is not just computers running things, it’s inter-person communications. Quality history, and the richness that branches and merges give us is part of that tapestry of communication with persons past, present and future.

Become good at it and the objections melt away.

Why do automated tests matter?

Tim Abell — Wed, 27 Mar 2024 00:00:00 +0000

It might seem a bit odd to write a post on software tests after so many years and so much content, yet to this day I see well meaning developers writing software without adequate test coverage. In fact I will share that I myself have been very late to enlightenment on this front. Sure I’ve been “writing tests” for well over a decade, but I was missing the mental framework that would make those efforts coherent, complete and effective.

In my view the mental model of why we would write tests is more important than the detailed “how” of writing unit/integration tests, because without that it can only be at best haphazard whether the testing really serves its true purpose.

I’ll share with you here the keystone ideas that make it all sit together properly, just like the keystone in a stone archway. Then from there, all the tests you write will sit in their true and proper place within that framework, or be obviously either waste or missing.

Yes I’m aware this particular archway in the photo doesn’t actually have a keystone, but I took the pic and I rather like it so it’s staying.

The Before Time: TDD & Unit Testing

I started my career in software development way back in 2000 when only a few very forward thinking people were doing high quality automated testing, and it really hadn’t reached the broader consciousness off the software development community.

Like a large number of software devs of that era, I became aware of “software testing” through the “TDD” movement (test-driven development). I believe a lot of the noise about this came out of the Ruby on Rails community (no coincidence, there’s no compile step so there’s more reliance on good tests in ruby/rails than in compiled languages). My experience of this was that it was largely talked about and taught as a bottom-up approach to testing:

“Here’s how you write a unit test for a class.”
“Here’s how you do assertions.”
“Here’s why you should write the test before the code (red-green-refactor).”
And in C# land “here’s how you do dependency injection”.

This is all good and important to know but is not enough in itself.

This was also a time when SOLID was big in the OO consciousness, so classes, isolation, encapsulation and all the small pieces of the machine you were building was the focus, and testing those pieces was often enough to claim you were “doing testing”. Interviews often being more about “do you TDD” than “will your system break for end users”.

“Integration Testing” (i.e testing more than one piece together) was certainly in my awareness, but mostly as a TDD++, and was usually missing any kind of coherent “why”. You might test some code with a real database instead of a mocked persistence layer, or check that all your pure code classes didn’t blow up when they were wired back together. In C# world a lot of mental overhead was created by trying to work out how to test code that used Microsoft’s not very test-friendly .NET Framework standard library code.

So I bumbled along, more or less successfully writing tests for my software, but always feeling like I didn’t have a really solid argument for the big picture. For me it was mostly “aim for 100% test coverage” and you’ll catch/prevent lots of bugs, plus it was good at driving software architecture in the minutia by preventing things being too coupled together.

More recently I’ve had the pleasure on working with people who are super-keen on “outside-in” testing and considerably less keen on acres of unit tests, and it finally dawned on me what I’ve been missing all these years. What follows is the missing piece:

The “Why” of Testing

So let’s take a step back for a moment. Why do we write tests at all, what is the big goal that makes this all worthwhile.

There’s long been discussions of the “cost/benefit” of tests, and there is research that shows teams that write tests are more productive, which is good, but a bit abstract when it comes to what we actually need to write to be effective.

Preventing Regressions

The goal of writing any software is working software for users and businesses; and they aren’t going to be too happy when something that worked fine on Monday is now broken on Tuesday.

I don’t think it’s news to anyone that the goal of software tests is to prevent regressions.

Axioms

There are two pieces of the testing puzzle that “preventing regressions” alone didn’t make immediately clear to me, and these are they key reasons I’m taking the time to write this post at all. They drive a subtle but foundational shift in what our writing of tests actually looks like and the magnitude of their effectiveness.

They may seem trivial and obvious at first sight, but do not ignore them so quickly as they are the axioms upon which the whole approach rests. They are:

What matters is not whether your class has “tests” but whether the software as perceived by the user does the job they need.
When you add your first feature it’s easy to manually verify it’s behaviour. When you write your ninety-ninth feature you still want to be certain that the other ninety-eight features are all still intact.

Most systems have many thousands of “features”, especially if you include “non-functional” cross-cutting requirements such as acceptable performance for each capability.

Full, Automated Coverage

Every time you touch your software, there is a non-zero risk that something that used to work will no longer work. Anyone who’s written software for any length of time will laugh at the idea that “a change in component A has no possibility of breaking something unrelated in component B”. And as a more rigorous colleague pointed out, we have problem of “emergence” in complex systems which makes it increasingly hard to predict behaviour. A good measure of your tests is how afraid you are of upgrading third-party dependencies, running you tests and immediately shipping the result without further manual verification.

The only sustainable solution to the presented axioms combined with the need to add more features over time is:

Ensure every feature the user cares about has a test.
The software is tested from the perspective of the user.
The software is tested after every change.
This testing is fully automated.

Insufficient Test Automation

If you write insufficient automated tests there are only two things that can happen, both bad:

You spend exponentially more time manually verifying all expected behaviour before any change is passed on for the user to use; bugs get through anyway. Or
You give up trying to test what was supposed to work, and some of it stops working.

Full automated test coverage of all delivered features to date is the only solution to this problem. Any shortcoming in this coverage is a subclass of the generalized “technical debt” problem that quickly results in a catastrophic drop-off of ability to delivery anything at all if it is allowed to grow:

What Kind of Tests

So, if you accept all of the above, what does that mean when you actually fire up your editor and wonder what test to write?

You already write unit tests, that’s enough, right? … Bzzzzt, Wrong! That fails the “from the user perspective” need.

If the only thing that matters is that a feature works from the users perspective then the only automated test that matters is one that tests a feature from the users perspective. In practice that means things like:

End to end tests
Browser automation
Smoke tests
Platform tests
Performance tests
Outside-in tests

And what doesn’t matter a hoot is things like unit tests, integration tests and component tests, i.e. the very things that we were taught in “TDD school”, and then left to figure out “the real world” on our own. (I’m not bitter, just wish I’d figured this out 20 years earlier).

Pushing Tests Down the Pyramid

In an ideal world these fully-integrated end-to-end tests of entire features from the user’s perspective would run instantly and reliably on the developer’s machine the moment the code was written to disk, and we could have vast numbers of them at no extra cost of effort or speed. Here in the real world as we know from bitter experience, the most valuable tests are also the most troublesome. They are the slowest, most fragile, most prone to flap, they have the most difficult dependencies to unpick, and are the most subject to the combinatorial explosion in numbers as soon as there’s a few possible code paths to take or inputs to handle.

Therefore as a matter of pragmatism we are forced to push some of our testing down the testing pyramid towards component, integration and unit tests.

If we do not lose sight of the highest goal of software testing, then this practically turns out to be be fine, and we can continuously tune the balance between the layers.

If however we do lose sight, and retreat into the lower levels of testing because it’s hard or slow to create the necessary full system coverage, then we start to slide up the technical debt scale, and will pay the price sooner than we’d like.

On Business Driven Development (BDD)

This is one that went off the rails as a concept. I see so many teams entirely miss the point of this movement, which is conceptually a good thing, but: IT’S NOT ABOUT GHERKIN SYNTAX, AND IT’S NOT ABOUT BROWSER AUTOMATION. Teams constantly get the technology confused with the intent and cargo-cult their way to an unholy mess of low quality high volume waste.

It’s actually about writing down what the users expect of your software, in terms they’d understand, and making sure that those expectations are never broken unintentionally, which is actually a very good thing.

BDD ends up using gherkin and browser automation because gherkin allows plain English explanations that can be turned into executable tests, and users often interact with software via a browser these days.

Gherkin (specflow etc) and browser automation test frameworks are tools for achieving BDD, not the definition of BDD.

Sadly by its nature the gherkin tools end up requiring maintenance of “step definitions” which is a hard cost to bear unless you are very careful what you use it for.

Outside-In Testing

Outside-In testing is a term and concept that I came across recently that really aligns with what I have laid out here. It emphasises what the user experiences (even if it’s an API user rather than a web user).

I think once the concepts in this blog post are internalised, then outside-in is a good shorthand for a good approach to achieving the regression testing goals that have been laid out in this post.

These are a couple of good posts on the concept:

Mindset and Culture

In the end, whether the right sort of automated regression tests are written comes down to the core beliefs of your individual engineers writing the systems and the teams they operate in.

If there is any part of them then that isn’t 110% on-board with what I’ve written here, then test coverage ends up being haphazard and incomplete, and over time gets steadily less able to prevent regressions.

If there is not a full belief in the need for high quality automated testing then even a small friction in the way (e.g. a difficult 3rd party dependency, or the need for complex multi-team multi-service regression test), then teams will generally quietly give up on full feature coverage from the user’s perspective and satisfy themselves with unit or component level tests. Unhelpfully, to the untrained eye a large quantity of tests looks similar to “good coverage”, and its less obvious whether it actually tests anything users care about. We must be constantly vigilant for this as the message on this is still relatively weak in the industry, and often confused by the (valuable) talk of detailed test approaches.

Hero culture

If you are particularly unlucky, or have set up perverse incentives, you risk a harder problem to resolve which is the embedding of a self-perpetuating “hero culture” whereby engineers take the fastest path to delivering percieved value (whilst incurring significant technical debt and unseen bugs), and then take credit again by rushing to highly visibly fixing the problems that they themselves created.

Good regression coverage, like much good engineering, takes time to build in the short term for a payoff of peace, reliability and sustained velocity in the long term. Which is incompatible with hero-ing it.

Beware the “rock star” programmer.

Programmer Excuses

“I don’t have time”

The client/boss is paying for your time, and the boss/client would like the long term benefits, mkay?

“The boss/client/manager won’t let me”

The “should I do a good job” question, as I like to call it.

“Which is quicker? Do that one.”

~ Your client / boss

I often see all but the most experienced software engineers falling into the trap of confusing authority with expertise.

I often hear engineers complaining that “they aren’t given the time” to write tests, or do a proper job on some aspect of the software they are writing.

In reality what has always happened is the engineer has presented two options to a client/employer who doesn’t know or care about how software is built:

I can do this feature properly with all the tests and great architecture etc etc (client glazes over) it’ll be amazing, or
I can just do the feature without any of that proper stuff. Which one do you want?

The client / boss / project manager responds [whilst thinking “I have no idea what you are talking about, or why you are asking] “Which is quicker? Do that one.”

Then the engineer gets the hump that they can’t do a good job.

Or worse, the engineer assumed the answer for them without even asking.

Put it this way, if a plumber came to a house, should they ask the homeowner whether to earth the pipes, or should they just do it as part of the cost of the job. The homeowner might not understand why that’s part of the job, but they probably also don’t want to be electrocuted by the radiator.

It is perfectly reasonable for an experienced software engineer, who is the expert in their trade, to just include automated regression tests as part of the job of feature delivery. To not even bring it up in conversation. The client/employer NEVER wants to pay for a feature, think its done, only to have it break again four features later. From their point of view, and quite rightly, that’s just shoddy software.

This is not to say I’m secretive - I work in the open and am open to people asking why things are done the way they they are, and am happy to explain to the client why it’s in their best interest. But whether it happens is not up for discussion.

Hiring a QA team - Just Don’t

Some organisations seem to think the answer is to take the regression test problem off developer’s plates by hiring less skilled individuals to do this apparently mundane work. Some even hire SDETs to write automated tests for the developers.

This is a fundamentally flawed approach in the same way that hiring an Operations teams as a separate function was accepted as a bad idea and replaced with an integrated DevOps product team. (Well, apart from the anti-pattern of a “DevOps” role, but that’s another post).

It’s flawed in at least the following ways:

It embeds the “manual testing is okay” approach in the culture, but adds some extra brute force people-power in the hope that this will prevent the eventual arrival of the 99th-feature problem. (The number just gets bigger, it’s still there). This is pretty much like trying to outrun your own shadow.
It encourages developers to consider testing (automated or otherwise) “that other team/role’s problem”.
It adds significant delays and an additional silo between writing code and delivering value to users and getting valuable feedback. Lengthening feedback cycles. This goes against everything we have learned from toyota/kanban/lean etc.
QA people cannot actually improve “quality” - they are not the developers working on the actual code, and can at best catch the worst errors.
The people who could improve quality (developers who actually write the code) lose extremely important feedback signals on how their software behaves when tested and used.

I have also seen QAs used as a “minimum quality gate” that “allows” an organisation to hire an army of poor-to-mediocre programmers. Fairly quickly that goes spectacularly badly.

Summary

So the important take-away here is:

Tests must test what your users actually care about.
Those tests must be automated to sustain your velocity.
The testing pyramid is good and valid.
Start from the outside with your testing and work in only as much as you must.

And why we do it that way is:

The features for users is more important than the internal workings.
Without full-automated coverage we cannot continue to confidently add features.

Templated repos with dotnet new

Tim Abell — Wed, 06 Mar 2024 00:00:00 +0000

I’ve been digging in to making dotnet new templates and it turns out to be a remarkably capable bit of tooling.

It’s particularly useful when you want to build a load of similar microservices with their own git repos.

It can:

Rename files.
Rename strings (variables, class names etc).
Preserve case style of renamed strings (through “derived” replacements).
Create named command line arguments for your template string replacements, e.g. donet new mytemplate --myswitch MyReplacementValue.
Exclude entire blocks of code and files based on switches that user can provide.
Be installed an tested from a local folder with dotnet new install <path>.
Be built into a nuget package and published on private or public feeds.

Importantly, there’s nothing in the way that it works that stops you from making your template code build/test/run.

And because it’s a CLI tool, if you want to an update an existing generated repo with a new version of a template you can just run it again with --force, and use git to pick through what changes you want to take into the generated repo.

This really takes a lot of the toil out of the copy-paste-modify you would have to do otherwise.

Learn more:

New tool: sln-items-sync for Visual Studio solution folders

Tim Abell — Sat, 13 Jan 2024 00:00:00 +0000

How and why I created sln-items-sync - a dotnet tool to generate SolutionItems from filesystem folders.

If you want to skip the backstory head over: https://github.com/timabell/sln-items-sync

15 years of minor irritation

Faced with another set of microservice repos written in dotnet-core, with .sln files in various states of tidiness I found my self for the 1000th time in 15+ years manually pointy clicky adding fake solution-items folders and subfolders and then toiling away adding files just so I could search them, click them and view them from within Visual Studio or Rider.

There must be a better way by now I thought, so I went hunting.

All I turned up was a lot of people asking the same thing and some dead tooling from years ago. Here’s the stackoverflow from 2008 with 90k views and 180 upvotes: https://stackoverflow.com/questions/267200/visual-studio-solutions-folder-as-real-folders, which didn’t really help in spite of having 23 answers. Not to mention the slew of linked questions where people are asking the same thing with different words.

(Solution folders aren’t to be confused with adding files to a project which used to be an equal nightmare before Microsoft saw sense and just included what’s on the filesystem. There are many old stackoverflow questions on that too from frustrated devs around the world.)

So with the programmer war cry of “how hard can it be, I’ll knock this out in a couple of evenings…” I set about on what turned out to be a significant exercise in yak-shaving in order to sort it out myself once and for all.

“How hard can it be, I’ll knock this out in a couple of evenings…”

~ Me. Again.

What to build?

I did briefly look at writing an IntelliJ (aka Rider) plugin but that turned out quickly to be a daunting thing so I put that idea down sharpish.

I use Rider in preference to Visual Studio and VSCode for C# so didn’t even look at that side. VSCode didn’t even bother with .sln files last I checked.

Next step was to write a CLI (command-line interface, aka terminal) tool to do it. (sln + filesystem in, mutated sln out, easy…)

I have recently written command line tools in both GoLang and Rust, but given this is a tool that would only be useful to Microsoft developers I figured I’d do this one in C#. I do actually like C# as a language for all my interest in other things, and thanks to dotnet-core and Rider I can actually write the whole thing on Linux Mint Cinnamon where I like to be.

Parsing and writing .sln files

I then hunted around for any nuget packages that might do the grunt work of reading/writing the sln format. Surely after 20-something years there must be something, right? Well, kinda. The VS parsing code is locked away in some windows dll nastyness, probably in C++ and COM or something evil. It even predates XML as a format, never mind JSON.

What I did find was the SlnParser nuget package, which someone had kindly written and open-sourced, and after quick test I could see it did a decent job of turning .sln files into an in-memory C# object model (a Solution class, with lists of things as properties).

So major yak number one was to fork SlnParser and turn it into a two-way tool. This I did with a lot of hackery and created SlnEditor nuget package which I published on nuget and github with the same Unlicense licensing as the original. Perhaps others will find this gift to the world useful in its own right.

Creating sln-items-sync

Finally with that working I was able to create the CLI tool I wanted, which I named sln-items-sync. This was more work than I expected, but I got a first cut working reasonably quickly.

Tests

I put a good amount of effort into good end to end test coverage on both the parser and the tool itself because I am now a true believer that without tests you will be unable to make future changes and dependency upgrades with speed and confidence.

I.e. lack of tests is the epitome of technical debt.

In fact let me give that a block quote because it’s such an important point:

Lack of tests is the epitome of technical debt.

(p.s. why is Epitome spelled that way but pronounced epitomy. English. Sigh.)

This has paid off in spades as the amount of work to get it satisfactorily “done” grew and grew the closer I got to finished.

The tests in both projects focus on “outside-in” testing rather than mockist unit testing. As such you can see at a glance the overall behaviour, spot any unexpected/unwanted output, and easily write new tests for new desired behaviour, being able to eyeball them easily for correctness. I won’t include one here as they are a bit lengthy, but you can go and look at the source repos on github.

This is made a bit easier on this tool because the only interfaces to the world are:

a text format (easy to string-compare expected versus actual)
a filesystem (I went for creating real file trees in tests which worked well and gives even more confidence)
the command line interface (for the sync tool)
the API (for the parser lib)

First contact with real .sln files

As I am doing some work for a C# contracting client currently I was able to try it out on gnarly real solution files, with a view to submitting some small cleanup pull requests that could created really quickly.

Stable ordering

The first attempt was a complete failure because the generated patch re-wrote the entire sln file in a completely different order, resulting in a sea of red/green lines full of GUIDs and other cryptic changes in the git diff. While the solution items were updated as intended and could be seen in Rider etc., this was not a patch that could be submitted to the team, or that I would put my name to.

Getting stable ordering between parsing and writing turned out to be a huge amount of work and refactoring, largely in the SlnEditor lib.

The key to making stable-ordering work was to add an int sourceLine property to almost everything when parsing, and to sort by that before rendering back out again. This had the desired effect of keeping everything in the original order no matter how it was mutated, and new items are added to the end (by replacing default 0 with Int.MaxValue before sorting).

Phew, another yak shaved, lost count now, but got more xmas hols so keep going….!

Many bugs and gaps

It surprised me a bit just how many little niggles, edge cases, and small omissions there were that had to be sorted out before I could use it to submit quality patches to client .sln files for real. Even the ever present byte-order-marker (BOM) was causing unwanted diffs because I hadn’t included it in the render, but .sln files seem to have them.

Pleasingly I’ve resolved everything I came across, apart from making the parent/child guid mapping order stable which didn’t seem to be worth the effort seeing as they are completely incomprehensible anyway.

Making a dotnet-tool

Once it was working, it was only a couple of rounds of building and copying the exe to bin/ before I got fed up with that approach to distribution.

Amazingly it turns out to be pretty simple to build and publish tools to the dotnet tool ecosystem, they are actually just slightly special nuget packages, and you only have to add a couple of properties to the .csproj file.

Making a dotnet-tool worked great, and is a great user experience for installing and running the tool. It even does updates for no extra effort!

Github-actions

To make both of these tools even easier to work on and maintain longer term, I wanted to have a good github action (aka CI) to build and run the tests.

Build and test is trivial, you can pretty much click the default workflow button for .net in an empty github actions page and it just works.

I wanted to also automate the nuget publishing of both from github-actions, as although I had a sh file to upload them from my machine that’s a faff and tends to stop working after a few machine rebuilds. Amazingly the author of SlnParser has taken an interest and provided a PR that gave me a ready-made github-action to push to nuget for every release tag! So that’s now in place, and to release a new version I can just git tag v1.2.3 && git push --tags and github does the rest.

The end, I need a lie down

So after all that, I’m not sure it was all worth it, but it’s done and I’m justifying it as a holiday hobby project and a gift to the dotnet developers of the world. I will certainly enjoy it every time I find an out of sync SolutionItems folder in future and run my tool so that I can ship a patch for it in seconds flat. I also learned a few things and got kata-like practice on shipping quality things at speed.

So with that, Merry Xmas and a happy new 2024. May all your solution folders be tidy and complete.

git - what do ‘base’ ‘local’ ‘remote’ mean?

Tim Abell — Fri, 20 Oct 2023 00:00:00 +0000

The terminology for 3-way git merge, rebase and cherry-pick conflict files is very confusing, particularly because they flip direction between rebase and merge.

When you run git mergetool it will spit out 4 files that look like this, and then pass them as arguments to your merge tool of choice:

$ gs 
## HEAD (no branch)
UU src/gitopolis.rs
?? src/gitopolis_BACKUP_1585963.rs
?? src/gitopolis_BASE_1585963.rs
?? src/gitopolis_LOCAL_1585963.rs
?? src/gitopolis_REMOTE_1585963.rs

In kdiff3 (by far the best 3-way merge algorithm out there) it looks like this:

Here’s how I think of it

Merge

You are on the target branch (local), and the patches are coming from the branch you are merging in (remote), kinda like this:

git checkout local-branch
git merge remote-branch

Cherry-pick

Same direction as merge.

You are on the target branch (local), and the patch is coming from the commit you are cherry-picking (remote), kinda like this:

git checkout local-branch
git cherry-pick some-remote-commit-ref

Rebase

Opposite direction to merge.

You start on your own branch that you want to rebase, but…

When you start the rebase you end up temporarily while the rebase is running in “detached HEAD” on the branch you are rebasing onto (often origin/main), so: You are on the target branch (local), and commits to rebase are coming from the branch you are rebasing (remote), kinda like this:

git checkout your-remote-branch
git rebase target-local-branch

Terminology

“ BASE ”: before anyone changed it (in all cases)
When merging (other branch coming to me):
- LOCAL : branch I’m on
- REMOTE : branch I’m merging in
When cherry-picking (other commit coming to me):
- LOCAL : branch I’m on
- REMOTE : commit I’m merging in
When rebasing (my own branch coming to me):
- LOCAL : branch I’m rebasing on to (checked out as detached head mid-rebase)
- REMOTE : my commits on branch I’m rebasing

Refs

https://stackoverflow.com/questions/20381677/in-a-git-merge-conflict-what-are-the-backup-base-local-and-remote-files-that

DEV Community: Tim Abell

📹 Video: How I implement David Allen's GTD® system in Obsidian

[Boost]

Existential Dread and the End of Programming

David Whitney ・ Feb 18

Automatic release notes with semantic commits and git-cliff

Backstory

A real example

[1.13.0] - 2025-11-28

Features

Bug Fixes

Setting up & configuring

Open source library projects

Commercial projects with Jira

cliff config example

Crafting commit messages

Mainline & pull requests

Not every commit is worthy of inclusion

Who are you writing release notes for

Ticket URLs

Get started then iterate

Over to you

Further reading / references

Addendum — em-dashes

Using Gitopolis to Manage Multiple Git Repositories

Day One: Setup

Day Two: Payoff

Day Three: A New Starter

Day Four: Evolving

Should you create tickets for tech tasks?

In short

Who cares?

Turning to GPT for explanations

Tech-lead-GPT Says…

Default friendly GPT explains fully

Why user stories should be shippable increments

The problem with non-shippable tickets

Incomplete Features Impact User Experience

Difficult to Test and Validate

Complicates Future Development

Increases Technical Debt

Blocks Related Development

Reduces Team Efficiency

Unclear Ownership and Accountability

Inconsistent System Behavior

Hinders Deployment and Releases

Wasted Effort and Resources

Unmet Stakeholder Expectations

Difficult to Manage in Agile Processes

In short

How to create small shippable batches

Use Vertical Slicing

Define Clear Acceptance Criteria

Focus on the minimum viable product

Implement Feature Flags

Break Down Complex Features

Avoid Over-Engineering

Make Use of Stubs and Mocks

Communicate Dependencies Clearly

Collaborate in Cross-Functional Teams

Cross-functional brainstorming of story creation

Summary

Running a beefy virtualbox dev server

Why

GNU’s Not Unix and neither is Windows

Basic setup

Remote access - linux

Remote access windows vm

1. Host connection

2. Remote access to virtualbox

3. Remote access to windows vm

Remote management tips

Was it worth it?

The end

Trunk based (mainline) development is (mostly) wrong

Mainlining

Does pairing remove the need for PRs?

Why branches and merge-commits matter

When to mainline, branch or PR

Split up commits before you write more code