DEV Community: Tim Abell

[Boost]

Tim Abell — Fri, 13 Mar 2026 22:29:56 +0000

Existential Dread and the End of Programming

David Whitney ・ Feb 18

Should you create tickets for tech tasks?

Tim Abell — Mon, 07 Oct 2024 00:00:00 +0000

In the manner of choosing the colour to paint the bikeshed, the decision of whether to create a ticket for every single tiny commit, no matter whether it’s a giant feature or the tiniest whitespace cleanup in the readme file continues to consume countless hours. This question is not really that important when it comes to delivering software, yet it still comes up from time to time and there is a right and wrong answer, which should be modified only with consideration for the current situation.

In short

Correct: create tickets only for features and larger technical work. Allow engineers to commit smaller improvements ad-hoc.
Incorrect: create an atmosphere of stifling bureaucracy where ll attempts to keep the work area tidy will require sign-off from the CFO and approval from the product owner who has never written a line of code (some have, but that’s a rare wonder).

Who cares?

If anyone wonders why engineers get so bent out of shape about this seemingly harmless request, imagine if someone asked you to fill out a form and submit it for every single piece of your work that you currently don’t have to. Want to pick up the phone?… create a ticket. Want to send an email?…

create a ticket. Can you imagine that this might make you a) miserable, b) less productive, and c) give up trying to do more than the bare minimum. By requesting tickets for everything you are introducing friction directly into the path of a programmer’s most important output - the codebase. We instinctively know that fast iteration and short cycle times lead to better delivery of value to users, and instinctively dislike anything that gets in the way. Note that the “auditing” and “why it changed” are already covered by source control (git) history and “pull requests” so adding tickets is redundant toil.

Turning to GPT for explanations

And now, given that there can be reasons for adding a ticket link to everything (for example generating change logs from git), and in the modern tradition, we turn to ChatGPT for a properly argued discussion of both sides of the argument.

The following is an edited result of my discussion with ChatGPT on the subject. Should you find yourself in the situation where someone has decided that every brush stroke during the painting of the bikeshed should require a JIRA ticket then this article will provide a handy reference to all the arguments that can be made around this oh-so-trivial subject.

I particularly like the fact you can ask GPT to write in the styles of different personas, so first we have the curt “lead dev” version that gets to the point and is devoid of superfluous wording, empathy, and any other affordances to anyone who thinks in any other way than the tech lead; and then we have the more wordy default style of GPT that explains the points a bit more fully for the benefit of those who need a bit more explanation around the subject. Both are useful depending on the audience, so take your pick.

Tech-lead-GPT Says…

Following the usual wordy explanation I fed GPT this follow-up prompt:

“write it in the style of a busy lead dev engineer - still too fancy and formatted. plain ascii, missing capitalisation in places, nothing more than the bare facts”

And got this straight-to-the-point explanation, lazy capitalization and all:

requiring a ticket for every tech improvement pr has upsides but will slow us down in key areas. here’s why:

pros:

prioritization: a ticket makes sure we focus on impactful work. it forces us to ask if this improvement is really needed now or if it can wait.
consistency: everyone follows the same process. good for cross-team work or when other stakeholders need visibility.
accountability: tickets clarify why a change is being made, what the risks are, and who owns it.

cons:

overhead: creating tickets for minor improvements (e.g. small refactors or cleanup) adds unnecessary steps. this slows devs down and eats up time for actual coding.
discourages quick fixes: devs spot opportunities to fix things as they go. forcing them to stop and make a ticket breaks their flow and makes it less likely they’ll bother with small but important changes.
ticket bloat: if every tiny improvement needs a ticket, we’ll end up with a cluttered backlog full of minor stuff, making it harder to see what’s actually important.
developer autonomy: devs should be trusted to make small improvements without red tape. too many rules can kill morale and make them feel micromanaged.

suggestion:

use tickets for significant changes (major refactors, cross-team impact, etc.), not small things like code cleanup or minor improvements.
use pr descriptions to document and discuss small technical changes. keeps things lightweight without losing track of what’s been done.
let devs decide when a ticket is needed vs when a quick pr will do. they’ll know best.

bottom line: tickets are useful for big changes but requiring them for every single tech improvement will slow us down more than it helps. let’s keep the process flexible where it makes sense.

Default friendly GPT explains fully

This part is in GPT’s default style, with some editing of what points to include. It’s pretty much as it was generated, and I think does a good job of explaining the trade-offs for discussion:

Requiring a ticket for every technical improvement pull request is a policy with both advantages and drawbacks, and it’s important to weigh these carefully to ensure the team’s workflow is as efficient and effective as possible. Pros of Requiring Tickets for Technical Improvements

Improved Prioritization and Planning Tickets create a formal process for prioritizing work. When technical improvements require a ticket, it forces a discussion on whether the improvement is truly necessary or how urgent it is. This helps teams focus on the most impactful changes and avoid unnecessary or lower-priority adjustments, allowing better alignment with the overall project goals.
Consistency Across Teams In larger teams or organizations, requiring tickets for all work (including technical improvements) helps ensure consistency. Everyone follows the same process, which makes it easier to manage cross-team dependencies, ensure the necessary approvals, and communicate changes to stakeholders.
Clearer Communication and Accountability Tickets provide a clear space for discussing the reasoning behind technical improvements, including potential risks, trade-offs, or alternatives. This promotes better communication among developers, product managers, and other stakeholders, and helps ensure that everyone is on the same page. It also adds accountability, as each improvement is documented and has a clear owner.

Cons of Requiring Tickets for Technical Improvements

Increased Administrative Overhead One of the biggest downsides is the added overhead. Many technical improvements are small, straightforward fixes or optimizations that don’t warrant the time and effort needed to create, manage, and track a ticket. Forcing developers to do this extra work can slow down progress and take time away from actual coding. Over time, this could lead to frustration or a slowdown in the pace of incremental improvements.
Discourages Quick, Organic Improvements Developers often spot opportunities for small technical improvements while they’re working on other tasks. Having to create a ticket for each of these can disrupt their workflow, making them less likely to act on these improvements when they notice them. This can lead to missed opportunities to clean up code, reduce technical debt, or optimize performance in small but meaningful ways.
Potential for Ticket Bloat In a fast-moving development environment, the requirement for tickets on small improvements could create a flood of minor issues in the ticketing system. This can make it harder to manage more critical work and obscure visibility into the progress of higher-priority features or bugs. The sheer volume of tickets can become difficult to sort through, adding cognitive load to project management.
Less Developer Autonomy Developers often thrive in environments where they can take initiative to improve the codebase. Requiring tickets for every small change might reduce their sense of ownership, as it introduces a layer of bureaucracy between identifying an issue and resolving it. This could dampen their motivation to take care of small, yet important, aspects of code quality.

A Balanced Approach

Given the pros and cons, there may be a middle ground that allows the team to capture the benefits of ticketing while avoiding its downsides:

Ticketing for Significant Changes: Tickets could be required for larger technical improvements that have broader implications, such as major refactors, changes that affect multiple teams, or improvements that touch critical parts of the system. For smaller changes, like simple code cleanup, minor refactoring, or bug fixes, developers could be allowed to open a pull request without the overhead of a ticket.
Use Pull Requests for Tracking: The pull request itself can serve as a lightweight way to document and discuss technical improvements. Detailed PR descriptions, along with the review process, can provide enough visibility and communication for smaller improvements without the need for a formal ticket.
Developer Discretion: Allow developers to decide whether a ticket is needed based on the scope of the improvement. They are often best positioned to know whether a change is minor enough to be handled via a simple PR or whether it requires more discussion and prioritization through a ticket.

Conclusion

Requiring tickets for every technical improvement has clear advantages in terms of prioritization, consistency, and accountability, particularly in larger teams or complex projects. However, the added overhead, potential for slowing down minor improvements, and impact on developer autonomy are significant downsides to consider. A balanced approach that applies ticketing requirements more selectively, while still ensuring visibility and discussion through PRs, could be a better fit for fostering both productivity and quality in the development process.

Why user stories should be shippable increments

Tim Abell — Thu, 12 Sep 2024 00:00:00 +0000

Sometimes I see people create work items / jira tickets / user- stories that if they were implemented and merged in to the codebase would leave the system in a half-finished state, either broken or inconsistent.

The problem with non-shippable tickets

This to me is instinctively a “bad thing”, however as not everyone has grokked this important fundamental of team software delivery, I turned to chat gpt for an explanation of why I think this is important, and I have to say it did a stellar job of clearly explaining the things that I have absorbed over my decades of commercial software development. So here you go, here’s why non-shippable tickets are a problem:

Incomplete Features Impact User Experience

A half-finished feature can confuse users or negatively impact their experience. If the feature is exposed to users before it’s fully functional, it may appear broken, resulting in dissatisfaction or mistrust in the system’s reliability.

Difficult to Test and Validate

Testing half-finished work is challenging, as it’s not in a stable or coherent state. Quality Assurance (QA) teams need a fully functional piece of work to validate it against the requirements. When a ticket is only partially completed, it’s hard to determine if it meets the intended functionality or introduces bugs.

Complicates Future Development

Leaving an unfinished task can create dependencies and conflicts with future work. If another developer later works on a related feature, they might have to deal with the incomplete state, which could introduce confusion or lead to mistakes in integrating related functionality.

Increases Technical Debt

Incomplete features often accumulate as technical debt, which means extra work will be needed later to finish, refactor, or fix them. This can slow down future progress, as more time will need to be spent revisiting and cleaning up the half-finished work.

Blocks Related Development

A half-finished ticket can block progress on other features that depend on it. For example, if a partially completed feature introduces bugs or changes a shared system in an untested way, it could delay the development of other tickets or cause cascading issues across the system.

Reduces Team Efficiency

When a ticket is left in a half-finished state, developers may need to pick it up again later or pass it to another developer, leading to context-switching. This results in inefficiency, as time is wasted on re-clarifying the problem, re-understanding the code, and figuring out where the previous developer left off.

Unclear Ownership and Accountability

If a ticket is left unfinished, ownership and accountability become unclear. Other team members may not know who is responsible for completing it or what still needs to be done, which can cause confusion and reduce productivity.

Inconsistent System Behavior

Partial implementation can cause the system to behave inconsistently. For example, a feature that’s half-completed might trigger errors, create discrepancies in data, or cause parts of the system to function incorrectly or unpredictably.

Hinders Deployment and Releases

Half-finished tickets often disrupt deployment cycles, as incomplete features cannot be included in production. This results in unnecessary delays or the need to work around unfinished tickets, making releases more complex and error-prone.

Wasted Effort and Resources

When a ticket is not fully completed, there’s a risk that the work may be scrapped or need significant changes by the time it’s revisited. This can result in wasted development effort, resources, and time.

Unmet Stakeholder Expectations

Leaving a ticket incomplete prevents stakeholders from seeing tangible progress, undermining confidence in the team’s ability to deliver. Stakeholders expect finished, usable deliverables that they can review and assess, not half-completed tasks that require further work.

Difficult to Manage in Agile Processes

In Agile and Scrum methodologies, the goal is to deliver shippable increments of work at the end of each sprint. Leaving a ticket half-finished breaks this cycle, making it hard to close sprints successfully and creating spillover tasks that carry into future iterations.

In short

Avoiding half-finished tickets ensures that the system remains stable, progress is clear, and each ticket represents a functional, complete deliverable. This promotes better workflow, higher quality, and more reliable software.

How to create small shippable batches

That’s all very well you say, but I don’t want to end up with giant, big-bang stories where engineering vanish for a month or six until every last thing works (if it ever does), so what to do?

Again I’ve turned to gpt as a starting point, and it’s not done a bad job, I’ve done a bit more editing to this set and added some of my own thoughts:

Use Vertical Slicing

Vertical slicing means breaking down features into small, independent pieces that cut through all layers of the stack (UI, backend, database, etc.) and provide end-to-end functionality. Each slice delivers a fully functional piece of the feature, even if it’s a smaller or simpler version of the final product. Example: Instead of building a complete “user profile” page, deliver a ticket for “adding a profile picture” with all the components needed (UI, API, storage).

Define Clear Acceptance Criteria

Each ticket should have clear, well-defined acceptance criteria that detail exactly what constitutes a “done” state. These criteria ensure that the ticket can be completed independently and is deliverable as a unit.

Focus on the minimum viable product

Like the Minimal Viable Product (MVP) concept, you can apply the same thinking to individual product increments.

When designing features, focus on delivering the smallest possible version that is still valuable to users. This forces the team to identify the core functionality that is immediately useful, allowing for iterative improvement later.

Example: Instead of delivering a fully-featured analytics dashboard, start with a basic version that shows just the key metrics, and expand later.

Implement Feature Flags

Use feature flags to allow partially completed features to be merged into the main codebase without affecting the user experience. This ensures that work can be released incrementally but hidden or disabled until it is complete.

Example: A half-built “comments section” can be developed over several tickets but hidden from users until all parts are ready.

Break Down Complex Features

Complex features should be split into smaller sub-features that can each be delivered independently. Ensure that each sub-feature is fully functional, even if it doesn’t have all the final functionality.

Avoid Over-Engineering

Start with the simplest solution to a problem, and iterate.

Avoid adding extra complexity upfront by sticking to what’s necessary to fulfill the ticket’s requirements for delivery.

This includes “gold plating” and “future proofing” when implementing the ticket. Just do the minimum high-quality code that you know you need now.

Make Use of Stubs and Mocks

To avoid leaving a ticket half-finished when other dependencies aren’t ready, use stubs or mocks for parts of the system that are still under development. This allows you to complete and deliver your ticket while waiting for those dependencies.

Example: If the backend API isn’t ready, a front-end ticket can use mock data to simulate responses, allowing the UI to be built and delivered without waiting.

This is more appropriate for very early product development rather than for modifying an established system.

Communicate Dependencies Clearly

If a feature is dependent on other parts of the system, make sure the dependencies are identified and managed.

Sometimes it makes sense to coordinate related tickets so they’re developed together or in the right sequence.

Example: If a new API endpoint is needed for a UI feature, ensure the API ticket is prioritized so the front-end team isn’t blocked, or vice versa.

Collaborate in Cross-Functional Teams

Ensure cross-functional teams collaborate on the entire stack of a feature (UI, backend, database, etc.) within the same sprint or ticket. This ensures no part of the feature is half-done or stuck waiting for another team to finish their part.

Example: A ticket to create a new form should include front-end developers, back-end developers, and testers working together to ensure it’s functional from end to end.

Cross-functional brainstorming of story creation

There’s nothing like all members of a team, including product, engineering, UX etc all sitting round a table and throwing around ideas to see if they can come up with a creative way to break down a problem into smaller but still shippable pieces.

I’ve repeated this exercise with many clients and the flashes of inspiration that come out of it cannot be replicated by thinking harder on one’s own while starting at Jira.

Summary

By structuring work this way, teams can maintain a balance between small, manageable batches of work and ensuring that each piece is fully deliverable and functional on its own.

This approach maximizes delivery velocity while maintaining system integrity and delivering value incrementally.

Running a beefy virtualbox dev server

Tim Abell — Fri, 07 Jun 2024 00:00:00 +0000

Why

Sometimes I have to work on hairy old sprawling legacy code bases in ye-olde c#, which means, unfortunately, no shiny new dotnet-core with Rider on linux and all the speed gains that brings.

So having fallen over another codebase that takes unbearably long to build and work on when combined with the need to run visual studio in a windows virtualbox vm on top of my linux box I’ve shelled out for a desktop pc with a top-end cpu to use as a dev box for those trickier builds. But who wants to give up the joy of a portable laptop, so here I am with the challenge of connecting to virtualbox from another machine. Turns out there’s a few non-obvious things and options that may or may not work for you, so here’s what worked for me.

GNU’s Not Unix and neither is Windows

Aside - windows will never again have ownership of my computer, I relegated it to a VM for emergencies (i.e. clients paying money), where it can stay within those steely confines unable to bother me the rest of the time. But that’s another blog post really.

Basic setup

This is pretty simple:

Install Linux Mint because home servers are a lot easier to deal with when they’re actually just normal desktop operating systems that are switched on more.
1. Enable full disk encryption during setup (advanced > lvm > encrypt) (this is a bit of a nuisance for remote management but important for security if the machine is stolen)
Turn on automatic updates
Install virtualbox and openssh-server
Copy the “Virtualbox VMs” folder across by opening sftp://ur-server-here/ in nemo (the file browser) on the source machine. It’s not the quickest but it did 200gb in 5 hours, and I have other things to do in my life so that’s fine.
Fire it up on the desktop with the virtualbox gui just to check it’s fine

Remote access - linux

To get some remote control of the machine simple ssh and ssh -X are enough for cli actions and some gui actions respectively (-X tunnels x-windows over ssh, meaning you can launch graphical programs on the remote machine and they pop up on the local display. Magic.)

Remote access windows vm

This proved to be less obvious.

There are three possible layers to remote in:

Remote access to the host machine UI
Remote access to the virtualbox process
Remote access to the windows vm

Only number 3, directly connecting to the guest windows vm with RDP worked for me.

1. Host connection

Remote desktop in linux is still a bit of a mess.

VNC is klunky old, and I don’t even know if it’s secure any more, it’s also not available out of the box. I didn’t even try.

There’s some more or less proprietary ones like nx (nomachine?), rustdesk, go-to-my-pc etc, which I either haven’t had any luck with previously or haven’t tried / don’t trust.

I tried running the virtualbox command gui over ssh, but it had two problems:

It was too laggy / slow updating the screen to be useable for intense visual studio coding work
Disconnecting killed the virtualbox process and terminated the vm without proper shutdown

I quickly gave up on this and tried #2…

2. Remote access to virtualbox

Tantalizingly virtualbox has RDP support available…. but the punchline is that it’s not part of the open source project, and is a proprietary “extension” that requires agreement to a personal/evaluation license that explicitly forbids commercial use, so that was out. Darn. If you’re only doing personal things you could try this, but I’m not so can’t.

3. Remote access to windows vm

This was the winner in the end but requires a few steps to get set up:

Change the virtualbox vm network settings to “bridged” instead of the default “NAT” so that the vm is available directly on the network.
Grab the generated ipv6 address for the machine (or give it a static ip4 lease)
In the windows guest:
1. Enable remote desktop
2. Give the user account a password if it doesn’t already have one (you can’t connect remotely with a passwordless user).
Start the vm with the virtualbox cli instead of the gui (avoids it being terminated when the shell session ends):
1. vboxmanage startvm WinDev2404Eval --type headless
On the client laptop install the remmina rdp client with apt
Connect to the vm in remmina with the ip6 address of the guest machine (not the host desktop)
Enter the windows user’s username & password when prompted

Remote management tips

Here’s some more useful commands for managing the server machine remotely over ssh:

List virtual machines (useful for getting the name):

vboxmanage list vms

Start a virtual machines

vboxmanage startvm WinDev2404Eval --type headless

Shut down the host

sudo shutdown -h now

(see https://explainshell.com/explain?cmd=shutdown+-h+now)

Suspend the host:

systemctl suspend

(ref https://askubuntu.com/questions/1792/how-can-i-suspend-hibernate-from-command-line/1795#1795

Was it worth it?

Build time of a gnarly project on laptop:

========== Rebuild completed at 02:44 and took 20:08.230 minutes ==========

And the same vm running the same full rebuild remoted into the new desktop:

========== Rebuild completed at 02:28 and took 04:15.142 minutes ==========

You can even copy paste text like this straight out of the RDP’d VM which will be immensely useful.

The end

That’s all for this one. An obscure technical thing, that was useful to me, and just tricky enough to be worth documenting for future-me and the rest of the interwebs in case anyone finds it useful. Hurrah for blogs and zero cost of replication.

Till next time. 👋

Trunk based (mainline) development is (mostly) wrong

Tim Abell — Thu, 18 Apr 2024 00:00:00 +0000

Mainlining

Some very experienced developers, some of whom I’ve heard it from in-person, strongly advocate what is often called “mainline” or “trunk-based” development, meaning that the git history is a series of commits directly to the main branch, with no pull requests or merge commits in sight. This is often held up as the panacea for achieving fast, high-quality delivery.

I. Do. Not. Agree. The absolute rule of “no merge commits on main” (aka straight-line history), sometimes enforced by github configuration, is absolute balderdash. This assertion is typical of the black-and-white presentations of topics that seem oh-so-appealing when presented in the conference talk circuit where there is no room for nuance and actual trade-offs made in the trenches, and where big extreme statements win the game of attention.

There is a place for thoughtful, individual mainline commits, so don’t take this as saying the opposite extreme of “only merge commits / PRs on main”. A small-to-medium internally consistent patch (say 1 to 200 lines of diff at most), can be quite a reasonable thing to just push to main.

Does pairing remove the need for PRs?

For anything more than trivial changes in your production code and test code you probably want two pairs of eyes on it, and pairing is a great way to achieve that in real-time. But does pairing remove the need for a PR entirely? Some say yes, but I say it depends on the actual patch …

The question of pairing and the question of “mainline vs pull request” are almost entirely orthogonal. Pairing fixes the async-review problem, and certainly reduces the need for PRs to achieve peer-review, but that doesn’t mean we should reflexively throw out feature branches, merge commits and PRs with the proverbial bath water.

Why branches and merge-commits matter

Unconditionally mainlining everything loses some important capabilities:

The ability to group a series of related commits (e.g. reformat, refactor, add feature), into one coherent group by way of merging them to main in one hit. (With or without github and its PRs.) A future reader can use this to look at history at two levels of detail - using git log --first-parent to look first at the merges and commits to main, and then only if it’s interesting look at the series of patches in a feature branch that was merged in. And conversely it allows breaking down a patch that needs to go into main in one hit (to avoid breaking things by shipping half-finished work) into a series of easier to understand patches with meaningful commit messages.
The ability to check your changes with CI in github before merging to main to be sure you don’t break main for everyone else (plus the permanent record of another “Fix main build, oopsie” commit).
The ability to offload regression checks to github actions instead of having to always run all the tests locally.
The ability to create a series of commits, discover they are wrong several hours later, and throw them out or rewrite them before they ever hit main.

If you think mainlining generates a clean history, how often have you made a series of commits before realising that the shape of code isn’t right and now you have another series of commits to get it into a different shape. With mainline development that completely irrelevant first attempt is now in history to confuse future readers for ever more. With branch based development you can throw it out and pretend it never happened.

When to mainline, branch or PR

It’s a bit of an art, so here’s some rules of thumb to guide you through the mainline / branch / PR decisions as you write code:

Pair on everything you can, eliminating the need for a PR for peer-review. Only peel off to solo work when it’s painfully obvious that there’s zero value / knowledge-transfer / alignment to be gained by pairing (e.g. library upgrades).
Trivial, uncontroversial changes can be mainlined solo (if your org allows any solo pushes to main), especially if you run the tests locally before pushing. Watch out for linters and formatters in CI. Pre-commit / pre-push checks can help you avoid breaking the build with silly mistakes.
Slightly bigger changes can be pushed directly to main if you are pairing, but they should still be restricted to smaller, simple to understand patches; not entire large features.
When working on a larger feature, try and peel off as many unrelated pieces as you can and mainline or PR them separately as you go instead of lumping them in to your feature patch/branch, rebasing your feature patch/branch onto the updated main.
If a piece of work requires a few logically coherent steps to make the change, that are related or build on each other, then group them together in a feature branch. If you are pairing then you can run your tests & lint locally, create a merge commit on main that merges your branch in, and push that to main. No PR needed as it’s already been reviewed.
You may opt to open a PR in any of the above circumstances anyway in order to:
1. Get a run of CI
2. Get some async input from the broader dev team or across team boundaries.
3. To put a novel idea/technique into public and give it time to be considered by yourself and others. The draft PR feature of github can be useful for this. Also useful for “spikes” (experiments in code to learn something or try something out that aren’t necessarily production quality or ready to merge).
Beware of long running branches with many commits, even if they are lots of well crafted commits. It’s a sign your deliverable increment is too large (at feature branch and/or story level), and you should look to take a step back and break it down into smaller increments; maybe even ditching the branch and starting again from main.

Note that you need to be constantly considering these rules of thumb as you make every change to the code because in the fluid development flow it’s normal to range across all sorts of types of changes in one coding session (e.g. cleanup, refactoring, bug fixes, feature changes, quality improvements, ci fixes, editor config changes, massive file/folder renames, and the actual features, etc etc), and each of these might require a separate approach to generating a high-quality history.

Split up commits before you write more code

Flipping to main and making a separate commit when you realise you want/need a change that is unrelated to your feature branch is a great habit/skill to build. It does however take discipline and practice.

I see many developers who default to branch-based development and just pile any old thing they need into their current branch, ending up with mammoth PRs that change far too much in one go. They often then complain that splitting it back up is too hard so they should just squash it. As the old saying goes “if I wanted to get to there I wouldn’t have started from here”; it’s far better to split into commits as you go rather than struggling with git’s tricky-but-powerful git rebase --interactive.

I’ve had devs say to me that “it’s not worth the time” to create a series of atomic coherent and well described commits. And for evidence they often point to their massive commit that does too many things, or their pile of incoherent “wip” commits to a branch, combined with the effort it takes to split and recombine commits into something better. In commercial projects I often relent because at that point the horse has bolted, and they are right that it isn’t worth spending that much client/employer time rescuing their history aside from the opportunity to demonstrate rebase-interactive to them. (Besides, it’s their name on that commit forever, not mine). That doesn’t however make it good enough, or the correct conclusion. The correct answer is that you should work on getting the series of patches right as you go, and the cost of interactive rebases largely goes away.

Feature flags DO NOT replace feature branches

Some claim that feature flags (the ability to toggle capabilities on and off at runtime) are the reason you no longer need branches.

To be sure, they are a useful thing, and an answer to long-lived feature branches that can’t be merged because the feature isn’t ready to ship yet.

But just because you know how to use feature flags, still doesn’t mean you should throw out the richness and tools available to you with merge commits to main and PRs. The fact that some teams have painted themselves into a corner of PR-hell doesn’t mean you should throw out PRs entirely. You can have feature flags and the option to use branches, merge commits and PRs when it is the right tool for the job.

“But no-one cares about the history”

Someone said this to me today. They aren’t the first person to say it to me either. This is empirically wrong. A friend of mine calls people with this mindset pejoratively “BSBITS devs - Big Save Button In The Sky Developers” meaning that they think that the only thing that matters is the current production code, and that source control is a fancy save/share system.

Here are some of the ways that I have seen first-hand many many developers including myself “care” about history:

Understanding why the current code is how it is in order to know how/whether to change it.
Understanding the changes another developer made in order to replicate those changes in a different microservice in a different team.
Reviewing a large PR with 1000+ lines of code changed, where some of it is refactoring and reformatting and some is functional changes - looking at the commits of the PR/branch allows you to see the intended functional change (sometimes just a few lines) separately from large rote refactors etc.
To judge the capabilities and productivity of a developer with a view to hiring/firing. Yes this actually happens, I’ve seen it. And guess what, that developer doesn’t get to explain their poor quality history, and they can be judged harshly for it. Are you really sure no-one cares?
To get a feel for the capabilities of a co-worker.
To keep up to date with changes to a project that you are either involved in directly or indirectly, perhaps a repo belonging to another team.
Principle developers that operate cross-team looking for best/worst practices that they need to take action on. (They often have significant HR influence, do you want them judging your poor history badly because “no-one cares”?

Ironically I’ve even seen some of the people who’ve said to me “no-one cares about history” then go on at a later date to look at the git history in order to figure something out while I’m on a screenshare with them. It’s all I can do not to point out the hypocrisy.

Build that habit

In the end, a lot of the arguments I see come from people who haven’t mastered the skill of creating high-quality history with commits, branches and thoughtful merges.

The answer in my eyes is not to declare half of the tooling we have off-limits, and to claim that the generated history is unimportant, it is instead to get really good at creating good history as you go so that it become effortless.

Treat generation of history just like generation of code. It’s the most visible and permanent record of what you do as a developer of software, and you should treat it with the same pride and diligence as you do the code that runs in production.

Fin

Coding is not just computers running things, it’s inter-person communications. Quality history, and the richness that branches and merges give us is part of that tapestry of communication with persons past, present and future.

Become good at it and the objections melt away.

Why do automated tests matter?

Tim Abell — Wed, 27 Mar 2024 00:00:00 +0000

It might seem a bit odd to write a post on software tests after so many years and so much content, yet to this day I see well meaning developers writing software without adequate test coverage. In fact I will share that I myself have been very late to enlightenment on this front. Sure I’ve been “writing tests” for well over a decade, but I was missing the mental framework that would make those efforts coherent, complete and effective.

In my view the mental model of why we would write tests is more important than the detailed “how” of writing unit/integration tests, because without that it can only be at best haphazard whether the testing really serves its true purpose.

I’ll share with you here the keystone ideas that make it all sit together properly, just like the keystone in a stone archway. Then from there, all the tests you write will sit in their true and proper place within that framework, or be obviously either waste or missing.

Yes I’m aware this particular archway in the photo doesn’t actually have a keystone, but I took the pic and I rather like it so it’s staying.

The Before Time: TDD & Unit Testing

I started my career in software development way back in 2000 when only a few very forward thinking people were doing high quality automated testing, and it really hadn’t reached the broader consciousness off the software development community.

Like a large number of software devs of that era, I became aware of “software testing” through the “TDD” movement (test-driven development). I believe a lot of the noise about this came out of the Ruby on Rails community (no coincidence, there’s no compile step so there’s more reliance on good tests in ruby/rails than in compiled languages). My experience of this was that it was largely talked about and taught as a bottom-up approach to testing:

“Here’s how you write a unit test for a class.”
“Here’s how you do assertions.”
“Here’s why you should write the test before the code (red-green-refactor).”
And in C# land “here’s how you do dependency injection”.

This is all good and important to know but is not enough in itself.

This was also a time when SOLID was big in the OO consciousness, so classes, isolation, encapsulation and all the small pieces of the machine you were building was the focus, and testing those pieces was often enough to claim you were “doing testing”. Interviews often being more about “do you TDD” than “will your system break for end users”.

“Integration Testing” (i.e testing more than one piece together) was certainly in my awareness, but mostly as a TDD++, and was usually missing any kind of coherent “why”. You might test some code with a real database instead of a mocked persistence layer, or check that all your pure code classes didn’t blow up when they were wired back together. In C# world a lot of mental overhead was created by trying to work out how to test code that used Microsoft’s not very test-friendly .NET Framework standard library code.

So I bumbled along, more or less successfully writing tests for my software, but always feeling like I didn’t have a really solid argument for the big picture. For me it was mostly “aim for 100% test coverage” and you’ll catch/prevent lots of bugs, plus it was good at driving software architecture in the minutia by preventing things being too coupled together.

More recently I’ve had the pleasure on working with people who are super-keen on “outside-in” testing and considerably less keen on acres of unit tests, and it finally dawned on me what I’ve been missing all these years. What follows is the missing piece:

The “Why” of Testing

So let’s take a step back for a moment. Why do we write tests at all, what is the big goal that makes this all worthwhile.

There’s long been discussions of the “cost/benefit” of tests, and there is research that shows teams that write tests are more productive, which is good, but a bit abstract when it comes to what we actually need to write to be effective.

Preventing Regressions

The goal of writing any software is working software for users and businesses; and they aren’t going to be too happy when something that worked fine on Monday is now broken on Tuesday.

I don’t think it’s news to anyone that the goal of software tests is to prevent regressions.

Axioms

There are two pieces of the testing puzzle that “preventing regressions” alone didn’t make immediately clear to me, and these are they key reasons I’m taking the time to write this post at all. They drive a subtle but foundational shift in what our writing of tests actually looks like and the magnitude of their effectiveness.

They may seem trivial and obvious at first sight, but do not ignore them so quickly as they are the axioms upon which the whole approach rests. They are:

What matters is not whether your class has “tests” but whether the software as perceived by the user does the job they need.
When you add your first feature it’s easy to manually verify it’s behaviour. When you write your ninety-ninth feature you still want to be certain that the other ninety-eight features are all still intact.

Most systems have many thousands of “features”, especially if you include “non-functional” cross-cutting requirements such as acceptable performance for each capability.

Full, Automated Coverage

Every time you touch your software, there is a non-zero risk that something that used to work will no longer work. Anyone who’s written software for any length of time will laugh at the idea that “a change in component A has no possibility of breaking something unrelated in component B”. And as a more rigorous colleague pointed out, we have problem of “emergence” in complex systems which makes it increasingly hard to predict behaviour. A good measure of your tests is how afraid you are of upgrading third-party dependencies, running you tests and immediately shipping the result without further manual verification.

The only sustainable solution to the presented axioms combined with the need to add more features over time is:

Ensure every feature the user cares about has a test.
The software is tested from the perspective of the user.
The software is tested after every change.
This testing is fully automated.

Insufficient Test Automation

If you write insufficient automated tests there are only two things that can happen, both bad:

You spend exponentially more time manually verifying all expected behaviour before any change is passed on for the user to use; bugs get through anyway. Or
You give up trying to test what was supposed to work, and some of it stops working.

Full automated test coverage of all delivered features to date is the only solution to this problem. Any shortcoming in this coverage is a subclass of the generalized “technical debt” problem that quickly results in a catastrophic drop-off of ability to delivery anything at all if it is allowed to grow:

What Kind of Tests

So, if you accept all of the above, what does that mean when you actually fire up your editor and wonder what test to write?

You already write unit tests, that’s enough, right? … Bzzzzt, Wrong! That fails the “from the user perspective” need.

If the only thing that matters is that a feature works from the users perspective then the only automated test that matters is one that tests a feature from the users perspective. In practice that means things like:

End to end tests
Browser automation
Smoke tests
Platform tests
Performance tests
Outside-in tests

And what doesn’t matter a hoot is things like unit tests, integration tests and component tests, i.e. the very things that we were taught in “TDD school”, and then left to figure out “the real world” on our own. (I’m not bitter, just wish I’d figured this out 20 years earlier).

Pushing Tests Down the Pyramid

In an ideal world these fully-integrated end-to-end tests of entire features from the user’s perspective would run instantly and reliably on the developer’s machine the moment the code was written to disk, and we could have vast numbers of them at no extra cost of effort or speed. Here in the real world as we know from bitter experience, the most valuable tests are also the most troublesome. They are the slowest, most fragile, most prone to flap, they have the most difficult dependencies to unpick, and are the most subject to the combinatorial explosion in numbers as soon as there’s a few possible code paths to take or inputs to handle.

Therefore as a matter of pragmatism we are forced to push some of our testing down the testing pyramid towards component, integration and unit tests.

If we do not lose sight of the highest goal of software testing, then this practically turns out to be be fine, and we can continuously tune the balance between the layers.

If however we do lose sight, and retreat into the lower levels of testing because it’s hard or slow to create the necessary full system coverage, then we start to slide up the technical debt scale, and will pay the price sooner than we’d like.

On Business Driven Development (BDD)

This is one that went off the rails as a concept. I see so many teams entirely miss the point of this movement, which is conceptually a good thing, but: IT’S NOT ABOUT GHERKIN SYNTAX, AND IT’S NOT ABOUT BROWSER AUTOMATION. Teams constantly get the technology confused with the intent and cargo-cult their way to an unholy mess of low quality high volume waste.

It’s actually about writing down what the users expect of your software, in terms they’d understand, and making sure that those expectations are never broken unintentionally, which is actually a very good thing.

BDD ends up using gherkin and browser automation because gherkin allows plain English explanations that can be turned into executable tests, and users often interact with software via a browser these days.

Gherkin (specflow etc) and browser automation test frameworks are tools for achieving BDD, not the definition of BDD.

Sadly by its nature the gherkin tools end up requiring maintenance of “step definitions” which is a hard cost to bear unless you are very careful what you use it for.

Outside-In Testing

Outside-In testing is a term and concept that I came across recently that really aligns with what I have laid out here. It emphasises what the user experiences (even if it’s an API user rather than a web user).

I think once the concepts in this blog post are internalised, then outside-in is a good shorthand for a good approach to achieving the regression testing goals that have been laid out in this post.

These are a couple of good posts on the concept:

Mindset and Culture

In the end, whether the right sort of automated regression tests are written comes down to the core beliefs of your individual engineers writing the systems and the teams they operate in.

If there is any part of them then that isn’t 110% on-board with what I’ve written here, then test coverage ends up being haphazard and incomplete, and over time gets steadily less able to prevent regressions.

If there is not a full belief in the need for high quality automated testing then even a small friction in the way (e.g. a difficult 3rd party dependency, or the need for complex multi-team multi-service regression test), then teams will generally quietly give up on full feature coverage from the user’s perspective and satisfy themselves with unit or component level tests. Unhelpfully, to the untrained eye a large quantity of tests looks similar to “good coverage”, and its less obvious whether it actually tests anything users care about. We must be constantly vigilant for this as the message on this is still relatively weak in the industry, and often confused by the (valuable) talk of detailed test approaches.

Hero culture

If you are particularly unlucky, or have set up perverse incentives, you risk a harder problem to resolve which is the embedding of a self-perpetuating “hero culture” whereby engineers take the fastest path to delivering percieved value (whilst incurring significant technical debt and unseen bugs), and then take credit again by rushing to highly visibly fixing the problems that they themselves created.

Good regression coverage, like much good engineering, takes time to build in the short term for a payoff of peace, reliability and sustained velocity in the long term. Which is incompatible with hero-ing it.

Beware the “rock star” programmer.

Programmer Excuses

“I don’t have time”

The client/boss is paying for your time, and the boss/client would like the long term benefits, mkay?

“The boss/client/manager won’t let me”

The “should I do a good job” question, as I like to call it.

“Which is quicker? Do that one.”

~ Your client / boss

I often see all but the most experienced software engineers falling into the trap of confusing authority with expertise.

I often hear engineers complaining that “they aren’t given the time” to write tests, or do a proper job on some aspect of the software they are writing.

In reality what has always happened is the engineer has presented two options to a client/employer who doesn’t know or care about how software is built:

I can do this feature properly with all the tests and great architecture etc etc (client glazes over) it’ll be amazing, or
I can just do the feature without any of that proper stuff. Which one do you want?

The client / boss / project manager responds [whilst thinking “I have no idea what you are talking about, or why you are asking] “Which is quicker? Do that one.”

Then the engineer gets the hump that they can’t do a good job.

Or worse, the engineer assumed the answer for them without even asking.

Put it this way, if a plumber came to a house, should they ask the homeowner whether to earth the pipes, or should they just do it as part of the cost of the job. The homeowner might not understand why that’s part of the job, but they probably also don’t want to be electrocuted by the radiator.

It is perfectly reasonable for an experienced software engineer, who is the expert in their trade, to just include automated regression tests as part of the job of feature delivery. To not even bring it up in conversation. The client/employer NEVER wants to pay for a feature, think its done, only to have it break again four features later. From their point of view, and quite rightly, that’s just shoddy software.

This is not to say I’m secretive - I work in the open and am open to people asking why things are done the way they they are, and am happy to explain to the client why it’s in their best interest. But whether it happens is not up for discussion.

Hiring a QA team - Just Don’t

Some organisations seem to think the answer is to take the regression test problem off developer’s plates by hiring less skilled individuals to do this apparently mundane work. Some even hire SDETs to write automated tests for the developers.

This is a fundamentally flawed approach in the same way that hiring an Operations teams as a separate function was accepted as a bad idea and replaced with an integrated DevOps product team. (Well, apart from the anti-pattern of a “DevOps” role, but that’s another post).

It’s flawed in at least the following ways:

It embeds the “manual testing is okay” approach in the culture, but adds some extra brute force people-power in the hope that this will prevent the eventual arrival of the 99th-feature problem. (The number just gets bigger, it’s still there). This is pretty much like trying to outrun your own shadow.
It encourages developers to consider testing (automated or otherwise) “that other team/role’s problem”.
It adds significant delays and an additional silo between writing code and delivering value to users and getting valuable feedback. Lengthening feedback cycles. This goes against everything we have learned from toyota/kanban/lean etc.
QA people cannot actually improve “quality” - they are not the developers working on the actual code, and can at best catch the worst errors.
The people who could improve quality (developers who actually write the code) lose extremely important feedback signals on how their software behaves when tested and used.

I have also seen QAs used as a “minimum quality gate” that “allows” an organisation to hire an army of poor-to-mediocre programmers. Fairly quickly that goes spectacularly badly.

Summary

So the important take-away here is:

Tests must test what your users actually care about.
Those tests must be automated to sustain your velocity.
The testing pyramid is good and valid.
Start from the outside with your testing and work in only as much as you must.

And why we do it that way is:

The features for users is more important than the internal workings.
Without full-automated coverage we cannot continue to confidently add features.

Templated repos with dotnet new

Tim Abell — Wed, 06 Mar 2024 00:00:00 +0000

I’ve been digging in to making dotnet new templates and it turns out to be a remarkably capable bit of tooling.

It’s particularly useful when you want to build a load of similar microservices with their own git repos.

It can:

Rename files.
Rename strings (variables, class names etc).
Preserve case style of renamed strings (through “derived” replacements).
Create named command line arguments for your template string replacements, e.g. donet new mytemplate --myswitch MyReplacementValue.
Exclude entire blocks of code and files based on switches that user can provide.
Be installed an tested from a local folder with dotnet new install <path>.
Be built into a nuget package and published on private or public feeds.

Importantly, there’s nothing in the way that it works that stops you from making your template code build/test/run.

And because it’s a CLI tool, if you want to an update an existing generated repo with a new version of a template you can just run it again with --force, and use git to pick through what changes you want to take into the generated repo.

This really takes a lot of the toil out of the copy-paste-modify you would have to do otherwise.

Learn more:

New tool: sln-items-sync for Visual Studio solution folders

Tim Abell — Sat, 13 Jan 2024 00:00:00 +0000

How and why I created sln-items-sync - a dotnet tool to generate SolutionItems from filesystem folders.

If you want to skip the backstory head over: https://github.com/timabell/sln-items-sync

15 years of minor irritation

Faced with another set of microservice repos written in dotnet-core, with .sln files in various states of tidiness I found my self for the 1000th time in 15+ years manually pointy clicky adding fake solution-items folders and subfolders and then toiling away adding files just so I could search them, click them and view them from within Visual Studio or Rider.

There must be a better way by now I thought, so I went hunting.

All I turned up was a lot of people asking the same thing and some dead tooling from years ago. Here’s the stackoverflow from 2008 with 90k views and 180 upvotes: https://stackoverflow.com/questions/267200/visual-studio-solutions-folder-as-real-folders, which didn’t really help in spite of having 23 answers. Not to mention the slew of linked questions where people are asking the same thing with different words.

(Solution folders aren’t to be confused with adding files to a project which used to be an equal nightmare before Microsoft saw sense and just included what’s on the filesystem. There are many old stackoverflow questions on that too from frustrated devs around the world.)

So with the programmer war cry of “how hard can it be, I’ll knock this out in a couple of evenings…” I set about on what turned out to be a significant exercise in yak-shaving in order to sort it out myself once and for all.

“How hard can it be, I’ll knock this out in a couple of evenings…”

~ Me. Again.

What to build?

I did briefly look at writing an IntelliJ (aka Rider) plugin but that turned out quickly to be a daunting thing so I put that idea down sharpish.

I use Rider in preference to Visual Studio and VSCode for C# so didn’t even look at that side. VSCode didn’t even bother with .sln files last I checked.

Next step was to write a CLI (command-line interface, aka terminal) tool to do it. (sln + filesystem in, mutated sln out, easy…)

I have recently written command line tools in both GoLang and Rust, but given this is a tool that would only be useful to Microsoft developers I figured I’d do this one in C#. I do actually like C# as a language for all my interest in other things, and thanks to dotnet-core and Rider I can actually write the whole thing on Linux Mint Cinnamon where I like to be.

Parsing and writing .sln files

I then hunted around for any nuget packages that might do the grunt work of reading/writing the sln format. Surely after 20-something years there must be something, right? Well, kinda. The VS parsing code is locked away in some windows dll nastyness, probably in C++ and COM or something evil. It even predates XML as a format, never mind JSON.

What I did find was the SlnParser nuget package, which someone had kindly written and open-sourced, and after quick test I could see it did a decent job of turning .sln files into an in-memory C# object model (a Solution class, with lists of things as properties).

So major yak number one was to fork SlnParser and turn it into a two-way tool. This I did with a lot of hackery and created SlnEditor nuget package which I published on nuget and github with the same Unlicense licensing as the original. Perhaps others will find this gift to the world useful in its own right.

Creating sln-items-sync

Finally with that working I was able to create the CLI tool I wanted, which I named sln-items-sync. This was more work than I expected, but I got a first cut working reasonably quickly.

Tests

I put a good amount of effort into good end to end test coverage on both the parser and the tool itself because I am now a true believer that without tests you will be unable to make future changes and dependency upgrades with speed and confidence.

I.e. lack of tests is the epitome of technical debt.

In fact let me give that a block quote because it’s such an important point:

Lack of tests is the epitome of technical debt.

(p.s. why is Epitome spelled that way but pronounced epitomy. English. Sigh.)

This has paid off in spades as the amount of work to get it satisfactorily “done” grew and grew the closer I got to finished.

The tests in both projects focus on “outside-in” testing rather than mockist unit testing. As such you can see at a glance the overall behaviour, spot any unexpected/unwanted output, and easily write new tests for new desired behaviour, being able to eyeball them easily for correctness. I won’t include one here as they are a bit lengthy, but you can go and look at the source repos on github.

This is made a bit easier on this tool because the only interfaces to the world are:

a text format (easy to string-compare expected versus actual)
a filesystem (I went for creating real file trees in tests which worked well and gives even more confidence)
the command line interface (for the sync tool)
the API (for the parser lib)

First contact with real .sln files

As I am doing some work for a C# contracting client currently I was able to try it out on gnarly real solution files, with a view to submitting some small cleanup pull requests that could created really quickly.

Stable ordering

The first attempt was a complete failure because the generated patch re-wrote the entire sln file in a completely different order, resulting in a sea of red/green lines full of GUIDs and other cryptic changes in the git diff. While the solution items were updated as intended and could be seen in Rider etc., this was not a patch that could be submitted to the team, or that I would put my name to.

Getting stable ordering between parsing and writing turned out to be a huge amount of work and refactoring, largely in the SlnEditor lib.

The key to making stable-ordering work was to add an int sourceLine property to almost everything when parsing, and to sort by that before rendering back out again. This had the desired effect of keeping everything in the original order no matter how it was mutated, and new items are added to the end (by replacing default 0 with Int.MaxValue before sorting).

Phew, another yak shaved, lost count now, but got more xmas hols so keep going….!

Many bugs and gaps

It surprised me a bit just how many little niggles, edge cases, and small omissions there were that had to be sorted out before I could use it to submit quality patches to client .sln files for real. Even the ever present byte-order-marker (BOM) was causing unwanted diffs because I hadn’t included it in the render, but .sln files seem to have them.

Pleasingly I’ve resolved everything I came across, apart from making the parent/child guid mapping order stable which didn’t seem to be worth the effort seeing as they are completely incomprehensible anyway.

Making a dotnet-tool

Once it was working, it was only a couple of rounds of building and copying the exe to bin/ before I got fed up with that approach to distribution.

Amazingly it turns out to be pretty simple to build and publish tools to the dotnet tool ecosystem, they are actually just slightly special nuget packages, and you only have to add a couple of properties to the .csproj file.

Making a dotnet-tool worked great, and is a great user experience for installing and running the tool. It even does updates for no extra effort!

Github-actions

To make both of these tools even easier to work on and maintain longer term, I wanted to have a good github action (aka CI) to build and run the tests.

Build and test is trivial, you can pretty much click the default workflow button for .net in an empty github actions page and it just works.

I wanted to also automate the nuget publishing of both from github-actions, as although I had a sh file to upload them from my machine that’s a faff and tends to stop working after a few machine rebuilds. Amazingly the author of SlnParser has taken an interest and provided a PR that gave me a ready-made github-action to push to nuget for every release tag! So that’s now in place, and to release a new version I can just git tag v1.2.3 && git push --tags and github does the rest.

The end, I need a lie down

So after all that, I’m not sure it was all worth it, but it’s done and I’m justifying it as a holiday hobby project and a gift to the dotnet developers of the world. I will certainly enjoy it every time I find an out of sync SolutionItems folder in future and run my tool so that I can ship a patch for it in seconds flat. I also learned a few things and got kata-like practice on shipping quality things at speed.

So with that, Merry Xmas and a happy new 2024. May all your solution folders be tidy and complete.

git - what do ‘base’ ‘local’ ‘remote’ mean?

Tim Abell — Fri, 20 Oct 2023 00:00:00 +0000

The terminology for 3-way git merge, rebase and cherry-pick conflict files is very confusing, particularly because they flip direction between rebase and merge.

When you run git mergetool it will spit out 4 files that look like this, and then pass them as arguments to your merge tool of choice:

$ gs 
## HEAD (no branch)
UU src/gitopolis.rs
?? src/gitopolis_BACKUP_1585963.rs
?? src/gitopolis_BASE_1585963.rs
?? src/gitopolis_LOCAL_1585963.rs
?? src/gitopolis_REMOTE_1585963.rs

In kdiff3 (by far the best 3-way merge algorithm out there) it looks like this:

Here’s how I think of it

Merge

You are on the target branch (local), and the patches are coming from the branch you are merging in (remote), kinda like this:

git checkout local-branch
git merge remote-branch

Cherry-pick

Same direction as merge.

You are on the target branch (local), and the patch is coming from the commit you are cherry-picking (remote), kinda like this:

git checkout local-branch
git cherry-pick some-remote-commit-ref

Rebase

Opposite direction to merge.

You start on your own branch that you want to rebase, but…

When you start the rebase you end up temporarily while the rebase is running in “detached HEAD” on the branch you are rebasing onto (often origin/main), so: You are on the target branch (local), and commits to rebase are coming from the branch you are rebasing (remote), kinda like this:

git checkout your-remote-branch
git rebase target-local-branch

Terminology

“ BASE ”: before anyone changed it (in all cases)
When merging (other branch coming to me):
- LOCAL : branch I’m on
- REMOTE : branch I’m merging in
When cherry-picking (other commit coming to me):
- LOCAL : branch I’m on
- REMOTE : commit I’m merging in
When rebasing (my own branch coming to me):
- LOCAL : branch I’m rebasing on to (checked out as detached head mid-rebase)
- REMOTE : my commits on branch I’m rebasing

Refs

https://stackoverflow.com/questions/20381677/in-a-git-merge-conflict-what-are-the-backup-base-local-and-remote-files-that

Enabling modern app security

Tim Abell — Wed, 14 Jun 2023 00:00:00 +0000

A broad-view of improving security in any organisation.

An inspirational panel discussion

Yesterday I went to a panel discussion hosted by eSynergy, “Innovation at its safest: Excellence in Software Engineering through Integrated Security Best Practices”

The whole event was live-streamed, watch the panel discussion recording here

For me who lives in developer-land, it was a useful broadening of perspectives around app security. What follows are some bits that I took away from the discussions, which I think provide a useful starting point for anyone tasked with running any modern software systems in this increasingly hostile security environment.

Who’s who at the event

The speakers at the event were as follows:

Intro from Ulrike Eder (eSynergy) [00:03:47]
“Beyond OWASP Top 10” from Grant Ongers from OWASP and secure delivery @rewtd@defcon.social / @rewtd [00:06:37]
Grant was then joined for the panel discussion by: [00:19:40]
- Salman Iqbal, Principal Consultant, DevOps and ML Security at esynergy (hosting the panel)
- Teresa Wu VP Engineer at J.P. Morgan
- Ben Burdsall, Chief Technology Officer at dunnhumby, non-exec at eSynergy
- Tom Harris Chief Technology Officer at ClearBank, BuildCircle, ex-JustEat

Takeaways

There’s a bewildering array of things you can / should / must do for the security of your systems, users and company.

Within this article you’ll find some starting points for your onwards security journey

Levels

There are two reasons for thinking of security in layers or levels:

Your business needs, risks, regulatory environment and finances
Your security maturity

Some businesses are more at risk such as banks and thus need (and can afford) a more significant investment in security measures (such as multi-layered cloud infrastructure defenses), whereas some have less budget and less risk and so can operate at the simpler levels of security.

If you are currently very poor on security then there’s little point sprinkling some advanced things on top, it’s important to properly address each layer of security capability on the way up.

Regardless of your business needs, perfect security is always an unattainable ideal, but a worthy target nonetheless. The Unicorn project calls this kind of never-quite-attainable perfection an “Ideal”. The Unicorn Project and the Five Ideals: Interview with Gene Kim

Who’s job is security anyway?

The directors are “accountable”.
The developers, product etc. are “responsible”.

While developers can and should write “secure code” (SQL Injection vulnerabilities is still on the top 10 list), it’s important that everyone plays their part.

Notably the product function (“product owners”), as they are the decision makers for balancing the competing demands placed on delivery/development teams, including how much to invest in security defenses. (Much nodding in the audience at this one!)

Developers

Tools help but the developers need to understand what is required.

At ClearBank there is:

“Lunch and learn” sessions from AppSec team.
Training with “HackSplaining”.

How ClearBank leveled-up app dev security

Added security training
Required pull-request approval from someone with security training
This created a temporary bottleneck, which encouraged everyone to do the security training
Incubated an AppSec team to “reduce the cognitive load of security in collaboration with CISO and CTO (Tom)
1. Enthusiastic internal devs
2. Additional external resource
Collaboration at the top then filters down to all the teams

Justifying security investment

The board of directors now face criminal penalties (i.e. jail time) if they don’t properly approach security. It used to be just financial penalties but that wasn’t enough as they could just be absorbed as a “cost of doing business”.

If you need the C-suite or board to take security sufficiently seriously you can remind them of the legal penalties and costs!

Do security right because it’s the right thing to do and you care about your customer and their data.
There’s the “daily mail test” - how would we feel if there was a breach and it hit the papers?
Put a cost on breaches, e.g. probability of breach multiplied by cost of breach.
Use the “house fire” analogy. No-one thinks that insuring your house against fire is a bad investment. The same is true for investing in security before you have a incident or breach.

Lead from the top

Leaders should do the training too, no-one is too important and it sets the tone and culture, encouraging everyone down to the devs to do the training too.

Shift left

Shift Left: “take a task that’s traditionally done at a later stage of the process and perform that task at earlier stages”

~ https://devopedia.org/shift-left

Move security left. Nuff said. Dev+Security not Dev versus Security.

Check security and licenses at build time. Gives assurance of security for customers.

The Top 10 is not enough

The OWASP Top 10 is a good tool for awareness and generating conversations, but addressing these is only the lowest “level” of security.

A much broader view of security is provided by the OWASP Application Security Verification Standard (ASVS). It is also broken down into levels to allow you to start at the bottom and work up as your security capabilities mature, and decide what level your business needs to attain based on the relevant risks and regulations. Banks for example would go all the way to level 3.

There are also per-environment lists. E.g. OWASP Mobile Application Security for mobile app development.

Pen tests

Don’t just tick off “pen test”, ask your pen test providers how they work.

Do they just cover the OWASP Top 10?
Do they just cover the SAMM Top 20?
Do they go deeper than the Top-n?
Do they look at ASVS?
What tools do they use?
Do the tools report against the ASVS? (If not talk to the tool provider!)

Threat modelling

Use threat modelling, assess and then defend against that.

Red/blue teams

Can be effective, but also very expensive. Do the basics first (e.g. sql-injection training!)

Tools & resources to level-up security

Training and assessments from Secure Delivery. They provide security training and assessments for everyone in the business, not just developers.
OWASP Software Assurance Maturity Model (SAMM) A “measurable way for all types of organizations to analyze and improve their software security posture”
Slim Toolkit - had a massive impact in reducing vulnerabilities at dunnhumby.
HackSplaining Security training, “Learn to Hack”. In use at ClearBank.
Snyk (pronounced “sneak”) - security integrated with CI pipelines
Bug Bounties - good bang for buck, often find privilege escalation at the app level, even for as little as £3k per found vulnerability.
OWASP Cornucopia physical card game (also available online - cornucopia online, cornucopia game source code)
OWASP London Chapter Meetups

AI, LLMs & ChatGPT

There are new threats and risks with the new AI tools:

Developers incorrectly using information provided by the LLMs
ChatGPT allows attackers to accelerate, particularly social engineering. E.g. asking ChatGPT for an org chart instead of having to trawl for data manually, then using that in social engineering attacks. This might make it quicker for an attacker to use someone’s manager’s name to lend authority.
Developers etc accidentally exfiltrating sensitive data such as private keys and passwords by providing it as inputs to an LLM such as ChatGPT that then integrates the data into its model in a way that allows extraction by a malicious third-party.

Currently use of ChatGPT is blocked at some big companies.

The change to the security landscape is a bit like asking “how did the creation of the internet change theft”.

OWASP has used LLM technology to help make it easier for clients to decide which of the 150 tools they have are most appropriate.

Text-based tools - the ultimate format for everything

Tim Abell — Thu, 01 Jun 2023 00:00:00 +0000

Having lived in the world of technology for two to three decades now, I’ve come to a fundamental truth: text formats are the ultimate format.

“text formats are the ultimate format”

~ Me, just now

It’s funny really because for everything we’ve invented, of every level of complexity, usability, shinyness etc, when it comes down to it, texts is still king, just like it was in 1980 when I was still learning to talk.

Properties of text formats

Things that make text inevitably superior to all other more complicated formats:

Simple - nothing to go wrong.
Use any text editor you like - vim, vscode+vim, intellij+vim are my gotos, but there are soooo many.
Sync, backup and restore are trivial - try as they might, nothing beats a folder-tree of text files.
They are ultimately portable - no change in technology (windows to linux, desktop to cloud, laptop to mobile) requires you to change anything, text is text, just copy them across and carry on, the ultimate defense against the ever-present pernicious vendor-lockin.
Conflict resolution is always possible - edited two out of sync copies? No problem, there’s a plethora of tools (kdiff3 is my favourite), or you can just do it manually if you wish.
Version control supported - text files are trivially versionable in tools like git, everything understands it and can show diffs etc.
Simple conventions like markdown, yaml, toml, and even slightly more complicated things like json don’t fundamentally break any of the above.
With some lightweight processing and structure (noteably markdown), the same basic format can be automatically converted to a plethora of rich and beautiful forms, and with so many tools understanding formats like markdown you are spoilt for choice.
Supports emoji - this one is more modern, but its usefulness is not to be underestimated, and thanks to utf-8 and unicode the plain-old-text-file can have rich emotions and symbols too.
You can use all sorts of interesting tools to process text files, many from the linux cli stack such as sed, grep (or ag), plus full-on shell scripting to automate repetitive tasks such as making a new blog post.

Amazing things you can do with text files

The below are all things I personally swear by and use daily. I wish more things were like this.

Markdown is by far my favourite text format, and it’s incredibly versatile. In my crusade to basically convert everything to plain text / markdown files having been repeatedly burnt by fancy binary formats (.doc anyone?). GraphViz (“dot” format) is also a notably powerful text-based system.

Blogging

As per this blog, see “Setting up a static website/blog with jekyll” from 2019. No regrets there. Writing this in vim in a terminal.

Slide decks

reveal.js can parse markdown files with a sprinkling of html & css allowed inline (very handy) and turn them into stunning modern presentations with slick animations and multi-step reveals, amazing.

I was trying to create some slides in google-slides thinking that would be the quick way, ran into some bizarre formatting limitation and went hunting for alternatives. I haven’t looked back, at least for things I don’t need real-time collaboration on.

You can see what I managed to do with reveal.js for the Rust Workshop - here’s one of the source slide markdown files

Note taking

Markdown, VSCode with some markdown plugins, maybe even a markdown-wiki tool. Markor on android. Syncthing to keep them in sync across devices. Works for me, and any conflicts due to editing files out of sync is easier to deal with than tomboy’s nasty XML format (yes I know XML is text but it’s still naaaasty).

Coding

This entry is only half tongue-in-cheek. I think it’s worth pointing out that programmers have, after flirting with many other approaches, settled on plain old ASCII as being the one-true-format for explaining to a computer (and other programmers) what the computer is supposed to be doing. Pay attention to what programmers have learnt, there is much depth here on managing vast amounts of precise information in text form. Especially if you are not a programmer or not used to text tools there is much to learn from this world. You might thing programmers are odd creatures that thrive on unnecessary complexity; nothing could be further from the truth, they (we) are obsessive about solving problems once and for all and being ruthlessly efficient in all things. The fact that programmer practices are seen as odd by the general public is more a sign of just how far programmers have optimised their lives away from the unthinking defaults of the masses than it is of any peculiarity of whim or culture.

Graphs & flowcharts

The GraphViz dot format is amazing, it takes a bit of getting used to, but once you’ve got it then you can rearrange your flow chart with vim in a few keypresses and have the whole thing rearranged in milliseconds. Amazing.

There’s even some neat web based real-time renderers:

The yucky bits

The almost-rans:

Email’s mbox format is kinda text, but due to the way it’s set up is horrible for sync
vcf for contacts, what happened there then?!
ical for calendars, what a disaster, so close but yet never works, shame
XML - nice try, turned out to be horrible in hindsight, but not before we’d written almost all software to use it (.docx anyone?)

The text world is a bit short on collaborative real-time editing - google-docs is still king on that one, though it would be perfectly possible for equivalent tools to be created for the above text formats and tools. Watch this space.

Crappy half-arsed implementations of markdown, looking at you Jira/Confluence/Slack (not really a problem of text, more something where we’re almost there with and then crappy WYSIWIG implementations wreck it).

Maintaining software - a bare minimum

Tim Abell — Mon, 22 May 2023 00:00:00 +0000

All the press goes to new features, but there’s a lot that has to happen just to stand still in software development.

None of the following results in “shiny new feature that everyone is excited about”. It’s the ongoing work that anyone who’s not in day-to-day software development might not appreciate, sometimes questioning where the time is going.

Here’s a catalog of things that eat engineering time, but that are eventually unavoidable if you don’t want to grind to a halt under a mountain of tech debt:

Non-feature work

1) Bugs

Customers (or your monitoring) notice something that’s not working:

Investigate and ship a fix,
or worse, spend time investigating only to discover it can’t / won’t be changed or fixed.

2) Minor dependency upgrades

e.g. upgrading xUnit from v2.4.0 to v2.4.2

These are usually trivial if your tests are good and the authors respect Semantic Versioning. They still need to be done regularly to keep the impact small.

3) Major dependency upgrades

e.g. upgrading MediatR from v9.x to v10.0.0

“This release includes the following breaking changes in the API …”

~ MediatR release notes

4) Platform upgrades

e.g.

There is often significant changes, including removal and changing (sometimes called “breaking” or “breaking changes”) things that your code relies on.

You might be tempted to put these off. Don’t. The longer you leave it, the worse your problem becomes, eventually becoming insurmountable.

5) Fundamental shifts

Sometimes there’s an enormous shift in technology, e.g.

On-premise compute to cloud compute.
Desktop to mobile.
Server-rendered web to API + Single Page Applications (SPAs).
More recently, the shift from servers to serverless.
Data storage (SQL vs NoSql, vs Graph databases).
New hosting and technology platforms.

If you don’t keep up to date then you find it increasingly hard to operate what you have (no engineers want to work with the old tech, the online world no longer supports you with information and tooling, etc). And your customers expectations start to demand things that your outdated approaches are just unable to support.

Have a plan for regularly considering these and taking action. You might spin up new teams to try them out, or give people “Friday time” to explore new things. The only thing you mustn’t do is “nothing”.

Why is keeping on top of upgrades important?

Why not just ignore the upgrades till you need them?

Two reasons:

Security fixes
The longer you let it pile up, the harder it gets (exponentially so).

Keeping changes small

If you allow upgrades to pile up for a month or so, you’ll have one big patch that upgrades many things. If something breaks (even with good test coverage) it can be a lengthy process to figure out which upgrade broke it and what to do about it.

If you do this regularly (weekly at least), then you’ll only be upgrading a few minor versions at a time, and it will be immediately obvious where to start looking if something breaks (i.e. roll back, then upgrade the 5 dependencies one at a time, and look at the changelog of the one that breaks it.)

Test coverage

Upgrades are a key reason that good test coverage (and the functionality level) are very important. Without these you will have a significant manual testing effort for every upgrade. Relying on manual testing results in avoiding upgrades for longer, and breakages making it to production unnoticed.

Monitoring

Good exception monitoring and telemetry in production will improve your ability to catch any oddities that slip through your test coverage.