DEV Community: Ben Selby

Two weeks with the GitHub Coding Agent

Ben Selby — Sun, 29 Jun 2025 13:51:44 +0000

For the last two weeks, we've been learning to work with the new GitHub Coding Agent. This is a shift for me. Moving from working with AI in my IDE, to actually delegating entire tasks to an agent that works asynchronously in the cloud. The aim is to be more efficient, and provide a bigger impact for the business.

From now on we will call the GitHub Coding Agent "Mx Robot". Why? I think you need to think of this as a colleague that may or may not deliver what you need. I see in the community we are comparing a Coding Agent to be like a Junior Developer, and this makes sense to be at the moment. I'm learning new ways to think of solving the issue, which you also get with Junior engineers (fresh eyes!).

How we did it

First off, I'm in a team of two people at the moment. So the first week, it was very much an after thought. "Oh I could have given this to AI". Only 7% of the pull requests merged were from Mx Robot. The second week we were much more intentional. Each morning at 9am, we had an informal review of the backlog and agreed what to delegate to Mx Robot. We currently work in Jira, with no integration with the coding agent. This meant we had to re-write/copy/paste the Jira ticket into a GitHub Issue, or the Copilot agent window on GitHub.

Once the prompt had initiated work, we copied our prompt back to the Jira ticket so we had a reference of what was asked. The intention here is to learn which phrasing provides the most success.

The process took an hour each morning. This was mainly due to learning and then demoing our process to another team. In reality this could have been 30 minutes. I would say roughly 60% of the work last week was ear marked as "AI achievable". This means that we believed we could delegate to AI if we wanted to and get some level of success.

We agreed to only have 2 tickets in progress with Mx Robot at a time. Maybe it's a coincidence, or maybe 1 AI ticket per dev (plus their own work) is a workable heuristic. We haven't gathered enough data to make a determination yet. Our team grows to three devs next week, so we can see if this holds true.

We also agreed to only have three interactions with Mx Robot on the pull request. If we couldn't get it over the line with three interactions then we either:

Scrap it entirely and get humans to do the work, or
Take over from the pull request if there was a good foundation to build on.

Lastly, and maybe we were too pessimistic, we agreed that green field development that needed a wide range of thought we would leave to the humans.

In week two, 24% of our pull requests were created by Mx Robot. We went from a statistic of 1.1 tickets per human per day, to 1.8. This is over a 3 week rolling average.

What worked

We've given Mx Robot the following types of work

Refactoring configuration files.
Removing old feature flags.
Adding indexes.
Adding throttling configuration to API endpoints.
Fixing bugs in a user interface.
A very small feature that builds on top of an existing data structure we have.

Most of those changes were then reviewed, tested, approved, and merged within three interactions. I personally feel like I've had more of a "testing" mindset when reviewing the work from Mx Robot. I don't trust that it will have caught everything, so there is more testing overhead here. Keep this in mind when thinking about productivity. You don't get the results entirely for free without human productivity helping.

How did it all play out then?

11% of pull requests were abandoned as way off the mark.
22% of pull requests had 2 comments to get the change ready for human review.
22% of pull requests had no feedback, were tested locally, and then merged.
56% of pull requests had one piece of feedback.
22% of pull requests took a further 2 commits to get them over the line.
- Looking at the feedback, this was consistently due to bad linting and/or fixing tests.
- This was the case even though we have a copilot-instructions.md file explaining all linting and tests need to pass for each change.

What didn’t work

The flow of going from Jira to a GitHub Issue or Copilot window is very clunky. So much so, that we aim to trial not working in Jira for an entire week. We want to bring the product managers and designers closer to Mx Robot, not make engineers a proxy from ticket to pull request. "Pushing left" is still a thing in this new era.

The other big take away from the two weeks would be specificity. When we were super clear what we wanted, it did well. Where we were more vague we had to be more involved with the pull request. I suspect this is the distinction between AI Assisted and Vibe Coding. We did care about the code, and the solution, in comparison to vibe coding where you're less concerned with the output of code.

Next experiments

Next week we are going to mix it up a little.

Remove the 9am call and let each engineer decide what is given to Mx Robot. We believe we learnt a lot last week to the point where pairing on delegation is less fruitful.
We are going to try giving Mx Robot some more complicated work.
We may try being more vague again on the assignment. We need to find a balance here.
We are going to keep a good and a bad change and then review. The aim is to see what changes we can make to hit the target quicker.
Delve a little deeper into Copilot Spaces.

Speaking for myself, I believe it increased the cognitive load and context switching to some extent. However, I think this is about letting the flow bed in. Removing the sync call at 9am will give us some time back, that means we can spread the AI reviews throughout the day a little more.

Tips

Be direct. We asked "Can you create a pull request", and the answer was "Yes". Did it create the pull request - No. Changing this to "Create a pull request that..." helped.
Keep tasks small and scoped. I personally like creating lists. It turns out, AI like lists too.
Review the PRs like you would a junior dev’s. It’s good, but not infallible.

It’s early days, but I can see this becoming a core part of my workflow. Not because it’s flashy, but because it quietly gets things done. I'm curious to see when our Mx Robot might be up for promotion too.

... and yes, the banner image was created by Copilot (and Leonardo Da Vinci).

Dependency Management

Ben Selby — Sat, 18 Jan 2025 21:40:05 +0000

This post is going to be an advocate for managing your project dependencies and ensuring they are up to date. This seems to be a controversial topic, so I suspect we won't all agree. I think it comes down to your personal experience as to how you might view dependencies.

Back in my early career days, I had to deal with fixing dependencies when a security exploit was discovered in one of them. It was a real pain, and I never wanted to do it again under those circumstances: super stressful and time-sensitive. So, I made a firm decision: keeping dependencies up to date was the way to go.

Checkout the Code maintainability capability on the DORA website for a more robust argument.

What is a project dependency?

A dependency, or library, is essentially someone else's code. You would install this via your language's package manager, for example npm (Node), composer (PHP), or pip (Python).

The dependency is likely to give you some functionality that you do not want to write yourself. This could be as grand as a framework, such as React, Django, or Symfony. Or, it could be a dev tool like a code linter, and everything in between.

You essentially outsource the responsibility of logic to another project or engineer to perform a capability for you. In most cases, you will interact with it through its public API or contract within your project.

Before you keep adding dependencies to your project, you may want to consider if it’s better to use a third-party project, or to write it yourself. Whilst it is quicker to install other libraries into your project, in the long run it may actually be harder to support. We delve into this next.

So why do we need to keep dependencies up to date?

It's a valid question. For me, it breaks down as follows:

Security - If a dependency has a security exploit, you want to be able to update it as soon as possible. This is to ensure your project is not vulnerable to attack. The longer you have left the dependency to decay, the harder it could be to update it. For example if there have been major version changes, you might need to refactor your code to integrate with the new version.

Bug fixes - It's likely that your dependency has been worked on by the project owners. Just like your code, the library may have had bug fixes, and performance improvements applied. Your application could benefit from these changes.

Improvements - Maybe the library has had some new features added to it, that could improve your project. By updating frequently, the changeset you need to understand is minimal. You might be able to do small refactors to your existing code to take advantage of the new features.

Compatibility - If you're using a library that is not being updated, you could find yourself in a situation where you're unable to update other dependencies. This could be because the old library version is no longer compatible with the newer versions of other dependencies.

Maintenance - Regular updates prevent the accumulation of technical debt, making it easier to maintain your project in the long run.

How to keep the dependencies up to date?

We have a few options available to us here. As mentioned above, really think about if a dependency is even required.

If you really need the dependency, then you can either update your dependencies manually, or you can automate the process.

I'd recommend automating the process, which takes the burden away from keeping them up to date (almost).

Some options would include:

Dependabot - Built within the GitHub product set, this will create pull requests for you to update your dependencies.
Renovate - A vendor agnostic solution, that can be used with GitHub, GitLab, Bitbucket, and Azure DevOps.

I've personally only ever worked with Dependabot, and never really had any issues with it.

An example of what a simple Dependabot configuration looks like is as follows. This will monitor your npm dependencies on a weekly schedule and raise pull requests for you, if your dependencies are out of date. Check out the docs to see if your package repository is supported.

# .github/dependabot.yml
version: 2
updates:
  # Maintain dependencies for npm
  - package-ecosystem: 'npm'
    directory: '/'
    schedule:
      interval: 'weekly'
    commit-message:
      prefix: 'npm'

However you update your dependencies, I would recommend at least reviewing the changelog for the updated version. Automation is fantastic, and you can get all your automated tests running, and merge without a human being involved. However, you lose context. You would not find out about new features and improvements unless you review the changelog. It's also a good habit to know what you're installing into your project.

Summary

In this post, we explored the importance of managing and regularly updating your project dependencies. Keeping dependencies up to date is crucial for maintaining security, fixing bugs, accessing new features, ensuring compatibility, and improving performance. Outdated dependencies can lead to significant issues such as security vulnerabilities, compatibility problems, and technical debt.

I highly recommend you get them up to date, and then automate the process for keeping them up to to date. Good luck.

Photo by Mike Hindle on Unsplash

Stop tracking changes to certain files in Git

Ben Selby — Sat, 04 May 2024 07:34:30 +0000

If you've used git before, you're probably aware of the .gitignore file. This is what the manual has to say:

A gitignore file specifies intentionally untracked files that Git should ignore.

This file is usually tracked in your Git repo, so every contributor ignores the same files. Interestingly, you can also have a global ignore file in your home directory too ~/.gitignore. I have a global ignore file for ignoring basic IDE files.

Recently, I had a situation whereby the team was split on whether a file should, or should not, be ignored. Tricky, however, there is a way around it.

Assume unchanged

Let's say a file is not going to be ignored in the repo, but you specifically don't want to track changes to this file locally, you can run the following command.

git update-index --assume-unchanged [file]

This is what the manual has to say:

When this flag is specified, the object names recorded for the paths are not updated. Instead, this option sets/unsets the "assume unchanged" bit for the paths. When the "assume unchanged" bit is on, the user promises not to change the file and allows Git to assume that the working tree file matches what is recorded in the index.

OK, so you're promising that you won't change the file. I think this works for our situation, as the promise is really "I don't mind if Git doesn't track this file locally". This does not impact anyone else on your project either, so it's win win.

OK, let's say you change your mind and you do want to commit changes to the file. Here's what you do to track the file locally again.

git update-index --no-assume-unchanged [file]

What if you've completely forgotten what files you are, or are not, assuming unchanged?

git ls-files -v | grep "^[[:lower:]]"

The -v flag will alter the ls-files output to put a lowercase letter before the file if the assume-unchanged bit is set. e.g.

h untracked-file.ts
H tracked-file.ts

This means we can then grep on lowercase letters to see which files are assume-unchanged.

Summary

Hopefully, this solution helps you ignore files that you don't want to commit by accident, without them being in the .gitignore file for your project.

Working with Zed for a week

Ben Selby — Thu, 01 Feb 2024 19:47:32 +0000

I took Zed for a spin this week, and this post covers what I noticed whilst using it as my day-to-day editor.

I love it, and I'm probably going to stay with it (but I miss a debugger).

This is what Zed is:

Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter. It's also open source.

Projects worked on:

These are the projects I managed to work on this week with Zed:

Python project - 4m+ lines of code
Go project - 7k+ lines of code
Markdown - This blog post

No debugger

Honestly, I thought I would miss this a lot more than I did. However, to remove VS Code from my setup, I'd need the debugger. Firing up a second editor, to find the files, to then debug is a little tedious.

It's fast

Apart from searching for a symbol, which feels slower than VS Code, Zed seems quicker overall. I prefer the slim-down nature of it in comparison to VS Code. It certainly reminds me of Sublime Text, which I loved (and also now struggle to remember why I stopped using it!).

Collaboration UI

I managed to get a colleague to pair with me for 15 minutes using Zed, and it took us a wee while to get used to what the user interface was telling us. It's handy though:

Screensharing - Although we couldn't seem to configure which screen to share, so it shared my work diary, not the screen with the editor on. You live and learn.
Audio 1:1 calling - This was the first thing we tried, and the quality levels were very good.
Channel audio - It seems like you have to join the audio even if you just want to use the text chat. Maybe this was an oversight by us though?
Channel messaging - Slight bug that you couldn't copy the messages from the chat window.

I have one slight concern about this feature: Normally all this context is in Slack (for me at least), but it would be in Zed. Does this lose a collaboration touch point with the rest of the team who aren't in Zed? One to ponder. You may want to pull in a Product Manager, or a designer into the conversation (They are generally saviours to any given situation). I highly doubt they would install an editor to do that. It does, however, remind me of the early 2000's when engineers had to use IRC to communicate with each other. It's a little bit secret, which makes it fun.

Git integration

This was a surprise to me, as I hadn't anticipated this impacting me. I use git on the CLI. I've never used a UI, and I struggle with them for git. However, what I did miss was the little "git insights" you can get in VS Code. e.g. "Developer Alice modified this code last week".

This gives you a sense of "freshness" of the code. Back to git blame.

Keymaps

Although it says I was using the VS Code one, there were a few keymap changes I needed to make. No big drama.

Outline view

It would be nice to have the Outline View as a scrollable dock item, rather than a modal. It feels prohibitive as a modal, and it doesn't allow you to collapse certain symbols. So in a codebase that may have 15 classes in a file, you couldn't focus on the class you want to look at. It's all expanded.

Unused variables

It would be good to highlight unused variables, assuming this comes from the language server. It wasn't a biggy, as pre-commit hooks captured it, but it slows you down ever so slightly.

Copilot

Easily configured, but I did miss having Copilot Chat. You can use the Open AI integration, but I already have a license for GitHub Copilot, so not sure I want to pay for both services.

It seems to "Just work (tm)"

I've pretty much automated the install for VS Code, but it feels like there is a "lot going on". That doesn't feel the same with Zed. I opened a Python file for the first time, and it downloaded the language server. Same with the Go codebase.

Apart from the debugger, it had the right level of tools I needed. I also work on TypeScript and PHP projects, so that will be the next test to see how it handles those languages.

Summary

I'm going to stick with it. I suspect I will fire VS Code up when I need to debug something, but let's see how we go with Zed for a little bit longer.

Photo by Diego Jimenez on Unsplash

Show your working out

Ben Selby — Mon, 22 Jan 2024 19:22:35 +0000

Does anyone remember what your Maths teacher used to say?

Show your working out!

I mean, I bet they said more than that, but this is what I remembered. I think we used to get marked down if you just provided the answer. You got more marks when you showed how you derived your answer.

This post is a plea for software engineers to do the same when answering technical questions, or debugging an issue in group communication channels.

Let's take a standard question we are probably asked from time to time: "Can we see if there is a performance issue on our application right now, please?"

It's a typical question, so let's show differing levels of answering this question.

1 - Just the answer 🙂

You're asked, and you answer.

No, I cannot see any performance issues at the moment.

Helpful to the originator of the question, they have their answer. Perhaps less useful to the next engineer to come along?

2 - The answer and visuals 😊😊

You're asked, you answer, and you provide some evidence to back up your opinion.

No, I cannot see any performance issues at the moment. See the attached graphs for the relevant systems.

Graphs attached 📈📊💹📉

With any luck, the graph will provide some hints to the next engineer where you got the data from. The next engineer may be able to come to the same conclusion. Maybe next time, they can help answer the question (Multiplier effect).

3 - The answer, the visuals, and links to the original data source 😊👑🧠

You're asked, you answer. This time, you provide clickable links to the original data source that backs up your opinion ❤️.

Looking at this graph for system A, and this graph for system B, I cannot see any performance issues in the last hour. Interestingly, if you scale out here for 1 week, we are running a little quicker than the norm.

See the attached graphs for the relevant systems.

Graphs attached 📈📊💹📉

With this level of context, many engineers can now walk in your shoes as to how you answered the question (Multiplier effect unlocked). Ideally, for those without direct access to monitoring systems, you provide visuals that are useful within in your communication format (be it Slack, email etc).

For me, this is the best type of answer to the situation. Sure, it takes longer, but perhaps this unlocks others to start answering the same questions in the future. The more you share knowledge around engineers, the quicker we can collectively respond to questions.

Summary

I'd recommend we strive for example 3 above. If you provide enough context, others can learn from your experience. This unlocks their potential, and allows everyone to deliver at pace.

Photo by Susan Q Yin on Unsplash

A case for using a Makefile in your project

Ben Selby — Wed, 17 Jan 2024 20:12:23 +0000

This post is going to present a case for using a Makefile in your project. This will be the top-level task runner for your project. You can use make to run all of your project tasks such as building, linting, testing, and deploying your project. It doesn’t matter what language your project is written in.

What is a Makefile?

Make gets its knowledge of how to build your program from a file called the makefile, which lists each of the non-source files and how to compute it from other files. When you write a program, you should write a makefile for it, so that it is possible to use Make to build and install the program.

Source

Straight out of the gate, this doesn’t necessarily seem like something we could make use of, does it? However, at its core, a Makefile allows you to define a bunch of instructions together. Each set of instructions can then be given a name, and this becomes the “target”. We use a tool called make to run one of the targets defined in a Makefile.

If you’re working on JavaScript projects, for example, you may be using tools like npm, yarn, or pnpm to run your tasks. A Makefile is a very similar concept, and I’d recommend you have both.

The reason for this recommendation is abstractions.

Abstraction

We can use make to define an abstraction layer between your internal project decisions, and the outside world (a contract). The outside world could include what gets run in your CI/CD pipeline, or what engineers run in their terminals to build and test the project. By defining and making use of a Makefile, you can change your internal project workings, but leave the contract intact.

A concrete example for a JavaScript project could be switching your project from using npm to pnpm. If you’re using tasks within your package.json you have probably exposed npm run ... tasks in your CI/CD pipeline, your README, and your engineers will be running those commands. If you change your tool to pnpm you suddenly need to change a lot more code to bed this in. Your engineers also need to change their muscle memory to use the new version of the command. However, if you had a Makefile as a facade for your project, your make build task would stay the same to the outside world, but the internals of what make build does would change to use pnpm.

Another example could be for a project written in Go, where there really isn’t a task runner, and you may need to remember a rather complicated command to build your Go binaries. By having a Makefile you can document all the key commands in one place, and reference them.

Example

Let’s take a look at a single make target from this repo, which is a Go project.

.PHONY: test
test: ## Run the unit tests
  go test ./... -coverprofile=coverage.out
  go tool cover -func=coverage.out

Above shows a single make target called test. We can execute this target by running make test.

❯ make test
go test ./... -coverprofile=coverage.out
ok      github.com/benmatselby/walter/cmd       0.504s  coverage: 18.2% of statements
ok      github.com/benmatselby/walter/cmd/board 0.243s  coverage: 75.7% of statements
ok      github.com/benmatselby/walter/cmd/search        0.844s  coverage: 90.6% of statements
ok      github.com/benmatselby/walter/cmd/sprint        1.052s  coverage: 85.5% of statements
go tool cover -func=coverage.out
github.com/benmatselby/walter/cmd/board/board.go:9:             NewBoardCommand100.0%
github.com/benmatselby/walter/cmd/board/issues.go:15:           NewIssueCommand33.3%
github.com/benmatselby/walter/cmd/board/issues.go:33:           ListIssues     93.8%
github.com/benmatselby/walter/cmd/board/list.go:13:             NewListCommand 33.3%
github.com/benmatselby/walter/cmd/board/list.go:30:             DisplayBoards  100.0%

As you can see, we have output from the two commands defined in the test target. This is the same command engineers use when testing the software on their machines, to what is run in the CI/CD platform for quality checks.

If we wanted to change the options to go test, we could do this in one place, and it not impact everywhere else. It would be a trivial encapsulated change. We could also switch out go test for another test runner, and it wouldn’t impact anything else either.

Automation

Taking this further, if you can define a standard Makefile for all of your projects with the same target names, you can probably automate more things at a higher level. For example, you could do the following:

Clone your all git repos for your solution.
Iterate over each one and install their dependencies (make install).
Run the unit tests to make sure they work (make test).
Run each application (make run).

Perhaps this could be part of an automated system to spin up your entire project stack on your local machine. It wouldn’t matter if some of those projects were Go, React, Python, PHP for example, as the public contract (The Makefile) is the same.

Summary

As with most things, you have to decide if adding an extra abstraction layer is worth it. I’ve been using a Makefile in my projects since about 2013 when we transitioned a PHP project away from ant/phing. These projects have ranged in the language used, but what is consistent is the name of the Make targets. I can pretty much clone any personal repo, and remember all the make commands irrelevant if the project is PHP, Python, JavaScript, Go etc.

As outlined in this post (2018) using make to define how to run and test your project has gone into a full ecosystem to build development environments for engineers.

Automating the startup of a dev workflow

Ben Selby — Tue, 02 Jan 2024 19:09:39 +0000

Hopefully, this is the first of many small posts showing how I automate my workflow.

Back around 2008, I had a mentor/buddy for my new job. Rys. He was a wonderful, knowledgeable, and supportive buddy who always seemed to be one step ahead.

One day he noticed that I came into the office each day, opened the terminal, and created x amount of tabs, which I then named (So I knew what each terminal was there for, e.g. logs, database, git, and codebases etc).

He asked me:

How much time would you save if you could automate that?

Sitting there, stunned, I wasn't sure how to do what he said, but that question has stuck with me throughout my career. Then over the years, I've started to automate more and more, because I would remember Rys asking me how much time I would save.

So, I now have an alias defined as follows:

start-company() {
  open -a Music
  open -a Slack
  open -a "Google Chrome"
  open -a Obsidian
  open -a Notion
  open -a Miro
  open -a "Visual Studio Code"
}

Where company is the name of the company I work for. This may seem small, and maybe even obvious, but running start-company each time I boot my machine is substantially quicker than opening each application in turn.

Running this command is the first of many stacked habits I have in the morning. For more information on stacking habits, check out Atomic Habits. I'm hopefully going to write up other habits I have for my workflow.

So I have my terminal and other applications open, but how about naming those darn terminal windows Rys asked me about all those years ago?

Well, I now use tmux and tmuxinator. I have had many failed tmux attempts over the years, but I'm firmly bedded in now.

What is tmux?

tmux is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.

That nicely brings us to tmuxinator:

Create and manage tmux sessions easily.

Therefore I can start a tmux session, via tmuxinator, which has a defined configuration. So one example would be:

name: dotfiles
root: ~/

windows:
  - bms: cd ~/git/github/benmatselby
  - work: cd ~/git/github/[company-name]
  - dotfiles: cd ~/git/github/benmatselby/dotfiles
  - life: cd ~/git/github/benmatselby/life

When I now run tmuxinator start dotfiles (I have this aliased to txs), I get 4 named windows/tabs in a tmux session, all in the correct working directory. This is a massive time saving for me. I have many tmuxinator configuration files now. Each project generally gets its own configuration.

Hopefully, this setup is useful, or maybe interesting to you. Happy automating folks 😊

Photo by Joshua Sortino on Unsplash

Setting up k9s skins for different Kubernetes clusters

Ben Selby — Sat, 01 Apr 2023 07:18:07 +0000

This post is going to show you how to define different skins for k9s, so you can quickly see which environment you're working on.

k9s is a terminal application that allows you to manage your Kubernetes clusters. So if you're managing your production cluster, for example, you probably want to take extra care of what you're doing.

Every time I jump onto a production database for nearly two decades, I have set the background to red. Over the years better tooling has come out, so I now have iTerm profiles that allow me to quickly change the colour of the background when in different environments. Recently I was doing a bit of work on a Kubernetes platform using k9s and was doing the same thing, setting my iTerm profile. Then I checked on the k9s docs and realised you can do this within k9s. More importantly, it would be automatic based on your cluster name, which sounds even better.

So, I've set up the following skins:

A standard skin
A testing skin
A production skin

You can check out the official skins on GitHub. I've based my skins on monokai.yml.

Once you have picked your base skin, save it to this path:

~/Library/Application\ Support/k9s/skin.yml

When you now fire up k9s, you should have the skin you have picked. This is what mine looks like as standard.

Now we want to move on to the good stuff and have different skins for different clusters. Well luckily, k9s supports this. It's about naming the skin after the cluster name. So, I have a cluster called prod, therefore I've created a file called:

~/Library/Application\ Support/k9s/prod_skin.yml

The theme is still the base monokai.yml skin, but I changed the background value to:

background: &background '#41000c'

So now, when I connect to the prod cluster using k9s, my experience looks like this:

I've also gotten into the habit of having a specific colour for testing environments. So basically only my local development environment looks "normal". Everything else has a colour that can imply how worried I should be about making changes. My testing skin has a dark purple background.

All of these skins are defined in my dotfiles repo and you can see I symlink the skins, so I can easily track changes in git.

Hopefully, you found this useful.

Photo by Venti Views on Unsplash

How useful are code coverage reports?

Ben Selby — Sun, 06 Nov 2022 14:37:49 +0000

This post is going to talk about code coverage reports from unit tests. However, to provide some context first, I'm going to reminisce about 2010.

A story

Back in 2010, I was working for Plusnet, an ISP in the UK (Referral link). The engineering team were still getting to grips with unit testing our code. What we saw time and time again was many tests being written, high coverage reports, and management was happy. However, if you checked out a test, we saw that tests were running, and executing code, but they were not asserting anything. This meant the tests were more of a linter. So I wrote the --assert-strict feature in PHPUnit (which has subsequently become fail-on-incomplete). This helped us raise the quality of our tests at Plusnet.

Why mention this? Opinions mostly come from experience, and I wanted to share mine before talking about this.

Tools

I value coverage reports. It's a tool in the toolbox, it is not the tool in the toolbox! It is also not perfect (nothing is).

When I started a new job this year I found the project had thousands of tests, which is lovely. As a new engineer, I wanted to see what that looked like in terms of coverage. Did we have thousands of tests and high coverage or thousands of tests and low coverage? Why does it matter? For me, it gives me the confidence to make changes in the code. Once I have the confidence, it unlocks the ability to move at pace.

Getting the coverage report is not easy, as we lack test suites to help figure this out. I briefly mentioned we could work on this, and a colleague mentioned that coverage reports aren't helpful, and folks end up chasing the metric.

At this point, I want to agree with the latter statement. I value coverage reports, but I do not think you should chase unrealistic metrics.

Many testing tools allow you to ignore code in coverage reports. I'm genuinely against this concept because I want to gain confidence. I'm personally not after 100%. I'm after the highest coverage you can get, for the maximum benefit, and no more. If there is a gap in coverage, I'm fine with that, as long as it's clear and upfront.

It's also worth noting that bikeshedding over the last 5% of code coverage, is more than likely not going to add overall value. You will get diminishing returns, in my opinion.

Metrics

So, back to coverage reports. Line coverage is handy, but try and get a branch coverage report. This shows which paths through the code are tested. That's two metrics that can and should be used with many other metrics you can get from your code, e.g. Cyclomatic Complexity, and Static Analysis. No single metric can tell the entire story. Let's take a well-known example from the past that no one uses anymore. If you judged a software engineer on one single metric: lines of code written, you are not getting a realistic picture of that engineer!

Software is no different, you need to weigh all these metrics together. Each metric provides a different lens to look through in terms of quality.

So, meh, does it really matter?

Importance

Well, I would argue that if you don't have visibility of what code your tests are covering, you're now going to rely on monitoring and analytics platforms to identify quality issues with your code. When we say monitoring and analytics platforms, we actually mean our users. Our users are doing things on our platform, and the monitoring systems alert the engineers when things fail.

This week, as a relevant anecdote, we saw some notifications from Rollbar that showed us something was wrong with a new feature. We managed to get a coverage report for a single module of our code (took a little while), and we saw that the bugs identified in Rollbar were actually not covered by tests.

I personally want to know about this in development, not from our users, via Rollbar. It's more costly to fix it for one thing, and it's also impacting our customers, which isn't great. It's also stressful working in an environment where you know you have production issues impacting real people.

Summary

In conclusion, this is how I would summarise my views on this.

I value a holistic set of data points that help us understand quality in software development.
Code coverage is a single metric that can be part of that set of metrics you monitor.
No single metric can stand by itself, and be meaningful.
Nothing is perfect, which is why we should value a toolbox.
I don't believe in gaming the system and "hiding" uncovered code to get to 100%.
- You need engineering teams who are prepared and confident enough to publicly share their coverage reports. This sets the tone of the culture.
Context is needed, always.
- There will be reasons why the coverage is as it is.
Use tools that help engineering teams with confidence/delivering at pace and ultimately delivering customer satisfaction.
You cannot compare reports from different teams or projects.
- More than likely the same with most metrics.
Striving to improve is a mindset we should engage with, so use tools that help that improvement.

Debugging Python with a virtual environment in VSCode

Ben Selby — Sun, 30 Oct 2022 15:09:19 +0000

My current job is mainly Python, so there was a steep learning curve. I have done some Python previously, but mainly for writing Sublime Text plugins back in the day.

Over the last few months, I’ve set up my VSCode environment on my work machine. Last night I was wanting to do something on a different machine. I could not remember how to debug Python code inside a virtual environment within VSCode!

I kept getting this error

Exception has occurred: ModuleNotFoundError
No module named 'django'

The error could have been a different module, but for me, it was django.

I sync all my VSCode settings via my dotfiles, so I could not understand why it worked on my work machine and not my personal one. I did know it was a virtual environment issue though.

After some digging, I found out you have to pick your default interpreter. In VS Code open the Command Palette (⇧ ⌘ P). Then select “Python: Select Interpreter".

This will produce a list of Python installations that you can pick from.

Pick your local virtual environment Python interpreter. For me, this is the one that has a star next to it.

This then allows you to use your debug configuration. For completeness, mine was:

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Shell plus",
      "type": "python",
      "request": "launch",
      "program": "${workspaceFolder}/manage.py",
      "args": ["shell_plus"],
      "env": {
        "DOMAIN": "localhost:8888",
        "CLIENT_DOMAIN": "localhost:8080",
        "DYLD_FALLBACK_LIBRARY_PATH": "/opt/homebrew/lib/",
      },
      "django": true,
      "justMyCode": true
    }
  ]
}

That's it. Hopefully, I will remember this next time, or at least know I've written it up here.

Photo by Jonny Gios on Unsplash

Mongo shell commands to get you through the day

Ben Selby — Sun, 01 May 2022 16:39:12 +0000

I recently started a new job. The data store is MongoDB, which I have barely used before. This post aims to document what I've learnt to get me through the day. Please feel free to tweet me if there are discrepancies.

The first thing that I became aware of, is that there are/were two cli tools:

The legacy mongo shell, which was deprecated in MongoDB v5.0.
The new, and active, mongosh shell, which replaced mongo. The documentation for this shell is here.

This post will discuss mongosh and assume you have MongoDB running on your machine (or access to an instance).

Connecting

If you're wanting to connect to a local database instance, then by running mongosh it will connect to localhost as default.

A more complete connection string would be:

mongosh "mongodb://localhost:27017" -u [user]

This will prompt for the password, so it's not stored in your ~/.[zsh|bash]_history file.

Once you have connected, you can run help to get some basic commands:

> help

  Shell Help:

    use                Set current database
    show               'show databases'/'show dbs': Print a list of all available databases.
                       'show collections'/'show tables': Print a list of all collections for current database.
                       'show profile': Prints system.profile information.
                       'show users': Print a list of all users for current database.
                       'show roles': Print a list of all roles for current database.
                       'show log <type>': log for current connection, if type is not set uses 'global'
                       'show logs': Print all logs.

    exit               Quit the MongoDB shell with exit/exit()/.exit
    quit               Quit the MongoDB shell with quit/quit()
    Mongo              Create a new connection and return the Mongo object. Usage: new Mongo(URI, options [optional])
    connect            Create a new connection and return the Database object. Usage: connect(URI, username [optional], password [optional])
    it                 result of the last line evaluated; use to further iterate
    version            Shell version
    load               Loads and runs a JavaScript file into the current shell environment
    enableTelemetry    Enables collection of anonymous usage data to improve the mongosh CLI
    disableTelemetry   Disables collection of anonymous usage data to improve the mongosh CLI
    passwordPrompt     Prompts the user for a password
    sleep              Sleep for the specified number of milliseconds
    print              Prints the contents of an object to the output
    printjson          Alias for print()
    cls                Clears the screen like console.clear()
    isInteractive      Returns whether the shell will enter or has entered interactive mode

  For more information on usage: https://docs.mongodb.com/manual/reference/method

Listing the databases

Using the output from help, let's list all the databases stored in the instance we are running on.

> show dbs

We want to be able to switch between the databases, so lets switch to the admin database.

> use admin

Creating a database

So we have seen how to list databases, but we want to create a new one to store our data. To do this, we can run:

> use entertainment

You may notice that we are using the same use command. By switching to a non-existent database, the database will be created, when we create a collection.

Deleting a database

Note: I do not recommend testing this on production!

Make sure you are on the database you want to drop:

> use entertainment
switched to db entertainment

And then let's drop the database.

> db.dropDatabase()
{ ok: 1, dropped: 'entertainment' }

Creating a collection

Using the commands from above, create a database called entertainment again. Then switch to that database. We now want to create a collection. Let's just make sure we are on the same page again. Let's check the name of the database we are both on.

> db.getName()
entertainment

Your output should match mine, entertainment. Now, let's create a collection called films.

> db.createCollection("films")

For further options/configuration on creating collections, please review the docs.

Showing collections

Let's see what collections we have in the entertainment database.

Note: If you're coming from a SQL based background, collections are the MongoDB equivalent of tables.

> show collections
films

Inserting documents

Now we have a database (entertainment), and a collection (films), let's create some documents.

> db.films.insertOne({ title: "The Big Lebowski", url: "https://en.wikipedia.org/wiki/The_Big_Lebowski" })
{
  acknowledged: true,
  insertedId: ObjectId("626eaad80dd6e8884b86fda3")
}

This has created a document in the films collection with two attributes: title, and url. Behind the scenes, MongoDB will add an additional attribute: _id which cannot be changed.

You can also create many documents at the same time.

> db.films.insertMany([{ title: "How to loose a guy in 10 days", url: "https://en.wikipedia.org/wiki/How_to_Lose_a_Guy_in_10_Days" },
{ title: "Father of the Bride", url: "https://www.imdb.com/title/tt0101862/" }])
{
  acknowledged: true,
  insertedIds: {
    '0': ObjectId("626eaae00dd6e8884b86fda4"),
    '1': ObjectId("626eaae00dd6e8884b86fda5")
  }
}

For further options/configuration on inserting documents, please review the docs.

Getting documents

We should now have three documents in our films collection, within the entertainment database. Let's check:

> db.films.find({})
[
  {
    _id: ObjectId("626eaad80dd6e8884b86fda3"),
    title: 'The Big Lebowski',
    url: 'https://en.wikipedia.org/wiki/The_Big_Lebowski'
  },
  {
    _id: ObjectId("626eaae00dd6e8884b86fda4"),
    title: 'How to loose a guy in 10 days',
    url: 'https://en.wikipedia.org/wiki/How_to_Lose_a_Guy_in_10_Days'
  },
  {
    _id: ObjectId("626eaae00dd6e8884b86fda5"),
    title: 'Father of the Bride',
    url: 'https://www.imdb.com/title/tt0101862/'
  }
]

Filtering documents

It's unlikely that we always want to get all of the documents, so let's filter films with the word the in them:

> db.films.find({ title: { $regex: /the/i }})

You can see we have introduced a $regex keyword to provide a SQL LIKE command, if you're use to SQL.

If you know the id of the document:

> db.films.find({ _id: ObjectId("626eaae00dd6e8884b86fda5") })
[
  {
    _id: ObjectId("626eaae00dd6e8884b86fda5"),
    title: 'Father of the Bride',
    url: 'https://www.imdb.com/title/tt0101862/'
  }
]

You can see that we are returning an array of documents. But what if we just want the one document?

> db.films.findOne({ _id: ObjectId("626eaae00dd6e8884b86fda5") })
{
  _id: ObjectId("626eaae00dd6e8884b86fda5"),
  title: 'Father of the Bride',
  url: 'https://www.imdb.com/title/tt0101862/'
}

For further options/configuration on finding documents, please review the docs.

Updating documents

Let's say we want to give a film a rating. We can do this via an update.

db.films.updateOne({ title: "The Big Lebowski" }, { $set: { rating: 5 }})
{
  acknowledged: true,
  insertedId: null,
  matchedCount: 1,
  modifiedCount: 1,
  upsertedCount: 0
}

The benefit of a NoSQL data store is the fact that documents can have varying fields. Let's checkout what the documents look like now:

> db.films.find({})
[
  {
    _id: ObjectId("626eaad80dd6e8884b86fda3"),
    title: 'The Big Lebowski',
    url: 'https://en.wikipedia.org/wiki/The_Big_Lebowski',
    rating: 5
  },
  {
    _id: ObjectId("626eaae00dd6e8884b86fda4"),
    title: 'How to loose a guy in 10 days',
    url: 'https://en.wikipedia.org/wiki/How_to_Lose_a_Guy_in_10_Days'
  },
  {
    _id: ObjectId("626eaae00dd6e8884b86fda5"),
    title: 'Father of the Bride',
    url: 'https://www.imdb.com/title/tt0101862/'
  }
]

You can see that only one film document has a rating field.

For further options/configuration on updating documents, please review the docs.

Deleting documents

Deleting documents is very much akin to finding documents. So let's say we want to delete one document:

> db.films.deleteOne({_id: ObjectId("626eaae00dd6e8884b86fda5")})
{ acknowledged: true, deletedCount: 1 }

For further options/configuration on deleting documents, please review the docs.

Configuration file

We can create a configuration file called ~/.mongoshrc.js to customise the prompt. An example could be:

{
  prompt = () => {
    return `${db.getName()} with ${db.stats().objects} documents> `
  }
}

If you save this file, and then reconnect you should see your prompt as:

entertainment with 2 documents>

For further information on the configuration file, please review the docs.

Tidbits

mongosh has tab completion, which really helps discoverability of commands. If you run db.[tab][tab] you will see a list of commands on the db object. Likewise, if you do db.films.[tab][tab] you will see a list of commands on the collections object.

I hope this provides some useful knowledge to get you through your day using MongoDB. It should provide a grounding of basic commands, and provide links to give you further reading for more advanced commands/options.

Photo by Alfons Morales on Unsplash

Use Docker Context to switch between different solutions

Ben Selby — Wed, 17 Nov 2021 18:58:17 +0000

You may be on the lookout for a replacement to Docker Desktop at the moment. If you are, you may be wanting to trial the different solutions. You're going to need to know what is involved, and what the different cost models include. This post is going to show you how you can use docker context to run the different systems in parallel. The two solutions this post will cover are minikube and multipass.

Getting minikube up and running

This post assumes the host machine is a Mac. Therefore, we can use brew.sh to install the necessary packages.

brew install hyperkit
brew install minikube
brew install docker
brew install docker-compose

Once this is complete, you can start the minikube application.

minikube start --driver hyperkit

This will now be running a Virtual Machine we can use for Docker.

Getting multipass up and running

Let's use brew.sh again to install the necessary packages.

brew install multipass

Multipass allows us to use cloud-init to define a the environment we can use. This will configure the virtual machine for you. The definition I use is defined below.

users:
  - name: ubuntu
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh-authorized-keys:
      - ssh-rsa .....
package_update: true
packages:
  - apt-transport-https
  - avahi-daemon
  - ca-certificates
  - curl
  - docker
  - gnupg
  - lsb-release
runcmd:
  - sudo curl -fsSL https://get.docker.com | sudo bash
  - sudo systemctl enable docker
  - sudo systemctl enable -s HUP ssh
  - sudo groupadd docker
  - sudo usermod -aG docker ubuntu

It is saved to $HOME/git/github/benmatselby/dotfiles/common/multipass-docker.yml. You can save this file where you want. Add the contents of your ~/.ssh/id_rsa.pub file to the ssh-authorized-keys array above. This allows your host machine to ssh into the virtual machine multipass will create.

Now we want to spin up the multipass environment:

cd $HOME/git/github/benmatselby/dotfiles/common/
multipass launch -c 2 -m 2G -d 10G -n docker-multipass 20.04 --cloud-init multipass-docker.yml

Make sure you cd into the directory where you saved the cloud-init script. This command will launch a virtual machine using Ubuntu 20.04, with 10GB of disk space (-d), 2GB of RAM (-m), 2 CPUs (-c), and named docker-multipass. You can change the values as you see fit. You will need the name of the instance for later.

You may also want to mount some local folders to make life a little easier:

multipass mount /Users docker-multipass
multipass mount /Volumes docker-multipass

You can see the information about the VM you created by running:

multipass info docker-multipass
Name:           docker-multipass
State:          Running
IPv4:           192.168.64.5
                172.17.0.1
Release:        Ubuntu 20.04.3 LTS
Image hash:     a83b747df657 (Ubuntu 20.04 LTS)
Load:           0.19 0.14 0.05
Disk usage:     1.8G out of 9.5G
Memory usage:   232.8M out of 1.9G
Mounts:         /Volumes => /Volumes
                    UID map: 501:default
                    GID map: 20:default
                /Users   => /Users
                    UID map: 501:default
                    GID map: 20:default

Using docker context

We now have two environments that can be used instead of Docker Desktop: minikube and multipass. The best way to switch between the two systems is to use docker context. Let's now create a context for multipass, and set it as the default.

docker context create multipass \
  --description "Multipass Docker Desktop" \
  --docker "host=ssh://ubuntu@docker-multipass.local"
docker context use multipass

The name provided in the multipass launch command, is what becomes the ubuntu@[multipass name].local value. So if you named your multipass instance "lebowski" you would use host=ssh://ubuntu@lebowski.local as your --docker context. I used docker-multipass in the examples above.

The last thing we need to do is ssh into into the VM so we can accept the authenticity of the host.

ssh ubuntu@docker-multipass.local

You can now leave the VM, exit.

Now run docker run hello-world. This will run the hello-world container via multipass.

You can see what contexts you have by running:

$ docker context ls
NAME            DESCRIPTION                               DOCKER ENDPOINT
default         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux                                             unix:///Users/Ben/.docker/run/docker.sock
multipass *     Multipass Docker Desktop                  ssh://ubuntu@docker-multipass.local

Let's now add the minikube context. First, we need some data from minikube to build the context:

minikube -p minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.64.10:2376"
export DOCKER_CERT_PATH="/Users/Ben/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

We need this information for the context connection string. Let's run:

docker context create minikube \
  --description "Minikube Docker Desktop" \
  --docker "host=tcp://192.168.64.10:2376,cert=/Users/Ben/.minikube/certs/cert.pem,ca=/Users/Ben/.minikube/certs/ca.pem,key=/Users/Ben/.minikube/certs/key.pem"

You can see how we have built up the --docker string by using the environment variables provided by the minikube command. Let's now check the contexts we have:

docker context ls
NAME            DESCRIPTION                               DOCKER ENDPOINT
default         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux                                             unix:///Users/Ben/.docker/run/docker.soc
minikube        Minikube Docker Desktop                   tcp://192.168.64.10:2376
multipass *     Multipass Docker Desktop                  ssh://ubuntu@docker.local

We can then use docker context use [name] to switch between the contexts. Let's switch to minikube and run hello-world.

docker context use minikube
docker run hello-world

You have now run the hello-world container using both docker contexts.

Conclusion

Docker Desktop is an easy to use system for the Mac. For personal projects, it is great. With the new licensing agreement, businesses may need to review how their engineers run containers during development. This post showed how you can use docker context to switch between different back ends to help trial alternatives to Docker Desktop.

Photo by Alexander Schimmeck on Unsplash

DEV Community: Ben Selby

Two weeks with the GitHub Coding Agent

How we did it

What worked

What didn’t work

Next experiments

Tips

Dependency Management

What is a project dependency?

So why do we need to keep dependencies up to date?

How to keep the dependencies up to date?

Summary

See also

Stop tracking changes to certain files in Git

Assume unchanged

Summary

See also

Working with Zed for a week

Projects worked on:

No debugger

It's fast

Collaboration UI

Git integration

Keymaps

Outline view

Unused variables

Copilot

It seems to "Just work (tm)"

Summary

See also

Show your working out

1 - Just the answer 🙂

2 - The answer and visuals 😊😊

3 - The answer, the visuals, and links to the original data source 😊👑🧠

Summary

A case for using a Makefile in your project

What is a Makefile?

Abstraction

Example

Automation

Summary

See also

Automating the startup of a dev workflow

See also

Setting up k9s skins for different Kubernetes clusters

How useful are code coverage reports?

A story

Tools

Metrics

Importance

Summary

See also

Debugging Python with a virtual environment in VSCode

Mongo shell commands to get you through the day

Connecting

Listing the databases

Creating a database

Deleting a database

Creating a collection

Showing collections

Inserting documents

Getting documents

Filtering documents

Updating documents

Deleting documents

Configuration file

Tidbits

See also

Use Docker Context to switch between different solutions

Getting minikube up and running

Getting multipass up and running

Using docker context

Conclusion

See also