DEV Community: James Wade

Measuring performance in agile

James Wade — Mon, 12 Sep 2022 12:24:17 +0000

It seems to be a common theme that after a while of implementing agile, management wants to measure performance.

Should you use story points as a performance measurement?

Management thinker Peter Drucker is often quoted as saying that "you can't manage what you can't measure." Drucker means that you can't know whether or not you are successful unless success is defined and tracked.

Business people want estimates. They want to know how much it’s going to cost them to get a solution, and they want to know how likely it is to come in on time and on budget. And of course, quality is not negotiable.

Estimating work

In agile and scrum teams, we give stories points as a way to estimate the effort involved in a piece of work.

Unfortunately humans aren’t very good at accurately estimating how long something will take.

Story points do not equate to time exactly. They are based on effort, complexity and doubt.

Complexity is the "stuff we have to figure out." We know we can solve the problem, and we probably have a decent feel for how we'll approach it, but we still have to figure it out.

Effort is the sheer amount of stuff that needs done. For me, that example is configuring SharePoint lists, because I knew exactly how to solve everything, and I knew how many there were, but it still took time to run through them.

Doubt is about the stuff we don't actually know if it can be done. We may suspect we're on the wrong track, that the technology isn't up to it, or some other factor that would cause us to churn for a while before we figure out if we can actually do the work.

Arguably it’s all effort, and by removing complexity and doubt, you’re able to give a better estimate of the effort involved in a story.

As humans, we are good at comparing stuff. Pointing gives a sense of relative size, giving us a sense of the difference between a house and a skyscraper, we should be less concerned about 12 floors vs 14 floors.

Mike Cohn summarises story points best: “Story points are a unit of measure for expressing the overall size of a user story, feature or other piece of work.” Story points tell us how big a story is, relative to others, either in terms of size or complexity. Mike often refers to “dog points” when helping teams understand the concept of relative sizing.

A 2-point (small) dog would be a Chihuahua. A 13-point (big) dog would be a Great Dane. With those two guides in mind, it’s fairly easy to size the other dog breeds relative to a Chihuahua or Great Dane. A Beagle, which is about twice as big as a Chihuahua, might be a 5. A Labrador, which is bigger than a Beagle but smaller than a Great Dane, might be an 8.

So how do you get developers to get more done? How do you go faster?

Getting More Done

I often hear management people say things like:

“We need to get the developers doing more story points” or
“How can we get developers to do more work?” or
“Can we use story points to measure an individual’s performance?”

I get it. We want to go faster. We want to understand how to get more work done.

You might think that's simple, just put more work in and more work will get done, right? Unfortunately, of course, it’s not that simple.

Nobody would expect to put more work on the existing team and expect things to get better. So how can we get the outcomes we want?

Let’s say we want a developer to do 13 points per day. Of course, this can be achieved when a story is a developer just writing code at 100% capacity. However, when a developer is always busy, all of the time, we find that the wait time becomes longer. The effect of this is what the Japanese call Mura or 'waste' in the manufacturing process.

To explain this simply, I will use the analogy of a paper shredder:

They have a maximum limit. If you put too much in, they will jam up. If you constantly feed in the maximum or more, they will overheat or worse, break.
They can continually sustain a reasonable number at a time, at a reasonable rate, no problem.

Ultimately, in both paper shredders and developers, you need to control the flow, you need to limit the work in progress.

As we know, it's not just about writing code, it has no business value until it's in production. If it gets stuck in the process, it won't get into production or the developer will end up waiting for the business operations to become ready.

Everyone needs idle time or slack time. If there's no slack time, work will get stuck in the process. Any more than 80% capacity will result in longer wait times before things get processed.

For example, let’s assume each developer can do 13 points in a two week sprint:

100%	90%	80%	70%	60%	50%
13	11.7	10.4	9.1	7.8	6.5

If we imagine that a two-pizza sized team (let’s say 6 people) can deliver a feature per two week sprint, and each feature is about 20-40 points of effort, and each feature is broken down into stories of 3-8 points, then you would expect the team to deliver 10-20 stories of business value into production per sprint, providing you’ve sufficiently broken down the work.

Just measuring story points and expect things to get better isn’t going to be enough. Work needs to be broken down. When work is broken up into small bite sized chunks, it can get through the process quicker. For example, one 13 point story done, but not in production does not provide value to the customer compared to one 5 point story complete and in production delivering value.

If it’s not simply a case of putting more work in to get more work out, what is it?

Culture

It’s really easy to destroy the culture of an agile team with metrics. We need to be sure that what we measure encourages the right behaviour.

Using a team’s velocity as a performance measurement comes with a strong warning label:

“Scrum’s team-level velocity measure is not all that meaningful outside of the context of a particular team. Managers should never attempt to compare velocities of different teams or aggregate estimates across teams.

Unfortunately, we have seen team velocity used as a measure to compare productivity between teams, a task for which it is neither designed nor suited. Such an approach may lead teams to “game” the metric, and even to stop collaborating effectively with each other.

In any case, it doesn’t matter how many stories we complete if we don’t achieve the business outcomes we set out to achieve in the form of program-level target conditions”

We’ve all heard about working smarter, not harder, yet by focusing on story points as a measurement, we find that although in the short term we will succeed at getting people to complete more story points by simply working harder, this approach will not necessarily achieve the outcomes that we want.

People will always find a way to “work smarter”, however, we find that by focusing on story points it encourages the wrong behaviour over time.

“Estimate inflation is when the estimate assigned to a product backlog item (usually a user story) increases over time. For example, today the team estimates something will take five points, but previously they would have called it three points.” - Mike Cohn, Agile Alliance and Scrum Alliance founding member

We need to ensure that we encourage the correct behaviours, working smarter to achieve the outcomes that we need, rather than working smarter to game the system to achieve more story points.

“Putting too much emphasis and concern on velocity will often result in gaming the system through the use of velocity inflation” - Tom Smallwood, Agile Coach, Consultant at Smallwood Software Solutions, Inc

I’ve witnessed it first hand. If you aim to increase velocity the team will look to provide bigger estimates to make them look better or worse chasing points. It means you spend more time arguing over estimates, repointing, and administration to chase every single point. And worse, once these behaviours take their hold, it becomes difficult to get people to break out of it.

“Estimation is a waste. Estimation of tasks in hours is utter waste, people spend hours debating minutia, when they would better off just starting” - Certified Scrum Trainer Mark Levison

This doesn’t mean that we shouldn’t estimate at all. It means estimation, as estimating with story points is relative, it’s only really useful to the people who are making the estimates. If the team finds it useful to estimate, then you should do it. It should be used as a tool to help understand what is involved in a volume of work and how to make it doable.

Chasing points also discourages the right behaviour. Once people understand that their individual performance will be based on how many story points they complete, it simply becomes a case of just doing their work, then slinging it over the fence, to testing and/or ops to do their bit. You end up with silos and no collaboration.

It shouldn’t be about story points. Story points are simply a tool for a team to use to help them understand the effort involved in a piece of work. If we want to encourage the right behaviours, we have to ensure that whatever we measure will encourage the right behaviours.

I remember Dan North explaining a story where a company was measuring story points and had concluded that the solution was to fire the person who wasn’t completing any story points. Dan says that he said this would be a terrible idea as this particular developer was acting like a multiplier, helping the other developers in the team to go faster.

We want to encourage people to work together as a team. We want to encourage people to work smarter, not just game the system. We want people to be effective at delivering value, not just good at scoring points.

Measuring “work done” is misleading. How much work we’ve done pales in comparison to actually delivering the right things.

“Efficiency is doing things right; effectiveness is doing the right things” - Peter Drucker, founder of modern management

So how do you measure the team's effectiveness and nurture the right culture? Ask the team.

“Imagine that your new boss allows you to choose which metrics will be used to assess your individual performance in your role as a senior developer. Which metrics would you suggest?”

I asked this question to a team of engineers (developers and testers) and discovered that they already knew which metrics were the most important.

They voted the most important metrics as: the ability to deliver on schedule/within budget; customer satisfaction ratings for the product/service; and revenue performance of the product/service. A few people valued these metrics: Number of bugs found; peers’/teammates’ rating of my performance; performance of product/service against industry benchmarks.

These metrics only got one vote each: Frequency of major product revisions/releases; my own self-rating of my performance; number of hours worked/billed. However, nobody thought that these metrics were important: Frequency of check-ins/commits; manager’s rating of my performance; number of lines of code written.

From this, I discovered that it’s important that we understand what metrics we can use to measure effectiveness, but also ask the team what they think is important so we know where the knowledge gaps are, so that we are aligned and can focus on what we know matters.

Predictability

In an agile team, a key health metric is predictability, and by improving our ways of working we can become more predictable.

For example, If you’re being asked to “burn down” or “burn up” stories, then this signals that stories are taking too much time. They are spending too much time in progress because they are incorrectly estimated or simply too big. We should spend more time breaking down and/or unpacking the stories.

A manual/artificial burn-down is a bit of an anti-pattern. Stories should be small enough, made smaller or sub tasked. If we aim for consistency, then we should end up with stories of roughly the same size.

We should be aiming to improve the value curve by completing stories, rather than artificially burning down based on a guess.

Don't get bogged down in the implementation detail. Don't point to the best case scenario, be realistic, what would be worse case be and work from there.

Pointing should give you a sense of relative size, for example, we know the difference between a house and a skyscraper, we’re less concerned about 12 floors vs 14 floors.

Continuous Improvement

The agile manifesto explains “The team reflects on what happened in the iteration and identifies actions for improvement going forward”.

In Scrum, a retrospective touches on all three pillars of Scrum (Inspect, Adapt & Transparency) and is a key component that insights reflection and progressive improvement of processes and working agreements.

In Lean, it’s described as Continuous improvement, or Kaizen, a method for identifying opportunities for streamlining work and reducing waste.

At Spotify they introduced a “health check model” to help the teams self organise and identify areas to focus on, such as support, teamwork, control, mission, health of codebase, suitable process, delivering value, learning, speed of development, ease to release and fun.

All of these techniques and methodologies talk about the same thing, continuous improvement.

The Health and Safety Executive (HSE) and the Chartered Institute of Personnel and Development (CIPD) talk about a healthy team being empowered and able to review processes to see if work can be improved.

Retrospectives provide useful feedback and the health check model does give you a sense of whether things are getting better or worse. This puts people at the centre of what we do and puts them in control of their own improvements.

“Remember, outcomes are what matters - not the process, not controls, or, for that matter, what work you complete”, The Project Phoenix, 2013

Although some might say it’s about how much work you get done and others think it’s how well the process goes, we can all agree that it’s outcomes that matter. Not just building the right way, but building the right thing. We need to ask the right questions. What do your users think? What are you working on bringing value to the company?

“The key to creating a lean enterprise is to enable those doing the work to solve their customers’ problems in a way that is aligned with the strategy of the wider organisation” - Jez Humble, Vice President of Chef

If you really want to improve the performance of an agile team, then measuring story points isn’t going to encourage the correct behaviours. Instead by focusing on outcomes and asking the right questions:

How do we know right now that what we're working on right now is the most important thing?
What goals, objectives and measurements for this year? Who is accountable?
Which measure does the business care most about?
Which products/projects relate to which business goals and measurements?
Who is accountable for each product?

Fortunately the agile methodology, SCRUM framework, Lean Enterprise and the DevOps movement all encourage the same behaviours, because they are founded on the same principles. Your application is only as good as your team. If your team follows these principles, then you will get quality.

Quality

Now we know that it’s not about story points or doing more work, but instead doing the right work by focusing on the right thing. By understanding that through continuous improvement and focusing on the outcomes we want, we can reach our goals. We can really begin to think about how we measure our success.

“The paradox is that when managers focus on productivity, long-term improvements are rarely made. On the other hand, when managers focus on quality, productivity improves continuously” - John Seddon

From my experience, most developers when we talk about quality go straight to talking about the quality of code or quality of the process, or such. However, my view is that you could have the best code in the world that does absolutely nothing. So what is quality?

Most of the time when we talk about quality we think about testing. Quality assurance shouldn’t just be about testing. It’s not just about making sure we are building the product correctly, it’s about making sure we are building the correct product.

We learn that testing is not supposed to be something that is just tacked on at the end, you have to build quality in from the beginning. We begin to think about the process, and the standards, knowledge and tools we use to improve the process from the beginning.

We think about how to measure the success of our processes. How fast our builds run, how long our tests take, how many bugs we have, how many fatal errors there are in production, what our availability is, our time to recovery, how performant our application platform is, how much test coverage our code. However, this is how you measure the quality of a process.

It’s not about how many bugs we’ve found, how fast or often we deploy, how long work takes, the quality of our code. Sure all of these things help improve processes, however these only measure the quality of the process, they don’t measure the quality of the application.

How do you measure the quality of a product? Quality is hard to objectify, it’s not a specific thing, you can’t point at it and say that’s quality, or can you?

I’ve always had a fascination with process, ever since I learned about the success of Henry Ford and the production line. It’s interesting then, that when we talk about quality, we look to Japan, we talk about Kaizen, the Japanese word for “continual improvement”, and in software development, we talk about “The Toyota Way”. Are Ford and Toyota quality brands?

When I think of quality, I think of contemporary brands like Apple or popular car brands like BMW. They build quality products.

“Apple’s market share is bigger than BMW’s or Mercedes’ or Porsche’s in the automotive market. What’s wrong with being BMW or Mercedes?” - Steve Jobs, 2004.

We don’t talk about quality in terms of processes, it’s the end result that’s important. Quality then, is something you measure against another thing of a similar kind. It is knowing what the standard is and making it better.

Remember, outcomes are what matters - not the process, not controls, or, for that matter, what work you complete.

Commitment

Unlike Kanban, Scrum-style “sprints” promotes “commitment-driven planning” rather than “velocity-driven planning”.

The agile methodology says nothing about velocity or commitment, the SCRUM framework however is quite prescriptive about it.

Scrum demands a full-team commitment: If you’re behind, I’ll help, and I know you’ll do the same for me. It’s not “these are my tasks” and “those are yours.” - Mike Cohn

Using a commitment-driven method means that the team is able to function as a team, be autonomous and continuously improve.

Velocity is variable, it does go up and down, so it only really becomes useful in the long term as a rough guideline. It may eventually become useful for a long term team on a long running product, but this is rarely the case.

Just like trying to be accurate with estimates, you can fall into the trap of becoming too scientific about velocity - you don’t need to be.

“When a measure becomes a target, it ceases to be a good measure.” - Goodhart's Law

The problem I find with velocity-driven planning is that it makes it about story points again, rather than a commitment to the outcomes we want.

Although the agile methodology doesn’t say anything about velocity or commitment, it does say “Projects are built around motivated individuals, who should be trusted”.

We really need to trust our teams of motivated individuals to commit to building the product, rather than prescribing a velocity.

I can understand that trust might be an issue for some teams, so I would suggest that you build trust by using measurements that matter to help to demonstrate the team’s performance.

Progress

In the agile methodology it says “working software is the primary measure of progress”.

It’s interesting because it puts a big emphasis on working software being a priority but progress being the goal. I therefore think it’s really worth thinking about “progress” being a measurement and how you can best demonstrate that.

“Release early, release often and listen to your customer” - The Cathedral and the Bazaar, Eric Raymond

In a traditional “waterfall” approach, it would be quite difficult to demonstrate or even measure progress, however in this scenario, you would be able to get feedback from the customer as soon as the software is working.

In order to truly honour the principle set out in the agile methodology, to show progress you need to release early and release often so that you can demonstrate working software to the customer to get their feedback.

It must be true that, in order to demonstrate working software, it must be of a quality where it does not fail. We must have some measures in place to ensure it is working. We need to know that our code is of good quality, there are no bugs that prevent it from working, the pipeline does not fail to build. Fixing issues then becomes a top priority.

We also need to measure progress, so how long does a unit of work take to go from conception to inception where the user can use it - this is known as “lead time”. But what about how long it takes a developer? That’s cycle time. Build time is another good one. Anything we can measure to demonstrate that we’re able to progress quicker through work is a good metric.

Measuring Success

We need a way to know if things are getting better or worse, we know that working software is the primary measure of progress.

We need to ensure that we understand the business metrics used to measure the success or failure rate of our platform.

Lead Time

Lead time tells us if we have an efficient process

In software development, one metric that matters is lead time. If our lead time is good, then we know we have an efficient process. If anything affects our lead time, then we need to address those things at the root, but we can't improve what we can't measure. If development is taking too long, this will highlight that.

Working Software

“Working software is the primary measure of progress” - Agile Manifesto, 2001

Have we built the right thing, the right way? We need to know what our customers think.

You could have the best continuous integration pipeline, the best code and 100% test coverage, but if the software doesn’t do what customers want then it’s not of quality. Product quality is relative to the customer's expectations.

By having strong customer feedback loops from the beginning, we're able to understand the business impact sooner.

Team Health

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Team health checks help us to identify areas to improve on and give us a good idea of if things are getting better or worse. It means we have to work together to be successful.

I found that lots of people have experimented with ways of measuring and visualising how their teams were doing, often by showing progression through various levels called a “maturity model”.

“A maturity model is a tool that helps people assess the current effectiveness of a person or group and supports figuring out what capabilities they need to acquire next in order to improve their performance” - Martin Fowler

Maturity models are great, but they can easily be misused and sound a bit patronising. Instead I wanted something that would help the team understand where they felt they were and what areas they needed to improve on.

I found that retrospectives provide useful feedback and the health check model does give you a sense of whether things are getting better or worse, but it doesn’t really help with line management.

One to one meetings are proven to be a great way to maintain good, open communication, and continue to build a relationship. It’s important to meet on a regular basis to maintain alignment. Setting objectives helps to bring focus and alignment.

The stronger alignment there is, the more autonomy will be granted.

The biggest challenge with bringing alignment is that it requires a clear mission, objectives, strategy and tactics that everyone can agree to and buy in to. Without this, you find that the left hand is digging a tunnel while the right is building a bridge.

However, one to one's and objectives setting is not indicative of the culture of a team.

I was asked what role I will play and how would I measure my success. So how would I measure success?

As we know, success should be defined and tracked. We need to have a sense of if things are working (or not).

So let’s look at that: “Engagement through staff feedback” Are we engaged in what we do? Are we measuring the right things? Well. What do our staff say? We need to ask them…

What I need is a way to know whether our culture is getting better or worse. Culture is intangible and hard to change, but it can be measured.

A model created by sociologist Ron Westrum suggests that the biggest predictor of job satisfaction is how effectively organisations process information. You can start to measure this at a team level by asking the following questions to the team members on a quarterly basis:

Rate how strongly you agree (7) or disagree (1) to the following statements:

On my team, information is actively sought.
On my team, failures are learning opportunities, and messengers of them are not punished.
On my team, responsibilities are shared.
On my team, cross-functional collaboration is encouraged and rewarded.
On my team, failure causes enquiry.
On my team, new ideas are welcomed.

Alternatively the Gallup Q12 is said to be a good way to measure engagement.

However, it doesn’t need to be complicated, a simple dot-voted “pulse” on a scale of 1 to 10 taken at retrospective each sprint is quick and easy, and is a simple way to get started to give you a gauge of how a team is doing.

Conclusion

One of the biggest traps I see teams going through an "agile transformation" do is obsess over story points, burn down charts and velocity and I think this is a mistake.

Transforming a team from a traditional waterfall approach to an agile approach takes more than just understanding velocity, it requires a complete change in culture, one of continuous improvement and psychological safety.

Remember what it is you're trying to achieve and measure what really matters, focus on the outcomes rather than story points.

Don't write code for computers

James Wade — Fri, 09 Apr 2021 08:22:04 +0000

When trying to keep code simple, you may find yourself torn as to what is more simple, less code or more verbose code?

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand” - Martin Fowler

The problem

Often we attempt to be explicit by adding more documentation or comments, however this doesn’t always end up the way we hope. The comments end up implying what the code will do, rather than the code being self documenting and explicit.

“Redundant comments are just places to collect lies and misinformation.” ― Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship

Comments become redundant because they often differ from the code itself, especially over time. The more complex the code, often the more comments there are and the more it is likely to differ over time.

You have to put yourself in the shoes of the next person to read your code. They are unlikely to read your comment, less understand it, so will simply read what the code is doing instead and then change the code to whatever they need it to do, without updating the comments.

Docblocks are different to comments and often a place to put annotations, but even those where possible are favourable to put into the actual language, than in the docblock. Although docblocks can be used to describe methods, they shouldn’t describe the logic itself, the code should do that.

The solution

Rather than writing comments, see if you can refactor the code to make it more obvious. Break out complex logic into a new method. Use variable names to give context to its value. Avoid magic numbers and strings, instead use well defined constants.

It may not feel like writing mode code is the most pragmatic approach as you will write more code, however it will save you time in the long run and you won't need to refactor as much.

When you're explicit upfront, you'll find it much easier to write tests, debug and ultimately maintain rather than an implicit approach which leads to ambiguity and uncertainty.

“When you feel the need to write a comment, first try to refactor the code so that any comment becomes superfluous” - Martin Fowler

If you find yourself in a situation where you can choose whether to go with strict typing, you should because it means you can be explicit.

In 1974, Liskov and Zilles defined a strongly-typed language as one in which "whenever an object is passed from a calling function to a called function, its type must be compatible with the type declared in the called function."

Being explicit also puts less burden on the resources as it doesn't have to guess what you mean.

Similarly to keeping it simple, when you're thinking “clever” or “smart” by using implicit methods, see if you can opt for idiot proof and explicit method instead.

First, if the code is clear, and uses good type names and variable names, it should explain itself. Second, comments aren't checked by the compiler, so there is no guarantee they're right, especially after the code is modified. A misleading comment can be very confusing. Third, the issue of typography: comments clutter code. - Rob Pike, February 21, 1989

It's good to be explicit when it comes to standards too. Having implicit standards will soon get lost as new people begin to put their own spin on the code. That's why it's best to choose and define the coding standards and stick to that. It will save a lot of time.

Have you had experience with comments vs code? Do you have any examples? I'd love to hear about them in the comments.

Who’s bug is it anyway? Aka handling the on-call rota

James Wade — Fri, 21 Aug 2020 16:56:05 +0000

Being the person responsible for dealing with issues as they come in (aka on-call) is probably one of the most disliked aspects of the job, with feedback ranging from “it sucks” to “it’s alright”, so you know it’s always a challenging topic of conversation.

I’ve spoken to a few people about on-call, and the general feedback is that it doesn’t work for them. From the feedback, complaints ranged from, the rotas aren’t right to, not being paid enough for the volume of issues especially out of hours calls, to, people don’t know how to fix all the problems so don’t want to get involved.

The reality is, you won’t be able to solve these problems head on without compromise. Instead, it’s best to accept this and optimise for what will encourage the right behaviours.

Encouraging the right behaviours

The burden of issues should not be on one individual or a siloed team. Supporting products, services, and features should belong to the teams that build them.

Teams need both the autonomy to release and operate new products and features and the responsibility for supporting them. We shouldn’t treat support any differently than application configurations or infrastructure. It’s a requirement.

Developers should be responsible for responding to issues, outages and being on call, as well as designing and evolving the monitoring and alerting systems, and the metrics behind them. This means that the developers control the entire lifecycle, including operational tasks like backup, monitoring and, of course, on call rotation.

This way, they become more aware of the pitfalls of running code in production as they are the ones handling the incidents (waking up to alerts, etc).

This approach means that they are incentivised to put sufficient measures in place to avoid it happening. Things like monitoring (metrics and alerts), logging, maintenance (updating and repairing) and scalability become key considerations behind every line of code they write.

In the event of an incident that touches multiple teams, the issue is manually escalated to an Incident Manager, typically a Department Head or nominated representative, who would remain involved until the incident is addressed. We then utilise the root cause analysis approach to a blameless post mortem, to diagnose and address the problem.

I appreciate that this approach might not be popular with everyone and that people may be sceptical but this is a tried and tested option that should be considered.

What does not work

Let's face it, nobody likes being on-call. In the feedback, nobody said anything good about it, on a scale of OK to it sucks, you can guess where most people landed.

By attempting to manage on-call rotas at management level we set ourselves up for failure as it doesn't empower the people on-call, it requires continual intervention and will not scale. Sounds great in theory, but can it work in practice? Encouraging the right behaviours would only be theoretical if it wasn't feasible, let me explain...

One option to encourage the right behaviour would be to pay more for on-call duty. The problem is that this is favoured in only one way. That is, the developer gets paid for issues. This is not encouraging.

An issue not addressed by many rotas is that the switchover at midnight isn't practical, although sunrise is suggested instead, it doesn't solve the actual problem.

Two issues that most people are concerned about are that they feel that they don't know enough to solve the problems or it doesn't fit with their lifestyle. So how do we solve those issues?

What happens elsewhere

At Google, they hire dedicated Site Reliability Engineers. Once products or services become stable they are handed over to them to manage. This is a very practical option but it is probably more expensive and can lead to developers simply throwing work over the wall to operations, rather than taking responsibility. It can also cause an “us” and “them” type culture, which is not great for culture.

At Spotify they tried this and found that it led to a bottleneck in their Ops team. They instead opted for a redistribution of ownership. Now each team is responsible for support.

At UK government digital services, they too had a dedicated technical support team but found that in practice it didn't work. One of the problems was knowledge sharing. They have worked really hard to try and solve this problem but admit that it's still work in progress.

Netflix is probably most famed for their approach of "NoOps" to this problem. An engineer will generally have pagerduty 1 out of N weeks where N is the number of engineers. Some teams include QA engineers and managers in the on-call rotation and others don't. Some teams only include key engineers on pagerduty rotation while others will include all engineers on the team.

At Etsy, all their engineers are on call too. They have created an open source tool called Opsweekly that allows you to both organise your team into one central place, but also helps you understand and improve your on call rotations through the use of a simple on call "survey", and reporting as a result of that tracking.

Meanwhile at Facebook, two or three times a year, Facebook engineers have to completely rearrange their lives for two weeks while they serve something called "oncall duty." Basically, during that time, they are responsible for keeping the service up-and-running. Oncall duty lasts for two weeks and it rotates between Facebook's engineering teams. a Facebook engineer named Keith Adams says it is "the worst thing about working at Facebook."

At Amazon, it's less clear, they seem to rotate on-call to teams every few months for a week at a time and this approach isn't very popular from what I've read.

How to solve the problem

We want to reduce the burden on the person in the team with the most Site Reliability Engineering experience and encourage more engineers on the rota.

All of the platforms I mentioned seem to have one thing in common, that is on-call is a mandatory part of the job.

You'll notice another theme is that on-call is assigned at team level and the teams take responsibility for choosing their rota.

My view is that every engineer is responsible for support and monitoring including on-call and that rotas are coordinated at team level. My suggestion would be to rotate them every two weeks across teams in line with a typical sprint length.

If it’s not practical to get everyone on-call, then you've really got to question why it’s not practical. If you’re not addressing the underlying issues head on, then any alternative solution is just a compromise and is going to be more expensive and will be to the detriment of the team and culture.

If it’s not practical to get everyone on-call, then you will need more budget and resources to cope with the same recurring issues.

Return early

James Wade — Fri, 13 Mar 2020 09:39:31 +0000

All projects should take the same approach and stick to one set of Coding Standards. This solves many common problems, especially when projects which have more than one developer.

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand” - Martin Fowler

There are some things that are not covered by Coding Standards which are mostly subject of preference and not directly related to format of the code.

While, Coding Standards are strict guidelines that we should be stick to, relating to the way code is written and its readability, Best Practices are recommendations that may help with security, performance, or architectural patterns that are solutions to a commonly occurring problems.

Often these best practices are left solely on the developer to decide. One such best practice is “return early”.

This is another in a series of Best Practices that have served me well over the years and have stood the test of time, ones that I use or refer to on a regular basis.

What is an early return?

An early return, or “return early” is an approach to keep readability in functions and methods.

It is always considered a wise choice to return early if simple conditions apply that can be checked at the beginning of a method.

Why return early?

The motivation behind this is to avoid nesting too deeply and return early instead.

Too many if-else statements can make your code hard to follow. Explicit is better than implicit.

It's better to return early, keeping indentation and brain power needed to follow the code low.

Example

Here’s an example.

Bad: Not returned early

To keep readability in functions and methods, it is wise to return early if simple conditions apply that can be checked at the beginning of a method.

<?php

function foo($bar, $baz)
{
    if ($foo) {
        // method logic goes here
        return $calculated_value;
    } else {
        return null;
    }
}
?>

Good: Returned early

It's better to return early, keeping indentation and brain power needed to follow the code low.

<?php

function foo($bar, $baz)
{
    if (!$foo) {
        return null;
    }
   // method logic goes here
    return $calculated_value;
}
?>

Why wouldn’t you return early?

Functions that exit at multiple places using return statements may be confusing and can be hard to refactor.

The exception to this is usually when using early returns for type validation or an input mismatch.

When trying to keep it simple, you may find yourself torn as to what is more simple. Less code or more verbose code? Do whichever makes it easier to maintain.

What if we don't have time? We often find ourselves compromising on quality in order to deliver value sooner, this is fine initially but over time it becomes much more difficult to maintain.

Conclusion

There’s usually good reason to return early and rarely an exception.

My view is that returning early is a best practice that you should adopt as a style guideline. I tend to recommend it as I find it makes the code much easier to read and keeps methods short.

In summary, anything that helps to make your code more human readable is a good thing, including returning early.

Resources

Be bold

James Wade — Fri, 07 Feb 2020 09:18:39 +0000

Are you bold enough?

We can all be a little bolder sometimes.

Being bold is often what holds us back. As children we’re taught to “look before you leap”, which is great as a child, yet risk taking is an important part of learning.

I'm pretty pragmatic, however sometimes being sensible and realistic isn't bold enough, and find myself assessing risk when it probably isn't necessary.

“Be bold” is a term I picked up on as an editor on Wikipedia. I enjoyed learning about various subjects and began improving the entries as I learned more about the subject matters.

But it's more than just a style guide for Wikipedia, it's a style guide you can adopt for everyday life, a philosophy.

The Wikipedia entry for “Be bold” starts off by explaining that you should “go for it”. If you want to change something, do it. Less talk, more do. If someone doesn’t like it, it can be easily undone.

Rather than embarking in an edit war, you’re encouraged to follow the “Bold, revert, discuss” cycle, which encourages disagreements to turn into productive discussions.

In software engineering, we can attribute this type of approach to Grace Hopper, who was the developer of the first compiler for a computer programming language, by asking for forgiveness rather than permission.

"It's easier to ask forgiveness than it is to get permission" -- Grace Hopper, U.S. Navy's Chips Ahoy magazine (July 1986).

If you think it's a good idea, then do it. It is much easier to apologise later if it doesn't work out than it is to get permission before you do it.

In leadership, we should seek success through bold action and risk-taking, not security through the status quo.

Life is like a wild tiger.
You can either lie down
and let it lay its paw upon your head
Or sit on its back and ride it.
Ride the wild tiger.

Are you bold enough to ride the wild tiger?

Rule of three

James Wade — Tue, 10 Dec 2019 15:08:46 +0000

Sometimes called “1, 2, refactor”, the “rule of three” is code refactoring rule of thumb to decide when a replicated piece of code should be replaced by a new procedure. It states that the code can be copied once, but that when the same code is used three times, it should be extracted into a new procedure.

“Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior”
― Martin Fowler

This is a rule that has really served me well in recent years when I’ve come up against challenges on when something should be refactored.

As developers it’s all too easy to spend too much time trying to engineer perfection, when really, all we needed to do was what was asked of us.

We also have to recognise that we can’t always plan for the future and by refactoring or abstracting code too early, you may find that you back yourself into a corner which results in, you guessed it, two versions of the code, each with slight differences to accommodate the tight coupling.

It really only makes sense to defer the decision to refactor once you know a bit more about what it is you’re trying to do.

After all, you can’t plan for what you don’t know. You can only deal with the information that you have right now, so we should programme for that.

“Whenever I have to think to understand what the code is doing, I ask myself if I can refactor the code to make that understanding more immediately apparent.”
― Martin Fowler

This is, being a rule of thumb means that there are exceptions to this rule, there are good times to refactor as well as bad. This rule prevents you refactoring before you have enough information.

If you find yourself in a situation where you’re asking whether it’s too early to refactor, it probably makes sense to apply this rule.

Exceptions are meant to be exceptional

James Wade — Wed, 04 Dec 2019 14:24:52 +0000

Now and again I come across patterns that I see in codebases and pull requests, especially when the project or team is maturing. This covers one of them.

I've found that developers that have picked up more advanced patterns and become familiar with a codebase that is full of compromises made during it’s organic growth, they begin to apply those same patterns to solve the problem with a broad stroke, sometimes at the risk of not fixing the underlying issue.

One such pattern is try-catch. Once the power of a try-catch is learned, it starts being used everywhere. I’ve seen the whole contents of a method wrapped in a try-catch, catching anything it can. When all you have is a hammer, everything looks like a nail.

When you ask why they have used the try-catch, the answer is usually that they are trying to avoid the application producing unexpected errors. You might ask, "Why don't you wrap the whole application in a try-catch?", though it could be mistaken as flippant, it does promote the right thinking.

Uncle Bob explains that if you’re having to write in exception handling in your method, then you’re probably doing it wrong. Exceptions are meant to be exceptional.

Error handling is “one thing”.
— Uncle Bob Martin (@unclebobmartin) June 18, 2018

Similarly, catching an exception to then just report it back to the user isn’t great because you’re treating it as if it’s expected behaviour, when it isn’t. Exceptions should be exceptional.

Catching an exception only to report it is a sure overkill. Simply because uncaught exception is a fatal error already, and it will be reported by itself. Without that blunt try/catch/die sequence, making your code much cleaner.

Sadly, the PHP manual is especially bad at it, showing such examples all over the place. On the one hand, it is understandable as such an example is guaranteed to output an error message, regardless of the user settings. On the other hand, this when this approach gets mindlessly copy-pasted into the live code, it turns out to be superfluous, user-unfriendly and harmful.
https://phpdelusions.net/top#try_catch

Catching all “exceptions” or “catchables” makes it really hard to debug what’s going wrong. Avoid catching all “exceptions” or “catchables”, instead use exception handlers. This will maintain that exceptions are exceptional and handled exceptionally, and makes the application easier to understand and debug in the event of an issue.

If you’re validating some data, you usually shouldn’t be using exceptions to signal validation failures, if you’re expecting the failure, then perhaps it’s not an exception. Exceptions are exceptional.

“if a failure is expected behavior, then you shouldn't be using exceptions”
https://www.martinfowler.com/articles/replaceThrowWithNotification.html

You shouldn’t need to catch all exceptions. You’d have to have an exceptional reason to catch an exception, so use try-catch sparingly. Next time you see a try-catch, think, how can I refactor this so that the try-catch becomes redundant. Remember, exceptions are meant to be exceptional.

Exceptions come at a cost, and they are expensive. They have a performance impact on the software and a maintenance overhead.

Is no QA the way to go?

James Wade — Fri, 22 Nov 2019 11:47:09 +0000

There's no getting away from it, quality is a whole team responsibility. If you're aiming for Continuous Delivery, then you'll recognise one of the core principles of Continuous Delivery is to "Build quality in".

If you've heard of lean development, then you will no doubt have heard of the principle of "The Toyota Production System" of “building quality into” software.

It's inevitable then, that there will be a tension between filling that "QA" role and building a team of "T-Shaped people" who treat quality as a first class citizen.

Put quality first

“Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” - Dr. W. Edwards Deming

Quality is not something we should just tack on the end, it should help us to "shift left" – unfortunately, that’s often exactly where we seem to identify that support is needed – at the end, making sure the developers have met the acceptance criteria and haven't introduced any bugs. You have to ask, why?

To get new features to market quickly, we often trade off quality for higher velocity. This is a sensible and rational decision. But at some point, the complexity of our systems becomes a limiting factor on our ability to deliver new work, and we hit a brick wall.

When exploring, there is a tension between the need to experiment by building MVPs, and building at high levels of quality through practices such as test automation.

Test automation is still controversial in some organisations, but it is impossible to achieve short lead times and high-quality releases without it.

“The paradox is that when managers focus on productivity, long-term improvements are rarely made. On the other hand, when managers focus on quality, productivity improves continuously” - John Seddon, inventor of the Vanguard Method

Statistical analysis revealed that when engineering teams hold themselves accountable for the quality of their code through peer review, lead times and release frequency improved considerably with negligible impact on system stability.

“The difficulty in defining quality is to translate future needs of the user into measurable characteristics, so that a product can be designed and turned out to give satisfaction at a price the user will pay”. - Walter Shewhart, known as the father of statistical quality control

Werner Vogels, CTO of Amazon, says, “You build it, you run it”. This, along with the rule that all service interfaces are designed to be externalizable, has some important consequences. As Vogels points out, this way of organizing teams “brings developers into contact with the day-to-day operation of their software.

It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.”

Influence the culture

NUMMI was a broken organisation was reformed under a new leadership and management paradigm. Despite rehiring the same people, NUMMI achieved extraordinary levels of quality and productivity and reduced costs. In an article for MIT Sloan Management Review, John Shook, Toyota City’s first US employee, rected on how that cultural change was achieved:

What my NUMMI experience taught me that was so powerful was that the way to change culture is not to first change how people think, but instead to start by changing how people behave—what they do.

Those of us trying to change our organizations’ culture need to define the things we want to do, the ways we want to behave and want each other to behave, to provide training and then to do what is necessary to reinforce those behaviors. The culture will change as a result… What changed the culture at NUMMI wasn’t an abstract notion of “employee involvement” or “a learning organization” or even “culture” at all.

What changed the culture was giving employees the means by which they could successfully do their jobs. It was communicating clearly to employees what their jobs were and providing the training and tools to enable them to perform those jobs successfully.

As we do with functional and performance quality, we build evidence of compliance into our daily work so we don’t have to resort to large batch inspections after most of the work has been done.

Make it OK to fail

There are people that are really good at acceptance testing and exploratory testing, that have a keen eye for issues and aesthetics which often developers just simply lack, plus they often come at a piece of work from a different angel. A person in a QA role are a good sounding board and a valuable member of the team, and would help improve automated testing.

However, it can mean that someone in a QA role are at risk of becoming a bottleneck, a gate. Adding more capacity to the bottleneck isn’t the solution here. There's many ways you can improve throughput without adding more resources to a bottleneck.

A QA should not be a blocker to getting work released, otherwise they are a gate and gates become bottlenecks. The question is, can the system survive without a QA present?

It has to be OK to fail. It's easy to recognise that developers don’t like to fail or even get things wrong. By always relying on a QA to qualify work, having someone who will always catch the issues before they go out will mean that, as developers, we will never learn. Instead, developers will need to take ownership of their work, end to end. If it goes wrong, it should be on them to fix it, nobody else.

Sure, I’m not suggesting to adopt Facebook’s “move fast and break things” mantra, but you can make it safe to fail so long as you learn. It’s OK to fail, so long as we continuously improve and quality goes up. It’s easy to get stuck on quality at the last mile, which often means we don’t stop to figure out how to improve our processes upstream.

Root cause analysis is a great example of the type of learning that can be done after a failure. What we find is that they did do the right thing the right way and evaluated the risks at the start, but they still failed. The problems were in quality control – the process. Therefore it's a process that needs improvement, not the people.

Behaviours not roles

It's no surprise that by hiring someone to fill a QA role, the process may not improve and lead time can actually get worse. That's why it's key to hire people who will continue to improve processes and lead time so that quality can improve.

My view is that the better the lead time, the faster we can learn, the faster we can deal with problems, the faster we can get better and fix the process.

Not hiring a QA is a contrarian view but it’s something that every team needs to consider. That doesn't mean fire your QA, no. It means you need to have people who can do the work you need to do, not just tick the “QA” box. There needs to be "quality advocates" on your team, remembering that quality is a whole team responsibility.

If you need to invest in automation testing, then it's a good idea to hire people who have that skill set or invest in your existing team and train your QA to do more than just manual testing.

“If you can’t measure it, you can’t improve it” – Peter Drucker.

If you're trying to decide whether to hire a QA or not, it’s a good idea to agree on a definitive way to measure success, quality and lead time before hiring as that will tell you if it's working or not.

Resources

Here's some further reading on the matters discussed in this article.

DEV Community: James Wade

Measuring performance in agile

Estimating work

Getting More Done

Culture

Predictability

Continuous Improvement

Quality

Commitment

Progress

Measuring Success

Lead Time

Working Software

Team Health

Conclusion

Don't write code for computers

The problem

The solution

Who’s bug is it anyway? Aka handling the on-call rota

Encouraging the right behaviours

What does not work

What happens elsewhere

How to solve the problem

See also

Return early

What is an early return?

Why return early?

Example

Bad: Not returned early

Good: Returned early

Why wouldn’t you return early?

Conclusion

Resources

Be bold

Rule of three

Exceptions are meant to be exceptional

Is no QA the way to go?

Put quality first

Influence the culture

Make it OK to fail

Behaviours not roles

Resources

Presentations

Articles