Purpose of agile and the many misconceptions about it

Introduction

There have been many years since Agile went viral in the IT world and most organizations have adopted it. The reality is that the understanding is often limited, the expectations wrong and the disappointments from big. I'm writing this post because the successor of hyped Agile is DEVOPS and the same things are happening as well.

It starts with the fact that Agile was commercialized and sold into organizations by means of training, trainers, tools and certifications. Companies were created with the only goal to sell agile and it is not cheap. As years passed, the next steps were scaled agile because we all had mastered agile in one team. Big business if you ask me. In order to convince an organization to justify the expenses, a bit of tweaking and twisting happened to be able to sell the usual promise of doing things cheaper. And this is where all starts going wrong. People invest a lot of money, into making their processes cheaper with a process that is not even targeting cost.

As a disclaimer, I've followed two official training for agile, one as a Scrum Master from Scrum Alliance and one for Scaled Agile Framework (SAFe). They had their usefulness for sure but I can't fully justify their cost, especially with how costly SAFE is as it extends to the whole organization.

What should you expect from agile?

If you ask me, the purpose of agile from the perspective of business is to increase predictability and transparency. If cost is reduced as part of the process then it is a bonus. In fact, this costs more money to execute but it arguably saves money by avoiding delays and broken promises. It is all about moving expenses and profits around for a better result. if the result is your business goal, then the cost is justified. So, whether it is cheaper or not, it really depends on each case but most importantly it is not the primary driver. If only the execution cost is calculated then best to stay away from agile because it will disappoint you or you will disappoint others.

How does it work?

I'll not get into the technical wording, I'll keep it simple to explain this for everyone regardless of background. I'm also not going to dive deep into the SCRUM or Kanban methodologies because it is not relevant. It is not my goal with this post to teach agile. There is plenty said about this topic after all. My goal here is to explain the basic principles, many of them found in common sense and real life, to help you understand the basics.

Imagine that a business wants to implement a feature. After some analysis, the teams break down the work into stories also known as the backlog. Consider a story as an atomic unit of work with clear goals, clear validation and a size that is reasonable to achieve within a few days. As part of the breakdown, the team discusses the story and figures out what the risks are and achieves alignment. This process is also known as grooming, although you should know that the term is removed from the official wordbook but still most people use it. Last part of grooming, is the estimation of the story, also known as story-pointing. Each person in the team votes based on his/her perception about the effort required, risk, complexity etc. The only requirement is that his/her estimation is relative to another story's estimation that all agree upon. When all votes are similar, then the vote with the highest occurrence is chosen as the estimate also known as the story points. The backlog actually tells a story of how the team is going to achieve the feature and also what constitutes a successful delivery. Teams don't estimate for long into the future, because of the uncertainties and fast evolution of the software engineering industry. It is proven to be futile. Still, within the timeline of estimations, the total story points reflect an estimation of effort, which is implicitly is an estimation of time required. When working with SCRUM, each team works in small periods of time also known as sprints. For each sprint, the team commits to fully implement some stories. Usually, a person, also known as the product owner, will drive which stories to include based on their estimates and the availability of the team. Once there is consensus over the commitment, the sprint starts. Sprints typically last between one to three weeks, and most do two weeks. A few weeks later, the team presents its efforts to the stakeholders by means of a demo. The ceremony is actually called Demo. The team gets the chance to showcase their progress, raise awareness and the stakeholders get also the chance to ask questions and if necessary adapt their plans. This is something that is very typical with software engineering and flexibility is very important. It can be after all that the team went to a completely wrong direction but it is always better to find this early and have the chance to correct instead of figuring out one day before delivery. This is also known as failing early. After the demo, the team has a retrospective, also known as Retro where they get to discuss things that went good or bad with the ultimate goal being to improve for the next iteration. After a couple of sprints, the teams figure out their velocity which is how many story points they burn per sprint.

How is this predictable?

Story points are all arbitrary numbers to protect the teams of connecting effort and complexity with time and obligations. It doesn't take rocket science to do the math and connect the two, but in my experience, it helps the engineers to do much better estimations. Sometimes they are overestimated and sometimes underestimated. It doesn't matter much because it turns out that the with time, the estimations average out while the team learns how to better organize and estimate. To achieve this and minimize wrong estimations, the stories usually represent a small amount of work, hence the requirement for small stories with reasonable size. If the story is too big or too complicated, then chances are that the estimate will be wrong with big variances that have much more impact on the average. The goal of agile is to allow a person representing the business, to predict when a feature will be complete or chose which parts can be successfully implemented given a certain deadline. This is achieved by dividing total estimation with sprint velocity to calculate how many sprints it will take. This is an estimation but it is one much more trustworthy than asking someone to estimate how long something complicated will take. This is how predictability is achieved.

What about all these meetings?

Notice that there is some overhead in the process and many people don't agree with it. Especially at the beginning, it seems like the team went to a bureaucratic frenzy, where backlogs are managed and too many meetings take place. It feels counterintuitive. It looks like project management was dropped also into the team's responsibilities and that is partly true because part of agile is to allow teams to self organize. With time, respect and commitment to self-improvement, the team finds its way and this becomes part of work. In time, teams figure out which parts of the process are implicit and they tend to minimize the overhead. In any case, you should expect the cost of implementation to rise, especially during the first period.

The demo - My holy grail

Without jumping into much detail about how it helps the engineers, there are two processes that are crucial for the development of every person. The demo is, in my opinion, the single most important ceremony when doing agile. It requires a preparation from the team and should be always performed by different people. A demo man should never happen. Besides the usefulness in transparency and communication, it helps people learn how to explain a topic to an audience that is not an expert as they are. It also helps the non-experts understand a bit the hurdles faced during development. Overall, people learn how to be on stage. This is something missing from many technical profiles who would historically avoid the spotlight. It can be a difficult journey and it takes time to master. It also requires a couple of failure as part of the learning process. Being able to explain something is one of the most important indications of good knowledge. And the demo makes people more knowledgeable and overall help with their professional development.

Scaling agile

How does agile for the people not in the teams? This is a challenge indeed but more or less similar processes are followed. Multiple features are discussed on a high level and estimated are provided. At this moment, the estimated have significantly high chances of being wrong but as with the low-level stories they tend to average out. The team usually consists of product managers and architects and they both try to figure out how a high-level feature fits into the puzzle. This is not just an effort and complexity estimation but also acknowledging team availabilities, unknown topics, training etc. There are some best practices on how to choose which feature to implement next but at the end of the day, it is a matter of experience of the people involved in the discussion and business analysis.

Where does it go wrong?

It all starts with the expectation to reduce cost. Agile is a culture and as such, it can never solve the inherent problems of an organization. In fact, it will magnify the issues and create more problems if the organization is not willing to adapt. Agile is about culture and way of working. The goal is simple and the means less structured and less controlled. Although it is intended to help business, its focus is on the team and enabling the team to be the most productive. Hence, agile works well only in organizations were the drivers are relevant to technology and efficiency. It also requires consistency and high alignment in the team and rarely works with members having a mix of different employers, locations and type of engagement because their incentives are different. So when agile is run by business as some sort of project management, it usually goes wrong. There are too many interferences and too many decisions that don't enable the process to work. There is always a good excuse but overall it takes lots of discipline from business and a lot of learning to avoid the usual mistakes that lead to an ineffective team. Such a team is costly without any results.

Typically, at this point, agile is blamed because it didn't deliver on the original selling point and that is also not a fault of agile. The root cause is again the culture which is thinking like buying something, plug it in and voila things happen faster. You don't buy agile, you figure out a way and you become agile. So the most typical reason to fail with agile is when business and technology are not aligned and are not thinking with the same goals. This is something that has been recognized and led to the introduction of the DEVOPS culture wherein the simplest terms, there is no split between business and technology. With technology being so dominant nowadays, this means that technology becomes the business as much as the traditional business was. This requires significant changes in the culture and mindset of an organization. This is the most challenging task that organizations will have to face in order to survive in this new extremely fast-paced technological era. You can read more about this in my post A high-level overview of DevOps.

I'm not saying that agile companies and consultancies are wrong. But, it needs to be understood that they are eventually selling agile as a commodity when they can only help guide you into agile and then you figure things out. I've seen such consultants come into organizations only to sell training, certifications and often their own tools.

Indications of bad agile

With everyone doing agile, how does someone recognize agile misunderstandings and failures? What are the indications?

• When cost is too often the reason to disapprove of agile.
• When the demo ceremony is not done.
• When the product owners representing the business are not within the same organization structure as with the rest of the engineers.
• When there are too many different roles in the process.
• When there are too many non-relevant people in the process.
• When fixing agile is considered as the magic wand you wave that will solve everything.
• When you hear the us and them too often.
• When everything is about process and process has taken over.
• When certain outage or problems tend to happen too often with a similar pattern.
• When after an outage or a problem there is not post-mortem.
• When everything is in email or similar or in people's head.
• When project management is dominant and very close to the teams.
• When projects drive team composition and capacity.
• When tools and processes are more important than people.
• When the scrum master is a project manager.
• When agile is too often mentioned. As strange as it sounds, agile is implicit. Mentioning repeatedly agile means that you are not there.

I consider the role of the Scrum Master as anti-agile. The role is very much necessary to introduce a team in agile but contrary to common practices, I think the role becomes obsolete once the team picks up and paves its own path. If there is a scrum master, then it should be the person responsible for all teams. I've come to this conclusion because I've worked with very effective teams where actually all that needed to be done was handled by any member of the team. In the meantime, all issues that could not be resolved due to culture in the organization, could not be resolved by the scrum master as well, effectively negating the usefulness of the role. At the end of the day, if a team mentions a couple of times that something is wrong or when something goes wrong a couple of times, then with or without a scrum master, it is just a matter of culture in the organization to listen and make a decision (or not) to provide a solution. There is no place for a 100% role of scrum master. This is only in the books and the sellers. My reasoning is reflected by a simple question of whether the scrum master is technical or not:

• When not, which is normally the case, then he/she is often an interference in the team. As the role rarely justifies a full-time involvement, often the scrum master role is mixed with project management responsibilities. A good project manager is an extremely important role but a very bad one to have inside an agile team.
• When yes, then why waste his/her skills when anyone in the team can actually plan some meetings.