People talk a lot about "lean". Maybe they've read The Machine That Changed The World and were inspired by how lean manufacturing transformed the motor industry. They read how Japanese companies became incredibly efficient and blew the competition out of the water and then they take that to their software development team.
This sounds good but too often I feel like people miss the real genius of lean manufacturing and continue to run their organisation with a top-heavy, command and control structure; the only difference is a constant message of
Be more efficient! Stop being wasteful! LEEEEEAN
Agile, Devops and other buzzwords clearly get a lot of their inspiration from lean manufacturing and these ways of working gain their efficiencies (and other, bigger benefits) broadly by empowering development teams and giving them responsibility.
A primer on mass production
Before Henry Ford’s assembly line, cars were made to order by craftsmen. It took a long time to make them and were reserved for the wealthy.
The assembly line made it possible to make cars cheaply which made them accessible to a far broader market.
How?
It’s a big subject, but here are some pithy bullet points
- In order to reduce cost he needed to produce cars at high volume with low-skilled workers, not craftsmen.
- The assembly line was a collection of expensive, inflexible machinery that scaled to volume and could be operated by cheap labour doing very simple tasks.
- Emphasis on standardisation, being able to replace one worker with another cheaply and easily.
Problems
The assembly line was very intolerant to disruption
- Had to add lots of buffers such as storing supplies, space and additional workers to smooth production
- Better to accept defects and fix them after the car was made than to stop the line
The work was very dispiriting. Ford even passed on some of the savings and profits to his workers by heavily increasing their salaries but people just didn’t want to do the same boring task 100s of times a day.
The assembly line being heavily optimised for volume with expensive, specific machines meant it is very inflexible so creating different cars was a very difficult and costly endeavour.
When designing new cars it took what we would recognise as a very waterfall approach, of a project being passed around many departments and problems being unsurfaced way too late due to poor feedback loops and not a good structure of ownership.
Lean manufacturing
Japanese car makers studied the production line and tried a different way. When you hear about kanban and kaizen in agile circles, well it all came from here.
From Wikipedia
For many, lean is the set of "tools" that assist in the identification and steady elimination of waste. As waste is eliminated quality improves while production time and cost are reduced.
There isn't a canonical "lean" way of working, but the one that resonates strongest with me is "The Toyota Way"
The Toyota Way, in which the focus is upon improving the "flow" or smoothness of work, thereby steadily eliminating mura ("unevenness") through the system and not upon 'waste reduction' per se... The implementation of smooth flow exposes quality problems that already existed, and thus waste reduction naturally happens as a consequence. The advantage claimed for this approach is that it naturally takes a system-wide perspective, whereas a waste focus sometimes wrongly assumes this perspective
How does lean try to reduce waste?
Another big subject and there are good resources on it with a google search. The main drive of it from my perspective is actively trying to identify waste and then fix it. Lean describes 3 types
- Mura: Waste due to changes in demand. Lean advocates a "pull" system rather than "push". So you only make a car if there is demand for it rather than trying to forecast demand which is often difficult and makes supply inconsistent. In software this would be akin to perhaps making a cheap prototype to prove an idea before committing to a bigger endeavour.
- Muri: Waste by trying to do too much stuff at once. This is why if you're doing Kanban and not observing work-in-progress limits you're not doing Kanban. It can sometimes feel counter-intuitive that you can go faster by doing less work but you've probably worked in a situation where a team is working really hard and everything feels chaotic and unreliable. This is also why smaller teams are generally preferable.
- Muda: Non-value adding work. A classic example in software would be developers constantly struggling with flaky builds and not addressing the underlying causes.
Lean emphasises the idea of "flow", optimising the processes for delivery and reducing interruptions in the production process. (think about trunk based development vs pull requests here)
Quality
For quality you adopt Kaizen, striving for perfection and hunting down the root causes of issues and fixing them. You adopt a culture of quality across the value stream, not after the product is "finished".
Command and control vs autonomy and empowerment
What I mainly took from the book "The Machine That Changed The World" was how it contrasted mass manufacturing and lean manufacturing in respect to management.
Lean manufacturing acknowledges that it's impossible to heavily micro-manage the production of a complicated product like a car with thousands of people working on it.
The best way for Kaizen to work is to coach and empower the do-ers to find the best ways to deliver.
Signs you're not in a lean company
People with job titles of "manager" or "leader" but are nothing more than coordinators
When designing new cars there is the idea of the Shusa (not sushi!) who in short was actually in charge of things. They could bring people to help and make decisions and were not worried about being overruled by the whims of upper management; they were empowered to make the project work. People would say that these people actually made their mark on a project "oh you can tell that's a Dan car because of X Y and Z".
Contrast that to places I've worked where people have had very lofty job titles but no actual empowerment and were nothing more than a communication funnel between the people who actually make things and the people who could actually make decisions.
How common is it in our industry for tech leads to be miserable simply because they're promoted into what they'll actually call a "shit shield". Seriously, can we stop saying that as if it's a good thing. It's a problem.
A leader should be able to lead
The development team has no say on priorities
Lean famously relies on Kaizen; the concept of continually striving for perfection. In lean plants workers can stop the plant if they find a problem and they work hard to make sure it doesn't happen again.
If you see something that is technically really wrong and is slowing your team down, how empowered are you to fix it? Do you work with a scrum master who will humour you but probably thinks "technical stories" are "waste" ?
Being lean and efficient is not simply done by only working on the most important feature at a given time. It's about the builders working hard to create a system and environment that can be worked on efficiently. That takes the form of the usual best practices; tightening and improving feedback loops, continuous delivery, easy to setup, simple delivery pipeline, excellent code, etc.
The lean plants worked hard to reduce problems and defects and to give themselves flexibility. This only happened because the workers were given the autonomy to make it happen; not micromanagement from above.
How does your team compare? Are you able to fix problems when you see them (which is the best time) or do you have to wait for the "technical debt sprint"?
All teams have a standard, immutable way of working
Toyota didn't just make up lean and be successful, they had to practice and iterate at it to figure out how to be good. Your IT teams need to practice and work at being "lean"
You also have to understand that we dont have teams of resources, they are individuals working on different problems. This means if you want to get the most out of people you need to give them autonomy to figure out how they can work together most effectively.
How does a team get better at working together if they cant gather feedback by retrospecting and then improving the way they work?
Larger organisations often talk about standardisation to allow developers to move between teams. That in itself isn't a bad thing but it's not important as autonomy.
If you want teams to be empowered enough to practice Kaizen and let them find how to work efficiently you have to accept that teams will work differently.
This is fine and in my experience moving between teams is always a challenge even if everyone works the same way, because teams are more complicated than a set of tools and processes.
The people who write the software are not responsible for QA or running it in production
In mass manufacturing the assembly line optimised itself for volume and would defer any kind of quality problems to the end of the line where another team would pick things up and fix them. The problem of this is the feedback loop just didn't exist and systemic problems would never get addressed, which brings waste.
I consider a QA column in a kanban wall as a smell. QA in lean worlds is a part of the entire development process, not something tacked on at the end. QA is a responsibility and an inherent quality of a good team.
From the continuous delivery website
It’s much cheaper to fix problems and defects if we find them immediately—ideally before they are ever checked into version control, by running automated tests locally. Finding defects downstream through inspection (such as manual testing) is time-consuming, requiring significant triage. Then we must fix the defect, trying to recall what we were thinking when we introduced the problem days or perhaps even weeks ago.
If you're throwing your software over a wall to an ops team then how do you expect to write a good system?
Boredom
If you're trying to turn your development team into a homogeneous assembly lines pumping out JIRA tickets at volume you might start to bore some people (and probably build some crappy products).
You probably pay your engineers a lot of money because they're presumably quite clever, so why are you denying them the space to think and take responsibility?
I've worked on projects which should be to be honest quite boring but turned out to be great fun because I worked in an engaged, empowered and fun team who took pride in doing things the "right" way and practiced Kaizen.
Not only was it great for us but we delivered the project which was initially seen as very risky and tricky with very little stress. Not only that but what we learned in respect to shipping software was adopted by other teams in the organisation.
Kaizen is fun, rewarding and will improve your outcomes.
Too scared to release and get feedback
In the book Lean Thinking: Banish Waste and Create Wealth in Your Corporation it says
Identify value from the customer's perspective
Doing this with software products is notoriously difficult but it's made even more difficult if you dont have the courage to release your software to the world and gather feedback.
Too often software is managed as a project which "ends" (this is nonsense, obviously) and no one takes any lessons from the customers. This is the ultimate form of waste that lean strives to fight against.
The transformational lean technologies and methodologies
I think it’s interesting to think about the transformative technologies over the past few years and how they relate to lean. I think a lot of things appear big and transformative but are they solving real problems? Are they helping you be lean?
Docker
Docker is an empowering technology allowing developers to use technology far more freely. Rather than asking for permission to use a technology and have it installed in production etc we can use whatever we like because we just ship containers
The main benefit for me though is it has made the difference between a local computer and production much smaller, which allows developers to take more responsibility for the quality of their software.
Synonymous with moving away from highly specialised, inflexible tooling in the mass manufacturing world
Devops
Devops is lean. It is about tightening feedback loops, improving the quality of our tooling and processes and taking responsibility of our software. Read The Phoenix Project. Just do it.
Cloud computing (e.g AWS)
Again this is another empowering development which lets the shop floor workers easily adapt the technology landscape to solve problems. Synonymous with having more flexible machinery in the lean car factories as opposed to highly specialised ones from mass manufacturing.
Wrapping up - we're not building cars
Agile and DevOps have clearly taken a lot of lessons from the lean manufacturing of cars. All of them are about building complicated things with lots of people which requires a different way of thinking vs mass manufacturing if you wish to do it efficiently with a happy workforce.
But when you're reading about lean it's important to remember that we software developers have different goals
- We're not trying to build software at volume
- We're trying to deliver the promise of software, products that are malleable and can change to users needs quickly and cheaply.
A big thing about lean is trying to identify value from a customer's perspective and eliminating anything that doesn't contribute to it. People take this lesson in software to aggressively deprioritise anything that isn't obviously a feature.
The thing is, this is just a very immature way of looking at what the stakeholders of software are. The "real" customer does benefit from:
- Making it more operable so it has less downtime
- Improving the test suite so the software can be changed easily to meet customer needs quickly
- etc etc
I think examining the "value stream" for waste is a great thing, there's nothing more dispiriting than working on something for months with no real evidence it'll bring any benefit to anyone.
Too often though this is led by "product" people with little collaboration from the technical side. The team is unable to practice Kaizen and instead wastes its time working around technical debt, slow builds, poor feedback loops, manual testing etc. This results in a product which eventually becomes too hard to change and cant react to what the user wants in the long-run. A far cry from the flexible, malleable and efficient development system we're all supposedly striving for.
This comes back to what I wrote earlier, is your tech lead empowered to lead? Or is she just making sure you put estimates on JIRA tickets? Is she treated as an equal when deciding priorities for work? Does she have to argue constantly for scraps of time for the team to practice Kaizen whilst passive-aggressively being accused of being wasteful for even suggesting non-feature work?
Final final thought
From Wikipedia again
The cultural and managerial aspects of lean are arguably more important than the actual tools or methodologies of production itself. There are many examples of lean tool implementation without sustained benefit, and these are often blamed on weak understanding of lean throughout the whole organization.
Replace the word lean with "agile" and I think you'll have a lot of software developers nodding with knowing looks on their faces.
Top comments (2)
This was a great read and resonated a lot with some other teams I have worked on. Really well done, ty.
A great article. I could identify where my current team is diverging from being agile. This is surely going to help out in improving and doing things the 'lean' way.
Thanks