These are my notes on Nicole Forsgren, Jez Humble and Gene Kim's Accelerate, building and scaling high performing technology organizations. Note that there is an additional chapter on the book website, under the "BONUS MATERIAL" section.
The findings of this book are based on the research done using the data from the State of DevOps survey from 2014 to 2017.
It has been a great read, and a lot of the leadership advice is in line with Gerald Weinberg's "Becoming a technical leader".
In fact it has been such a nice read that my life as a CTO is going to start by trying to implement all the practices outlined in it.
Key Insights
- Quality == Speed. Throughput and stability move together.
- Software delivery performance impacts organization performance, including non commercial.
- Measure:
- Performance:
- Delivery lead time.
- Deployment frequency.
- Stability:
- Time to restore service.
- Change fail rate.
- Performance:
- Deming: Fear invites wrong figures.
- Ron Westrum generative culture.
- Good information: Answers what and when needs to be answered, in a way that can be effectively used.
- Change culture by first changing what people do, not how people think.
- Technical practices are vital. Not optional, not secondary.
- Gerald Weinberg: Quality is value to some person.
- Architecture's most important characteristic: loosely coupled, which means:
- Most testing can be done without an integrated environment.
- Release independently of other apps it depends on.
- Architects should focus on engineers and outcomes, not tools or technology.
- The Rugged Manifesto.
- Transformational Leaders:
- Should change environment, not "fix" the person.
- People are an organization's greatest asset.
- Focus on followers identifying with the organization.
- Common causes for transformation program to fail:
- Treat the transformation program as a project with an end date.
- Implement it top-down, without input from those affected.
- Not setting measurable business and organizational outcome for the transformation effort.
The capabilities
Highly recommend to download the OVERALL RESEARCH PROGRAM AND HIGH PERFORMANCE BEHAVIORS AND PRACTICES as it contains all the practices and what their impact is, plus the table "High-Performance Team, Management, and Leadership Behaviors and Practices" by Steve Bell and Karen Whitley Bell.
- Continuous Delivery capabilities:
- Test Automation.
- Deployment Automation.
- Trunk-Based Development.
- Shift Left on Security.
- Continuous Integration.
- Continuous Delivery.
- Version Control.
- Test Data Management.
- Architecture Capabilities:
- Loosely Coupled Architecture.
- Empowered Teams.
- Product and Process capabilities:
- Small Batches.
- Make Flow of Work Visible.
- Customer Feedback.
- Team Experimentation.
- Lean Management and Monitoring capabilities:
- Limit Work in Progress.
- Production Monitoring (to inform business decisions).
- Visualizing Work.
- Lightweight Change Approval.
- Proactive Notifications.
- Cultural capabilities:
- Foster generative culture.
- Encourage learning.
- Collaboration amongst teams.
- Job Satisfaction.
- Support Transformational leadership.
TOC
- Preface
-
Part I: What We Found
- 1 - Accelerate
- 2 - Measuring Performance
- 3 - Measuring and Changing Culture
- 4 - Technical practices
- 5 - Architecture
- 6 - Integrating InfoSec into the Delivery Lifecycle
- 7 - Management practices
- 8 - Product Development
- 9 - Making Work Sustainable
- 10 - Employee Satisfaction, Identity and Engagement
- 11 - Leaders and Managers
- Part II: The Research
- Part III: Transformation
- Bonus Material - How to transform
Preface
- 24 capabilities to drive improvement.
- Research based on 23k Surveys, 2k organizations.
- Applicable to:
- Orgs of any size (< 5 to > 10k).
- Greenfield and legacy.
- Any industry.
- Throughput and stability move together.
- Software development and delivery can be measured in a statistically meaningful way.
- Organization ability to make software positively impacts profitability and market share.
- Culture and technical practices matter. And they can be measured.
Part I: What We Found
Chapter 1 - Accelerate
- Maturity models are not the appropriate tool to use or mindset to have.
- Capabilities model of measurement is essential
| Maturity | Capability |
|---|---|
| There is a fixed goal. You "arrive" |
You can always improve. Never done |
| Lock-step. All orgs and all teams are treated the same |
Multidimensional. Dynamic. Team and Org context important |
| Vanity metrics | Outcome based metrics |
| Static Levels | Dynamic levels |
- Focus on right capabilities.
Chapter 2 - Measuring Performance
- Measuring performance in SW is hard:
- Inventory is invisible
- Breakdown of work arbitrary
- Design and delivery done simultaneously
- Design will change as we implement
- Flaw on previous attempts:
- Focus on outputs rather than outcomes.
- Focus on individuals/local, rather than team/global.
- Examples: LoC, velocity, utilization
- Proposed:
- Delivery lead time:
- From code committed to running in production.
- Deployment frequency:
- Proxy metric for batch size. More deploys, smaller batch size.
- Time to restore service.
- Change fail rate, changes that cause:
- Outages.
- Hot fixes.
- Rollback.
- Fix-forward.
- Patches.
- Delivery lead time:
- Delivery lead time and deployment frequency measure Tempo/Performance/Throughput.
- MTTR and Change fail rate measure Stability/Reliability.
- MTTR better than MTBF to measure reliability as "Failure is inevitable".
- No trade off between improving performance and achieving stability/quality.
- SW delivery performance impacts organization performance, including non commercial.
- Learning culture must exist before starting measuring.
- Deming: Fear invites wrong figures.
Chapter 3 - Measuring and Changing Culture
- Organization culture, three levels of culture:
- Basic assumptions: Invisible, things that we just "know".
- Values: more "visible".
- Artifacts:
- Mission statements.
- Technology.
- Formal procedures.
- Heroes.
- Rituals.
- Focus on second level.
- Ron Westrum, three types:
| Pathological | Bureaucratic | Generative |
|---|---|---|
| Power oriented | Rule oriented | Performance oriented |
| Low cooperation | Modest cooperation | High cooperation |
| Messengers "shot" | Messengers neglected | Messengers trained |
| Responsibilities shirked | Narrow responsibilities | Risks are shared |
| Bridging discouraged | Bridging tolerated | Bridging encouraged |
| Failure leads to scapegoating | Failure leads to justice | Failure leads to inquiry |
| Novelty crushed | Novelty leads to problems | Novelty implemented |
- Good information flow is critical:
- Answers what and when needs to be answered, in a way that can be effectively used.
- Org culture predicts performance outcomes.
- Bureaucracy is not bad. Ensures fairness.
- Good culture:
- Trust and cooperation.
- Higher quality decision making.
- Easy to reverse bad decisions.
- Higher job satisfaction.
- Change culture by first changing what people do, not try to first change how people think (John Shook).
Chapter 4 - Technical practices
- Technical practices are vital. Not optional, not secondary.
- Key principles of Continuous Delivery (CD):
- Build quality in.
- Small batches.
- People solve problems, computers do repetitive tasks.
- Continuous improvement.
- Everyone is responsible.
- Foundations of Continuous Delivery:
- Comprehensive configuration management: everything in VCS.
- Continuous Integration.
- Continuous testing.
- Jerry Weinberg: Quality is value to some person.
- CD brings higher quality:
- Less rework or unplanned work.
- Less bugs
- Practices:
- Version control: including system and app config.
- Test Automation:
- Reliable tests.
- Writen by devs.
- TDD is important, but not mandatory.
- Test Data management.
- Trunk-Based development:
- No branches older than one day.
- Shift left on Security.
Chapter 5 - Architecture
- Most important characteristic: loosely coupled, which means:
- Most testing can be done without an integrated environment.
- Release independently of other apps it depends on.
- Lower performers more likely to be integrating with custom SW developed by another company (or outsourced) or working on mainframe systems.
- Biggest contributor to CD:
- Be able to do work without needing to talk with anybody outside the team. Aka, proper cross-functional teams.
- Read Steve Yegge's "Platform rant".
- Additionally, loosely coupled architectures enable engineering organizations scaling.
- Teams that can chose their own tools are more performant, but standardize around architecture and configuration of infrastructure.
- Architects should focus on engineers and outcomes, not tools or technology.
Chapter 6 - Integrating InfoSec into the Delivery Lifecycle
- Typical InfoSec ratio: 1 per 10 infra per 100 devs.
- OWASP Top 10.
- Building security into SW improves delivery performance and security quality.
- Shift left on security:
- Security experts involved earlier and through the whole dev/ops process.
- Security provides tools/libraries/processes that are secure.
- Security provides devs the means to build security in.
- The Rugged Manifesto.
Chapter 7 - Management practices
- Derived from the Lean movement
- Practices:
- Limit WIP - not enough on its own.
- Visual management: Key quality and productivity metrics.
- Feedback from production: to make business decisions.
- Intrateam code reviews (which includes pair programming) are the best way of change approval.
Chapter 8 - Product Development
- Eric Ries - The Lean Startup: Lightweight approach to explore new business models and product ideas.
- Lead Product Development:
- Small batches:
- Less than a week.
- MVP.
- Make Flow of work visible:
- From Business to customers.
- Status of products and features.
- Customer feedback:
- Satisfaction metrics.
- Actively seeking the feedback.
- Implementing the feedback.
- Team experimentation:
- Without external approval.
- Team can change requirements as they learn.
- Small batches:
Chapter 9 - Making Work Sustainable
- Where code deployments are more painful, you will find the poorest organization performance.
- To reduce deployment pain, use same technical practices as to improve delivery speed and stability.
- Reduce burnout:
- Environment emphasizes learning over blaming.
- Strong sense of purpose.
- Invest on employment development: time, space, resources.
- Remove obstacles.
- Authority to make decisions that they are affected by.
- Burnout risks:
- Overwork.
- Lack of control.
- Insufficient rewards.
- Breakdown of community.
- Absence of fairness.
- Value conflicts.
- Leaders should change environment, not "fix" the person.
- Official vs Real organization values.
Chapter 10 - Employee Satisfaction, Identity and Engagement
- Better employee Net Promoter Score (eNPS) == better business outcomes.
- eNPS:
- "How likely is it that you would recommend our org/team to a friend or colleague?"
- 9-10 -> Promoter.
- 7-8 -> Passive.
- 0-6 -> Detractors.
- People are an organization's greatest asset.
- Diversity in gender/minorities achieve better outcomes.
Chapter 11 - Leaders and Managers
- Leadership: inspiring and motivating those around you
- Essential for:
- Establishing high-trust cultural norms.
- Support dev productivity.
- Support team experimentation and innovation.
- Work across org silos to achieve strategic aligment.
- Transformational Leaders:
- Vision: where the org is going and where it should be in 5 years.
- Inspirational communication.
- Intellectual stimulation.
- Supportive leadership: cares about followers' personal feelings and needs.
- Personal recognition.
- Transformational Leadership:
- Appeal to followers' values and sense of purpose.
- Focus on followers identifying with the organization.
- Culture of Learning:
- Create a training budget and advocate to use it.
- Google 20% time.
- Safe to fail.
- Weekly lightning talks.
- Brown bag sessions.
- Demo days.
- Mini-conferences.
Part II: The Research
Skipped this part.
Part III: Transformation
Chapter 16 - High-Performance Leadership and Management
- Spotify Model.
- External coaches to challenge assumptions.
- Squad stand-up -> Tribe stand-up -> Senior Leadership stand-up.
- Technical problems shared with Chapter, business problems with Tribe.
- "Help me better understand the problems you're encountering", instead of "Why isn't this getting done?"
- Teams own their job. They can decide not to release.
Bonus Material - How to Transform
- Common causes for transformation to fail:
- Treat the transformation program as a project with an end date.
- Implement it top-down, without feedback from those affected.
- Not setting measurable business and organizational outcome for the transformation effort.
- Executing Continuous Improvement:
- From the Lean Enterprise book.
- Four steps:
- Three Planning steps:
- Understand the direction or challenge.
- Inspirational goal that seems impossible.
- Gasp current condition.
- Establish next target condition.
- Understand the direction or challenge.
- One Execute:
-
Iterate towards target condition:
- Scientific approach: Plan, Do, Check, Act.
- Improvement Kata 5 daily questions:
- What is the target condition?
- What is the actual condition now?
- What obstacles do you think are preventing you from reaching the target condition? Which one are you addressing now?
- What is your next step? What do you expect?
- When can we go and see what we have learnt from taking that step?
-
Iterate towards target condition:
- Three Planning steps:
- Principles of Effective Org Change Management:
- You are never done with improvement work.
- Leaders and team agree on measurable outcome. Teams discover how to achieve it.
- Achieve Large-Scale Change Iteratively and Incrementally.
- What works in an org does not need to work in others.
Top comments (5)
Ive found this book very helpful and we have implemented the learnings form it in our organisation.
I particularly like how it "redefines" dev-ops. Many associate it with being able to stand up servers or use new and modern CI tools. But all it really is, is a set of practices that allow devs to operate their systems. Create automated test, easy to deploy and maintain systems, develop the tools to manage those systems, and plug them together. So even a specialised person in the team might be undertaking the tasks of setting up those tools, is the whole team that benefit from it, provided they also do their part (coding, testing, etc).
I also like how it moves away from "vanity metrics" like code coverage for example. What matters is how often you break things, how long it takes you to recover from it, etc.
We are in the process of implementing those, how has been your experience? Have you noticed (or measured) an improvement?
We had a great experience so far. We noticed a great improvement in the failure rate for example.
We have gone quite far in terms of shaping our processes. We basically shaped a Lean process that meet our needs.
We start with a Roadmap that maps our Product Strategy.
Each Roadmap Item (RMI) is a Feature or Improvement, that we shape and define in terms of what we want to achieve with it.
We then give this RMI to a team member to lead the solution design for it, and we team up to implement it, test it, deploy it, and operate it. We have an end-to-end approach in our team involvement with the work required. No handover across silos (Designer -> Dev -> QA -> Ops), we are all involved the whole time.
This means we have effectively shifted left, or built quality into our process, which reduced the failure rate, and also improved our recovery time.
In terms of release frequency and delivery lead time, we follow a Kanban plus "Release Train" approach. We have a weekly release cycle, that works well for us with the -ever growing- level of test automation. This means that features, fixes, or any work is released within a week of being "ready".
Being "ready" means it is coded correctly, covered by automated test, and most importantly, solves the problem it set to tackle.
Hope this helps
Wow! Congrats for the transformation. Your story is really inspiring!
If you don't mind me asking, how to do you measure "Delivery Lead Time"? Do you look at git commit dates? Do you look at jira/github issue open/close dates?
Thats probably the hardest one to measure. I thought about using git dates for it, but then I realised something.
In the definition it says: how long it takes for code that is "ready" to reach production. And that's the key, when its "ready".
As I mention we work on a Kanban flow with weekly Release Trains. Which means that when the code is "ready" to be released (not necessarily when its written) it will have less than a week before it reaches production.
And since we all work together in a cross functional fashion (ie: no handover of tasks from dev to qa for testing), there usually is no much lead time from when the dev considers its work done until is merged and in the next release train