(Image credit: midjourney)
Trunk What?
Trunk Based Development (TBD) is committing small, frequent code-changes either directly to main branch or using few branches that are merged to main in less than a day. This is a way of working that is scientifically shown to be a predictor of high performing teams, with high performing teams having twice the impact on their organizations along metrics such as profitability, productivity, number of customers, customer satisfaction, and more:
ℹ️ Teams that did well had fewer than three active branches at any time, their branches had very short lifetimes (less than a day) before being merged into trunk and never had “code freeze” or stabilization periods. It's worth re-emphasizing that these results are independent of team size, organization size, or industry.
- Forsgren PhD, Nicole; Humble, Jez; Kim, Gene. Accelerate: The Science of Lean Software and DevOps
Scientifically speaking we want the time it takes from code committed to code running in production to be less than an hour (and ideally deploys happen per-commit).
If you're curious about the actual science you can find it all documented in the amazing book Accelerate: The Science of Lean Software and DevOps, and you can follow their yearly Accelerate State of DevOps Report publications that track trends. This research has consistently shown for a decade that, amongst many other fascinating findings, trunk based development is a practice that is likely to cause an increase in team performance.
A few people I talk to have had very bad reactions to this research, reacting with anger at being told how some ways of working are better than others. But let's cut this short: Accelerate is the most trustworthy analysis of software development patterns, and it has consistently found that teams that use Trunk Based Development do better. It's not a guarantee of success; it's "just" a statistically meaningful pattern that appears consistently among several tens of thousands of data points… but that should be plenty for us to take notice.
Most of us are thankfully fine with the idea of optimizing how we to work to find ways of shortening feedback loops, and Trunk Based Development is merely one such technique out of many to experiment with.
What's the trick then?
The essence of TBD is committing straight to main, with no branches or pull-requests or other external gating getting in the way. There are several variants in this topic, but I think the most useful is to discuss an ideal Trunk Based Development workflow first before getting into any potential compromises.
Committing straight to main can sound scary, but it's really quite manageable with the right local workflows.
What's in a branch?
First of all, we need to establish local workflow that provide great safety in only letting good commits onto main. To do that we can consider the positive aspects of using branching and other gating mechanisms and then reimagining them as local workflows.
A log-hanging fruit is running tests: We'll run tests locally prior to shipping code, to extract that aspect of safety out of branching. But that is just an example of the more general principle: We should reimagine all checks currently done in branch as something we run and do locally.
There's one aspect that often trips up this topic though: What do we do about the review? For a lot of programmers that's the whole point of a PR: Getting those green checkmarks from your colleagues.
The answer is simple, but can be uncomfortable to some: Reviewing must also be reimagined into a local activity. How? By doing pair- and team-programming: You review what your buddy writes, they review what you write.
Continuous reviewing can be a big topic and we'll avoid the details in this article to stay focused on introducing the core concepts. But one way or another reviewing cannot be left as a slow asynchronous affair where changes are left in branches to rot for hours, days, or even weeks.
No More Gates
And finally, if you're unlucky enough to come from a context where there are even more gates beyond the PR such as Change Approval Boards (CAB) then note this part of the research:
ℹ️ Approval by an external body (such as a manager or CAB) simply doesn't work to increase the stability of production systems, measured by the time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.
- Forsgren PhD, Nicole; Humble, Jez; Kim, Gene. Accelerate: The Science of Lean Software and DevOps
So for the purposes of Trunk Based Development there are no more gates beyond pushing to main: The change should automatically get deployed to production.
Green field
Finally, I must say I've unfortunately found it very difficult to retrofit existing processes towards TBD. It very quickly feels scary for a team to move away from what they perceive is working, and so my experience has been it is easiest to do this from e.g. a new service: Lock down all the Trunk Based Development practices right from day one. That allows a team to learn in a safe space and get comfortable with TBD, and perhaps in time those lessons can be applied back to legacy services.
Conclusion
We've covered the problem statement of how scientific analysis of software development has found patterns that indicate Trunk Based Development causes teams to more positively impact their organizations.
And we've covered the principles to pursue for practicing Trunk Based Development: Establish strong local workflows that reimagine the safety-benefits of branches and pull-requests in a local context, and do away with later quality gating such as manual steps or Change Approval Boards.
So what's left? I like getting practical, so I've also written a follow-up article to tackle the actual nuts and bolts of how one can set up a Trunk Based Development-based workflow. Please read on if you're curious for more.
Top comments (3)
Love how dynamic this makes the deployment process. Also that it embraces that error are OK and can be fixed. This gate keeping we have with PRs often feels like perfectionism.
I am suffering from Change Approval Boards, as you so accurately put it, and agree, most of the time the peeps are too far away from what I am doing to truly understand the list of changes.
Anyone not closely involved with the concepts behind my PR will only be able to comment on form, which is a whole different monster which fuels my imposter syndrome.
Besides the automatic test, do you also combine this with user tests? Not sure what your needs are, but for games I could imagine that this would mean we needed more in-person game testing rounds.
Could not agree more on external reviewing, it's a net-negative every time.
I started my career in gaming, and I found it a difficult space to remain agile in because every feature can impact the whole. It's so much art intersecting with tech it was difficult to preserve simplicity.
But I don't think the point of test-automation is to test everything. A lot of release-processes guard themselves behind heavy end-to-end testing, with humans meticulously going through every screen and clicking every detail before a change is brought to production. And it's the wrong response. It's like an autoimmune disease where the life-giving response is out of proportion and mistargeted. Hiding behind thicker and thicker armor weighs down the processes until the product suffocates.
I think it's very important to ambitiously imagine how a process would look if anchored around releasing code both quickly and safely. Because the research also show that speed and quality go hand in hand: High performing teams are both faster and safer than other performance-groups. What would that look like in games? If the game is long-lived and will be iterated on, then perhaps feature toggles could allow changes to be deployed instantly and then progressively rolled out? It would eliminate things like separate builds for user tests, and probably most changes are improvements that can be rolled out automatically.
I of course don't know your context and I've been out of games for a decade now so regardless my knowledge is likely out-of-date, but I hope there are many exciting ways to reimagine developments because otherwise games will miss out on some really amazing gains in job satisfaction, business impacts, and more.
And thanks for reading!
True, safeguarding is not the point here.
Feature flags and roll-outs are some safety measures we employ as well. If a bug is discovered in the rollout it can indeed be fixed quickly.
Something we found is helpful though is when the team play tests the game regularly or are playing the rollout. Not as testers running endless test scenarios but as players. The advantage is that the team understands the user perspective which improves decision making during development which in turn has an impact on development time.
Thank you for taking the time to answer in such detail!