DEV Community

Cover image for Moving Fast: A Retrospective on Trunk-based Development
Basti Ortiz
Basti Ortiz

Posted on

Moving Fast: A Retrospective on Trunk-based Development

As the final output for a series of software engineering courses at my university, my team developed DocTrack: an installable, mobile-first rewrite of the university's document tracking system. Unlike its PHP-based predecessor, the DocTrack front end is a single-page Svelte application that leverages Progressive Web App technologies such as offline caching, web push notifications, and background synchronization. In the back end, we use PostgreSQL (as the database) and Deno (for the REST API and the static file hosting of the web server).

GitHub logo BastiDood / doc-track

A mobile-first document tracking system built for the modern age.

DocTrack Logo

DocTrack

A mobile-first document tracking system built for the modern age. 🚀

Runtimes

Operation

Link to Website

Introduction

DocTrack is a robust, open-source document tracking system that utilizes a modern web stack to meet the demands of document management in the modern age. It offers a powerful and intuitive platform to efficiently manage and monitor documents within an organization or in any collaborative environment.

With a strong focus on modern web development paradigms, DocTrack is a proof-of-concept on the usage of modern web technologies (such as Svelte + Typescript in the front-end and Deno + PostgreSQL in the back-end) to develop a Progressive Web Application that can handle offline usage, deferred operations, and resource caching.

The online version of DocTrack is hosted through Deno Deploy and is accessible here.

Highlights
doc-man   paper-trail
ofline-usage   notifs-alert
barcode-qr   qr-scan
office-man   user-man
metrics   categories

Program Features

  1. 📄 Document Management - Store, organize, and send documents between registered offices and organizations.
  2. 🔍 Document Tracking - Track the lifecycle of…

With all of these moving parts and interconnected features, how were we able to complete the scope of our project within a single semester? In this article, I reflect on my experiences with trunk-based development as the project lead for DocTrack.

The Git Workflow

Before embarking on the project, we needed to decide on our Git workflow. Having experienced the unnecessary indirection and bureaucracy of Gitflow, I immediately proposed and implemented a trunk-based strategy instead.

Why not Gitflow?

In a nutshell, Gitflow is a branching strategy that mainly involves four types of branches:

  1. The Trunk (e.g., main, master, etc.)
  2. Development Branch (e.g., develop)
  3. Feature Branches
  4. Hotfix Branches

Here, develop first branches from main. Developers then branch off of develop to work on specific feature branches. As feature branches are merged into develop, it is worth noting that develop eventually becomes a long-lived branch ahead of main. In a perfect world where no merge conflicts exist, the develop branch can finally be merged into main as a trivial fast-forward operation upon completion of the sprint1 requirements.

But alas, we do not live in a perfect world. When hotfix branches step into the picture (which are allowed to directly branch from and merge into main), long-lived develop branches present themselves as a mine field of merge conflicts, especially when large diffs and refactors are involved.

This may be a non-issue for most experienced industry professionals, but considering that my team consists of undergraduate students (initially) without much experience with Git in a collaborative setting, I wisely decided against subjecting ourselves under the madness of large-scale merge conflict resolution.

Trunk-based Development

Instead, we implemented a trunk-based strategy, where there are only two types of branches: the trunk (i.e., main) and the feature/hotfix branches. The setup is simple:

  1. Pull the latest changes from main.
  2. Create a new feature/hotfix branch off of main.
  3. Apply relatively small self-contained additions, deletions, and changes.
  4. Submit a pull request.
  5. Have someone review the pull request.
  6. Merge the feature/hotfix branch back into main.
  7. Delete the now-merged branch.

Most notably, we have no develop branch. There are no levels of indirection. Everything branches from and merges into main. One may naturally ask then: what makes this better than Gitflow?

Merging Often with Atomic Commits

The primary key to success is the short lifetime of branches. By enforcing self-contained units of work, we are able to merge pull requests directly into main without worrying about conflicts because small scopes tend to affect fewer files. Another key assumption is that everyone has the latest version of main anyway, which further reduces the likelihood of merge conflicts due to unsynchronized histories. In summary, small scopes and frequent merges facilitate conflict-free development.

To more effectively apply the principle of small scopes and self-contained work, we enforce Conventional Commits in the repository. In DocTrack, conventional commits encourage the atomicity of progress. The easiest pull requests to review are the ones with commit histories that read like a storybook. With all pull requests being small and self-contained, the velocity at which we integrated new features and bug fixes cannot be overstated.2

Getting Used to Merge Conflicts

But alas, we do not live in a perfect world. Even with all the aforementioned measures in place, merge conflicts are still an inevitable reality of the Git workflow. In our case, merge conflicts typically occur when two developers branch off of main and work on independent features that just happen to touch a common region in one or more files. When attempting to merge these two branches into main, one of them succeeds while the other bears the brunt of the conflict.

Fortunately, when this happens, the conflicts are often minimal and easy to reason about due to the enforcement of small scopes and self-contained work. That is to say, they are not as mind-bending as resolving a long-lived develop branch into a long-diverged main branch as in Gitflow.

To further increase productivity, I often take these moments of conflict as opportunities to teach my team how to handle merge conflicts in the future. The minimal diffs are instructive examples that empower my team to resolve more complex situations. Indeed, merge conflicts became less of an issue in the latter half of the semester.

Resolving Conflicts with Git Rebase

At some point, I finally introduced the concept of a Git rebase. In short, a Git rebase allows us to yank a branch off of its base and replay its commits (one by one) on top of another branch (e.g., the latest version of main). This one feature alone is our primary weapon against merge conflicts. When conflicts arise, we rebase feature/hotfix branches on top of the latest version of main and then resolve from there.

Despite rewriting the Git history, the rebase is often a safe operation because feature/hotfix branches are typically assigned to a single "owner", thereby granting an artifical mutual exclusivity over the branch for the meantime. Until a pull request is published, we refrain from touching the branches of others.3

Note that we never rebase main itself—only the feature/hotfix branches reserve that right.

Another use case for Git rebase arises when a pull request depends on a preceding (but pending) pull request. Consider a situation when a feature is implemented in PR #1. A developer then branches from PR #1 and works on PR #2. Now, the latter cannot be merged until the former is merged. When this happens, we simply set the latter pull request as a "draft" and wait for the former pull request. Once merged into main, rebasing #2 on top of the now-merged main should be trivial since the work of #2 derives from that of #1.4

Although the standard Git merge is sufficient for many cases, I insisted on a Git rebase because it results in a cleaner linear commit history, where there are no internal merge commits that pollute an otherwise self-contained pull request. In the long run, this made it easier to review said pull requests. In this setup, merge commits that only performed a fast-forward operation were allowed.5

Eventually, we developed clever merge strategies along the way. We realized that smaller pull requests must be merged first before the larger feature-heavy ones. Should the need arise, it is simply easier to resolve merge conflicts in a large pull request if these were caused by small preceding pull requests. This is less true the other way around.

Following this insight, we prioritize and batch the minor bug fixes first over feature-based pull requests. We then rebase the latter pull requests accordingly. In most cases, the rebase results in a trivial fast-forward anyway. In other times, the merge conflicts are minimal and self-contained.

Automating Continuous Integration

Given our pace of development, we require extra safety nets to maintain confidence in the correctness of our codebase. This is why early on, we set up some GitHub Actions workflows that lint the implementation, format the code, build the project, run unit tests, execute end-to-end tests, and deploy to production. The repository blocks a pull request if any of the mandatory workflows fails, thereby enforcing the invariant that main must always be in a deployable state.

A neat consequence of this setup is that the main branch serves as an anchor from which all debugging begins. When a new feature is introduced with regressions, a cursory inspection of the GitHub Actions logs is often sufficient to find the bug. For larger pull requests, however, main becomes a natural endpoint for bisection (by assumption).

Admittedly, the safety nets did come with some trade-offs. In particular, the maintenance of automated tests certainly slowed down the development of features in the back end due to the additional work. If it's any consolation, however, the overhead of test-driven development was alleaviated by implementing the features and their test cases in lockstep. Peak productivity was achieved when the test-driven methodology was like hitting two birds with one stone.

With that said, the time "spent" on writing tests is less "spent" but rather "invested" in determinism and debuggability. Although it may not be apparent to me here and now, an alternate universe where we did not invest in an automated testing infrastructure may have resulted in hours of wasted productivity due to aimless debugging.

Conclusion

Trunk-based development is a perfectly reasonable Git workflow, especially for small teams. The simple workflow, the minimal bureaucracy (beyond the mandatory code reviews), and the necessary automations empowered the DocTrack team to move fast with great confidence. These were the keys to our success with trunk-based development:

  • Small, self-contained commits and pull requests.
  • Frequent merges to main ensure that everyone has the latest version.
    • Decreases the likelihood of merge conflicts.
    • Lessens the severity of merge conflicts (if any).
  • Automation of tests and deployment infrastructure.
    • Ensures that main is always in a deployable state.
    • Instills confidence in the health of the codebase (even in fast pacing).

Although the mandatory maintenance of test suites slowed down productivity to some extent, I still believe that this was a worthwhile investment that yielded a net positive over time. The assurance and saved time (from aimless debugging) more than made up for what was lost in rapid (but reckless) feature implementation.


  1. The word "sprint" here is used as a catch-all term to refer to a single development cycle. It is not meant to be taken literally as in agile development frameworks such as Scrum. 

  2. Considering that we had other coursework to attend to, I must say that our pace was exceptionally impressive. 

  3. There are times when a rebase actually affects future patches in a pending pull request. When this happens, a simple git rebase origin/<feature> resolves the incompatible histories. 

  4. I am willing to concede that this may not be the most effective workflow, but it certainly worked for my team. I am aware that we could have just superseded Pull Request 1 with Pull Request 2, but I insisted on preserving the atomicity of the pull requests. 

  5. Admittedly, some internal merge commits still managed to slip through, but we honestly could not be bothered to rebase them again. 😅 

Oldest comments (7)

Collapse
 
petercodeson profile image
Petercodeson

really interesting post

Collapse
 
freddyhm profile image
Freddy Hidalgo-Monchez

Awesome retro! I've reached the same conclusions with my teams. The benefits in speed and agility are so high that it does completely change the velocity of a project.

I'm curious as to how long your branches were alive for on average? Were there any concerns from your team?

I find it sometimes hard to sell to my colleagues the importance of opening PRs quickly (say at least at the end of the day) even if the overall feature is not complete. I find a lot of engineers find psychological safety in keeping their branches alive until their changes are "perfect".

Collapse
 
somedood profile image
Basti Ortiz • Edited

I'm curious as to how long your branches were alive for on average? Were there any concerns from your team?

We were still getting the hang of things in the first half of development. At the time, the PR turnaround times were typically one to three days. The bigger PRs lasted five days at most. As the days went by, I gradually realized how unsustainable it had become that feature branches were creeping up to be long-lived branches. Not only did we fall behind our internal deadlines, but it also became trickier to merge dependent PRs due to diff conflicts.

So at around the halfway point of the semester, I had to step in and enforce what may be the greatest team policy change throughout the project: the 24-12-6 Rule. We agreed on the following:

  1. The maximum turnaround time for any PR must be 24 hours.
    • All PRs must be merged/resolved/closed within 24 hours at most.
  2. The average turnaround time for any PR must be 12 hours.
    • All PRs must be merged/resolved/closed within 12 hours in the average.
  3. The average response time for any PR must be 6 hours.
    • A PR must receive a reply or a review at least once in any 6-hour window.
    • This encourages rapid feedback.

Although these measures may seem draconian in hindsight, this policy change was exactly the spur we needed to get back on track. What used to be two-day turnaround times became twelve-hour turnaround times. Suddenly, we were churning out features and bug fixes multiple times in a single day. It would be an understatement to say that our main branch was "active".

So to directly answer your question, our branches typically lived for 18 hours during asynchronous work. But during our synchronous sessions, we managed to get all of our changes into main within 2 hours. We strived to keep the branch lifetimes as short as possible.

As an aside, seeing multiple PRs merged into main interestingly served as a morale boost of sorts. It was a satisfying feeling to see the backlog of PRs gone, which I would like to believe further contributed to our success.

Collapse
 
wkbaran profile image
Bill Baran

I don't know why anyone would be masochistic enough to use gitflow.
In my mind it means there's something else wrong with your development process, probably a combination of too many cooks in the kitchen and junior developers doing 'experiments' that aren't closely related to a user story

Collapse
 
somedood profile image
Basti Ortiz • Edited

In defense of Gitflow, the attributed creator Vincent Driessen posted a "note of reflection" in the blog post that started it all.

During those 10 years, Git itself has taken the world by a storm, and the most popular type of software that is being developed with Git is shifting more towards web apps—at least in my filter bubble. Web apps are typically continuously delivered, not rolled back, and you don't have to support multiple versions of the software running in the wild.

This is not the class of software that I had in mind when I wrote the blog post 10 years ago. If your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team.

If, however, you are building software that is explicitly versioned, or if you need to support multiple versions of your software in the wild, then git-flow may still be as good of a fit to your team as it has been to people in the last 10 years.

I wouldn't say Gitflow enjoyers are "masochistic". We're just caught up in our own bubble of web development, which is why we can't fathom any other workflow aside from CI-centric ones.

Collapse
 
rillus profile image
Riley

In practical terms, your workflow doesn't differ too much from how I implement Git Flow, the key points being:

  • atomic commits
  • small features
  • short-lived branches

With develop being deployed to a preview environment, it provides an extra opportunity to verify the build before moving to main... but it just depends on your build and deployment processes.

I enjoy the safety of having main as the last stable deployed branch - if anything happens to develop (or in your case main), you have a quick roll-back route. This is only necessary for me though if the project is live with an active user base. For early stage projects you can forego the live branch and run a much more stripped back workflow.

In short, I think Git Flow as described is a complex system which you can adapt according to your needs (as long as you get the team to buy-in and agree to adhere), just as with all your Agile processes.

Collapse
 
kfazz profile image
kfazz

We currently use a hybrid approach at work. The applications are web mvc frameworks with different business domains as submodules, where we use the trunk approach, and the parent project pretty closely follows gitflow. Merge conflicts are pretty painless, as they're usually just advancing a submodule pointer, or updating project specific css or controllers.