I am a fan and supporter of the #buildInPublic trend. Rome wasn’t built in a day, and neither was any meaningful software product. To some degree, the build in public trend is even more important than ever as a signal that what you are shipping is not haphazardly built AI slop thrown together in a weekend. Real products take time, thought, and iteration. So I wanted to tell you about what I am building and where I currently am in the process.
The project that I am spending most of my development time on lately is MergeMeter. It is a software engineering metrics platform designed to help leaders use data to better understand the flow of work through their teams and have conversations about reducing bottlenecks.
This project is also scratching an itch for me personally. In past roles, I spent countless hours obsessing over the data I could glean from various systems about my teams. It helped me put data behind my gut feelings about where bottlenecks were and led to some great conversations with team members and stakeholders about the state of the department.
I think metrics can be very powerful, but it is important to keep in mind that they are best when used to seed conversations and not treated as the whole story. Metrics should help leaders understand systems, not judge individuals.
Over the years, I often ended up building my own small tools or scripts to gather some of this information because the commercial tools on the market were either extremely expensive or focused on measuring individual productivity instead of understanding system flow. MergeMeter is an attempt to make those types of insights more accessible.
How MergeMeter Works
As of now, the product focuses on the data that can be gleaned from pull request metadata, along with optional surveys related to the pull request.
The metrics are gathered via a GitHub Action added to a code repository. The action is triggered when a pull request is merged, sending meaningful metadata, never source code, to MergeMeter.
If configured, surveys regarding things such as confidence level, risk potential, and AI usage are sent to the pull request author and pull request approvers.
Users can then log into MergeMeter using their GitHub login to view metrics about the pull requests in their organization.
I cannot overstate how valuable pull request data is on its own. It can seed great conversations by providing context about many aspects of the software development process, including delivery patterns, review behavior, and change size.
Metrics Available Today
The current metrics available include:
- Pull request throughput
- Slicable by month, quarter, week, and committer
- Pull request size
- Pull request merge confidence
- Days of the week when most pull requests are opened and closed
- Pull request cycle time
- Pull request approver and reviewer counts
- Files changed per pull request
- Time until first pull request feedback
- and more.
The Metrics That Still Matter at the Dawn of the AI Era
There has been a lot of discussion lately about measuring AI usage and AI return on investment inside engineering organizations. I hear it in conversations with peers, at conferences, and across many of the articles and podcasts circulating in the industry. It makes sense that leadership teams and boards are interested in it. AI is clearly changing how software is written.
That said, the industry is still figuring out what meaningful measurement actually looks like. Counting prompts, tracking tool usage, acceptance rate of AI suggestions, or estimating the percentage of AI generated code does not necessarily tell you whether your engineering system is actually improving. Usage without context does not say very much.
If a team is generating large amounts of AI assisted code but that code shows up as massive pull requests that take days to review, then the net impact on the system may not actually be positive. In some cases the bottleneck simply moves somewhere else in the development process.
You still need to track the flow of work through the system to provide context around its impact. Is the team deliverying more work? Are reviews happening faster? Are bottlenecks improving or shifting to new places?
This is where pull request flow metrics become incredibly valuable. Pull requests sit at the end of the software development assembly line. They track when work starts, when collaboration occurs, and when code is committed to the repository.
Even if AI agents eventually write a majority of the code, teams will still rely on pull requests and reviews to move work from idea to merged code
….
The full article is available at https://sabiventures.com/mergemeter/building-mergemeter-in-public-pre-mvp/
Top comments (0)