DEV Community: Greg Foster

The "Mom Test" in software development: asking good questions when everyone is lying to you

Greg Foster — Thu, 08 Feb 2024 17:39:56 +0000

Graphite.dev is currently a thriving company developing a code review platform used by tens of thousands of top developers. But, like many dev tools startups, it didn't start that way.
In this post, I want to focus on the wisdom from a particular book that's been a cornerstone of my learning: "The Mom Test." This book is a short and essential read for anyone venturing into the world of DevTools startups, open-source contributions, or creating developer-focused products.

From Airbnb to startup founder

Before working at Graphite, I was an infrastructure engineer at Airbnb. There, I learned countless lessons that helped teach me how to become a great software engineer. However, I would come to learn that when building something from scratch, it hardly matters how well you build it. What matters much more is picking the right thing to build. This realization became particularly evident as I embarked on the startup path, and co-founded Graphite.

Two pivots

Our team has always been passionate about the software quality and release processes. The first prototype my co-founders and I built was a “bug capture” tool. Despite getting a prototype in the hands of a wide network of friends, LinkedIn connections, and even random engineers online, the reception was immediately underwhelming. We pivoted into our next product, centered around iOS rollbacks, and faced the same challenges.

Frankly, no one (outside of a select few users) cared about what we were building. We started asking why — this is where “The Mom Test” came in.

Embracing the "Mom Test" in product development

Had we asked the right questions upfront, we could have saved ourselves years of toil. It might sound silly, but asking good questions can be really, really difficult.

"The Mom Test" book explains the solution - it’s about framing your questions in such a way that you get truthful, unbiased feedback, even from those who are inherently supportive, like your mother. When I used to share my excitement about our iOS rollbacks concept, the usual response was overwhelmingly positive, but in hindsight it was probably just folks being nice to me.

Of course, my friend would tell me, “iOS rollbacks sound like a great idea!” But what I should have asked was:

“When was the last time you googled for a way to roll back your iOS app?”

“Did you try the answers online?”

“You’re a great engineer, tell me about how you’ve hacked a solution here.”

“Given that you’ve hacked a solution, do you still google from time to time for something better?”

The key is to ask questions that dig deeper, inquiring about actual behavior, like whether someone has ever actively sought a solution like yours. Furthermore, software engineers as a user group are some of the most capable people in the world at solving their own problems. If they haven’t already tried scripting a solution, was it that big of a problem in the first place?

Existing vs. nonexistent solutions

In developing a product for developers, it’s crucial to check one key detail: Does a solution already exist?

One of two things must be true if a solution doesn't exist. Either the idea doesn’t solve a big enough pain point (such as in our case with rollbacks and bug-capture), or there’s a fundamental reason why it hasn’t been able to exist yet. In-memory databases have always been a great idea, but it took RAM getting cheap enough for the idea to become unlocked. Chatbots are wonderful, but were blocked on advancements in large language models.

If you want to pursue building a dev tool that has never been built before, be very clear on why it can only be built now, and be ready to fight many competitors who have also just been unlocked.

A great example of being unlocked by the advancement of technology is MemSQL:

On April 23, 2013, SingleStore launched its first generally available version of the database to the public as MemSQL.[9] Early versions only supported row-oriented tables, and were highly optimized for cases where all data can fit within main memory. This design was based on the idea that the cost of RAM would continue to decrease exponentially over time, in a trend similar to Moore's law. This would eventually allow most use cases for database systems to store their data exclusively in memory.
https://en.wikipedia.org/wiki/SingleStore

The alternative (and in my opinion better) scenario is when the tool already exists but it has some glaring flaws. You can often find an open source project, a blog, or an internal big company tool that matches your idea. This can be some of your best validation that folks care about the problem enough to have tried building it before. The only thing left for you to figure out is: do users still feel some pain around the tool? How can you do better?

PagerDuty is a great example of this scenario:

We spent that first month of ‘09 thinking of ideas and doing research. One of the ways we thought of ideas was by thinking of internal tools that bigger companies had built in-house (like Amazon, where all three of us had worked prior) that other companies of all sizes would need… Amazon had built an internal tool to handle on-call scheduling and alerting via pagers. This tool was bolted on top of their internal ticketing and monitoring systems, so when critical issues were detected, the right people were paged… After doing a bit of research, we realized that it wasn’t just Amazon that built an internal tool for going on call—Google and Facebook both built their own versions. It seemed like there was a clear need here.
https://www.pagerduty.com/blog/decade-of-duty/

In the case of Graphite, our third and final pivot, the tool also already existed. Stacked diffs and alternative code review platforms were already invented and beloved at bigger companies like Google and Facebook. Phabricator and Gerrit were open source. Users actively searched for solutions, wrote scripts, tried self-hosting, and more. Each month someone on Twitter would tweet at GitHub asking for them to build stacked diffs natively. All the while, users craved more. It was the perfect opportunity for a new dev tool.

Why did our first two attempts fail to gain traction, while our third succeeded? It certainly wasn't that the quality of our engineering doubled over night; that remained steady. The answer was that unlike our previous two ideas, people actually felt pain in the problem we were solving.

The Litmus test for dev tools

Fundamentally, Graphite passed The Mom Test in a way that our previous ideas hadn’t. Had we clearly and unbiasedly asked the following questions, we could have picked the right product to build from the start:

"When was the last time you googled for such a tool?”
“Did you try installing what you found?”
“Tell me about a script you hacked together here.”
“Since hacking together a script, when was the last time you still googled for a better solution?”

What ideas have passed this test historically? PagerDuty. Merge queues. Metrics dashboards. Slack bots. CI test runners. The list goes on. If it exists in a big company but doesn’t externally, that's a great starting place. If no company has ever built it internally, you might want to check your rose-tinted glasses.

Closing thoughts

Years into working on Graphite, the insights from "The Mom Test" remain invaluable to me. The whole team references it weekly when seeking genuine, unfiltered feedback, ensuring that Graphite focuses on developing features that address real needs. I highly recommend this book to anyone in the field of product development, especially in the realm of DevTools. It's not just a guide to asking the right questions – it's a roadmap to understanding and meeting your users' true needs.

The ideal PR is 50 lines long

Greg Foster — Tue, 25 Jul 2023 18:50:56 +0000

Most engineers intuitively know that smaller code changes are better than big ones. The logical arguments flow easy - small pull requests are easier to review, less likely to have bugs, and are faster from inception to deploy. There are a few papers around this that I love - see the references section at the end of this post for further reading.

But how small is small? Can PRs be too small? And if a PR is better than a big PR, how much better?

The claim: the ideal PR is 50 lines long

After pulling the numbers, the ideal code change is 50 lines long.

50-line code changes are reviewed and merged ~40% faster than 250-line changes. They’re 15% less likely to be reverted than 250-line changes and have 40% more review comments per line changed. If your median PR is 50 lines long, you’re probably shipping 40% more total code than your teammate writing 200+ line PRs.

50 lines is a sweet spot across speed, review comments, revert rate, and total coding volume. If you’re willing to accept a range, I can recommend 25-100 lines per PR. According to the data, we see that time-to-review, time-to-merge, and review comments per line all get better the smaller you make your PRs. There is a limit though: under 25 lines, and you start suffering a higher revert rate, as well as a lower total code shipped.

Let’s talk about why and back this claim with some data.

Our sample set

All of the data-based statements in this piece are made using private and public PRs and repos that have been synced with Graphite. To figure out the ideal PR size, I took a look at four main metrics and how they correlated with PR size:

Time-to-review/merge
Revert rate
Average number of inline comments
Total code changed over a year

Caveats

As with all collected data, there are some caveats to be aware of when extrapolating meaning from the numbers:

I used inconsistent, non-linear bucket sizes corresponding to intuitive PR size ranges. Linear buckets would be too granular, and exponential bucket sizes lose too much nuance.
I used median PR size rather than average PR size to avoid outlier refactors from skewing the data.
Reverts were defined as PRs with the word “Revert” in the title. This felt like a safe assumption because generated GitHub reverts automatically have the word prepended to the title.
We find that Graphite users tend to create smaller PRs in general, since many organizations use a trunk-based development style (which encourages smaller PRs)

Time-to-review and time-to-merge

Let’s dig into the data. Starting with time-to-review and time-to-merge, we see that the smallest PRs are almost five times faster than 2-5k line PRs. Intuitively this makes sense - smaller PRs mean less lines of code, and less code means a lower chance of having destructive or meticulous changes which in turn leads to a faster review.

What’s fascinating to see play out is that post-5k lines, PRs start getting faster again. I can only assume this is a combination of blind stamps, assumed-safe refactors, package additions or generated changes; perhaps both authors and reviewers start shrugging when they can’t even scroll to the bottom of the change.

Note, we assume that we care about the time per PR more than we do the time per line. If we care about landing a PR as fast as possible, the data tells us to make it as small as possible. But if we want to land as high volume of code as fast as possible, a 2000-line PR merges at a rate of about 12 lines per hour, whereas a 10-line PR merges at about 0.25-2 lines per hour.

Revert rate by PR size

The revert rate demonstrates the same high-level conclusion: smaller PRs are reverted less than large PRs, the least reverted PRs being those that fall between 25-50 lines of code. But once again, the edges of the graph are interesting. Sub-10 line PRs are reverted noticeably more than 10-100 line PRs. If I had to guess, I’d say that sub-10 line PRs get into the realm of dangerous config changes, but I’d be curious if this finding held true after normalizing for language.

Once PRs start to exceed 10k lines of code, they seem to become slightly “safer.” I suspect this is because the extreme end of PR sizes includes refactors, which maybe start including less functionality change and therefore have a slightly lower chance of breaking. Alternatively, engineers may become progressively more reluctant to revert PRs after 10k lines because of emotional anchoring and merge conflicts.

Average number of inline comments by PR size
Depending on what you’re optimizing for (a fast merge or an in-depth code review), you can choose whether or not you want to split your PR into smaller changesets. If you want to get the most feedback on a single PR, write a length 1-2k diff. If you want the highest chance of a blind stamp, keep it under 10 lines. This information is useful to know when all you care about is getting a single specific change out the door with as little debate as possible.

Here we also see that massive PRs start getting progressively less engagement. There’s a practical limit to how much code your reviewer is willing to read - I suspect that 2k lines is the point at which “reading” a PR becomes “skimming” a PR.

Inline comments by pr size

If what you care about is maximizing the amount of engagement and feedback on your code over the long run, you’re best off writing as small of PRs as possible. They’re more digestible, and you can approach a max rate of one comment on every 39 lines of code you write. Alternatively, if you hate written feedback, make your PRs larger than 10k lines, and you’ll start receiving stray comments on only every 6000+ lines of code you change - take this with a grain of salt though, since people rarely put up PRs for the sole purpose of getting feedback/engagement.

Total code output

Some folks might wonder if writing small PRs leads to less code output in total. We all want to be high velocity, but there are times when you need to be high volume. Consistently writing sub-20 line PRs will have a significant impact on your net coding ability - but interestingly, so will writing PRs greater than 100+ lines. The highest volume coders and repositories have a median change size of only 40-80 lines. I suspect this is because the volume of code is change-size * speed of changes. Too small of changes, and the faster merge time doesn't make up for it. Too large, and you start getting weighted down by slower review and merge cycles.

Conclusion

The average developer coding with a team at a tech company should aim for a median PR size of 50 lines - it's generally the size of PR we stick to here at Graphite. Obviously this comes with some edge cases that were discussed above and specific changes may require you to flex up and down in size, but know that doing so may come at a noticeable cost with respect to review quality, speed, and the chance that changes may need to be reverted.

Stacked changes: how Facebook and Google engineers stay unblocked and ship faster

Greg Foster — Wed, 17 Nov 2021 16:39:58 +0000

POV: waiting for your pull requests to get reviewed

You just finished writing the new comments feature for your web app: you wrote the code, tested it, and put up the PR. You're under a deadline and your next highest priority task is to add reactions to those comments.

But wait a minute - the reactions code will depend on the comments code, which isn't yet merged to main. Which means you're blocked.

If you've ever found yourself in this position, you know you have three options:

Work on something lower-priority while you wait for your comments PR to be approved (and risk missing the deadline).
Interrupt your teammate to ask for a faster review.
Make your comments PR bigger by merging what should have been a separate reactions PR into it (which will make it harder to review and increase the chance of bugs).

Stacking your changes

“But what about the fourth option?" - you might ask. Why can’t you write the reactions branch on top of the comments branch to stay unblocked without compromising on the size of your PRs?

The blocker here isn’t Git. In fact, the reason you can’t go with option #4 is that your code review tools probably don’t support it:

How do you tell reviewers that your reactions PR depends on the comments PR?
How will reviewers see the relevant diff for the reactions PR (excluding the comments PR changes)?
How can you propagate changes to the reactions PR if you ever need to update the comments PR before you land it (i.e. to address review comments)?

But suppose we did have the right tooling to make all of this easy - would you stop at just 2 PRs stacked on top of each other? Should comments really be just one PR? Or would you break it into the series of atomic changes you originally wrote: a DB change, a new endpoint to the API schema, the implementation of that endpoint, and the front-end changes that call that endpoint?

No one likes a 4,000-line pull request

I think most developers agree that smaller PRs are easier to work with:

Smaller changes help you get faster, more thorough reviews on your code, be more targeted in who you tag (why do backend engineers and designers both need to review the same change?), and experience fewer merge conflicts.

By stacking your changes, you can get all of these benefits while staying unblocked and not needing to thrash your teammates.

Introducing Graphite

This idea isn't new, and if you've worked at a company like Facebook, Google, Uber, Dropbox, or Lyft (among others), you've either had first-class support for stacked changes built into your code review platform (i.e. Phabricator at Facebook, Critique at Google) or scripts built on top of it to enable stacking. Many engineers who come from those companies seem to miss this workflow, but it has yet to make its way into the public toolchain.

We're among the engineers who want stacked changes back - so much so that we're building the tooling to make it accessible for everyone else.

Graphite - our platform for stacked changes - is built directly on top of Git and seamlessly syncs with GitHub. You can use it without needing anyone else on your team to change their workflows - they'll see normal PRs from you in GitHub, which they can comment on and review as they always have.

Graphite is currently in closed beta, but it's already used by engineers at some of the largest companies every day. If you're interested in being among the first to try it out, you can sign up for our waitlist starting today.

How to use native git as a key-value store

Greg Foster — Thu, 02 Sep 2021 21:51:43 +0000

Most engineers are familiar with creating branches and making commits in Git. The tool is notoriously unintuitive but has become universal in software engineering. But did you know that you can store more than just snapshots of code?

There is already a long history of tools hacking small amounts of metadata into Git. For example, the open-source code review tool Gerrit ingests pull requests through git push by allowing the user to encode data in the name of the remote ref. The command git push gerrit HEAD:refs/for/master would open a pull request against master on Gerrit rather than writing a new ref.

Another example of metadata hacking is the practice of adding unique IDs to commits. Maintaining association between a proposed code change and a specific commit can be hard because a git commit ID can change between revisions. In response, both Gerrit and another code review tool, Phabricator, leverage the commit message as a metadata store. Using a commit hook or alternative source control CLI, they add a unique ID to each commit message which proves stable across rebases and amendments.

The open-source CLI I work on, Graphite, has a different form of metadata it needs to track. To create stacks of branches, the tool needs to map branches to their parents. Storing a reference to the name of a parent branch in a commit message wouldn't work because no one commit is stable over the life of a branch.

After investigating various mechanisms, we landed on using git's object database directly to store branch metadata. The command [git hash-object](https://git-scm.com/docs/git-hash-object) allows a user to write any string Git's object database and returns an ID. A second command, [git update-ref](https://git-scm.com/docs/git-update-ref) allows you to create or update any ref to point to the stored object by its ID. Used together, we had a dead simple mechanism for storing JSON blobs in Git's native database:

const objectId = execSync(`git hash-object -w --stdin`, {
  input: JSON.stringify(metadata),
}).toString();

execSync(`git update-ref refs/branch-metadata/${branchName} ${objectId}`, {
  stdio: "ignore",
});

Storing data is of no use if we can't read it back. Luckily, the read operation is even easier using [git cat-file](https://git-scm.com/docs/git-cat-file)

const metadata = execSync(
  `git cat-file -p refs/branch-metadata/${branchName} 2> /dev/null`
).toString();

With these two code blocks, we have everything necessary to read and write any data to Git's object database. The advantages of this approach are plentiful:

The metadata refs are plainly visible to users by running ls .git/refs/branch-metadata
The data can be inspected, modified, and removed using native git commands.
The refs can be pushed and pulled from remote repositories, allowing easy syncing.

Graphite simply stores small JSON blobs keyed on branch names, but this approach could be used to store any data under any keys while remaining accessible to a tool as common as Git. For example, Graphite has already started caching open PR statuses through git hash-object. By asynchronously fetching and storing PR information, Graphite is able to print elegant log outputs like:

․  ◯ gf--fix_cycles_disallow_meta_parent_cycl PR #238 (Approved)
․  │ fix(cycles): disallow meta parent cycles
․  │ 68 minutes ago
․  │ * f0e3e7 - fix(cycles): disallow meta parent cycles
․  │
◌──┘

You can read Graphite's full implementation of metadata handing here.

How to visualize stacked git branches

Greg Foster — Thu, 26 Aug 2021 18:21:25 +0000

The standard workflow in git is to create feature branches off of a trunk branch (usually called main), submit a pull request, and merge approved changes back into the trunk. This is fairly straightforward and follows the patterns of most pre-git version control systems. However, as any experienced developer can tell you, this workflow is not without issues. One of the biggest challenges of the feature branch workflow is that it often induces developers to create massive pull requests once they've finished building. If you're working on a large project over many weeks or months, your feature branch can easily contain 100s of commits, which correspond to 1000s of lines of code changes. Software engineering wisdom suggests that "any pull request longer than 400 lines is unmanageable" - so how can we improve on this widely-used workflow?

To reduce the pain of large pull requests, engineers at many top companies (i.e. Facebook, Uber, Dropbox, Google) have moved towards "stacking" their git branches. Stacking is the practice of creating new branches off of existing feature branches instead of the trunk branch. This allows developers to break down large features into smaller, more manageable changesets, each of which can be reviewed, tested, and landed incrementally. While stacking initially gained popularity at larger companies as a way to help engineers stay unblocked while they waited for code reviews, engineers at companies of all sizes have found value and velocity from writing and reviewing stacked changes (shhh... we are some of the converts). For more on the fundamentals of stacking git branches, you can read more! here, here, here, and here.

Although stacked changes have benefits for both authors and reviewers, they sadly lack first-class support from git and many of the popular git GUIs. Perhaps the biggest conceptual change is that the state in git can no longer be represented as a list of commits on a feature branch + a list of feature branches - instead we need to think of our git state as a DAG (Directed Acyclic Graph) of branches. Interestingly, while many major dev tools (i.e. ETL platforms, deployment pipelines, CI runners) support visualizing DAGs, this is almost entirely missing from git-related tooling.

While developing the CLI for our code review platform Graphite, we wanted to provide an intuitive, DAG-based visualization of stacked git branches.

Let's take a look at the problem more closely - as mentioned, visualizing a list of branches is easy:

git branch

However, git has no native command to visualize a DAG of branches.

Our Solution

After playing around a lot with git internals, we developed a good solution. Let’s start small and consider commits. Each commit references its parents - you can print them by running:

git rev-parse ${SHA}^

PS: The --help description of git rev-parse is comically cryptic. It seems like we're not alone in thinking this is weird.

We can also associate branches to revision SHAs by running:

git show-ref --heads

Now we can run a breadth-first search from a branch and stop when we see a revision matching a known branch. Unfortunately, walking through many commits with a series of shell executions isn’t very fast, and doing so doesn't help us discover branch children.

A more efficient approach here would then be to load the commit-graph into memory before walking it. Fortunately, git has a helpful command (with a much more human-readable description) for listing all commits and their parent or child SHAs:

git rev-list --parents --all

This provides all the nodes and edges needed to construct an in-memory graph of all commits, from which we can also construct the graph of all branches. Getting the rev-list for child commits is as easy as changing “--parents” to “--children”.

Unfortunately, loading all commit relationships into memory is often a time-intensive operation. This would be prohibitively slow in a monorepo with millions of commits. If we’re looking to build a performant CLI, we can do better by loading only a subset of commits.

A key simplifying assumption we can make is that all our branches were created off of the trunk branch. In this case, the minimum subset of commit relations needed for computing a branch parent is:

git rev-list --parents child_branch ^$(git merge-base child_branch main)~1

This command roughly translates to “List commits reachable from “child_branch,” but not reachable by the trunk. This lists enough commits to calculate a branch’s parent and nothing more - but if we want the minimum set needed to calculate the graph of all branches, we need to tune it further:

git rev-list --parent ^$(git merge-base --octopus branch_1 branch_2 branch_3...)~1 branch_1 branch_2 branch_3...

Which roughly translates to "all commits unreachable by the oldest shared commit of all branches." This final command is how our open-source CLI tool Graphite currently computes the relationships between stacks of branches. Once we have the full graph of all branches, we can finally visualize complex stacks of branches as a graph:

gt log

You can see the full implementation of our branch graphing command gt ls on our Github repo.

If you want to see our visualizations in action, simply run:

brew install screenplaydev/tap/graphite && cd <some_repo> && gt ls