The Real Costs of Sub-Optimal Product Backlog Ordering

A quick post while I’m thinking about real costs (see the previous post, real costs of context-switching). The role of the Product Owner is to “maximize the value of the team”. Paradoxically, she does not do that by altering the team itself. The flip side of the coin is that the team of creators do not control their own fate when it comes to maximizing their own value. They can deliver high quality software soon and often, and still be a very low-value team, because they do not control what they work on.

The synergy of the Product Owner and Development Team (or creators) is a major component of what makes Scrum work — and by “work” I mean deliver 400% more value than non-Scrum teams, or the same team’s pre-Scrum days. The Product Owner owns the Product Backlog , which can be considered the “on ramp” of the production highway. While the creators are focusing on “this mile of traffic” the PO is queueing up the next batch of work to merge onto the highway once the lane is clear. This both protects the team from distracting work (and distracting people) and preserves their time during the Production Episode so that they can Focus.

The Product Owner herself is also in a bind. She does not control the work she’s given to do! There are caveats to this. A great Product Owner is much more than an order taker. The most valuable thing a Product Owner can say is “No”. But I will touch on this another time. For our limited scope, we will treat the Product Owner as simply an order taker. Since the PO has no control over the work requested, and the creators have no control over the work the PO serves up, how can the Value possible be optimized or sub-optimized?

One good question to ask yourself when wondering if something can be improved is, “Could I take what is given and make it worse?”

The answer is often “Yes”, because a given system or collection of goods is most likely to be in a random order. If your PO simply orders the stories in the Product Backlog by their arrival time, or the importance of the customer, or the height of the customer, or any other arbitrary criteria, we have no reason to assume there is a relationship between the Ordering of stories in the PBL and their Valuation. If we could intentionally rearrange that order to make things worse, then we could intentionally rearrange it for the better.

If the PBL is ordered by anything other than Valuation, the most likely outcome of long-run production is the average value of all stories selected randomly.

In other words, not properly ordering the PBL is a guarantee the value of the Development Team will be average.

Hard to imagine? Picture a black back with a collection of ping-pong balls. On each ball is a $ value. If you randomly reach into the bag, over time, the value you pluck out will approach the average value in the bag. You might get lucky and pull out $1000 first, but you might be unlucky and pull out $1. What kind of bag would you like to use? (What’s the 1st pillar of Empiricism?) A Transparent bag. So what would a transparent bag mean, it means picking the Story order by looking at what’s relevant.

Baby Steps

“But we don’t put a dollar denominated Valuation on our Product Backlog Items”, the patient says.

That’s OK. Can your customers or stakeholders give your PO an estimate, even a very rough estimate of dollar value? Can they even give you an estimated order of value, absent any realistic dollar value?

“But what’s the point of making up numbers? Didn’t you just say we want to avoid getting random value out of the team? All of the stakeholders are just going to put a large number on everything!”

That’s OK. These are baby steps. Imagine customer Carl can tell our PO, “Well, I’ve given you three portfolios of work, and they’re worth about $1M, $2M, and $3M respectively”. This is a great first step. Now our PO can work with Carl to decompose the individual stories within each of the three portfolios and ask, “Now within Portfolio A, can we order these by value? Then let’s put some pretend value numbers next to each one in descending order”. Do that three times and you have three portfolios broken down into individual PBIs, each of which has a (made-up) number representing a value relative to the other stories within that portfolio; but also against all other stories in all three of the portfolios.

If our intrepid PO serves four customers, and does this with each, we now have 12 individual portfolios, with all of the decomposed stories represented by a relative value. We’re making huge strides in improving the long-run value of our Development Team. Now, it’s true that those individual customers may have exaggerated their base numbers. They could be gaming the system against the other managers! But we can work on improving this system later. This is enough to get us started.

So What?

How much difference can the order of the items in the PBL matter? (By the way, this also applies to the Sprint Backlog, but we’ll tackle DOWP later). The short answer is “a lot”. If you’re incredibly lucky, you might pick the most valuable PBI from the top of the PBL and the SBL. If so, you immediately win! Because the next best possible move would be selecting the second most valuable PBI instead. The value of your win is the difference in value between the two. But the most likely outcome is picking a random, average-value story first, and a random, average-value story second.

Here is a sample Product Backlog:

That’s some expensive fruit.

You’ll notice these are mostly thrown in order of the total portfolio’s value. There’s one monkey in the wrench, where our Blue and Orange portfolios snuck up to the top, maybe Ms. Blue is a Very Important Manager, or Mr. Orange brought our PO her favorite coffee. We can also see where the organization decided on an arbitrary cut-off of “Phase I” and what will be part of “Phase II” and “Phase III”. It seems reasonable that someone would think of working each project in this order, because each portfolio is (mostly) more valuable than the one below it, and if you compare the individual stories, they are also more valuable on a one-to-one basis (I used starting values of 1, 2, 3 etc. to make the case clear).

If we calculate the value of working the PBL in this order, how does it compare to ordering by individual story value? And how do both compare to throwing the PBL into a random order?

The graph is pretty. The results are not.

The graph tells the story. Blue, Green and Yellow show the comparison between Portfolio order, individual PBI order, and Random order for good measure (there are many variations of the random order. This particular case was by RNG). The Story order (green) versus Portfolio order (blue) is the first thing to note. Both start out the same, because we’re tackling our $600M story first either way. But then we immediately see the cost of taking the second most valuable story in the pink portfolio instead of the second most valuable story overall. Yellow is our random order, which start out close to zero (a $60K story was in first place), jumps a decent amount by the time the fourth story is a $150M payoff. As we go further in time, the average value delivered converges.

Loss

Right out of the gate, within 1-3 items we have an enormous difference in value delivered between the three ways of ordering the PBL. The dark red line shows the value sub-optimized ($178M is a substantial difference) between the Story-ranked PBL and the Portfolio-ranked PBL. The difference in Story-ranked and Random (thin red line) is even worse, essentially foregoing close to $600M right out of the gate, and not catching up until well into Phase II, or by the time we complete PBI 33.

Questions:

1. What should our next baby steps be?
2. This is just a snapshot in time with 6 portfolios or epics. What happens over time as more epics are submitted to our Product Owner?
3. What other benefits would come from ordering by valuation of individual stories, aside from maximizing the dollar denominated production value of the team?
4. We would have reached “Phase II” at item #25, and “Phase III” at item #32. What observations can we make while thinking about this? What about the time before it; the time after it?
5. Why is the timeline important when considering the comparative value lost?
6. What happens if you miss nearly $600M in value within the first 4-5 items delivered? Could this make or break a project in a way that isn’t fundamentally related to it’s intrinsic value?
7. How much value did your Development Team deliver last year? Does anyone know?