Every billing team has run the same experiment: change an email subject line, watch recovery for a fortnight, feel vaguely encouraged, and ship it. Nothing was learned, because nothing was defined. There was no hypothesis worth the name, no sample size worked out in advance, no decision rule for what result would change behaviour, and no guard against the oldest trap in the field — comparing a test window full of payday Fridays against a control window that had none. Dunning testing fails quietly, which is worse than not testing at all, because it manufactures confidence.
A real test starts with a single sentence: if we change this one thing, this one number should move by roughly this much, for this reason. Retry timing is the richest vein — moving a second attempt from day two to the morning after the most common payday is a mechanical change with a mechanical theory behind it, and the theory can be wrong in an interesting way. Message changes are weaker candidates; they are easy to run and hard to read, because tone effects are small next to timing effects and drown in weekly noise.
Sample size is where honesty enters. A store recovering forty failed payments a month cannot detect a three-point improvement in anything within a reasonable window, and pretending otherwise produces coin-flip decisions dressed as data. The discipline in planning dunning experiments that can actually be read is mostly arithmetic done in advance: how many failures flow through each week, how large an effect would matter commercially, and therefore how long the test must run before anyone is allowed to look at the answer. Writing that number down before launch is the whole game.
Two design rules protect the result once it starts. First, randomise at the subscriber level and hold the assignment — a subscriber who sees variant A in the first email and variant B in the third is not in a test, they are in a mess. Second, freeze everything else: no simultaneous price change, no new retry added mid-flight, no promotional weekend in one window only. Each extra change is a confession you will have to make later when someone asks why the numbers moved.
The metric deserves the same care as the design. Recovery rate alone will lie to you in both directions: an aggressive variant can recover more payments this month while generating more cancellations and chargebacks next month, and a gentler variant can look worse on recovery while keeping subscribers who pay happily for another year. Read at least one harm metric alongside the headline — cancellations during the sequence, support complaints, refunds — and decide in advance how much harm trades against how much recovery.
One more protection is worth naming, because it is the one teams skip when a test starts well: decide the stopping rule before the midpoint peek. Early results in dunning tests are dominated by whichever subscribers happened to fail first — often the same payday cluster, the same card network, the same plan tier — and a variant can look brilliant on day four and ordinary by day twenty. Peeking is human; acting on the peek is what converts a test into an anecdote. Write the end date and the minimum sample on the plan, and let the calendar, not the dashboard, say when the answer exists.
When the window closes, the most valuable output is often a negative result written down properly. A timing change that did nothing, tested cleanly, closes a question the team would otherwise re-litigate every quarter. File it with the hypothesis, the window, and the numbers, and the next person inherits knowledge instead of folklore.
Test less, define more. One honest experiment a quarter will teach a billing team more than a year of hopeful tweaking.
Published by Swipe Again, the flat-fee failed payment recovery service for subscription businesses.
Top comments (0)