Dunning advice is full of testable claims: send the first email immediately versus the next morning, lead with the amount or the deadline, three touches or five. Most merchants who try to test them discover the same arithmetic problem. A subscription business with a few thousand customers might see thirty failed payments a month. Split thirty failures across two variants and three decline families, and each cell holds about five people. Five people cannot tell you anything; the "winner" of such a test is whoever the calendar happened to favor.
That does not make testing pointless. It makes sequencing the whole game.
Test one variable at a time, and pick variables in the order the failure queue actually flows. Timing beats copy, because timing decides whether the message is read at all — a subject line cannot rescue an email sent after the grace period effectively ended. Channel beats both: adding an in-app alert or an SMS for the failures email cannot reach changes the population itself, which swamps any wording effect. Copy comes last, and within copy, the ask comes before the tone — a clear button and a prefilled update page move more recoveries than any rewrite of the first paragraph.
Pool your failures honestly. The legitimate way to get sample size is time and aggregation: run a variant for a full quarter, compare it against the same quarter's decline mix, and pre-commit to the metric — recoveries within the grace window, per first failure, segmented by decline family. The illegitimate ways are quieter and more common: peeking weekly and stopping when one side looks good, letting the variants drift so that variant B also got a new retry schedule, or counting a day-40 winback as a dunning recovery for whichever variant sent the last email.
There is one more trap specific to billing: your variants share a sender reputation and a card file. A more aggressive variant does not just perform differently — it can train spam filters and annoy the exact customers the other variant then emails. Treat send-volume and complaint signals as guardrail metrics, not afterthoughts.
What to do while a test is running: nothing, on purpose
The hardest part of a small-sample test is the quarter of disciplined inaction it demands. The practical compromise is a two-track queue. Track one is the test itself, frozen: no copy edits, no schedule changes, no "small" improvements that contaminate both variants equally in theory and neither in practice. Track two is everything that does not touch the tested surface — card-file hygiene, updater coverage, the support macros, the update page's instrumentation — where work continues and improvements are real, merely unattributed to the test. Teams that park all billing work for a quarter in the name of purity fall behind; teams that change everything weekly learn nothing. The two-track split is how a small business gets both.
There is also an honest stopping rule worth writing down before the first send: the result will be read as a direction, not a verdict. With pooled quarters and segmented cells, a variant that wins consistently across two decline families earns adoption; a variant that wins in one family's noise earns another quarter, not a rollout. Deciding that in advance is what separates a testing program from a recurring argument with charts attached.
If you want the discipline written as a worksheet rather than a warning, a planner for A/B testing dunning copy without poisoning the result walks through variable order, pooling windows, and the pre-commitment fields before the first variant sends.
Most dunning tests fail before they start, in the planning, not the sending. That is also where Swipe Again's recovery work begins: one change at a time, measured on a frozen cohort, so that when a number moves you know what moved it.
Top comments (0)