A/B testing cold email subject lines without fooling yourselfNovember 02, 2026Testing two subject lines on a small list
You changed one word in your cold email subject line, sent it to 40 prospects, got two replies instead of one, and now you're convinced you've struck gold. But with samples that small, you're likely fooling yourself. Here's how to A/B test subject lines without drowning in false confidence.
Small Samples Produce Confident Nonsense
When you test a subject line on 40 sends split into two groups of 20, a single reply difference feels like a clear signal. Statistically, it's noise. With a baseline reply rate of 5%, you'd need hundreds of sends per variant to detect a meaningful lift with any confidence. At 40 a day, you're not running an experiment—you're flipping a coin and calling it strategy.
The fix isn't to stop testing; it's to test sequentially and accumulate evidence over weeks, not hours. Treat your inbox as a slow-motion laboratory where each day's sends add to a running total. Only declare a winner when you've hit a pre-calculated sample size—not when the early numbers look exciting.
- Calculate required sample size before you start: for a 5% baseline and a 20% relative lift, you need roughly 1,500 sends per variant.
- At 40 sends a day, that's about 75 days per variant if you split 50/50—so test one variable at a time and be patient.
- Use a sample size calculator (e.g., Evan Miller's) to set expectations and avoid peeking.
Sample Size for a 40-a-Day Inbox
If you can only send 40 emails a day, you have two options: run a simultaneous A/B test with 20 sends per variant, or run a sequential test where you send variant A for a week, then variant B for a week, and compare. Simultaneous tests control for time-based variables (like day of week) but require longer to reach significance because each variant gets only half your volume. Sequential tests reach the same total sample size faster but risk confounding from time trends.
For most small teams, sequential is more practical. Send variant A to all 40 contacts for two weeks (about 560 sends), then switch to variant B for two weeks. Compare reply rates at the end. This gives you 560 sends per variant—still short of the 1,500 ideal, but enough to detect large differences (e.g., 5% vs. 10%). If you need smaller lifts, you'll need to run multiple sequential cycles or increase volume.
- Simultaneous test: 20 sends per variant per day; requires ~75 days for 1,500 per variant.
- Sequential test: 40 sends per variant per day; requires ~38 days for 1,500 per variant, but watch for seasonality.
- Aim for at least 500 sends per variant to detect big swings; 1,500+ for subtle ones.
Measuring replies, not opens
Reply Rate Beats Open Rate Every Time
Open rates are inflated by privacy features like Apple Mail Privacy Protection, which pre-fetches images and marks emails as opened even when they're not. Reply rate is the only metric that directly ties to pipeline: a prospect who replies is a real human engaging. Clicks can be bot-driven or accidental; opens are increasingly meaningless. Measure replies per variant, and track positive replies separately if you can.
When you test subject lines, the goal is to get more replies—not more opens. A subject line that boosts opens but tanks replies (because it overpromises) is a loss. Always decide the winner based on reply rate, and use opens only as a diagnostic to understand why a variant failed.
- Define success as replies per 100 sends, not opens or clicks.
- Segment replies into positive, neutral, and negative to avoid celebrating a spike in unsubscribes.
- Ignore open-rate differences under 10 percentage points—they're likely noise.
Sequential tests on a small sending volume
Sequential vs Simultaneous Tests
Simultaneous tests are the gold standard because they eliminate time-based confounding: both variants run under the same market conditions. But with 40 sends a day, you'd need to split 20/20, doubling the time to reach sample size. Sequential tests are faster but risk bias if your prospect list changes over time (e.g., you target different industries week to week).
To run a clean sequential test, keep your targeting criteria constant across both periods. Use the same lead sources, the same personalization approach, and the same follow-up sequence. If you must change something, note it and consider restarting the test. For most B2B teams, sequential is the pragmatic choice—just be honest about the limitations.
- Use sequential when volume is low and you need faster feedback.
- Use simultaneous when you can afford longer test cycles or have higher volume.
- Keep all other variables constant—don't test subject line and offer at the same time.
What Not to Test
Don't test things that can't plausibly move reply rate enough to matter. Tiny tweaks like adding an emoji, changing punctuation, or swapping one synonym for another are usually wasted effort—the expected lift is too small to detect with your sample size. Similarly, avoid testing multiple variables at once (subject line + body copy + send time). You'll never know what caused the difference.
Also skip testing subject lines that violate deliverability best practices (ALL CAPS, spammy phrases, misleading claims). Even if they win in a small test, they'll hurt your sender reputation long-term. Focus on big swings: personalization vs. generic, question vs. statement, benefit-led vs. curiosity-led. These are the levers that can produce double-digit percentage differences—large enough to detect with realistic sample sizes.
- Don't test minor word changes—test structural differences.
- Don't test more than one variable per experiment.
- Don't test anything that risks spam complaints or unsubscribes.
Stop letting small samples fool you. Define your metric, calculate your sample size, and run sequential tests that respect your daily send limit. Your pipeline will thank you.
Useful links
- Evan Miller Sample Size Calculator
- HubSpot: A/B Testing Email Subject Lines
FAQ
How many sends do I need for a reliable A/B test on subject lines?
Aim for at least 500 sends per variant to detect large differences (e.g., 5% vs. 10% reply rate). For subtle lifts, you'll need 1,500+ per variant. Use a sample size calculator to set expectations.
Can I test subject lines if I only send 40 emails a day?
Yes, but use sequential testing: run variant A for a period, then variant B, keeping everything else constant. It takes weeks to accumulate enough data, so be patient and avoid peeking at early results.
Why is reply rate better than open rate for cold email tests?
Open rates are unreliable due to privacy features that auto-load images. Reply rate measures actual engagement and directly correlates with pipeline generation.
What should I avoid testing in cold email subject lines?
Avoid tiny tweaks like emojis or punctuation, and never test multiple variables at once. Also skip anything that could harm deliverability or brand reputation.
See the data behind this
- B2B data coverage by industry and country
- Verified contacts in United States
- Verified contacts in Germany
- Verified contacts in Poland
The data behind this
- templates
- advertising contacts
Originally published on BatScout — live B2B data: companies, suppliers and creators with verified contacts.
Top comments (0)