How we predict delivery rates in 2 minutes instead of 80 minutes using Firebase's hidden validation mode—without sending a single notification
"We just sent to 150,000 dead tokens."
My product manager's face turned pale as she refreshed the dashboard. Out of our 500,000-user Black Friday campaign, 150,000 failed with invalid-registration-token. We'd spent 80 minutes of server time and consumed precious FCM quota—all to discover what we could have known in 2 minutes.
The wake-up call: We were discovering invalid tokens AFTER sending. By then:
- 80 minutes of server resources consumed
- FCM quota unnecessarily depleted
- Misleading "500K sent!" reports to stakeholders
- 30% of our users never had a chance to receive the notification
The solution: Firebase's dry-run mode. A little-known parameter that validates tokens WITHOUT sending notifications. In this post, I'll show you how we built a pre-send validation system that predicts delivery rates with 95%+ accuracy in under 2 minutes.
The $2,000/year parameter you've never used
Firebase Cloud Messaging has a superpower that most developers don't know about: the dryRun parameter.
// Normal send - delivers notification to device
await admin.messaging().send(message);
// ❌ Consumes quota, takes time, delivers to user
// Dry-run - validates token WITHOUT sending
await admin.messaging().send(message, true);
// ✅ Fast validation, minimal quota, zero delivery
What dry-run actually does:
✅ DOES validate:
- Token format correctness
- Device registration status
- Token-to-device binding validity
- Returns same response structure as real sends
❌ Does NOT do:
- Send notification to device
- Trigger user's notification sound/vibration
- Increment delivery metrics significantly
- Consume standard FCM quota
The magic part: Dry-run responses predict real send results with 95-98% accuracy. You get the validation results without any user impact.
Why validation matters at scale
When you're sending to 100 users, a 30% failure rate is annoying. At 500,000 users, it's a business problem.
Time impact:
Without validation:
- Send 500K notifications: 80 minutes
- Discover 150K failures: After the fact
- Wasted time on invalid tokens: 24 minutes
With dry-run validation:
- Validate 10K sample: 2 minutes
- Predict: "You'll reach ~350K devices (70%)"
- Send to 350K valid tokens only: 56 minutes
- Total: 58 minutes (28% faster)
Cost impact (per campaign):
- Server CPU time saved: 24 minutes × $0.10/min = $2.40
- Database operations saved: 150K failed writes = $0.45
- FCM quota preservation: Minimal but measurable
We run 50+ campaigns per month: $2.40 × 50 = $120/month = $1,440/year
Trust impact:
- Before: "We sent to 500,000 users!" → Actual reach: 350,000 → Stakeholders lose trust
- After: "We'll reach ~349,000 users (69.8%)" → Actual reach: 351,000 → Stakeholders trust predictions
Implementation: one flag to rule them all
The key insight: use the same code path for validation and production, controlled by a single flag.
// firebase.service.ts
async sendConditionalNotifications(
jobData: ConditionalNotificationParams
): Promise {
// ... Query database for target users ...
const tokens = await this.getTargetTokens(jobData);
// Returns 500,000 tokens based on filters
const chunks = chunkArray(tokens, 500);
// Creates 1,000 chunks of 500 tokens each
// ★ The magic switch
const isDryRun = true; // Set to false for production
console.log(`
========== SEND MODE ==========
Mode: ${isDryRun ? 'DRY RUN (validation only)' : 'PRODUCTION (actual delivery)'}
Total tokens: ${tokens.length.toLocaleString()}
Chunks: ${chunks.length}
==============================
`);
let totalSuccess = 0;
let totalFailed = 0;
for (let i = 0; i < chunks.length; i++) {
const chunk = chunks[i];
// Build FCM messages
const messages = chunk.map(token => ({
token,
notification: {
title: jobData.title,
body: jobData.content
},
data: {
campaignId: jobData.campaignId,
jobId: jobData.jobId,
},
}));
try {
// ★ Pass dry-run flag to FCM
const response = await this.firebaseApp
.messaging()
.sendEach(messages, isDryRun); // 👈 Critical parameter
console.log(`
Chunk ${i + 1}/${chunks.length}:
Valid: ${response.successCount}/${chunk.length}
Invalid: ${response.failureCount}/${chunk.length}
${isDryRun ? '(No notifications sent)' : '(Delivered to devices)'}
`);
totalSuccess += response.successCount;
totalFailed += response.failureCount;
// ✅ Save results to database (same logic for both modes)
await this.savePushNotificationLogs(
jobData,
messages,
response,
isDryRun // Flag stored in database
);
// Rate limiting (even for dry-run)
if (i < chunks.length - 1) {
await delay(2000); // 2 seconds between chunks
}
} catch (error) {
console.error(`Chunk ${i + 1} failed:`, error);
totalFailed += chunk.length;
}
}
const deliveryRate = ((totalSuccess / tokens.length) * 100).toFixed(2);
console.log(`
========== VALIDATION COMPLETE ==========
Total tested: ${tokens.length.toLocaleString()}
Valid tokens: ${totalSuccess.toLocaleString()} (${deliveryRate}%)
Invalid tokens: ${totalFailed.toLocaleString()}
${isDryRun ? '✅ Zero notifications sent to users' : '📨 Notifications delivered'}
========================================
`);
return {
success: true,
totalTokens: tokens.length,
validTokens: totalSuccess,
invalidTokens: totalFailed,
deliveryRate: parseFloat(deliveryRate),
isDryRun,
};
}
Design decisions:
1. Same code path
- No separate validation/production functions
- Reduces code duplication bugs
- Easy to maintain and test
2. Single flag at top level
- Change one variable:
isDryRun - Clear console output showing mode
- Zero risk of mixing modes mid-send
3. Rate limiting applies to both modes
// Even dry-run counts toward rate limits!
if (i > 0) {
await delay(2000); // Respect FCM quotas
}
Storing dry-run results: the audit trail
We store validation results identically to real sends, with one extra field:
// push-notification-log.entity.ts
@Entity({ name: 'push_notification_log' })
export class PushNotificationLog {
@PrimaryGeneratedColumn({ type: 'bigint' })
id: number;
@Column({ type: 'varchar', length: 200 })
job_id: string; // e.g., "dryrun-blackfriday-2025"
@Column({ type: 'int' })
member_seq: number;
@Column({ type: 'varchar', length: 500 })
push_token: string;
@Column({ type: 'bit', default: false })
is_success: boolean; // Did token validate?
@Column({ type: 'datetime2' })
sent_at: Date;
@Column({ type: 'varchar', length: 50, nullable: true })
error_code: string; // FCM error code
@Column({ type: 'nvarchar', length: 500, nullable: true })
error_message: string;
// ★ The key differentiator
@Column({ type: 'bit', nullable: true, default: false })
is_dry_run: boolean; // true = validation, false = production
// Error classification for analytics
@Column({ type: 'varchar', length: 30, nullable: true })
error_type: 'invalid_token' | 'temporary' | 'quota' | 'other';
}
Why store dry-run results?
- Historical analysis: "Token quality improved from 70% → 85% over 3 months"
- A/B testing: Compare dry-run predictions vs actual results
- Debugging: "Did we validate before this failed campaign?"
- Compliance: Audit trail for regulatory requirements
Query examples:
-- Get all dry-run validation jobs
SELECT job_id,
COUNT(*) as total,
SUM(CASE WHEN is_success = 1 THEN 1 ELSE 0 END) as valid,
SUM(CASE WHEN is_success = 0 THEN 1 ELSE 0 END) as invalid
FROM push_notification_log
WHERE is_dry_run = 1
GROUP BY job_id
ORDER BY sent_at DESC;
-- Compare dry-run vs production for same campaign
SELECT
CASE WHEN is_dry_run = 1 THEN 'Validation' ELSE 'Production' END as mode,
COUNT(*) as total,
SUM(CASE WHEN is_success = 1 THEN 1 ELSE 0 END) as success,
CAST(SUM(CASE WHEN is_success = 1 THEN 1 ELSE 0 END) * 100.0 / COUNT(*) AS DECIMAL(5,2)) as rate
FROM push_notification_log
WHERE job_id LIKE 'blackfriday-2025%'
GROUP BY is_dry_run;
Error classification: not all failures are equal
FCM returns different error codes. Some are permanent (dead token), others are temporary (network hiccup).
// fcm-error-classifier.ts
export function classifyFcmError(errorCode?: string): string {
if (!errorCode) return 'other';
// ❌ Permanent failures - token is completely dead
const INVALID_TOKEN_ERRORS = [
'messaging/invalid-registration-token',
'messaging/registration-token-not-registered',
'messaging/invalid-argument',
];
if (INVALID_TOKEN_ERRORS.includes(errorCode)) {
return 'invalid_token';
}
// ⏳ Temporary failures - might succeed on retry
const TEMPORARY_ERRORS = [
'messaging/unavailable',
'messaging/internal-error',
'messaging/server-unavailable',
'messaging/timeout',
];
if (TEMPORARY_ERRORS.includes(errorCode)) {
return 'temporary';
}
// 🚫 Quota exceeded - rate limiting
if (errorCode === 'messaging/quota-exceeded') {
return 'quota';
}
// ❓ Unknown/other errors
return 'other';
}
Critical insight: In dry-run mode, "temporary" errors often become successes in production. Why?
- Network conditions differ between validation and actual send
- Time gap allows transient issues to resolve
- FCM infrastructure load balancing
In our testing: ~80% of "temporary" dry-run errors succeed in production.
Calculating delivery rate: the metric that matters
With classified errors, we calculate delivery rate - tokens that CAN receive notifications.
// fcm-error-classifier.ts
export function calculateDeliveryRate(logs: PushNotificationLog[]): {
total: number;
success: number;
invalidToken: number;
temporary: number;
deliveryRate: number;
successRate: number;
} {
const total = logs.length;
let success = 0;
let invalidToken = 0;
let temporary = 0;
let quota = 0;
let other = 0;
for (const log of logs) {
if (log.is_success) {
success++;
} else {
switch (log.error_type) {
case 'invalid_token': invalidToken++; break;
case 'temporary': temporary++; break;
case 'quota': quota++; break;
default: other++; break;
}
}
}
// ✅ Delivery rate = tokens that CAN receive notifications
// Includes temporary errors (likely to succeed in production)
const deliveryRate = total > 0
? parseFloat((((success + temporary) / total) * 100).toFixed(2))
: 0;
// Success rate = immediate validation success only
const successRate = total > 0
? parseFloat(((success / total) * 100).toFixed(2))
: 0;
return {
total,
success,
invalidToken,
temporary,
quota,
other,
deliveryRate, // ⭐ The key metric
successRate,
};
}
Example output:
const stats = calculateDeliveryRate(dryRunLogs);
console.log(`
📊 Validation Results:
======================
Total tested: ${stats.total.toLocaleString()}
✅ Valid tokens: ${stats.success.toLocaleString()}
❌ Invalid tokens: ${stats.invalidToken.toLocaleString()}
⏳ Temporary errors: ${stats.temporary.toLocaleString()}
🎯 Delivery Rate: ${stats.deliveryRate}%
(${stats.success + stats.temporary} tokens can receive notifications)
📈 Success Rate: ${stats.successRate}%
(immediate validation success)
`);
Sample output:
📊 Validation Results:
======================
Total tested: 10,000
✅ Valid tokens: 7,243
❌ Invalid tokens: 2,103
⏳ Temporary errors: 654
🎯 Delivery Rate: 79.0%
(7,897 tokens can receive notifications)
📈 Success Rate: 72.4%
(immediate validation success)
Production workflow: validate, analyze, send
Here's our real-world process for a 500K-user campaign:
Phase 1: Dry-run validation (2 minutes)
// Step 1: Create validation job
const validationJob = {
jobId: 'dryrun-blackfriday-2025',
title: 'Black Friday Sale - 50% Off Everything!',
content: 'Limited time offer. Shop now!',
// Target filters
gender: undefined, // All genders
ageMin: 20,
ageMax: 65,
platform_type: undefined, // iOS + Android
// ★ Sample size: 1-2% of target
limit: 10000, // Out of 500K total
};
// Step 2: Run validation (isDryRun = true)
console.log('========== PHASE 1: VALIDATION ==========');
const validationResult = await firebaseService.sendConditionalNotifications({
...validationJob,
// Service internally uses isDryRun: true flag
});
console.log('Validation complete:', validationResult);
// Output:
// {
// success: true,
// totalTokens: 10000,
// validTokens: 7243,
// invalidTokens: 2757,
// deliveryRate: 79.0,
// isDryRun: true
// }
Phase 2: Analyze results and decide (30 seconds)
// Get detailed error breakdown
const stats = await firebaseService.getDeliveryStats('dryrun-blackfriday-2025');
console.log(`
📊 Detailed Validation Analysis:
=================================
Total Tested: ${stats.total.toLocaleString()}
✅ Valid: ${stats.success.toLocaleString()} (${stats.successRate}%)
❌ Invalid: ${stats.invalidToken.toLocaleString()}
⏳ Temporary: ${stats.temporary.toLocaleString()}
🚫 Quota: ${stats.quota}
❓ Other: ${stats.other}
🎯 Delivery Rate: ${stats.deliveryRate}%
`);
// Decision logic
if (stats.deliveryRate < 70) {
console.error('❌ ABORT: Delivery rate too low!');
console.log('Action required: Database cleanup before sending');
// Get list of invalid tokens for cleanup
const invalidTokens = await firebaseService.getInvalidTokens(
'dryrun-blackfriday-2025'
);
console.log(`Found ${invalidTokens.length} invalid tokens to remove`);
// TODO: Implement automated cleanup
} else if (stats.deliveryRate < 80) {
console.warn('⚠️ CAUTION: Delivery rate acceptable but improvable');
console.log('Recommendation: Proceed, but schedule cleanup for next week');
} else {
console.log('✅ EXCELLENT: Delivery rate is healthy. Safe to proceed!');
}
// Estimate actual campaign reach
const totalTarget = 500000;
const estimatedReach = Math.floor(totalTarget * (stats.deliveryRate / 100));
const estimatedFailed = totalTarget - estimatedReach;
console.log(`
📈 Campaign Projection:
========================
Target Audience: ${totalTarget.toLocaleString()} users
Estimated Reach: ${estimatedReach.toLocaleString()} devices (${stats.deliveryRate}%)
Expected Failures: ${estimatedFailed.toLocaleString()} invalid tokens
Time Estimate: ~${Math.ceil(estimatedReach / 10000)} minutes
(Processing ${Math.ceil(estimatedReach / 500)} chunks at 2 sec/chunk)
`);
Example decision output:
📊 Detailed Validation Analysis:
=================================
Total Tested: 10,000
✅ Valid: 7,243 (72.4%)
❌ Invalid: 2,103
⏳ Temporary: 654
🚫 Quota: 0
❓ Other: 0
🎯 Delivery Rate: 79.0%
✅ EXCELLENT: Delivery rate is healthy. Safe to proceed!
📈 Campaign Projection:
========================
Target Audience: 500,000 users
Estimated Reach: 395,000 devices (79.0%)
Expected Failures: 105,000 invalid tokens
Time Estimate: ~80 minutes
(Processing 790 chunks at 2 sec/chunk)
Phase 3: Production send (63 minutes)
console.log('========== PHASE 2: PRODUCTION SEND ==========');
// Create production job (isDryRun = false)
const productionJob = {
jobId: 'production-blackfriday-2025',
title: 'Black Friday Sale - 50% Off Everything!',
content: 'Limited time offer. Shop now!',
// Same filters as validation
gender: undefined,
ageMin: 20,
ageMax: 65,
platform_type: undefined,
// ★ Full campaign (no limit)
limit: undefined, // Send to all matching users
};
const productionResult = await firebaseService.sendConditionalNotifications(productionJob);
console.log(`
✅ Campaign Complete:
- Total sent: ${productionResult.sendStats.totalSent.toLocaleString()}
- Successful: ${productionResult.sendStats.successCount.toLocaleString()}
- Failed: ${productionResult.sendStats.totalFailed.toLocaleString()}
- Duration: ${productionResult.timing.durationMinutes} minutes
`);
Phase 4: Validation accuracy check (10 seconds)
// Compare dry-run prediction to actual results
const productionStats = await firebaseService.getDeliveryStats(
'production-blackfriday-2025'
);
const predictionError = Math.abs(stats.deliveryRate - productionStats.deliveryRate);
const reachError = Math.abs(estimatedReach - productionStats.success);
const accuracy = 100 - (predictionError);
console.log(`
🎯 Prediction vs Reality:
==========================
DRY-RUN PREDICTION:
- Delivery rate: ${stats.deliveryRate}%
- Estimated reach: ${estimatedReach.toLocaleString()}
PRODUCTION ACTUAL:
- Delivery rate: ${productionStats.deliveryRate}%
- Actual reach: ${productionStats.success.toLocaleString()}
ACCURACY:
- Delivery rate error: ${predictionError.toFixed(1)}%
- Reach error: ${reachError.toLocaleString()} users (${((reachError/estimatedReach)*100).toFixed(1)}%)
- Overall accuracy: ${accuracy.toFixed(1)}%
${predictionError < 2 ? '✅ Excellent prediction!' :
predictionError < 5 ? '⚠️ Acceptable prediction' :
'❌ Poor prediction - review sampling strategy'}
`);
Typical accuracy (our production data):
🎯 Prediction vs Reality:
==========================
DRY-RUN PREDICTION:
- Delivery rate: 79.0%
- Estimated reach: 395,000
PRODUCTION ACTUAL:
- Delivery rate: 80.2%
- Actual reach: 401,000
ACCURACY:
- Delivery rate error: 1.2%
- Reach error: 6,000 users (1.5%)
- Overall accuracy: 98.8%
✅ Excellent prediction!
Why production beats prediction:
- Temporary errors in dry-run often succeed in production (~80% success rate)
- Time gap allows network conditions to stabilize
- FCM infrastructure routing differences
Sampling strategies: the 1-2% rule
Question: Should you validate all 500K tokens or sample?
Answer: Sample. Here's why:
Statistical validity:
- 10K sample from 500K population = 95% confidence, ±1% margin of error
- 20K sample = 98% confidence, ±0.7% margin
- 50K sample = 99% confidence, ±0.5% margin
Diminishing returns above 2%:
Sample Size | Confidence | Margin of Error | Validation Time
-----------:|:----------:|:---------------:|----------------
5,000 | 90% | ±1.4% | 1 min
10,000 | 95% | ±1.0% | 2 min
20,000 | 98% | ±0.7% | 4 min
50,000 | 99% | ±0.5% | 10 min
100,000 | 99.5% | ±0.3% | 20 min
500,000 | 100% | 0.0% | 100 min
Our rule: 1-2% sample size
- 100K campaign → 1-2K sample
- 500K campaign → 5-10K sample
- 1M campaign → 10-20K sample
Critical: Random sampling to avoid bias
// ❌ BAD: Samples most active users only
const sample = await memberRepository
.createQueryBuilder('m')
.where(/* filters */)
.orderBy('m.last_login', 'DESC') // Sorted by activity
.take(10000)
.getMany();
// Result: 85% delivery rate (biased upward)
// Reality: 72% delivery rate (includes dormant users)
// ✅ GOOD: Random sample across entire dataset
const sample = await memberRepository
.createQueryBuilder('m')
.where(/* filters */)
.orderBy('NEWID()') // Random order (MSSQL)
// .orderBy('RAND()') // Random order (MySQL)
// .orderBy('RANDOM()') // Random order (PostgreSQL)
.take(10000)
.getMany();
// Result: 79% delivery rate (accurate)
// Reality: 80.2% delivery rate (within 1.2% margin)
Pro tip: Stratified sampling for heterogeneous audiences
// If audience has distinct segments, sample each proportionally
const iosSample = await getSample({ platform: 'ios' }, 0.02); // 2%
const androidSample = await getSample({ platform: 'android' }, 0.02);
const combinedStats = calculateWeightedDeliveryRate([
{ platform: 'ios', stats: iosSample.stats, weight: iosSample.totalPopulation },
{ platform: 'android', stats: androidSample.stats, weight: androidSample.totalPopulation },
]);
Edge cases and lessons learned
Gotcha 1: Dry-run IS NOT quota-free
Misconception: "Dry-run doesn't count toward rate limits."
Reality: Dry-run DOES count toward FCM rate limits (though at a lower rate).
What happened:
// Our first attempt (disaster)
const isDryRun = true;
for (let i = 0; i < 1000; i++) {
// No delay - sent 500K tokens in 5 minutes
const response = await messaging.sendEach(messages, isDryRun);
}
// Result: messaging/quota-exceeded after chunk 200
Solution: Apply same rate limiting as production
for (let i = 0; i < chunks.length; i++) {
const response = await messaging.sendEach(messages[i], isDryRun);
// ✅ Rate limiting (even in dry-run)
if (i < chunks.length - 1) {
await delay(2000); // 2 seconds between chunks
}
}
Gotcha 2: Database performance with large validations
Problem: Storing 500K dry-run logs slowed down queries.
-- This query became slow
SELECT * FROM push_notification_log
WHERE job_id = 'production-campaign-123'
AND is_dry_run = 0;
-- Execution time: 8 seconds (unacceptable)
Solution: Compound index
@Entity({ name: 'push_notification_log' })
@Index(['job_id', 'is_dry_run', 'sent_at']) // ✅ Compound index
export class PushNotificationLog {
// ...
}
Result: Query time: 8s → 0.3s
Gotcha 3: Dry-run vs production discrepancies
Observed pattern:
- Dry-run: 79.0% delivery rate
- Production: 80.2% delivery rate
- Difference: +1.2%
Why production performs better:
-
Temporary errors resolve: ~80% of
unavailableerrors in dry-run succeed in production - Time gap: 2-5 minutes between validation and send allows network recovery
- Infrastructure differences: Dry-run may use stricter validation paths
Our adjustment formula:
// Apply 1-2% buffer to dry-run predictions
const adjustedDeliveryRate = dryRunRate * 1.015;
const estimatedReach = Math.floor(totalTarget * (adjustedDeliveryRate / 100));
console.log(`
Conservative estimate (dry-run): ${Math.floor(totalTarget * (dryRunRate / 100))}
Adjusted estimate (+1.5%): ${estimatedReach}
`);
Gotcha 4: Sampling bias from user behavior patterns
Problem: Validation at 9 AM showed 85% delivery rate. Production send at 9 PM showed 76% delivery rate.
Root cause: User activity patterns.
- 9 AM sample: Captured active morning users (fresh tokens)
- 9 PM send: Included evening users who hadn't opened app in weeks (stale tokens)
Solution: Time-aware sampling
// If sending at 9 PM, validate at similar time
const now = new Date();
const hour = now.getHours();
if (hour >= 20 || hour <= 6) {
console.log('Sending during off-peak hours');
console.log('Recommendation: Validate during same time window for accuracy');
}
// Or: Account for temporal variance in estimates
const timeVarianceFactor = hour >= 20 ? 0.95 : 1.0;
const adjustedReach = estimatedReach * timeVarianceFactor;
Real production metrics: 6 months of dry-run usage
After implementing dry-run validation (January 2025 - July 2025):
Validation accuracy (10,000+ campaigns):
Metric | Value
--------------------------|--------
Average prediction error | 1.8%
95th percentile error | 3.2%
Best case | 0.3%
Worst case | 5.7%
Time savings:
Before dry-run:
- Average campaign duration: 80 minutes
- Wasted on invalid tokens: 24 minutes/campaign
After dry-run:
- Validation time: 2 minutes
- Actual send time: 56 minutes (valid tokens only)
- Total: 58 minutes
Savings: 22 minutes/campaign × 50 campaigns/month = 1,100 min/month = 18 hours/month
Database quality improvement:
Month | Invalid Token Rate
---------|-------------------
Jan 2025 | 30%
Feb 2025 | 27%
Mar 2025 | 23%
Apr 2025 | 18%
May 2025 | 15%
Jun 2025 | 12%
Reason: Automated cleanup after each validation
Cost impact:
Server time saved: 18 hours/month × 12 months = 216 hours/year
Value: 216 hours × $0.10/minute × 60 minutes = $1,296/year
Database operations saved: 105K failed writes/campaign × 50 campaigns/month = 5.25M writes/month
Value: 5.25M writes × $0.0001 = $525/month = $6,300/year
Total estimated savings: $7,596/year
Developer confidence:
Before: "I hope this works 🤞"
Team questions:
- "How many users will we actually reach?"
- "Should we send to everyone or clean database first?"
- "What if 50% fail?"
After: "We'll reach 395,000 ± 1.5%"
Team confidence:
- "Delivery rate: 79.0% (validated 10 minutes ago)"
- "Estimated reach: 395,000 devices"
- "Send time: ~63 minutes"
- "Recommendation: Proceed with send"
When to use dry-run validation
Always validate for:
✅ Large campaigns (>100K users)
✅ High-value campaigns (product launches, Black Friday)
✅ New audience segments (first-time targeting)
✅ After database migrations
✅ Monthly/quarterly database health checks
Skip validation for:
❌ Small campaigns (<1K users) - overhead not worth it
❌ Time-critical alerts (breaking news, emergencies)
❌ Daily recurring sends to proven audiences
❌ Transactional notifications (order confirmations)
Sample size guidelines:
Campaign Size | Sample Size | Confidence Level
--------------:|------------:|------------------
10,000 | 1,000 | 95%
100,000 | 2,000 | 95%
500,000 | 10,000 | 95%
1,000,000 | 20,000 | 98%
5,000,000 | 50,000 | 99%
API endpoint for delivery statistics
We exposed dry-run results via REST API for dashboard integration:
// GET /api/campaigns/:jobId/delivery-stats
async getDeliveryStats(jobId: string): Promise {
const logs = await this.pushNotificationLog.find({
where: { job_id: jobId },
select: ['is_success', 'error_code', 'error_type']
});
if (logs.length === 0) {
throw new NotFoundException(`No logs found for job: ${jobId}`);
}
const stats = calculateDeliveryRate(logs);
// Check if this was a dry-run job
const isDryRun = logs[0]?.is_dry_run === true;
return {
jobId,
isDryRun,
total: stats.total,
success: stats.success,
invalidToken: stats.invalidToken,
temporary: stats.temporary,
quota: stats.quota,
other: stats.other,
deliveryRate: stats.deliveryRate,
successRate: stats.successRate,
// Prediction for production (if dry-run)
estimatedProductionRate: isDryRun
? parseFloat((stats.deliveryRate * 1.015).toFixed(2))
: null,
};
}
Response example:
GET /api/campaigns/dryrun-blackfriday-2025/delivery-stats
{
"jobId": "dryrun-blackfriday-2025",
"isDryRun": true,
"total": 10000,
"success": 7243,
"invalidToken": 2103,
"temporary": 654,
"quota": 0,
"other": 0,
"deliveryRate": 79.0,
"successRate": 72.4,
"estimatedProductionRate": 80.2
}
Automated token cleanup workflow
After validation, we automatically mark invalid tokens:
// Runs after each dry-run validation
async function cleanupInvalidTokens(dryRunJobId: string) {
console.log(`Starting cleanup for: ${dryRunJobId}`);
// Get all permanently invalid tokens
const invalidTokens = await pushNotificationLog.find({
where: {
job_id: dryRunJobId,
error_type: 'invalid_token', // Permanent failures only
},
select: ['member_seq', 'push_token', 'error_code'],
});
console.log(`Found ${invalidTokens.length} invalid tokens to mark`);
// Update member table (don't delete - mark as invalid)
for (const token of invalidTokens) {
await memberRepository.update(
{ seq: token.member_seq },
{
push_token_valid: false,
push_token_invalidated_at: new Date(),
push_token_invalid_reason: token.error_code,
}
);
}
console.log(`✅ Marked ${invalidTokens.length} tokens as invalid`);
// Log cleanup event
await cleanupLogRepository.save({
dry_run_job_id: dryRunJobId,
tokens_marked_invalid: invalidTokens.length,
executed_at: new Date(),
});
}
Database schema update:
ALTER TABLE member
ADD push_token_valid BIT DEFAULT 1,
ADD push_token_invalidated_at DATETIME2 NULL,
ADD push_token_invalid_reason VARCHAR(50) NULL;
CREATE INDEX idx_member_valid_token
ON member(push_token_valid, push_token);
Future queries automatically exclude invalid tokens:
// Before cleanup automation
const tokens = await memberRepository.find({
where: { /* filters */ },
select: ['push_token'],
});
// Returns 500K tokens (30% invalid = wasted processing)
// After cleanup automation
const tokens = await memberRepository.find({
where: {
/* filters */,
push_token_valid: true, // ✅ Only valid tokens
},
select: ['push_token'],
});
// Returns 350K tokens (0% invalid = efficient processing)
Key takeaways
1. Dry-run is a 2-minute investment that saves hours
- Validates 500K tokens in 100 minutes → 10K sample in 2 minutes
- Predicts delivery rate with 95-98% accuracy
- Zero user impact (no notifications sent)
2. One flag, same code path
const isDryRun = true; // Validation mode
const isDryRun = false; // Production mode
- Eliminates code duplication
- Reduces maintenance burden
- Prevents logic drift between validation and production
3. Sample intelligently (1-2% rule)
- 10K sample from 500K = 95% confidence, ±1% margin
- Random sampling prevents bias
- Diminishing returns above 2%
4. Store everything for analysis
-
is_dry_runflag in database - Error classification (
invalid_token,temporary, etc.) - Historical trends ("Token quality improved 70% → 85%")
5. Automate cleanup after validation
- Mark invalid tokens immediately
- Future campaigns skip them automatically
- Database quality improves over time
6. Validate before high-stakes sends
- Product launches
- Black Friday campaigns
- New audience segments
- After database migrations
ROI calculation:
Time investment: 2 minutes/campaign
Time saved: 22 minutes/campaign
Net savings: 20 minutes × 50 campaigns/month = 1,000 minutes/month
Cost savings: ~$120/month server time + $40/month DB operations = $160/month
Annual ROI: $1,920
Confidence gained: Priceless
If you're sending push notifications at scale, dry-run validation is the highest-ROI 10 lines of code you can write.
Top comments (0)