Part 6 of CrisisCore Build Log — Start here: https://dev.to/crisiscoresystems/testing-privacy-preserving-analytics-verifying-that-insights-dont-leak-identity-e37
We test crisis detection thoroughly, but how do we test that the system actually facilitates recovery?
We've spent thousands of engineering hours perfecting crisis detection. We can identify a panic attack from navigation patterns. We catch dissociation from unexplained inactivity. We detect sensory overload from preference churning.
If you want privacy-first, offline health tech to exist without surveillance funding it: sponsor the build → https://paintracker.ca/sponser
But here's the uncomfortable question: does any of this actually help people get better?
Detection without recovery facilitation is just sophisticated surveillance. We're not building a system to watch people suffer more accurately—we're building a system to help them heal. That means we need to test not just "did we detect the crisis?" but "did the user's trajectory improve over time?"
This is the hardest testing we do.
The Longitudinal Testing Challenge
Unit tests run in milliseconds. Integration tests run in seconds. Recovery happens over weeks and months.
// This is NOT how recovery works
test('user recovers from crisis', async () => {
triggerCrisis();
await wait(100);
expect(user.isRecovered).toBe(true); // ❌ Recovery takes weeks, not milliseconds
});
We need a different approach: longitudinal pattern simulation.
interface LongitudinalSimulation {
userId: string;
duration: { weeks: number };
// Weekly snapshots
weeklySnapshots: WeeklySnapshot[];
// Trajectory metrics
trajectoryMetrics: {
crisisFrequency: number[]; // Crises per week
crisisSeverity: number[]; // Average severity when crises occur
recoveryTime: number[]; // Hours to return to baseline
featureEngagement: number[]; // Features actively used
selfReportedWellbeing: number[]; // If available
};
// Recovery indicators
recoveryIndicators: {
sustainedImprovement: boolean; // 3+ weeks of improvement
setbackHandling: 'resilient' | 'fragile' | 'unknown';
copingStrategyAdoption: string[];
};
}
interface WeeklySnapshot {
week: number;
entriesLogged: number;
crisisEvents: CrisisEvent[];
featuresUsed: string[];
preferenceChanges: PreferenceChange[];
sessionPatterns: SessionPattern[];
}
Simulating Recovery Trajectories
Real recovery isn't linear. People have setbacks. Good weeks follow bad weeks. We model this:
class RecoveryTrajectorySimulator {
/**
* Simulate a realistic recovery trajectory over time
*/
simulateRecovery(config: RecoverySimulationConfig): LongitudinalSimulation {
const snapshots: WeeklySnapshot[] = [];
// Recovery follows a noisy downward trend in crisis frequency
const baselineImprovement = config.expectedImprovementRate; // e.g., 0.15 = 15% weekly improvement
for (let week = 1; week <= config.durationWeeks; week++) {
// Base crisis probability decreases over time
const baseCrisisProbability = config.initialCrisisRate *
Math.pow(1 - baselineImprovement, week - 1);
// Add realistic noise (setbacks happen)
const noise = this.generateRealisticNoise(week, config);
const actualCrisisProbability = Math.max(0.05,
Math.min(0.9, baseCrisisProbability + noise));
// Simulate the week
const snapshot = this.simulateWeek(week, actualCrisisProbability, config);
snapshots.push(snapshot);
}
return this.buildLongitudinalSimulation(snapshots, config);
}
private generateRealisticNoise(week: number, config: RecoverySimulationConfig): number {
// Setbacks are more likely early in recovery
const setbackProbability = 0.3 * Math.exp(-week / 8);
if (Math.random() < setbackProbability) {
// Setback: increase crisis probability temporarily
return 0.2 + Math.random() * 0.3;
}
// Normal variation
return (Math.random() - 0.5) * 0.1;
}
private simulateWeek(
week: number,
crisisProbability: number,
config: RecoverySimulationConfig
): WeeklySnapshot {
const sessionsThisWeek = 5 + Math.floor(Math.random() * 10);
const crisisEvents: CrisisEvent[] = [];
const featuresUsed = new Set<string>();
// Core features always available
featuresUsed.add('painLevel');
featuresUsed.add('quickLog');
for (let session = 0; session < sessionsThisWeek; session++) {
// Determine if this session includes a crisis
if (Math.random() < crisisProbability / sessionsThisWeek) {
crisisEvents.push(this.generateCrisisEvent(week, config));
}
// Feature usage expands as recovery progresses
this.addProgressiveFeatures(week, featuresUsed, config);
}
return {
week,
entriesLogged: sessionsThisWeek,
crisisEvents,
featuresUsed: Array.from(featuresUsed),
preferenceChanges: this.simulatePreferenceChanges(week, crisisEvents.length),
sessionPatterns: this.simulateSessionPatterns(sessionsThisWeek, crisisEvents)
};
}
private addProgressiveFeatures(
week: number,
features: Set<string>,
config: RecoverySimulationConfig
): void {
// Features unlock based on recovery progress and user readiness
const featureTimeline = {
2: ['notes', 'moodTracking'],
3: ['medicationLog', 'sleepQuality'],
4: ['triggerIdentification', 'patternView'],
6: ['trendAnalysis', 'correlationInsights'],
8: ['predictiveAlerts', 'exportReports'],
12: ['socialSharing', 'providerIntegration']
};
for (const [unlockWeek, weekFeatures] of Object.entries(featureTimeline)) {
if (week >= parseInt(unlockWeek)) {
// Not all users adopt all features - simulate realistic adoption
for (const feature of weekFeatures) {
if (Math.random() < config.featureAdoptionRate) {
features.add(feature);
}
}
}
}
}
}
Testing Progressive Feature Exposure
A key recovery mechanism is gradual exposure to complexity. Users in crisis need simplicity. Recovering users can handle—and benefit from—more sophisticated tools:
describe('Progressive Feature Exposure', () => {
it('limits features during early recovery', async () => {
const simulation = simulator.simulateRecovery({
durationWeeks: 12,
initialCrisisRate: 0.6,
expectedImprovementRate: 0.12,
featureAdoptionRate: 0.7
});
// Week 1: Only essential features
const week1Features = simulation.weeklySnapshots[0].featuresUsed;
expect(week1Features).toContain('painLevel');
expect(week1Features).toContain('quickLog');
expect(week1Features).not.toContain('triggerAnalysis');
expect(week1Features).not.toContain('patternRecognition');
expect(week1Features).not.toContain('predictiveAlerts');
});
it('expands features as recovery progresses', async () => {
const simulation = simulator.simulateRecovery({
durationWeeks: 12,
initialCrisisRate: 0.6,
expectedImprovementRate: 0.12,
featureAdoptionRate: 0.7
});
// Week 4: Intermediate features available
const week4Features = simulation.weeklySnapshots[3].featuresUsed;
expect(week4Features).toContain('patternView');
expect(week4Features).toContain('triggerIdentification');
// Week 8+: Advanced features for recovered users
const week8Features = simulation.weeklySnapshots[7].featuresUsed;
expect(week8Features.length).toBeGreaterThan(week4Features.length);
});
it('correlates feature expansion with crisis reduction', async () => {
const simulation = simulator.simulateRecovery({
durationWeeks: 12,
initialCrisisRate: 0.6,
expectedImprovementRate: 0.12,
featureAdoptionRate: 0.7
});
const earlyWeeks = simulation.weeklySnapshots.slice(0, 4);
const lateWeeks = simulation.weeklySnapshots.slice(-4);
const earlyCrisisRate = earlyWeeks.reduce((sum, w) => sum + w.crisisEvents.length, 0) / 4;
const lateCrisisRate = lateWeeks.reduce((sum, w) => sum + w.crisisEvents.length, 0) / 4;
const earlyFeatureCount = earlyWeeks.reduce((sum, w) => sum + w.featuresUsed.length, 0) / 4;
const lateFeatureCount = lateWeeks.reduce((sum, w) => sum + w.featuresUsed.length, 0) / 4;
// More features AND fewer crises = recovery working
expect(lateCrisisRate).toBeLessThan(earlyCrisisRate);
expect(lateFeatureCount).toBeGreaterThan(earlyFeatureCount);
});
});
Recovery Metric Validation
How do we know our metrics actually measure recovery? We validate against multiple signals:
interface RecoveryMetrics {
// Primary metrics
crisisFrequencyTrend: TrendDirection; // Should decrease
crisisSeverityTrend: TrendDirection; // Should decrease
recoveryTimeTrend: TrendDirection; // Should decrease (faster recovery)
// Secondary metrics
sessionConsistency: number; // Regular usage = engagement
featureDepthScore: number; // Using advanced features
selfEfficacyIndicators: number; // Taking control of tracking
// Validation metrics
selfReportCorrelation: number; // Do our metrics match user's perception?
clinicalCorrelation?: number; // If available: match clinical assessment
}
class RecoveryMetricValidator {
validateMetrics(simulation: LongitudinalSimulation): ValidationResult {
const metrics = this.calculateRecoveryMetrics(simulation);
return {
// Primary validation: Are trends in the right direction?
primaryValidation: {
crisisFrequencyDecreasing: metrics.crisisFrequencyTrend === 'decreasing',
severityDecreasing: metrics.crisisSeverityTrend === 'decreasing',
recoveryFaster: metrics.recoveryTimeTrend === 'decreasing'
},
// Secondary validation: Engagement patterns
secondaryValidation: {
consistentUsage: metrics.sessionConsistency > 0.6,
featureProgression: metrics.featureDepthScore > 0.4,
increasingAutonomy: metrics.selfEfficacyIndicators > 0.5
},
// Cross-validation: Internal consistency
crossValidation: {
metricsCorrelate: this.checkMetricCorrelation(metrics),
noContradictions: this.checkForContradictions(metrics)
}
};
}
private checkMetricCorrelation(metrics: RecoveryMetrics): boolean {
// If crisis frequency is down, severity should also tend down
// If recovery time is improving, feature engagement should be up
const correlations = [
{ a: metrics.crisisFrequencyTrend, b: metrics.crisisSeverityTrend, expected: 'same' },
{ a: metrics.recoveryTimeTrend, b: 'decreasing', expected: 'same' },
];
return correlations.every(c =>
c.expected === 'same' ? c.a === c.b : c.a !== c.b
);
}
}
Testing Habit Formation
Recovery isn't just about fewer crises—it's about building sustainable coping strategies:
describe('Habit Formation Testing', () => {
it('detects emerging positive habits', async () => {
const simulation = simulator.simulateRecoveryWithHabits({
durationWeeks: 8,
habitFormationRate: 0.7
});
const habitDetector = new HabitDetector();
const detectedHabits = habitDetector.analyze(simulation);
// Look for consistent usage patterns
expect(detectedHabits).toContainEqual(
expect.objectContaining({
type: 'regular_logging',
consistency: expect.any(Number),
weekFormed: expect.any(Number)
})
);
});
it('measures coping strategy adoption', async () => {
const simulation = simulator.simulateRecoveryWithHabits({
durationWeeks: 12,
copingStrategiesIntroduced: ['breathing_exercise', 'grounding_technique', 'distraction_activity']
});
const copingAnalyzer = new CopingStrategyAnalyzer();
const adoption = copingAnalyzer.measureAdoption(simulation);
// At least one strategy should be adopted by week 8
expect(adoption.strategiesAdoptedByWeek8.length).toBeGreaterThan(0);
// Adopted strategies should show usage during crisis
for (const strategy of adoption.strategiesAdoptedByWeek8) {
expect(strategy.usedDuringCrisis).toBe(true);
expect(strategy.reportedHelpfulness).toBeGreaterThan(0.5);
}
});
it('verifies app encourages sustainable practices', async () => {
const simulation = simulator.simulateRecoveryWithHabits({
durationWeeks: 16,
habitFormationRate: 0.8
});
// Check that app doesn't create dependency
const dependencyIndicators = analyzeDependency(simulation);
// Users should be able to skip days without anxiety
expect(dependencyIndicators.anxietyOnMissedDays).toBeLessThan(0.2);
// Usage should feel voluntary, not compulsive
expect(dependencyIndicators.compulsiveUsagePatterns).toBe(false);
// Skills should transfer outside the app
expect(dependencyIndicators.reportedRealWorldCopingImprovement).toBeGreaterThan(0.6);
});
});
class HabitDetector {
analyze(simulation: LongitudinalSimulation): DetectedHabit[] {
const habits: DetectedHabit[] = [];
// Check for regular logging habit
const loggingConsistency = this.measureLoggingConsistency(simulation);
if (loggingConsistency.isHabit) {
habits.push({
type: 'regular_logging',
consistency: loggingConsistency.score,
weekFormed: loggingConsistency.formationWeek,
sustainedWeeks: loggingConsistency.sustainedWeeks
});
}
// Check for proactive feature use (not just reactive)
const proactiveUse = this.measureProactiveEngagement(simulation);
if (proactiveUse.isHabit) {
habits.push({
type: 'proactive_tracking',
consistency: proactiveUse.score,
weekFormed: proactiveUse.formationWeek,
sustainedWeeks: proactiveUse.sustainedWeeks
});
}
// Check for coping strategy habit
const copingHabits = this.measureCopingStrategyHabits(simulation);
habits.push(...copingHabits);
return habits;
}
private measureLoggingConsistency(simulation: LongitudinalSimulation): HabitMeasurement {
const weeklyEntries = simulation.weeklySnapshots.map(w => w.entriesLogged);
// A habit is formed when variance drops and mean stabilizes
const recentWeeks = weeklyEntries.slice(-4);
const variance = this.calculateVariance(recentWeeks);
const mean = recentWeeks.reduce((a, b) => a + b, 0) / recentWeeks.length;
// Low variance + consistent entries = habit
const isHabit = variance < 2 && mean >= 3;
// Find when habit formed (first sustained low-variance period)
let formationWeek = -1;
for (let i = 3; i < weeklyEntries.length; i++) {
const window = weeklyEntries.slice(i - 3, i + 1);
if (this.calculateVariance(window) < 2) {
formationWeek = i - 2;
break;
}
}
return {
isHabit,
score: isHabit ? 1 - (variance / 10) : 0,
formationWeek,
sustainedWeeks: formationWeek > 0 ? weeklyEntries.length - formationWeek : 0
};
}
}
Relapse Detection Testing
Recovery isn't always forward. We need to detect when someone is backsliding:
describe('Relapse Detection', () => {
it('detects early warning signs of relapse', async () => {
const simulation = simulator.simulateRecoveryWithRelapse({
durationWeeks: 16,
relapseWeek: 12,
earlyWarningWeek: 10 // Signs appear 2 weeks before full relapse
});
const relapseDetector = new RelapseDetector();
const warnings = relapseDetector.analyze(simulation);
// Should detect warning signs at week 10-11
const earlyWarnings = warnings.filter(w => w.week >= 10 && w.week < 12);
expect(earlyWarnings.length).toBeGreaterThan(0);
// Warning should come BEFORE full relapse
const firstWarningWeek = Math.min(...warnings.map(w => w.week));
expect(firstWarningWeek).toBeLessThan(12);
});
it('distinguishes setback from relapse', async () => {
// Setback: temporary increase in symptoms that resolves
const setbackSimulation = simulator.simulateRecoveryWithSetback({
durationWeeks: 12,
setbackWeek: 6,
setbackDuration: 1 // Resolves in 1 week
});
// Relapse: sustained return to earlier crisis patterns
const relapseSimulation = simulator.simulateRecoveryWithRelapse({
durationWeeks: 12,
relapseWeek: 6,
relapseDuration: 4 // Persists for 4+ weeks
});
const detector = new RelapseDetector();
const setbackResult = detector.classify(setbackSimulation);
const relapseResult = detector.classify(relapseSimulation);
expect(setbackResult.classification).toBe('setback');
expect(relapseResult.classification).toBe('relapse');
});
it('triggers appropriate interventions for different severities', async () => {
const simulations = [
{ type: 'mild_setback', severity: 0.3 },
{ type: 'moderate_relapse', severity: 0.6 },
{ type: 'severe_relapse', severity: 0.9 }
].map(config => simulator.simulateWithSeverity(config));
const interventionEngine = new InterventionEngine();
for (const sim of simulations) {
const intervention = interventionEngine.recommend(sim);
if (sim.severity < 0.4) {
// Mild: Gentle reminder, no major UI changes
expect(intervention.type).toBe('gentle_reminder');
expect(intervention.uiChanges).toEqual([]);
} else if (sim.severity < 0.7) {
// Moderate: Offer support resources, consider simplification
expect(intervention.type).toBe('support_offer');
expect(intervention.uiChanges).toContain('show_coping_resources');
} else {
// Severe: Full crisis support
expect(intervention.type).toBe('crisis_support');
expect(intervention.uiChanges).toContain('simplify_interface');
expect(intervention.uiChanges).toContain('show_crisis_resources');
}
}
});
});
class RelapseDetector {
// Early warning signs we look for
private warningSignals = [
'decreased_logging_frequency',
'increased_severity_variance',
'abandoned_coping_strategies',
'feature_avoidance',
'session_shortening',
'crisis_signal_emergence'
];
analyze(simulation: LongitudinalSimulation): RelapseWarning[] {
const warnings: RelapseWarning[] = [];
for (let week = 3; week < simulation.weeklySnapshots.length; week++) {
const recentTrend = this.calculateTrend(simulation, week);
const signals = this.detectWarningSignals(simulation, week);
if (signals.length >= 2 || (signals.length >= 1 && recentTrend === 'worsening')) {
warnings.push({
week,
signals,
trend: recentTrend,
confidence: this.calculateConfidence(signals, recentTrend),
recommendedAction: this.recommendAction(signals)
});
}
}
return warnings;
}
private detectWarningSignals(
simulation: LongitudinalSimulation,
week: number
): string[] {
const currentWeek = simulation.weeklySnapshots[week];
const previousWeeks = simulation.weeklySnapshots.slice(Math.max(0, week - 3), week);
const signals: string[] = [];
// Check logging frequency
const avgPreviousEntries = previousWeeks.reduce((s, w) => s + w.entriesLogged, 0) / previousWeeks.length;
if (currentWeek.entriesLogged < avgPreviousEntries * 0.6) {
signals.push('decreased_logging_frequency');
}
// Check feature usage (avoiding features they used to use)
const previousFeatures = new Set(previousWeeks.flatMap(w => w.featuresUsed));
const currentFeatures = new Set(currentWeek.featuresUsed);
const droppedFeatures = [...previousFeatures].filter(f => !currentFeatures.has(f));
if (droppedFeatures.length >= 2) {
signals.push('feature_avoidance');
}
// Check for crisis signals re-emerging
const avgPreviousCrises = previousWeeks.reduce((s, w) => s + w.crisisEvents.length, 0) / previousWeeks.length;
if (currentWeek.crisisEvents.length > avgPreviousCrises * 1.5 && avgPreviousCrises > 0) {
signals.push('crisis_signal_emergence');
}
return signals;
}
}
Gradual Exposure Testing
As users recover, we can safely increase complexity. Testing this requires care:
describe('Gradual Exposure', () => {
it('increases complexity only when user is ready', async () => {
const simulation = simulator.simulate30DayRecovery();
// Week 1: Only essential features
expect(simulation.week1.featuresUsed).toContain('painLevel');
expect(simulation.week1.featuresUsed).not.toContain('triggerAnalysis');
// Week 4: More features as confidence grows
expect(simulation.week4.featuresUsed).toContain('patternRecognition');
expect(simulation.week4.crisisFrequency).toBeLessThan(simulation.week1.crisisFrequency);
});
it('backs off complexity during setbacks', async () => {
const simulation = simulator.simulateRecoveryWithSetback({
setbackWeek: 3
});
// Week 2: Features were expanding
const week2Complexity = measureComplexity(simulation.weeklySnapshots[1]);
// Week 3: Setback occurs
const week3Complexity = measureComplexity(simulation.weeklySnapshots[2]);
// Week 4: System should have reduced complexity
const week4Complexity = measureComplexity(simulation.weeklySnapshots[3]);
expect(week4Complexity).toBeLessThanOrEqual(week3Complexity);
expect(week4Complexity).toBeLessThan(week2Complexity);
});
it('never introduces triggering content without preparation', async () => {
const simulation = simulator.simulateRecoveryWithTriggerContent({
triggerType: 'detailed_pain_visualization',
durationWeeks: 8
});
for (const week of simulation.weeklySnapshots) {
// If trigger content was shown
if (week.featuresUsed.includes('detailed_pain_visualization')) {
// User must have opted in
expect(week.userOptIns).toContain('advanced_visualizations');
// System must have provided preparation
expect(week.preparationShown).toContain('visualization_intro');
// User must have had 3+ stable weeks
const weekIndex = simulation.weeklySnapshots.indexOf(week);
const previousWeeks = simulation.weeklySnapshots.slice(Math.max(0, weekIndex - 3), weekIndex);
const allStable = previousWeeks.every(w => w.crisisEvents.length === 0);
expect(allStable).toBe(true);
}
}
});
});
function measureComplexity(snapshot: WeeklySnapshot): number {
const featureWeights: Record<string, number> = {
'painLevel': 1,
'quickLog': 1,
'notes': 2,
'moodTracking': 2,
'medicationLog': 3,
'triggerIdentification': 4,
'patternView': 4,
'trendAnalysis': 5,
'correlationInsights': 5,
'predictiveAlerts': 6,
'detailed_pain_visualization': 7
};
return snapshot.featuresUsed.reduce((sum, feature) =>
sum + (featureWeights[feature] || 3), 0
);
}
The Recovery Dashboard
We surface recovery metrics in development for validation:
function RecoveryTestingDashboard() {
const [simulation, setSimulation] = useState<LongitudinalSimulation | null>(null);
const [metrics, setMetrics] = useState<RecoveryMetrics | null>(null);
const runSimulation = async (config: RecoverySimulationConfig) => {
const sim = await simulator.simulateRecovery(config);
setSimulation(sim);
const validator = new RecoveryMetricValidator();
setMetrics(validator.calculateRecoveryMetrics(sim));
};
return (
<div className="p-6 space-y-6">
<h2 className="text-xl font-bold">Recovery Trajectory Testing</h2>
{/* Configuration */}
<SimulationConfigPanel onRun={runSimulation} />
{/* Timeline visualization */}
{simulation && (
<div className="space-y-4">
<h3 className="font-semibold">Weekly Progression</h3>
<div className="flex gap-2 overflow-x-auto">
{simulation.weeklySnapshots.map((week, i) => (
<WeekCard
key={i}
week={week}
index={i}
showCrisisIndicator={week.crisisEvents.length > 0}
/>
))}
</div>
</div>
)}
{/* Metrics */}
{metrics && (
<div className="grid grid-cols-3 gap-4">
<MetricCard
label="Crisis Frequency Trend"
value={metrics.crisisFrequencyTrend}
good={metrics.crisisFrequencyTrend === 'decreasing'}
/>
<MetricCard
label="Feature Depth"
value={metrics.featureDepthScore.toFixed(2)}
good={metrics.featureDepthScore > 0.4}
/>
<MetricCard
label="Self-Efficacy"
value={metrics.selfEfficacyIndicators.toFixed(2)}
good={metrics.selfEfficacyIndicators > 0.5}
/>
</div>
)}
</div>
);
}
Conclusion
Testing recovery is fundamentally different from testing features. We're not checking if a button works—we're checking if a human trajectory improves over time.
Key takeaways:
Simulate time: Recovery happens over weeks, not milliseconds. Build longitudinal simulators.
Model realistic trajectories: Recovery isn't linear. Include setbacks, plateaus, and breakthroughs.
Validate metrics internally: Do your recovery metrics correlate with each other? Are there contradictions?
Test habit formation: Are we building sustainable practices, not just detecting crises?
Detect relapse early: Warning signs appear before full relapse. Test that we catch them.
Manage complexity carefully: Gradual exposure helps recovery. Test that we never expose too much too fast.
The ultimate test isn't "did our algorithm fire correctly?" It's "did this person's life get better?" We can't measure that directly in a test suite, but we can build the scaffolding that makes it more likely.
This is Part 8 of our series on building trauma-informed healthcare applications. Previous posts covered crisis detection, testing strategies, visual regression, ethics of simulation, performance, and cross-crisis calibration.
Coming Next: "Internationalization of Trauma: Testing Across Cultural Contexts"
Tags: #recovery #longitudinal-testing #healthcare #trauma-informed #testing #habit-formation #react #typescript
Support this work
- Sponsor the project (primary): https://paintracker.ca/sponser
- Star the repo (secondary): https://github.com/CrisisCore-Systems/pain-tracker
- Read the full series from the start: https://dev.to/crisiscoresystems/testing-privacy-preserving-analytics-verifying-that-insights-dont-leak-identity-e37
Top comments (0)