"You improved" is a statistical claim, and on eight attempts it is usually false

CogniPrep shows people whether their practice scores are going anywhere. That sounds like a charting problem and it is not. The moment you put an arrow next to someone's history you have made a claim, and with the sample sizes a practice product actually has, eight attempts, maybe twelve, the honest answer is very often "we cannot tell yet".

This is the module that decides, and the one performance bug that hid inside it for months.

The claim has to survive a threshold

A linear regression over a score history always returns a slope. It is never exactly zero, so if you render the sign of the slope you will tell every user a story, every time.

private determineDirection(slope: number): 'improving' | 'stable' | 'declining' {
  const threshold = 0.5; // points per session

  if (slope > threshold) return 'improving';
  if (slope < -threshold) return 'declining';
  return 'stable';
}

Half a point per session is a product decision, not a statistical one. It is roughly "would this person notice the difference after ten more attempts". Below it, "stable" is both true and more useful than a direction that will flip next week.

Confidence that cannot be talked into certainty

private calculateConfidence(dataPoints: number, rSquared: number): number {
  if (dataPoints < 2) return 0;

  // Base confidence from data points (logarithmic).
  // Reaches 0.7 at 30 points, 0.8 at 50 points.
  const dataConfidence = Math.min(0.8, Math.log(dataPoints + 1) / Math.log(51));

  // Fit confidence from R-squared, max 0.2.
  const fitConfidence = Math.max(0, rSquared) * 0.2;

  return Math.min(1, dataConfidence + fitConfidence);
}

Two properties worth stealing.

Volume is capped at 0.8. No number of sessions alone buys certainty, because a long history of noisy scores is still a noisy history. The remaining 0.2 has to be earned by the fit.

The fit contributes at most 0.2. A tidy R² on four points is a coincidence, and the cap is what stops that coincidence from outvoting the sample size. The two terms are deliberately unequal: quantity dominates, quality adjusts.

The logarithm means the curve is steep exactly where it matters. Going from 3 sessions to 8 moves confidence a long way. Going from 40 to 50 barely moves it at all, which is correct, because by then the uncertainty is not about how much data you have.

The comparison is Welch, not Student

The improvement check does not use the regression at all. It splits the history and compares two periods:

// Recent: last 30% of sessions (minimum 1, maximum 10)
const recentCount = Math.max(1, Math.min(10, Math.ceil(totalScores * 0.3)));

Proportional so it means the same thing for a 10 session history and a 50 session one, capped at 10 so a heavy user's "recent" stays recent, floored at 1 so the function has something to work with.

Then Welch's t-test rather than the standard independent samples version. That matters here specifically: someone who has genuinely improved usually got more consistent at the same time, so the two samples have different variances, which is exactly the assumption Student's t-test makes and Welch's does not.

And the verdict takes two conditions, not one:

const improved = pValue < 0.05 && percentageChange > 0;

A significant change downward is significant. It is not an improvement. Checking significance alone is one of the easiest ways to ship a cheerful message about someone getting worse.

"Insufficient data" is a return value, not an error

if (scores.length < 2) {
  return {
    improved: false,
    percentageChange: 0,
    significanceLevel: 0,
    comparisonPeriod: 'insufficient data',
  };
}

Every branch that cannot make a claim returns the same shape as every branch that can, with a period string that says so. No nulls, no throwing, no caller writing its own guard, which is how one page ends up saying "not enough data yet" and another says "0% improvement" about the same user.

The trend side does the same, and it distinguishes two cases rather than collapsing them: zero sessions gets confidence: 0, one session gets confidence: 0.5. One attempt is not nothing, it just cannot have a direction.

Now the bug

The engine originally fetched its own rows:

const { sessions: scores } = await gameSessionRepository.getUserSessions(userId, {
  gameId,
  limit: 50,
});

Reasonable. Except the route also needs those rows, because the response includes the chart points, not just the verdict:

const dataPoints = scores.map((score) => ({
  date: score.completed_at?.toISOString() || score.started_at.toISOString(),
  score: /* ... */,
  percentile: /* ... */,
}));

So every request ran the identical query twice. Worse, it ran them sequentially, because the route awaited the engine before fetching its own copy. Two round trips to Postgres for one set of rows, on a route that is called on every dashboard view.

The fix is an optional parameter, which is not elegant and is the right answer:

async computeTrend(userId: string, gameId: string, preloadedSessions?: SelectGameSession[]) {
  const scores =
    preloadedSessions ??
    (await gameSessionRepository.getUserSessions(userId, { gameId, limit: 50 })).sessions;

and a route that fetches once and hands the rows down:

// Fetch the session history ONCE and share it with the engine, rather than
// having the engine query it and then querying the identical rows again.
const { sessions: scores } = await gameSessionRepository.getUserSessions(userId, {
  gameId,
  limit: 50,
});

const trend = await analyticsEngine.computeTrend(userId, gameId, scores);

The obvious objection is: why not memoise the repository call? In a React Server Component you would, with cache(). But this is a route handler, and cache() is scoped to a render pass. There is no render here, so there is nothing for the memo to live in. This surprises people who have internalised "Next.js dedupes fetches for you", and it is worth knowing where that stops: it covers the React rendering path, not your API routes.

The general shape is one I keep meeting. A function that fetches its own data is easier to call and impossible to compose. Making the data a parameter with a fetching fallback keeps the easy call site and gives the caller that already has the rows a way to say so.

Try it

The provider hubs at cogniprep.app/games list what there is to practise, 24 providers as I write this. The trend and improvement analysis sits behind an account, because it is computed per person over their own history: sign up at cogniprep.app, play the same game a few times, and watch the confidence value refuse to get excited until there is enough history to justify it. That restraint is the feature.