Building a 29-skill map from an append-only answer log

I build satultra, a digital SAT practice site. After a practice test, most tools lead with one number, and a 1290 says how you did, not what to study next.

The digital SAT is built around 29 skills: College Board's question bank lists 10 in Reading and Writing and 19 in Math. If every answer is filed under one of them, the results page can show where the points went. Here's how that works in a React and Firestore app: one taxonomy file, an append-only event log, pure reducers and a planner.

One taxonomy file, one canonical id

Over several imports, the same skill showed up as "Central Ideas and Details", "central ideas & details" or just 1. So the first rule was one source of truth, subcategoryTaxonomy.json: eight domains and 29 skills, each with a kebab-case id, a domain and its aliases.

{
  "id": "central-ideas-details",
  "numericId": 1,
  "name": "Central Ideas and Details",
  "domain": "information-ideas",
  "aliases": ["central ideas and details", "central ideas & details"]
}

The kebab id is canonical everywhere: Firestore document ids, events, the planner. numericId survives only for older hand-authored questions, and toCanonicalSubcategoryId() maps any alias or legacy id to the canonical one. The API keeps a copy of the file, and a verify script keeps the two in sync.

If I did it again, I'd fix the canonical id before the second import. Our question documents still carry three versions of the field: subcategory, subCategory and subcategoryId.

Log every answer as an event

Every answer that counts, from a quiz, a drill or a full practice test, goes into an activityEvents collection that clients can append to but never edit. Quiz and test completions, lesson views and flashcard sessions go there too. When a student deletes a test result, the server removes its events and replays the rest.

// a question_attempt event's payload
{
  source: 'smartquiz',          // or exam, drill, lesson_quiz, word_quiz
  questionId: 'q_8f2c',
  subcategoryId: 'words-in-context',
  correct: false,
  timeSpentMs: 48000,
  parentId: quizId,
}

The per-skill numbers are derived from that log. Because the log is the source of truth, the derived state can be rebuilt at any time by replaying events, and a backfill script does exactly that for older history.

Not every finished sitting is evidence

A test module left blank, more than half unanswered, or clicked through in under two minutes is judged once, when the result is saved. Its answers never become skill evidence, though the sitting itself is still logged. A five-question quiz "done" in under a minute gets the same treatment.

Blank questions inside a real attempt count as wrong for the test score, but they aren't attempts, so they don't drag down skill accuracy. Before these rules, skipped modules produced skill cards like "Linear Functions: 0% over 17 questions".

Pure reducers turn the log into skill state

Derived state lives at users/{uid}/skillState/{skillId}. It's computed by pure functions: no I/O and no Date.now(), since timestamps come in on the event. The same module runs in the browser for live updates and in Node for replays, so both paths apply the same logic, and a replay can rebuild whatever a failed live update left behind.

export function reduceSkillState(prev, a) {
  const s = { ...prev, bySource: { ...prev.bySource } };
  s.attempts += 1;
  if (a.correct) s.correct += 1;
  s.accuracy = pct(s.correct, s.attempts);

  s.lastResults = [...prev.lastResults, !!a.correct].slice(-LAST_N); // last 20
  s.accuracyLast10 = accuracyOf(s.lastResults.slice(-10));
  s.rollingAccuracy =
    prev.rollingAccuracy === null
      ? (a.correct ? 100 : 0)
      : Math.round(EWMA_ALPHA * (a.correct ? 100 : 0) + (1 - EWMA_ALPHA) * prev.rollingAccuracy);
  s.trend = computeTrend(s.lastResults);
  s.lastPracticedTs = a.ts || null;
  // ...per-source counts and average time per question
  return s;
}

Lifetime accuracy is kept, but the weakest-skills list and the score estimate use accuracyLast10. A student who was weak at Boundaries a month ago and has since fixed it shouldn't keep seeing it at the top of the list.

Is a skill improving or slipping?

The trend compares the last 10 answers in a skill with up to 10 before them:

export function computeTrend(lastResults) {
  if (!lastResults || lastResults.length < 10) return 'insufficient';
  const recent = lastResults.slice(-10);
  const prior = lastResults.slice(0, Math.max(0, lastResults.length - 10));
  if (prior.length < 5) return 'insufficient';
  const delta = accuracyOf(recent) - accuracyOf(prior);
  if (delta >= TREND_DELTA_PTS) return 'improving'; // 15 points
  if (delta <= -TREND_DELTA_PTS) return 'declining';
  return 'stable';
}

So a trend needs at least 15 answers and a swing of 15 points. Anything smaller is treated as noise at this sample size.

A declining skill is labeled "slipping". A skill with five or more attempts and nothing in the last 8 days is labeled "fading". Neither label changes the numbers: idle time doesn't decay accuracy, it only flags the skill for a short review.

Below the skill level, individual concepts get the same treatment. A concept the student struggled with and then answered right three times in a row is marked recovered, and two misses in a row after that raise a regression flag. Those flags rank first in a four-item list of what to work on next, followed by repeated misses, slipping skills, fading skills and the lowest recent accuracy. The built-in coach reads the same derived state.

From skill accuracy to a study plan

The planner, generatePlan(), is deliberately simple:

1. Sort the 29 skills weakest first by recent accuracy. Skills with no attempts count as 50%, so they sit between real weaknesses and real strengths.
2. Give each skill a 20-minute lesson, skipped if already done, and a 15-minute, 10-question practice set.
3. Lay that queue across the days from tomorrow to the day before the test, skipping rest days, up to the student's minutes per day.
4. Keep the last two days for reviews of the weakest domains.
5. If it doesn't all fit, keep the weakest skills and set truncated, so the page can say so.

When a student finishes a lesson or quiz elsewhere in the app, the plan reconciles on the next load and ticks off the matching task. Each quiz can credit only one task.

A prerendering bug worth knowing about

The public pages are part of the same Vite SPA, served by one Express app on Render. For crawlers and a faster first paint, a build step opens 13 public routes in headless Chrome with Puppeteer and saves the rendered HTML. Express serves those snapshots, and the SPA boots on top.

The snapshot also captured scripts that SDKs had injected at runtime. Serialized back to HTML, an injected <script src> loses the implicit async it had and becomes a parser-blocking script at the end of <head>. PostHog's config script did exactly that: the parser stalled on a round trip to a new origin, the async module entry ran before #root existed, and createRoot(null) threw React error #299. On fast connections, the app never mounted.

The fix has two parts. Before saving a snapshot, drop every script that isn't in the built index.html template; the SDKs inject themselves again when the app boots. Then strip async from the module entry so it stays deferred. As a backstop, the entry itself waits for DOMContentLoaded before mounting.

Stack

• React 18 and React Router 6 with lazy-loaded pages, built with Vite
• Firebase Auth and Firestore, with firebase-admin on the server
• One Node/Express app on Render serving both /api and the built site
• Stripe Checkout for one-time passes, no subscriptions
• Puppeteer for prerendering

satultra is free to start, with the diagnostic, three full-length tests, the question bank and the planner, at satultra.com. Lessons come with a paid pass. If you've built progress tracking for a learning app, I'd like to hear how you handled recent versus lifetime accuracy, and whether you decay old answers.