DEV Community: Ahmet Koc

Every question on my IELTS site has to survive a blind solver first

Ahmet Koc — Fri, 23 Oct 2026 21:39:41 +0000

Disclosure: this post was written with an AI assistant, working from the project's code and docs.

I run ultraielts, a free site where people preparing for IELTS take full mock tests in the computer-delivered format. The interface is in Turkish because that's who it's for. The tests are in English, as on the real exam.

The worst bug a practice test can have isn't a crash. It's a question with two defensible answers, or an answer key that's simply wrong. A student marks NOT GIVEN, the key says FALSE, and from then on they stop trusting every score the site gives them. The tests on the site are original (written for it, no third-party material), and a pipeline helps write them, so I needed a check that is stricter than me rereading the key at 2 a.m.

So every listening part and reading passage goes through a blind solver before it can be published.

The solver sees only what a candidate sees

For each reading passage or listening part, a model gets exactly what a candidate gets:

the passage, or for listening, the transcript as "what the candidate hears"
the questions without answers, produced by the same function that strips the keys before the site sends a test to the browser
any figures (maps, diagrams, charts) as images

It's told to answer as a band 9 candidate would, using only the text, obeying each group's word limit, and to give a confidence level, a reason of at most 25 words that quotes the decisive words, and a note when something about the question itself makes it impossible or ambiguous.

Reusing the public "strip the answers" function matters more than it sounds. If the solver got a slightly different view than students do, it would be testing a different exam.

Scored by the site's own scoring code

The solver's answers aren't compared to the key with string equality. They're scored by the same scoreGroup function that scores real students. Word limits like "NO MORE THAN TWO WORDS AND/OR A NUMBER", spelling variants, optional words in brackets and TRUE/FALSE/NOT GIVEN aliases are all judged exactly as they would be for a person.

That removes a whole class of false alarms. "colour" against a key of "color", "7:30" against "7.30", or "forty-five" against "45" isn't a mismatch, because the scorer accepts all of those from a student. When the scorer wouldn't accept the solver's answer, I want to know before a student finds out.

Two runs, three kinds of disagreement

The solver runs twice, at two different reasoning-effort settings. Comparing the runs with the key gives three useful signals:

Strong mismatch: both runs disagree with the key.
Weak mismatch: only one run does.
Consensus: both runs give the same wrong answer. This is the interesting one. It usually means the key is missing an acceptable variant, or the key is wrong.

A reviewer decides what the mismatch means

A mismatch is a symptom, not a verdict. A second pass, the reviewer, sees the whole unit with the keys, the deterministic validator's findings, and every mismatch with both runs' answers and reasons. It reports issues with a severity and a suggested fix, and gives each mismatch a verdict: the key is wrong, a variant should be added, the question is ambiguous, the key is right after all, or it's unsure.

The verdict sets the severity:

Reviewer verdict	Severity
key wrong / add a variant / ambiguous	error
key is correct	warning
unsure or no verdict	error if the mismatch was strong, warning if weak

It also looks for things the solver can't: answers that break the word limit, NOT GIVEN vs FALSE confusion, listening answers that come out of order in the recording, and distractors too weak to distract anyone.

Repairs are allowed, within strict rules

Errors, and warnings the reviewer marks as auto-fixable, go to a repair step, one question group at a time. The repair has to return the corrected group plus precise edits: exact find-and-replace edits for passages, line edits for transcripts, and a changelog. A repair is rejected if:

it changes any question numbers,
its edits don't apply cleanly to the current text, or
the result has more validator errors than before.

After a round, only the groups that changed are solved and reviewed again. Anything the checkers no longer report is marked fixed. There are at most three rounds. Anything still open stays open, and the test waits for a human.

Each issue's id is a fingerprint of where it was found and what it is (source, test, section, issue code, group, questions). The same finding keeps the same id across rounds and across later QA runs, so once I've looked at something and waived it, it stays waived instead of coming back every time.

The publish gate

Publishing is a separate command that refuses when:

the test would be public but isn't original or rights-cleared (this one can't be waived),
the validator still finds errors, or
the test's QA job has open error-level issues, unless I waive them by code.

The rights rule is the one I care about most. Test-prep sites are full of "real past papers" of doubtful origin. Here, anything imported from a book stays private, as a share link for one's own students, and only original or rights-cleared tests can be public.

What I took away

Don't let the checker see more than the user does. A solver that sees the key, or a different rendering of the question, is checking something else.
Score with production code. If your checker has its own notion of a correct answer, you end up with two notions of correct.
Separate symptoms from verdicts. The solver only finds disagreements; a different pass with more context decides what each one means.
Constrain the fixer harder than the finder. Most of the safety comes from what a repair is not allowed to do.

The site is live, free and needs no signup: ultraielts. The interface is Turkish, but the tests are English. If you write exam questions of any kind, I'd like to hear how you check them.

Building a 29-skill map from an append-only answer log

Ahmet Koc — Fri, 23 Oct 2026 07:34:14 +0000

I build satultra, a digital SAT practice site. After a practice test, most tools lead with one number, and a 1290 says how you did, not what to study next.

The digital SAT is built around 29 skills: College Board's question bank lists 10 in Reading and Writing and 19 in Math. If every answer is filed under one of them, the results page can show where the points went. Here's how that works in a React and Firestore app: one taxonomy file, an append-only event log, pure reducers and a planner.

One taxonomy file, one canonical id

Over several imports, the same skill showed up as "Central Ideas and Details", "central ideas & details" or just 1. So the first rule was one source of truth, subcategoryTaxonomy.json: eight domains and 29 skills, each with a kebab-case id, a domain and its aliases.

{
  "id": "central-ideas-details",
  "numericId": 1,
  "name": "Central Ideas and Details",
  "domain": "information-ideas",
  "aliases": ["central ideas and details", "central ideas & details"]
}

The kebab id is canonical everywhere: Firestore document ids, events, the planner. numericId survives only for older hand-authored questions, and toCanonicalSubcategoryId() maps any alias or legacy id to the canonical one. The API keeps a copy of the file, and a verify script keeps the two in sync.

If I did it again, I'd fix the canonical id before the second import. Our question documents still carry three versions of the field: subcategory, subCategory and subcategoryId.

Log every answer as an event

Every answer that counts, from a quiz, a drill or a full practice test, goes into an activityEvents collection that clients can append to but never edit. Quiz and test completions, lesson views and flashcard sessions go there too. When a student deletes a test result, the server removes its events and replays the rest.

// a question_attempt event's payload
{
  source: 'smartquiz',          // or exam, drill, lesson_quiz, word_quiz
  questionId: 'q_8f2c',
  subcategoryId: 'words-in-context',
  correct: false,
  timeSpentMs: 48000,
  parentId: quizId,
}

The per-skill numbers are derived from that log. Because the log is the source of truth, the derived state can be rebuilt at any time by replaying events, and a backfill script does exactly that for older history.

Not every finished sitting is evidence

A test module left blank, more than half unanswered, or clicked through in under two minutes is judged once, when the result is saved. Its answers never become skill evidence, though the sitting itself is still logged. A five-question quiz "done" in under a minute gets the same treatment.

Blank questions inside a real attempt count as wrong for the test score, but they aren't attempts, so they don't drag down skill accuracy. Before these rules, skipped modules produced skill cards like "Linear Functions: 0% over 17 questions".

Pure reducers turn the log into skill state

Derived state lives at users/{uid}/skillState/{skillId}. It's computed by pure functions: no I/O and no Date.now(), since timestamps come in on the event. The same module runs in the browser for live updates and in Node for replays, so both paths apply the same logic, and a replay can rebuild whatever a failed live update left behind.

export function reduceSkillState(prev, a) {
  const s = { ...prev, bySource: { ...prev.bySource } };
  s.attempts += 1;
  if (a.correct) s.correct += 1;
  s.accuracy = pct(s.correct, s.attempts);

  s.lastResults = [...prev.lastResults, !!a.correct].slice(-LAST_N); // last 20
  s.accuracyLast10 = accuracyOf(s.lastResults.slice(-10));
  s.rollingAccuracy =
    prev.rollingAccuracy === null
      ? (a.correct ? 100 : 0)
      : Math.round(EWMA_ALPHA * (a.correct ? 100 : 0) + (1 - EWMA_ALPHA) * prev.rollingAccuracy);
  s.trend = computeTrend(s.lastResults);
  s.lastPracticedTs = a.ts || null;
  // ...per-source counts and average time per question
  return s;
}

Lifetime accuracy is kept, but the weakest-skills list and the score estimate use accuracyLast10. A student who was weak at Boundaries a month ago and has since fixed it shouldn't keep seeing it at the top of the list.

Is a skill improving or slipping?

The trend compares the last 10 answers in a skill with up to 10 before them:

export function computeTrend(lastResults) {
  if (!lastResults || lastResults.length < 10) return 'insufficient';
  const recent = lastResults.slice(-10);
  const prior = lastResults.slice(0, Math.max(0, lastResults.length - 10));
  if (prior.length < 5) return 'insufficient';
  const delta = accuracyOf(recent) - accuracyOf(prior);
  if (delta >= TREND_DELTA_PTS) return 'improving'; // 15 points
  if (delta <= -TREND_DELTA_PTS) return 'declining';
  return 'stable';
}

So a trend needs at least 15 answers and a swing of 15 points. Anything smaller is treated as noise at this sample size.

A declining skill is labeled "slipping". A skill with five or more attempts and nothing in the last 8 days is labeled "fading". Neither label changes the numbers: idle time doesn't decay accuracy, it only flags the skill for a short review.

Below the skill level, individual concepts get the same treatment. A concept the student struggled with and then answered right three times in a row is marked recovered, and two misses in a row after that raise a regression flag. Those flags rank first in a four-item list of what to work on next, followed by repeated misses, slipping skills, fading skills and the lowest recent accuracy. The built-in coach reads the same derived state.

From skill accuracy to a study plan

The planner, generatePlan(), is deliberately simple:

Sort the 29 skills weakest first by recent accuracy. Skills with no attempts count as 50%, so they sit between real weaknesses and real strengths.
Give each skill a 20-minute lesson, skipped if already done, and a 15-minute, 10-question practice set.
Lay that queue across the days from tomorrow to the day before the test, skipping rest days, up to the student's minutes per day.
Keep the last two days for reviews of the weakest domains.
If it doesn't all fit, keep the weakest skills and set truncated, so the page can say so.

When a student finishes a lesson or quiz elsewhere in the app, the plan reconciles on the next load and ticks off the matching task. Each quiz can credit only one task.

A prerendering bug worth knowing about

The public pages are part of the same Vite SPA, served by one Express app on Render. For crawlers and a faster first paint, a build step opens 13 public routes in headless Chrome with Puppeteer and saves the rendered HTML. Express serves those snapshots, and the SPA boots on top.

The snapshot also captured scripts that SDKs had injected at runtime. Serialized back to HTML, an injected <script src> loses the implicit async it had and becomes a parser-blocking script at the end of <head>. PostHog's config script did exactly that: the parser stalled on a round trip to a new origin, the async module entry ran before #root existed, and createRoot(null) threw React error #299. On fast connections, the app never mounted.

The fix has two parts. Before saving a snapshot, drop every script that isn't in the built index.html template; the SDKs inject themselves again when the app boots. Then strip async from the module entry so it stays deferred. As a backstop, the entry itself waits for DOMContentLoaded before mounting.

Stack

React 18 and React Router 6 with lazy-loaded pages, built with Vite
Firebase Auth and Firestore, with firebase-admin on the server
One Node/Express app on Render serving both /api and the built site
Stripe Checkout for one-time passes, no subscriptions
Puppeteer for prerendering

satultra is free to start, with the diagnostic, three full-length tests, the question bank and the planner, at satultra.com. Lessons come with a paid pass. If you've built progress tracking for a learning app, I'd like to hear how you handled recent versus lifetime accuracy, and whether you decay old answers.