When I started building IntelligenceMax, I faced a question that every cognitive training product has to answer: how do you measure whether someone is getting better at reasoning?
The obvious answer is "track their accuracy." The obvious answer is wrong.
The problem with raw accuracy
If a learner gets 8 out of 10 questions right on day 1 and 9 out of 10 on day 10, did they improve? Maybe. But you cannot know without knowing the difficulty of those questions. Easy questions inflate accuracy. Hard questions depress it. Raw accuracy confounds ability with the difficulty of the specific questions you happened to ask.
This is why standardized tests do not report raw accuracy. They report scaled scores estimated from the difficulty of the items answered. The SAT does not tell you "you got 47/58." It tells you "your score is 720," where 720 is an estimate of your ability that accounts for which questions you got right and wrong.
What IRT actually does
Item Response Theory (IRT) models the probability that a person with a given ability level answers a given item correctly. The simplest model, the 1PL or Rasch model, uses one parameter per item: difficulty. The probability of a correct answer is a logistic function of the difference between the person's ability and the item's difficulty.
$$P(\text{correct}) = \frac{1}{1 + e^{-(\theta - b)}}$$
Where θ is the person's ability and b is the item's difficulty. Higher ability relative to difficulty means higher probability of a correct response.
The 2PL model adds a discrimination parameter. The 3PL model adds a guessing parameter. For our use case (LLM-generated MCQs where guessing is not the primary concern), we use a 2PL model.
The benefit: ability estimates are comparable across different sets of questions. If you answer 7 hard questions correctly and I answer 9 easy questions correctly, IRT can tell you your ability is higher than mine, even though my raw accuracy is higher.
Adaptive difficulty
IRT also enables adaptive testing. Once you have an ability estimate, you can select the next question to maximize information about that person's ability. For a 2PL model, the most informative question for a person at ability θ is one where the difficulty b is close to θ.
This is what CAT (Computerized Adaptive Testing) does. The GRE and GMAT use it. We use a simplified version: generate questions at a target difficulty based on the current ability estimate, and adjust after each response.
The benefit for the learner: every question is at the edge of their ability. Not too easy (boring), not too hard (frustrating). This is the zone of proximal development, operationalized psychometrically.
What IRT does not do
Here is where I want to be careful, because the cognitive training industry has a history of overclaiming.
IRT does not measure intelligence. It measures performance on a specific set of items. If those items are reasoning MCQs, IRT estimates your ability at answering reasoning MCQs. It does not estimate your general intelligence (g). Ability on a specific reasoning task and general intelligence are correlated but not identical.
IRT does not prove transfer. Getting better at our reasoning MCQs, even with adaptive difficulty and accurate scoring, does not mean you have improved at other reasoning tasks. The evidence for far-transfer from cognitive training to general cognition is weak (Melby-Lervåg & Hulme, 2013; Sala & Gobet, 2017). We do not claim our platform improves general cognitive ability. We claim it provides adaptive practice at a specific skill, scored with a method that accounts for item difficulty.
IRT does not make claims about IQ. IQ is a specific construct measured by specific standardized tests. Our ability estimate is not an IQ score. It is a performance estimate on our specific item pool. Conflating the two is what brain training companies do, and it is why the field has a credibility problem.
Why we chose IRT anyway
Because it is the honest way to measure what we are doing. We are providing practice at a specific reasoning skill (distinguishing supported arguments from plausible ones). To know whether that practice is working, we need to measure ability changes over time. Raw accuracy cannot do that because question difficulty varies. IRT can, because it separates ability from difficulty.
The alternative is to not measure at all, or to report raw accuracy and pretend it means something. Neither is honest.
The engineering trade-off
IRT requires item parameter estimation. You need enough responses per item to estimate difficulty and discrimination. For a platform generating fresh questions with LLMs, this is a challenge: each question is new, so there are no historical parameters.
Our approach: we generate questions at a target difficulty (determined by prompt engineering), then calibrate item parameters from response data as it accumulates. Early on, we rely on prompt-specified difficulty. Over time, the data corrects.
This is a known trade-off in adaptive testing with generated items. The alternative is to use a static item bank with pre-calibrated parameters, but that sacrifices freshness and variety. We chose variety over parameter precision, and we are transparent about that.
The bottom line
IRT-based scoring is a tool for honest measurement of a specific skill. It does not make IntelligenceMax a brain training app, an IQ test, or a general intelligence improver. It makes it a platform that practices one reasoning skill, measures performance on that skill accounting for item difficulty, and adapts difficulty to the learner.
That is a narrow claim. I think narrow claims are the honest ones.
Top comments (0)