Three current-year exams, one per track, came back from calibration with a maximum item discrimination well below what the archive exams of the same tracks had shown. The question I was handed was whether the new questions were worse. The new exams had also been taken by far fewer people: 434, 521 and 1,134 students, against 6,342, 9,222 and 8,648 on the archive exams I compared them with.
Finding out what that maximum measured took six posts. IRT Calibration Notes is my record of the investigation, done on an online exam platform's analytics data, and it keeps the explanations that failed next to the ones that held. I wrote it for backend and data engineers, people who run assessment platforms, and developers curious about psychometrics. Part 1 opens with a short refresher on the two-parameter logistic (2PL) model, so you can follow the series without a psychometrics background.
The six parts
Part 1: When a low discrimination score is a pipeline problem
The puzzle and the 2PL refresher come first, then the early surprises. The result tables stored a Lambda of 0.25 that was never estimated, and the service function with "3pl" in its name fitted a 2PL. The idea that the new cohorts simply varied less did not survive the official statistics: in the main mathematics lesson of each track, the new exam had the larger raw-score spread and the lower stored maximum discrimination. Four estimator files shared by two codebases turned out to be byte-identical, which proved that those four files matched and very little else. The part ends with the idea the rest of the series builds on: a stored coefficient is the output of a pipeline, and each stage of that pipeline can be checked on its own.
Part 2: Who actually entered the calibration
One exam and lesson had four student counts, 521, 530, 522 and 523, and each was right for what it counted. This part pins down the population. Only completed analysis requests count, and a lesson counts only when it applied to the student: one lesson of a separate curriculum track went from 8,148 users with seen rows to 328 after that filter. Then comes the rule for what a zero means. Seen-but-blank answers stay as zeros, unseen questions are left out, and nothing is invented where no record exists. The post also covers a library helper that turns every non-correct status into 0 and silently drops duplicate pairs, and why read-only queries under READ UNCOMMITTED still needed frozen snapshots.
Part 3: The prior called vague
The estimator's prior mode was named "vague", yet it put Normal(0, 0.1) on both discrimination and difficulty, and that 0.1 is a standard deviation. The experiment that convinced me held 30 archive questions fixed and changed only the number of students. Mean discrimination went from 0.7665 with 434 students to 1.4986 with 9,099, while the correct-response rate stayed near 9%. Widening only the discrimination prior to an SD of 0.5, on one response set of 523 people, moved the maximum from about 0.93 to about 2.64. A simulation on generated data then asked whether 434 students can support high discrimination at all. With a true mean of 2.75, the local code returned about 0.68 in all three runs, and a small marginal maximum likelihood comparator returned 2.59 to 2.84.
Part 4: Hierarchical priors and marginal maximum likelihood
Part 4 tries two alternatives to the narrow prior. The library's hierarchical branch changes both the model and the guide, and in one small-sample run its latent scale came out reversed, so the questions looked defective until the orientation was fixed. After that correction its size effect on the fixed questions was much smaller than the narrow-prior branch's, but one negative discrimination remained and the branch stayed exploratory. GIRTH 0.8.0, which fits the same 2PL by marginal maximum likelihood, gave larger maxima on all six mathematics lessons and a lower held-out conditional log loss on all six, with one split per lesson. I count that as promising validation evidence, and the post explains why it still does not make the larger coefficients true.
Part 5: Twenty samples per size in a reproducible sampling study
One subset per sample size shows a direction. Part 5 replaces it with a repeated-sampling study over nine exams and 52 lesson-year groups, with sizes from 434 to 8,000 and 20 nested replicates per size. Seeds come from NumPy's SeedSequence, and fit keys expose reused fits: 4,852 successful requests came down to 4,760 distinct fits. Much of the post is engineering, from atomic checkpoints with a retry for a Windows rename, to UTF-8 mode after Persian text broke logging in worker processes, to a single CSV whose rows mean different things depending on their record type. The comparisons use paired t-tests and Wilcoxon checks with Benjamini-Hochberg control. All 175 discrimination contrasts showed a positive mean change from the base size, and on one 1403 mathematics lesson the mean discrimination went from 0.5882 to 1.0881 between 434 and 4,000 students while the samples' raw scores stayed the same on average.
Part 6: Equal sample sizes are not equating
The last part compares lessons across 1403, 1404 and 1405 with the sample size held equal within each track: 1,950 for Mathematics, 2,038 for Experimental Sciences and 2,198 for Humanities. Getting there meant fixing a coverage bug. A Humanities lesson had part of its responses stored under subtopic keys, so a main-lesson query found 17 of its 30 questions, and a missing-question assertion stopped the run before any fit started. The post compares equal-lesson bar charts with equal-question boxplots, explains what a circle in a boxplot does and does not say about a question, and treats the HIGH, MEDIUM and LOW information labels as ranks inside one lesson. Then it shows why equal N is still not equating: the 2PL scale has a free origin and unit, and linking years takes common-item anchors or a deliberate design, neither of which this project performed. It closes with what was not done and with two problems that remain open: a formal link between the years' scales, and a production estimator validated well enough to replace the current one.
How the series treats evidence
Each part keeps what I observed apart from how I read it and from what I propose, and says where a result stops applying. The numbers come from several phases of the work with different extractions and populations, so I report each one with its population and never merge numbers across phases, even when two of them look close. The platform and the organization are not named, no student data appears, and the code is shown in generic form.
Where to start
If you read only one part, make it part 3: the fixed-question experiment and the simulation carry most of the argument. Parts 2 and 5 are the most practical if you build data pipelines, and part 6 matters most if you report comparisons across years.
Part 1 is also cross-posted on DEV, and the whole series is available in Persian and Turkish. For how the same item parameters drive question selection in adaptive tests, see Backend Notes 2. If you run calibration on an assessment platform and want a second look at the pipeline behind your numbers, that is the kind of backend and data work I take on: Work with me.
Top comments (0)