DEV Community: Isaac Land

IBM Updates Three Granite Speech Models

Isaac Land — Sat, 07 Nov 2026 12:05:47 +0000

IBM Updates Three Granite Speech Models

Speech recognition models process speech picked up by microphones (illustrative photo). Photo: Panos Sakalakis; source: Wikimedia Commons; license: CC BY 2.0.

On October 21, 2026, IBM published an update to its Granite Speech models on the Hugging Face blog. It added granite-speech-5.0-2b and updated two existing models, Turbo CTC and 4.1 2B Plus. All three are released under the Apache 2.0 license. On the Open ASR Leaderboard's English short-form audio benchmark, 5.0 2B has an average word error rate of 5.06%. By the leaderboard data, that places it 8th overall and 2nd among open-weight models, but the ranking mixes averages taken over different numbers of test sets.

What Was Released and What Each Model Is For

IBM's Granite speech team produced the release, and the blog byline includes researchers such as George Saon and Brian Kingsbury. The blog gives each of the three models a use case. 5.0 2B is for high-accuracy multilingual transcription and translation. Turbo CTC is for high-throughput and on-device transcription. 4.1 2B Plus is for multi-speaker recordings such as meetings and phone calls. The dates do not fully agree. The blog is dated October 21, but the 5.0 2B model card says October 20, and the discussion on the leaderboard PR shows the model was still not public on October 17. Which date counts as the official public release has yet to be verified.

Model	Status in this release	Parameter size	Supported languages	Main tasks
granite-speech-5.0-2b	New release (model card date 2026-10-20)	Named 2B; listed as 2.31B in leaderboard data	Transcription: English, French, German, Spanish, Portuguese, Japanese; translation: between each of these languages and English, plus English-to-Italian and English-to-Chinese	Speech recognition, speech translation, keyword biasing (up to 2000 words), punctuation and capitalization output on prompt
granite-speech-5.0-470m-turboctc	Multilingual update (2026-10-21; English version first released 2026-08-25)	470 million	English, French, German, Spanish, Portuguese, Italian, Japanese	Speech recognition only; outputs punctuation and capitalization directly
granite-speech-4.1-2b-plus	Weight update (first released 2026-04-28, updated 2026-10-19)	Named 2B; same architecture as 4.1 2B; exact parameter count yet to be verified	English, French, German, Spanish, Portuguese	Speech recognition, speaker-attributed transcription, word-level timestamps, keyword biasing; no punctuation or capitalization output

The information in the table comes from the three official model cards, and all three models are licensed under Apache 2.0. Note that "Attribute" (speaker attribution) in the blog title refers to a capability of 4.1 2B Plus; the 5.0 2B model card does not list speaker labeling. 5.0 2B uses an intermediate checkpoint of granite-4.0-1b-base as its language model backbone. It was trained on about 390,000 hours of public audio, plus synthetic data created for Japanese recognition, keyword biasing, and translation. The model card recommends cutting long recordings into segments of about 2 minutes or less before transcription.

Results Reported by IBM

All figures in this section come from IBM's blog and model cards and are officially reported results. For 5.0 2B, the blog says its average word error rate (WER, lower is better) on the Open ASR Leaderboard's English short-form public test sets is 5.06%, which ranked 2nd among open-weight models as of October 21, 2026. For punctuation and capitalization, the blog says capitalization F1 ranges from 89.7 to 99.5. However, the model card table lists 5.0 2B's values as 90.16 to 99.51, and 89.71 is actually the score of the previous-generation 4.1 2B. Punctuation error rate went down on 7 of 9 test sets and rose slightly on CommonVoice English and French. Results for noisy and far-field conditions (the FFASR leaderboard), multilingual recognition, and speech translation appear only as charts in the model card. This article could not read the exact values, so they have yet to be verified.

For Turbo CTC, the model card says the new multilingual version reaches an English average WER of 5.51%, better than the English-only version's 5.78%. It also notes that 5.51% is an unofficial result and that the leaderboard still lists the English-only version. The blog says the model reaches 19,870 RTFx on the leaderboard (processing speed as a multiple of the audio's real-time length), about 1.6 times the 12,600 it had at launch. In the leaderboard data read for this article, however, the entry is listed as 21,134.31. The two figures do not match, and the reason has yet to be verified.

For 4.1 2B Plus, the blog compares the old and new versions (all figures are error rates, so lower is better):

Metric	Old version	New version
Speaker attribution error rate WDER	4.40	4.10
Error rate for transcription with speaker labels cpWER	30.50	28.48
Long-form audio WER	13.53	10.22
Keyword WER with keyword biasing (20-word list)	11.79	6.48
Average timestamp shift AAS (all speech)	210.86	98.54

The per-test breakdown in the model card shows the gains are not across the board. WDER got worse on FISHER (from 0.9 to 1.0) and CALLHOME English (from 2.2 to 3.0) but improved on AMI-SDM and GALE. On the leaderboard's revised English test sets, average WER rose from 6.01% to 6.10%. The model card does not explain how the blog's summary WDER values of 4.40 and 4.10 were calculated, so this has yet to be verified.

Where It Stands on the Open ASR Leaderboard

The Open ASR Leaderboard is hosted under Hugging Face's hf-audio organization, and its result data are public in the open-asr-leaderboard-results dataset. On October 22, 2026, this article read english_short_latest.csv from that dataset. At the time, the latest commit was f3f9949, the one that added the 5.0 2B results. The English short-form leaderboard currently uses 9 public test sets: cleaned versions of AMI, Earnings22, Gigaspeech, and VoxPopuli; LibriSpeech clean and other; SPGISpeech; URGENT2024; and Voice Arena Monsoon (Indian English). The leaderboard does not state a formal methodology version number, so this has yet to be verified.

This result does not rest on IBM's word alone. On October 14, IBM researcher gsaon opened PR #245 in the leaderboard's code repository and attached run logs from Hugging Face Jobs that showed an average WER of 5.06% across the 9 test sets. On October 19, repository collaborator Deep-unlearning said in a comment that they had run the public test sets independently on an H200 and got the same 5.06%. The PR was merged on October 21 and the result went live. However, the reproduction logs were not attached to the PR, and no results on the leaderboard's private test sets have been published. Both points have yet to be verified.

When the CSV's average WER column is sorted from lowest to highest, 5.0 2B (5.06%) is 8th. Of the 7 entries above it, 6 are closed-source commercial services, and OtterAI speech-v3.1 (3.88%) is 1st. The only open-weight model among them is Microsoft's Phi-4-multimodal-instruct (5.02%, MIT license). Going by this column, the blog's claim of "2nd among open-weight models" matches the leaderboard data.

The ranking does not compare like with like, however. This article recalculated each row from the per-test-set values in the CSV and found that the average column covers different numbers of test sets. For 5.0 2B it covers all 9. Phi-4-multimodal has no URGENT2024 result, so its average covers 8. For older entries such as the previous-generation 4.1 2B and Qwen3-ASR-1.7B-hf, the listed values match the average over 7 test sets, with the two LibriSpeech test sets left out. Averaged over the same 9 test sets, Qwen3-ASR-1.7B-hf scores 4.94%, lower than 5.0 2B's 5.06%. On the 8 test sets where Phi-4-multimodal has results, 5.0 2B averages 4.42%, lower than Phi-4-multimodal's 5.02%. In other words, with a consistent method 5.0 2B is still 2nd among open-weight models, but the model ahead of it is Qwen3-ASR-1.7B-hf. Its 8th place overall also depends on the method. These figures are this article's own calculations, not the leaderboard's official ranking. Why the leaderboard mixes different methods has yet to be verified.

Compared with the previous generation on the same 9 test sets, 4.1 2B has an average WER of 5.35% and 5.0 2B has 5.06%. WER went down on 8 of the 9 test sets; only LibriSpeech other rose slightly, from 2.19% to 2.21% (this article's calculation from the CSV). The two models are about equally fast, with RTFx of 546.84 and 544.52 in the CSV, respectively. IBM reported an RTFx of 570.17 for 5.0 2B in the PR log. Whether the gap from the leaderboard figure comes from different hardware has yet to be verified.

Spectrogram of the English word "open" being spoken (illustrative image). According to the Granite Speech 5.0 2B model card, its speech encoder takes log-mel spectrogram features as input, which are the same kind of time-frequency representation. Author: Kelly Fitz; source: Wikimedia Commons; license: CC BY-SA 3.0.

References

IBM Granite. Granite Speech 5.0: Transcribe, Translate, and Attribute more Accurately and Faster. Hugging Face Blog, 2026-10-21.
IBM Granite. ibm-granite/granite-speech-5.0-2b model card. Hugging Face, accessed 2026-10-22.
IBM Granite. ibm-granite/granite-speech-5.0-470m-turboctc model card. Hugging Face, accessed 2026-10-22.
IBM Granite. ibm-granite/granite-speech-4.1-2b-plus model card. Hugging Face, accessed 2026-10-22.
Hugging Face hf-audio. open-asr-leaderboard-results: english_short_latest.csv. Commit f3f9949, read 2026-10-22.
huggingface/open_asr_leaderboard. Add granite-speech-5.0-2b (PR #245). GitHub, merged 2026-10-21.
Smart Academy · Your own academy for the age of intelligence
Compiled and published by Scholay All-in-One Academic Platform. If this content infringes your rights, please let us know and it will be taken down.
Also published on EnterScholar, a community of scholars for the age of intelligence

Bootstrap: Understanding Sampling with Replacement Through Five Numbers

Isaac Land — Thu, 22 Oct 2026 06:46:57 +0000

Bootstrap: Understanding Sampling with Replacement Through Five Numbers

How can we estimate how much a sample mean might vary when we have only one sample? The bootstrap treats the observed sample as a small empirical distribution that we can sample from repeatedly. Draw the same number of records, calculate a mean, and repeat. The spread of these means gives us information about uncertainty.

One resample from five numbers

Take the teaching example 2, 3, 4, 4, 7. Its mean is 4. Each of the five records has its own position: the two 4s are separate records. Randomly select one of the five positions, leaving that position available for later draws. Draw five times to obtain one bootstrap sample.

For example, one resample might be 2, 2, 4, 7, 7. The original 3 is absent, 2 and 7 each appear twice, and only one of the two records containing 4 is selected. A selected record stays in the pool and can be selected again. After five draws, the sample size is still 5. This resample has a mean of (2 + 2 + 4 + 7 + 7) / 5 = 4.4. The next resample may have a different combination and a different mean.

A statistic for each resample

If the target is a mean, calculate a mean each time. If the target is the correlation between two measurements, resample complete records and then calculate the correlation. Resampling the two measurements independently would destroy the pairing that we want to study.

A histogram of the repeated statistics can show where they concentrate, whether the distribution is skewed, and whether a few extreme results occur. Their standard deviation can estimate the bootstrap standard error. More resamples make the computation more stable; the original sample still contains only the observations we started with.

What should we resample?

The simplest ordinary bootstrap applies when records can be treated as independent and identically distributed observations. Seven diary entries from one person belong to the same person; students in the same classroom may also be correlated. The research design should determine whether to resample people, groups, or time blocks. A row in a spreadsheet is not automatically the right resampling unit.

Software interfaces reflect this distinction. R's boot function takes data, a statistic function, and a number of replications. In ordinary resampling, row indices are passed to the statistic function. Understanding what each index represents matters more than increasing the replication count first.

Report the settings and the limitations

Record the original sample size, target statistic, resampling unit, number of replications, and random seed, then report the results. If constructing a confidence interval, specify the percentile, BCa, or other method used. Different constructions should not all be described simply as a "bootstrap interval."

The bootstrap recombines records that have already been observed. If a sample misses a group of people, repeated resampling cannot create those missing people. For targets that depend strongly on the tails, such as a maximum or a range, check whether the method is suitable. Start with how the sample was obtained and what you want to estimate, then decide how to resample.

Sources

Translated and adapted from the original Chinese article on Metho.

Metho · Research Methods — Rituals can be copied. Understanding cannot be faked.

Discuss this article on EnterScholar — A scholarly community for the age of intelligence.

How Wald Inferred Vital Areas from Returning Aircraft

Isaac Land — Thu, 22 Oct 2026 03:19:40 +0000

How Wald Inferred Vital Areas from Returning Aircraft

According to the Wikimedia Commons description, this is a B-17 of the U.S. 398th Bomb Group whose nose was destroyed by flak during a bombing raid on Cologne, Germany, on October 15, 1944. The plane still made it back to base. Image: U.S. Air Force (National Museum of the U.S. Air Force), via Wikimedia Commons; license: public domain.

In 1943, Abraham Wald of Columbia University's Statistical Research Group (SRG) wrote a set of memoranda on how to estimate, using only the bullet holes on returning aircraft, the probability that a plane would be shot down after being hit in each of its parts. The popular story reduces this to a one-line quip: “reinforce wherever there are no holes.” Checked against the original memoranda and Mangel and Samaniego's 1984 interpretation, the actual reasoning was more subtle, and many of the details in circulation cannot be verified.

Columbia University's Statistical Research Group

The Statistical Research Group (SRG) was established on July 1, 1942, at Columbia University in New York. It was funded by the Applied Mathematics Panel (AMP) of the National Defense Research Committee (NDRC), which sat under the U.S. Office of Scientific Research and Development (OSRD). W. Allen Wallis, the group's director of research, recorded several figures in a 1980 retrospective article. Over its lifetime the group had 18 principal members and a total of 78 staff. At most about 50 people were on duty at once, including about 30 young women who worked only on computation. During the war the group produced 572 reports, memoranda and letters. Its members included Friedman, Savage, Mosteller, Wolfowitz and others who later became leading figures.

Wald was born in 1902 in Cluj, in present-day Romania. He worked on pure mathematics in Vienna in his early years and turned to mathematical statistics in the mid-thirties. Later, after losing his position in Austria because he was Jewish, he made his way to the United States. Wallis recalled that in the fall of 1939, when Wald had just begun teaching at Columbia, he and Wolfowitz often attended Wald's evening lectures. Wald was a part-time member of SRG and took part in its work for about 30 months in total. His best-known achievement there was sequential analysis, which originated at SRG in 1943. According to Wallis, the classification of sequential analysis as secret had a side effect: Wald's naturalization application was approved quickly, making him one of the few “enemy aliens” to obtain U.S. citizenship during the war.

Aircraft vulnerability was one of the topics that grew out of SRG's air-warfare research. In SRG's final report, written in 1945, Wallis wrote that this problem led the group to devise a method of “judging vulnerability from the damage sustained by our own planes.” Mangel and Samaniego summarize the operational problem of the time this way: commanders had to decide which tactics to use and which parts of the aircraft to give extra protection, but added armor increases weight and reduces a plane's ability to carry out its mission. The real difficulty was that no one could see how the bullet holes were distributed on the planes that failed to return.

The Reasoning and Numerical Example in the Memoranda

Wald's work took the form of a set of technical memoranda under the overall title “A Method of Estimating Plane Vulnerability Based on Damage of Survivors.” According to the Defense Technical Information Center's entry, these are eight memoranda that SRG wrote in 1943 for the National Defense Research Committee. They were not published during the war and were reprinted only in July 1980, by the Center for Naval Analyses (CNA), as CRC 432. The first few parts do not distinguish among parts of the aircraft. Using only the number of sorties, the number of planes that returned, and the number of hits on each returning plane, they derive the probability that a plane is downed by the i-th hit, along with upper and lower bounds on that probability. Part Five divides the aircraft into several “equal-vulnerability regions” and estimates the vulnerability of each part. Part Six deals with sampling error and gives 95% and 99% confidence limits.

The numerical example in Part Five shows the line of reasoning most clearly. Wald supposes that 400 planes fly a bombing mission and 359 return. Of the returning planes, 240 were not hit, while 68, 29, 12 and 10 planes took 1 to 4 hits respectively, for a total of 202 hits. He adopts simplifying assumptions, such as “every hit has the same probability of downing the plane, regardless of how many hits came before.” Under these assumptions he first solves for the probability that a plane survives a single hit, which comes to about 0.85. He then divides the aircraft into the engine, the fuselage, the fuel system and other parts. He assumes that the probability of a single hit landing on a given part equals that part's share of the projected area facing the direction of enemy attack: engine 26.9%, fuselage 34.6%, fuel system 15.4%, other parts 23.1%.

The hits on the returning planes, however, did not follow these proportions. The engine accounted for only 19.3% of them (39 hits), the fuselage 38.6% (78), the fuel system 15.4% (31), and other parts 26.7% (54). If a part's share of hits on surviving planes is lower than the share it “should” receive based on its area, then more of the planes hit there failed to return. By Wald's formula, the probability of surviving a single hit on a given part equals 0.85 multiplied by “actual share ÷ expected share.” That gives about 0.61 for the engine, about 0.95 for the fuselage, 0.85 for the fuel system, and about 0.98 for other parts. On this basis the memorandum states that, for the data of this hypothetical example, the engine area is the most vulnerable and the fuselage is relatively less vulnerable.

The hypothetical example in Part Five of Wald's memoranda. The engine's share of hits on returning planes is lower than the share its area would predict, and its estimated single-hit kill probability is the highest. Data from Wald's (1943) memoranda, CNA 1980 reprint; chart drawn with code for this article.

What Survivorship Bias Means and What It Assumes

This is what later came to be called survivorship bias: whether an object makes it into the sample depends on the very outcome we want to study. The returning planes are not a random sample of all the planes that flew. They are only the ones that “could still fly home after being hit.” If you count holes on them, the places with the most holes are often the places that can best withstand fire. In a short 1980 piece responding to readers' comments, Wallis put the point plainly. The military was inclined to add protection to the parts of returning planes that had been hit most often. Wald, however, had good reason to assume that hits in combat were spread roughly evenly over the whole aircraft. Hits on vital parts would therefore be underrepresented on the returning planes, because planes hit in those parts were less likely to come back and supply data.

Still, “reinforce where there are no holes” is only shorthand, and it holds only under one condition: we must first know how hits would have been distributed if no planes had been shot down. In Part Five, Wald specifically points out that the damage on returning planes alone cannot be used to estimate both the hit distribution and the vulnerability of each part. He gives an example. Suppose all the holes on returning planes fall in Region One. That could mean Region Two simply cannot be hit, or it could mean that any plane hit in Region Two goes down. The data cannot tell the two explanations apart. He therefore required the hit distribution to be known in advance. It could be inferred from the dispersion of enemy fire, or measured in firing tests with “dummy bullets.” In the numerical example, the proportions of projected area serve in its place.

In other words, Wald was not comparing “holes versus no holes.” He was comparing “how many fewer holes than expected.” In his example the engine still had 39 holes, just fewer than its area would predict. If a spot has few holes only because it is small and rarely hit, that is no reason to conclude that hits there are more deadly. In their 1984 article, Mangel and Samaniego re-examined these memoranda using modern statistical theory and discussed them within the framework of missing data and approximation methods. They concluded that Wald's treatment of these problems “was definitive.”

Exaggerations and Unsupported Details in Popular Versions

The popular version goes roughly like this. Looking at the pattern of holes on returning bombers, the military was about to reinforce the heavily hit fuselage. Wald pointed out on the spot that the sparsely hit engines were what needed reinforcing, and his advice was adopted. In 2016, the mathematician Bill Casselman checked this story point by point in his column for the American Mathematical Society. When it is compared with the original sources, its claims fall into three categories.

The documented parts: Wald did write a set of memoranda at SRG on inferring vulnerability from damage to surviving aircraft. Wallis's 1980 rejoinder did say that the military was inclined to reinforce the parts of returning planes that had been hit most often, and it stated that these methods were later used by the Navy and the Air Force. CNA's note to the reprint and Mangel and Samaniego's article both mention that the method was still used to analyze data from the Korean and Vietnam Wars.

The exaggerated parts: the memoranda are rigorous and highly technical mathematical derivations, and Casselman notes that they make no recommendation about what the military should do. The claim that “the engine is the most vulnerable” comes from the hypothetical data in Part Five. The text says explicitly that this conclusion applies to “this hypothetical example” and is not a finding measured in any actual campaign. Telling the story as a flash of insight also leaves out the assumptions Wald made to solve the identification problem, as well as the separate treatment he gave to sampling error.

The unsupported parts: no original record currently supports the conversations and meeting scenes between Wald and military officers, or the so-called “Wald's hand-drawn bullet-hole diagram” that circulates online. Casselman originally found that the only reliable material on this work was two brief passages in Wallis's retrospective article that do not mention Wald by name, plus the memoranda themselves. He later added Wallis's rejoinder in an addendum. The red-dotted airplane diagrams commonly seen online were all drawn by others later, and some even show a C-47 transport, a plane that rarely saw combat.

Chance also played a part in the memoranda becoming public nearly forty years late. Casselman quotes Phil DePoy of the Center for Naval Analyses on how this happened. When Wallis was moving house, he came across a box of loose papers left over from when SRG shut down in 1946 and handed it to DePoy. DePoy kept only one packet from it, Wald's material, edited it lightly, and published it under Wald's name in July 1980. Wald himself had died in 1950 in a plane crash in the mountains of southern India, and he left no recollections of his own about this work.

References

Wald, A. (1980). A Reprint of "A Method of Estimating Plane Vulnerability Based on Damage of Survivors" by Abraham Wald (CRC 432). Center for Naval Analyses. Original memoranda written in 1943. https://www.cna.org/reports/1980/0204320000.pdf
Defense Technical Information Center. Entry ADA091073. https://apps.dtic.mil/sti/citations/ADA091073
Mangel, M., & Samaniego, F. J. (1984). Abraham Wald's Work on Aircraft Survivability. Journal of the American Statistical Association, 79(386), 259–267. https://doi.org/10.1080/01621459.1984.10478038
Wallis, W. A. (1980). The Statistical Research Group, 1942–1945. Journal of the American Statistical Association, 75(370), 320–330. https://doi.org/10.1080/01621459.1980.10477469
Wallis, W. A. (1980). The Statistical Research Group, 1942–1945: Rejoinder. Journal of the American Statistical Association, 75(370), 334–335. https://doi.org/10.1080/01621459.1980.10477472
Casselman, B. (2016, June). The Legend of Abraham Wald. AMS Feature Column. https://www.ams.org/publicoutreach/feature-column/fc-2016-06
Research Methods · Rituals can be imitated; understanding cannot be faked
Compiled and published by Scholay, a one-stop academic platform. If this infringes your rights, please let us know and it will be taken down.
Also published on EnterScholar, a community for scholars in the age of intelligence

Originally published: How Wald Inferred Vital Areas from Returning Aircraft

Discuss on EntScholar (Chinese)

Metho · Research Methods — Rituals can be copied. Understanding cannot be faked.

The 1884 Weight Experiment Randomized with Playing Cards

Isaac Land — Thu, 22 Oct 2026 02:59:45 +0000

The 1884 Weight Experiment Randomized with Playing Cards

Charles Sanders Peirce (1839–1914), photographed in 1891. Image: Napoleon Sarony; original held by the New York Public Library; source: Wikimedia Commons; license: public domain.

In October 1884, Peirce and his student Jastrow reported a set of weight-perception experiments to the US National Academy of Sciences: the experimenter used the color of cards from a shuffled pack to decide whether the pressure would be “first increased, then decreased” or “first decreased, then increased,” while the subject made judgments from behind a screen. Even when the subjects believed they had no confidence at all, their judgments were still clearly better than guessing. The experiment is often regarded as one of the earliest to use an explicit randomization scheme, but what it randomized was the order of trials, not group assignment.

The Dispute over the Difference Threshold, and the Apparatus

From the mid-19th century onward, the psychophysics of G. T. Fechner assumed the existence of a “difference threshold” (Unterschiedsschwelle): when the difference between two stimuli falls below a certain proportion, people can no longer tell them apart. Peirce, who had long studied observational error at the US Coast Survey, held that the errors made in comparing two sensations arise from a great many tiny chance disturbances along the path of nerve conduction and should follow the error curve (the normal distribution); in that case, there would be no threshold. The two views yield different testable predictions. If a threshold exists, then for differences below it, right and wrong judgments should each make up half in the long run. If the theory of errors holds, then no matter how small the difference, the proportion of correct answers should be above half, and that proportion can be calculated in advance from the theory of errors.

According to Hacking's research, the experiments were carried out between 10 December 1883 and 7 April 1884. Jastrow, then Peirce's student at Johns Hopkins University, recalled in 1914 that the problem, the apparatus, the method and the way of arriving at the results were all supplied by Peirce, and that the apparatus was set up in his own room. Both men served as subjects, each making 150 judgments in a session lasting about two hours.

The apparatus was adapted from a Fairbanks postal scale. One end of the beam carried a small block fitted with a rubber cap, which pressed up from below against a finger of the subject's left hand; with a 1-kilogram weight on the scale pan, the finger bore a pressure roughly equal to one quarter of that weight. A separate small pan holding the “difference weight” could be lowered on or lifted off by a fine rubber band, increasing or decreasing the pressure without any sound or vibration. The difference weights were 60, 30 and 15 parts per thousand of 1 kilogram; later on, Jastrow switched to 20, 10 and 5 parts per thousand. In each trial, the subject had to judge whether the pressure had “increased then decreased” or “decreased then increased,” and was not allowed to answer “can't tell.” At the same time, the subject reported confidence on a scale from 0 to 3: 0 meant no inclination at all, and 3 meant the strongest assurance one could have about a sensation of this kind.

Ordering by Playing Cards and Concealment by a Screen

The order in each trial was determined by playing cards. Each set of 25 trials used a pack of 25 cards, 12 red and 13 black, or the reverse, so that for a given difference weight, the 50 trials of one session had exactly 25 red cards and 25 black. Before each set, the pack was shuffled thoroughly, and the experimenter used the colors of the cards, turned up one after another, to decide whether each trial would be increase-then-decrease or decrease-then-increase. After the subject answered, the experimenter laid the card face down or face up depending on whether the answer was right or wrong, and, according to the confidence level, set it so that the corner bearing 0 to 3 punched holes pointed in a fixed direction. This meant no separate notes were needed and the pace was not slowed; the results were recorded all together once a set was finished.

A postal scale made by the Fairbanks company, in the collection of the Smithsonian National Postal Museum (date of manufacture unverified). Peirce and Jastrow modified a postal scale of the same brand; this image is for reference only and is not the scale used in the experiment. Image: Smithsonian National Postal Museum; license: CC0 1.0.

The reason given in the paper was that this prevented the subject from making any psychological guess about which change the experimenter “might be likely to choose.” The authors also wrote about a side effect of the random sequence: long runs of the same kind of change, occurring by chance, would confuse the subject. They believed, however, that if long runs were deliberately excluded and the subject knew this, the resulting disturbance would be even greater. Hacking points out that the problem of long runs remained contested later, when Fisher promoted randomization, and that Peirce's use of two small packs with opposite numbers of red and black cards had in effect already limited the imbalance between red and black within a session.

As for concealment, a screen hid all of the experimenter's movements, and the paper states that they took ample precautions against “unconsciously received hints.” It should be noted that this blinded only the subject. The experimenter could see the card colors and knew exactly how to add or remove weight each time, so he was not blind. The role of randomization was to take “which order to choose” out of the experimenter's hands, so that even if he fell into some rhythm, consciously or unconsciously, it could not affect the sequence of stimuli. Hacking's summary of the experiment is exactly this: weights were presented to subjects who could not see the manipulation, and randomization was used to guard against unconscious suggestion.

Accuracy of Low-Confidence Judgments

The paper compared the number of errors in each set with the expected value calculated from the theory of errors, and concluded that, as far as could conveniently be traced, the error counts all conformed to the probability curve, thereby “destroying every presumption in favor of a difference threshold.” Even more striking is the confidence record in footnote 4 of the original:

Subject and difference weights	Trials	Overall error rate	Share with confidence recorded as 0	Error rate at confidence 0
Peirce, 15/30/60 parts per thousand	1,125	29%	63%	38%
Jastrow, 15/30/60 parts per thousand	1,975	23%	57%	30%
Jastrow, 5/10/20 parts per thousand	1,675	32%	93%	34%

The last row is the most telling: when the difference weight was only 5 to 20 parts per thousand, Jastrow marked more than nine in ten of his judgments as “no inclination at all,” yet about two-thirds of those judgments were still correct. On this basis, the authors argued that confidence reflects a “secondary sensation” of the difference between the two sensations. This secondary sensation likewise has no threshold, but subjects often overlook it, so much so that confidence was still recorded as 0 even when the rate of correct answers reached three-fifths. They also found that average confidence roughly followed m = c log(p/(1−p)), where p is the probability of a correct answer. As cited by Hacking, Stigler considered this possibly the earliest attempt to elicit subjective probability experimentally.

The final paragraph of the paper pushed the conclusion further: people may gain knowledge of other people's minds through faint sensations of which they are not conscious, and “women's insight” and certain “telepathic” phenomena might perhaps be explained in this way. This paragraph is the authors' speculation; the experiment itself did not test it.

Relation to Fisher's Randomization, and Limitations

Calling it “one of the earliest randomized experiments” rests mainly on research in the history of science. As cited by Hacking, Stigler wrote in 1978 that this was the first experiment he knew of to be conducted according to a “precise, mathematically sound randomization scheme.” Hacking sums up its design as three innovations: deliberate randomization, not allowing subjects to answer “can't tell,” and confidence ratings from 0 to 3. By way of contrast, in Fechner's earlier weight-lifting experiments, Fechner himself served as the subject and knew in advance which box was heavier. Hacking also notes that the French physiologist Charles Richet independently used random card drawing in telepathy tests at about the same time, so “earliest” should be understood as among the earliest known.

The practice was not handed down all the way to Fisher. Hacking found that the later psychophysics literature almost never mentioned randomization again. Fullerton and Cattell's 1892 study says only that trials were conducted in an “irregular order”; when Jastrow later gave a systematic account of the method, he did not mention randomization either; and the debate over Peirce's method centered on “not allowing ‘can't tell’ answers.” Fisher first publicly argued for the central role of randomization in 1926 in “The Arrangement of Field Experiments,” and his 1935 The Design of Experiments made it textbook material. His reasons belonged to his own system of statistical inference: randomization gives error estimation and significance testing an objective basis and can improve the efficiency of experiments. Hacking states explicitly that he does not believe early psychical research had any important “influence” on Fisher; he points out only that the concern about chance long runs in the “lady tasting tea” example of The Design of Experiments had already been raised by Peirce.

Judged by the standards of modern design, the experiment has several limitations. First, it randomized the order of trials within the same subject; it did not randomly assign subjects to treatment and control groups. Dehue 1997 argues that the random group design took shape gradually in educational psychology during the 20s of the 20th century, and that McCall appears to have been the first to propose forming groups at random. Second, there were only two subjects, who were the authors themselves, and the experimenter was not blind. Third, the balanced packs themselves left a clue: a footnote in the paper admits that Jastrow's observational errors in the last two stages were abnormally small, partly because he unconsciously kept track of how many times each of the two changes had appeared in a set and answered accordingly. Fourth, practice effects made Jastrow's errors fall steadily, so the authors could only divide his data into four stages and compare them separately; and in the early experiments of December 1883 and January 1884, the procedure had not yet been perfected. Finally, the paper describes its data analysis in less detail than its data collection. Its inference relied mainly on comparing error counts with the theoretical values of the error curve, and the role of randomization in inference was not explicitly argued in the way Fisher would later argue it.

References

Peirce, C. S., & Jastrow, J. (1885). On Small Differences of Sensation. Memoirs of the National Academy of Sciences, 3(1), 73–83 (read 17 October 1884). PhilArchive scan; Full text at Classics in the History of Psychology
Hacking, I. (1988). Telepathy: Origins of Randomization in Experimental Design. Isis, 79(3), 427–451. doi:10.1086/354775
Stigler, S. M. (1978). Mathematical Statistics in the Early States. The Annals of Statistics, 6(2), 239–265. doi:10.1214/aos/1176344123
Dehue, T. (1997). Deception, Efficiency, and Random Groups: Psychology and the Gradual Origination of the Random Group Design. Isis, 88(4), 653–673. doi:10.1086/383850
The Collected Papers of Charles Sanders Peirce, Vol. 7, editors' notes (including Jastrow's 1914 recollection). Electronic edition
Research Methods · Rituals can be imitated; understanding cannot be faked
Compiled and published by Scholay, the one-stop academic platform. If any content infringes your rights, please let us know and it will be taken down.
Also published on EnterScholar, a community for scholars in the age of intelligence

Originally published: The 1884 Weight Experiment Randomized with Playing Cards

Discuss on EntScholar (Chinese)

Metho · Research Methods — Rituals can be copied. Understanding cannot be faked.

How Data Journalists Spotted Fabricated Polls

Isaac Land — Thu, 22 Oct 2026 02:59:06 +0000

How Data Journalists Spotted Fabricated Polls

On October 20, 2026, Wedgewood Polls, an emerging American polling outfit, admitted on the social platform X that the vast majority of the polls it had published had no real survey data behind them. The admission was prompted by an investigation by the election-data site FiftyPlusOne: after obtaining its raw data, reporters found the sample was abnormally “clean,” and the respondent IDs and transaction IDs did not follow the patterns of the survey platform it claimed to use. This article reviews what happened, how the data were checked and how polling aggregators screen pollsters, and lists items readers can check for themselves.

The “Vote Here” sign outside the polling place at Woodbury City Hall, Minnesota, on U.S. Election Day, November 3, 2020. Photo: Tony Webster. Source: Wikimedia Commons. License: CC BY 2.0.

What Happened with Wedgewood

According to Newsweek and The Guardian, Wedgewood Polls appeared in June 2026, describing itself as an independent election pollster. One of its earlier polls, on Louisiana’s Republican Senate primary, showed John Fleming leading Julia Letlow by 4 percentage points, while in the actual result on June 27 Letlow won by nearly 14 percentage points. On July 10, Washington Monthly columnist Bill Scher pointed out that the new outfit’s first poll had been wildly wrong.

On August 20, Straight Arrow News reported on its background: its website was built on Google Sites and listed no staff names, phone number or location; the site was registered on June 26, and its only social media account, on X, was also created in June. Wedgewood described itself as a two-person team that had not yet incorporated as a company. It declined to reveal who its members were on the grounds that one of them had been harassed online, and it also refused video calls. At the time, The New York Times’ polling database already included its results, but with the note “this pollster does not meet our criteria for select pollsters.”

In mid-October, Wedgewood released a Mississippi Senate poll: conducted October 12 to 14 among 511 likely voters, it showed incumbent Republican Senator Cindy Hyde-Smith leading Democrat Scott Colom by only 52% to 48%. According to Magnolia Tribune, Colom’s campaign promptly cited the poll in a press release. FiftyPlusOne’s Mary Radcliffe, G. Elliott Morris and Cooper Burton had received multiple tips about Wedgewood before, but this time the tip came with evidence, so they asked Wedgewood for the poll’s raw data. On October 20, FiftyPlusOne sent its findings to Wedgewood for comment. Rather than answer the questions, Wedgewood posted a statement on X admitting that the vast majority of its polls were not backed by real data. It called this a “social experiment” to expose gaps in how media outlets and aggregators vet pollsters, and said that only one crowdfunded Nebraska poll was real. For now, both the “social experiment” motive and the claim that the Nebraska poll was real rest only on Wedgewood’s own account.

Afterward, The New York Times removed all of Wedgewood’s polls from its poll tracker, Decision Desk HQ deleted 17 of them from its race lists and averages, Silver Bulletin announced a permanent ban, and Newsweek retracted 3 articles and corrected other coverage. Michigan State Senator Mallory McMorrow said that a Wedgewood poll in late June showing her support falling into single digits dried up her campaign donations and hurt her volunteers’ morale; she withdrew from the Senate primary on July 5. This account of the poll’s impact is her own.

FiftyPlusOne’s Data Check

According to FiftyPlusOne’s report (as quoted on Gelman’s blog), the reporters’ first impression on obtaining the raw data was that it was “surprisingly clean.” The unweighted sample was 67% white and 31% Black, already quite close to reasonable estimates for Mississippi (about 60% and 37%); the poll’s design effect was only 1.18. The design effect measures “how hard” the weighting has to work to make the sample representative: the more skewed the sample, the more spread out the weights must be, the higher the design effect, and the smaller the effective sample size. For example, if a sample has far too few young people, the few young respondents must be given very large weights, which amounts to letting a handful of people speak for a large group. By the usual conversion, 511 respondents with a design effect of 1.18 amount to an effective sample of about 433, a very small loss for an online survey.

A low design effect does not by itself mean fabrication. If quotas are set during recruitment, requiring the platform to return respondents from each group in proportion to the population, the raw sample can also be very close to the population. FiftyPlusOne therefore asked Wedgewood whether it had used quotas, because as far as they understood, quotas seemed to be the only way to control sample composition on Pollfish while a survey is in the field. Wedgewood said it had not, which made the unusually low design effect harder to explain. But the reporters also stated that this was still not enough to prove the data were made up.

The decisive evidence came from IDs automatically generated by the survey platform. Wedgewood said the poll had been conducted through the online survey platform Pollfish, so FiftyPlusOne paid out of its own pocket to replicate the Mississippi survey on Pollfish as closely as possible and obtain real data for comparison. In the real data, about half of the respondent IDs were plain strings of digits (e.g., “1234567890”) and half had a hyphen prefix (e.g., “-1234567890”); the transaction IDs tracked the time respondents submitted the questionnaire almost perfectly (in the report’s words, “almost 1:1 correlated”). The reporters confirmed with Pollfish that this was exactly the expected behavior of these two columns, and checked data from another organization that had used the platform, finding the same patterns. The data Wedgewood provided matched neither pattern, nor the platform’s description of these two columns.

Example scatter plots for different correlation coefficients: at the far left of the first row the correlation is 1 and the points fall on a straight line; in the middle it is 0 and the points form a patternless cloud. According to the report, the relationship between transaction IDs and submission times in real Pollfish data is close to the former. This is a generic illustration, not data from FiftyPlusOne or Wedgewood. Image: DenisBoigelot (original uploader Imagecreator). Source: Wikimedia Commons. License: CC0.

It is a bit like checking a receipt: a forger can make the amount and items look very convincing, but is unlikely to think of imitating the fixed format of the cash register’s serial numbers or the way they increase over time. FiftyPlusOne stated that the ID patterns were only two of the “many factors” that raised its suspicions. The rest is in the paywalled part of the original article, which the author of this article was unable to read. In the freely available part, the report does not mention statistical tests such as checking the distribution of final digits.

Before this data check, FiftyPlusOne says it had already checked Wedgewood’s identity: Wedgewood provided two names, one of whom was independently confirmed to have worked in polling, and the reporters also read the methodology description on its website closely. In its statement, Wedgewood accused FiftyPlusOne, The New York Times and other aggregators of failing to vet properly; FiftyPlusOne says this claim is false. Silver Bulletin also says Wedgewood had previously answered questions from several aggregators about its identity, methods and funding, which is why it was included. Judging by the outcome, vetting based on self-reported materials was circumvented, and what ultimately worked was checking at the level of raw data.

Aggregators’ Screening Rules and the Blog Commentary

Polling aggregators usually do not audit raw data poll by poll; instead they decide whether to include a pollster, and how much weight to give it, based on the pollster’s characteristics. The explanation on The New York Times’ polling page says that pollsters that meet at least two of the following three criteria and conduct surveys for nonpartisan sponsors are classified as “select pollsters”: an accurate track record in recent elections; membership in professional polling organizations; and use of probability sampling. These factors affect each poll’s weight in the average. Wedgewood did not meet this standard, so although it was included, its polls always carried the “does not meet the criteria” notice.

Silver Bulletin’s explanation says its averages distinguish between registered voters and likely voters, adjust for each pollster’s systematic lean (house effect), and give more weight to more reliable polls; most of its election polls are drawn from The New York Times’ database. Rules like these rely mainly on pollsters’ self-reported information and track records. A new pollster has no track record yet and can fabricate its self-reported information, so this kind of screening can only do so much. FiftyPlusOne also warns in its report that AI tools make it increasingly easy to generate websites, PDFs and tables, so fakers can make their materials look as polished as those of legitimate research organizations.

On his blog, Gelman praised this as a successful application of data-journalism methods and offered two personal comments. First, Wedgewood said it was an “experiment” only on the very day it was exposed, which he considers “a bit suspicious.” Second, he reiterated a view he first put forward in 2004: there are too many polls in the United States, even counting only legitimate ones. The sheer number of polls wastes respondents’ time and also makes people less willing to take part in surveys that serve the public good, such as public health surveys. Polls and “horse-race” election coverage that cares only about who is ahead feed each other: polls supply headlines for such coverage, and such coverage spawns more polls. Fake polls mixed in only make things worse. These are Gelman’s views, not conclusions of the investigation itself.

Items Readers Can Check

The American Association for Public Opinion Research (AAPOR) Transparency Initiative lists the elements that should be disclosed when survey results are published; the initiative only requires disclosure and does not judge methodological quality. In light of this episode, when readers come across an unfamiliar poll, they can check the following items one by one. Note that these are only screening clues: meeting all of them does not prove the data are real. Wedgewood itself passed some vetting.

Who conducted it and who paid: whether the sponsor and the organization that carried out the survey are named; whether it was commissioned by a candidate, party or advocacy group.
Whether the organization can be verified: whether there are verifiable names of those in charge, an address and a phone number; when the website and social media accounts were created; whether it belongs to professional polling organizations or the AAPOR Transparency Initiative.
Track record: how far the organization’s past polls were from actual election results. Wedgewood’s earlier Louisiana poll even got the leader wrong.
Sampling method: whether it used probability sampling or an opt-in online sample; which platform, list or panel the sample came from.
Questionnaire, timing and population: whether the exact question wording, interview dates, target population (all adults, registered voters or likely voters) and sample size are published.
Weighting and error: whether it explains which variables were used for weighting; whether the margin of error accounts for the design effect. Under AAPOR’s requirements, a nonprobability sample that does not describe the model used should not report a measure of precision, and should instead explain why none is provided.
Data quality control: whether it describes measures such as attention checks, removing respondents who answer too quickly, and preventing duplicate responses, and whether AI was used.
Whether the results and timing are too convenient: whether the results happen to play into hot topics in public discussion, and whether the release timing is suspiciously convenient; when the raw data are available, whether the sample is abnormally “clean.” Clues like these only signal that further checking is needed; they are not, on their own, evidence of fabrication.

Generally only journalists and aggregators can check raw data, and only if they can obtain the data and have comparable real data as a reference. In this case, FiftyPlusOne ran its own survey on the same platform and compared the suspect data against the “fingerprint” of real data. For ordinary readers, the more practical course is to pay attention to how aggregators label pollsters and to treat polls of unclear origin with only a single result with caution.

References

Andrew Gelman. Real political reporters use data journalism techniques to catch fakers at “Wedgewood Polls”. Statistical Modeling, Causal Inference, and Social Science, 2026-10-20.
Mary Radcliffe, G. Elliott Morris, Cooper Burton. FiftyPlusOne just caught a pollster faking data — and they copped to it. FiftyPlusOne, 2026-10-20.
Lucy Campbell. Pollster admits to fabricating data for ‘social experiment’ to expose flaws of industry. The Guardian, 2026-10-21.
Newsweek Editors. Fake Polling Firm Duped Media in ‘Social Experiment’. Newsweek, 2026-10-21.
Azi Paybarah. Pollsters weather storm of fake data, secret sponsorships, with some afraid to give names. Straight Arrow News, 2026-08-20.
Frank Corder. Poll pushed by Colom claiming Senate race with Hyde-Smith is “tied” was fake, fabricated. Magnolia Tribune, 2026-10-20.
Nate Silver, Eli McKown-Dawson. 2026 Midterm Election Forecast: Senate & House. Silver Bulletin, updated 2026-10-21.
The New York Times. President Trump Approval Rating: Latest Polls (About our polling averages). Publication date not stated.
American Association for Public Opinion Research. Transparency Initiative. Accessed 2026-10-22.
Research Methods · Rituals can be imitated; understanding cannot be faked
Compiled and published by Scholay, the all-in-one academic platform. If any content infringes your rights, please let us know and it will be taken down.
Also published on EnterScholar, a community for scholars in the age of AI

Originally published: How Data Journalists Spotted Fabricated Polls

Discuss on EntScholar (Chinese)

Metho · Research Methods — Rituals can be copied. Understanding cannot be faked.

Regression discontinuity designs break in six common ways

Isaac Land — Wed, 21 Oct 2026 11:33:56 +0000

Regression discontinuity (RD) is one of the most credible designs for causal inference from observational data. When a rule assigns treatment based on whether a score crosses a cutoff (a scholarship for test scores above 80, a programme for towns under a population threshold, a policy on one side of a border), people just above and just below the cutoff should be comparable. Any jump in outcomes at the cutoff can then be attributed to the treatment.

But a jump in a plot is not the same thing as a credible effect. This post goes through the mistakes that most often undermine RD analyses and the tools that help avoid them.

A quick refresher

The variable that determines treatment is the running variable (also called the forcing variable or score). RD comes in two forms:

Sharp RD: treatment switches on completely at the cutoff.
Fuzzy RD: crossing the cutoff only changes the probability of treatment. The effect is estimated as the jump in the outcome divided by the jump in treatment take-up, which is essentially an instrumental-variables estimate at the cutoff. It needs a non-zero first stage, an exclusion restriction, and monotonicity, and it identifies a local effect for compliers near the cutoff.

The key identifying assumption is not that the observed outcome is smooth. It is that the potential outcomes, meaning what would happen with and without treatment, change smoothly through the cutoff. Lee and Lemieux (2010) give an accessible treatment, and the design itself goes back to Thistlethwaite and Campbell (1960).

Mistake 1: fitting high-order global polynomials

A common older practice was to fit a 4th-, 5th-, or 6th-degree polynomial to the whole range of the running variable on each side and read off the gap at the cutoff. Gelman and Imbens (2019), in the Journal of Business & Economic Statistics, argue strongly against this, for three reasons:

Global polynomials put surprisingly large and erratic weight on observations far from the cutoff.
Estimates are sensitive to the polynomial order.
Confidence intervals tend to be too narrow, so real coverage falls short of the nominal level.

Better: use local linear (or local quadratic) regression within a bandwidth around the cutoff, with a triangular kernel that gives more weight to points near the threshold.

Mistake 2: choosing the bandwidth by eye (or by result)

How much data on either side of the cutoff should you use? A narrow window reduces bias but increases variance. A wide window does the opposite. Trying several bandwidths and reporting the one with the cleanest result is a garden of forking paths.

Better: use a data-driven, MSE-optimal bandwidth selector and report it up front. Then show sensitivity to, say, half and double that bandwidth as a robustness check, not as a menu to choose from.

Mistake 3: conventional confidence intervals with an optimal bandwidth

Here is a subtle one. MSE-optimal bandwidths are, by construction, wide enough that some bias remains. Conventional confidence intervals ignore that bias, so they under-cover. Calonico, Cattaneo and Titiunik (2014), in Econometrica, proposed robust bias-corrected confidence intervals that account for this. They are now standard, and implemented in the rdrobust package for R, Stata, and Python:

library(rdrobust)
out &lt;- rdrobust(y = outcome, x = score, c = 0)   # local linear, MSE-optimal bandwidth
summary(out)          # report the robust bias-corrected CI
rdplot(y = outcome, x = score, c = 0)            # binned scatter with local fits

Report the point estimate together with the robust confidence interval, not the conventional one.

Mistake 4: not checking for manipulation of the running variable

If people can precisely control their score (retaking a test until they pass, or misreporting income to stay under a threshold), those just above the cutoff are no longer comparable to those just below. A tell-tale sign is bunching: a spike in the density of the running variable on one side.

McCrary (2008) introduced the standard density test. A more recent local-polynomial version is available in the rddensity package. A histogram of the running variable with fine bins is a cheap first look that every RD paper should show.

Note the asymmetry. A clean density test does not prove the absence of sorting, but clear bunching is a serious warning.

Mistake 5: skipping covariate balance checks

If the design works, pre-treatment characteristics should not jump at the cutoff. Run the same RD estimator with baseline covariates (age, prior outcomes, demographics) as the outcome. A significant jump in a variable the treatment could not have affected suggests sorting or a second policy that shares the same cutoff.

Mistake 6: over-extrapolating the result

An RD estimate is a local effect at the cutoff. A scholarship's effect on students scoring exactly 80 says little about students scoring 50. Reporting it as "the effect of scholarships" stretches the design beyond what it identifies. Be explicit about the population the estimate applies to.

A cautionary case: the Huai River heating study

China historically provided free winter heating north of the Huai River and not south of it, which creates a geographic discontinuity. Chen, Ebenstein, Greenstone and Li (2013), in PNAS, used this border to estimate the effect of air pollution on life expectancy. The paper drew wide attention, and also methodological criticism, notably from Andrew Gelman, centring on the use of a global cubic polynomial in distance from the river and how sensitive the estimate was to that choice. A follow-up study, Ebenstein, Fan, Greenstone, He and Zhou (2017), also in PNAS, revisited the question with more data and different specifications.

Whatever your view of the substantive result, the episode is a useful teaching case. It shows why specification choices in RD (polynomial order, bandwidth, what counts as "near" the cutoff) need to be justified and stress-tested rather than picked once.

A practical RD checklist

Confirm the assignment rule exists independently of the outcome data. Don't go hunting for a kink first and call it a policy cutoff afterwards.
Plot the raw data with rdplot or a binned scatter.
Check the density of the running variable for bunching.
Check covariate balance at the cutoff.
Estimate with local linear regression, an MSE-optimal bandwidth, and robust bias-corrected CIs.
Show sensitivity to bandwidth and kernel; avoid high-order global polynomials.
For fuzzy RD, report the first stage.
Run placebo cutoffs where no effect should exist.
State clearly that the estimate is local.

I wrote a longer version of this discussion, with more on the Huai River debate, in this article on common regression discontinuity mistakes.

References

Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82(6), 2295–2326.
Chen, Y., Ebenstein, A., Greenstone, M., & Li, H. (2013). Evidence on the impact of sustained exposure to air pollution on life expectancy from China's Huai River policy. PNAS, 110(32), 12936–12941.
Ebenstein, A., Fan, M., Greenstone, M., He, G., & Zhou, M. (2017). New evidence on the impact of sustained exposure to air pollution on life expectancy from China's Huai River Policy. PNAS, 114(39), 10384–10389.
Gelman, A., & Imbens, G. (2019). Why high-order polynomials should not be used in regression discontinuity designs. Journal of Business & Economic Statistics, 37(3), 447–456.
Lee, D. S., & Lemieux, T. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355.
McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), 698–714.
Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto test. Journal of Educational Psychology, 51(6), 309–317.

EEG classifiers can score well by recognising the person

Isaac Land — Wed, 21 Oct 2026 02:00:22 +0000

Many EEG machine-learning papers report accuracies that look too good. One common reason has nothing to do with the brain signal you care about: the model has learned to recognise who a recording came from. If the same person's data sit in both the training and the test set, the test score answers an easier question than the one the paper claims to answer. This post walks through how that happens, what a recent small replication found, and how to set up the split so your number means what you think it means.

How windowing creates the problem

A typical EEG pipeline cuts each recording into many short windows (say, 4 seconds each), extracts features per window, and treats each window as a sample. A study with 36 participants can easily end up with a couple of thousand windows. That looks like plenty of data.

The trap is the next step. If you shuffle all the windows and do a random 80/20 split, windows from the same person land on both sides. Consecutive windows from one recording are highly similar, and so are different recordings from the same head. A flexible classifier can then learn person-specific boundaries ("for this participant, the alpha power shifts like this between rest and task") that do not carry over to anyone new.

The randomisation unit decides what you are estimating:

Window-level split: how well the model classifies new windows from people it has probably already seen.
Subject-level split: how well the model transfers to people it has never seen.

Neither is wrong in itself. The problem is reporting the first and claiming the second.

What a minimal replication showed

In October 2026, Renish Joshi published a deliberately simple replication in Towards Data Science. He used the public EEGMAT dataset on PhysioNet (EEG during mental arithmetic): 36 healthy students, each with a resting recording and a serial-subtraction recording, sampled at 500 Hz with 10/20 electrode placement (Zyma et al., 2019).

The pipeline was kept fixed:

19 scalp channels, 1–40 Hz band-pass filter, non-overlapping 4-second windows (2,134 windows in total).
Welch power spectra, relative power in theta, alpha, beta and low-gamma bands, giving 76 features per window.
Three classifiers: class-weighted logistic regression, an RBF-kernel SVM, and a 300-tree class-weighted random forest. Scaling sat inside a scikit-learn pipeline, so it was fit on training data only.

Only the evaluation changed. The window-level protocol used 10 stratified random 80/20 splits. The subject-level protocol used 6-fold GroupKFold with participant ID as the group, so no test participant contributed a single training window.

Balanced accuracy dropped for every model:

Classifier	Random windows	Held-out subjects
Logistic regression	0.770	0.648
RBF SVM	0.864	0.602
Random forest	0.664	0.532

ROC-AUC moved the same way. Because a linear model, a kernel method and a tree ensemble all showed the pattern, it is hard to blame one model family.

Do the features know who you are?

A score drop alone doesn't explain the mechanism, so Joshi ran a second test: use the same 76 spectral features to predict which of the 36 participants a window came from. Chance is about 2.8%. The identity classifier reached 95.6%. On a class-balanced subset (15 rest and 15 task windows per person, 1,080 windows), it still reached 94.0%, with 99.5% top-3 accuracy.

He is careful about what this shows. It does not prove the task classifier used identity as its decision rule. It shows the features carry a lot of person-specific information, which is enough to make overlapping splits optimistic. The window-versus-subject gap also persisted on the balanced subset, so class imbalance alone doesn't explain it.

The averages also hide a wide spread. Per-participant balanced accuracy for the subject-level logistic regression ranged from 0.289 to 0.956, and 5 of 36 participants scored below chance.

This isn't a one-off

The replication follows Brookshire et al. (2024) in Frontiers in Neuroscience, who reran two published convolutional networks. For Alzheimer's disease versus subjective cognitive decline, test accuracy was 99.8% with segment-level splits and 53.0% with subject-level splits, where the confidence interval included chance. For seizure detection on the Siena Scalp EEG database, accuracy fell from 79.1% to 65.1%. In a non-exhaustive review of 63 deep-learning EEG papers on disease identification, they found only 17 (27.0%) that clearly avoided this leakage.

Setting up the split correctly

The rule is simple to state: match the split to the generalisation claim. If you claim the model works on new people, no data from a test person may reach training, and that includes anything learned from data along the way.

In scikit-learn, that means grouped cross-validation, with every learned step inside the pipeline:

from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# X: (n_windows, n_features), y: labels, groups: participant ID per window
pipe = make_pipeline(
    StandardScaler(),
    LogisticRegression(class_weight="balanced", max_iter=1000),
)

cv = GroupKFold(n_splits=6)
scores = cross_val_score(pipe, X, y, groups=groups, cv=cv,
                         scoring="balanced_accuracy")
print(scores.mean(), scores.std())

A few things to check beyond the outer split:

Tuning: if you tune hyperparameters, do it with an inner grouped split inside the training participants only (nested CV), not on the outer test folds.
Feature selection, PCA, ICA, normalisation: anything fit on data belongs inside the pipeline so it only sees training participants.
Report per-subject results: show the spread across held-out people, not just the mean.
Run the identity check: if your features can name the participant, window-level results are likely to be inflated.

Window-level evaluation is still fine when the question really is "new sessions from people already enrolled", for example a personalised model that only ever serves the users it was trained on. Just say so.

Limits worth keeping in mind

Joshi's study is one 36-person dataset with hand-crafted spectral features. The two protocols also differ in resampling structure (10 random splits versus 6 grouped folds), so the gap is descriptive, not a formal causal estimate of leakage. Where the person-specific signal comes from (physiology, electrode placement, session effects or residual artefacts) isn't resolved. And as of mid-October 2026, the article page did not link a public code repository, so the numbers can't yet be rerun independently. Brookshire et al. did publish code in their supplementary material, and EEGMAT is freely downloadable.

I first wrote up this case in Chinese, with the figures from Brookshire et al., on Scholay's research-methods section. The same logic applies to any data with repeated samples per person: patients, users, sensors, or schools.

References

Brookshire, G., Kasper, J., Blauch, N. M., Wu, Y. C., Glatt, R., Merrill, D. A., Gerrol, S., Yoder, K. J., Quirk, C., & Lucero, C. (2024). Data leakage in deep learning studies of translational EEG. Frontiers in Neuroscience, 18, 1373515. https://doi.org/10.3389/fnins.2024.1373515
Joshi, R. (2026, October 17). When EEG models see the same people in training and testing. Towards Data Science. https://towardsdatascience.com/when-eeg-models-see-the-same-people-in-training-and-testing/
Zyma, I., Seleznov, I., Popov, A., Chernykh, M., & Shpenkov, O. (2018). EEG during mental arithmetic tasks (version 1.0.0). PhysioNet. https://doi.org/10.13026/C2JQ1P
Zyma, I., Tukaev, S., Seleznov, I., Kiyono, K., Popov, A., Chernykh, M., & Shpenkov, O. (2019). Electroencephalograms during mental arithmetic task performance. Data, 4(1), 14. https://doi.org/10.3390/data4010014
scikit-learn developers. Cross-validation: evaluating estimator performance (section on cross-validation iterators for grouped data). https://scikit-learn.org/stable/modules/cross_validation.html

Cross-validation fails when data leaks between folds

Isaac Land — Tue, 20 Oct 2026 12:44:31 +0000

A model that scores well on the data it was trained on has shown that it can memorise. Cross-validation is the standard way to estimate how it will do on data it has never seen. It only works if one rule holds, though: no information from the validation data may reach the model-building process by any route. Break that rule, which is easy to do without noticing, and your estimate turns optimistic. This post covers k-fold and leave-one-out cross-validation and the leakage patterns that most often break them.

Why not just score on the training set?

The scikit-learn user guide gives the extreme case: a model that simply repeats the labels of samples it has already seen gets a perfect training score and is useless on new data. That is overfitting in its purest form.

The obvious fix is a held-out test set. With small datasets, though, holding data out shrinks the training set further, and the score depends heavily on which rows happened to land in the test split. Cross-validation reuses the data more efficiently and gives a more stable estimate.

The idea is old. Stone (1974), in the Journal of the Royal Statistical Society, Series B, used cross-validation for both choosing a prediction method and assessing it. The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, chapter 7) keeps those two jobs separate:

Model selection: using CV to pick a model or tune hyperparameters.
Model assessment: using CV to report the final model's error on new data.

Mixing the two in a single CV loop is one of the leakage sources covered below.

k-fold and leave-one-out

k-fold CV randomly splits the data into k roughly equal folds. In each round, one fold is held out for validation and the model is trained on the other k − 1. Average the k validation scores and you have the CV estimate. Every observation is used for validation exactly once.

Leave-one-out CV (LOOCV) is the special case k = n: each round holds out a single observation.

Choosing k is a bias–variance trade-off. ESL notes that LOOCV is approximately unbiased for expected prediction error, but its n training sets are almost identical, so the estimate can have high variance. It can also be expensive. Five- or ten-fold CV is the common default and usually a sensible compromise.

In scikit-learn:

from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import LogisticRegression

cv = KFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=cv)
print(scores.mean(), scores.std())

Leakage pattern 1: preprocessing before splitting

This is the classic mistake. You standardise features, impute missing values, or run feature selection on the whole dataset, and only then run cross-validation. Every validation fold has now influenced the means, the imputation values, or the selected features.

Feature selection is the worst offender. ESL section 7.10.2, "The Wrong and Right Way to Do Cross-validation", describes screening thousands of predictors for those most correlated with the label and then cross-validating a classifier on the survivors. Even when labels are pure noise, this procedure can report error rates far below chance. Ambroise and McLachlan (2002) documented the same selection bias in gene-expression studies.

The fix is to put every data-dependent step inside the CV loop. In scikit-learn, that is what Pipeline is for:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif

pipe = make_pipeline(
    StandardScaler(),
    SelectKBest(f_classif, k=20),
    LogisticRegression(max_iter=1000),
)
scores = cross_val_score(pipe, X, y, cv=cv)  # scaler and selector refit per fold

Leakage pattern 2: tuning and reporting on the same folds

If you try 50 hyperparameter settings and report the best CV score, that score is biased upwards: you picked the winner partly because it got lucky on those folds. Varma and Simon (2006) showed how large this bias can be in small-sample settings.

The remedy is nested cross-validation. An inner loop tunes hyperparameters and an outer loop estimates the performance of the whole tuning procedure:

from sklearn.model_selection import GridSearchCV

inner = KFold(5, shuffle=True, random_state=1)
outer = KFold(5, shuffle=True, random_state=2)
search = GridSearchCV(pipe, {"logisticregression__C": [0.01, 0.1, 1, 10]}, cv=inner)
nested_scores = cross_val_score(search, X, y, cv=outer)

Leakage pattern 3: grouped data split row by row

When several rows come from the same unit (multiple scans per patient, many EEG windows per subject, several photos of the same object), a random row-level split puts the same unit in both training and validation. The model can learn to recognise the individual rather than the pattern you care about. The score then reflects identity matching, not generalisation to new people.

Split by group instead:

from sklearn.model_selection import GroupKFold

scores = cross_val_score(pipe, X, y, groups=subject_ids, cv=GroupKFold(n_splits=5))

A useful sanity check: if the group-wise score falls sharply compared with the row-wise score, your earlier number was mostly measuring leakage.

Leakage pattern 4: time

For time series, a random split lets the model train on the future and predict the past. Features built with rolling windows or "future-aware" aggregates make this worse. Use forward-chaining splits such as TimeSeriesSplit, which only ever validate on data that comes after the training window, and add a gap between them if features look back over several periods.

Leakage pattern 5: features that encode the target

Kaufman, Rosset, Perlich and Stitelman (2012) describe leakage as including information that would not be legitimately available at prediction time. Typical examples are an ID field that correlates with outcome because of how records were created, a "date of treatment" column when predicting diagnosis, or a field that is filled in only after the event you are predicting. No splitting scheme protects you from this. You have to ask, for each feature, "would I actually have this value at the moment of prediction?"

It is not just a Kaggle problem

Kapoor and Narayanan (2023), in Patterns, surveyed published research that used machine learning across many scientific fields. They found leakage to be a widespread cause of overstated results, and proposed "model info sheets" so authors could document how they had ruled it out. If you are using ML in a research setting, it is worth reading their taxonomy before submitting anything.

A short checklist

Every step that learns from data (scaling, imputation, encoding, feature selection, resampling such as SMOTE) sits inside the CV loop.
Hyperparameter tuning happens in an inner loop, and the reported score comes from an outer loop or a truly untouched test set.
If rows share a subject, site, or source, split by group.
If time matters, validate only on the future.
Each feature is available at prediction time.
The test set is used once, at the end.

For a broader look at how validation choices affect what a study can claim, I keep notes on research-method topics like this one on Scholay.

References

Ambroise, C., & McLachlan, G. J. (2002). Selection bias in gene extraction on the basis of microarray gene-expression data. PNAS, 99(10), 6562–6566.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.), chapter 7. Springer.
Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9).
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4).
Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B, 36(2), 111–147.
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, 91.

Nested data needs a multilevel model

Isaac Land — Tue, 20 Oct 2026 12:43:16 +0000

When your data come in groups (students inside classrooms, patients inside hospitals, repeated measurements inside the same person), ordinary regression quietly assumes something false: that every row is independent. This post walks through why that matters, how a multilevel (mixed-effects) model fixes it, and how to fit one in R with lme4.

The problem: rows that know each other

Imagine 3,000 students spread across 100 classrooms. Students in the same classroom share a teacher, a timetable, and often a neighbourhood. Their test scores will be more alike than scores drawn at random from the whole population.

Ordinary least squares (OLS) does not know that. It treats 3,000 rows as 3,000 independent pieces of information. In reality, part of the information is duplicated within each classroom, so OLS standard errors come out too small, confidence intervals too narrow, and p-values too optimistic. Classroom-level predictors such as "teacher has a master's degree" are hit hardest, because there are really only 100 independent units for that variable, not 3,000.

Measuring the similarity: the intraclass correlation

The standard way to quantify "how alike are members of the same group" is the intraclass correlation (ICC). Start with an empty (intercept-only) multilevel model:

y_ij = γ00 + u_j + e_ij

u_j is classroom j's deviation from the grand mean, with variance τ²
e_ij is student i's deviation from their classroom mean, with variance σ²

Then

ICC = τ² / (τ² + σ²)

It is the share of total variance that lies between groups, and it also equals the expected correlation between two observations from the same group.

From ICC to the design effect

Survey statisticians have long used the design effect to translate the ICC into lost precision. Following Kish (1965), for groups of average size m:

deff ≈ 1 + (m − 1) × ICC

A purely hypothetical illustration: 100 classrooms of 30 students and an ICC of 0.2 give deff = 1 + 29 × 0.2 = 6.8. The effective sample size is about 3000 / 6.8 ≈ 441, and naive standard errors would be roughly 1/√6.8 ≈ 0.38 of what they should be. Even a modest ICC of 0.05 gives a design effect of 2.45 with classes of 30. Small ICCs matter when groups are large.

Three ways to handle groups

Gelman and Hill (2007) frame the choice as three options:

Complete pooling. Ignore groups and fit one regression. Simple, but it erases between-group variation (which is often the very thing you care about) and gives overconfident standard errors.
No pooling. Fit each group separately, or add a dummy variable per group. Small groups get noisy estimates, and group dummies are perfectly collinear with any group-level predictor, so you cannot estimate the effect of, say, school type at the same time.
Partial pooling (multilevel model). Each group gets its own parameters, but those parameters are drawn from a common distribution. Groups with little data are "shrunk" towards the overall mean; groups with lots of data mostly keep their own estimates.

Partial pooling is the core idea. It is a data-driven compromise between "all groups are the same" and "all groups are unrelated".

Random intercepts and random slopes

A random-intercept model lets each group have its own baseline:

y_ij = γ00 + γ10 x_ij + u0j + e_ij

A random-slope model also lets the effect of a predictor vary by group:

y_ij = γ00 + (γ10 + u1j) x_ij + u0j + e_ij

For example, the link between hours of homework and test scores might be steeper in some classrooms than in others. The variance of u1j tells you how much that slope varies, and the covariance between u0j and u1j tells you whether high-baseline groups also have steeper slopes.

Fitting it in R with lme4

The lme4 package (Bates, Mächler, Bolker & Walker, 2015) is the workhorse for this in R. It ships with the sleepstudy dataset: reaction times for 18 participants measured over 10 days of sleep restriction, so measurements are nested in people.

library(lme4)

# Complete pooling: one line for everyone
m_ols &lt;- lm(Reaction ~ Days, data = sleepstudy)

# Random intercept per participant
m_ri &lt;- lmer(Reaction ~ Days + (1 | Subject), data = sleepstudy)

# Random intercept and random slope for Days
m_rs &lt;- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)

summary(m_rs)

Things worth comparing yourself:

The standard error of Days in m_ols versus m_rs. The OLS version treats 180 rows as independent.
The Subject variance components in summary(m_ri). Plug them into the ICC formula above.
coef(m_rs)$Subject against per-subject lm() fits. You will see shrinkage: participants with extreme individual slopes are pulled towards the average.

lmer() uses REML (restricted maximum likelihood) by default, which gives less biased variance estimates than plain ML. If you compare models that differ in their fixed effects with a likelihood-ratio test, refit with REML = FALSE first. anova() in lme4 does this automatically for merMod objects.

Common mistakes

Too few groups. Variance components are estimated from the number of groups, not the number of rows. Maas and Hox (2005) found that with very few groups the standard errors of the group-level variances were noticeably biased. Five schools with 1,000 students each is a weak basis for a random effect.

Ignoring the design and just clustering standard errors. Cluster-robust standard errors can fix inference for fixed effects, which is a reasonable choice in some settings. But they do not give you estimates of between-group variation, group-specific predictions, or random slopes. Pick the tool that answers your question.

Adding random slopes for everything. Models with many random slopes and correlations often fail to converge or produce singular fits (isSingular() returns TRUE). Start from a theoretically motivated structure and simplify when the data cannot support it.

Forgetting centring. With a random slope on x, the intercept variance depends on where x = 0. Centring predictors, either at the grand mean or within each group, changes what the intercept means. Within-group centring also separates within-group and between-group effects, which are often different things entirely.

Treating the ICC as a test of whether you "need" a multilevel model. A small ICC can still produce a large design effect when groups are big, and the grouping is part of the design whatever the ICC turns out to be.

When to reach for a multilevel model

Use one when:

observations are grouped, and the grouping comes from how the data were collected (sampling, randomisation, or repeated measurement);
you care about variation between groups, or about predictors measured at the group level;
some groups are small and you want stable estimates for them.

If you want a longer, education-focused walk-through of the same ideas, including a worked example with classroom data, the original article on nested data and multilevel models goes into more detail.

References

Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.
Gelman, A., & Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press.
Kish, L. (1965). Survey Sampling. Wiley.
Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods (2nd ed.). Sage.