In news stories, grant applications, and everyday arguments, "peer reviewed" is often treated as a stamp: if a paper has passed that test, the finding can be treated as settled. The phrase does real work. It also promises more than the process can deliver. The useful habit is to ask what peer review actually is, before treating the label as a verdict on the world.
The core idea is this: peer review is a limited reading of a submitted manuscript by a small number of other researchers. It can catch some problems and improve a paper. It does not certify that the results are true, complete, or reproducible.
In the common journal workflow, an editor sends a manuscript to one or more outside reviewers in a related field. Those reviewers read what the authors have written — the argument, the methods as described, the tables and figures, the claims in the discussion. They may recommend publication, revision, or rejection, and they often ask for clarifications or further analysis. The editor then decides. None of this is trivial. A careful review can expose a muddled comparison, an overstated conclusion, a missing control, or a gap between the abstract and the results. There is also evidence that papers can become clearer between submission and publication, in part because authors have to answer criticisms. For a reader, "peer reviewed" usually means at least this: the work was not merely posted by its authors; someone with relevant training was asked to look at the manuscript and comment.
That is already a meaningful filter. It is not, however, the same thing as a re-run of the experiment. Reviewers typically do not receive the raw data, do not visit the laboratory, and do not repeat the measurements. They work, as the former BMJ editor Richard Smith has put it, largely on trust. They can ask whether the design as described makes sense. They cannot, in the ordinary process, confirm that every number in a table came from the patients or samples the authors claim. Fraud and serious data problems are sometimes noticed by a reviewer, but they are more often found later, when others try to reuse the data, repeat the analysis, or simply fail to reproduce the result. Treating "peer reviewed" as a fraud detector will systematically over-trust the literature.
Even as a check on honest errors, peer review is uneven. In 2008, Sara Schroter and colleagues reported a study of 607 BMJ reviewers who were sent three test manuscripts, each seeded with nine major and five minor methodological errors. On average, reviewers found about 2.6 to 3.1 of the nine major errors, depending on the paper. A short training package produced only small improvements. The authors' conclusion was blunt: editors should not assume that reviewers will detect most major errors. Other work has made a similar point. The process can help. It does not reliably empty a manuscript of important flaws.
Reviewers also do not speak with one voice. In 2010, Lutz Bornmann, Rüdiger Mutz, and Hans-Dieter Daniel published a meta-analysis in PLOS One of inter-rater reliability in journal peer review: 48 studies, more than 19,000 manuscripts. They found that agreement among reviewers was low, reporting a mean intra-class or Pearson coefficient of 0.34 and a mean Cohen's kappa of 0.17. Disagreement in science is not in itself a scandal; fields advance by arguing. But it does undercut the idea that a published paper has been certified by "the field." It has usually been read by a few people, who may reasonably differ, and then accepted by an editor who has to weigh those differences. A second set of reviewers might have asked different questions.
So what should the label be used for? It is fair to treat peer-reviewed publication as a sign that a manuscript has passed a formal, if imperfect, scrutiny of its internal argument. It is a reason to take the paper seriously enough to read the methods, not a reason to skip them. It is not a guarantee that the sample was adequate, that the statistical choices were the best ones, that competing explanations were ruled out, or that another team will see the same effect. Those questions are answered, when they are answered at all, after publication: by replication, by re-analysis, by letters, by data sharing, and by the slow accumulation of related studies.
None of this means that an unreviewed draft and a reviewed paper are interchangeable. Informal comments from colleagues, preprint discussions, and journal review all belong on a spectrum of scrutiny, and formal review is usually more structured than an unsolicited comment online. The mistake is not in valuing that scrutiny. The mistake is in collapsing it into a binary: reviewed, therefore established; not yet reviewed, therefore worthless. Plenty of important work has been improved by review. Plenty of reviewed work has later been corrected, limited, or overturned. Both can be true at once.
A more precise way to talk about a paper is therefore also more modest. Instead of "It is peer reviewed, so it is true," it is better to say what the reviewers could actually have seen: a written account of a study, judged by a few colleagues to be clear enough, plausible enough, and relevant enough to enter the public record. The record remains open to correction. That is not a dismissal of peer review. It is a description of what the process is for — and of the work that still belongs to readers after the stamp has been applied.
More research explainers: https://www.crushsci.com
Top comments (0)