How peer review works

The label peer-reviewed is one of the most quoted labels in science, and one of the least understood. It does not mean a study is true. It means that other experts read it before it was published and that an editor weighed their views. Knowing what that process involves helps a reader decide how much weight a paper can bear.

What peer review is

Wikipedia's article on peer review defines it as the evaluation of work by one or more people with similar competencies to the people who produced it, and describes it as a form of self-regulation by qualified members of a profession. In academia, it explains, scholarly peer review is typically used to determine whether a paper is suitable for publication, with reviewers who are experts in the topic and anonymous. The article credits Henry Oldenburg, a seventeenth-century German-born British philosopher, as the figure often seen as the father of modern scientific peer review, and notes that the practice developed over the following centuries, with the journal Nature making it standard practice in 1973.

The people involved

Editors

Every journal has editors who receive submitted manuscripts. Their first job is triage: deciding whether a paper fits the journal's subject, meets its basic standards and is worth sending out at all. Many papers are declined at this stage without external review. For those that go forward, the editor chooses reviewers, reads their reports, and makes the decision. The Wikipedia article describes this role: peer review helps the publisher, in practice the editors, decide whether work should be accepted, accepted with revisions, or rejected.

Reviewers

Reviewers, also called referees, are researchers working in the same or a closely related field. They are usually unpaid, and they review as part of the shared obligations of an academic career. A reviewer reads the manuscript closely and writes a report that assesses whether the question is worthwhile, the methods are appropriate, the results support the conclusions, and the paper is clearly written. The report usually ends with a recommendation, but the decision belongs to the editor, who may receive conflicting advice from different reviewers.

Authors

Authors receive the reports and the editor's decision. If revisions are requested, they revise the paper and write a response explaining what they changed and, where they disagree with a reviewer, why. A paper may pass through several rounds before a final decision.

Who knows whose name

Review systems differ in who is told whose identity. In single-blind review, the most traditional arrangement, reviewers know who the authors are but authors are not told who reviewed them. In double-blind review, neither side is told; the Wikipedia article calls the case where the identities of authors and reviewers are not revealed to each other dual-anonymous peer review. The aim is to reduce bias from an author's reputation, institution or country, although in small fields reviewers can often guess. In open review, some or all identities are known, and in some journals the reports themselves are published alongside the paper. Each system trades one kind of fairness or accountability against another, and none is perfect.

What review can catch

A good reviewer can find many problems: a missing control, a statistical method that does not fit the data, a conclusion that goes beyond the results, an important earlier study that was ignored, or a description too vague to repeat. Reviewers often improve papers considerably, and the knowledge that work will be read critically shapes how researchers design and report it in the first place.

What review cannot catch

Review has real limits, and the Wikipedia article lists the main criticisms: failure to prevent the publication of invalid research or scientific fraud, low correlation with a paper's future impact, slowness, and lack of accountability. It also notes that impartial review can be hard to achieve, especially for interdisciplinary work, and that it requires a community of experts in what is often a narrowly defined field.

The reason for most of these limits is simple. Reviewers see only what the authors send them. They do not usually repeat the experiments, inspect the raw data or visit the laboratory. A study that was carefully faked, or one where an honest error lies hidden in unreported steps, can pass review. Reviewers are also busy people working for free, and the depth of review varies widely between journals and between individual reviewers.

What this means for a reader

Peer review is best thought of as a filter, not a guarantee. A peer-reviewed paper has passed a check that an unreviewed one has not, and that is worth something. But confidence in a finding comes from more than one paper: from independent studies reaching similar results, from methods that other laboratories can repeat, and from the slow accumulation of evidence. Reviews that bring many studies together, such as those cited throughout this guide, are valuable precisely because they weigh a body of work rather than a single result.

It also means reading papers with a reviewer's eye. The questions a reviewer asks, whether the method fits the question, whether the results support the conclusion and what the limitations are, are questions any reader can ask.