Peer Review Limitations

Peer review is science's quality filter. It's also a small club where everyone knows each other.

The idea sounds solid: before research gets published, other experts in the field check it. But in practice, most specialised fields are tiny. The reviewers and the authors compete for the same grants, publish in the same journals, and attend the same conferences. They know whose paper they're reviewing even when it's "anonymous." They have opinions about who should be right. And they have strong career reasons to protect the theories their own work is built on.

Reviewers are unpaid, overworked, and rarely replicate the experiments they're evaluating. They check whether the methods sound reasonable and the conclusions follow from the data presented — but they can't detect p-hacking, selective reporting, or outright fabrication from a manuscript alone. High-profile fraud cases have sailed through peer review at top journals. So have studies with basic statistical errors that undergraduates could catch.

None of this means peer review is useless. It catches obvious mistakes and improves papers at the margins. But it's a filter, not a guarantee. When someone says a finding is "peer-reviewed" as though that settles the matter, they're confusing a minimal check with a thorough validation. Peer review means someone glanced at it. It doesn't mean it's true.

---

References

• John Ioannidis — Why Most Published Research Findings Are False (2005)
• Ben Goldacre — Bad Science (2008)
• Thomas Kuhn — The Structure of Scientific Revolutions (1962)