From The Truth Wears Off by Jonah Lehrer in The New Yorker:

An excellent article on the publication bias- keep in mind that in order to be peer reviewed it has to be published- although up to a third of articles claimed to be peer reviewed in one of the IPCC report were not. IN Mann’s case 43 of the peer reviewed had also co – authored studies with him- again I question some of the peer review process. It is supposed to be squeaky clean. If it is not then there is corruption that is quite damaging to the scientific profession.

Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for. A “significant” result is defined as any data point that would be produced by chance less than five per cent of the time. This ubiquitous test was invented in 1922 by the English mathematician Ronald Fisher, who picked five per cent as the boundary line, somewhat arbitrarily, because it made pencil and slide-rule calculations easier. Sterling saw that if ninety-seven per cent of psychology studies were proving their hypotheses, either psychologists were extraordinarily lucky or they published only the outcomes of successful experiments. In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology.

Stated in another way- 97% of egg-lovers claim eggs are awesome. 97% of the Flat Earth Society believe the Earth is flat. 97% of the Labour Party believe in Labour. 97% of musicians think that people need music in their lives. 97% of climate scientists believe in anthropogenic global warming. 97% of YouTube commenters believe that they know everything. I think there’s a pattern here.

Lehrer further illuminates-

Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe. ♦

Tag Archives

The Publication Bias

Recent Posts

Recent Comments