The real risks of cherry picking scientific data by Matt Ridley and his blog The Rational Optimist
The Tamiflu tale is that some years ago the pharmaceutical company Roche produced evidence that persuaded the World Health Organisation that Tamiflu was effective against flu, and governments such as ours began stockpiling the drug in readiness for a pandemic. But then a Japanese scientist pointed out that most of the clinical trials on the drug had not been published. It appears that the unpublished ones generally showed less impressive results than the published ones.
Imbued as we are with an instinctive tendency to read meaning into nature, we find it counter-intuitive that many experiments get significant results by chance and that the way to check if this has happened is to repeat the experiment and publish the result. When the drug company Amgen tried to replicate 53 key studies of cancer, they got the same result in just six cases. All too often scientists publish chance results, or “false positives”, like gamblers or fund managers who tell you about winners they backed.
Outside medicine, we popular science authors are probably guilty of too often finding startling results in the scientific literature and drawing lessons from them without waiting for them to be replicated. Or as Christopher Chabris, of Union College in Schenectady, New York, harshly put it about the pop-psychology author Malcolm Gladwell: cherry-picking studies to back his just-so stories. Dr Chabris points out that a key 2007 experiment cited by Gladwell in his latest book, which found that people did better on a problem if it was written in hard-to-read script, had been later repeated in a much larger sample of students with negative results.
We seem unsettled by not knowing. We are also today drowning in data that will support conclusions that can prove to be very deceptive. This requires us to be more skeptical, not less. If we do not try to duplicate research results aggressively we risk drawing a lot of wrong conclusions. This is also easily abused by interested parties who can prey on the statistical ignorance of even our most educated leaders. When these conclusions support political objectives, hostility greets skepticism and verification.
Google and our mobile devices gives us all the answers and endless information at the touch of a finger. The questions and the wisdom to understand what we read is still up to us. Without this most human part of intelligence all we have done is speed up ignorance.