Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions. By Richard Harris. New York: Basic Books. 2017. ISBN: 0465097901. 288 pp. Hardcover, $28.
Richard Harris, a distinguished science journalist with National Public Radio, presents a deeply skeptical view of science in this journalistic but accurate depiction of the “other side” of science.
The story opens: “Each year about a million biomedical studies are published in the scientific literature. And many of them are simply wrong.” Harris describes the experience of C. Glenn Begley, a scientist at biotech company Amgen, who discovered that much of the academic research his company relied on could not be replicated. In 2012, Begley published a highly cited commentary in Nature calling for better preclinical research. Other scientists began publishing similar results. In 2015, Leonard Freedman (of the Global Biological Standards Institute in Washington, D.C.) summarized estimates by other scientists that between 51 and 89 percent of studies were in some way unreproducible, at an estimated cost to the U.S. taxpayer and industry of about $28 billion per year. Harris tells of other scientists who came to similar conclusions.
Harris personalizes his story by recounting tragedies to patients in clinical trials of drugs that do not work as a result of, he suggests, poor decisions by drug companies based on unreliable preclinical science. He describes an unsuccessful $20 million clinical trial of a drug to combat amyotrophic lateral sclerosis (ALS): “The results: fail, fail, fail, fail, fail, fail, fail. Science administrators had assumed that the academic scientists had all done the legwork carefully. They had not.”
Other chapters recount a variety of loosely related issues to the reliability of biomedical research. These include the unreliability of animal and in vitro (cellular) studies for drug development, use of genetic markers from tumors to help choose therapy for patients, and contamination of cell lines—all real problems that are discussed in the scientific literature. Harris personalizes these issues with accounts of victims of bad science and honest scientists who are trying to set matters straight.
In a long section that is generally accurate but highly oversimplified, Harris discusses the root cause of much false discovery in science: inappropriate use of null hypothesis significance testing and the criterion of p<0.05 to establish “statistical significance” as the threshold for publication of results. Statisticians have been lecturing scientists for decades about the fallacy of this approach, but it remains routine practice for many scientists. This issue turns on what, to most readers, will be arcane issues in statistics. From my experience as a journal editor, the issues are not widely understood even by scientists themselves, and as a journal editor I constantly struggle with manuscripts that draw invalid conclusions from weak data using inappropriate statistical methods.
Harris suggests some solutions to the problem. He praises scientists who work endlessly to get experiments right by identifying unanticipated sources of variability in their experiments and work to improve the standards of reporting of science. Both funding agencies, including the U.S. National Institutes of Health (NIH), and scientific publishers have developed programs to encourage authors to clean up their acts, such as identifying sources of the cell lines and other research materials they used—which astonishingly many authors fail to report in their papers. Reviewers of NIH grant applications (myself included) are now asked to assess the rigor of proposed studies. But scientists are often looking for small effects in complex systems, and the results of an experiment can depend on tiny details of experimental technique. There is only so much a reviewer can do.
Is the problem really as bad as Harris lets on? Yes and no. The reliability of science depends a lot on where it is done and for what purposes. Pharmaceutical companies, faced with rapidly escalating costs once a potential drug has passed the preclinical stage, have much to gain by careful quality control even in preclinical work. Moreover, drug development and testing is subject to stringent controls of good laboratory practice that, while imposing burdensome paperwork requirements, at least keeps error rates down as well. While poor quality preclinical work might lead to development of ineffective drugs, there are many reasons only one out of ten experimental drugs that enter Phase I clinical trials eventually receives FDA approval (see, for example, https://www.amplion.com/inthenews/report-suggests-drug-approval-rate-now-just-1-in-10). There are simply too many variables at play when scaling research from animal studies to clinical trials—complexities that Harris alludes to but that are difficult to control or predict. Drug companies would dearly love to increase their success rate, but improving the reliability of preclinical research is only part of the solution.
The situation in academic labs is often different. Scientists want to come up with new ideas, do exciting research, publish papers in high-status journals, give featured lectures at conferences, and write grant applications that will excite reviewers and maintain the flow of funding. Postdocs must publish during their relatively short appointments if they hope to have a research career. Scientists need to get product (papers) out the door. As Harris correctly points out, the hypercompetitive atmosphere of most academic science encourages scientists to cut corners. In the September 12, 2018, issue of Nature, John Ioaniddis and colleagues reported that 9,000 scientists around the world each have published a paper on the average of once every five days in at least one recent year. Do they even have time to read their own papers?
Clearly the reliability of much published research is poor, and the problem is becoming much worse with the flood of new online journals, many lacking careful peer review. Here are two examples. A search on Google Scholar will reveal thousands of studies that propose methods to diagnose disease using data from online databases and machine learning software. How many of these diagnostic methods will prove successful in real-life medicine? Not many, I’d guess.
A second example is a research field of mine—biological effects of electromagnetic fields, including cell phone radiation. The literature is swamped with low-quality studies reporting biological effects of electromagnetic fields. Many of the studies appear simply to be one-shot experiments, reporting small effects in noisy data using inappropriate statistics, with study designs lacking such elementary precautions as use of concurrent sham controls. Health agencies generally give such papers little weight in their reviews of the literature, and they consistently find no clear evidence for health risks of electromagnetic fields within international exposure limits. For their part, people concerned about what they consider health risks of electromagnetic fields can cherry pick this diverse literature to support alarmist positions.
I invited Lorne Trottier, a nonscientist who has been active in the skeptics’ community for many years, to comment further:
We Need to Put the Reproducibility Problem in Perspective
“Let us hope it is not true, but if it is, let us pray it does not become widely known.” This (perhaps apocryphal) quote was supposedly made by the wife of the Archbishop Wilberforce about Darwin’s theory of evolution. While interpretations vary, most take this remark to reflect her fear that science will damage people’s belief in religion (at least as she viewed it). In the present context it might be used to express fear that Harris’s skepticism about science will damage people’s faith in science itself.
At a conference on the reproducibility of science I attended several years ago, a participant made a similar remark to that attributed to Mrs. Wilberforce. I share this concern about the credibility of the scientific enterprise in the eyes of the public. We live in a time when established science is under attack by creationists, vaccine and climate change deniers, and other purveyors of pseudoscience. The fact that the majority of published scientific studies are wrong in some way could provide a major boost to science deniers of all sorts.
Defenders of the scientific enterprise need to be prepared to put the problem of reproducibility in perspective, while working actively to improve the quality and reproducibility of research. We need to stress that science does yield reliable knowledge. Despite fits and starts, science identified the infectious agent for AIDS, developed sensitive tests for the virus (protecting countless recipients of blood transfusions), and found ways to stop the replication of the virus (allowing individuals infected with HIV to live normal lives). We have reliable knowledge that vaccines do not cause autism, Laetrile does not cure cancer, and smoking does cause lung cancer. This knowledge comes through the collective enterprise of many scientists, whose work, individually and in some respects, may be flawed. That said, scientists should work to reduce these flaws and improve scientific rigor.
Skeptics now have to confront a different kind of skepticism, against science itself, and we need to be prepared to argue our case for science.
Lorne Trottier lives outside of Montreal, Quebec, Canada.