A Scientist Pushes Psychology Journals toward Open Data

Tomasz Witkowski

The March 9 Nature reported some disturbing news: “An editor on the board of a journal published by the prestigious American Psychological Association (APA) has been asked to resign in a controversy over data sharing in peer review.” The controversy arose when psychologist Gert Storm declared that he would review only papers whose data he could see. His declaration is not some whim of a fractious scientist. He is one of a few hundred scientists to proclaim that, starting in 2017, they will begin rejecting papers whose authors refuse to publicly share the underlying data or explain why they can’t. But the story of Gert Storms and other scientists asking for data to be shared is not a new phenomenon.

It has been more than a half a century since Leroy Wolins, a psychologist from Iowa State University, authorized one of his students to write a letter to thirty-seven authors of original research articles asking them to submit the raw data that their studies were based on. The student intended to make practical use of the data in a study of his own. Out of thirty-seven authors, thirty-two replied to the request. However, twenty-one of those thirty-two researchers who replied informed “with tremendous regret” that their data had been accidentally destroyed, lost, or archived in such a way that it was impossible to retrieve. Only nine researchers (24 percent of the initial group) appeared to be willing to make their data available. Wolins, an expert in statistics, took a closer look at the data received and concluded that only seven of them met the requirements of what we might call a reliable statistical analysis (Wolins 1962). Surprised by the scientists’ responses, Wolins described the experience in American Psychologist, thus triggering a heated scientific debate over the availability of raw data.

In 1973, James R. Craig and Sandra C. Reese decided to replicate Wolins’s “accidental” study. They thoroughly planned and designed a new study to check whether there had been any improvement in the availability of raw data since Wolins’s attempt more than a decade earlier. They asked fifty-three researchers to provide them with the raw data that they had used in their research. The results were more optimistic than those presented by Wolins. Nine researchers refused to reveal the data, claiming it had been lost, destroyed, or was otherwise unavailable. Only about half of the researchers declared their readiness to cooperate. Twenty of them sent analyzed or summarized data. Seven declared they would cooperate but only under certain conditions (Craig and Reese 1973).

Warren Eaton, a scientist from the University of Manitoba, reported even more optimistic results in 1984. His success rate of 73.5 percent (twenty-five of thirty-four) was much better than that of Wolins and Craig and Reese (Eaton 1984). However, in 2006, a group of Dutch psychologists from Amsterdam obtained diametrically opposite results, virtually identical to those of Wolins from 1962. In an attempt to reanalyze data sets to assess the robustness of the research findings, they contacted 141 authors of 249 studies. They received thirty-eight positive answers and managed to obtain actual data sets from sixty-four studies. This figure represents 25.7 percent of the total number of 249 data sets; 73 percent of the authors did not share their data (Wicherts et al. 2006).

Inspired by the examples of Wolins and other researchers, I decided to replicate their studies on my own. To do so, I randomly selected fifty empirical studies, all published in the past twelve months. I requested that the authors share the raw data from their studies. Out of fifty requests, I received twenty-seven replies, which accounted for 54 percent of the entire sample. I received only seven raw data sets. A further seven responses could be labeled as “willing to cooperate.” My success rate was 30 percent of the total sample. It is remarkably close to the results presented by Wolins in 1962 and subsequently by Wicherts et al. in 2006.

I wrote a commentary article about my study and submitted it to American Psychologist, where discussion of this particular topic started over half a century ago. To my disbelief, I was told that this particular subject has actually been completely explored and further discussions from now on will be closed: “As you know, every member of APA receives American Psychologist. The limited space we have available constrains us, and we can accept only articles that are of interest to a broad range of psychologists and that have broad consequences for the science and practice of psychology. . . . The American Psychologist has recently published a comment on this topic, and the present proposed comment does not really add any new additional information to the discussion” (Witkowski and Zatonski 2015).

In the light of such explanations, it is not surprising that Gert Storms has been forced to resign after his refusal to review papers without raw data. How does one dare to reopen an issue already resolved once and for all? Fortunately, Storms is not alone. The idea, called the Peer Reviewers’ Openness Initiative, was launched by psychologists hoping to increase transparency in the field of psychology, so often shocked by scientific controversies and scandals. On the other side there are representatives of the APA acting in accordance with previously established rules and laws but not necessarily in accordance with scientific ethics. In this game, the stakes are high: truth and transparency.


References

  • Craig, J.R., and S.C. Reese. 1973. Retention of raw data: A problem revisited. American Psychologist 28: 723.
  • Eaton, W.O. 1984. On obtaining unpublished data for research integrations. American Psychologist 39: 1325–1326.
  • Wicherts, J.M., D. Borsboom, J. Kats, et al. 2006. The poor availability of psychological research data for reanalysis. American Psychologist 61: 726–728.
  • Witkowski, T., and M. Zatonski. 2015. Psychology Gone Wrong: The Dark Sides of Science and Therapy. Boca Raton: Brown Walker Press, 51–60.
  • Wolins, L. 1962. Responsibility for raw data. American Psychologist 17: 657.

The March 9 Nature reported some disturbing news: “An editor on the board of a journal published by the prestigious American Psychological Association (APA) has been asked to resign in a controversy over data sharing in peer review.” The controversy arose when psychologist Gert Storm declared that he would review only papers whose data he …

This article is available to subscribers only.
Subscribe now or log in to read this article.