Like all human endeavors, science is flawed. There are biases, poor practices and downright fraud that undermine the integrity of science. A solution that has been increasingly accepted is to adopt the principles of “open science” that are intended to remedy many of the limitations of science as traditionally practiced. But open science is no panacea, as a recent article in the journal Industrial and Organizational Psychology by Richard Guzzo, Benjamin Schneider and Haig Nalbantian point out. These experienced scientists point out limitations to the open science practices that don’t fit all research situations. In a commentary to this article, I noted that the problems with scientific publication cannot be fixed by adopting a new set of rules. These problems exist because of the reward system for scientists. If you follow the money, you will get to the root of the problem, and that is not where open science is focused.
What Is Open Science
Open science is a list of proposed publication policies that scientific research journals are urged to require of authors. If a scientist wants to publish, they should adhere to these practices intended to improve the accuracy and integrity of what is reported. The basic principle is that scientists should be more transparent about their methods and results. These practices include:
- Pre-registering research methods and proposed analyses prior to conducting the study. You submit your research and data analysis plan to the journal before conducting the study.
- Keeping and sharing detailed research notes compiled as the study is conducted.
- Posting copies of computer code for analyses conducted.
- Posting datasets so others can check analyses and reinterpret results.
It is difficult to fault the idea that scientists should be more open about their methods and analyses. Guzzo’s concern is that total openness is not always possible. For example, sometimes revealing information publicly might cause harm. My issue is that rules alone do not fix problems.
Open Science Is No Panacea
The idea that scientists should be open about their methods and results is at the core of what makes science a science. With science, conclusions are based on evidence and not opinion. Too often, however, scientists are not willing to be entirely open about what they have done. I recall once finding a discrepancy in the analysis of a published paper. When I wrote the author about it, he brushed me off and denied there could be a problem. Furthermore, scientists are famously reluctant to share data when requested. This occurs even in journals that have policies requiring data sharing.
One of the main problems that open science is intended to fix is the disconnect between how scientists conduct their research and how they report it in published articles. These include claiming that you specified hypotheses in advance when you did not (HARKing) and manipulating the data to provide results that you want–referred to as torturing the data until it confesses (p-hacking). These widely-used practices occur because they make it possible for scientists living in a publish or perish world to get articles accepted in high prestige journals.
It has raised people’s awareness of limitations in science, but open science is no panacea. To be most effective, its various proposals need to be applied with flexibility with the research context taken into consideration. To solve integrity problems in science we need to change the reward system, and that is far more difficult that writing new rules.
Photo by Markus Spiske from Pexels
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE