Like all human endeavors, science is flawed. There are biases, poor practices and downright fraud that undermine the integrity of science. A solution that has been increasingly accepted is to adopt the principles of “open science” that are intended to remedy many of the limitations of science as traditionally practiced. But open science is no panacea, as a recent article in the journal Industrial and Organizational Psychology by Richard Guzzo, Benjamin Schneider and Haig Nalbantian point out. These experienced scientists point out limitations to the open science practices that don’t fit all research situations. In a commentary to this article, I noted that the problems with scientific publication cannot be fixed by adopting a new set of rules. These problems exist because of the reward system for scientists. If you follow the money, you will get to the root of the problem, and that is not where open science is focused.
What Is Open Science
Open science is a list of proposed publication policies that scientific research journals are urged to require of authors. If a scientist wants to publish, they should adhere to these practices intended to improve the accuracy and integrity of what is reported. The basic principle is that scientists should be more transparent about their methods and results. These practices include:
- Pre-registering research methods and proposed analyses prior to conducting the study. You submit your research and data analysis plan to the journal before conducting the study.
- Keeping and sharing detailed research notes compiled as the study is conducted.
- Posting copies of computer code for analyses conducted.
- Posting datasets so others can check analyses and reinterpret results.
It is difficult to fault the idea that scientists should be more open about their methods and analyses. Guzzo’s concern is that total openness is not always possible. For example, sometimes revealing information publicly might cause harm. My issue is that rules alone do not fix problems.
Open Science Is No Panacea
The idea that scientists should be open about their methods and results is at the core of what makes science a science. With science, conclusions are based on evidence and not opinion. Too often, however, scientists are not willing to be entirely open about what they have done. I recall once finding a discrepancy in the analysis of a published paper. When I wrote the author about it, he brushed me off and denied there could be a problem. Furthermore, scientists are famously reluctant to share data when requested. This occurs even in journals that have policies requiring data sharing.
One of the main problems that open science is intended to fix is the disconnect between how scientists conduct their research and how they report it in published articles. These include claiming that you specified hypotheses in advance when you did not (HARKing) and manipulating the data to provide results that you want–referred to as torturing the data until it confesses (p-hacking). These widely-used practices occur because they make it possible for scientists living in a publish or perish world to get articles accepted in high prestige journals.
It has raised people’s awareness of limitations in science, but open science is no panacea. To be most effective, its various proposals need to be applied with flexibility with the research context taken into consideration. To solve integrity problems in science we need to change the reward system, and that is far more difficult that writing new rules.
Photo by Markus Spiske from Pexels
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE
Well said. A thoughtful essay on a sensitive topic.
I can think of two factors that inhibit researchers from making their data available. One is the fear that the next researcher who accesses the data the original researchers collected will have a bias that gives rise to a publication with a misleading conclusion or to a conclusion that varies from the original conclusion. I have had that fear when making data available.
The other factor is a sensitivity that I observed in a colleague, a senior researcher, back when I was starting out. The senior researcher expressed resentment about someone conducting a secondary analysis of the data the senior researcher and his team assembled. He and his team put in a great deal of labor assembling the original data and writing the papers that flowed from those data. The senior researcher resented the secondary analyst who would publish a paper based on those data while having expended a fraction of the original labor.
These are human feelings. They are understandable. I agree that it would be helpful if the reward system in science were changed. Even if, in the unlikely event that the reward system in science were changed, I suspect that the feelings I describe above would persist.
Ditto Paul and Irvin.
If someone wants to cheat, they will find a way to do so regardless of the rule and even while seemingly following them.
Moreover, the assumption that those who want to replicate are bias-free and are more trustworthy than the original researcher is flawed.
Data takes so many resources to gather that having no control over who else uses it and for what purpose is unfair to the original researcher who invested all these resources.
These widely-used practices occur because they make it possible for scientists living in a publish or perish world to get articles accepted in high prestige journals… These problems exist because of the reward system for scientists. If you follow the money, you will get to the root of the problem, and that is not where open science is focused… To solve integrity problems in science we need to change the reward system, and that is far more difficult than writing new rules (with the open science movement).
— very well said!