One of the tools of the open science movement, designed to improve the integrity of research, is pre-registration. The idea is that researchers would pre-register their protocols for design, assessment, and analysis, and then not deviate from the plan. While pre-registration is a worthy idea, it relies on good faith–that people won’t game the system and pre-register after results are known, a phenomenon that I call PARKing. I suspect that PARKing is another questionable research practice (QRP) that occurs because such practices are widespread.
Faculty Are Pressured To Engage In Questionable Research Practices
To be fair, deans and editors don’t pressure faculty to engage in specific practices. However, how colleges and journals are run produces the pressure that underlies them. Publish or perish is very real for academics at research-oriented universities, as it is a requirement for tenure and promotion. On the university side, that pressure is compounded by requiring faculty to publish in a small list of targeted journals, like the Financial Times 50. Journals do their part by adopting a one-size-fits-all requirement for publication where every paper has to adopt a deductive, theory testing approach. Combined these pressures produce the perfect storm that encourage QRPs. Add to it that they are so widespread and it is easy to rationalize doing them.
The Open Science Movement fights QRPs by encouraging greater transparency, and one tool is pre-registration. Ideally one pre-registers all methods prior to conducting the study. Originally a researcher would submit the introduction and method to a journal for peer review. The journal would make an acceptance decision based on the submitted plan without knowing the results. Once accepted, the researcher would conduct the study and submit the completed manuscript, knowing that it was already accepted. Unfortunately, few journals in the organizational sciences have adopted that model. Rather they rely on researchers pre-registering their plan on a publicly available platform, like OSF. You then can refer to the plan in the method section of a paper so readers can see that it was pre-registered.
PARKing Is Another Questionable Research Practice
Pre-registration is a good idea but it is not enough. It is too easy for researchers to treat it as another box to check, along with all the check boxes that journals now require when you submit articles. It is as if checking a box would prevent someone from engaging in a QRP. People engage in QRPs because of the pressure they are under, and as pre-registration becomes more widespread, I am afraid that we will be adding PARKing to the list. The Open Science Movement has raised awareness of the problem, and it has provided some useful tools to improve science. But to finish the job we will need to change the culture.
Culture Change in the Organizational Sciences
The old expression “culture eats strategy for lunch” applies to our research practices. We can create all the policies and check-boxes we want, but that is not enough to overcome evolved practices among researchers. What we need is culture change–the adoption of values and norms that govern our behavior.
- Research integrity should be our foundational value. We should all focus on contributions to true scientific knowledge rather than contributions to our CVs.
- Hold one another accountable. We should all push back on our coauthors and colleagues who engage in QRPs. If a coauthor wants to p-hack, speak up and share your discomfort.
- Editors should hold the line. Editors should make it clear to authors that they are NOT to take reviewer advice to add/drop hypotheses or try different analyses unless what was done is clearly wrong. They should also give feedback to reviewers when suggestions are inappropriate.
- Deans and department chairs should focus on contribution. I know that it is easier to count FT50s, but a paper in a top journal is not guaranteed to have impact on the field, nor is a paper in a lesser outlet going to be ignored. People’s work should be evaluated on its merits rather than outlets.
QRPs undermine the integrity of science, leading to erroneous findings being published that cannot be replicated. It will take a community effort to overcome their widespread use so in the future I can write that PARKing is another questionable research practice that no longer exists.
Photo by Tomas Ryant from Pexels
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE