I have noticed an alarming trend recently. Peer reviewers are ignoring that the statistical analyses used in the manuscripts that they reviewed were pre-registered. Rather, they are asking the authors to analyze data a different way. Statistical analyses are based on probabilities, and once you start analyzing data in more than one way, those probabilities become distorted. If I analyze twice, and get different results, I cannot know which gave me the correct answer, but I can choose the one that gave me the better chance of getting my results published. Pre-registration (publicly announcing your analysis plan before data are analyzed) was created to combat the temptation to try different things in data analysis and report the best results. For that reason peer reviewers should respect pre-registration.
P-Hacking Is a Widespread Practice
P-hacking is the practice of analyzing data in different ways in order to achieve statistical significance. It is how we “torture the data until it confesses”. Research has shown that if you try enough things, you can raise your chances of finding statistical significance by mistake from the accepted 1 chance in 20 (p < .05) to near certainty. Imagine you are trying to decide whether you or your friend are going to pay for lunch, so you flip a coin. This is fair (assuming it isn’t a trick coin), because the chance of each of you winning is 50/50. But suppose you lose and then insist on a do-over so you get a second chance? What if you keep insisting on a do-over until you finally win? This is analogous to p-hacking.
This practice of repeated analysis has undoubtedly contributed to the replication crisis–the realization that many established phenomena reported in the literature cannot be replicated and are apparently bogus. A clever study comparing dissertations to journal versions of the same research found that among faculty at top business schools publishing in prestigious journals, nearly 3 of 4 papers showed evidence of p-hacking. This level of engaging in what everyone considers a questionable research practice is a concern for the integrity of our science.
Pre-registration To the Rescue
Pre-registration is a practice intended to combat p-hacking. The idea is that you log into a publicly accessible website and specify the methods you will use in your study, including the data analysis plan. Assuming people are honest and truly pre-register prior to analysis, this cuts down on p-hacking because the researchers have committed themselves to follow a particular plan. Pre-registering and following the plan is analogous to flipping the coin once. It respects the .05 statistical significance level.
There has been a movement in many fields to encourage pre-registration in order to reduce the number of erroneous findings (significance due purely to chance rather than a real phenomenon). If everyone follows the statistical rules, only 1 in 20 findings should be significant by chance, but that is not the case in our academic literature where rates can be much higher.
Peer Reviewers Should Respect Pre-registration
The Open Science movement has raised awareness of the destructive nature of p-hacking. Today authors are encouraged to avoid the practice, and I have noticed that more and more papers report that their procedures and analyses were pre-registered. Unfortunately, there are counter-forces in the peer review process. Although peer reviewers should respect pre-registration, too often they do not and ignore that the analysis plan was pre-registered. All the p-hacking attention so far as been on the behavior of authors, but it is time to pay attention to the role of peer reviewers in the questionable research practices that are so widespread.
The review process in the organizational sciences is seriously broken for a variety of reasons, and how authors are treated borders on the unethical. Today reviewers feel empowered to insist that authors not only fix data analysis errors, but that they totally change their approach to the one that each reviewer feels is optimal, which can be quite subjective and based on common practice idiosyncratic in a field. One thing I’ve learned over the years is that there are many ways to analyze data and each has strengths and limitations. There is generally no one best way, but rather there are ways that might favor one limitation over the other. Issues I have seen reviewers raise include parametric vs. nonparametric statistics, eliminating vs. not eliminating outliers, adding versus deleting control variables, logistic vs. least squares regression, and using vs. not using transformations to normalize data. Each of these approaches has pros and cons, and each have their proponents.
When analyses are pre-registered, the question of whether the approach used was the “best” one should be off the table. Rather the reviewer’s role should be to answer three questions, with the last one optional.
- Was the analysis done correctly? In my experience as a reviewer, it is rare to catch mistakes, but it does happen. Authors can report results that seem unlikely. For example, the mean of one group is 4 and the other is 5, but they report the mean difference of 2.1.
- Did the analysis make sense? Given the research questions or hypotheses being tested, is the analysis able to provide a meaningful answer? It the hypothesis is that a treatment condition in an experiment will have a higher score than a control condition, there are a variety of analyses that can provide an answer.
- Are there confounds or alternative explanations for the results? The issue here is not one of changing the analyses, but of testing alternative explanations–adding not subtracting. If the reviewer can provide a reasonable rationale that a third factor might account for results, and that factor was measured, it is perfectly reasonable to ask for an additional analysis controlling for that factor and then reporting both. The discussion section can then note that the alternative was ruled in or out as a possible explanation. What isn’t reasonable is to make a general statement that the authors should have including unspecified control variables just because most papers have control variables, that is, control variables for the sake of control variables.
What peer reviewers should avoid is asking for a different approach when what was reported was done correctly and makes sense for the question being addressed. When peer reviewers violate the pre-registration plan, editors should be quick to tell authors to ignore the suggestion. In my experience they are not likely to do so. Both editors and peer reviewers should respect pre-registration because it is more important to defend research integrity than try to make every paper each reviewer’s personal version of perfect.
Image from Pexels.
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE