Statistical Control Is Academic Rigor Theater

The incorporation of statistical control variables into our analyses has been common practice in many social science fields including the organizational sciences and psychology. The other day I came across an interesting paper from 2020 admonishing us to be more careful in our use of these variables. Anna Wysocki, Katherine Lawson and Mijke Rhemtulla provided a nice technical explanation of pitfalls of the intemperate use of control variables, joining papers taking a more conceptual approach. I’ve written papers on the topic myself, and I have come to the conclusion that statistical control is academic rigor theater. Researchers throw in control variables (and reviewers ask them to do so if they did not) because it gives the illusion that the research is more rigorous than it is. After all, our conclusions must be correct if the relationships between target variables still exist after controlling for a handful of other stuff. Aren’t they?

A Brief History of Control Variable Concerns

Cautionary papers about using control variables go back a century. Some of my favorites:

• Barbara Burks in 1926: This was the earliest I could find, and if the field had listened to her, we wouldn’t have had all the papers that followed.
• Paul Meehl in 1971: In “A Reply to Schwarz” Meehl admonished the field to stop blindly controlling for demographic variables because it can lead to erroneous inference.
• Paul Spector et al. in 2000: Our “Baby with the Bathwater” paper that explains why we should not automatically control for negative affectivity in stress studies.
• Tom Becker in 2005: This began a series of papers in the journal Organizational Research Methods critical of the use of control variables.
• Tom Becker et al. in 2016: Becker assembled six methodologists who had written on the topic to write a best practices-type paper.
• Anna Wysocki et al. 2020: Explains what inclusion of inappropriate control variables does to estimates of regression coeficients.

What is remarkable is that these methodologists, coming from different fields, have all reached the same conclusion. There is consensus about the impact of control variables on results and conclusions. What is curious is why their routine use continues despite a century-long parade of admonishments not to.

Statistical Control Is Academic Rigor Theater

Control variables can be helpful if used appropriately, but most uses are far from ideal. We generally see them used in the hope of providing a more accurate indication of how two variables are related. Mike Brannick and I referred to this as the purification principle—the belief that adding a bunch of control variables will reveal a more accurate picture of how variables relate to one another. They can be used to “purify” the correlation between two variables, or they can be introduced into complex structural equation models for the same purpose. But as the Wysocki team points out, adding control variables is only helpful if they are causes of the other variables you are studying. If you include variables that are not causes, including controls makes your results inaccurate, and can lead to erroneous conclusions.

But if control variables are only helpful under limited and uncommon conditions, why are they used almost universally? In part it is because science can be driven by common practices such as including control variables–the “everyone else is doing it” phenomenon. Most researchers will include them in papers submitted for publication in a scientific journal, but if they do not, peer reviewers or editors are likely to ask for them. The reason is the false belief that including control variables increases accuracy of results and therefore conclusions are more trustworthy when they are not. In other words statistical control is academic rigor theater. It provides the illusion of rigor without the reality of it.

Photo by Monica Silvestre from Pexels

SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE