Last week I decide to update the readings for my USF executive Doctor of Business Administration class on organizational climate and culture. I have been teaching the class for about 10 years, and periodically I will replace older readings with newer ones. My intent with the reading list is to represent a wide range of approaches and methodologies, both qualitative and quantitative, so students can be exposed to a variety of methodological tools. It was not very hard 10 years ago, but now I am struggling. Papers have become entirely formulaic, using the same statistical approach, even in cases where the research design didn’t support it. It has me wondering, has organizational research become meaningless because it is using statistical methodologies on data that it isn’t suited for.
Has Organizational Research Become Meaningless?
Ten years ago when I created the original reading list, I had no trouble finding papers that used simple and straight-forward statistics. Papers used simple correlations, multiple regression, and analysis of variance. The statistics used matched the research designs. If the design was a cross-sectional survey where everyone was surveyed one time, the analyses were correlations. If the study involved an intervention where efforts were made to change climate, mean levels were compared with an analysis of variance. These studies used straight-forward approaches, and it wasn’t so hard to find relevance to organizational practice.
Today every paper I found uses some form of complex statistical modeling that is designed to test causal models that explain how a process unfolds over time. For example, we might want to know how leadership practices lead to climate, and how climate leads to employee behavior. This would made sense if we were able to observe how changing leadership leads to climate that leads to behavior over time. If we could see that leadership changes first, then climate changes second, we would have some confidence in our model. In such cases complex modeling could be helpful in analyzing the data.
Unfortunately, statistical modeling is applied to cases where all we can show is that leadership, climate, and behavior are correlated. The common saying that correlation does not imply causation applies. If you cannot establish the order in which things occur, and you cannot establish that connections are causal, the modeling is not helpful. That’s why I ask has organizational research become meaningless. The analyses are just a device to get papers publishable without really providing new insights.
The Logic of Modeling
The logic underlying statistical modeling is based on deductive inference. A deductive inference is a logical conclusion based on premises or assumption. This is a simple if A then B logic. If a given model is correct, we would expect that our statistical modeling would have a particular pattern of relationship. But if A then B doesn’t work backwards. If A then B doesn’t mean if B then A. Finding expected results of a statistical model does not mean that the model is correct. It winds up that when we conduct a model test, there are many alternative models that would produce the same pattern of results. For example, a model that says leadership leads to climate leads to behavior will produce the same pattern if reversed as behavior leads to climate leads to leadership. The correlation pattern cannot tell us which is correct. To know that would require different kinds of studies.
Analysis Must Match the Design
I sometimes wonder if the invention of the computer has been good or bad for organizational science. It has made it easy to perform complex statistical modeling and other complex statistical analyses. The ease with which we can conduct modeling analyses has led to an over-reliance on complex statistics, and a neglect of field experiments that would actual help us understand what leads to what in organizations. It has fueled the misuse of statistical methods by applying methods that are not justified by the research design. This has led to a growing divide between academics and practitioners who also ask has organizational research become meaningless.
Photo by Pavel Danilyuk from Pexels
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE
Paul, I love this piece. This is something I felt when I was a full time academic professor. The inaccessibility of the work, even to my own graduate students, was frustrating. Even if we solve distribution of science with more open access, what does it matter if 99% – perhaps even in the academic community – can’t interpret or understand the methods? Unfortunately a single researcher alone will struggle to operate differently because the systemic incentives in publishing reinforces the alienation.
Maybe Necessary Condition Analysis -NCA can give the answer: “If not A then not B”. No confounding issues, no complex statistical modeling, no complex analysis, no unrealistic assumptions. Just a different perspective on causality and data analysis. (https://www.sciencedirect.com/science/article/pii/S014829632400122X)