More than a century ago, a Swiss patent clerk presented a theory that revolutionized physics. Einstein’s relativity theory continues to be challenged empirically as technological advances and the imagination of physicists apply new research methods. This application of method diversity is the basis of theory testing in all sciences. That point seems to have been overlooked in the development of the organizational sciences in industrial-organizational (I-O) psychology and management where the focus is more on developing new theory and invoking theories (theory salting) rather than inventing new ways of testing existing theories to verify their predictions.
The Changing Role of Theory in the Organizational Sciences
A few years ago I did a historical content analysis of the leading I-O journal, Journal of Applied Psychology. In 1971 there were only a handful of theories in the field, and many of the papers reported new methods for testing those theories. One paper was an attempt to test a theory using a new method that overcame limitations in the prior tests of that theory. Fast forward to the early 2000s and Suzy Fox and I submitted a paper to the same journal using the same rationale. At that time, all papers concerning counterproductive work behavior (misbehavior by employees at work), relied on anonymous self-report surveys where employees were asked about their own bad behavior. We applied a different method by having supervisors report on the behavior of a direct report. We were able to verify that results found in prior studies was not due to biases in the self-reports, often referred to as common method bias. Reviewers complained that our paper provided no new insights. Merely testing a theory with a different method was not seen as a contribution to science if it merely confirmed that theory.
Put Science Back in Organizational Science
It is past time that the organizational sciences went back to scientific basics. As I have written before, the field has become imbalanced by so much emphasis on developing new theories and theoretical ideas that the important work of testing existing theories has gone by the wayside. Testing old theories is not valued as it is in other scientific disciplines, so we are at a point where stating theories is seen as the only goal. We need to accept that before a theory can be taken seriously, it must be tested using methods that are different from those used to develop the theory. We must embrace that method diversity is the basis of theory testing.
Method Diversity Is the Basis of Theory Testing
Science is a three-step process of exploration, explanation, and confirmation. It begins with exploratory research to better understand a phenomenon. There can be theoretical elements in exploratory research, but the goal is not to test theories but to generate new knowledge and insights. Explanation is where theory is developed. A theorist reviews the existing research on a phenomenon and comes up with a theory to explain it. That theory is a tentative explanation as we have no idea if it is correct or not. It merely describes the current state of knowledge while offering little in the way of valid inference. Theory testing is needed to confirm that the predictions of the theory have merit, and they tell us that the theory is scientifically useful.
Testing a theory using the same methods that were used in the research it was based on is not helpful. Merely showing that studies derived from the theory using the same methods as in the past is only telling us that results can replicate. We need to apply different methods that show us if predictions of the theory can be supported. In the organizational sciences, most research is based on surveys, usually of the employees themselves. This is an important first step, but we need to go beyond self-reports to test theories that are based on self reports. Possibilities include:
- Other Sources of Data. Data from records or other sources that are not based on surveys can be compared to self-reports to confirm theoretical predictions. I suspect that we do not see more use of alternative sources because often they yield weaker results than surveys, making it more difficult to publish. Perhaps one reason for the weaker results is that many of our theories are incorrect.
- Other-Report Surveys. Getting data from people other than the target employees is a good way to incorporate a different method. Sometimes results with other-reports are similar to self-reports and sometimes they are not. The comparison is a good way to apply method diversity.
- Observational Studies. It is labor intensive, but sending trained observers to watch people work can be a good way to introduce a different method. I recall a study by Michael Frese in a factory that did just that–observers rated the working conditions of employees and compared it to their self-reports.
- Intervention Studies. Nothing screams method diversity more than an intervention study. The best way to test a theory based on survey research is to put it to a rigorous experimental test. The intervention study does that by manipulating the workplace to see if the effects confirm theoretical predictions.
The hyper-competitive world of publication in the organizational sciences where academics are pressured to publish in a limited set of journals that are on “the list” has inhibited creativity and innovation in how methods are applied to test theory. With top journals demanding theory for theory’s sake, it is difficult for individuals to break the mold. If the field is to advance as a science and if we are serious about bridging the academic-practice divide, it will be necessary to incorporate more method diversity into the organizational sciences.
Photo by JESHOOTS downloaded from Pexels
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE
Yea verily!