Of all the things I study, I am most passionate about research methods—doing research on how we do research. I have written about how we design our studies, how we measure our variables, and how to use statistics to analyze data. Last week I had the privilege of giving a virtual talk (the video is posted on You Tube) sponsored by the University of Melbourne and Society for Quantitative Methods SQM. The talk was about how research methods determine scientific inference.
What Is Scientific Inference?
Scientific inference is about reaching conclusions based on a particular scientific study. A scientific study is one in which we use structured procedures to collect information (data), designed to minimize subjectivity. How we collect information or data, how we measure the variables we study, and how we analyze them determines the conclusions, or inferences, that we can reach. Although it might seem simple and straight-forward, in practice scientific inference is complicated, as there can be multiple interpretations and possibilities. For example, if I survey a sample of employees, asking about how stressful their jobs are and their level of burnout, I would almost certainly find that people who report high levels of stress will report high levels of burnout. But is it because experiencing stress leads to burnout, burnout leads people to appraise their jobs as stressful, or something else? My talk is about the different possibilities and how we figure out which is most likely correct.
Three Possible Connections
In our burnout example, it is pretty easy to show with a survey that burnout is associated with people’s appraisals of stressful job conditions. What is not so easy to determine is why we might find that connection. There are three possibilities:
- There is a direct connection between the two—either stressful conditions drive burnout or the reverse.
- There are shared biases that affect people’s reports of stressful conditions and burnout. Referred to as method variance, these biases make some people prone to report negatively—their jobs are stressful and they experience burnout—whereas others are prone to deny negative things. Such people’s jobs and burnout might in fact be the same, but it is the shared bias or common method variance that determines how they respond to the survey.
- The relationship between stressful job conditions and burnout is caused, not by their direct connection, but by some shared external factor. Spurious relationships are those in which the relationship between two variables is due to something else. In this case people who report higher stress have jobs that are more stressful, and those who report more burnout really are burned out. However, it is something else determining their connection, such as a personality that leads people to experience burnout and gravitate to jobs that are stressful.
Research Methods Determine Scientific Inference
What makes science challenging is that there can be many reasons that you find two (or more) variables are related. Testing possibilities and ruling them in or out requires a variety of research strategies. Some possibilities include:
- Direct connection: Experiments are the best means of establishing direct connections. Create an intervention to reduce the stressfulness of jobs, and see it is affects burnout over time.
- Method variance: Using different methods to assess your variables can eliminate shared biases. Measure burnout from an employee survey but measure the stressfulness of the job by having researchers observe people at work.
- Spuriousness: One way to test for possible spuriousness is to identify what is most likely to be affecting both variables and use statistical methods to examine its effect. For example, we might see if people with certain personalities are in stressful jobs and report burnout.
Science is like a criminal court case. We collect evidence that leads us to the most likely explanation, knowing that our inferences might not be correct. It is the process of discovering a phenomenon and then conducting additional research to explore possible explanations that makes it challenging and fun.
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE