Scientists often repeat studies to determine if they can be replicated, that is, if the same results can be found across studies. This is important in fields that rely on statistical analysis because results are based on probabilities rather than certainties. If I conduct a drug trial, for example, to see if a new medication is effective in treating depression, I need to be sure that positive results in one trial isn’t a fluke. Perhaps I made a mistake that distorted the results and gave an erroneous result. Sometimes there are enough replications of a particular issue that it makes sense to combine results statistically using the method of meta-analysis. But what is a meta-analysis?
Replication in Science
In many areas of science, important questions will be studied repeatedly as scientists want to be sure they can produce the same results as other scientists. They want to be sure that there wasn’t an error or even that someone was less than honest in their reporting. For example, in my field of organizational behavior, employee surveys are popular, and it is not uncommon for there to be dozens of studies of the same question. For example, I study the connection between stressful job conditions, like interpersonal conflict among employees, and job satisfaction (whether people like their jobs).
What Is a Meta-Analysis?
A meta-analysis is simply a quantitative analysis of the results across studies. For example, in 2013 we conducted a simple meta-analysis of how often nurses are assaulted at work. We found more than 100 studies that reported the percentage of nurses in a sample who experienced this at work. We computed the mean percentage across studies, showing that about a third of nurses have been assaulted.
There are many ways to conduct a meta-analysis. The main goal is to determine the size of the effect across studies. There are different types of effect sizes, depending on the type of study.
- Group Differences. This occurs with intervention studies like drug trials where we compare those who receive the treatment (experimental drug) with those who do not (control group). Each study will report the difference in means between the two groups on the outcome of interest (e.g., level of depressive symptoms) to see if the treatment is effective compared to the no treatment control. Because not every study uses an identical measure for the outcome, we must convert our outcome measure to a standard scale. Typically, we use a d statistic that expresses the difference in standard deviation units.
- Variable Relationships. This occurs with survey studies in which people are asked to indicate their levels on several variables (e.g., stressful job conditions and job satisfaction) by making ratings along a pre-determined scale. For example, people might rate each item along a 6-choice scale ranging from strongly disagree to strongly agree. By quantifying each variable along a continuum from low to high, it is possible to compute correlation coefficients that indicate the extent to which they are related. In other words, do people who report high levels of interpersonal conflict report low levels of job satisfaction? The meta-analysis involves computing the mean correlation among studies. For example, Nathan Bowling and Terry Beehr found in their meta-analysis that the mean correlation between conflict and job satisfaction was -.29 across 20 studies.
How To Conduct a Meta-Analysis
Conducting a meta-analysis requires several steps.
- Literature Review. A literature review is conducted with a data base that is appropriate for the topic. Some are freely available, such as Google Scholar. Others require a paid subscription or access to a library with a subscription. To conduct the review, enter relevant keywords (e.g., “interpersonal conflict” AND “job satisfaction”) and the data base will provide a list of possibilities. You scan them to find sources that provide results of relevant studies.
- Inclusion Rules. Decide what sorts of studies you want to include. For me, they usually must be studies of employees. Sometimes they must be in a specific industry or job. Sometimes the measures used must meet certain specifications, for example, if I am studying employee job performance, the measure must be from supervisors and not the employees themselves.
- Coding of Articles. Once you identify relevant research reports, you must read through them to verify that they measure the variables of interest in a way that makes sense. If so, you extract relevant statistics.
- Statistical Analysis. The goal of the meta-analysis is to determine the mean effect size (d or correlation). Typically researchers go farther and conduct a series of computations that provides a complex consideration of the results. For example, one might want to know if the effects of an experimental drug are the same for men and women, young and old, or people with versus without some underlying health conditions.
Meta-analysis has become a major tool that is frequently used in fields that rely on statistical methods. It has its limitations, however, so results should always be taken with a healthy dose of skepticism, as is true of all scientific work. The adage “garbage in, garbage out” applies as a meta-analysis is only as good as the studies available to the researcher. Nevertheless, a well-conducted meta-analysis can provide important insights about many of the phenomena we study.
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE
Very clearly explained.
Thanks Irvin