Three Forms of Analytics Evidence

We have all heard the call to use analytics to support decision-making. That means practicing evidence-based management to base decisions on information rather than gut feelings or intuition. Doing so can increase the chances that our decisions lead to desirable outcomes. But evidence is not all created equal. There are three forms of analytics evidence that we can use.

Three Forms of Analytics Evidence

Helena Kraemer and her colleagues distinguished three forms of evidence that vary in the extent to which they are helpful in supporting decisions. This evidence comes from collecting data using a variety of research design strategies. Suppose you are interested in raising sales volume for your organization. You are looking for an effective strategy to do so, and you want to base that strategy on sound evidence. What are the three forms that you might look at?

1. Association. This is something that relates to the outcome in question. You might use analytics to search for factors that are associated with sales to provide a hint of what sorts of actions might be effective. For example, your company conducts an annual employee survey and you find that satisfied employees have higher sales.
2. Prediction. This is something that can predict your outcome over time. For example, you find that today’s sales can be predicted by last year’s employee survey.
3. Intervention. An intervention means that you manipulate something to see if it leads to the desired outcome. You institute a program designed to raise employee job satisfaction to see if it will affect subsequent sales.

Limitations of the Three Forms

Organizations and people are so complex, that no action is certain. What worked yesterday won’t necessarily work tomorrow because things change. However, we use our evidence to make the best decisions possible with the available information at hand to increase the odds of a good outcome. These three forms of evidence have limitations.

1. Association. Just knowing that something relates to our outcome does not tell us why. Further, we generally do not know what to do, only where to focus our attention. Knowing that satisfied employees have more sales does not mean that if we raise satisfaction sales will increase. This is because it is possible that sales drive satisfaction rather than the opposite. Individuals who sell more get rewarded, and those rewards lead to satisfaction.
2. Prediction. Something that can predict over time is compelling, but that is not enough. Prediction does not mean that changing that factor will lead to our outcome. This is because we do not know if the predictor drives sales, or if it is something associated with it. For example, suppose sales skill is the driver of both sales and satisfaction. People who are skilled enjoy the job because they are well suited for it, and they sell more because they have the talent. Thus satisfaction predicts sales because it is associated with sales skill, which is the real driver. Doing something to raise satisfaction would not affect sales unless it also raises sales skill.
3. Intervention. An intervention means that the factor in question was manipulated to see if it would drive the outcome. If we design an intervention to raise job satisfaction, it will produce better sales. This is a direct test of the very actions we would take, and might be considered a pilot test. The main limitation is that we are typically not sure if the target of the intervention (job satisfaction) is really what drove the outcome, or if it was something else. Often interventions have more than one effect, so there are a variety of possible reasons it worked. For example, our satisfaction intervention might have enhanced sales skill, or encouraged more teamwork among sales reps. From a practical standpoint, demonstrating that an intervention has desirable effects is highly useful, even if there is a possibility that we do not really understand the reasons. Additional analytics work could be conducted to figure that out.

Making the Most of Evidence

Evidence-based decision making means collecting the best information that you can to support a decision. That information might come from inside the company, such as company records and evaluation studies that you conduct. Information can also come from external sources, such as the academic research literature. Keep in mind that those published studies in most cases are limited to association and prediction forms, which is one reason that practitioners often complain that they contain little that is actionable. Such studies offer hints about where to start looking for solutions, and do not provide the solutions themselves. This is often a good place to start so you do not waste time looking in the wrong place for a solution. But ultimately if you want to know if your intervention will work, you need to collect intervention evidence.

A big part of effective decision-making is understanding the limitations of the information at your disposal so you can place it in perspective. Knowing the strengths and limitations of the three forms of analytics evidence is a good place to start.

Photo by Miguel Á. Padriñán from Pexels

To Subscribe Enter Your Email and Click SUBSCRIBE