I recently read this article by Nancy Doyle who writes about neurodiversity for Forbes. She notes how neurodiverse people can have difficulties with some employment selection tests. She talks about an issue with a situational judgment test in which people are given a scenario and are asked to pick the best action to take. Where this is a relatively straight-forward task for a neurotypical person, it is not always an easy task for someone who is neurodiverse. He or she might question the premise of a question or find none of the choices to be adequate. This idea that items designed to assess a given construct might not work well in all applicant populations has me wondering about how far this phenomenon might extend beyond selection tests and people who are neurodiverse. The article has we pondering the question, can personality destroy test validity?
Manifest Versus Latent Content
Early in my career I conducted an informal focus group about the Marlowe-Crowne social desirability scale with two friends. We were talking about some research I was doing, and they found it intriguing and wanted to take the scale. As luck would have it, they scored on opposite ends, and when I gave them their scores it sparked a lively discussion. The husband, Dave, scored very low, whereas his wife Ellen scored at the top of the scale. Dave took each item literally, and he argued that if an item asked if you always did something (e.g., thoroughly investigated each candidate before voting), that meant if even one time you failed to do the investigation, you would have to mark false. Ellen had a different view. From her perspective, the word “always” in an item did not mean literally every time. It meant that this was your typical behavior—what you did most times. For her “always” meant “usually”. Their differences reminded me of the distinction between manifest and latent content. Dave was responding to the literal or manifest content of each item. Ellen was thinking about the underlying meaning or latent content. She was reading more into the item than was contained in the words. Ellen responded true that she always investigated candidates, but when I asked her if there was ever a time that she did not, she said yes there was. But she also argued that was not what the item asked.
Can Personality Destroy Test Validity?
Doyle makes a convincing argument that people who are neurodiverse might not always respond to items in the same way that most neurotypical individuals would. I wonder how far this phenomenon goes and whether it extends to measures that goes beyond situational judgement and other abilities we use for employee selection. Would it hold for measures of job attitudes such as job satisfaction? Would it hold for measures of employee engagement? What about perceptions organizational climate?
Even further, does this phenomenon of people interpreting items and using different response strategies extend beyond the neurodiversity realm? Research has shown how personality differences are related to a host of psychological measures. We assume that this reflects the relationship between the underlying personality characteristics and these other underlying constructs. However, can it also reflect that people with different personalities might interpret and respond to items differently? After all, Marlowe and Crowne who developed the most popularly used social desirability scale argued that it reflected the personality trait of need for approval. Individuals who were high on the need were likely to deny negative things about themselves.
This blog raises more questions than it answers, but Doyle’s point that individuals can differ in how they respond to test items is compelling. This idea resonates with me as I have often wondered with the scales my colleagues and I have developed what the people who take them are actually telling us. Do they see the items as we intended, or is there something more complex going on? Does personality act as a biasing factor that affects what test scores tell us? If so then we need to take a harder look at the role of personality in our research.
Photo by Skitterphoto from Pexels