Questionnaires about feelings and perceptions are just that
I keep seeing people generalize and overrate polls/questionnaires. Someone may read some poll stats and then assume that these emotional answers explain something major and factual about the world. This is seldom the case as people are just too biased to give you the facts head-on. Qualitative studies show you how people think they feel about a subject, but they don’t explain much of anything. In this blog post I’ll try to explain why qualitative studies often mislead.
Experiments showing how biased we are
When was the last time you told the 100% objective truth when asked an emotional question? Think back, someone asked you why you failed an exam or didn’t get somewhere on time. What was your answer? Were you holding back or being 100% direct? What actually often happens in these cases is that your brain tries to come up with a self-serving answer to explain the event after it happened. And the explanation will fit into your worldview, self-perception, and often be an explanation that makes you feel good about yourself whether it’s true or not.
One of the thousands of factors that make us lie to ourselves is the actor–observer bias that proposes that we have a tendency to blame other people’s problems on laziness while we blame our own problems on external factors. The opposite is the case with success where we feel like we worked hard to get somewhere while we assume other successful people were just lucky in life. Adults are basically small spoiled lying kids who don’t know what they don’t know and largely don’t even care as they think they are objective. Our brain is not evolved to see facts. It’s evolved to solve problems in our natural social environment and sometimes that requires misleading ourselves or other people to gain resources.
This is all very hard to believe as we see ourselves as rational spirits. But I think the split-brain experiment clearly shows how biased we actually are, look at the image below. A split-brain person may be shown a chicken foot to the side of the brain that speaks and a snowy landscape to the side that doesn’t speak. This is done by showing an image to one eye only. Then the person is told to select or draw figures based on what he has seen. The language brain part will afterward be asked to explain why the person picked the 2 images. The language brain part is now in control and can explain why it picked its own image, the chicken, but it won’t know why the other hand picked that weird random image. That’s no problem at all as it will come up with a story on the spot. As you see in the image the story after the fact and the fact itself are 2 vastly different things. The story is just something we tell ourselves to rationalize our life even if we don’t understand it. This rationalization is always there and you cannot turn it off.
People lie and they lie systematically. You lie to yourself more than to anyone else. Often the same event may even be perceived and described in vastly different ways and it’s not too difficult to even implant false memories in people as we see in this study:
About one-third of the people who were exposed to a fake print advertisement that described a visit to Disneyland and how they met and shook hands with Bugs Bunny later said they remembered or knew the event happened to them.
This is why making up questions and translating them for polls is so complicated. A small word may completely change the perception of the whole question. This is because we feel for the right answer. We don’t just logically calculate things about past experiences.
Below is yet another study illustrating how people can systematically be manipulated to give specifically biased answers about exactly the same event. If researchers search for a specific effect in a qualitative study it’s not too hard to force the right answer out of people.
A study showing differences in perception
There are also many specific group differences in biased thinking too. Let’s look into a famous perception study about biological differnces.
In this study men and women saw videos of: an unacquainted man and female discussing things, a female student talking to a male professor, and a male store manager talking to a female cashier. In all 3 studies, the male viewers had a tendency to say that the female in the video was flirting. Women largely perceived it as friendliness. The study comes to this conclusion:
These results support the idea that some of the less severe forms of sexual harassment in business and academic settings may be better understood eventually through research and theory development that considers these sex differences in social perceptions.
The bias is called sexual overperception bias in men and sexual underperception bias in women. It’s the same bias, it just goes in the opposite direction for each gender. Both genders are born to see the world in a “wrong” way that helps them reproduce their genes in the most effective manner. Then imagine creating a poll about flirting between sexes without explaining these biological sex differences to readers. How would laymen read the poll numbers? Well, anyway that would fit into their biased worldview.
But while we know a lot about biological gender differences there is another area that is much harder to study, race perception. This topic is also very emotional so it’s bound to produce biases results in questionnaires which is exactly what I want to look into. Try to keep all my study examples in mind as you read about these poll studies.
Discrimination polls
What do they tell us about society overall?
Let’s look into the sort of questionnaires one could use to study discrimination. I picked Pew Research Center as they make easy-to-share images. I just found a Pew site with a collection of polls on race discrimination and used some of them here. They may feel a bit random but I’ll try to explain why I picked them.
Blacks say they have faced discrimination
The first poll asked black Americans how often they experience discrimination and many claimed it was quite often. What can one conclude based on these results?
- One person may read this chart and claim that blacks clearly are discriminated against in a systematic way.
- Someone else may read the chart and claim that blacks are just overly eager to complain.
Neither claim is supported by the data in this chart alone. We don’t know if the answers explain anything real about the world. It may just be perceptions of discrimination or pure lies. We just don’t know what these perceptions are caused by. Something cultural or something inborn? To what degree? And of course, we can’t even know if this is a bigger perceived effect than in other races. So let’s answer this simple question first.
Perceived discrimination is a subjective measure
To compare thinking among races we need to look into a poll that actually does try to investigate perceived discrimination amongst different races.
Here we do see differences. For example, Asians are most likely to claim they have been subjected to slurs or jokes. But yet again we don’t really know what this means or what effect it has overall on anything. In this poll we yet again cannot see if these things are based on reality or are just biased mental perceptions of experiences. What is the actual level of discrimination? Do some races exaggerate their experience or overlook things? Which race does what? The poll-takers are not answering “this is what the actual number is”, but rather “this is what I feel the number is”. We cannot conclude that these claims are close to being factual numbers of something objectively measured. Or brain does not measure emotional experiences objectively so these numbers are bound to be wrong somehow. I’ll try to explain how I think some of the race perception biased may appear in a questionnaire even if there is no actual discrimination differences.
Different cultures and genes cause different perceptions
Different cultures will have different ways to describe similar cultural events. “Sup white man” may seem like a friendly gesture to a white man. While the similar “Sup black man” may have very negative connotations to a black man. And the intentions may even be the opposite of how they are perceived. The intention may be to make fun of a white man and be polite to a black man. This sort of communication is not easily studied or described by asking people about their feelings about their experiences. Intentions behind something offensive may be well-meaning, but this is completely overlooked in most questionnaires. Not all discrimination is purposeful or even harmful and this yet again is not really something we often look into.
Some left-leaning groups may also use a peculiar definition of racism where the groups in power are said to be unable to be subjected to racism. In this case a white person describing to that definition may experience greater discrimination than a black person, but just refuse to describe it as such when asked about it. So a white progressive may be called “Stupid cracker” and still not see it as racist discrimination, but define it as just an emotional outburst. This factor alone could in principle cause the black vs. white numbers in the poll to differ.
In the same way creating an award show for only white people may be seen as racist while creating an award show for only black people may be seen as anti-racism. So blacks may see a group of only whites as discrimination while whites may see a group of only blacks as just normal friendship. This would make it hard to fairly compare discrimination perception between races as people just have vastly different perceptions of fairness depending on which racial group suffers the harm. You may have seen examples of that online where a comment is perceived as overly sexist/racist by one group, but by another group isn't considered offensive in any way.
A group may also just gain resources from lying about being discriminated against. Affirmative action gestures and laws may make it easier for you to get into college and get a public job if you can claim some historical suffering. So blaming problems in your life on external factors may gain you extra help. Or course, as I mentioned before, just blaming external factors for your personal problems is in itself something we automatically do to ease our mental suffering. So either way it would be a thing we lie about in some degree.
Then there is the whole memory thing. Our memory is not created to remember specific details and exact numbers. It’s actually pretty unspecific and we restore memories every time we think about them. So once you recall a memory in an emotional state that influences the memory long-term. To understand the bad and biased memory that goes into such a poll try answering this question: “How many times have you eaten junk food last year?” How exact will the answer be? The answer may depend as much on your perception of yourself as the actual junk food amount you ate. Or your memory may just not be detailed enough to remember exact numbers. Even an estimate may be very wrong. For you “a lot” may mean you eat fast food once a week. For others, it implies that they eat fast food every day. Different people may also either have inborn or learned perception differences. Some races can’t handle alcohol or milk and may even see a bit of that as a lot. So how can be compare unique experiences in specific races?
Above is a chart trying to fix some of the problems I mentioned as here different races are asked about the same race. Both black and white people feel that black people are discriminated against more. Of course, it’s still an estimate based on feelings. We still don’t really know what this implies or if the effects are inborn or cultural. So let’s dig into real numbers to see if we can find any real-life effect that correlates with these subjective answers.
A better method to investigate effects in society
What do factual measures tell us about race differences? While questionnaire polls study emotions and subjective thinking about subjective experiences, quantitative measures measure factual things. We know for a fact that there are gender differences in wages caused largely by how much women stay at home with their kids. We know for a fact that Asian men and women outearn all other races in USA because they have higher inborn IQ. Though other factors may play some minor role too. Unfortunately, even factual data can be made to support subjective worldviews. Here the title of the chart is a subjective point of view in that white men are for some reason used to explain the data even though they don’t have the highest wage. So the data itself is good here, but you still need to beware of what info they force onto the data. A biased scientist will find a way to convey her moral point of view onto the world somehow. And just one such biased title makes me doubt all their qualitative studies. This alone makes me doubt anything Pew may write from now on unless they fire the employee who made this title and apologize for the bias. But even though this alone makes me distrust Pew overall I still pretty much trust the underlying data itself as I have not seen any clear data faking from them. So let’s continue.
Let’s look into another random chart and compare the results here to the wage numbers. Here yet again the perceived discrimination differences make very little sense to me. People think being Asian doesn’t help quite as much as being white. But as we read before Asian men earn 117% compared to white men. It’s not clear why the group that is seen as the most privileged is far below the most economically privileged group. Most factors are yet again unexplored and unexplained unless you just build a just-so-story after having read the numbers which would make you a terrible scientist. But largely these perception differences may just be based on shared anecdotes and myths about races. What Richard Dawkins calls memes. It all may be one huge just-so-story about races. Xenophobia.
What do we want?
Okay, let’s go a step further and look into wants and needs and see if we can uncover some hidden motivations that may tell us more about the world. We know from a former chart that black people have the lowest wages on average. We also know that people, in general, feel that blacks are discriminated against. We might therefore logically assume that black people hate discrimination and just want us to stop talking about races. That’s at least the effect I would predict if discrimination was an actual issue for a given race.
That’s not the case here. Blacks want to focus more on race than any other race. Maybe this factor is causing discrimination perception? There is a need to want to focus on race that then makes some groups perceive more discrimination? Could be. Or it may the other way around? Or there may be no relationship between these factors or they could be caused by a third factor like a heritable trait. These kind of studies don’t really answer such causality questions. We don’t know if it is whine or reason. But it could be the case that some races take advantage of misperceptions and affirmative action. If you pay people to do something they will do more of that thing. It’s the number 1 layman rule in economics. Whole economic book are written about this rule and how it often destroys cities and countries when politicians overlook it. As Goodhart’s law proposes:
“When a measure becomes a target, it ceases to be a good measure.”
If we give out money for discrimination we are bound to see a higher perception of discrimination. Affirmation action creates more stories about racism as that is what we pay people to think about and discuss. And then, of course, there may be a negative learned helplessness effect from seeing yourself and your race as losers. But we cannot assume that this learned helplessness thinking, in this case, has any measurable effect on life.
How would we explain these differences?
What do people think causes race differences? Will the chart above finally reveal something about causality? Well, no. Zero questions here are about genetics so we don’t really know what people actually overall think as they are forced to pick between these specific answers. The big nature-nurture debate is completely ignored in all their questions. Since in principle nature may explain 100% of the differences both Democrats and Republicans may be 100% wrong.
We know that all traits in humans are heritable in some degree as this gigantic meta-analysis published in Nature illustrates:
We report a meta-analysis of twin correlations and reported
variance components for 17,804 traits from 2,748 publications
including 14,558,903 partly dependent twin pairs, virtually
all published twin studies of complex traits. Estimates of
heritability cluster strongly within functional domains,
and across all traits the reported heritability is 49%.
The study concludes:
Our results provide compelling evidence that all human traits are
heritable: not one trait had a weighted heritability estimate of zero.
At least in some degree, inborn mental differences cause discrimination perception. It may be just an inborn perception effect or inborn life trajectories later causing perceptions to differ. If people don’t know much about heritability they may just assume all the differences in racial outcomes are cultural and therefore just invent just-so-stories to explain away the effect.
If a collection of qualitative studies don’t even consider the effect of heritability I consider them useless at explaining anything as 49% of human traits are heritable and basically nothing is just environment. We need to explore what causes what factor. Some traits like personality are only 50% heritable while intelligence is 60–80% heritable. This is why a 40-minute culturally unbiased intelligence test predicts your work performance better than any other measured factor including job interviews that in themselves also partly measures intelligence. That’s because that concrete test measures something inborn and important instead of something fluid and everchanging. Job interviews are like questionnaires. It’s not surprising that when biased people are made to ask questions of biased people that the answers tell us very little about actual facts.
Conclusion
Qualitative studies and questionnaires are not really saying much about the world by themselves. They can’t investigate causality and often don’t even ask questions that could explore causality. They may be measuring inborn differences or cultural stories. If you don’t know the actual quantitative numbers behind the effects you won’t get much out of qualitative studies and they may just confuse you. Unless you study the heritability of these things it’s impossible to say what this all means. While only 49% of the average trait is explained by heritability much of the remaining effect is explained by random variability. It’s very seldom we actually find any concrete environmental effect that we can control in any way. So unless hundreds of studies show a very clear effect from a certain cultural variable I remain unconvinced of such a just-so-story effect. Discrimination surely has some effect on something somewhere, but unless we find it and conclusively illustrate it we cannot assume it’s causing anything noticeable. We need to pinpoint the causal effects to understand the real world and we can’t do that by asking random people emotional questions while depending on their memory to be perfect.
I like reading polls. I like charts. Just as I like listening to engaging personal anecdotes and fictional tales. But it’s all emotional tales about personal experiences not a mirror image of the real world. It’s after-rationalizations created to fit the world into our very small heads. And boy is that head small.
