Qualitative anecdote studies in social science have low validity and reliability
The replication crisis decimated especially soft subjects like social psychology and left it as a hollow husk of its former glory. In cognitive psychology, 50% of the retested studies could be replicated with the same positive results. But just 25% of social psychology findings could be replicated. That’s a scientific field? At that point just having neutral experts guessing on results may be more consistent.
These results may come as a shock for people who see social science as a hopeful and glorious field that is always on the cusp of changing our world into a utopia. But unfortunately, much of it are glorified pop-sci books made to sell instead of tell.
Pop-sci morality priests know that if you control the study participants and the report you control the message. In most cases they know that qualitative studies focused on emotional points are hard to retest.
Anecdotes are seldom completely false — but often mislead
Most of the anecdotal studies try to explain a real effect, but vastly overstate it in a way that unfortunately make the examples misleading. Oftentimes when you spot a huge effect it’s caused by several factors or a manipulated study. So explaining it from only one point of view is oversimplified thinking.
According to legend Galileo Galilei’s put the sun in the center of our solar system and was therefore put under house arrest by the church. This story supposedly illustrates how modern science is an anti-thesis to religion. But there are several factors to the story that make it a less than stellar example of this point. Firstly, when he was under house arrest he had the time to do his very best work. So the Pope didn’t exactly shut down a genius, but rather ignited him. Secondly, Galilei made the Pope look like a fool in his book and then when asked to take it back doubled down. It would be like pointing out an important uncovered thing about the company you work for while calling your boss an idiot for ignoring it.
I do personally love the Galileo anecdote, but today we have better examples of morality priests destroying good scientists. Basic religion no longer controls liberal sciences and the popes of today are leaders with cosmic egalitarian moral values that kick out social scientists for studying the morally wrong things.
The morality priests have a clear model they follow in social science: Have an all-encompassing hypothesis that is interesting and feels morally true. Control and manipulate the qualitative test-design from start to finish, have few top-down controlled participants to create a personal story, focus on the positive results only and tell a clear story from start to finish. Afterwards: “sell, sell, sell”. This is how prosecutors win in trials. They create a story while leaving the opposition without a story. Jury members seem to to pick the story over the non-story even without evidence for it.
Psychology anecdote examples
All these examples featured in the psychology textbooks we read at my university.
In reality you need minimum 30 random people to test an effect. And you need 2 groups to see if the effect is caused by your factor or maybe just a random factor like time.
1.) Zimbardo’s Stanford prison experiment
Prison guards easily become evil and this experiment/anecdote proves it?
No, it’s a one-time experiment which means it’s not worth much. Right, remember the replication crisis? And it’s an experiment the lead scientist took part in and then even could describe and sell as he wanted. Qualitative experiments seem to be more misses than hits as they are in many ways glorified anecdotes. It is true that people adapt to the environment and the conditions at hand. But we are not that moldable. This study was created to create an anecdote by having people follow basic acting orders.
A woman got murdered and 38 witnesses didn’t call the cops because of apathy?
Yes, according to the news article misconstruing facts. Fake news. In reality, the number of people who saw and understood the attack is vastly overstated. Some saw a woman get up and walk away while the attacker disappeared. Her gay friend saw her getting attacked the second time, but got too scared to call for help. So we have another effect influencing the anecdotal point, fear.
Varied lighting conditions make workers more productive? Oh, wait, actually any change makes workers more productive here? Oh, maybe observing workers makes them more productive?
This experiment is famous because it’s a great example of a bad study design where scientists influenced the results by selecting good leaders who could then select good workers. It’s an old example of “we found a positive effect caused by…” When you just test 1 specifically selected group you cannot know what causes the effect. This is why we need 2 random groups, then change a factor for one group and compare the groups. It’s easy to state that your snake-oil pills cure depression when time itself is the greatest healing factor.
Rosenhan had a hypothesis way before he had a study: insane asylums make you worse, not better.
It’s easy to “prove” this hypothesis by only having 9 test subjects after removing the test subjects that got better in the mental asylums. And no one could interview the subjects or look into his data because he never revealed who these participants were. Did he misconstrue results? Seems very likely. Did he make up some of the test subjects? Maybe. But either way it’s a weak study design set up to create an anecdote instead of reliable results. It’s true that mental disorders are not clear and easily measurable things. But most people don’t go into psychiatry to harm unfortunate people. Most people try to help the best way we can and social contact does in fact help a bit.
5.) Robbers Cave Experiment by Muzafer Sherif
2 groups of boys playing games against each other created a Lord of the Flies scenario.
This is another arbitrarily created anecdote. The Sherifs created conditions that made the 2 groups bicker while constantly influencing the conditions. Afterward, they could describe the events in their own way.
It is true that primitive male groups tend to become very ingroup focused and start fighting over resources. There are chimpanzee studies and anthropology tribe studies that found this effect. Male groups tend to kill males in other groups and kidnap the females. The naturally observed examples are great anecdotes as they describe instead of prescribe.
John B. Watson tried to demonstrate that humans are endlessly malleable by doing a simple study with 1 single baby.
At that time Americans felt they had created a glorious nation and could drag the country away from conservative European ideas towards a matriarchal utopia state. These ideas gave a lot of blank slate voices a lot of power. But their utopia ideas like the alcohol prohibition ended up making things much worse.
7.) The Milgram “Shock Experiments”
Milgram tried to show that regular civil people follow orders just like the Nazi soldiers did.
Unfortunately, a small study doesn’t really prove any wide-ranging theory about the world. Plenty of people follow orders in an experiment because they trust the scientists in such a degree that they think they will improve the world via rationality by creating new data. And we may as well focus on the participants who refused to continue past a point. These people are everywhere.
