psychology experiment results, are you kidding?

No not mine. I like the ATS community and all but not like that.

I guess a bunch of psychologist got together and tried to reproduce the results of about 100 claims in various research journals and the results weren't good.

Of 100 studies published in top-ranking journals in 2008, 75% of social psychology experiments and half of cognitive studies failed the replication test.

All of the experiments the scientists repeated appeared in top ranking journals in 2008 and fell into two broad categories, namely cognitive and social psychology. Cognitive psychology is concerned with basic operations of the mind, and studies tend to look at areas such as perception, attention and memory. Social psychology looks at more social issues, such as self esteem, identity, prejudice and how people interact.

This is pretty concerning to me for a couple reasons. First the major failure came with social psychology and we all know that social anything seems to take front and center. The other concerning thing to me is that these were published in 2008. How many other psychologists have been working with what may be completely inaccurate information. Trying to drive policy based on wrong information? Trying to help patients based on wrong information?

They do offer some explanations as to why the results were so poor, methods, etc. But this is pretty concerning. What type of crazyness has been pushed into acceptance since 2008 from the mental health industry based on false data?

www.theguardian.com...

a reply to: Reallyfolks

I saw that article earlier today.

It certainly underscores the importance of reproducibility in scientific experimentation.

And it does make you wonder. I recall all the recent news stories about how studies determined this or that about people of various ideologies or with faith or without it for example.

As much as I did like what some of those stories had to say ... the phrase "junk science" did come to mind when reading these stories.

originally posted by: Reallyfolks
No not mine. I like the ATS community and all but not like that.

I guess a bunch of psychologist got together and tried to reproduce the results of about 100 claims in various research journals and the results weren't good.

Of 100 studies published in top-ranking journals in 2008, 75% of social psychology experiments and half of cognitive studies failed the replication test.

All of the experiments the scientists repeated appeared in top ranking journals in 2008 and fell into two broad categories, namely cognitive and social psychology. Cognitive psychology is concerned with basic operations of the mind, and studies tend to look at areas such as perception, attention and memory. Social psychology looks at more social issues, such as self esteem, identity, prejudice and how people interact.

p://www.theguardian.com/science/2015/aug/27/study-delivers-bleak-verdict-on-validity-of-psychology-experiment-results

There are more ways to look at it, especially in psychology. Who knows, it could be the peers who were not up to scratch, like this example, going back many years but still relevant.

"Ninety years ago, Stanford psychologist Lewis Terman began an ambitious search for the brightest kids in California, administering IQ tests to several thousand of children across the state. Those scoring above an IQ of 135 (approximately the top 1 percent of scores) were tracked for further study. There were two young boys, Luis Alvarez and William Shockley, who were among the many who took Terman’s tests but missed the cutoff score. Despite their exclusion from a study of young “geniuses,” both went on to study physics, earn PhDs, and win the Nobel prize.

How could these two minds, both with great potential for scientific innovation, slip under the radar of IQ tests? One explanation is that many items on Terman’s Stanford-Binet IQ test, as with many modern assessments, fail to tap into a cognitive ability known as spatial ability. Recent research on cognitive abilities is reinforcing what some psychologists suggested decades ago: spatial ability, also known as spatial visualization, plays a critical role in engineering and scientific disciplines. Yet more verbally-loaded IQ tests, as well as many popular standardized tests used today, do not adequately measure this trait, especially in those who are most gifted with it."

Shoehorning comes to mind here.

a reply to: ketsuko

I kind of thought the same thing. Wonder if any of the studies showing people in this political/social/religious group are xyz failed the reproducibility tests.

a reply to: Reallyfolks

Reefer madness anyone?

Couldn't resist... thanks for posting!

This comes as no surprise to me.
I taught research methods in social science
and edited several research method college textbooks.

I can tell you that most researchers do all they can
to make sure that whatever they want to prove
is "proved", and they will use all sorts of tricks
to make their theory "true".

The statistics can be easily manipulated,
we even go over in class how to make the stats
prove what you want it to prove in class. I did that
to show how bad the majority of social science
research really is.

As a social science textbook editor I saw way
too many peer reviews done by friends of the
authors. I know for certain that "peer review" is
normally code for you rubber stamp my research
and I'll rubber stamp yours.

This doesn't only go on in Social Science, but
unfortunately also in "hard" science as well,
which unlike Psychology, can lead to tragic
results because the vast majority of Americans
believe as absolutely and unequivocally true
any "studies" or "research" that is published.

Most people on ATS scream to high heaven
when I point out that one should never
ever believe one study on any subject.
But it is true. Without a study being
replicated multiple times, you can not
at all be in any way sure the results are
accurate.

a reply to: grandmakdw

Thank you for your insight and honesty.

a reply to: Reallyfolks

Psychology isn't a science.

It isn't even a priori.

That doesn't mean it can't be useful but, scientific repeatability is impossible.

originally posted by: greencmp
a reply to: Reallyfolks

Psychology isn't a science.

It isn't even a priori.

That doesn't mean it can't be useful but, scientific repeatability is impossible.

I know many who would agree, but no matter what you think of it may be driving policies, pushed into schools to change teaching methods, pushed into business environments , etc and may still affect you. I would love to see the list of those failing and see if any were then used in any manner since 2008. It's not what you or I believe about psychology , it's how it still may affect us.

No surprise here. They could test me everyday and get different results each time based on how much I carwd that day or how good my diet was or how good I slept.

They also couldn't touch my creative ability or anything really.

a reply to: Reallyfolks

I agree, I was just making the comment that someone attempting to repeat those experiments is wasting their time.

You are right that pseudoscience is being applied to public policy and creating bad law which is a threat to the respect for and, ultimately, the intellectual defensibility of, just law.

Science as a concept is a brilliant thing, but once you get people involved it's much less so. If only Science would do itself and just leave us out altogether.

Without looking at each experiment and the replication or retest I can't comment on the reliability of the experiments either way. The original research may be dodgy or the retest could be dodgy. The original research will say whether there could possibly be an error and why. Does the retest offer any reason why the original might be wrong? It is very difficult to replicate an experiment in every detail and if you want it to look bad it is extremely easy to do so. I bet I could pick holes in the replicated experiment. Things that could go wrong in any research include things like experimenter bias and sampling bias. Often the research participants are other psychology students who may perform differently to a general population sample for example. I would be just as wary of any so-called replication experiments as the so-called dodgy one as researchers are always trying to point out that the research of others is suspect. I have taken part in some perception research and yes it was replicable and got similar results. Some wasn't but that might have been our fault.
But all science is like that. Until recently for example scientists were saying that white skin in humans was only about 5000 years old. Now they have found it is at least 20000 years old. We can only keep learning.

a reply to: grumpy64

In the article, at least one of the scientists doing the retest was retesting a study he had helped to do in the first place and it failed.

The problem is that people in high places and even the average Joe on the street has the idea that if science says it and scientists did it, it must be definitive. They us it to justify the policy positions they take or endorse, but here we see that from a social science perspective, those studies may not even be reproducible in the broad sense in further cases.

So the idea that science and technology are able to centrally plan and control our every move is still very much in doubt because at least some aspects of science still have a long way to go to unravel the mysteries of the human animal.

Half the scientific studies out there have such tight parameters that they can't be applied appropriately to real life. Now if they hired people to go through a pile of research and try to decipher what is really happening, they could do better. But interpretation is based on a person's belief and expectations from what knowledge they possess so that might be difficult to get the proper interpretation that applies to a variety of situations.

Maybe a bunch of seniors with no knowledge of the subject who have experienced a lot of things in life should be used along with someone who is both able to understand the results and is open minded to evaluate the practicability of the information.

a reply to: ketsuko
The main problem seems to be in the area of social psychology, how people are influenced in the presence of others. This is always going to be a difficult area as it contains a healthy component of sociology. My wife is doing honours in sociology and it is quite different to psychology in the way research is approached. Psychology appears to be closer to other sciences in the way it performs research I believe. I had this discussion the other night. All scientists should be sceptics and be critical of their own work and base their research on the principles of determinism and empiricism. Psychology is no exception.

I agree with your last point completely. Science still has a long way to go regarding the complexities of the human mind. But people in high places often cherry-pick the research that agrees with what they want to do. Often the results are taken out of context and exaggerated. I can remember scientists claiming that sea levels would be 10' higher by 2010.
I thought that was rubbish at the time.

But remember that principles of psychology are used all the time by TPTB to influence who we vote for, what we buy, what we think and how we behave. I always wonder if the researchers are unduly influenced by whoever is paying the grant for the research.

a reply to: Reallyfolks

I liked this quote:
[exThe only finding that will replicate 100% of the time is likely to be trite, boring and probably already known: yes, dead people can never be taught to read.
Source

And psychology has been associated with a lot of really awful studies. If you want to read some hilarious ones, go and read the ones that were used to support the 'ban the lads mags' campaign. That said, a huge issue is that sometimes, particularly in psychology, you'll just get anomalous results from small sample groups. A lot of the more complex studies use less than fifty people. You run that study 8 times, and if the results are on a scale (1 to 5 instead of yes and no) for example, you're going to get some statistically variant results.

Psychology is one of those fields that could severely benefit from publishing failed studies. Currently they only attempt to falsify something once its been 'found'. I'm looking at you psychological priming.

a reply to: Reallyfolks

I'm sure the big thing to come out of this is , if they ever find a completely well adjusted sane person , they can cure them .

a reply to: Pinke

And I have seen many failed studies that are published. They are not a lot of use though- unless you are trying to disprove someone's theories in favour of your own. They simply say that the null hypothesis is true and the findings were not replicated. But perhaps that is not interesting enough to publish. And yes sample size is always a problem. You have to have a large enough sample to make the research valid. I wonder who paid for the retesting. It would have cost a small fortune and how boring would that be? And who did the retesting? Starving uni students no doubt!

a reply to: Reallyfolks

Any good scientist knows to conduct, document, experiments in a manner that can be reproduced.