It looks like you're using an Ad Blocker.

Please white-list or disable AboveTopSecret.com in your ad-blocking tool.

Thank you.

 

Some features of ATS will be disabled while you continue to use an ad-blocker.

 

Odds Are, It's Wrong - Science fails to face the shortcomings of statistics

page: 1
4

log in

join
share:

posted on Nov, 19 2010 @ 01:50 PM
link   

For better or for worse, science has long been married to mathematics. Generally it has been for the better. Especially since the days of Galileo and Newton, math has nurtured science. Rigorous mathematical methods have secured science’s fidelity to fact and conferred a timeless reliability to its findings.

During the past century, though, a mutant form of math has deflected science’s heart from the modes of calculation that had long served so faithfully. Science was seduced by statistics, the math rooted in the same principles that guarantee profits for Las Vegas casinos. Supposedly, the proper use of statistics makes relying on scientific results a safe bet. But in practice, widespread misuse of statistical methods makes science more like a crapshoot.


Source

Here is a new article that can act as a compliment to the thread on falsification of data. I'm not quite sure how to feel about this. My personal field of research is in psychology, so I use a lot of statistics and I thought I had a pretty good grasp on the concepts. While this article seems to be more focused on medical science, one must ask the question as to whether or not statistics are actually hindering the advancement of science, instead of helping it progress. I'm going to need to mull this over a bit before I give a more in depth reaction to the article, but I know there are more than a few people on here who will find this of interest.
edit on 19-11-2010 by Xcalibur254 because: (no reason given)




posted on Nov, 19 2010 @ 01:56 PM
link   
reply to post by Xcalibur254
 


I've always been a math geek more than anything...
I help my wife with her college algreba by figuring out the answer in my head to confirm what she takes forever doing by writing it out on paper...

but I have NEVER understood statistics like I do math.

statistics to me is just a way to lump things together and not pay any head to the unique qualities of life, which if we paid a bit more attention to, we would find that more and more things are unique in their own way

but then again, I never got very far with actually studying statistics, all the symbols hurt my head lol



posted on Nov, 19 2010 @ 10:03 PM
link   
reply to post by Xcalibur254
 

I have often thought this myself, and usually about medicine.

I think the eraly success of statistical analysis in correctly identifying the association between smoking and lung cancer, even when the mechanism by which smoking causes cancer was not known, sparked a slow-building explosion in the field of statistical medicine. Nowadays, it seems that the majority of popular news items on health, diet, etc., are based on statistical studies or metastudies (studies of studies). That's how we learn that drinking so many millilitres of red wine a day makes people less prone to heart attacks, that people who exercise live longer, and so on. For all we really know, none of it is true.

I tend not to believe what I read in the media about medical subjects for precisely this reason--most of what is presented as near-certain is based merely on some statistical correlation, without any causal mechanism established to back it up.

This over-reliance on statistics is not an indictment of science in general, by the way--merely of those who place too much confidence in dubious statistical analysis. A lot of these people are scientific careerists, desperate to make careers by publishing something.

Thanks for the article. More people should read it.



posted on Nov, 19 2010 @ 11:41 PM
link   
The misuse of statistics is prevalent throughout society, and not just in the sciences (look at how the misuse of statistical models has afflicted finance...despite the many failings complex models are still being used as a source of truth when they are nothing of the sort). However the fact that many scientists have little statistical training means that the statistics and statistical models used in science seem particularly weak and prone to misuse.

Medical science is an obvious one but climate science is another where the statistical training simply isnt there to properly understand the shortcomings of what they are doing. They put faith in models which are not suited to what they are doing. The simple fact that the ability to hindcast does not demonstrate predictive power doesnt even seem well understood by most!

Physics has been afflicted with a slight variation of this issue, where the mathematical abstractions have become so complex that they have veered physicists away from reality and into the imagination.

The more "advanced" and "intelligent" we think we are, the more ignorant we become, we forget that we dont know far more than we know. Flawed statistical models aid this process greatly.



posted on Nov, 20 2010 @ 12:00 AM
link   
reply to post by Xcalibur254
 


With statistics you can make anything come true, that is why today's science likes it when looking for funding or lying to the public.



posted on Nov, 20 2010 @ 12:13 AM
link   
the biggest group to misuse statistic have been the delusional religious. I mean it takes somone with a small enough IQ to believe that the crap preposed by the bible, koran, or torah is truth.

Im pretty sure any mistakes made would more likely in the hands of fools like that. You have to do studies..and you have to work on statistic for NOW because biological interactions have so many variables you HAVE to simplify it.

crying over statistics like it invalidates all science is incredibly pitifull.
edit on 20-11-2010 by Wertdagf because: (no reason given)



posted on Nov, 20 2010 @ 06:06 AM
link   

Originally posted by Xcalibur254
Here is a new article that can act as a compliment to the thread on falsification of data. I'm not quite sure how to feel about this. My personal field of research is in psychology, so I use a lot of statistics and I thought I had a pretty good grasp on the concepts.


This is pretty old news, and really only comes down to misinterpretation of statistics and a less than great understanding of the philosophy of science. Not statistics' problem. For example, the author poses the common misinterpretation:

"Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke"

And this merely illustrates that a little bit of philosophy of science, along with better statistical teaching/understanding, would help no end at UG level (and above!).

Even Ron Fisher never made such errors (the major proponent of NHST), and was well aware that determining 'real' effects required repeated replication. As a psych student, you might enjoy this 1994 article by the fantastic Jacob Cohen:

The Earth is Round (p < .05)

And this is one reason the APA have been pushing for effect sizes for at least a decade. Reading Ronald Fisher's early work is pretty enlightening (he's also the sadistic guy you can despise for ANOVA). NHST can actually be informative (which is what stats is all about), but on its lonesome it's pretty weak sauce (but better than no sauce, which is almost what we had pre-Fisher).

Some of the statements he makes are ridiculous: "a mutant form of math has deflected science’s heart from the modes of calculation that had long served so faithfully" - and just shows the author to be a typical journo-type.

Fisher was pretty clear (especially later, no so much in the early work) that simplistic confirm/reject decision-making is not really ideal for NHST:

"tests of significance are used as an aid to judgement, and should not be confused with automatic acceptance tests or 'decision functions'" (1958)

Which was aimed at the Neyman-Pearson perspective.

But, yeah, Bayesian approaches would be much better. They also tend to bring the advocate wine, attractive partners, and an awesome style which cannot be bought on the Milan catwalks.
edit on 20-11-2010 by melatonin because: Bayes, Bayes, my kingdom for some Bayes.



posted on Nov, 20 2010 @ 10:00 AM
link   
reply to post by Xcalibur254
 
Even if there's no falsification of data and the statistics is applied perfectly, I think there's an understanding gap between the creators of the statistics and some of the consumers of the statistics, especially among the general population.

For example, let's say 20 different studies are performed to a 95% confidence level.

On a typical sample of 20 studies, one of those conclusions can be wrong because of use of the 95% confidence level.

We could get that down to only 1 in 100 studies being wrong if we used a 99% confidence level test, but 95% confidence is much more common, isn't it? At least that's the case in the studies I look at but I'm not in the medical field.

So when 1 out of 20 studies reaches an incorrect conclusion based on 95% confidence tests, the statistics isn't flawed, it's working just the way it's applied and doing what it's supposed to do. Most non-statisticians and non-scientists I talk to, don't seem to understand this fact.

Here's a medical site that says that's what they use:

clinicalevidence.bmj.com...


We always try to provide 95% confidence intervals for results.



posted on Nov, 20 2010 @ 07:35 PM
link   
Give them enough time, and I'm sure a skilled statistician could make the existence of Santa Claus a probability and the existence of Nessie an absolute certainly.



posted on Nov, 20 2010 @ 07:40 PM
link   
Statistics are a way to visualize mathematics and data. In hypothesis testing, statistics are wonderful in thatvwe can make inferences about data populations and guess, with a greater probability than flipping a coin the out come. For me, that's a wonderful thing!



posted on Nov, 20 2010 @ 07:54 PM
link   

Originally posted by zvezdar
The misuse of statistics is prevalent throughout society, and not just in the sciences (look at how the misuse of statistical models has afflicted finance...despite the many failings complex models are still being used as a source of truth when they are nothing of the sort). However the fact that many scientists have little statistical training means that the statistics and statistical models used in science seem particularly weak and prone to misuse.


It is true, but the statistical capability is getting generally better.



Medical science is an obvious one but climate science is another where the statistical training simply isnt there to properly understand the shortcomings of what they are doing. They put faith in models which are not suited to what they are doing. The simple fact that the ability to hindcast does not demonstrate predictive power doesnt even seem well understood by most!


Climatology is not dependent on statistics, but on physics. That the ability to hindcast doesn't demonstrate predictive power ---by itself --- is perfectly understood by climatology. These models are not at all just data-driven statistical models, but physical models based on specific individual mechanisms consistent with the laws of chemistry and physics. And in this sense, the combination of the ability to hindcast plus evaluation of the fidelity and sensibility of the equations of motion give more confidence. Climatology is 95% physics and 5% statistics.

In medicine, there are far less secure and reliably predictive mechanistic models, and that's where we need to be more careful about statistical inference.

And yes, I fully support the Bayesian approach here, because it gives more useful and applicable results, evne with acknowledged judgemental inputs.

A clinician seeing a real patient always in the Bayesian fashion: given my experience and others of what things are likely to be, and given the symptoms the patient is presenting, what overall is my current best guess of what I should do next, given both probability and the risks of being mistaken.


Physics has been afflicted with a slight variation of this issue, where the mathematical abstractions have become so complex that they have veered physicists away from reality and into the imagination.


That's a substantially different issue. And most physicists try to stay close to reality as they can.
edit on 20-11-2010 by mbkennel because: (no reason given)



posted on Nov, 20 2010 @ 08:01 PM
link   

Originally posted by Astyanax
reply to post by Xcalibur254
 

I have often thought this myself, and usually about medicine.

I think the eraly success of statistical analysis in correctly identifying the association between smoking and lung cancer, even when the mechanism by which smoking causes cancer was not known, sparked a slow-building explosion in the field of statistical medicine. Nowadays, it seems that the majority of popular news items on health, diet, etc., are based on statistical studies or metastudies (studies of studies). That's how we learn that drinking so many millilitres of red wine a day makes people less prone to heart attacks, that people who exercise live longer, and so on. For all we really know, none of it is true.

I tend not to believe what I read in the media about medical subjects for precisely this reason--most of what is presented as near-certain is based merely on some statistical correlation, without any causal mechanism established to back it up.


True, but this can still be the right thing to do. Otherwise everybody would still be smoking and dying until every possible mechanism is elucidated.


This over-reliance on statistics is not an indictment of science in general, by the way--merely of those who place too much confidence in dubious statistical analysis.


How much is 'over-reliance' and how much is proper reliance?

In reality, professional statisticians know how to interpret statistical results the best and are the least fooled---they are the source of the remedy, not the problem.


edit on 20-11-2010 by mbkennel because: (no reason given)



posted on Nov, 21 2010 @ 12:08 AM
link   

Originally posted by mbkennel

Climatology is not dependent on statistics, but on physics. That the ability to hindcast doesn't demonstrate predictive power ---by itself --- is perfectly understood by climatology. These models are not at all just data-driven statistical models, but physical models based on specific individual mechanisms consistent with the laws of chemistry and physics. And in this sense, the combination of the ability to hindcast plus evaluation of the fidelity and sensibility of the equations of motion give more confidence. Climatology is 95% physics and 5% statistics.


I disagree, the only way that climate scientists have been able to make a tenuous case for disasterous climate change is by using statistical models to recontruct past climate via proxies. The idea being that to demostrate that current climate change is due to a different driver from the past requires a statistically significant deviation from past climate history. Without that all you have is a dynamic climate and no disasterous 'change'.

That is where the whole 'climate change' paradigm breaks down, the statistical models they have been using are completely inadequate for what they are trying to achieve and time after time it has been shown that the limitations of their models are poorly or incompletely understood. The only reason they pass the peer review system is because everyone else that is involved has an equally poor understanding of statistics.

The current radiative physics models are completely incapable of longer-term climate prediction (and the dynamic physics of the earth's climate remains poorly understood overall), hence a reliance on arguments based on the statistical reconstructions. Take away the reconstructions and all you have is short-term predictive power and no 'catastrophe' on the cards.

For me short-term climate modelling is 95% physics, but the 'climate change' issue iself is about 5% physics and 95% everything else (included in that 95% is poor statistics).



Another post above mentions confidence intervals, the use of 95% confidence intervals seems to come from the way statistics is taught in universities; the 95% level is always used to test a one-tailed null hypothesis. Its usually not questioned by students and one of those things that is accepted. However after seeing financial models built to 99.5% confidence fail during the finance crisis it makes the use of a 95% CI even more ridiculous as a benchmark for significance in important trials.



posted on Nov, 21 2010 @ 12:46 AM
link   
reply to post by mbkennel
 


True, but this can still be the right thing to do. Otherwise everybody would still be smoking and dying until every possible mechanism is elucidated.

I completely agree, though my own attempts to give up smoking have been hopeless...

I think the problem--it is a problem of public perception and behaviour--has at least three contributing factors. They are:

  • the public response to medical and health issues in general, which is governed by ignorance, fear and faddism. This really can't be helped--folks will be folks--so the problem has to be addressed on some other level.

  • the use made of medical and health (and yes, climatology) research by those who seek to attain some end by influencing public reactions to it. These people include journalists who need to find a hook for their stories in a ferociously competitive media environment, politicians and bureaucrats in charge of public-health and climate policy, opponents seeking to unseat those politicians at the next election, companies and states with commercial or economic interests that may be served or harmed by certain research, religious congregations whose doctrines may be threatened or supported by it, etc., etc. All these will represent or misrepresent the research to their advantage, further confusing the public.

  • scientists themselves, especially young ones in a hurry to publish and make a name for themselves, or attract funding to their research by raising public interest in it. I am not alleging a scientific conspiracy or cabal, but it is naive to imagine that scientists are somehow immune to human frailty.


How much is 'over-reliance' and how much is proper reliance (on statistics)?

As you point out, it is professional statisticians to whom we must turn for this. However, they are rarely consulted by those who feel they must make a personal life choice based on something they read in Newsweek.



posted on Nov, 21 2010 @ 01:47 PM
link   

Originally posted by zvezdar

Originally posted by mbkennel

Climatology is not dependent on statistics, but on physics. That the ability to hindcast doesn't demonstrate predictive power ---by itself --- is perfectly understood by climatology. These models are not at all just data-driven statistical models, but physical models based on specific individual mechanisms consistent with the laws of chemistry and physics. And in this sense, the combination of the ability to hindcast plus evaluation of the fidelity and sensibility of the equations of motion give more confidence. Climatology is 95% physics and 5% statistics.


I disagree, the only way that climate scientists have been able to make a tenuous case for disasterous climate change is by using statistical models to recontruct past climate via proxies. The idea being that to demostrate that current climate change is due to a different driver from the past requires a statistically significant deviation from past climate history. Without that all you have is a dynamic climate and no disasterous 'change'.


No that's not true. You have to demonstrate that you are changing the boundary conditions of the atmosphere in a way that hasn't existed in fifty to a hundred million years (check---the carbon being released has not been in the biosphere for many upon many ice age cycles), and that there are physical mechanisms, which have been verified, which will change the energy balance (check---radiative properties of greenhouse gases are very well known and verified experimentally).



That is where the whole 'climate change' paradigm breaks down, the statistical models they have been using are completely inadequate for what they are trying to achieve and time after time it has been shown that the limitations of their models are poorly or incompletely understood. The only reason they pass the peer review system is because everyone else that is involved has an equally poor understanding of statistics.


No, this isn't true. Climate models aren't statistical models, they're physical models.


The current radiative physics models are completely incapable of longer-term climate prediction (and the dynamic physics of the earth's climate remains poorly understood overall), hence a reliance on arguments based on the statistical reconstructions. Take away the reconstructions and all you have is short-term predictive power and no 'catastrophe' on the cards.


So, the best physical models which have been pretty adequate to explain everything we've seen in the modern instrumental era and are based on laws of phyisics which are fully proven show substantial to nearly catastrophic effects in the next few hundred years? (which is tiny next to geological time).

What is the best guess as to what will happen in the future in the next hundred years
a) most probably in the range of models
b) nothing at all, because the models aren't good enough to predict 50,000 years in the future

Do you "feel lucky today"?


For me short-term climate modelling is 95% physics, but the 'climate change' issue iself is about 5% physics and 95% everything else (included in that 95% is poor statistics).


Why? What about the ozone hole? How much paleoclimatic evidence was there regarding past behavior? Virtually none. Did that stop people from doing something about it? No. Why? Because the effect was very well proven to be a consequence of physics and chemistry (chemistry in cold icy particles, different from most lab experiments as it turned out). Based on this undersanding people predicted what would happen if we did Y, and we did Y, and what was predicted happened. The same is happening now with global warming, but people this time don't want to believe it.

The point is that even global planetary physics and chemistry is not arbitrary and random and unknowable, because just as physics leads us to adequate predictions of all sorts of things---with elementary or no statistics needed---the same applies to the basic properties of the Earth and other planetary bodies.

In fact, I wish the paleoevidence for climate wasn't so prominent, because then average people would think more productively about the issue.

Let's come back to basics: the increase in infrared radiation emitted back towards is increasing as a consequence of the increase in greenhouse gases. This is not a hypothesis, it is an experimentally measured fact. Another fact is that these greenhouse gases will be increasing substantially, and effect will thus increase. It is thus physically impossible for the climate not to change because the electromagnetic energy hitting the surface is overwhelmingly the most important influence on the climate.



Another post above mentions confidence intervals, the use of 95% confidence intervals seems to come from the way statistics is taught in universities; the 95% level is always used to test a one-tailed null hypothesis. Its usually not questioned by students and one of those things that is accepted. However after seeing financial models built to 99.5% confidence fail during the finance crisis it makes the use of a 95% CI even more ridiculous as a benchmark for significance in important trials.


The financial models failed because certain humans had a very strong interest in breaking them.
edit on 21-11-2010 by mbkennel because: (no reason given)

edit on 21-11-2010 by mbkennel because: (no reason given)

edit on 21-11-2010 by mbkennel because: (no reason given)



new topics

top topics



 
4

log in

join