I recently became a member of ATS and after posting a few threads, and responding to several posts, I noticed that many posters have a medicine of last resort, when their arguments don't convince others. Their response is something like this: "Your source hasn't been peer reviewed"
I'd like to know what others think.
For those not familiar with the jaded history of peer review, you might want to Google "Sokal affair", a famous hoax perpetrated on academia to expose peer review and its flaws. That was a non-sense paper that passed peer review. There are many such papers. In addition, there are works/papers that were rejected by peer review that ended up winning the Nobel Prize (Krebs Cycle-1937 for one). There were also Nobel Prize winners that did not go through the peer review process, such as Abdus Salam, “Weak and electromagnetic interactions” (1968), and Watson and Crick, 1951, a paper on DNA in nature. (reference-Peggy Dominy & Jay Bhatt,"eer Review in the Google Age" .

Despite of all the facts (or more properly, exceptions) you listed, peer review has its "usefulness". It has been an effective practice, although the outcome is not always correct. Some of the listed problems could be justified by "lack of knowledge of experience at that moment".

The thing I most disagree about this "peer review" of many journals is that they let the author to propose the "peer". In that case, many authors try to propose their "friends" or close colleagues as peer reviewers. This makes the practice lack of objectivity.

Originally posted by ProfEmeritus
I recently became a member of ATS and after posting a few threads, and responding to several posts, I noticed that many posters have a medicine of last resort, when their arguments don't convince others. Their response is something like this: "Your source hasn't been peer reviewed"
I'd like to know what others think.
For those not familiar with the jaded history of peer review, you might want to Google "Sokal affair", a famous hoax perpetrated on academia to expose peer review and its flaws. That was a non-sense paper that passed peer review. There are many such papers. In addition, there are works/papers that were rejected by peer review that ended up winning the Nobel Prize (Krebs Cycle-1937 for one). There were also Nobel Prize winners that did not go through the peer review process, such as Abdus Salam, “Weak and electromagnetic interactions” (1968), and Watson and Crick, 1951, a paper on DNA in nature. (reference-Peggy Dominy & Jay Bhatt,"eer Review in the Google Age" .

Peer review happens both before and after publication. It`s an ongoing process. Pre-publication peer review happens in some cases, but more often than not it`s a post-publication process to verify the results/observations/etc. Sometimes the whole process takes place within a single lab - getting another person to verify your conclusions. Sometimes papers are presented in public (thesis defense, symposium, conference) and questions are fielded on the spot. Sometimes debates rage in the literature via letters for years.

Taking a look at a couple of your examples:

The Abdus Salam article you mention appears here:

Elementary Particle Theory. Relativistic Groups and Analyticity. Proceedings of the eighth Nobel Symposium, Aspenäsgården, Lerum, Sweden, May 1968. Nils Svartholm, Ed. Interscience (Wiley), New York, and Almqvist and Wiksell, Stockholm, 1969.

This is an example of a symposium paper. It would have been first reviewed by the symposium organizers, at least in part, provided to attendees beforehand, and questions taken on the spot. After publication... 39 years of peer review thusfar.

I`m not sure what you (or Dominy and Bhatt) are referring to regarding the Watson and Crick paper, because the earliest paper I can find that they co-authored in Nature was in 1953. As their 1953 papers (quite short, the most important one is only a single page) detailed discoveries arrived at through current research, there may have been no pre-review process beyond checking the details of the paper before opening it up to comments. As far as it having had no peer-review though: they`ve had over 50 years of peer review of their work.

The Krebs article from 1937 was published in the journal Enzymologica, and it outlines the Krebs Cycle. Can`t comment on it being rejected without knowing the details. Regardless, it was published, and it`s had 70 years worth of peer review at this point, and still taught in basic chem classes.

I`m not sure where you`re going with this, exactly. There have been examples of hoaxes - although you (and Dominy and Bhatt) left out my favourite - the Flashman Papers - but for the overwhelming majority of cases, bad science, bad history, bad archaeology, everything but bad poetry tends to work very well via peer review, providing you remember that the peer review process does not end at publication.

Peer reviewed scholarly journals are generally more credible. Your argument will have greater weight if it is backed up with a factual proposition from a scholarly journal, rather than a factual proposition from some other source. This is not to say that a super market tabloid is always false, or that articles published in peer reviewed journals are always true. It's just that something that appears in a peer reviewd journal is more likely to be true.

Sorry I'm just getting back to ATS. We've been away for a few days visiting family.
Anyway, let me try to respond to the three posters. First, you all make some very good points. I'm not saying that peer review is so flawed that it shouldn't be used. Vox makes a good point, in that the longer the process goes on, the better the chances are that the work under review is fully vetted.
To answer the question about where I am going with this, here is my concern.
Radical new ideas are often met with total disdain, during the peer review process, because the work usually goes against established, some say "mainstream" ideas and theories. Theories, now accepted as most probably correct, such as plate tectonics(probably the MOST negatively reviewed theory by peers), solar-centric solar system, Daltons' atomic theory, and many others were rejected because of the peer process. Yes, time, many times wins out, but in some cases, over decades and centuries.
In other cases, "peer review" is nothing more than, as fuelcell points out, a more sophisticated rendition of the "good old boys" network. I'll review yours favorably, you review mine. (And please don't tell me it doesn't happen, because I witnessed a case where it did.)
Finally, in some cases, peer review is nothing more than political. Probably, the most recent issue involves "Global Warming". I don't want to get into that, as there are probably many threads on that already. However, at a time when, at least the Nobel Committee and the MSM have seemed to accept it as fact, there are many in the field that believe just the opposite, or in some cases, looking at subsets of that theory, that:
1.) carbon emissions have negligible effect on global warming
2.) there is no discernible global warming
3.) there is a more pronounced global cooling over a longer period of time

Again, I don't wish to get into the global warming scenario, as it is not a field that I am, in any way, an expert on. By the way, I don't think Al Gore has any academic credentials in that either, but ....

Now, to get to the point- Let's just say that you do come up with a theory that may lead to a cure for cancer. Furthermore, let's say that in the peer review process, it is shot down. Grant money probably will be hard to acquire. OK, you say, as time goes by, maybe it becomes accepted. (Plate tectonics took 50 years to reverse). The theory is finally accepted, (maybe 50 years from now?) A cure is finally developed. How many people died of cancer in that time that might have been saved over 50 years? What if it were your spouse, your child, your grandchild, someone that may have had some major breakthrough that led to other developments to benefit humanity.

Hypotheticals don`t work well with this type of thing - the field is simply too large. Your cancer situation is too laced with possibility and uncertainty to comment on - one may as well extend the problem to all aspects of life. I could have been CEO of company X, had I not been turned down at interview Y.

I`m sure that many ideas that could have led to groundbreaking research have been rejected. I`m also sure that 99% of the people I meet on a daily basis have had ideas that could have, with more research and a bit of cash, turned into something bigger. Is the peer review process to blame? Or is the peer review process simply a convenient scapegoat in this case, alongside high school guidance councilors, coaches, old boys networks and Catholic Mothers?

I`ve seen a number of papers blocked by initial peer review. Written a few of them myself. The fault, in my case, was with the writing style and presentation of the idea, among other things. If the idea is sound, then it`s worth pursuing. If an idea is based on faulty information, or the argument is easily negated by the existence of contrary information of which the author may or may not be aware (*cough* been there) then the author can re-evaluate their own research, and decide whether or not to pursue it.

Another thing to keep in mind is that the peer review process itself has been undergoing an ongoing ... er.. peer review. As technology improves, as the number of peers increases, as the number of potential publications and academic institutions increases, so does the efficiency of the process. In other words, there are more peers to review.

The tectonic plates issue is a particularly interesting one. The reluctance of academia to accept continental drift is actually tied more closely to the influence of Christian ideas of creation in the field of geology and the lack of a cohesive body of researchers than to anything that could be described as the modern peer review process. Geology, like most scientific disciplines, is actually fairly "young". At the time when the ideas of continental drift were first being proposed, the vast majority of research in the discipline was done by and for mining and oil companies, and by people who quite often were using the Bible`s version of creation as a jumping off point for their research. It`s a different ballgame now. I think that you have to keep that in mind when examining this issue - bringing Copernicus into a discussion of modern day peer review makes no sense. Most serious journals have abandoned the thumbscrew and trial by fire elements of the peer review process. Most.

Finaly, I think you may be confusing peer review and mainstream acceptance of an idea. The climate change issue - which I only mention as an example, I`m not willing to get into a debate on it here - is a very good example of the lag between these two processes. Just because there has been no mainstream acceptance of (or reporting on) the effect of solar output on global climate change, does not mean that there is no one conducting research on the issue - and the non-acceptance of current research on that topic does not necessarily mean that there is a conspiracy of scientists to retain their intellectual niche; it may mean, quite simply, that the research is a dead end. Or that it`s ongoing. Stranger things have happened.

Evolutionary biology is another example. The MSM seems to have a lag of nearly a decade in current research on this one. It`s almost to the point where we have two sciences: academic science and colloquial science - the former being what`s current, the latter being what the average joe thinks is current, courtesy of the discovery channel.

The peer review process ensures that all current work, even things that the media (in all it`s iterations) would have us believe are sacrosanct to the mysterious men in their lab coats, are under continuous evaluation. I`d be willing to bet that at this moment, there are scores of physicists toying with how Einstein`s theory of relativity can be disproved. Someday it may be. Some people will be reluctant to let go of it. But if the proof is sound, it will fall. The same can be said of all "radical" new ideas. If the proof is sound, if the research is sound, then they will be accepted - not as fact, but rather as another step on the ladder, from which new research will emerge. It`s a process. Today`s science is tomorrow`s bar trivia.

Moonvibe,
Thank you very much for the link. That says it better than I did.

You are so welcome!

As I was reseraching a topic I stumbled on that site, THEN I read this thread. There are no coincidences

Papers can appear on arXiv that would not appear in other sources.
If you wish to have your papers peer-reviewed, that is your prerogative. However, much of the peer-review, involves the "good old boys" network, and thus, the stamp of approval is meaningless. It doesn't mean that the paper is not worthwhile. It just means that the peer-review acceptance does not mean much.
On the other hand, papers that go against orthodoxy, have a problem right off the bat. Mavericks don't usually play with the "good old boys". In addition, if the paper challenges what they have accepted as fact all of their lives, then those papers pose a threat to them, to their funds and to their reputation.

I can see that you are not budging, but neither am I. I have experienced situations where the above has occurred. And no, they were not my papers, but papers of people that were brilliant, and attempting to open up new horizons.

Look, in most instances, most of the papers published are relatively insignificant in their impact on society. In fact, many of the papers are pure Bull----, to use the vernacular. But they seem to bring in grants. Much of it is a sick game. I say that, because those funds could be used to benefit mankind in much better efforts.
Those, however, that push the envelope are the ones that could potentially lead to huge breakthroughs, that benefit mankind.

Non-peer-reviewed outlets provide such papers the opportunity to do that.

Originally posted by ProfEmeritus
Non-peer-reviewed outlets provide such papers the opportunity to do that.

Do they?

The link posted by moonbeam - the one you said illustrates your point, presumably that peer-review is an enemy of progress - illustrates that they are even more susceptible to the stamp of the old boy`s network than anything else. I`ve never heard of a scientific journal admitting in their submissions guidelines that "in the past, our system has been arbitrary". Have you?

we have an internet now. And last time I looked, it was full of cures for cancer.

And how many "cures" for cancer are there in the Journal of the AMA or other "prestigious" medical journals? In fact, is there even ONE?

You see, THAT is my point. Most of the articles published in those journals are pure hogwash. Let me tell you what I mean by that. Most of those papers never result in ANY REAL benefit to mankind. Yes, there are some, but the vast majority of those papers are utter gibberish. Oh, I know, the equations are sound, the logic is sound, but as far as real progress for mankind, I don't think so. In fact, MOST real advances are produced by private enterprise that performs research to benefit their companies, and in the long term, those benefit mankind. Academia, in many cases wouldn't know a cure for cancer, if they were staring it in the face. Publish or Perish, is not the way to produce sound advances, it's just the way to fill paper with meaningless ink.

Originally posted by ProfEmeritus
And how many "cures" for cancer are there in the Journal of the AMA or other "prestigious" medical journals? In fact, is there even ONE?

You see, THAT is my point.

Well, the 'cures' on the internet don't work either. They only serve to give false hope to unhappy, desperate people.

On the other hand, peer-reviewed medicine has developed effective treatments that can halt or reverse the progress of many kinds of cancer; to offer just one example, the armoury of treatments that now exist for breast cancer has saved or prolonged many lives.

You haven't addressed my real point, which is that science without peer review is not science at all. Peer review is, quite simply, the way in which we verify that the claims of scientists are true. Without it, how will you tell truth from conjecture and quackery? Do you imagine that you can just put everything out in the public forum and let the people decide? Truth is not determined by majority vote, nor by market forces.

Eliminating peer review (assuming it were possible) would destroy science. It would render all scientific study equally untrustworthy. It would force scientists to become mummers cavorting in the media to attract attention and research grants. It would increase the pressure to portray every little completed study as resulting in a breakthrough, leading 'scientists' to falsify or exaggerate their results -- without peer review, who would be there to stop them? It would divert funds from the really important work that needs to be done (work that the public and politicians do not understand) to politically attractive crank projects that can be packaged interestingly for the man in the street. The few genuinely scientific projects that did get funds would have to waste a large part of them on advertising and public relations. Ultimately it would reduce science to the kind of rabble-babble you see in mass media and (especially) on the internet.

Science must always make some accommodation with the public: first, because much research is privately funded, and most of that money goes into research designed to produce something marketable; and second, because government funding is determined, in democracies at least, by what politicians think the public wants, or at least will stand for. A little accommodation of this kind may be good for science, but too much of it is destructive, and we have too much of it already. Eliminating peer review will bring more -- exponentially more. And what would be the corresponding benefit? None, except to cranks and frauds who now see their road to fame and fortune blocked by the barrier of peer review.

If you are a genuine scientist, you cannot possibly object to peer review per se. You will be aware that without it there can be no science. You may object to the process as it is currently conducted -- entry barriers, old-boy networks and so forth -- but that is an argument for reforming peer review, not for doing away with it.

Yes, of course, the cures for cancer on the internet do not work.
I will concede that what bothers me about peer review is the process as it exists today, not that it would not work in theory. I'm not sure we can ever get rid of the 'good old boy" network, or the "you review mine favorably, and I'll do the same for you". I also believe that the truly novel approaches to anything are going to be resisted, any time that the status quo is threatened. Three of the enemies to new ideas are the competition for funds, and the fear of loss of prestige, should that new approach negate previously accepted tenets.
In the last few days, there has been much press concerning the new study which seems to show that anti-depressant medications don't work any better than placebos. Let's see how that paper (Prof Irving Kirsch from the department of psychology at Hull University and colleagues in the US and Canada.) is accepted, and how quickly those that have an interest in continuing to prescribe the medications, are willing to accept the new "findings". I would bet that they are vilified, smeared and subjected to the modern-day equivalent of tar and feathering. I will freely admit that I am wrong if that does not happen.
Anyway, you did make some very good points, and I do admit that it is the process, as frequently practiced, that I have a problem with.

Peer review is generally useful. It gets dangerous when committee members start trading favors. But that tends to happen less in the larger conferences and more in the smaller circles of workshops. The more competitive a publication, the more prestigious it tends to be. Places that are easy to publish in are generally ignored by top tier academics.

Most of the professionals I know spend most of their time trashing each other on the committees of high-esteemed publications. There is little conspiracy. However, the communities I know tend to fracture into subgroups that largely oppose one another's papers, and this reduces the granularity of peer checking, thereby weakening the system. A good committee tries to limit such groups' power when it is known. Occasionally, these safeguards fail, but then the publication's standing tends to fall as it fails to publish the best work over a period of a few years.

This feedback loop tends to help journals, magazines, conferences from decaying too far into social cliques. But on occasion, when they do, new publication centers arise to replace them. Its surprisingly effective over time, despite all of the heartache and failure one can personally experience in the processes.

There will inevitably be landmark papers rejected by peer-review. Those papers, however, are generally regenerated later when peer-process is ready to accept them. (Though often not by the same authors within the same subcultural context.) Usually its another group that is more popular or better at expressing the idea in the language of a particular community that succeeds in publishing the idea first.

That's my 2 cents.

[edit on 29-2-2008 by Ectoterrestrial]

However, the communities I know tend to fracture into subgroups that largely oppose one another's papers, and this reduces the granularity of peer checking, thereby weakening the system.

Your above quote sounds just like the environment that I was in. In fact, it wasn't limited to papers, but extended to curriculum and courses. In fact, much of it was more juvenile than a middle school cafeteria food fight. I was only in academia for the last decade of my working life, having spent thirty some years in industry, but I have to admit that it was not a very pleasant experience, although I immensely enjoyed the teaching, and had great rapport with the students. The politics I could have done without.

Thanks for your comments.