Faith, Fantasy, And The Protean Peer Review Process

We frequently hear from those among us who place their faith in the scientific paradigm, about the joys and wonders of “peer review”. That blessed system, and paragon of virtue among our beloved academia that metaphorically, is equal to the holiest of the church presbytery itself. Without peer review, any unholy charlatan or crackpot could publish any pseudo-science paper he desired, and have it accepted by the public, just because it was published in a scientific journal. Without peer review, how would we maintain quality, accuracy, and scientific integrity?

To answer that, we must first understand what “peer review” is. You will find many references to “peer review”, but definitions can be somewhat varied, the harder you look. However, one definition that seems to be agreed upon is:

"Peer Review is a process that journals use to ensure the articles they publish represent the best scholarship currently available. When an article is submitted to a peer reviewed journal, the editors send it out to other scholars in the same field (the author's peers) to get their opinion on the quality of the scholarship, its relevance to the field, its appropriateness for the journal, etc."

Opinion? Wait a minute. Opinion!? What does that mean? Is that like an educated guess? Or is opinion, fact in this case? Hmmm.

"Peer review is at the heart of the processes of not just medical journals but of all of science. It is the method by which grants are allocated, papers published, academics promoted, and Nobel prizes won. Yet it is hard to define. It has until recently been unstudied. And its defects are easier to identify than its attributes. Yet it shows no sign of going away. Famously, it is compared with democracy: a system full of problems but the least worst we have."

Hard to define? How can a process that is so integral to the academic community, and so relied upon by those who keep up with the latest advancements, be “hard to define”?

"Peer review is thus like poetry, love, or justice. But it is something to do with a grant application or a paper being scrutinized by a third party—who is neither the author nor the person making a judgement on whether a grant should be given or a paper published. But who is a peer? Somebody doing exactly the same kind of research (in which case he or she is probably a direct competitor)? Somebody in the same discipline? Somebody who is an expert on methodology? And what is review? Somebody saying `The paper looks all right to me', which is sadly what peer review sometimes seems to be."
OR...
"...somebody pouring all over the paper, asking for raw data, repeating analyses, checking all the references, and making detailed suggestions for improvement? Such a review is vanishingly rare. What is clear is that the forms of peer review are protean."

Protean? This is hardly a word I would expect to hear in relation to peer review. The author of this particular article goes on to explain that “grant giving bodies” and journals differ in methodology, and some very much so. Including what is called by the author “the classic system”...

"The editor looks at the title of the paper and sends it to two friends whom the editor thinks know something about the subject. If both advise publication the editor sends it to the printers. If both advise against publication the editor rejects the paper. If the reviewers disagree the editor sends it to a third reviewer and does whatever he or she advises. This pastiche—which is not far from systems I have seen used—is little better than tossing a coin..."

Classic? This is sounding less like science, and more like nostalgia.

"...A systematic review of all the available evidence on peer review concluded that `the practice of peer review is based on faith in its effects, rather than on facts'..."

I think that statement needs a little clarification. The faith they’re talking about here is, that if three astrophysicists examine a paper written by a fourth, surely they are more likely to catch errors, poor procedure, lack of adherence to protocol, and so on. Surely that is a logical assumption. Or is it?

"At the BMJ we did several studies where we inserted major errors into papers that we then sent to many reviewers. Nobody ever spotted all of the errors. Some reviewers did not spot any, and most reviewers spotted only about a quarter. Peer review sometimes picks up fraud by chance, but generally it is not a reliable method for detecting fraud because it works on trust."

So what’s the point? The bottom line. Simply put...

"People have a great many fantasies about peer review, and one of the most powerful is that it is a highly objective, reliable, and consistent process."

The truth is, it isn’t. I looked at different articles online reviewing the peer review process. Some were unduly critical of it, and some went overboard with glowing reviews. This article, which I highly recommend you read before commenting in depth, seemed to be more realistic in its approach to explaining what peer review is, and what we can realistically expect from it. To me, the bottom line is the same as it is for anything else. Question everything. Research, backtrack, dig deeper. Just because it is “peer reviewed” doesn’t mean it is now the gospel, or the Holy Grail of scientific accuracy.

At this point, I’d like to give some examples, and pay a little homage to those who, in the past, suffered the wrath of their peers in the scientific and academic community of their day.

1847 - Ignaz Semmelweis - Semmelweis brought the medical community the idea that they were killing large numbers of new mothers by working with festering wounds in surgery, then immediately assisting with births without even washing hands. Such a truth was far too shameful for a community of experts to accept, so he was ignored. Semmelweis finally ended up in a mental hospital, and his ideas caught fire after he had died.

1964 - George Zweig - Zweig published quark theory at CERN in 1964 (calling them 'aces'), but everyone knows that no particle can have 1/3 electric charge. Rather than receiving recognition, he encountered stiff barriers and was accused of being a charlatan.

1982 - Prusiner, Stanley - Prusiner endured derision from colleagues for his prion theory explaining Mad Cow Disease, but was vidicated by winning the Nobel.

1827 - George S. Ohm - Ohm's initial publication was met with ridicule and dismissal; called "a tissue of naked fantasy." Approx. twenty years passed before scientists began to recognize its great importance.

1973 - Virginia Steen-McIntyre - Steen-McIntyre innocently stumbled into heresy when she found wide-ranging evidence that native settlements in the USA southwest were 300,000 years old. This damaged here career, since the dates acceptable to the archeologist community are more like 50,000BC.

And many more examples in the links I will provide at the end.

Continued...

...continued

In my personal opinion, the peer review process has at once been set on a pedestal as infallible, and at the same time unduly criticized. Two extreme views, both unbalanced. As a modern society that lives and breathes technology and advancement, as well as a species in search of our origins and history, the peer review process is valuable to the effect that it endeavors to create accountability among academia. It tries to maintain a standard of scholarship, ethics, and responsible reporting of properly conducted experiments, and observable evidence for the conclusions set forth in a hypothesis, and eventually, a scientific theory.

It is fallible to the extent that humans, with all of our imperfections, are the ones doing the reviewing. For many reasons, the peer review process is flawed. Jealousy, conflicts of interest, money, corporate and government interests, confirmation bias, consensus science, and those who have a vested interest in maintaining the status quo. In all fairness to those involved in the review process, I must also mention that many of these academics are not paid for their time reviewing other peoples work. They can easily become inundated with requests for reviews. It is partly because of this, that shortcuts are taken, and those with well known names have an easier time with the process. Where those with unrecognized names, or who are on the “fringes” can be shot down quickly. No matter how plausible and probable their conclusions may be.

The peer reviewed article, or publication should be questioned and researched no less than any other source of information.

Links:
Peer review: a flawed process...
Peer Review...
Ridiculed Discoverers...
5 famous scientists...

fortunately, 'piers' will collapse ...




as for a horror story concerning peer review,
who's the dr. that claimed he invented aids?

reply to post by Klassified

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

Arthur C. Clarke's First of the Three Laws

I notice this more and more with my peers. There seems to be a predilection as we age to assume we already know the answers just by reading an abstract or even skimming a headline. I don't like it, but what do we do? Humans tend to get set in their ways. It is easier to dismiss a fledgling idea than it is to really think on it to separate out the warts. Then factor in all the half-thought out papers that get tossed around due to "publish or perish" and we have a recipe for a culture that starts to think everything is nonsense. I find a dynamic state of tension where I don't even wholly trust myself makes me more honest. However this isn't a very comfortable position because it means constantly reassessing everything. I don't think the human psyche, let alone the human mind, is designed for such a thing. But nice write up. The problem is real and unfortunately there are no simple answers.

reply to post by Xtraeme

I think it is often the journals who have made the process out to be more than it is, because it is their business to do so. They have a vested interest in the longevity and standing of their publication.

Because of the very things you mention, it is the publics responsibility, at least those who read those journals and papers, to understand what the peer review process entails, and the limits of its value to the reader and researcher.

Thanks for commenting. It's nice to have the input of someone who is more intimately familiar with the inner workings of the system.

BTW. That avatar is awesome.

Funny thing...

Here's a definition of "peer" from dictionary.com:

a person who is equal to another in abilities, qualifications, age, background, and social status.

When someone discovers something new they often have no peers in their particular area of expertise.

Furthermore the scientific establishment is full of starved egos who make themselves feel higher by pulling others down the ladder which is of course a false sense of empowerment. This is why I find the concept of "submitting to peers" a bit ironic, and frankly a waste of time.

reply to post by circlemaker

Regardless of the human shortcomings, the bigger problem I personally see, is the portrayal of a review system as being equally scientific in methodology to the material it is reviewing. Which is obviously not the case. From my perspective, the journals themselves should shoulder the responsibility of making the reader aware of the process of review, and its innate limitations.

Originally posted by Xtraeme

I notice this more and more with my peers. There seems to be a predilection as we age to assume we already know the answers just by reading an abstract or even skimming a headline. I don't like it, but what do we do?

We could try requiring some sort of sensitivity training.

reply to post by tinhattribunal

as for a horror story concerning peer review,
who's the dr. that claimed he invented aids?

I have tried to find this, but haven't had any luck. If you find it, please post it.

Originally posted by circlemaker
When someone discovers something new they often have no peers in their particular area of expertise.

Stunningly simple and elegant way to state it. Danke.

People forget to look at what their heroes of discovery and deep thought in the past went through in their attempt to get the ideas that are taken for granted now through. A shocking number never saw full recognition by the experts or "peers" while alive, if any. Except from a few people who "got them" and continued filtering the ideas out.

All we have to do is see who is behaving like who *now* to know which side of the fence history will leave those perspectives on.

reply to post by ErgoTheConclusion

People forget to look at what their heroes of discovery and deep thought in the past went through in their attempt to get the ideas that are taken for granted now through. A shocking number never saw full recognition by the experts or "peers" while alive, if any.

Indeed. The list is quite extensive.

"Concepts which have proved useful for ordering things easily assume so great an authority over us, that we forget their terrestrial origin and accept them as unalterable facts. They then become labeled as 'conceptual necessities,' etc. The road of scientific progress is frequently blocked for long periods by such errors." - Einstein

Originally posted by circlemaker
Funny thing...

Here's a definition of "peer" from dictionary.com:

a person who is equal to another in abilities, qualifications, age, background, and social status.

When someone discovers something new they often have no peers in their particular area of expertise.

Furthermore the scientific establishment is full of starved egos who make themselves feel higher by pulling others down the ladder which is of course a false sense of empowerment. This is why I find the concept of "submitting to peers" a bit ironic, and frankly a waste of time.

You are kind of right in all you say.
Although for obvious reasons, the journals are not really obliged to publish
anything that has security and defense implications.

reply to post by Klassified


Let me tell you a secret. Scientists are humans too. And as such, they are prone to make errors.

Peer review, in its core, is just a way of proofreading. Instrumentalization or abuse of a control element like peer review to achieve certain goals is regrettable, but also very human and can be observed in pretty much all areas of human life.

So can we please get back on the ground.

reply to post by moebius

If you had read my whole OP, you would know that I covered this aspect, and more in the interest of balance and fairness to those involved in that process.

In my personal opinion, the peer review process has at once been set on a pedestal as infallible, and at the same time unduly criticized...

www.abovetopsecret.com...
The idea of this thread was to bring balance, and realistic expectation to peer review. Not to belittle it, or glorify it. Unfortunately, none of those who fall to the extremes want to broach this issue, and therefore it has been largely ignored.

However, the crux of this thread was to do exactly what you stated...

So can we please get back on the ground.

I had hoped this thread would spur more discussion than it has, but the OP is rather lengthy, and there is much else going on at present, so it's understandable.

To those who have participated, I do appreciate the comments and opinions. Thanks.

reply to post by Klassified

Those who understand it... don't have much reason to discuss it.

Those whose beliefs depend on rejecting it... won't touch it.

Welcome to being ahead of the curve.

The Inductive Method is a term that has in some circles has been jokingly related to the Scientific Method because, of the difficulties in applying deduction to nature.

Further reading

The concept of "bottom up" logic, is responsible for all kinds of problems. There was a time when even slavery was supported after peer review, because the brains of slaves were less developed. Science is an institution governed by people, trying to define reality.

reply to post by Klassified


Have you ever gotten a paper in (or through) a peer review process? If so, I'm wondering what your thoughts are on how YOU would improve it (knowing that each journal gets hundreds of papers submitted to it for publication.) A journal like JAMA (Journal of the American Medical Association) is read by doctors and hospital staff all over the world and is one of the main journals that gets flooded with all sorts of articles.

They can't publish them all, and picking the ones to publish based on "did the authors attend important schools" or "what else have the authors published" is a really lousy way to do it, IMHO.

So, if you've had your stuff run through the peer review mill, what's your suggestion for it?

(for the record, yes, I've gone through peer review several times for my own publications. And yep, one of my papers was rejected from JAMA but published in Texas Journal of Epidemiology after some revisions that made it a better paper thanks to the reviewer's comments.)

Originally posted by Klassified
reply to post by circlemaker

 

Regardless of the human shortcomings, the bigger problem I personally see, is the portrayal of a review system as being equally scientific in methodology to the material it is reviewing. Which is obviously not the case. From my perspective, the journals themselves should shoulder the responsibility of making the reader aware of the process of review, and its innate limitations.

Erm... they do. It's the "letters to the editor" section (no, I'm not kidding.) And I've seen some real scream-fests in them as well as at conferences (that's the other place where the audience really will jump on problematic papers.)

reply to post by Byrd

Thank you Byrd. I was hoping more folks with intimate knowledge of this process would chime in. Mostly in the interest of fairness and balance.

Have you ever gotten a paper in (or through) a peer review process? If so, I'm wondering what your thoughts are on how YOU would improve it (knowing that each journal gets hundreds of papers submitted to it for publication.)

I have not been through the process. Although the last time I had a hypothesis examined by a "peer", she found all kinds of errors. Some she was right on, some I felt were biased personal opinions straight from the consensus point of view. The part in bold. This is one of the cruxes of the problem, is it not? And one of the reasons I made my statement in the OP that:

In all fairness to those involved in the review process, I must also mention that many of these academics are not paid for their time reviewing other peoples work. They can easily become inundated with requests for reviews.

Add to that, every bit of time they spend reviewing other peoples work, is less time they have to spend on their own interests and work.

They can't publish them all, and picking the ones to publish based on "did the authors attend important schools" or "what else have the authors published" is a really lousy way to do it, IMHO.

Yes it is. But when you're looking at days worth of reviewing time. I can only imagine someone thinking "how can I weed out some of this, and lessen this mountain staring me in the face?"

Everything I've read, which has been quite a lot now, tells me there isn't an easy fix to the process. The best that can be hoped for, in my opinion only, is a refinement of the best parts of the process. Not that there aren't problems with that too.

First off, the reviewer(s) should be compensated for their time.

Second, trimming down the subjective factor. I realize it's always going to be there to some degree, but I think there are ways to lessen its effect...
Each paper on it's own merits. The reviewer doesn't need to know the authors name, or their background. Nor do they need to know the publication it is being reviewed for. Which rules out friends and aquaintances of the editor, and prejudiced reviews based on the bias of the publication itself.

Third. Reviewers should have guidelines established, and personal opinions should be noted as such. I would like to say reviewers should be certified by a board. But I think we both know that won't help the process. Those who overseer my field have done a lousy job of establishing a certification process that ensures those certified know what the hell they are doing when turned loose on the public.

The above ideas are brief notes I made to myself while reading. I don't have all the answers either. To be honest, I'm not so sure fixing the process is the whole answer. We improve what we can, and stop having unrealistic expectations of the reviewer, and the "process".

The problem is, so many who are not scientists, or academics, like myself (I'm self-employed, and I consult, as well as fix problems for consumers and businesses relating to computers and Audio-visual.) rely on peer review as something akin to the "good housekeeping" seal of approval. However, the more I read, and talk to a few folks I know, the more I have found that isn't what it is at all.

Too much has been made of peer review to the public. And not enough measures have been taken among academia to insure that good theories don't slip and fall through the cracks, because of the faults and limitations of the review process.

Erm... they do. It's the "letters to the editor" section (no, I'm not kidding.) And I've seen some real scream-fests in them as well as at conferences (that's the other place where the audience really will jump on problematic papers.)

Excellent point, and something I hadn't even considered. Evidently, those on ATS who see peer-review as the gospel, don't read those letters, or go to conferences.

I sincerely appreciate your input into this thread. I know it was lengthy, and not the most cohesive, but I'm glad you read it, and added your thoughts. Looking forward to more if you're so inclined. If not,