Originally posted by Klassified
I have not been through the process. Although the last time I had a hypothesis examined by a "peer", she found all kinds of errors.

Did you rework the hypothesis after that? That's what we have to do. I'm currently doing my PhD dissertation, and it's a hair-puller. I gave the committee MY research design (and grand ideas) and they came after it with chainsaws and hatchets. I moaned and wailed because they whacked down my Grand Idea into something that I thought was trivial -- BUT -- when I took their approach, I found that what they were pointing me at is a nice little foundational concept.

Then they told me that this would be one of the foundations of my "Life's Work." (oh sure. No pressure THERE, right??)

But when you're looking at days worth of reviewing time. I can only imagine someone thinking "how can I weed out some of this, and lessen this mountain staring me in the face?"

What'll kill a paper first is Bad Reseach Design (or if it's not explained well.) After that, it's Bad Statistical Analysis.

Research Design transparency is CRITICAL. If we can't tell how you did the research, we can't prove that you really have something good, and we can't confirm what you did by replicating your research. In a recent paper presented to the City of Dallas by the Centers for Disease Control (their assessment of the West Nile Virus aerial spraying), the CDC went over how they gathered and measured data in very great detail before they gave their conclusions. While the "science-y stuff" probably put everyone to sleep, it told me that the CDC didn't just show up and blow smoke at everyone. It confirmed some of the data I had gathered independently and highlighted the real problems cities have in dealing with mosquitoes (they don't know how to collect data or what to do with it when they get it.)

The reviewer doesn't need to know the authors name, or their background.

When a paper is passed for review, the names of the authors and their affiliations NEVER show up anywhere on the review copies.

Reviewers should have guidelines established, and personal opinions should be noted as such. I would like to say reviewers should be certified by a board.

Magazines do have rubrics in place, yes, and where appropriate, most reviewers are board certified. Some disciplines don't have boards, though.

The problem is, so many who are not scientists, or academics, like myself (I'm self-employed, and I consult, as well as fix problems for consumers and businesses relating to computers and Audio-visual.) rely on peer review as something akin to the "good housekeeping" seal of approval. However, the more I read, and talk to a few folks I know, the more I have found that isn't what it is at all.

Perhaps it's just that the failures are so loudly talked about, and the millions (literally) of good papers are ignored?

Too much has been made of peer review to the public. And not enough measures have been taken among academia to insure that good theories don't slip and fall through the cracks, because of the faults and limitations of the review process.

How do you know they are "good theories"?

Seriously.

Look around the board at all the theories you see here. None of them would get into any journals (including PLOS, and that's got VERY low standards of "peer review" (I hesitate to call it that, frankly.)) There's no methodology, there's no background, it's just "wow! I got a great idea!"

My "great ideas" (I haz them) have to be well documented and explained before they can get into a paper. When I don't have enough data (example: the "heat island effect" on ponds) then I will write an article for a magazine about the topic (because I'm retired, and all the science I do is funded solely by my not-very-glorious pension and by social security (I'm older than dirt.))

I have seen a lot of self-made mathematicians on the Internet who explain how they've solved all sorts of grand mathematical things. I'm married to a mathematician, so have gotten Math By Osmosis, and the people who are complaining that "academics are ignoring me" or "the government is suppressing this" would fail your average senior level college math course. Most of them would fail a junior level math course, and some of them couldn't pass a high school algebra course... yet they're telling us they've solved everything with math.

So... how do you know good ideas are suppressed?

Evidently, those on ATS who see peer-review as the gospel, don't read those letters, or go to conferences.

True. We talk about it, but it's not as exciting as mentioning Kim Kardassian's newest boyfriend or bathing suit. There's a zillion conferences with exciting stuff and journals and all, but the latest moral outrage gets a lot more attention.