November 30, 2009

Peer-Review--a magical spell...





Shannon Love, No One Peer-Reviews Scientific Software:

...Too often these days when people want to use a scientific study to bolster a political position, they utter the phrase, "It was peer reviewed" like a magical spell to shut off any criticism of a paper's findings.

Worse, the concept of "peer review" is increasingly being treated in the popular discourse as synonymous with "the findings were reproduced and proven beyond a shadow of a doubt."

This is never what peer review was intended to accomplish. Peer review functions largely to catch trivial mistakes and to filter out the loons. It does not confirm or refute a paper's findings....

[...]

....Custom-written scientific programs are much, much larger and much, much more complex than any spreadsheet. It would take a huge amount of time for a peer reviewer to go through the code line by line to see if the software had any faults. Normally, peer reviewers work for only a token payment and they work in isolation. They don't have the time or resources to actually check out a complex piece of software. Further, there is no guarantee that a peer reviewer in a particular field is competent to judge software. That is like assuming that a biologist who understands everything about the Humboldt squid can also rebuild any automotive transmission.

The practical inability of peer reviewers to verify scientific software doesn't mean much in reality, because scientific institutions never even developed the standard that experimenters had to make the code for their software available to reviewers in the first place!

This raises a troubling question: When scientists tell the public that a scientific study that used a large, custom-written piece of software has been "peer reviewed" does that mean the study faced the same level of peer scrutiny as did a study that used more traditional hardware instruments and procedures?

Clearly not.

Scientists have let a massive flaw slowly creep into the scientific review system as they have ignored the gradually increasing significance and complexity of computer software...
Posted by John Weidner at November 30, 2009 12:13 PM
Weblog by John Weidner