From stem cell fraud to acupuncture, peer review can save us from ourselves

The publication of a paper in Nature describing “stimulus-triggered acquisition of pluripotency” (STAP) caused an international sensation. It was potentially the Holy Grail of stem cell research - how to create stem cells with a benchtop technique that seemed simple, reliable and cheap compared to other ways that were known to researchers.

The researcher involved was a scientific wunderkind named Haruko Obokata. Obokata had published this revolutionary research in the most prestigious peer-reviewed journal of all only 3 years after completing her PhD in cell biology. It seemed like a fairytale rise.

It didn’t take long for cracks to appear in the tale. Excited scientists all over the world tried and failed to replicate the results. If the breakthrough had truly been all it promised, and more, there would be rapid replication by other labs within weeks, and frenzied efforts to improve and enlarge upon the new knowledge.

Sadly, none of that happened.

Weeks ticked by and nobody could get the same results. Irregularities in the paper were noted and explanations were demanded. Some of Obokata’s colleagues in the field suggested the paper should be retracted until it could be replicated. Retraction of a paper is an unusual event in serious science, and when done tends to call into question the academic capability of the authors of the paper and the peer review processes of the journal involved. It is sometimes done for ethical reasons, but retraction due to fraud or misconduct is rare and devastating to the careers of those involved. Some of the retraction debate is well summarized here.

Then today comes the news that Obokata has been found by her research institute to have manipulated the results of her experiments to a degree that amounts to faking them. This is sad and disappointing news, but in many ways does not come as a surprise. As any science-watcher knows, extraordinary claims must be backed by extraordinary evidence, and error or misconduct by one group of researchers is always more likely than all other researchers in the same field being mistaken or ignorant.

Scientific scandals are not new, and tend to involve cutting-edge areas. The cold fusion debacle springs to mind. In that case, Pons and Fleischmann described a very simple method which had supposedly been missed by other researchers for creating almost unlimited nuclear energy at room temperature. Unlike Obokata, they didn’t bother trying to publish in a prestigious journal but went straight to a press conference. Replication was also the rock upon which cold fusion was dashed, as doubts were raised in both practical and theoretical spheres in the weeks that followed.

The process of peer review is a massive part of the reason that the scientific method is so dependably credible. Having to convince a determinedly sceptical group of colleagues who will hold you to pre-agreed high standards in your argument produces convincing work. It seems extraordinary that a possibly concocted paper could have made it all the way through to publication in Nature but even the best reviewers can lose objectivity when confronted by such a remarkable claim. We all desperately want good things to be true.

Purveyors of poor quality science have an easier time of it if they can avoid peer review. They might do this by going straight to the media with an engaging story that they hope will be reported uncritically by busy journalists who won’t fact-check with experts. They might present the data as a poster at a conference and not bother submitting to a journal that has robust peer-review. They may even publish a very conservative, minimally controversial paper and then make more sweeping claims in the press release announcing their publication. I’ve seen all these used to promote research that goes on to be shredded by peer review and justly ignored by science-based healthcare providers.

The recent story in the Australian media about acupuncture used for acute pain relief in emergency departments is a good example of research which seems to have avoided peer review and made a bid to capture the popular consciousness. The headline reports that the study has demonstrated a positive result in favour of acupuncture. Does it really? We have no idea, because in the third paragraph of the story, we learn that the study hasn’t even been written up for publication yet, let alone peer reviewed. In my opinion, that should have been the point when a responsible journalist would have turned off the recorder, put away her pen and politely requested that the interviewee come back when they actually have a story. If it hasn’t been peer-reviewed, it’s just hearsay and third-hand gossip as far as serious clinicians and academics are concerned.

The acupuncture study in the article is known about in rough terms from the previous publication of the proposed protocol in 2011. The study design has the potential for numerous serious flaws, any one of which could reduce its real-world usefulness to zero. Dr Rachael Dunlop has neatly summarized the possible methodological flaws in a blog post here. It’s academically irresponsible to discuss research publicly that has not been through peer review. That’s not just my heartfelt opinion, it’s in the Australian Code for the Responsible Conduct of Research as well.

Aggressive, robust peer review will always be applied to things that look too good to be true. Nobel Prizes in Medicine are hard-earned, because to win one you have to produce research that is not only game-changing, but which stands up to replication and peer review. Quality research breeds new knowledge, explains previously misunderstood problems and creates whole new industries. Pseudoscience is sterile and leads only to repetition and elaboration of delusions. The stakes are high, which is why the knowledge at the root of new discoveries has to be demonstrably sound.

If you have described reality accurately, you will be rewarded generously, both financially and socially. If you have not, your research will be met with a response that is red in tooth and claw. The public image of stem cell research may take a hit over the STAP fakery, but it shouldn’t. The whole saga shows the scientific community doing what it does every day of the year. There are hundreds of other reputable researchers who will simply pick up where they left off, incrementally advancing the collective knowledge of the field. Their work will go on, just as the careers of those who have misled them will be over. Long may it remain so.