Sections

Services

Information

UK United Kingdom

Science is in a reproducibility crisis – how do we resolve it?

Over the past few years, there has been a growing awareness that many experimentally established “facts” don’t seem to hold up to repeated investigation. This was highlighted in a 2010 article in the New…

Scientists are often untrained in methods to make their research replicable. Pulpolux !!!

Over the past few years, there has been a growing awareness that many experimentally established “facts” don’t seem to hold up to repeated investigation.

This was highlighted in a 2010 article in the New Yorker entitled The Truth Wears Off and since then, there have been many popular press accounts of different aspects of science’s current reproducibility crisis.

These include an exposé of the increasing number of retractions by scientific journals and damning demonstrations of failures to replicate high profile studies.

Articles in recent days have discussed how the majority of scientists might be more interested in funding and fame than “truth” and are becoming increasingly reluctant to share unpublished details of their work.

So why exactly is science in such a crisis - and where do we start fixing it?

What caused the reproducibility crisis?

Flood

In each discipline, there have been different triggers. In psychology, it was an unreplicable study about extrasensory perception (ESP); in medicine, it was unreplicable cancer studies.

Behind these (somewhat arbitrary) triggers, however, are the same underlying causes: a combination of mechanised reporting of statistical results and publication bias towards “statistically significant” results.

Problems with traditional significance testing and publication bias have already been addressed on The Conversation.

So is the crisis a result of scientific fraud?

Not really. Well, maybe a bit. The number of known cases of outright fraud is very low. But what we might consider softer fraud — or “undisclosed flexibility” in data collection — is well documented and appears to be very widespread.

There can be little doubt that the “publish or perish” research environment fuels this fire. Funding bodies and academic journals that value “novelty” over replication deserve blame too.

While no-one knows the true level of undetected scientific fraud, the best way to deal with this problem is to increase the number of replication studies.

Articulate Matter

How do we fix it?

Some initiatives are already underway. In psychology, there’s the Reproducibility Project, which has previously been covered by The Conversation.

In biomedicine, there’s the Reproducibility Initiative. It’s backed by the Science Exchange, the journal PLOS ONE, Figshare, and Mendeley. It will initially be accepting 40 to 50 studies for replication with the results of the studies to be published in PLOS ONE.

There are also various other proposals such as

The proposals and initiatives mentioned above draw attention to improving methodological protocols, and require a more thoughtful approach to statistical reporting practises.

We might broadly consider these to be issues of researcher integrity. But instruction in research ethics alone is unlikely to be sufficient. Enabling others to replicate studies published across all areas of science will also require changes in the way scientists prepare, submit and peer review journal articles, as well as changes in how science is funded.

This points to a new way of doing science, which can loosely be called “open science”. This could include new practices such as open peer-review, and open notebook science and there are already platforms being developed to support these approaches.

… such as this one: Open Science Framework.

Publishing computer source code and supporting data sets with academic articles will be an important change in making research more reproducible. This is a pressing issue with the increasing use of large data sets, computer simulation and sophisticated statistical analysis across many areas of science.

Although some fields of science have developed further in this direction than others, there has recently been a proliferation of services to support scientists publishing data and source code. This includes services such as Figshare, RunMyCode and the Dataverse Network.

In addition there is currently a push to give researchers a greater incentive to publish their data by making scientific datasets citable contributions to the scholarly record and with associated journals such as GigaScience and Earth System Science Data.

Thomas Leuthard

While opportunities to share raw data associated with a journal publication are growing, currently only around 9% of articles do so.

Before we assume this is a moral failing on the part of the authors of these articles, we should consider that there are many practical hurdles involved. In many areas of science, researchers are not trained in data curation, version control of source code or other methodologies required for research to be replicable.

Meeting the challenge

Data sharing and other procedures outlined here can be time-consuming, and currently provide little academic reward. Instruction in these skills will eventually need to become part of mainstream science education.

Methodology and statistics courses are one obvious place for them to find a home. The ethics of the reproducibility and open science movements are hard to dispute, but success will depend on how well we rise to meet associated practical and pedagogic challenges.

Join the conversation

32 Comments sorted by

  1. John Newton

    Author Journalist

    I wonder whether the authors of this article would like to comment on the following statement by Ignacio Chapela, a Mexican biologist and one of the authors of a paper on the genetic contamination of the traditional Mexican corn crop published in Nature who subsequently lost his position at Berkley University and had his paper retracted after a number of complaints by biotech stooges (later uncovered).

    “There was a time” Chapela said “when science and the university loudly proclaimed their independence from governmental, military and industrial institutions. That’s over, not only because scientists depend on industry to survive, but because they themselves are part of industry.”

    Is it also a factor that independence of science and industry is over – because many scientists are part of industry?

    report
    1. Gavin Moodie
      Gavin Moodie is a Friend of The Conversation.

      Adjunct professor at RMIT University

      In reply to John Newton

      I suggest that scientists' engagement with industry is very patchy. In Australia business expenditure on research and development is lower than other OECD countries and is falling. Most Australian science is supported directly and indirectly by government grants.

      However, industry funding is significant and arguably too influential in some fields: pharmaceuticals and medical products are the most commonly given examples.

      report
    2. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to John Newton

      Maybe. But in my opinion, it's only half the story. For example, the situation in psychology appears to have nothing to do with industry. Industry involvement is certainly not a prerequisite to ending up in this position.

      report
    3. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to John Newton

      Sorry, I posted my reply on the wrong comment. This reply will appear below as well, but it was written for you (John Newton)!

      Maybe. But in my opinion, it's only half the story. For example, the situation in psychology appears to have nothing to do with industry. Industry involvement is certainly not a prerequisite to ending up in this position.

      report
  2. Gavin Moodie
    Gavin Moodie is a Friend of The Conversation.

    Adjunct professor at RMIT University

    The authors offer no evidence to establish that science is in crisis. 'Many popular press accounts of different aspects of science’s current reproducibility crisis' is just journalists recycling a few cases, some of them poorly established. Retractions have increased over the last few years, but the number remains minuscule, surely <1% of papers. The media article cited as evidence that 'most scientists may be more interested in funding and fame than 'truth'' merely reports anecdotes.

    The authors' sensational claim seems to be the very attention seeking behaviour they criticise in scientists.

    If a paper reports apparently useful results they will be used in subsequent work. If the subsequent work does not succeed the dodgy results will be discarded and scientists will move on to more promising work. That seems to me just as effective and more efficient than reproducing all work in the way in which it is first reported.

    report
    1. Simon Brown
      Simon Brown is a Friend of The Conversation.

      Professor in Emergency Medicine, University of Western Australia

      In reply to Gavin Moodie

      I disagree with Gavin - I think that the authors are absolutely spot on. Unfortunately (at least in medicine) there is often huge resistance to replication studies -especially once a drug is registered- often via manipulated by pharmaceutical companies, including via their high profile "key opinion leaders" in clinical practice. Academic independence is fading due to increasing dependence on industry support for research. Add to this the resistance to making data available and hiding negative studies. All very real and well-documented. The British Medical Journal is leading the fight in this regard - worth a read.

      report
    2. Gavin Moodie
      Gavin Moodie is a Friend of The Conversation.

      Adjunct professor at RMIT University

      In reply to Simon Brown

      All that may be true, but it does not amount to crisis in science as the authors claim. It is a problem in pharmacology, which as important as it is, is only a small part of science.

      report
    3. Simon Brown
      Simon Brown is a Friend of The Conversation.

      Professor in Emergency Medicine, University of Western Australia

      In reply to Gavin Moodie

      All of that *is* true Gavin, and reflects similar problems in other areas of science, particularly those dealing with a combination of complexity (eg biological and ecological systems) and high financial stakes. The increasingly competitive funding environment, the need to "publish or perish" and to make one's "academic mark", and industry involvement at various levels (study design, interpretation and promotion) are having a not so subtle impact. When dealing with complex systems, replication studies and a constant search for potential contaminating bias is incredibly important. It's not just medicine.

      report
    4. Gavin Moodie
      Gavin Moodie is a Friend of The Conversation.

      Adjunct professor at RMIT University

      In reply to Simon Brown

      Yet we can all think of counter examples. Most prominently, a lot of environmental research is counter to the immediate interests of huge vested interests, yet it continues and expands.

      The article and some posters are making very big claims. I will not accept those without evidence. That is, I ask the authors to meet the same standards that they accuse all, most, much or some scientists of breaching.

      report
    5. Paul Savage

      logged in via LinkedIn

      In reply to Gavin Moodie

      I agree wholeheartedly Gavin. Both of your comments are right on the money. The overwhelming majority of articles that appear on Retraction Watch are clinical biology, pharmacology, and other medical sciences (setting aside plagiarism and crudely fabricated data, which probably equally plagues all branches of science).

      My hypothesis is that this is because biology research, generally, tends to be harder to reproduce due to the inherent complexity of variables, and the subjectivity in measuring…

      Read more
    6. Sue Ieraci

      Public hospital clinician

      In reply to Gavin Moodie

      I don't see any ''crisis'' as being in the science, but perhaps in the public perception - driven by mass media reporting - of the significance of individual studies.

      This is how I see it:

      When all original research was published on paper in medical journals, it was mostly accessed by a readership with expertise in that area, and knowledge of the entire body of previous research in that area. So, a study could be discussed, evaluated, accepted or rejected in the light of what was already known…

      Read more
    7. Gavin Moodie
      Gavin Moodie is a Friend of The Conversation.

      Adjunct professor at RMIT University

      In reply to Sue Ieraci

      In my observation journalists are sent a copy of the full paper they are asked to report as an attachment to the press release. Some journalists seek comments on the paper from people they identify as experts in the field and often send the paper. To get a comment published the expert normally has to respond quite quickly, which for me usually means dropping everything immediately to read the paper and formulate a response.

      But even were all those steps followed fully there is still much scope for inaccurate reporting and understanding. Over the last few years the media have telescoped the time between receiving information on a paper and reporting it. But other changes in the media may allow that time to increase to allow at least the experts to respond with more care.

      The Conversation is a laudable exception to the media's standard practice, of course, which is presumably why we are here. Hopefully its example is strong enough to influence other media.

      report
    8. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Gavin Moodie

      The reproducibility crisis--as we see it and describe it--is not about any single result not being reproduced. It's about systematically discouraging replication (by funding and publishing only 'novel' studies) and publication bias (which means only statistically significant studies make it to the literature). Our references to popular media articles were simply an attempt to show how widespread and well known the problem is. Unfortunately, they seem to have been a controversial distraction from our actual story!

      report
    9. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Simon Brown

      Thanks. Resistance to replication studies and the inability to get negative results published plagues psychology too, and some areas of biological science. In those cases, there isn't even industry pressure (usually). I think psychology, in particular, is reaching a tipping point, and that there will be major institutional changes soon.

      report
    10. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Sue Ieraci

      Our concern is not with studies which are replicated and then rejected. It's also not with how the media portray such outcomes. It's about practices within science that discourage replication studies and biases in what is published (i.e., only statistically significant studies, not negative results).

      report
    11. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Paul Savage

      I do agree that the problem is related to measurement and statistical analysis. But the interaction is complex. In disciplines where there are fewer 'natural' or 'universal' units of measurement, statistical significance testing may entrench itself. Psychology is a perfect example. And where statistical significance testing dominates, publication bias is almost always found. And where there is publication bias, there are likely to reproducibility problems...

      Experiments that don't replicate because of differences in culture, time or geography are *not* part of the problem here. Those are examples of moderating variables. The problem here is false positive results--that enter the literature at an alarming rate when publication bias exists--and hang around because of a research climate that discourages replication.

      report
    12. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Gavin Moodie

      I've written a general comment that addresses this, and specifically our use of the term crisis. And I'm sure we haven't accused anyone of attention seeking behaviour by simply pointing out related popular press articles.

      report
  3. Bill Skinner

    Research Professor at University of South Australia

    "So is the crisis a result of scientific fraud?
    Not really. Well, maybe a bit. The number of known cases of outright fraud is very low."

    There is a reason that the "outright" fraud statistics are low. Many journal article retractions occur before final publication (even before online availability) and many (I'm sure) retractions have reasons given such as "we spotted a mistake in our experimental procedure" or somesuch, thus covering fraudulent behaviour with a plausible, human error. If serious…

    Read more
    1. Bill Skinner

      Research Professor at University of South Australia

      In reply to Fiona Fidler

      No worries Fiona. The growing emphasis on open access to journal articles has included a similar debate on open access to research data. There may be an argument for the latter, particularly for publicly-funded research. However it may be a double-edged sword.

      Positively, a wider peer review of data will make it easier to confirm the result and conclusion of the work in question (in part, addressing the reproducibility aspect of your article) and making authors think twice about publishing flawed…

      Read more
  4. Peter Bowdidge

    logged in via LinkedIn

    I was under the impression that science advanced through reproduction - or rejection - of results. The great strength of the scientific method is that for every published finding, a horde of researchers wait in the wings looking to replicate or disprove it.

    A (purported) higher frequency of published hypotheses later disproven isn't a crisis; it's how science works.

    report
    1. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Peter Bowdidge

      I completely agree that science *should* work that way! The problem is that replications are so rarely done, and that they have been systematically discouraged by funding and publishing institutions. There has not been "a horde of researchers waiting in the wings" because replication work has brought so little reward.

      report
  5. Mike Brisco

    Scientist at Flinders University of South Australia

    The writers should have done a bit more homework, before concluding the whole of science is in crisis.

    The claim, comes from an article in Salon.com, which refers to an commentary in Nature, (a commentary, note, not a research paper) from C Glenn Begley, a consultant to a private corporation Amgen, and Lee Ellis. ( Nature 483, 531–533 doi:10.1038/483531a).

    Amgen are a private pharmeceutical company, developing new drugs to treat cancer - a commercially risky endeavour. There is a research…

    Read more
  6. Mike Brisco

    Scientist at Flinders University of South Australia

    Sorry, correction due. In my over-long post, I said Begley and Amgen claimed they could only repeat 11 of 53 papers.

    In fact they said, they could only repeat 6 of 53. which was 11%. I had mistaken the percentage, for the number.

    Sorry for the inaccuracy, don't think it affects my main points. .

    report
  7. Fiona Fidler

    Senior Research Fellow at University of Melbourne

    Thanks everyone for your comments, criticisms and suggestions. We welcome them all! Let me address a couple of points directly.

    First, we employ the term ‘crisis’ to refer to the point precipitates a major shift, or in Thomas Kuhn’s words, a ‘scientific revolution’. Whilst open science wouldn’t technically consitute a paradigm shift, the initiatives we discuss would be a *big* change in the way science operates. To us, things appear to have reached a tipping point. Some comments suggest we’ve…

    Read more
    1. Gavin Moodie
      Gavin Moodie is a Friend of The Conversation.

      Adjunct professor at RMIT University

      In reply to Fiona Fidler

      'Crisis' may have a technical meaning in the philosophy of science, altho the support for using this term refers to ‘scientific revolution’, 'paradigm shift' and 'tipping point' rather than any specialised use of 'crisis' in the philosophy of science literature. But the Conversation is not part of the philosophy of science literature. Since the article didn't introduce any specialised meaning of 'crisis' readers appropriately understood it to have its common meaning.

      The difficulty with using the term is not only that it is wrong, but that it also gives succour to those who would deny science rather than improve or even reform it.

      report
    2. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Gavin Moodie

      Anticipate how your work might be misinterpreted and guard against it--that's not bad advice!

      And for the record, we love science and definitely want it improved. We are not trying to deny or destroy anything, or facilitate those who are.

      Still, message received!

      report
  8. Alex Cannara

    logged in via LinkedIn

    An interesting statistic published a year or so ago by the major journal AAAS Science showed that rejections are proportional to submissions and published articles. In other words,, bigger journals are no better than smaller ones in selecting good papers.

    And, realistically, the rejection rates are very high in established, reputable journals, so what's going may simply be on the authoring side -- publish or perish.

    Our research institutions, particularly our universities that depend greatly…

    Read more
  9. Jack Heinemann

    Professor of Molecular Biology and Genetics at University of Canterbury

    Fascinating initiative for academic science. Thanks for the article.
    It strikes me though that most science that makes immediate impact on most people is that which is done by industry for the government regulator. This in my experience is the least open about underlying assumptions and methodology, and the least likely to provide the necessary materials for replication that is necessary for verifying both claims of efficacy and claims of safety. Shouldn't we begin with this 'gray' literature?

    report
    1. Fiona Fidler

      Senior Research Fellow at University of Melbourne

      In reply to Jack Heinemann

      I'm sure you're right about the gray literature being less open. I suspect the pressures are different (no publication bias for example, but probably stronger external pressures). Let's hope the reforms filter down!

      report
  10. Philip Impey

    Architect+Urban Designer

    I experimental reproducibility was a key to a scientific fact (and therefore, truth) being established, Darwin's theory of biological evolution would fail. So too would the theory of AGW.

    report