tag:theconversation.com,2011:/global/topics/replicability-8238/articlesReplicability – The Conversation2018-09-30T15:14:39Ztag:theconversation.com,2011:article/1013032018-09-30T15:14:39Z2018-09-30T15:14:39ZOpening up the future of psychedelic science<figure><img src="https://images.theconversation.com/files/237197/original/file-20180919-158228-1p66ki0.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">There is a growing research literature suggesting psychedelics hold incredible promise for treating mental health ailments ranging from depression and anxiety to PTSD.</span> <span class="attribution"><span class="source">(Shutterstock)</span></span></figcaption></figure><p>Attempts to replicate classical scientific studies have been failing. These alarming failures have hit psychology, the life sciences and other fields, calling major findings into question. <a href="https://doi.org/10.1038/533452a">Scientists agree</a>: questionable research practices are <a href="https://doi.org/10.1371/journal.pmed.0020124">rife in many disciplines</a>. </p>
<p>We are two psychology PhD students with experience researching mindfulness. We echo the <a href="https://www.scientificamerican.com/article/wheres-the-proof-that-mindfulness-meditation-works1/">scathing criticisms levelled against poorly designed studies within the field of mindfulness research</a>. </p>
<p>As science is only trustworthy when consistent, we need to make sure future work can be replicated. As such, we have decided to spread the word about proper open scientific practice. This is especially important in the nascent interdisciplinary field of psychedelic science, in which we are now conducting research into the practise of “microdosing” substances like LSD (lysergic acid diethylamide) and “magic” mushrooms (psilocybin).</p>
<p>There is a growing research literature suggesting psychedelics hold <a href="https://theconversation.com/the-real-promise-of-lsd-mdma-and-mushrooms-for-medical-science-100579">incredible promise</a> for treating mental health ailments ranging from <a href="https://www.businessinsider.com/psychedelics-trip-therapy-2018-1">depression and anxiety</a> to <a href="https://www.cnn.com/2018/05/01/health/mdma-psychotherapy-ptsd-study/index.html">PTSD</a>. But how do we know for sure?</p>
<p>The way forward for psychedelics is through “open science.” Researchers should pre-register their plans and share their data, <a href="https://osf.io/g5cwy/">as we have in our own research</a>. </p>
<h2>Science must be consistent</h2>
<p>Science needs to have a strong foundation, but right now a lot of the research isn’t replicating. In 2015, the <a href="http://science.sciencemag.org/content/349/6251/aac4716">Reproducibility Project</a> tried to replicate 100 high quality psychology findings. Only <a href="https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248">39 of these findings were replicated</a> — that’s less than half! </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/237196/original/file-20180919-158237-i55cwn.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/237196/original/file-20180919-158237-i55cwn.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/237196/original/file-20180919-158237-i55cwn.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/237196/original/file-20180919-158237-i55cwn.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/237196/original/file-20180919-158237-i55cwn.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/237196/original/file-20180919-158237-i55cwn.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/237196/original/file-20180919-158237-i55cwn.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Mindfulness research lacks active control groups and has inconsistent definitions of mindfulness itself.</span>
<span class="attribution"><span class="source">(Shutterstock)</span></span>
</figcaption>
</figure>
<p>This phenomenon isn’t limited to psychology: findings from disciplines such as biology, medicine and chemistry can be hard to believe. For example, <a href="https://retractionwatch.com/2017/07/31/nearly-500-researchers-guilty-misconduct-says-chinese-govt-investigation/">almost 500 authors</a> were found guilty of misconduct by the Chinese government last year, <a href="https://retractionwatch.com/2018/09/04/cancer-journals-retract-10-papers-flag-8-more-and-apologize-for-the-delay/#more-70872">several cancer research papers</a> have been retracted recently and a recent report indicated that as much as <a href="https://pubs.acs.org/doi/pdf/10.1021/acs.jchemed.7b00907">80 per cent of chemists</a> have trouble replicating findings from the literature.</p>
<p><a href="https://theconversation.com/ca/topics/replicability-8238">Several great pieces</a> on <em>The Conversation</em> have tackled this issue so there is lots to review if replicability is new to you. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/why-students-are-the-answer-to-psychologys-replication-crisis-90286">Why students are the answer to psychology's replication crisis</a>
</strong>
</em>
</p>
<hr>
<p>Psychedelic research is an interdisciplinary field combining psychology, biology and medicine and so is an especially important field in which to implement “open science.” </p>
<h2>Open science <strong>= rigorous science</strong></h2>
<p>For <a href="https://www.sagepub.com/sites/default/files/upm-binaries/40007_Chapter8.pdf">statistics in science to work properly</a>, scientists need to guarantee that what they have studied is no more and no less than what they intended to study. </p>
<p>Instead of hiding inconvenient results or adding unplanned research conditions, scientists can use open science to demonstrate their integrity. Open science involves pre-registering hypotheses before doing research, and publishing the entire data set once the research is done. </p>
<p><a href="https://osf.io">Pre-registration happens online</a>. The content of the registration is locked and time stamped, then kept confidential until a set date, when it is released for the public to see. This is done so that the researcher can show they did exactly what they planned to do, which is how we all learned we are supposed to do science. Pre-registration is not even difficult, but researchers need to <a href="https://www.youtube.com/watch?v=kzUtpDBo8wk&index=1&list=PLMOU-iLiJIc0amNVabGXJ0liKwIwxqkO8">learn how to do it</a> and adjust.</p>
<p>Once the study has been published, the data set can be made public. This way, the entire scientific community can examine the data, serving at least two purposes. First, the scientific community can verify that the data supports the conclusions made in the study, ensuring no mistakes were made. Second, other scientists can explore for new patterns in the data to create new hypotheses for new studies, moving science forward faster. </p>
<p>Making the data public makes scientists publicly accountable, and is good for the scientific community at large.</p>
<h2>Co-operation over competition</h2>
<p>So far, most psychedelic research has not been pre-registered, which means it should be considered exploratory and, unfortunately, inconclusive. Some findings may have been by chance rather than clearly caused by the substances used, and these findings need to be replicated by independent labs to ensure they hold up. </p>
<p>A recent call for “<a href="http://chacruna.net/cooperation-over-competition-statement-on-open-science-for-psychedelic-medicines-and-practices/">Cooperation Over Competition</a>” has been made, but its impact remains to be seen. For now, we take the results on psychedelics that scientists have found on faith.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/237226/original/file-20180919-143281-1qtxrlg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/237226/original/file-20180919-143281-1qtxrlg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/237226/original/file-20180919-143281-1qtxrlg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/237226/original/file-20180919-143281-1qtxrlg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/237226/original/file-20180919-143281-1qtxrlg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/237226/original/file-20180919-143281-1qtxrlg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/237226/original/file-20180919-143281-1qtxrlg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">The way forward is for scientists to share their plans and data.</span>
<span class="attribution"><span class="source">(Shutterstock)</span></span>
</figcaption>
</figure>
<p>Pre-registration is the only way to ensure psychedelic science is conducted with a high level of integrity. Psychedelic science is in its infancy, much as mindfulness research was some few decades ago. We must learn from past mistakes if we do not wish to see the same harsh criticisms levelled upon this field in the future.</p>
<p>This will improve and maintain public trust in the scientific endeavour, especially important for these storied substances. As public consumers of science, we should all be critical of new research and remember the <a href="https://www.wired.com/story/sagan-old-interview/">Sagan Standard</a>: “Extraordinary claims require extraordinary evidence.”</p><img src="https://counter.theconversation.com/content/101303/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>To know the real promise of psychedelic substances like LSD, mushrooms and MDMA, researchers must embrace the principles and practise of ‘open science.’Thomas Anderson, PhD student, University of TorontoRotem Petranker, PhD student in Clinical Psychology, York University, CanadaLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/946742018-05-21T07:45:59Z2018-05-21T07:45:59ZThere’s a crisis in psychology – here’s how technology could provide a solution<figure><img src="https://images.theconversation.com/files/219215/original/file-20180516-155616-1je7p3m.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Technology involving virtual reality could teach us a lot about the human psyche.</span> <span class="attribution"><span class="source"> (U.S. Air Force photo by Airman 1st Class Keith Holcomb)</span>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Psychologists obediently follow the same rules as other scientists. But their efforts haven’t yielded equivalent progress. In fact, in the last decade, psychologists have realised that some of their most intriguing findings <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0056515">are not reliable</a> – when other researchers try to repeat the same study, they don’t find the same results. </p>
<p>Many people refer to this as a <a href="https://theconversation.com/is-psychology-really-in-crisis-60869">replication crisis</a> in the field. But what is to blame for this problem and what can we do about it? In a new review, <a href="https://www.research.manchester.ac.uk/portal/en/publications/the-assessment-and-modeling-of-perceptual-control(9a95a0bf-e3ae-4b42-90f6-b579496c5c4e).html">published in the General Review of Psychology</a>, we describe a promising technological solution.</p>
<p>Most psychologists are convinced that the widespread misuse of statistics and <a href="http://www.sciencemag.org/news/2012/09/harvard-psychology-researcher-committed-fraud-us-investigation-concludes">poor research integrity</a> – a euphemism for cheating – are ultimately to blame for the crisis. So, removing bad practices should solve the problem. Yet this often <a href="http://journals.sagepub.com/doi/10.1177/1745691616652873">doesn’t work</a> – seriously undermining confidence in the reliability of psychology.</p>
<p>We are convinced that tightening the regulation of research won’t fix the crisis. Instead, we need to go back over the past century to a crucial wrong turn in psychology that happened because of a limit in the technology of the time. </p>
<figure class="align-right ">
<img alt="" src="https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=777&fit=crop&dpr=1 600w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=777&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=777&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=976&fit=crop&dpr=1 754w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=976&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=976&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">William James.</span>
<span class="attribution"><span class="source">Notman Studios</span></span>
</figcaption>
</figure>
<p>In the late 19th century, the American philosopher <a href="https://en.wikipedia.org/wiki/William_James">William James</a> argued that the essence of psychology is hidden purpose. He famously described the purposeful behaviour of a frog held under water in an inverted glass. Despite attempts by the experimenter to stop it, the frog eventually found its way up to the air in surprising ways. James argued that the frog’s purpose was to get to the surface and it did this in different ways each time.</p>
<p>But it isn’t easy to test hidden purpose reliably in humans. Most research in psychology relies on getting large numbers of participants to provide data. The researchers then measure correlations, or the effects of experimental manipulations, in these groups. This research began before the time of the modern computer, when the researcher could simply present a “stimulus” to a participant and measure the response. And this approach <a href="https://www.researchgate.net/profile/Richard_Marken/publication/232499797_You_Say_You_Had_a_Revolution_Methodological_Foundations_of_Closed-Loop_Psychology/links/574da87108ae8bc5d15bce5f/You-Say-You-Had-a-Revolution-Methodological-Foundations-of-Closed-Loop-Psychology.pdf">persists today</a>, making up the vast majority of studies in psychology. </p>
<p>Unfortunately, it’s not reliable. One recent series of “stimulus-response” studies were set up so that participants could respond to an image on a screen by either pushing or pulling a joystick. They were presented with either “negative” or “positive” images or words (stimuli). The researchers proposed that viewing a negative stimulus (such as an angry face) unconsciously <a href="https://pdfs.semanticscholar.org/faf1/b1ebc5b361b56b22e2b29bae5792b0d72dda.pdf">activates the muscles that extend the arm</a>. This is because that’s how we push something away if we are faced with it in real life. The initial studies supported this account – participants were quicker to respond to negative stimuli when the response was to push the lever away from them than when it was to pull it.</p>
<p>However, a huge <a href="https://www.tandfonline.com/doi/full/10.1080/02699931.2014.968096">review of over 68 attempts</a> to test for this effect in more than 3,000 participants showed that this effect was not consistently repeated. Importantly, in tasks that were designed so that pushing the lever actually made the stimulus get closer, the opposite effect was found – negative stimuli were now associated with the response of pulling the lever. </p>
<p>The authors concluded that participants were actually controlling their perceived distance from the negative image through whatever action they could (just like James’s frog). But the traditional experimental design was simply not set up to test this.</p>
<h2>Embracing VR</h2>
<p>In our recent article, we bring together the advances that researchers have made using an approach known as <a href="https://www.pctweb.org">perceptual control theory</a>. It continues where James left off, assuming the hidden purposes of living things, but it tests for them using a sophisticated approach. It typically relies on computing capacity to measure people’s activities in virtual environments, and to build a computer model of the psychological processes within the individual. </p>
<p>The technique is based on creating a situation where the participant can pursue a goal, for example controlling the distance from a negative image on a screen using a joystick. It then measures every change that goes on in the situation continuously (for example by making real-time videos) – including disturbances that get in the way of the person’s goal, such as changes in the experimental set up or physical obstacles. All this data is then used to build a computer model of how each participant is pursuing their goal. </p>
<p>You can then repeat the situation, using the computer model to predict what the individual will do, and constantly compare with what they are doing. If the model fails, you improve it until you’ve got a good match – creating a “<a href="https://link.springer.com/article/10.3758/s13414-017-1398-2">personal profile</a>” for each individual. This can then be tested for replication over repeated sessions. You can also combine data for many individuals to look at mean effects to work out what goals are generally relevant to a given situation.</p>
<h2>Replication … at last?</h2>
<p>The result of this approach is typically a robust model of the psychological processes involved in an activity – such as tracking a target on a screen. These models have been shown to repeat a high level of accuracy over and over again, typically <a href="https://www.ncbi.nlm.nih.gov/pubmed/2943855">showing correlations over 0.98</a> – a perfect correlation is 1.0. A correlation shows the association between two different variables (for example stimuli and response). This is currently virtually unheard of in traditional psychology research, where correlations of as low as 0.3 are regarded as “statistically significant”.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Don’t worry, there’s hope.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/spider-bites-hand-man-afraid-662092573?src=5gS8jB4fJhKhOdOTnux-mw-1-69">Goncharov_Artem/Shutterstock</a></span>
</figcaption>
</figure>
<p>You might think that modelling of this kind is only suitable for simple tasks, but a similar approach has been applied to many areas, including <a href="https://escholarship.org/uc/item/0ch4g2x6">food competition in animals</a>. This used frame-by-frame video analysis to show that a rat holding food in its mouth continually reorients its body to maximise the distance between its food and a competing animal’s mouth. </p>
<p>The same assumptions have informed treatments of spider phobia, helping to build tasks in which the participant can <a href="https://www.sciencedirect.com/science/article/abs/pii/S088761851630408X">control their distance from a spider</a> in a virtual corridor. Facing fears in this way is a treatment known as exposure therapy. However, it was previously unknown what level of control over the exposure works best. The study using this technique showed for the first time that people who have a higher degree of control over the exposure actually ended up avoiding spiders less after the experiment than those who had little control.</p>
<p>There are areas where it will be more challenging to use this technique – such as complex tasks involving memory and reasoning. Nevertheless, it could be easily applied in many areas.</p>
<p>The replication crisis has been the wake up call psychological science needed to think differently – now it is time to embrace the advances in technology that allow us to improve the field.</p><img src="https://counter.theconversation.com/content/94674/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Vyv Huddy is affiliated with the British Association for Behavioural and Cognitive Psychotherapies Control Theory Special Interest Group. </span></em></p><p class="fine-print"><em><span>Warren Mansell does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Studies in psychology cannot be replicated in the same way as in other fields. But technology could change that.Warren Mansell, Reader of Clinical Psychology, University of ManchesterVyv Huddy, Lecturer in Clinical Psychology, UCLLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/902862018-02-21T23:58:58Z2018-02-21T23:58:58ZWhy students are the answer to psychology’s replication crisis<figure><img src="https://images.theconversation.com/files/207382/original/file-20180221-132680-1hql3pa.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Shutterstock</span> </figcaption></figure><p>“Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital,” Aaron Levenstein, a <a href="http://www.nytimes.com/1986/07/05/obituaries/prof-aaron-levenstein.html">business professor at Baruch College</a>, once said. </p>
<p>I first heard a version of this quote in an undergraduate social psychology class in 2003. Nearly a decade and a half later, psychology is having a replication crisis — and the “bikini” is largely to blame. </p>
<p>Recently, more than 270 psychologists set out to <a href="http://science.sciencemag.org/content/349/6251/aac4716">repeat 100 experiments</a> to see if they could generate the same results. They successfully replicated only 39 of the 100 studies. </p>
<p>Over several years, <a href="http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.html">failed attempts to replicate published studies</a> have caused generally accepted bodies of research to be called into question — or rejected outright. </p>
<p>One example is the idea that <a href="https://hbr.org/2016/11/have-we-been-thinking-about-willpower-the-wrong-way-for-30-years">your willpower is a limited resource that, like a muscle, becomes exhausted</a> when it is used. Another is that power posing — <a href="http://journals.sagepub.com/doi/full/10.1177/0956797614553946">standing like a superhero for two minutes</a> — makes you feel bolder, reduces stress hormones and increases testosterone. Both have fallen aside due to failed replications. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=300&fit=crop&dpr=1 600w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=300&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=300&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=377&fit=crop&dpr=1 754w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=377&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=377&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Psychology was wrong about the power pose.</span>
<span class="attribution"><span class="source">(Shutterstock)</span></span>
</figcaption>
</figure>
<p>These aren’t dusty, arcane findings limited to academic journals; a TED talk by social psychologist Amy Cuddy on the effectiveness of power posing has been viewed over 45 million times and is <a href="https://www.ted.com/playlists/171/the_most_popular_talks_of_all">near the top of the list of the most popular TED talks of all time</a>. </p>
<h2>Bad habits</h2>
<p>The “bikini” at the centre of the crisis refers to the way researchers collect and analyze data and report their results. Many important details and decisions are often concealed.</p>
<p>When carrying out experiments, <a href="http://journals.sagepub.com/doi/abs/10.1177/0956797611417632">researchers make decisions</a> about how much data to collect, whether some observations should be excluded from the analysis and what controls, if any, should be included in analyses. </p>
<p>After the data has been collected, researchers have additional, undisclosed, leeway. </p>
<p>They may “torture the data” <a href="https://theconversation.com/one-reason-so-many-scientific-studies-may-be-wrong-66384">until it reaches statistical significance</a> (a cut-off that suggests the real effect may not be zero), a practice called “p-hacking.” </p>
<p>Or they may <a href="https://www.nature.com/articles/s41562-016-0021">engage in the practice of “HARKing,”</a> short for “hypothesizing after results are known.” Creating a hypothesis to confirm a result that has already been found makes it easier to satisfy journal reviewers and editors who are interested in publishing statistically significant results.</p>
<p>In academia, where researchers are often under pressure to “publish or perish” to advance their careers and win grants, amassing publications is the route to success. </p>
<p>All told, this undisclosed flexibility can lead to extremely high rates of <a href="http://journals.sagepub.com/doi/full/10.1177/0956797616658563">false positive results</a>. A false positive is essentially claiming there is an effect when there isn’t one. An example would be concluding that standing up straight increases testosterone levels, when it doesn’t. </p>
<h2>A new research culture</h2>
<p>Despite all the upheaval, psychology’s replication crisis may have a silver lining. In a few short years, researchers have proposed many ideas and recommendations for <a href="http://science.sciencemag.org/content/sci/348/6242/1422.full.pdf">reforming research with the goal of improvement</a>. </p>
<p>Journals and granting agencies are demanding more from authors with respect to <a href="http://www.science.gc.ca/eic/site/063.nsf/eng/h_415B5097.html">openness</a> and <a href="https://www.psychologicalscience.org/publications/badges">transparency</a>. There are accessible online repositories, such as <a href="http://github.com">Github</a>, the <a href="http://osf.io">Open Science Framework</a> and <a href="http://opendoar.org">OpenDOAR</a>, that allow researchers to share their raw materials, exact protocols, scripts, data, code, etc. with anyone who has an internet connection. The aim is to essentially have nothing concealed in the scientific process.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=338&fit=crop&dpr=1 600w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=338&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=338&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Researchers who manipulate their data or engage in poor research practices will wind up with results that can’t be replicated.</span>
<span class="attribution"><span class="source">(Shutterstock)</span></span>
</figcaption>
</figure>
<p>Some journals, such as <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456"><em>Psychological Science</em></a>, and recently <a href="http://www.apa.org/news/press/releases/2017/08/open-science.aspx">American Psychological Association journals</a> are encouraging authors to store their data and code in these repositories and to disclose details about data collection decisions before submitting a manuscript for peer review. Researchers can also preregister their hypotheses. But something has been missing. </p>
<h2>The missing link</h2>
<p>While psychological science has been moving toward more open and transparent methods, graduate student training has been largely left out of discussions. </p>
<p>Many of the practices that created the crisis are embedded in our research culture: We do things a certain way because we have always done things this way and other people do too. Much of this culture is assimilated when researchers are in graduate school. </p>
<p>To sustain and maintain the momentum of positive change, it is important for graduate education to keep up with changes in the field. If training fails to keep up, graduate students may leave programs with antiquated ideas and practices. <a href="http://rsos.royalsocietypublishing.org/content/3/9/160384">These ideas and practices can proliferate</a> as students become faculty members, start their own labs and train graduate students in the same manner they were taught.</p>
<p>Part of educating students is ensuring they are aware of the changing cultural landscape, and then explicitly teaching them to follow open and transparent research practices and avoid bad habits. </p>
<h2>Finding the light</h2>
<p>In our department at the University of Guelph, a group of methodologically minded faculty have recognized the importance of tackling this problem head on. Our goal is to create positive change and take steps to avoid history repeating itself with the next generation of researchers. </p>
<p>We created “<a href="https://www.uoguelph.ca/psychology/graduate/thesis-statistics">Statistical methods in theses: Guidelines and explanations</a>” to help students when conducting their thesis research. Students can work through the guidelines with their advisors, allowing them to make better decisions in the planning stages of their research projects.</p>
<p>The document’s rather humble sounding purpose belies an unintended provocative side. The guidelines identify questionable research practices — to provide explanations and advice for students who wish to follow open and transparent research practices. Because some of the questionable practices it identifies may be standard, previously unquestioned — and sometimes taught — procedures, the document has the potential to be viewed, by some, as extreme. </p>
<p>Culture is not something that can be changed overnight. But with explicit efforts to cultivate a new research culture, change can be targeted and purposeful. </p>
<p>This crisis in psychology makes me think about a line in John Milton’s epic poem, <em>Paradise Lost</em>: “Long is the way and hard, that out of Hell leads up to Light.” </p>
<p>By acting on the crisis, psychology has embarked upon its symbolic journey back to “light.” It will be current and future graduate students that will decide how, and where, the journey ends.</p><img src="https://counter.theconversation.com/content/90286/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Jeffrey R. Spence works at the University of Guelph. </span></em></p><p class="fine-print"><em><span>David Stanley is affiliated with University of Guelph. </span></em></p><p class="fine-print"><em><span>Ian Newby-Clark does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Bad research techniques have called into question the results of many psychology studies. Fixing the problem starts with making sure students don’t pick up bad habits.Jeffrey R. Spence, Associate Professor, University of GuelphDavid Stanley, Associate Professor, University of GuelphIan Newby-Clark, Professor, University of GuelphLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/868652017-11-09T12:39:28Z2017-11-09T12:39:28ZScience’s credibility crisis: why it will get worse before it can get better<figure><img src="https://images.theconversation.com/files/193759/original/file-20171108-26972-17v1ar7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Science itself needs to be put under the microscope and carefully scrutinised to deal with its flaws.</span> <span class="attribution"><span class="source"> Nattapat Jitrungruengnij/Shutterstock</span></span></figcaption></figure><p>Science’s credibility crisis is making headlines once more thanks to a <a href="http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12461/full">paper</a> from John P. A. Ioannidis and co-authors. Ioannidis, an expert in statistics, medicine and health policy at Stanford University, has done more than anyone else to ring the alarm bells on science’s quality control problems: scientific results are published which other researchers cannot reproduce. </p>
<p>When the crisis erupted in the media in 2013 The Economist devoted it’s <a href="https://www.google.es/search?q=economist++science+goes+wrong&client=firefox-b&dcr=0&source=lnms&tbm=isch&sa=X&ved=0ahUKEwi-uu-D2JrXAhWFOhQKHYcZDF0Q_AUICigB&biw=1920&bih=971#imgrc=6VLfyJMejNEoVM:">cover</a> to “<a href="https://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needs-change-itself-how-science-goes-wrong">Wrong Science</a>”. Ionannidis’s <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124">work</a> was an important part of the background material for the <a href="https://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble">piece</a>.</p>
<p>In previous papers Ioannidis had mapped the troubles of fields such as <a href="https://www.nature.com/nature/journal/v483/n7391/full/483531a.html">pre-clinical</a> and <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002049">clinical</a> medical studies; commenting how, <a href="http://www.jclinepi.com/article/S0895-4356(16)00147-5/pdf">under market pressure</a>, clinical medicine has been transformed to finance-based medicine.</p>
<p>In this new <a href="http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12461/full">work</a> he and co-authors target empirical economics research. They conclude that the field is diseased, with one fifth of the subfields investigated showing a 90% incidence of under-powered studies – a good indicator of low-quality research – and a widespread bias in favour of positive effects. </p>
<p>The field of psychology had gone through a similar ordeal. Brian Nosek, professor of psychology at the University of Virginia and his co-workers ran a <a href="http://science.sciencemag.org/content/349/6251/aac4716">replication analysis</a> of 100 high-profile psychology studies and reported that only about one third of the studies could be replicated. </p>
<p>Several other instances of bad science have gained attention in the media.
The problems in <a href="https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/comment-page-1/">“priming research”</a>, relevant to marketing and advertising, prompted Nobel Prize winner Daniel Kahneman to issue a publicised statement of <a href="https://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535">concern</a> about the wave of failed replication. </p>
<p>And a study on “power poses”, which claimed that body posture influences a person’s hormones level and “feelings of power” went first viral on <a href="https://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are">TED</a> when it was published – then again when its replication <a href="https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html?_r=0&utm_content=bufferab1e2&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer">failed</a>. </p>
<p>We are observing two new phenomena. On the one hand doubt is shed on the quality of entire scientific fields or sub-fields. On the other this doubt is played out in the open, in the media and blogosphere.</p>
<h2>Fixes</h2>
<p>In his <a href="http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12461/full">newest work</a> Ioannidis sets out a list of remedies that science needs to adopt urgently. These include fostering a culture of replication, data sharing and more collaborative works that pool together larger data sets; along with pre-specification of the protocol including model specifications and the analyses to be conducted.</p>
<p><a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001747">Ioannidis</a> has previously proposed additional remedies to “fix” science, as have <a href="http://orca.cf.ac.uk/97336/">other investigators</a>. The list includes better statistical methods and better teaching of statistics as well as measures to restore the right system of incentives at all stages of the scientific production system – from peer review to academic careers. </p>
<p>Important work is already being done by committed individuals and communities, among them Nosek’s <a href="https://osf.io/ezcuj/">Reproducibility Project</a>, Ioannidis’ <a href="https://metrics.stanford.edu/">Meta-research innovation centre</a>, Ben Goldacre’s <a href="http://www.alltrials.net/">alltrials.net</a> and the activities of <a href="http://retractionwatch.com/">Retraction Watch</a>. These initiatives – which attracted <a href="https://www.wired.com/2017/01/john-arnold-waging-war-on-bad-science/">private funding</a> – are necessary and timely.</p>
<p>But what are the chances that these remedies will work? Will this crisis be solved any time soon?</p>
<h2>Methods, incentives and introspection</h2>
<p>Ioannidis and co-authors are aware of the interplay between methods and incentives. For example, they say they’d refrain from suggesting that underpowered studies go unpublished, “as such a strategy would put pressure on investigators to report unrealistic and inflated power estimates based on spurious assumptions”.</p>
<p>This is a crucial point. Better practices will only be adopted if new incentives gain traction. In turn the incentives will have traction only if they address the right set of science’s problems and contradictions. </p>
<p>Ethics is a crucial issue in this respect. And here is where research effort is lacking. The broader field of economics is aware of its ethical problems after Paul Romer – now chief economist of the World Bank – coined the new term “<a href="https://paulromer.net/wp-content/uploads/2015/05/Mathiness.pdf">Mathiness</a>”, to signify the use of mathematics to veil normative premises. Yet there seem to be some hesitation to join the dots from the methodology to the ethos of the discipline, or of science overall.</p>
<p>The book <a href="https://www.amazon.com/Rightful-Place-Science-Verge/dp/0692596380">Science on the Verge</a> has proposed an analysis of the root causes of the crisis, including its neglected ethical dimension. The formulation of remedial measures depends on understanding <a href="http://www.sciencedirect.com/science/article/pii/S0016328717301969">what happened to science</a> and how this reflects on its <a href="https://theconversation.com/to-tackle-the-post-truth-world-science-must-reform-itself-70455">social role</a>, including when science feeds into <a href="http://www.sciencedirect.com/science/article/pii/S0016328717300472">evidence based policy</a>. </p>
<p>These analyses are indebted to philosophers Silvio O. Funtowicz and Jerome R. Ravetz, who spent several decades studying the <a href="https://en.wikipedia.org/wiki/Uncertainty_and_quality_in_science_for_policy">science’s quality control arrangements</a> and how quality and uncertainty impacted the <a href="http://www.sciencedirect.com/science/article/pii/001632879390022L">use of science for policy</a>.</p>
<p>Ravetz’s book “<a href="https://en.wikipedia.org/wiki/Scientific_Knowledge_and_Its_Social_Problems">Scientific knowledge and its social problems</a>” published in 1971 predicted several relevant features of the present crisis.</p>
<p>For Ravetz it is possible for a field <a href="https://en.wikipedia.org/wiki/Scientific_Knowledge_and_Its_Social_Problems">to be diseased</a>, so that shoddy work is routinely produced and accepted. Yet, he notes, it will be far from easy to come to accept the existence of such a condition – and even more difficult to reform it. </p>
<p>Reforming a diseased field or arresting the incipient decline of another will be delicate tasks, adds Ravetz, which calls for a </p>
<blockquote>
<p>sense of integrity, and a commitment to good work, among a significant section of the members of the field; and committed leaders with scientific ability and political skill. No quantity of published research reports, nor even an apparatus of institutional structures, can do anything to maintain or restore the health of a field in the absence of this essential ethical element operating through the interpersonal channel of communication.</p>
</blockquote>
<p>Ravetz emphasises the loss of this essential ethical element. In later works he notes that the new social and ethical conditions of science are reflected in a set of <a href="http://www.andreasaltelli.eu/file/repository/Maturing_Contradictions_2011_1.pdf">“emerging contradictions”</a>. These concern the cognitive dissonance between the official image of science as enlightened, egalitarian, protective and virtuous, against the current realities of scientific dogmatism, elitism and corruption; of science serving corporate interests and practices; of science used as an ersatz religion. </p>
<p>Echoes of Ravetz’s analysis can be found in many recent works, such as on the <a href="http://www.hup.harvard.edu/catalog.php?isbn=9780674046467">commodification of science</a>, or on the present problems <a href="https://theconversation.com/science-in-crisis-from-the-sugar-scam-to-brexit-our-faith-in-experts-is-fading-65016">with trust in expertise</a>. </p>
<h2>A call to arms?</h2>
<p>Ioannidis and co-authors are careful to stress the importance of a multidisciplinary approach, as both troubles and solutions may spill over from one discipline to the other. This would perhaps be a call to the arms for social scientists in general – and for those who study science itself – to tackle the crisis as a priority. </p>
<p>Here we clash with another of science’s contradictions: at this point in time, to study science as a scholar would mean to criticise its mainstream image and role. We do not see this happening any time soon. Because of the scars of “science wars” – whose spectre is <a href="https://theconversation.com/science-wars-in-the-age-of-donald-trump-67594">periodically resuscitated</a> – social scientists are wary of being seen as attacking science, or worse helping US President Donald Trump. </p>
<p>Scientists overall wish to use their <a href="https://theconversation.com/forcing-consensus-is-bad-for-science-and-society-77079">moral authority</a> and association with Enlightenment values, as seen in the recent <a href="https://theconversation.com/a-scientists-march-on-washington-is-a-bad-idea-heres-why-73305">marches for science</a>. </p>
<p>If these contradictions are real, then we are condemned to see the present crisis becoming worse before it can become better.</p><img src="https://counter.theconversation.com/content/86865/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Andrea Saltelli does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>We are observing two new phenomena. On one hand doubt is shed on the quality of entire scientific fields or sub-fields. On the other this doubt is played out in the open, in the media and blogosphere.Andrea Saltelli, Adjunct Professor Centre for the Study of the Sciences and the Humanities, University of Bergen, University of BergenLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/809972017-07-19T17:01:16Z2017-07-19T17:01:16ZHere’s the three-pronged approach we’re using in our own research to tackle the reproducibility issue<figure><img src="https://images.theconversation.com/files/178674/original/file-20170718-31872-1uv1xdv.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Step one is not being afraid to reexamine a site that's been previously excavated.</span> <span class="attribution"><span class="source">Dominic O'Brien. Gundjeihmi Aboriginal Corporation</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span></figcaption></figure><p>If you keep up with health or science news, you’ve probably been whipsawed between conflicting reports. Just days apart you may hear that “science says” coffee’s good for you, no actually it’s bad for you, actually red wine holds the secret to long life. As <a href="https://www.youtube.com/watch?v=0Rnq1NpHdmw">comedian John Oliver put it</a>:</p>
<blockquote>
<p>“After a certain point, all that ridiculous information can make you wonder: is science bullshit? To which the answer is clearly no. But there is a lot of bullshit currently masquerading as science.”</p>
</blockquote>
<p>A big part of this problem has to do with what’s been called a “<a href="https://theconversation.com/us/topics/reproducibility-5484">reproducibility crisis</a>” in science – many studies if run a second time don’t come up with the same results. <a href="https://doi.org/10.1038/533452a">Scientists are worried</a> about this situation, and <a href="https://www.nature.com/collections/byblhcfwhw">high-profile</a> international <a href="https://doi.org/10.1126/science.aab2374">research journals</a> have raised the alarm, too, calling on researchers to put more effort into ensuring their results can be reproduced, rather than only striving for splashy, one-off outcomes.</p>
<p><a href="https://www.nytimes.com/2016/05/29/opinion/sunday/why-do-so-many-studies-fail-to-replicate.html">Concerns about</a> <a href="https://www.theatlantic.com/science/archive/2016/03/psychologys-replication-crisis-cant-be-wished-away/472272/">irreproducible results</a> <a href="http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html">in science resonate</a> <a href="https://fivethirtyeight.com/features/science-isnt-broken/">outside the ivory tower</a>, as well, because a lot of this research translates into information that affects our everyday lives. </p>
<p>For example, it informs what we know about how to stay healthy, how doctors should look after us when we’re sick, how best to educate our children and how to organize our communities. If study results are not reproducible, then we can’t trust them to give good advice on solving our everyday problems – and society-wide challenges. Reproducibility is not just a minor technicality for specialists; it’s a pressing issue that affects the role of modern science in society.</p>
<p>Once we’ve identified that reproducibility is a big problem, the question becomes: How do we tackle it? Part of the answer has to do with changing incentives for researchers. But there are plenty of things we in the research community can do right now in the course of our scientific work.</p>
<p>It might come as a surprise that <a href="https://doi.org/10.1007/s10816-015-9272-9">archaeologists are at the forefront</a> of finding ways to improve the situation. Our <a href="https://doi.org/10.1038/nature22968">recent paper in Nature</a> demonstrates a concrete three-pronged approach to improving the reproducibility of scientific findings.</p>
<h2>Going back to where it all started</h2>
<p>In our new publication we describe recent work at an archaeological site in northern Australia. The results of our excavations and laboratory analyses show that <a href="http://theconversation.com/buried-tools-and-pigments-tell-a-new-history-of-humans-in-australia-for-65-000-years-81021">people arrived in Australia 65,000 years ago</a>, substantially earlier than the previous consensus estimate of 47,000 years ago. <a href="http://theconversation.com/buried-tools-and-pigments-tell-a-new-history-of-humans-in-australia-for-65-000-years-81021">This date has exciting implications</a> for our understandings of human evolution.</p>
<p>A less obvious detail about this study is the care we’ve taken to make our results reproducible. Our reproducibility strategy had three parts: fieldwork, labwork and data analyses.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=906&fit=crop&dpr=1 600w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=906&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=906&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1138&fit=crop&dpr=1 754w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1138&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1138&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Ben Marwick and colleagues excavating at Madjedbebe.</span>
<span class="attribution"><span class="source">Dominic O'Brien. Gundjeihmi Aboriginal Corporation</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>Our first step toward reproducibility was our choice of what to investigate. Rather than striking out to someplace new, we reexcavated an archaeological site <a href="https://doi.org/10.1016/j.jhevol.2015.03.014">previously known to have very old artifacts</a>.</p>
<p>The rockshelter site Madjedbebe in Australia’s Northern Territory had been excavated twice before. Famously, excavations there in 1989 indicated that people had <a href="https://doi.org/10.1038/345153a0">arrived in Australia by about 50,000 years ago</a>. But this age was not accepted by many archaeologists, who refused to accept anything older than 47,000 years ago.</p>
<p>This age was controversial from its first publication, and our goal in revisiting the site was to check if it was reliable or not. Could that controversial 50,000-years age be reproduced, or was it just a chance result that didn’t indicate the true time period for human habitation in Australia?</p>
<p>Like many scientists, archaeologists are generally less interested in returning to old discoveries, instead preferring to forge new paths in search of novel results. The problem with this is that it can lead to many unresolved questions, making it difficult to build a solid foundation of knowledge. </p>
<h2>Double-check the lab tests</h2>
<p>The second part of our reproducibility strategy was to verify that our laboratory analyses were reliable.</p>
<p>Our team used <a href="https://www.thoughtco.com/luminescence-dating-cosmic-method-171538">optically stimulated luminescence</a> methods to date the sand grains near the ancient artifacts. This method is complex, and there are only a few places in the world that have the instruments and skills to date these kinds of samples.</p>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=766&fit=crop&dpr=1 600w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=766&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=766&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=963&fit=crop&dpr=1 754w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=963&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=963&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Zenobia Jacobs produced the new ages for the Madjebdebe site based on her work in the Luminescence Dating Laboratory at the University of Wollongong, Australia.</span>
<span class="attribution"><span class="source">University of Wollongong</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>We first analyzed our samples in our laboratory at the <a href="http://smah.uow.edu.au/sees/facilities/UOW002889.html">University of Wollongong</a> to find their ages. Then we sent blind duplicate samples to another laboratory at the <a href="https://www.adelaide.edu.au/ipas/facilities/luminescence/">University of Adelaide</a> to analyze, without telling that lab our results. With both sets of analyses in hand, we compared them; it turned out in this case that they got the same ages as we did for the same samples.</p>
<p>This kind of verification is not a common practice in archaeology, but because this site was already controversial, we wanted to make sure the ages we obtained were reproducible.</p>
<p>While this extra work involved some additional cost and time, it’s vital to proving that our dates give the true ages of the sediments surrounding the artifacts. This verification shows that our lab results are not due to chance, or the unique conditions of our laboratory. Other archaeologists, and the public, can be more confident in our findings because we’ve taken these extra steps. This external checking should be standard practice in any science where controversial findings are at stake. </p>
<h2>Don’t let the computer be a black box</h2>
<p>After we completed the excavation and lab analyses, we analyzed the data on our computers. This stage of our research was very similar to what scientists in many other fields do. We loaded the raw data into our computers to visualize it with plots and test hypotheses with statistical methods.</p>
<p>However, while many researchers do this work by pointing and clicking using off-the-shelf software, we tried as much as possible to write scripts in the <a href="https://doi.org/10.1038/517109a">R programming language</a>.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=402&fit=crop&dpr=1 600w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=402&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=402&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=505&fit=crop&dpr=1 754w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=505&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=505&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Could be the enemy of reproducibility if it helps obscure the steps in data analysis.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/erinkohlenbergphoto/5353222369">Erin Kohlenberg</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span>
</figcaption>
</figure>
<p>Pointing and clicking generally leaves no traces of important decisions made during data analysis. Mouse-driven analyses leave the researcher with a final result, but none of the steps to get that result is saved. This makes it <a href="https://theconversation.com/how-computers-broke-science-and-what-we-can-do-to-fix-it-49938">difficult to retrace the steps</a> of an analysis, and check the assumptions made by the researcher.</p>
<p>On the other hand, our scripts contain a record of all our data analysis steps and decisions. They’re like an exact recipe to generate our results. Other researchers not using scripts for their data analysis don’t have these recipes, so their results are much harder to reproduce. </p>
<p>Another advantage of our choice to use scripts is that we can share them with the scientific community and the public. We follow <a href="https://doi.org/10.1038/nn.4550">standard practices</a> by making our script files and main data files <a href="https://osf.io/qwfcz/">freely available online</a> so anyone can inspect the details of our analysis, or explore new ideas using our data.</p>
<p>It’s easy to understand why many researchers prefer point-and-click over writing scripts for their data analysis. Often that’s what they were taught as students. It’s hard work and time-consuming to learn new analysis tools among the pressures of teaching, applying for grants, doing fieldwork and writing publications. Despite these challenges, there is an accelerating shift away from point-and-click toward scripted analyses in many areas of science.</p>
<h2>Combating irreproducibility one step at a time</h2>
<p>Our recent paper is part of a new movement emerging in many disciplines to improve the reproducibility of science. Examples of recent papers that have made a commitment to reproducibility similar to ours have come from <a href="https://doi.org/10.1038/nature22975">epidemiology</a>, <a href="https://doi.org/10.1038/s41559-017-0160">oceanography</a> and <a href="https://doi.org/10.7554/eLife.20470">neuroscience</a>.</p>
<p>We hope our example will inspire other scientists to be strategic about improving the reproducibility of their research. Some of these steps can be difficult for researchers: It means learning how to use unfamiliar software, and publicly sharing more of their data and methods than they’re accustomed to. But they’re important for generating reliable results – and for maintaining public confidence in scientific knowledge.</p><img src="https://counter.theconversation.com/content/80997/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Ben Marwick receives funding from the Australian Research Council, the University of Wollongong, and the University of Washington. This work was supported in part by the University of Washington eScience Institute.</span></em></p><p class="fine-print"><em><span>Zenobia Jacobs receives funding from the Australian Research Council. </span></em></p>A team of archaeologists strived to improve the reproducibility of their results, influencing their choices in the field, in the lab and during data analysis.Ben Marwick, Associate Professor of Archaeology, University of WashingtonZenobia Jacobs, Professor, University of WollongongLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/768512017-05-30T01:49:32Z2017-05-30T01:49:32ZResearch transparency: 5 questions about open science answered<figure><img src="https://images.theconversation.com/files/171204/original/file-20170526-6389-1eepgnq.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Opening up data and materials helps with research transparency.</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/book-wisdom-life-read-magic-background-515241850">REDPIXEL.PL via Shutterstock.com</a></span></figcaption></figure><p><strong>What is “open science”?</strong></p>
<p><a href="https://osf.io/preprints/psyarxiv/ak6jr">Open science</a> is a set of practices designed to make scientific processes and results more transparent and accessible to people outside the research team. It includes making complete research materials, data and lab procedures freely available online to anyone. Many scientists are also proponents of <a href="https://sparcopen.org/open-access/">open access</a>, a parallel movement involving making research articles available to read without a subscription or access fee.</p>
<p><strong>Why are researchers interested in open science? What problems does it aim to address?</strong></p>
<p>Recent research finds that many published scientific findings might not be reliable. For example, researchers have reported being able to replicate <a href="https://elife.elifesciences.org/collections/reproducibility-project-cancer-biology">only 40 percent</a> <a href="https://doi.org/10.1038/nrd3439-c1">or less</a> of <a href="http://www.nature.com/nature/journal/v483/n7391/full/483531a.html">cancer biology results</a>, and a large-scale <a href="https://doi.org/10.1126/science.aac4716">attempt to replicate 100 recent psychology studies</a> successfully reproduced fewer than half of the original results.</p>
<p>This has come to be called a “<a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">reproducibility crisis</a>.” It’s pushed many scientists to look for ways to improve their research practices and increase study reliability. Practicing open science is one way to do so. When scientists share their underlying materials and data, other scientists can more easily evaluate and attempt to replicate them.</p>
<p>Also, open science can help speed scientific discovery. When scientists share their materials and data, others can use and analyze them in new ways, potentially leading to new discoveries. Some journals are specifically dedicated to publishing data sets for reuse (<a href="https://www.nature.com/sdata/">Scientific Data</a>; <a href="http://openpsychologydata.metajnl.com/">Journal of Open Psychology Data</a>). <a href="http://doi.org/10.5334/jopd.ac">A paper in the latter</a> has already been cited 17 times in under three years – nearly all these citations represent new discoveries, sometimes on topics unrelated to the original research.</p>
<p><strong>Wait – open science sounds just like the way I learned in school that science works. How can this be new?</strong></p>
<p>Under the status quo, science is shared through a single vehicle: Researchers publish journal articles summarizing their studies’ methods and results. The key word here is summary; to write a clear and succinct article, important details may be omitted. Journal articles are vetted via the peer review process, in which an editor and a few experts assess them for quality before publication. But – perhaps surprisingly – the primary data and materials underlying the article are almost never reviewed. </p>
<p>Historically, this made some sense because journal pages were limited, and storing and sharing materials and data were difficult. But with computers and the internet, it’s much easier to practice open science. It’s now feasible to store large quantities of information on personal computers, and <a href="https://www.nature.com/sdata/policies/repositories">online repositories to share study materials and data</a> are becoming more common. Recently, some journals have even begun to <a href="http://journals.plos.org/plosone/s/data-availability">require</a> or <a href="https://osf.io/tvyxz/wiki/5.%20Adoptions%20and%20Endorsements/">reward</a> <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456">open science practices</a> like publicly posting materials and data.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=397&fit=crop&dpr=1 600w, https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=397&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=397&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=499&fit=crop&dpr=1 754w, https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=499&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/171205/original/file-20170526-6402-1kb6dxp.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=499&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Open science makes sharing data the default.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/client-passing-documentation-binders-his-partner-330663044">Bacho via Shutterstock.com</a></span>
</figcaption>
</figure>
<p>There are still some difficulties sharing extremely large data sets and physical materials (such as the specific liquid solutions a chemist might use), and some scientists might have good reasons to keep some information private (for instance, trade secrets or study participants’ personal information). But as time passes, more and more scientists will likely practice open science. And, in turn, science will improve.</p>
<p>Some do view the open science movement as a return to science’s core values. Most researchers over time have <a href="https://doi.org/10.1525/jer.2007.2.4.3">valued transparency</a> as a key ingredient in evaluating the truth of a claim. Now with technology’s help it is much easier to share everything.</p>
<p><strong>Why isn’t open science the default? What incentives work against open science practices?</strong></p>
<p>Two major forces work against adoption of open science practices: habits and reward structures. First, most established researchers have been practicing closed science for years, even decades, and changing these old habits requires some upfront time and effort. <a href="https://osf.io">Technology</a> is helping speed this process of adopting open habits, but behavioral change is hard. </p>
<p>Second, scientists, like other humans, tend to repeat behaviors that are rewarded and avoid those that are punished. Journal editors have tended to favor publishing papers that tell a tidy story with perfectly clear results. This has led researchers to craft their papers to be free from blemish, omitting “failed” studies that don’t clearly support their theories. But real data are often messy, so being fully transparent can open up researchers to critique. </p>
<p>Additionally, some researchers are afraid of being “scooped” – they worry someone will steal their idea and publish first. Or they fear that others will <a href="http://www.nejm.org/doi/full/10.1056/NEJMe1516564">unfairly benefit</a> from using shared data or materials without putting in as much effort. </p>
<p>Taken together, some researchers worry they will be punished for their openness and are skeptical that the perceived increase in workload that comes with adopting open science habits is needed and worthwhile. We believe scientists must continue to <a href="https://osf.io/tvyxz/">develop systems</a> to <a href="http://www.ourdigitalmags.com/publication/?i=365522&article_id=2657445&view=articleBrowser&ver=html5#%7B%22issue_id%22:365522,%22view%22:%22articleBrowser%22,%22article_id%22:%222657445%22%7D">allay fears</a> and reward openness. </p>
<p><strong>I’m not a scientist; why should I care?</strong></p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=466&fit=crop&dpr=1 600w, https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=466&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=466&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=585&fit=crop&dpr=1 754w, https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=585&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/171145/original/file-20170526-6380-6rryx7.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=585&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Open access is the cousin to open science – the idea is that research should be freely available to all, not hidden behind paywalls.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/34070876@N08/3602393341">h_pampel</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<p>Science benefits everyone. If you’re reading this article now on a computer, or have ever benefited from an antibiotic, or kicked a bad habit following a psychologist’s advice, then you are a consumer of science. Open science (and its cousin, open access) means that anyone – including teachers, policymakers, journalists and other nonscientists – can access and evaluate study information.</p>
<p>Considering automatic enrollment in a 401k at work or whether to have that elective screening procedure at the doctor? Want to ensure your tax dollars are spent on policies and programs that actually work? Access to high-quality research evidence matters to you. Open materials and open data facilitate reuse of scientific products, increasing the value of every tax dollar invested. Improving science’s reliability and speed benefits us all.</p><img src="https://counter.theconversation.com/content/76851/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Elizabeth Gilbert supports the Society for the Improvement of Psychological Science and has published on replication efforts as part of the Open Science Collaboration. Along with Katherine Corker and Barbara Spellman, she has a chapter called "Open Science: What, why, how" forthcoming in the Stevens Handbook of Experimental Psychology and Cognitive Neuroscience.</span></em></p><p class="fine-print"><em><span>Katie Corker is on the executive board for the Society for the Improvement of Psychological Science (improvingpsych.org) and an ambassador for the Center for Open Science (cos.io). She is also an editorial board member for Scientific Data. All of these roles are pro bono.</span></em></p>Partly in response to the so-called ‘reproducibility crisis’ in science, researchers are embracing a set of practices that aim to make the whole endeavor more transparent, more reliable – and better.Elizabeth Gilbert, Postdoctoral Research Fellow in Psychiatry and Behavioral Sciences, Medical University of South CarolinaKatie Corker, Assistant Professor of Psychology, Grand Valley State University Licensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/769672017-05-04T22:18:47Z2017-05-04T22:18:47ZBehind closed doors: What the Piltdown Man hoax from 1912 can teach science today<figure><img src="https://images.theconversation.com/files/167589/original/file-20170502-17267-1pa48wd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">When new discoveries are jealously guarded under lock and key, science suffers.</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/rightee/260028769">Andy Wright</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span></figcaption></figure><p>In 1912, Charles Dawson, an amateur archaeologist in England, claimed he’d made one of the most important fossil discoveries ever. Ultimately, however, his “Piltdown Man” proved to be a hoax. By cleverly pairing a human skull with an orangutan’s jaw – stained to match and give the appearance of age – a mysterious forger duped the scientific world.</p>
<p>In the decades between the find’s unearthing and the revelation it was fraudulent, people in the United States and around the world learned about Piltdown Man as a “missing link” connecting ape and man. Newspaper articles, scientific publications <a href="http://www.hup.harvard.edu/catalog.php?isbn=9780674660410">and museum exhibitions</a> all presented Piltdown Man as a legitimate scientific discovery supporting a particular vision of human evolution.</p>
<p>Historians, science writers and others have <a href="https://www.amazon.com/Piltdown-Scientific-Forgery-Frank-Spencer/dp/0198585225">investigated the Piltdown Man controversy</a> over the years, shedding <a href="https://global.oup.com/academic/product/the-piltdown-forgery-9780198607809?q=Piltdown&lang=en&cc=us">new light on the fraud</a>. As we reconsider the nature of “<a href="http://www.cnn.com/2017/01/22/politics/kellyanne-conway-alternative-facts/">facts</a>,” “<a href="https://www.theguardian.com/media/2016/dec/18/what-is-fake-news-pizzagate">fake news</a>” and knowledge production, it’s worthwhile to revisit the Piltdown Man episode.</p>
<p>At issue was not just the deliberate hoax, but also the incomplete flow of information about the purported human ancestor. Soon after the discovery, access to the original materials in England was cut off by a few gatekeepers. Science is suffocated when researchers are unable to reliably corroborate claims made by others. The same issues arise today, with the research community grappling with what’s been called a <a href="http://www.nature.com/news/reproducibility-1.17552">reproducibility crisis</a>; scientists need access to evidence and data in order to replicate (or not) research results. The Piltdown Man controversy lends support to the modern <a href="https://osf.io/preprints/psyarxiv/ak6jr">open science movement</a>, with its call for transparency at every step of the scientific process. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=438&fit=crop&dpr=1 600w, https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=438&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=438&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=551&fit=crop&dpr=1 754w, https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=551&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/167586/original/file-20170502-17285-1pjrddd.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=551&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Piltdown Man believers kept tight control over who could get an up-close look at the fossils. Arthur Keith is pictured in the white coat, Charles Dawson over his left shoulder.</span>
<span class="attribution"><a class="source" href="https://commons.wikimedia.org/wiki/File:Piltdown_gang_(dark).jpg">John Cooke</a></span>
</figcaption>
</figure>
<h2>Limited firsthand access</h2>
<p>Experts immediately cited the discovery of a large human-like cranium with a primitive-looking, ape-like jaw as a major breakthrough. Influential anatomists such as Sir Arthur Keith hailed Piltdown Man as authentic. The popular press on both sides of the Atlantic described prehistoric archaeology as a dramatic hunt for a missing link and came to embrace Piltdown Man within an oversimplified framework of human evolution.</p>
<p>But there were some scientists – notably British Museum curator Reginald A. Smith – who were skeptical from the outset. Doubters noted the major find was attributed to a previously little-known archaeologist.</p>
<p>Curators in the United States impatiently hoped to learn more. But transatlantic requests were denied by their counterparts in Britain who controlled access to the cranium and jaw, moving the bones to a secure vault at the Museum of Natural History in London. Rumors swirled. </p>
<p>Controversial Smithsonian curator <a href="http://anthropology.si.edu/naa/fa/Hrdlicka_Ales.pdf">Aleš Hrdlička describes in an annual report</a> traveling to England himself:</p>
<blockquote>
<p>“Regrettably… the specimen was not yet available for examination by outsiders, and so no original opinion can be given concerning its status. It represents doubtless one of the most interesting finds relating to man’s antiquity, though seemingly the last word had not yet been said as to its date and especially as to the physical characteristics of the being it stands for.”</p>
</blockquote>
<p>Early in the 20th century, provocative claims about discoveries commonly circulated through letters, rumors and splashy newspaper articles suggesting major new finds. American museums were simultaneously intrigued and frustrated by word of significant finds like Piltdown Man. Some claims proved to be genuine, while many others were found to be falsified or misleading. With limited information, it was especially difficult to determine the validity of claims made by scientists abroad.</p>
<p>News about major discoveries might change planned exhibitions about human evolution or prehistory at museums in New York or Chicago, or influence what students were taught about human history. Uncertainty plagued museums in this regard, as their scientists tried to view skeletons firsthand on visits to European museums and to secure good casts or copies for their own collections. Even amid growing doubts, a major exhibition in San Diego that opened in 1915 <a href="http://www.hup.harvard.edu/catalog.php?isbn=9780674660410">prominently featured a Piltdown Man sculpture</a>.</p>
<h2>What’s the damage done?</h2>
<p>This lack of transparency resulted in an absence of accurate information in the scientific community.</p>
<p>It ultimately took until the later decades of the 20th century for the Piltdown bones to be fully discredited. The hoax was <a href="https://theconversation.com/solving-the-piltdown-man-crime-how-we-worked-out-there-was-only-one-forger-63615">likely created by Dawson himself</a>, though <a href="https://www.amazon.com/Piltdown-Men-Ronald-William-Millar/dp/057500536X">who exactly concocted the scam is still debated</a> – “Sherlock Holmes” author <a href="http://www.telegraph.co.uk/science/2016/08/10/sir-arthur-conan-doyle-cleared-of-piltdown-man-hoax/">Arthur Conan Doyle’s name has even been mentioned</a> as a <a href="https://theconversation.com/a-new-twist-to-whodunnit-in-sciences-famous-piltdown-man-hoax-64470">possible perpetrator</a>.</p>
<p>As Berkeley anthropologist <a href="http://anthropology.si.edu/naa/fa/spencer.pdf">Sherwood Washburn offered in a letter</a>, “My opinion is that if more people had seen the originals sooner the fake would have been recognized.” Confusion had arisen because <a href="https://www.youtube.com/watch?v=2LtOkhpR3hY">so few scholars were granted access</a> to the original evidence.</p>
<p>Part of what finally put Piltdown Man to rest was the nature of new discoveries emerging. They informed researchers’ developing understanding of the human past and began turning much scientific attention away from Europe toward Asia and Africa.</p>
<p>While it is impossible to know with certainty, the Piltdown Man episode likely slowed scientific progress in the global search for human ancestors. What is clear is that the claims worked to muddle popular knowledge about human evolution.</p>
<h2>Piltdown Man’s lessons for today</h2>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=587&fit=crop&dpr=1 600w, https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=587&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=587&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=738&fit=crop&dpr=1 754w, https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=738&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/167590/original/file-20170502-17267-1l3115g.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=738&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Museums still display Piltdown Man replicas, not as science but as cautionary reminder.</span>
<span class="attribution"><a class="source" href="https://commons.wikimedia.org/wiki/File:Sterkfontein_Piltdown_man.jpg">Anrie</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<p>The unknown forger behind Piltdown Man intentionally misled the world about human evolution. The false claims rippled through the news media and museum exhibitions. Without access to reliable sources, in this case the original bones, the fraudulent story of Piltdown Man spread like a slowly building wildfire. </p>
<p>The Piltdown Man controversy hints at the dangers of drawing conclusions based on limited or emerging information, for both the public and scientists. In some ways, the whole episode foreshadowed threats we face now <a href="http://www.slate.com/blogs/future_tense/2017/04/13/facebook_s_latest_attempt_to_fight_fake_news_makes_it_seem_more_helpless.html">from fake news</a> and the spread of misinformation about science and many other topics. It is hard to get to the truth – whether about a news story or scientific theory – without access to the evidence supporting it.</p>
<p>Certainly new information flows much more rapidly today – thanks to the internet and social media – potentially a partial corrective to the problems connected to misleading claims. However, scientists and others still need access to accurate and reliable information from original sources. With the Piltdown Man remains locked away in a secure museum vault, speculation and misinformation flourished.</p>
<p>Support is now building for an <a href="http://www.digital-scholarship.org/cwb/WhatIsOA.htm">open access</a> research model: When possible and appropriate, original materials, data and preliminary findings should be made available to others in the field. Scientists also work to balance <a href="https://www.theguardian.com/science/2015/oct/25/discovery-human-species-accused-of-rushing-errors">how quickly they publish new research</a>: It takes time to do careful work, but keeping finds hidden away for too long also impedes progress and understanding. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=450&fit=crop&dpr=1 600w, https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=450&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=450&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=566&fit=crop&dpr=1 754w, https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=566&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/167785/original/file-20170503-21649-1x026xi.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=566&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Excavations continue in the hobbit cave in Indonesia.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/12394349@N06/14748473277">Bryn Pinzgauer</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span>
</figcaption>
</figure>
<p>Consider a 2003 find from Indonesia that was as shocking as the discovery of Piltdown Man: a nearly complete female skeleton researchers suggested was from a tiny human ancestor they called <a href="http://humanorigins.si.edu/evidence/human-fossils/species/homo-floresiensis">Homo floresiensis</a> (commonly nicknamed “hobbit”). Media speculation ran wild early on about this new species added to our family tree, but paleoanthropology has evolved a great deal since Piltdown Man.</p>
<p>Scientists from several different groups worked to <a href="http://www.livescience.com/29100-homo-floresiensis-hobbit-facts.html">understand the discovery</a> – seeking related finds and going back to the original fossils to systematically assess the claim. Soon additional <a href="https://dx.doi.org/10.1038/nature02999">detailed scientific publications began to emerge</a>, allowing the scientific community to continue <a href="https://dx.doi.org/10.1038/nature04022">to add to the evidence</a> and better <a href="https://dx.doi.org/10.1038/4311029a">scrutinize the discovery</a>. To date, the teeth of at many as 12 individuals have been found.</p>
<p>Homo floresiensis are likely a genuinely groundbreaking discovery – hopefully the more transparent way the research unfolded makes this easier to untangle than Dawson’s claims a century ago. Thoughtful collaboration, making data available openly, more effective <a href="https://www.theguardian.com/science/political-science/2016/may/10/what-has-science-communication-ever-done-for-us">popular science communication</a> and multiple channels of accurate information may help us better respond to the next Piltdown Man.</p><img src="https://counter.theconversation.com/content/76967/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Samuel Redman does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>A century-old case of scientific fraud illustrates how hard it is to untangle the truth when access to new discoveries is limited.Samuel Redman, Assistant Professor of History, UMass AmherstLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/656192016-09-21T00:01:46Z2016-09-21T00:01:46ZWhy isn’t science better? Look at career incentives<figure><img src="https://images.theconversation.com/files/138450/original/image-20160920-11131-1alomb3.jpg?ixlib=rb-1.1.0&rect=49%2C65%2C5289%2C3660&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Experiment design affects the quality of the results.</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/iaea_imagebank/8147632150">IAEA Seibersdorf Historical Images</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>There are often substantial gaps between the idealized and actual versions of those people whose work involves providing a social good. Government officials are supposed to work for their constituents. Journalists are supposed to provide unbiased reporting and penetrating analysis. And scientists are supposed to relentlessly probe the fabric of reality with the most rigorous and skeptical of methods. </p>
<p>All too often, however, what should be just isn’t so. In a number of scientific fields, <a href="https://www.washingtonpost.com/news/speaking-of-science/wp/2015/08/28/no-sciences-reproducibility-problem-is-not-limited-to-psychology/">published findings turn out not to replicate</a>, or to have smaller effects than, what was initially purported. Plenty of science does replicate – meaning the experiments turn out the same way when you repeat them – but the amount that doesn’t is too much for comfort.</p>
<p>Much of science is about identifying relationships between variables. For example, how might certain genes increase the risk of acquiring certain diseases, or how might certain parenting styles influence children’s emotional development? To our disappointment, there are no tests that allow us to perfectly sort true associations from spurious ones. Sometimes we get it wrong, even with the most rigorous methods.</p>
<p>But there are also ways in which scientists increase their chances of getting it wrong. Running studies with small samples, mining data for correlations and forming hypotheses to fit an experiment’s results after the fact are <a href="http://fivethirtyeight.com/features/science-isnt-broken/">just some of the ways</a> to <a href="http://doi.org/10.1038/526182a">increase the number of false discoveries</a>. </p>
<p>It’s not like we don’t know how to do better. Scientists who study scientific methods have known about <a href="http://doi.org/10.1086/288135">feasible remedies for decades</a>. Unfortunately, their advice often falls on deaf ears. Why? Why aren’t scientific methods better than they are? In a word: incentives. But perhaps not in the way you think. </p>
<h2>Incentives for ‘good’ behavior</h2>
<p>In the 1970s, <a href="https://en.wikipedia.org/wiki/Campbell%27s_law">psychologists</a> and <a href="https://en.wikipedia.org/wiki/Goodhart%27s_law">economists</a> began to point out the danger in relying on quantitative measures for social decision-making. For example, when public schools are evaluated by students’ performance on standardized tests, teachers respond by teaching “to the test” – at the expense of broader material more important for critical thinking. In turn, the test serves largely as a measure of how well the school can prepare students for the test.</p>
<p>We can see this principle – often summarized as “when a measure becomes a target, it ceases to be a good measure” – playing out in the realm of research. Science is a competitive enterprise. There are <a href="http://doi.org/10.1038/520144a">far more credentialed scholars and researchers</a> than there are university professorships or comparably prestigious research positions. Once someone acquires a research position, there is additional competition for tenure, grant funding, and support and placement for graduate students. Due to this competition for resources, scientists must be evaluated and compared. How do you tell if someone is a good scientist?</p>
<p>An oft-used metric is the number of publications one has in peer-reviewed journals, as well as the status of those journals (along with related metrics, such as the <a href="https://en.wikipedia.org/wiki/H-index"><em>h</em>-index</a>, which purports to measure the rate at which a researcher’s work is cited by others). Metrics like these make it straightforward to compare researchers whose work may otherwise be quite different. Unfortunately, this also makes these numbers susceptible to exploitation. </p>
<p>If scientists are motivated to publish often and in high-impact journals, we might expect them to actively try to game the system. And certainly, some do – as seen in recent high-profile cases of scientific fraud (including in <a href="https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal">physics</a>, <a href="http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html">social psychology</a> and <a href="http://onlinelibrary.wiley.com/doi/10.1111/bcp.12992/full">clinical pharmacology</a>). If malicious fraud is the prime concern, then perhaps the solution is simply heightened vigilance.</p>
<p>However, most scientists are, I believe, genuinely interested in learning about the world, and honest. The problem with incentives is they can shape cultural norms without any intention on the part of individuals. </p>
<h2>Cultural evolution of scientific practices</h2>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=784&fit=crop&dpr=1 600w, https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=784&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=784&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=986&fit=crop&dpr=1 754w, https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=986&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/138454/original/image-20160920-11090-684nc6.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=986&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Scientists work within a culture of research.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/iaea_imagebank/8199500456">IAEA</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<p>In a <a href="http://rsos.royalsocietypublishing.org/lookup/doi/10.1098/rsos.160384">recent paper</a>, anthropologist <a href="http://xcelab.net/rm/">Richard McElreath</a> and I considered the incentives in science through the lens of <a href="http://www.oxfordbibliographies.com/view/document/obo-9780199766567/obo-9780199766567-0038.xml">cultural evolution</a>, an emerging field that draws on ideas and models from evolutionary biology, epidemiology, psychology and the social sciences to understand cultural organization and change.</p>
<p>In our analysis, we assumed that methods associated with greater success in academic careers will, all else equal, tend to spread. The spread of more successful methods requires no conscious evaluation of how scientists do or do not “game the system.” </p>
<p>Recall that publications, particularly in high-impact journals, are the currency used to evaluate decisions related to hiring, promotions and funding. Studies that show large and surprising associations tend to be favored for publication in top journals, while small, unsurprising or complicated results are more difficult to publish.</p>
<p>But <a href="http://dx.doi.org/10.1371/journal.pmed.0020124">most hypotheses are probably wrong</a>, and performing rigorous tests of novel hypotheses (as well as coming up with good hypotheses in the first place) takes time and effort. Methods that boost false positives (incorrectly identifying a relationship where none exists) and overestimate effect sizes will, on average, allow their users to publish more often. In other words, when novel results are incentivized, methods that produce them – by whatever means – at the fastest pace will become implicitly or explicitly encouraged.</p>
<p>Over time, those shoddy methods will become associated with success, and they will tend to spread. The argument can extend beyond norms of questionable research practices to norms of misunderstanding, if those misunderstandings lead to success. For example, despite over a century of common usage, the <em>p</em>-value, a standard measure of statistical significance, is still <a href="http://dx.doi.org/10.1080/00031305.2016.1154108">widely misunderstood</a>.</p>
<p>The cultural evolution of shoddy science in response to publication incentives requires no conscious strategizing, cheating or loafing on the part of individual researchers. There will always be researchers committed to rigorous methods and scientific integrity. But as long as institutional incentives reward positive, novel results at the expense of rigor, the rate of bad science, on average, will increase. </p>
<h2>Simulating scientists and their incentives</h2>
<p>There is ample evidence suggesting that publication incentives have been negatively shaping scientific research for decades. The frequency of the words <a href="http://dx.doi.org/10.1136/bmj.h6467">“innovative,” “groundbreaking” and “novel”</a> in biomedical abstracts increased by 2,500 percent or more over the past 40 years. Moreover, researchers often <a href="http://dx.doi.org/10.1126/science.1255484">don’t report when hypotheses fail to generate positive results</a>, lest reporting such failures hinders publication.</p>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=736&fit=crop&dpr=1 600w, https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=736&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=736&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=925&fit=crop&dpr=1 754w, https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=925&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/138455/original/image-20160920-11127-ntmb9h.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=925&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">There doesn’t need to be anything nefarious going on for scientists to stick with the suboptimal methods that help them get ahead.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/iaea_imagebank/8198415199">IAEA</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<p>We reviewed <a href="http://www.statisticsdonewrong.com/power.html">statistical power</a> in the social and behavioral science literature. Statistical power is a quantitative measurement of a research design’s ability to identify a true association when present. The simplest way to increase statistical power is to increase one’s sample size – which also lengthens the time needed to collect data. Beginning in the 1960s, there have been <a href="http://datacolada.org/wp-content/uploads/2013/10/3416-Sedlmeier-Gigerenzer-Psych-Bull-1989-Do-studies-of-statistical-power-have-an-effect-on-the-power-of-studies.pdf">repeated outcries that statistical power is far too low</a>. Nevertheless, we found that statistical power, on average, <a href="http://rsos.royalsocietypublishing.org/lookup/doi/10.1098/rsos.160384">has not increased</a>.</p>
<p>The evidence is suggestive, but it is not conclusive. To more systematically demonstrate the logic of our argument, we built a computer model in which a population of research labs studied hypotheses, only some of which were true, and attempted to publish their results.</p>
<p>As part of our analysis, we assumed that each lab exerted a characteristic level of “effort.” Increasing effort lowered the rate of false positives, and also lengthened the time between results. As in reality, we assumed that novel positive results were easier to publish than negative results. All of our simulated labs were totally honest: they never cheated. However, labs that published more were more likely to have their methods “reproduced” in new labs – just as they would be in reality as students and postdocs leave successful labs where they trained and set up their own labs. We then allowed the population to evolve.</p>
<p>The result: Over time, effort decreased to its minimum value, and the rate of false discoveries skyrocketed. </p>
<p>And replication – while a crucial tool for generating robust scientific theories – isn’t going to be science’s savior. Our simulations indicate that more replication won’t stem the evolution of bad science.</p>
<h2>Taking on the system</h2>
<p>The bottom-line message from all this is that it’s not sufficient to impose high ethical standards (assuming that were possible), nor to make sure all scientists are informed about best practices (though spreading awareness is certainly one of our goals). A culture of bad science can evolve as a result of institutional incentives that prioritize simple quantitative metrics as measures of success. </p>
<p>There are indications that the situation is improving. Journals, organizations, and universities are increasingly emphasizing <a href="http://www.psychologicalscience.org/index.php/replication">replication</a>, <a href="https://royalsociety.org/journals/ethics-policies/data-sharing-mining/">open data</a>, <a href="http://blogs.plos.org/everyone/2015/02/25/positively-negative-new-plos-one-collection-focusing-negative-null-inconclusive-results/">the publication of negative results</a> and more <a href="https://www.idrc.ca/sites/default/files/sp/Documents%20EN/Research-Quality-Plus-A-Holistic-Approach-to-Evaluating-Research.pdf">holistic evaluations</a>. Internet applications such as <a href="https://twitter.com/lakens/status/774953862012755968">Twitter</a> and <a href="https://www.youtube.com/watch?v=WFv2vS8ESkk&list=PLDcUM9US4XdMdZOhJWJJD4mDBMnbTWw_z">YouTube</a> allow education about best practices to propagate widely, along with spreading norms of holism and integrity. </p>
<p>There are also signs that the old ways are far from dead. For example, one regularly hears researchers discussed in terms of how much or where they publish. The good news is that as long as there are smart, interesting people doing science, there will always be some good science. And from where I sit, there is still quite a bit of it.</p><img src="https://counter.theconversation.com/content/65619/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Paul Smaldino does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Embracing more rigorous scientific methods would mean getting science right more often than we currently do. But the way we value and reward scientists makes this a challenge.Paul Smaldino, Assistant Professor of Cognitive and Information Sciences, University of California, MercedLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/608692016-06-22T15:33:00Z2016-06-22T15:33:00ZIs psychology really in crisis?<figure><img src="https://images.theconversation.com/files/127300/original/image-20160620-8861-ifipj6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">I just can't seem to get my replication studies published.</span> <span class="attribution"><a class="source" href="http://www.shutterstock.com/cat.mhtml?lang=en&language=en&ref_site=photo&search_source=search_form&version=llv1&anyorall=all&safesearch=1&use_local_boost=1&autocomplete_id=&searchterm=psychoanalysis&show_color_wheel=1&orient=&commercial_ok=&media_type=images&search_cat=&searchtermx=&photographer_name=&people_gender=&people_age=&people_ethnicity=&people_number=&color=&page=1&inline=314294129">Photographee.eu/Shutterstock.com</a></span></figcaption></figure><p>Modern psychology is apparently in <a href="http://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-016-0135-2">crisis</a>. This claim is nothing new. From phrenology to psychoanalysis, psychology has traditionally had an uneasy scientific status. Indeed, the philosopher of science, Karl Popper, viewed Freud’s theories as a typical example of pseudoscience because no test could ever show them to be <a href="https://en.wikipedia.org/wiki/Falsifiability">false</a>. More recently, psychology has feasted on a banquet of extraordinary findings whose scientific credibility has also been questioned. </p>
<p>Some of these extraordinary findings include <a href="http://psycnet.apa.org/journals/psp/100/3/407/">Daryl Bem’s experiments</a>, published in 2011, that seem to show future events influence the past. Bem, an emeritus professor at Cornell University, revealed that people are more likely to remember a list of words if they practise them <em>after</em> a recall test, compared with practising them before the test. In another study, he showed that people are significantly better than chance at selecting which of two curtains hide a pornographic image. </p>
<p>Then there’s Yale’s John Bargh who in 1996 <a href="http://www.yale.edu/acmelab/articles/bargh_chen_burrows_1996.pdf">reported</a> that, when unconsciously primed with an “elderly stereotype” (by unscrambling jumbled sentences containing words such as “Florida” and “bingo”), people subsequently walk more slowly. Add to this Roy Baumeister who in 1998 presented <a href="https://bama.ua.edu/%7Esprentic/672%20Muraven%20%26%20Baumeister%202000.pdf">evidence</a> suggesting we have a finite store of will-power which is sapped whenever we resist temptations such as eating chocolates. Or, in the same year, <a href="http://psycnet.apa.org/psycinfo/1998-01060-003">Ap Dijksterhuis and Ad Van Knippenberg</a> showing that performance on Trivial Pursuit is better after people list typical characteristics of a professor rather than those of a football hooligan. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/127306/original/image-20160620-8867-ajehxx.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/127306/original/image-20160620-8867-ajehxx.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/127306/original/image-20160620-8867-ajehxx.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/127306/original/image-20160620-8867-ajehxx.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/127306/original/image-20160620-8867-ajehxx.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/127306/original/image-20160620-8867-ajehxx.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/127306/original/image-20160620-8867-ajehxx.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Does thinking about him really make you better at Trivial Pursuit?</span>
<span class="attribution"><a class="source" href="http://www.shutterstock.com/cat.mhtml?lang=en&language=en&ref_site=photo&search_source=search_form&version=llv1&anyorall=all&safesearch=1&use_local_boost=1&autocomplete_id=&search_tracking_id=tN3OCDYxnajjaAjIbfrMCA&searchterm=professor&show_color_wheel=1&orient=&commercial_ok=&media_type=images&search_cat=&searchtermx=&photographer_name=&people_gender=&people_age=&people_ethnicity=&people_number=&color=&page=1&inline=330576656">Dragon Images/Shutterstock.com</a></span>
</figcaption>
</figure>
<p>These studies are among the <a href="http://digest.bps.org.uk/2014/09/the-10-most-controversial-psychology.html">most controversial</a> in psychology. Not least because other researchers have had difficulty replicating the experiments. These types of studies raise concerns about the methods psychologists use, but also more broadly about psychology itself. </p>
<h2>Do not repeat</h2>
<p>A survey of 1,500 scientists published in <a href="http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970?WT.mc_id=SFB_NNEWS_1508_RHBox">Nature</a> last month indicated that 24% of them said they had published a successful replication and 13% published an unsuccessful replication. Contrast this with over a century of psychology publications, where just <a href="http://pps.sagepub.com/content/7/6/537">1%</a> of papers attempted to replicate past findings. </p>
<p><a href="https://bmcpsychology.biomedcentral.com/articles/10.1186/2050-7283-1-2">Editors and reviewers</a> have been complicit in a systemic bias that has resulted in high-profile psychology journals becoming storehouses for the strange. Many psychologists are obsessed with the “impact factors” of journals (as are the journals) – and one way to increase impact is to publish curios. Certain high-impact journals have a reputation of publishing curios that never get replicated but which attract lots of attention for the author and journal. By contrast, confirming the findings of others through replication is unattractive, rare and relegated to less prestigious journals. </p>
<p>Despite psychology’s historical abandonment of replication, is the tide turning? This year, a crowd-sourced initiative – the <a href="http://science.sciencemag.org/content/349/6251/aac4716">OSC Reproducibility project</a> – attempted to replicate 100 published findings in psychology. The multinational collaborators replicated just over a third (36%) of the studies. Does this mean that psychological findings are unreliable? </p>
<p>Replication projects are selective, targeting studies that are cheaper and less technically complicated to replicate or those that are simply unbelievable. Other projects such as “<a href="http://econtent.hogrefe.com/doi/full/10.1027/1864-9335/a000178">Many Labs</a>” have reported a replication rate of 77%. All initiatives are non-random and headline replication rates reflect the studies that are sampled. Even if a random sample of studies were examined, we don’t know what would constitute an acceptable replication rate in psychology. This is not an issue specific to psychology. As John Ioannidis noted: “<a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124">most published research findings are false”</a>“. After all, scientific hypotheses are our current best guesses about phenomena, not a simple accumulation of truths. </p>
<h2>Questionable research practices</h2>
<p>The frustration of many psychologists is palpable because it seems so easy to publish evidence consistent with almost any hypothesis. A likely cause of both unusual findings and non-replicability is psychologists indulging in questionable research practices (QRPs). </p>
<p>In 2012, a <a href="https://www.cmu.edu/dietrich/sds/docs/loewenstein/MeasPrevalQuestTruthTelling.pdf">survey of 2,000 American psychologists</a> found that most indulged in QRPs. Some 67% admitted selectively reporting studies that "worked”, while 74% failed to report all measures they had used. The survey also found that 71% continued to collect data until a significant result was obtained and 54% reported unexpected findings as if they were expected. And 58% excluded data after analyses. Astonishingly, more than one-third admitted they had doubts about the integrity of their own research on at least one occasion and 1.7% admitted to having faked their data. </p>
<p>The problems associated with modern psychology are longstanding and cultural, with researchers, reviewers, editors, journals and news-media all prioritising and benefiting from the quest for novelty. This systemic bias, coupled with minimal agreement on fundamental principles in certain areas of psychology, means questionable research practices can flourish – consciously or unconsciously. Large-scale replication projects will not address the cultural problems and may even exacerbate them by presenting replication as something special that we use to target the unbelievable. Replication – whether judged as failed or successful – is a fundamental aspect of normal science and needs to be both more common and more valued by psychologists and psychology journals.</p><img src="https://counter.theconversation.com/content/60869/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Keith Laws does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Only 1% of published psychology research papers are ever repeated. If psychologists want their discipline to be taken seriously, they’ll need to get their house in order.Keith Laws, Professor of Cognitive Neuropsychology, University of HertfordshireLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/472492015-09-10T10:11:12Z2015-09-10T10:11:12ZReal crisis in psychology isn’t that studies don’t replicate, but that we usually don’t even try<figure><img src="https://images.theconversation.com/files/94294/original/image-20150909-18665-5n51wp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Run a study again and again – should the results hit the same bull's-eye every time?</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/richardofengland/6788829651">Richard Matthews</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span></figcaption></figure><p>Psychology is still digesting the implications of a large study published last month, in which a team led by University of Virginia’s Brian Nosek <a href="http://doi.org/10.1126/science.aac4716">repeated 100 psychological experiments</a> and found that <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">only 36% of originally “significant”</a> (in the statistical sense) results were replicated.</p>
<p>Commentators are divided over how much to worry about the news. Some psychologists have suggested that the <a href="https://www.psychologytoday.com/blog/the-nature-nurture-nietzsche-blog/201509/quick-guide-the-replication-crisis-in-psychology">field is in “crisis,”</a> a claim that others (such as Northeastern University psychology professor <a href="http://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html">Lisa Feldman Barrett</a>) have flatly denied.</p>
<p>What can we make of such divergence of opinion? Is the discipline in crisis or not? Not in the way that some seemed to suggest, but that doesn’t mean substantial changes aren’t needed.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Does the majority of psych research belong in the trash?</span>
<span class="attribution"><a class="source" href="http://www.shutterstock.com/pic-220895758/stock-photo-a-lot-of-wrinkled-paper-laying-in-and-around-a-wastepaper-basket-picture-is-toned.html">Wastebasket image via www.shutterstock.com</a></span>
</figcaption>
</figure>
<h2>Mixing up what the study really tells us</h2>
<p>Certainly the fact that 64% of the findings were found unstable is surprising and disconcerting. But some of the more sensational press response has been disappointing. </p>
<p>Over at The Guardian, a headline writer implied the study delivered a “<a href="http://www.theguardian.com/science/2015/aug/27/study-delivers-bleak-verdict-on-validity-of-psychology-experiment-results">bleak verdict on validity</a> of psychology experiment results.” Meanwhile an article in The Independent claimed that much of “<a href="http://www.independent.co.uk/news/science/study-reveals-that-a-lot-of-psychology-research-really-is-just-psychobabble-10474646.html">psychology research really is just psycho-babble</a>.”</p>
<p>And everywhere there was the term “failure to replicate,” a subtly sinister phrasing that makes nonreplication sound necessarily like a bad thing, as though “success” in replication were the goal of science. “Psychology can’t be trusted,” runs the implicit narrative here, “the people conducting these experiments have been wasting their time.”</p>
<p>Reactions like this tied themselves up in a logical confusion; to believe that nonreplication demonstrated the failure of psychology is incoherent, as it entails a privileging of this latest set of results over the earlier ones. This can’t be right: it makes no sense to put stock in a new set of experimental results if you think their main lesson is to cast doubt on <em>all</em> experimental findings.</p>
<p>Experiments should be considered in the aggregate, with conclusions most safely drawn from multiple demonstrations of any given finding. </p>
<p>Running experiments is like flipping a coin to establish whether it is biased. Flipping it 20 times, and finding it comes up heads for 17 of them, might start to raise your suspicions. But extreme results like this are actually more likely when the number of flips is lower. You would want to try that coin many more times before feeling confident enough to wager that something funny is going on. Failure to replicate your majority of heads in a sample of 100 flips would indicate just that you hadn’t flipped the coin enough to make a safe conclusion the first time around.</p>
<p>This need for aggregation is the basis of an <a href="http://www.theguardian.com/commentisfree/2015/aug/28/psychology-experiments-failing-replication-test-findings-science">argument advanced</a> by Stanford’s John Ioannidis, a medical researcher who <a href="http://doi.org/10.1371/journal.pmed.0020124">proposed 10 years ago</a> that most published research findings (not just those in psychology) are false. Ioannidis highlights the positive side of facing up to something he and many other people have suspected for a while. He also points out that psychology is almost certainly not alone among scientific disciplines. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=450&fit=crop&dpr=1 600w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=450&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=450&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=566&fit=crop&dpr=1 754w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=566&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=566&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">You can’t know if the study will produce similar results again and again unless you run it again and again.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/arripay/174695277">Tanya Hart</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<h2>Real crisis is we don’t try to replicate enough</h2>
<p>The fact is, psychology has long been aware that replication is a good idea. Its importance is evident in the longstanding practice of researchers creating systematic literature reviews and meta-analyses (statistical aggregations of existing published findings) to give one another broader understandings of the field. Researchers just haven’t been abiding by best practice. As psychologist Vaughan Bell <a href="http://mindhacks.com/2015/08/28/dont-call-it-a-comeback/">pointed out</a>, a big part of Nosek’s achievement was in the logistical challenge of getting such a huge study done with so many cooperating researchers.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=338&fit=crop&dpr=1 600w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=338&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=338&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Experiments shouldn’t be one and done – run once, then published as if set in stone.</span>
<span class="attribution"><span class="source">Maggie Villiger</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>This brings us to the <a href="http://nymag.com/scienceofus/2015/08/many-psychology-research-findings-may-be-false.html">actual nature of the crisis</a> revealed by the Science study; what Nosek and his colleagues showed is that psychologists need to be doing more to try to replicate their work if they want a better understanding of how much of it is reliable. Unfortunately, as journalist Ed Yong pointed out in his Atlantic coverage of <a href="http://www.theatlantic.com/health/archive/2015/08/psychology-studies-reliability-reproducability-nosek/402466/">the Nosek study</a> (and in <a href="http://www.theatlantic.com/notes/2015/09/sweeping-psychologys-problems-under-the-rug/403726/">a reply</a> to Barrett’s op-ed) there are several powerful professional disincentives to actually running the same experiments again. In a nutshell, the profession rewards publications and journals publish results which are new and counter-intuitive. The problem is compounded by the media, which tend to disseminate experimental findings as unquestionable “discoveries” or even God-given truths.</p>
<p>So though psychology (and very likely not only psychology) most certainly has something of a crisis on its hands, it is not a crisis of the discipline’s methodology or rules. Two of the study’s authors made some <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">suggestions for improvement</a> on The Conversation, including incentives for more open research practices and even obligatory openness with data and preregistration of experiments. These recommendations reiterate what methods specialists have said for years. Hopefully the discussion stirred up by <a href="https://osf.io/ezcuj/">Nosek and colleagues’ efforts</a> will also inspire others.</p>
<p>In essence, everyone agrees that experimental coin flipping is a reasonable way to proceed. This study exposed a flaw of the discipline’s sociology, of what people <em>actually</em> do and why they do it. Put another way, psychologists have already developed a perfectly effective system for conducting research; the problem is that so few of them really use it.</p><img src="https://counter.theconversation.com/content/47249/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Huw Green does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>The field of psychology is trying to absorb a recent big study that was able to replicate only 36 out of 100 major research papers. That finding is an issue, but maybe not for the reason you think.Huw Green, PhD Student and Trainee Clinical Psychologist at the Graduate Center, City University of New YorkLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/465962015-08-27T18:03:46Z2015-08-27T18:03:46ZWe found only one-third of published psychology research is reliable – now what?<figure><img src="https://images.theconversation.com/files/93195/original/image-20150827-368-orja2i.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">What does it mean if the majority of what's published in journals can't be reproduced?</span> <span class="attribution"><span class="source">Maggie Villiger</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span></figcaption></figure><p>The ability to repeat a study and find the same results twice is a prerequisite for building scientific knowledge. Replication allows us to ensure empirical findings are reliable and refines our understanding of when a finding occurs. It may surprise you to learn, then, that scientists do not often conduct – much less publish – attempted replications of existing studies.</p>
<p>Journals prefer to publish novel, cutting-edge research. And professional advancement is determined by making new discoveries, not painstakingly confirming claims that are already on the books. As one of our colleagues recently put it, “Running replications is fine for other people, but I have better ways to spend my precious time.”</p>
<p>Once a paper appears in a peer-reviewed journal, it acquires a kind of magical, unassailable authority. News outlets, and sometimes even scientists themselves, will cite these findings without a trace of skepticism. Such unquestioning confidence in new studies is likely undeserved, or at least premature.</p>
<p>A small but vocal contingent of researchers – addressing fields ranging from <a href="http://retractionwatch.com/?s=physics">physics</a> to <a href="http://doi.org/10.1038/nrd3439-c1">medicine</a> to <a href="http://economistsview.typepad.com/economistsview/2013/05/economics-needs-replication.html">economics</a> – has maintained that many, <a href="http://doi.org/10.1371/journal.pmed.0020124">perhaps most</a>, published studies are wrong. But how bad is this problem, exactly? And what features make a study more or less likely to turn out to be true?</p>
<p>We are two of the 270 researchers who together have just <a href="http://www.sciencemag.org/content/349/6251/aac4716">published in the journal Science</a> the first-ever large-scale effort trying to answer these questions by attempting to reproduce 100 previously published psychological science findings.</p>
<h2>Attempting to re-find psychology findings</h2>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=338&fit=crop&dpr=1 600w, https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=338&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=338&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/93197/original/image-20150827-358-1r7iij4.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">The results are bound and shelved – but are they reproducible?</span>
<span class="attribution"><span class="source">Maggie Villiger</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>Publishing together as the <a href="https://osf.io">Open Science Collaboration</a> and coordinated by social psychologist Brian Nosek from the <a href="http://centerforopenscience.org/">Center for Open Science</a>, research teams from around the world each ran a replication of a study published in three top psychology journals – Psychological Science; Journal of Personality and Social Psychology; and Journal of Experimental Psychology: Learning, Memory, and Cognition. To ensure the replication was as exact as possible, research teams obtained study materials from the original authors, and worked closely with these authors whenever they could.</p>
<p>Almost all of the original published studies (97%) had statistically significant results. This is as you’d expect – while many experiments fail to uncover meaningful results, scientists tend only to publish the ones that do.</p>
<p>What we found is that when these 100 studies were run by other researchers, however, only 36% reached statistical significance. This number is alarmingly low. Put another way, only around one-third of the rerun studies came out with the same results that were found the first time around. That rate is especially low when you consider that, once published, findings tend to be held as gospel.</p>
<p>The bad news doesn’t end there. Even when the new study found evidence for the existence of the original finding, the magnitude of the effect was much smaller — half the size of the original, on average.</p>
<p>One caveat: just because something fails to replicate doesn’t mean it isn’t true. Some of these failures could be due to luck, or poor execution, or an incomplete understanding of the circumstances needed to show the effect (scientists call these “moderators” or “boundary conditions”). For example, having someone practice a task repeatedly might improve their memory, but only if they didn’t know the task well to begin with. In a way, what these replications (and failed replications) serve to do is highlight the inherent uncertainty of any single study – original or new.</p>
<h2>More robust findings more replicable</h2>
<p>Given how low these numbers are, is there anything we can do to predict the studies that will replicate and those that won’t? The results from this <a href="https://osf.io/ezcuj/">Reproducibility Project</a> offer some clues.</p>
<p>There are two major ways that researchers quantify the nature of their results. The first is a p-value, which estimates the probability that the result was arrived at purely by chance and is a false positive. (Technically, the p-value is the chance that the result, or a stronger result, would have occurred even when there was no real effect.) Generally, if a statistical test shows that the p-value is lower than 5%, the study’s results are considered “significant” – most likely due to actual effects.</p>
<p>Another way to quantify a result is with an effect size – not how reliable the difference is, but how big it is. Let’s say you find that people spend more money in a sad mood. Well, <em>how much</em> more money do they spend? This is the effect size.</p>
<p>We found that the smaller the original study’s p-value and the larger its effect size, the more likely it was to replicate. Strong initial statistical evidence was a good marker of whether a finding was reproducible.</p>
<p>Studies that were rated as more challenging to conduct were less likely to replicate, as were findings that were considered surprising. For instance, if a study shows that reading lowers IQs, or if it uses a very obscure and unfamiliar methodology, we would do well to be skeptical of such data. Scientists are often rewarded for delivering results that dazzle and defy expectation, but extraordinary claims require extraordinary evidence.</p>
<p>Although our replication effort is novel in its scope and level of transparency – the methods and data for all replicated studies are <a href="https://osf.io/ezcuj/">available online</a> – they are consistent with previous work from other fields. Cancer biologists, for instance, have reported replication rates as low as <a href="http://www.nature.com/nature/journal/v483/n7391/full/483531a.html#t1">11%</a>-<a href="http://doi.org/10.1038/nrd3439-c1">25</a>%. </p>
<h2>We have a problem. What’s the solution?</h2>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=895&fit=crop&dpr=1 600w, https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=895&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=895&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1125&fit=crop&dpr=1 754w, https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1125&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/93196/original/image-20150827-364-1t8d6wo.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1125&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Recruitment of volunteers for new studies is ongoing. What about revisiting past findings?</span>
<span class="attribution"><span class="source">Maggie Villiger</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>Some conclusions seem warranted here.</p>
<p>We must stop treating single studies as unassailable authorities of the truth. Until a discovery has been thoroughly vetted and repeatedly observed, we should treat it with the measure of skepticism that scientific thinking requires. After all, the truly scientific mindset is critical, not credulous. There is a place for breakthrough findings and cutting-edge theories, but there is also merit in the slow, systematic checking and refining of those findings and theories.</p>
<p>Of course, adopting a skeptical attitude will take us only so far. We also need to provide incentives for reproducible science by rewarding those who conduct replications and who conduct replicable work. For instance, at least one top journal has begun to give special “<a href="http://www.psychologicalscience.org/index.php/publications/journals/psychological_science/badges">badges</a>” to articles that make their data and materials available, and the Berkeley Initiative for Transparency in the Social Sciences has <a href="http://www.bitss.org/2015/05/13/prize-competition/">established a prize</a> for practicing more transparent social science.</p>
<p>Better research practices are also likely to ensure higher replication rates. There is already evidence that <a href="http://doi.org/10.1371/journal.pone.0132382">taking certain concrete steps</a> – such as making hypotheses clear prior to data analysis, openly sharing materials and data, and following transparent reporting standards – decreases false positive rates in published studies. Some funding organizations are already demanding <a href="https://clinicaltrials.gov/ct2/manage-recs/background">hypothesis registration</a> and <a href="http://my.americanheart.org/professional/Research/FundingOpportunities/Open-Science-Policy-Statements-for-AHA-Funded-Research_UCM_461225_Article.jsp">data sharing</a>.</p>
<p>Although perfect replicability in published papers is an unrealistic goal, current replication rates are unacceptably low. The first step, as they say, is admitting you have a problem. What scientists and the public now choose to do with this information remains to be seen, but our collective response will guide the course of future scientific progress.</p><img src="https://counter.theconversation.com/content/46596/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Researchers from around the globe tried to replicate 100 published psychology studies. They were successful on only 36.Elizabeth Gilbert, PhD Student in Psychology, University of VirginiaNina Strohminger, Postdoctoral Fellow at the School of Management, Yale UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/288232014-07-09T05:16:41Z2014-07-09T05:16:41ZWhat lesson do rising retraction rates hold for peer review?<figure><img src="https://images.theconversation.com/files/53203/original/89z6qth4-1404747954.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Reading journals is not fun anymore.</span> <span class="attribution"><a class="source" href="http://en.wikipedia.org/wiki/Jack_Nicholson#mediaviewer/File:Little_Shop_of_Horrors_Nicholson.JPG">Roger Corman</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span></figcaption></figure><p>In January, Haruko Obokata and colleagues published two papers in the journal Nature suggesting that a simple acid bath can convert differentiated cells back to a stem-cell-like state. This finding, if true, would be revolutionary. Last week, however, after five months of debate among peers, the papers have been <a href="http://www.nature.com/news/stap-retracted-1.15488">retracted</a>.</p>
<p>This incident is part of a larger trend. The rate of retractions of scientific papers <a href="http://dx.doi.org/10.1038/478026a">has been growing</a> over the past decade, suggestive to some of a crisis of confidence in science. Can we no longer trust the scientific literature? Is <a href="https://theconversation.com/explainer-what-is-peer-review-27797">peer review</a> dysfunctional? </p>
<p>Retractions reveal both science’s weakness and its strength. Science frequently goes wrong; that’s its weakness. Then science corrects itself; that’s its strength. And yet there’s a lesson in the rising rate of retractions.</p>
<h2>Amplifying the noise in the system</h2>
<p>When a scientific finding is published, our major indicator of its reliability and importance is the prestige of the journal where it appears. So when Obokata’s findings appeared in Nature, one of the top journals, the world paid attention. The story was reported in mass media across the globe. It is difficult to estimate the cost of confusing the world with an incorrect a message at this scale.</p>
<p>The problem is not that science, for five months, was in a state of confusion about Obokata’s claims. Confusion in science is part of the process of working things out. The problem is that the message of the paper was amplified to global visibility, before the field could resolve its confusion.</p>
<p>In the current system of prepublication peer review, a paper is evaluated before publication by a small number of other scientists (typically three or four). Such reviews formed the basis for presenting Obokata’s claims, as fact, to the whole world.</p>
<p>When one of us makes a claim (by submitting a paper), it would seem wise not to blurt it out to the whole world after just four of us (the peer reviewers) have had a look at it.</p>
<p>There’s a clear lesson in the Obokata story and the general trend of rising retraction rates. It was prepublication peer review that failed to catch the error. And it was postpublication peer review, the open debate on the web, that corrected the path of science.</p>
<p>Nature, Science, and other <a href="https://theconversation.com/how-to-break-free-from-the-stifling-grip-of-luxury-journals-21669">prestige journals</a> are run by talented people who have every incentive to publish the best research. Their review process is professional and their reviewers are highly qualified. However, three or four reviewers asked to comment within a couple of weeks cannot achieve the breadth or depth of evaluation that an open discussion by hundreds of scientists can achieve over several months.</p>
<p>We need this sort of open evaluation among peers before we can justify alerting the entire world. The aura of prestige journals grossly overstates the actual confidence we can have in a scientific result when it first appears. Slight tweaks to the review process as discussed in a <a href="http://www.nature.com/news/stap-retracted-1.15488">Nature Editorial</a> reflecting on the Obakata story will not solve the problem. Even dramatic changes, such as doubling the number of reviewers or requiring independent replication, would fall short – as long as peer review is restricted to the prepublication phase.</p>
<h2>Prepublication peer review is inadequate</h2>
<p>Prepublication peer review is flawed for three reasons. First, it is restricted to a small number of people, the editors and peer reviewers. To bring the brain power of the entire community of peers into the evaluation process, the paper has first to be made publicly available – that is, published. Second, prepublication peer review is conducted in secret. Since the paper is not yet published, the review process as well is hidden from public scrutiny. Typically, the reviewers are anonymous and their reviews secret. There is thus no strong disincentive to self-serving or subtly biased reviewing. Third, the review process delays publication. When conducted quickly, it may lack thoroughness. When given more time, it slows down the progress of science. The present model suffers from both of these drawbacks.</p>
<p>Establishing the reliability of a finding is only half the challenge. The other half is assessing the implications and importance of a study. Prepublication peer review falls short on both counts. Understanding the full implications of a study, too, requires an open peer debate.</p>
<p>We’ve inherited the current system from the pre-internet age. Back when articles needed to be printed on physical paper, we needed to filter before publication to control the costs. Today the internet enables us to “publish then filter”, to use Clay Shirky’s useful phrase. This will revolutionise scientific publishing. For the moment, however, the current system is held in place by historical inertia, our habits, and the financial interests of the publishing industry.</p>
<h2>Open evaluation</h2>
<p>The emerging alternative model is <a href="http://www.frontiersin.org/computational_neuroscience/researchtopics/beyond_open_access_visions_for/137">open evaluation (OE)</a>, a transparent public process of peer review and rating after publication. All scientific papers, in such a system, would be instantly published in an open access model, where everyone can read them. They would then be vetted and ranked postpublication in an ongoing fashion.</p>
<p>The transition is not going to be easy or swift, but recent developments and a growing number of startup companies are moving in the right direction. Pubmed, a respository of science publications, has established a forum called <a href="http://www.ncbi.nlm.nih.gov/pubmedcommons/">PubMed Commons</a>, where scientists can leave comments on any paper. <a href="http://www.ploslabs.org/openevaluation/">PLOS Open Evaluation</a> provides a web-based system for sampling opinions on papers through ratings. New journals including <a href="http://f1000research.com/">F1000 Research</a> and <a href="https://www.scienceopen.com/home">ScienceOpen</a> rely entirely on postpublication peer review.</p>
<p>Once open evaluation ratings on published papers become available, scientists and journalists will no longer be dependent on the impact factor of the journal as the only immediately available indication to a new paper’s reliability and importance.</p>
<p>A decade from now, Nature, or its successor in prestige science publishing, might pick the most exciting among previously published studies that have fared well through months of open evaluation. With the evaluation taken care of, the publishers will focus on helping authors communicate the findings to an audience that extends to other fields and beyond science. Had Obokata and colleagues published their findings first for their peers, the flaws of the papers would have been exposed before alerting the world. It would have saved us a lot of confusion.</p><img src="https://counter.theconversation.com/content/28823/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Nikolaus Kriegeskorte has edited a collection of visions for open evaluation and postpublication peer review. He frequently argues in favour of reform for scientific publishing. He has served in editorial roles for Frontiers and PLoS Computational Biology, and is on the editorial board of ScienceOpen. He receives funding from the UK Medical Research Council, the European Research Council, and the Wellcome Trust.</span></em></p>In January, Haruko Obokata and colleagues published two papers in the journal Nature suggesting that a simple acid bath can convert differentiated cells back to a stem-cell-like state. This finding, if…Nikolaus Kriegeskorte, Programme leader, MRC Cognition and Brain Sciences Unit, University of CambridgeLicensed as Creative Commons – attribution, no derivatives.