tag:theconversation.com,2011:/au/topics/replication-20230/articlesReplication – The Conversation2023-07-31T05:49:54Ztag:theconversation.com,2011:article/2107002023-07-31T05:49:54Z2023-07-31T05:49:54ZViral room-temperature superconductor claims spark excitement – and skepticism<figure><img src="https://images.theconversation.com/files/540083/original/file-20230731-25689-3y1jr1.jpg?ixlib=rb-1.1.0&rect=0%2C0%2C2787%2C1671&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><a class="source" href="https://sciencecast.org/casts/suc384jly50n">Hyun-Tak Kim</a></span></figcaption></figure><p>Last week, a group of South Korean physicists made a startling claim. In <a href="https://arxiv.org/abs/2307.12037">two</a> <a href="https://arxiv.org/abs/2307.12008">papers</a> uploaded to the arXiv preprint server, they say they have created a material that “opens a new era for humankind”.</p>
<p>LK-99, a lead-based compound, is purportedly a room-temperature, ambient-pressure superconductor. Such a material, which conducts electricity without any resistance under normal conditions, could have huge implications for energy generation and transmission, transport, computing and other areas of technology.</p>
<p>The papers have sparked wild enthusiasm online, and several efforts to <a href="https://www.science.org/content/article/spectacular-superconductor-claim-making-news-here-s-why-experts-are-doubtful">replicate the work</a>. At the same time, there are <a href="https://www.asiafinancial.com/korea-superconductor-papers-published-without-consent-yonhap">reports</a> of disputes among the Korean researchers over whether the research should have been released at all.</p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1684044616528453633"}"></div></p>
<h2>Why superconductors are so super</h2>
<p>When an electric current flows through an ordinary conductor like a copper wire, the electrons bump into atoms as they jostle along. As a result, the electrons lose some energy and the wire heats up.</p>
<p>In a superconductor, electrons move without any resistance. Superconducting wires can transmit electricity without losing energy, and superconducting magnets are powerful enough to levitate trains and contain the fierce plasmas in fusion reactors.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/room-temperature-superconductors-could-revolutionize-electronics-an-electrical-engineer-explains-the-materials-potential-201849">Room-temperature superconductors could revolutionize electronics – an electrical engineer explains the materials' potential</a>
</strong>
</em>
</p>
<hr>
<p>However, all known superconductors require very low temperatures (typically lower than –100 °C) or extremely high pressures (more than 100,000 times ordinary atmospheric pressure). These restrictions make superconductors expensive and impractical for many applications.</p>
<p>Several teams of researchers have claimed to detect room-temperature superconductivity in various substances in the past, but none of the claims have withstood scrutiny. As recently as last week, a superconductivity paper by American physicist Ranga Dias was <a href="https://www.nature.com/articles/d41586-023-02401-2">retracted</a> amid suspicions of data fabrication. </p>
<p>So while a room-temperature superconductor would be an amazing discovery, we should meet the new claims with some skepticism.</p>
<h2>Bold claims</h2>
<p>The South Korean researchers say LK-99 can be made in a baking process that combines the minerals lanarkite (Pb₂SO₅) and copper phosphide (Cu₃P). They say the resulting material shows two key signs of superconductivity at normal air pressure and at temperatures up to 127 °C: zero resistance and magnetic levitation.</p>
<p>They propose a plausible theory of how LK-99 might display room-temperature superconductivity, but have not provided definite experimental evidence. The data presented in the papers appear inconclusive.</p>
<p>One of the signatures of a superconductor is the Meissner effect, which causes it to levitate when placed above a magnet. </p>
<p>In a <a href="https://sciencecast.org/casts/suc384jly50n">video demonstration</a>, the researchers position a piece of LK-99 over a magnet. One edge of the flat disk of LK-99 rises, but the other edge appears to maintain contact with the magnet. </p>
<p>We would expect a superconductor to display full levitation and also “quantum locking” which keeps it in a fixed position relative to the magnet. In a charitable interpretation, the behaviour we see in the video may be due to imperfections in the sample, meaning only part of the sample becomes superconductive.</p>
<p>So it is too early to say we have been presented with compelling evidence for room-temperature superconductivity.</p>
<h2>What’s next</h2>
<p>At present, all we know about LK-99 comes from the two arXiv papers, which have not been peer-reviewed. Both papers present similar measurements, though the presentation is unconventional. However, there are some differences in the content, and also in authorship, which does not inspire confidence.</p>
<p>So what happens now? The processes of science swing into action. </p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1684256538142154752"}"></div></p>
<p>Experts will closely review the papers. Researchers at other laboratories will attempt to reproduce the experiments described in the papers, and see whether they end up with a room-temperature superconductor. </p>
<p>These crucial steps are necessary to establish the validity and reliability of the LK-99 claims. If the claims are validated and confirmed, it could mark one of the most groundbreaking advancements in physics and materials engineering in the past few decades. </p>
<p>However, until the research undergoes rigorous review and testing, we should approach the claims with caution. We will all be awaiting the outcome of the verification process with great interest.</p><img src="https://counter.theconversation.com/content/210700/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Mahboobeh Shahbazi does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Room-temperature superconductors could transform technology – but the latest, much-hyped claims should be approached with caution.Mahboobeh Shahbazi, Senior Research Fellow, Materials Science, Queensland University of TechnologyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1377512020-05-06T12:20:11Z2020-05-06T12:20:11ZRemdesivir explained – what makes this drug work against viruses?<figure><img src="https://images.theconversation.com/files/332156/original/file-20200503-42903-l663fn.jpg?ixlib=rb-1.1.0&rect=0%2C18%2C4210%2C2746&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Remdesivir is an experimental medicine that is showing promise in clinical trials for COVID-19. </span> <span class="attribution"><span class="source">Photo by ULRIC PERREY/POOL/AFP via Getty Images</span></span></figcaption></figure><p>With the <a href="https://www.fda.gov/media/137564/download">FDA approving Gilead’s Remdesivir</a> as an emergency use treatment for the most acute cases of COVID-19, many people are wondering what type of a drug it is. </p>
<p>Remdesivir is a member of one of the oldest and most important classes of drugs – known as nucleoside analogue. Currently there are more than 30 of these types of drugs that have been approved for use in treating viruses, cancers, parasites, as well as bacterial and fungal infections, with many more currently in clinical and preclinical trials.</p>
<p><a href="https://chemistry.umbc.edu/seley-radtke-lab/">I am a medicinal chemist</a> who has worked in design and synthesis of these important drug treatments for over 30 years. I have written numerous reviews over the years about these drugs and their structure and function, and as a result have had many inquiries lately from friends, family and others not in the field asking me to explain what exactly is it about Remdesivir that makes it so effective, but also why it is so interesting. Understanding why means digging into the biochemistry of this class of drugs.</p>
<h2>Fake genetic building blocks</h2>
<p>The reason nucleoside analogues and a similar group called nucleotide analogues are so effective is that they resemble the naturally occurring molecules known as nucleosides – cytidine, thymidine, uridine, guanosine and adenosine. These are the essential building blocks for the DNA and RNA that carry our genetic information and play critical roles in our body’s biological processes. </p>
<p>Slight differences in the chemical structure of these analogues from naturally occurring compounds make them effective as drugs. If an organism like a virus incorporates a nucleoside analogue into its genetic material, rather than the real thing, even small changes to the structure of these building blocks prevent the regular chemistry from happening and ultimately foils the ability of the virus to replicate. </p>
<p><a href="http://doi.org/10.1016/j.antiviral.2018.04.004">The basic structure of a nucleoside</a> includes a sugar group and a base (A, C, G, T or U), and in the case of a nucleotide, a group containing a phosphate which is a collection of oxygen and phosphorus atoms. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/332157/original/file-20200503-42918-pj3qd6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/332157/original/file-20200503-42918-pj3qd6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=565&fit=crop&dpr=1 600w, https://images.theconversation.com/files/332157/original/file-20200503-42918-pj3qd6.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=565&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/332157/original/file-20200503-42918-pj3qd6.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=565&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/332157/original/file-20200503-42918-pj3qd6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=710&fit=crop&dpr=1 754w, https://images.theconversation.com/files/332157/original/file-20200503-42918-pj3qd6.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=710&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/332157/original/file-20200503-42918-pj3qd6.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=710&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Every building block of DNA is made from three parts: a sugar, a base (A, C, G, or T) and a phosphate group. Every building block of RNA is made from (A, C, G, or U).</span>
<span class="attribution"><span class="source">ttsz / Getty Images</span></span>
</figcaption>
</figure>
<p>The first nucleoside analogues were approved for medicinal use in the 1950s. The early nucleosides had only simple modifications, typically either to the sugar or the base, while today’s nucleosides, such as Remdesivir, typically have several modifications to their structure. <a href="http://doi.org/10.1016/j.antiviral.2018.04.004">These modifications are essential to their therapeutic activity</a>. </p>
<h2>How does Remdesivir work as antiviral therapeutic?</h2>
<p>This activity occurs because nucleoside/tide analogues mimic the structure of a natural nucleoside or nucleotide such that they are recognized by, for example, viruses. Due to those structural modifications, however, they stop or interrupt viral replication, which stops the virus from multiplying and infecting more cells in the body.</p>
<p>As a result, they are known as direct-acting antivirals, and this is the case for <a href="http://doi.org/10.1016/j.antiviral.2018.11.016">Remdesivir, which works by blocking the coronavirus’s</a> RNA polymerase – one of the key enzymes that this virus needs to replicate its genetic material (RNA) and proliferate in our bodies. Remdesivir works when the enzyme replicating the genetic material for a new generation of viruses accidentally grabs this nucleoside analogue rather than the natural molecule and incorporates it into the growing RNA strand. Doing this essentially blocks the rest of the RNA from being replicated; this in turn prevents the virus from multiplying. </p>
<p>The drug Remdesivir is basically an altered version of the natural building block adenosine – which is essential for DNA and RNA. Comparing the structure of Remdesivir with adenosine, one can see there are three key modifications that make it effective. </p>
<p>The first is that Remdesivir, as it is administered, is not the actual active drug; it is actually a “prodrug,” meaning it must be modified once in the body before it becomes an active drug. Prodrugs are used for many reasons, including protecting a drug until it reaches its site of action. The active form of Remdesivir contains three phosphate groups; it is this form that is recognized by the virus’s RNA polymerase enzyme. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=182&fit=crop&dpr=1 600w, https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=182&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=182&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=228&fit=crop&dpr=1 754w, https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=228&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/332108/original/file-20200502-42929-5gizi3.jpeg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=228&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">A naturally occurring nucleotide (left) which is a building block of RNA and DNA and Remdesivir (right) which is a variation on its natural counterpart.</span>
<span class="attribution"><span class="source">Katherine Seley-Radtke</span>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<p>The second important modification on Remdesivir is the carbon-nitrogen (CN) group attached to the sugar. Once Remdesivir is incorporated into the RNA growing chain, the presence of this CN group causes the shape of the sugar to pucker, which, in turn, distorts the shape of the RNA strand such that only three more nucleotides can be added. <a href="https://doi.org/10.2165/00003495-200969020-00002">This terminates the production of the RNA strand</a> and is what ultimately sabotages the replication of the virus. </p>
<p>The third important structural feature which makes Remdesivir differ from adenosine is the change of one particular chemical bond on the molecule. Rather than a bond linking a carbon and nitrogen atoms, chemists replaced the nitrogen with another carbon, creating a carbon-carbon bond. This is critical to the success of this drug because coronaviruses have a special enzyme that recognizes unnatural nucleosides and clips them out. But by changing this chemical bond, Remdesivir cannot be removed by the enzyme, allowing it to stay in the growing chain and block replication.</p>
<h2>Remdesivir trials</h2>
<p>Remdesivir originally was found during a drug discovery program at Gilead to search for inhibitors of the hepatitis C virus, which is another RNA virus. Although Gilead ultimately selected a different nucleoside analogue for treatment of hepatitis the company tested the drug to see if it was effective against other RNA viruses. Remdesivir <a href="https://doi.org/10.1038/nature17180">exhibited potent activity against Ebola</a> and <a href="http://doi.org/10.1073/pnas.1922083117">Middle Eastern respiratory virus</a>, <a href="http://doi.org/10.1126/scitranslmed.aal3653">among others</a>. </p>
<p>Now the drug is being tested against the SAR-CoV-2 virus in <a href="https://www.niaid.nih.gov/news-events/nih-clinical-trial-remdesivir-treat-covid-19-begins">the first clinical trial launched in the United States.</a> </p>
<p><a href="https://www.niaid.nih.gov/news-events/nih-clinical-trial-shows-remdesivir-accelerates-recovery-advanced-covid-19">According to the NIH</a>, patients who received Remdesivir had a faster recovery compared to those who received placebo; 11 days compared with 15 days for those who received the placebo. “Results also suggested a survival benefit, with a mortality rate of 8.0% for the group receiving Remdesivir versus 11.6% for the placebo group,” according to the NIH press release.</p>
<p>While these results are preliminary, there are <a href="https://www.gilead.com/purpose/advancing-global-health/covid-19/remdesivir-clinical-trials#">a plethora of clinical trials underway across the world</a>. Regardless, a certain amount of caution is still needed. As noted by <a href="https://www.today.com/video/dr-anthony-fauci-remdesivir-is-a-very-important-first-step-in-fighting-coronavirus-82800197863">Dr. Anthony Fauci on NBC’s “Today” show</a>, “the antiviral drug Remdesivir is the first step in what we project will be better and better drugs coming along” to treat COVID-19, but cautioned, “This is not the total answer.” </p>
<p>I share this view with many other scientists in the field. No matter what those results ultimately show, Remdesivir will mostly certainly be part of a cocktail of drugs, just as is standard for treating other viruses such as HIV and hepatitis C.</p>
<p>A combination, or cocktail, of drugs will provide a more effective and more complete therapy that blocks the virus from replicating. The other benefit of such a drug cocktail is that it lowers the chance the virus will develop resistance to the therapy. In the meantime, these early results for Remdesivir are proving to be an important source of hope for many of us across the world as we wait for this pandemic to subside.</p>
<p>[<em>You need to understand the coronavirus pandemic, and we can help.</em> <a href="https://theconversation.com/us/newsletters?utm_source=TCUS&utm_medium=inline-link&utm_campaign=newsletter-text&utm_content=upper-coronavirus-help">Read The Conversation’s newsletter</a>.]</p><img src="https://counter.theconversation.com/content/137751/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Dr. Katherine Seley-Radtke has previously consulted for Gilead Sciences and owns Gilead Sciences stock. She currently receives funding from the National Institutes of Health (NIH), the National Institute of General Medicine (NIGMS), the National Institute of Allergies and Infectious Diseases (NIAID). She is the President-elect of the International Society for Antiviral Research (ISAR) and is the Secretary and former President of the International Society of Nucleosides, Nucleotides & Nucleic Acids (IS3NA), both non-profit scientific professional societies.</span></em></p>Gilead’s drug Remdesivir showed preliminary positive results in clinical trials. But what is this drug and how, exactly, does it work?Katherine Seley-Radtke, Professor of Chemistry and Biochemistry and President-Elect of the International Society for Antiviral Research, University of Maryland, Baltimore CountyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1255682019-10-24T19:00:48Z2019-10-24T19:00:48ZPredicting research results can mean better science and better advice<figure><img src="https://images.theconversation.com/files/298477/original/file-20191024-31500-7c7lpw.jpg?ixlib=rb-1.1.0&rect=50%2C0%2C5635%2C3753&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Putting scientific results under the microscope before they are even collected could help improve science as a whole.</span> <span class="attribution"><span class="source">Konstantin Kolosov/Shutterstock</span></span></figcaption></figure><p>We ask experts for advice all the time. A company might ask an economist for advice on how to motivate its employees. A government might ask what the effect of a policy reform will be.</p>
<p>To give the advice, experts often would like to draw on the results of an experiment. But they don’t always have relevant experimental evidence.</p>
<p>Collecting expert predictions about research results could be a powerful new tool to help improve science - and the advice scientists give.</p>
<h2>Better science</h2>
<p>In the past few decades, academic rigour and transparency, particularly in the social sciences, have greatly improved. </p>
<p>Yet, as Australia’s Chief Scientist Alan Finkel <a href="https://theconversation.com/there-is-a-problem-australias-top-scientist-alan-finkel-pushes-to-eradicate-bad-science-123374">recently argued</a>, there is still much to be done to minimise “bad science”. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/there-is-a-problem-australias-top-scientist-alan-finkel-pushes-to-eradicate-bad-science-123374">'There is a problem': Australia's top scientist Alan Finkel pushes to eradicate bad science</a>
</strong>
</em>
</p>
<hr>
<p>He recommends changes to the way research is measured and funded. Another increasingly <a href="https://www.sciencemag.org/news/2018/09/more-and-more-scientists-are-preregistering-their-studies-should-you">common approach</a> is to conduct randomised controlled trials and pre-register studies to avoid bias in which results are reported.</p>
<p>Expert predictions can be yet another tool for making research stronger, as my co-authors Stefano DellaVigna, Devin Pope and I argue in a new article published in <a href="https://science.sciencemag.org/content/366/6464/428">Science</a>.</p>
<h2>Why predictions?</h2>
<p>The way we interpret research results depends on what we already believe. For example, if we saw a study claiming to show that smoking was healthy, we would probably be pretty sceptical.</p>
<p>If a result surprises experts, that fact itself is informative. It could suggest that something may have been wrong with the study design. </p>
<p>Or, if the study was well-designed and the finding replicated, we might think that result fundamentally changed our understanding of how the world works.</p>
<p>Yet currently researchers rarely collect information that would allow them to compare their results with what the research community believed beforehand. This makes it hard to interpret the novelty and importance of a result.</p>
<p>The academic publication process is also plagued by bias against publishing insignificant, or “null”, results. </p>
<p>The collection of advance forecasts of research results could combat this bias by making null results more interesting, as they may indicate a departure from accepted wisdom.</p>
<h2>Changing minds</h2>
<p>As well as directly improving the interpretation of research results, collecting advance forecasts can help us understand how people change their minds.</p>
<p>For example, my colleague Aidan Coville and I <a href="http://evavivalt.com/wp-content/uploads/How-Do-Policymakers-Update1.pdf">collected advance forecasts from policymakers</a> to study what effect academic research results had on their beliefs. We found in general they were more receptive to “good news” than “bad news” and ignored uncertainties in results. </p>
<p>Forecasts can also inform us as to which potential studies could most improve policy decisions.</p>
<p>For example, suppose a research team has to pick one of ten interventions to study. For some of the interventions, we are pretty sure what a study would find, and a new study would be unlikely to change our minds. For others, we are less sure, but they are unlikely to be the best intervention.</p>
<p>If predictions were collected in advance, they could tell us which intervention to study to have the biggest policy impact.</p>
<h2>Testing forecasts</h2>
<p>In the long run, if expert forecasts can be shown to be fairly accurate,
they could provide some support for policy decisions where rigorous studies can’t be conducted.</p>
<p>For example, Stefano DellaVigna and Devin Pope <a href="https://www.journals.uchicago.edu/doi/10.1086/699976?mobileUi=0&">collected forecasts</a> about how different incentives change the amount of effort people put into completing a task.</p>
<p>As you can see in the graph below, the forecasts were not perfect (a dot on the dashed diagonal line would represent a perfect match of forecast and result). But there does appear to be some correlation between the aggregated forecasts and the results.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=436&fit=crop&dpr=1 600w, https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=436&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=436&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=548&fit=crop&dpr=1 754w, https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=548&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/298204/original/file-20191022-55674-1kr6ucb.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=548&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Reproduced with permission from DellaVigna and Pope.</span>
</figcaption>
</figure>
<h2>A central place for forecasts</h2>
<p>To make the most of forecasts of research results, they should be collected systematically. </p>
<p>Over time, this would help us assess how accurate individual forecasters are, teach us how best to aggregate forecasts, and tell us which types of results tend to be well predicted. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/science-is-best-when-the-data-is-an-open-book-49147">Science is best when the data is an open book</a>
</strong>
</em>
</p>
<hr>
<p>We built a platform that researchers can use to collect forecasts about their experiments from researchers, policymakers, practitioners, and other important audiences. The beta website can be viewed <a href="http://socialscienceprediction.org/">here</a>. </p>
<p>While we are focusing first on our own discipline – economics – we think such a tool should be broadly useful. We would encourage researchers in any academic field to consider collecting predictions of research results.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=341&fit=crop&dpr=1 600w, https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=341&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=341&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=428&fit=crop&dpr=1 754w, https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=428&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/298205/original/file-20191022-55665-xrz1q2.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=428&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">The Social Science Prediction Platform, https://socialscienceprediction.org/.</span>
</figcaption>
</figure>
<p>There are many potential uses for predictions of research results beyond those described here. Many other academics are also exploring this area, such as the <a href="https://www.replicationmarkets.com/">Replication Markets</a> and <a href="https://replicats.research.unimelb.edu.au/">repliCATS</a> projects that are part of a large <a href="https://www.darpa.mil/program/systematizing-confidence-in-open-research-and-evidence">research initiative</a> on replication. </p>
<p>The multiple possible uses of research forecasts gives us confidence that a more rigorous and systematic treatment of prior beliefs can greatly improve the interpretation of research results and ultimately improve the way we do science.</p><img src="https://counter.theconversation.com/content/125568/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Eva Vivalt receives funding from the Alfred P. Sloan Foundation and the John Mitchell Economics of Poverty Lab. </span></em></p>Researchers rarely collect information that lets them to compare their results with what was believed beforehand. If they did, it could help spot new or important findings more readily.Eva Vivalt, Research Fellow and Lecturer, Australian National UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1141612019-04-01T10:40:12Z2019-04-01T10:40:12ZIs it the end of ‘statistical significance’? The battle to make science more uncertain<figure><img src="https://images.theconversation.com/files/266165/original/file-20190327-139371-15nimd0.jpg?ixlib=rb-1.1.0&rect=3%2C0%2C2625%2C1552&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Some scientists think it's time to hang up statistical significance.</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/white-robe-hanging-on-door-laboratory-321548114">mariakraynova/Shutterstock.com</a></span></figcaption></figure><p>The scientific world is abuzz following recommendations by two of the most prestigious scholarly journals – <a href="https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913">The American Statistician</a> and <a href="https://www.nature.com/articles/d41586-019-00857-9">Nature</a> – that the term “<a href="https://en.wikipedia.org/wiki/Statistical_significance">statistical significance</a>” be retired.</p>
<p>In their introduction to the special issue of The American Statistician on the topic, the journal’s editors urge “moving to a world beyond ‘p<0.05,’” <a href="http://www.haghish.com/resources/materials/Statistical_Methods_for_Research_Workers.pdf">the famous 5 percent threshold</a> for determining whether a study’s result is statistically significant. If a study passes this test, it means that the probability of a result being due to chance alone is less than 5 percent. This has often been understood to mean that the study is worth paying attention to.</p>
<p>The journal’s basic message – but not necessarily the consensus of the 43 articles in this issue, one of which <a href="https://doi.org/10.1080/00031305.2018.1518788">I contributed</a> – was that scientists first and foremost should “embrace uncertainty” and “be thoughtful, open and modest.”</p>
<p>While these are fine qualities, I believe that scientists must not let them obscure the precision and rigor that science demands. Uncertainty is inherent in data. If scientists further weaken the already very weak threshold of 0.05, then that would inevitably make scientific findings more difficult to interpret and less likely to be trusted. </p>
<h2>Piling difficulty on top of difficulty</h2>
<p>In the traditional practice of science, a scientist generates a hypothesis and designs experiments to collect data in support of hypotheses. He or she then collects data and performs statistical analyses to determine if the data did in fact support the hypothesis. </p>
<p>One standard statistical analysis is the <a href="https://en.wikipedia.org/wiki/P-value">p-value</a>. This generates a number between 0 and 1 that indicates strong, marginal or weak support of a hypothesis.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=396&fit=crop&dpr=1 600w, https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=396&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=396&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=498&fit=crop&dpr=1 754w, https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=498&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/266547/original/file-20190329-70999-ktz14q.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=498&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">A quick guide to p-values.</span>
<span class="attribution"><span class="source">Repapetilto/Wikimedia</span>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<p>But I worry that abandoning evidence-driven standards for these judgments will make it even more difficult to design experiments, much less assess their outcomes. For instance, how could one even determine an appropriate sample size without a targeted level of precision? And how are research results to be interpreted? </p>
<p>These are important questions, not just for researchers at funding or regulatory agencies, but for anyone whose daily life is influenced by statistical judgments. That includes anyone who takes medicine or undergoes surgery, drives or rides in vehicles, is invested in the stock market, has life insurance or depends on accurate weather forecasts… and the list goes on. Similarly, many regulatory agencies rely on statistics to make decisions every day.</p>
<p>Scientists must have the language to indicate that a study, or group of studies, provided significant evidence in favor of a relationship or an effect. Statistical significance is the term that serves this purpose.</p>
<h2>The groups behind this movement</h2>
<p>Hostility to the term “statistical significance” arises from two groups.</p>
<p>The first is largely made up of scientists disappointed when their studies produce p=0.06. In other words, those whose studies just don’t make the cut. These are largely <a href="https://www.researchgate.net/publication/319880949_Justify_your_alpha">scientists who find the 0.05 standard too high</a> a hurdle for getting published in the scholarly journals that are a major source of academic knowledge – as well as tenure and promotion. </p>
<p>The second group is concerned over the <a href="https://theconversation.com/a-statistical-fix-for-the-replication-crisis-in-science-84896">failure to replicate scientific studies</a>, and they blame significance testing in part for this failure.</p>
<p>For example, <a href="https://doi.org/10.1126/science.aac4716">a group of scientists</a> recently repeated 100 published psychology experiments. Ninety-seven of the 100 original studies reported a statistically significant finding (p<0.05), but only 36 of the repeated experiments were able to also achieving a significant result. </p>
<p>The failure of so many studies to replicate can be partially blamed on publication bias, which results when only significant findings are published. Publication bias causes scientists to overestimate the <a href="https://doi.org/10.4065/75.12.1284">magnitude of an effect</a>, such as the relationship between two variables, making replication less likely.</p>
<p>Complicating the situation even further is the fact that <a href="https://doi.org/10.1073/pnas.1313476110">recent research</a> shows that the p-value cutoff doesn’t provide much evidence that a real relationship has been found. In fact, in replication studies in social sciences, it now appears that p-values close to the standard threshold of 0.05 probably mean that a scientific claim is wrong. It’s only when the p-value is much smaller, maybe less than 0.005, that scientific claims are likely to <a href="https://www.nature.com/articles/s41562-017-0189-z">show a real relationship</a>. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/266166/original/file-20190327-139345-fw20gp.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">What do the data really say?</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/african-american-businessman-using-devices-business-1017688327">fizkes/shutterstock.com</a></span>
</figcaption>
</figure>
<h2>The confusion leading to this movement</h2>
<p>Many nonstatisticians confuse p-value with the <a href="https://en.wikipedia.org/wiki/Probability">probability</a> that no discovery was made.</p>
<p>Let’s look at an example from the Nature article. <a href="https://doi.org/10.1016/j.ijcard.2014.09.205">Two studies</a> examined the increased risk of disease after taking a drug. Both studies estimated that patients had a 20 percent higher risk of getting the disease if they take the drug than if they didn’t. In other words, both studies estimated the <a href="https://en.wikipedia.org/wiki/Risk_ratio">relative risk</a> to be 1.20. </p>
<p>However, the relative risk estimated from one study was more precise than the other, because its estimate was based on outcomes from many more patients. Thus, the estimate from one study was statistically significant, and the estimate from the other was not.</p>
<p>The authors cite this inconsistency – that one study obtained a significant result and the other didn’t – as evidence that statistical significance leads to misinterpretation of scientific results. </p>
<p>However, I feel that a reasonable summary is simply that one study collected statistically significant evidence and one did not, but the estimates from both studies suggested that relative risk was near 1.2.</p>
<h2>Where to go from here</h2>
<p>I agree with the Nature article and The American Statistician editorial that data collected from all well-designed scientific studies should be made publicly available, with comprehensive summaries of statistical analyses. Along with each study’s p-values, it is important to publish estimates of effect sizes and confidence intervals for these estimates, as well as complete descriptions of all data analyses and data processing. </p>
<p>On the other hand, only studies that provide strong evidence in favor of important associations or new effects should be published in premier journals. For these journals, standards of evidence should be increased by requiring smaller p-values for the initial report of relationships and new discoveries. In other words, make scientists publish results that they’re even more certain about.</p>
<p>The bottom line is that dismantling accepted standards of statistical evidence will decrease the uncertainty that scientists have in publishing their own research. But it will also increase the public’s uncertainty in accepting the findings that they do publish – and that can be problematic.</p><img src="https://counter.theconversation.com/content/114161/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Valen E. Johnson receives funding from the National Institutes of Health to perform biostatistical research on the selection of variables associated with cancer and cancer research. </span></em></p>Two prestigious journals have suggested abandoning the traditional test of the strength of a study’s results. But a statistician worries that this would make science worse.Valen E. Johnson, University Distinguished Professor and Department Head of Statistics, Texas A&M UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/946742018-05-21T07:45:59Z2018-05-21T07:45:59ZThere’s a crisis in psychology – here’s how technology could provide a solution<figure><img src="https://images.theconversation.com/files/219215/original/file-20180516-155616-1je7p3m.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Technology involving virtual reality could teach us a lot about the human psyche.</span> <span class="attribution"><span class="source"> (U.S. Air Force photo by Airman 1st Class Keith Holcomb)</span>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Psychologists obediently follow the same rules as other scientists. But their efforts haven’t yielded equivalent progress. In fact, in the last decade, psychologists have realised that some of their most intriguing findings <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0056515">are not reliable</a> – when other researchers try to repeat the same study, they don’t find the same results. </p>
<p>Many people refer to this as a <a href="https://theconversation.com/is-psychology-really-in-crisis-60869">replication crisis</a> in the field. But what is to blame for this problem and what can we do about it? In a new review, <a href="https://www.research.manchester.ac.uk/portal/en/publications/the-assessment-and-modeling-of-perceptual-control(9a95a0bf-e3ae-4b42-90f6-b579496c5c4e).html">published in the General Review of Psychology</a>, we describe a promising technological solution.</p>
<p>Most psychologists are convinced that the widespread misuse of statistics and <a href="http://www.sciencemag.org/news/2012/09/harvard-psychology-researcher-committed-fraud-us-investigation-concludes">poor research integrity</a> – a euphemism for cheating – are ultimately to blame for the crisis. So, removing bad practices should solve the problem. Yet this often <a href="http://journals.sagepub.com/doi/10.1177/1745691616652873">doesn’t work</a> – seriously undermining confidence in the reliability of psychology.</p>
<p>We are convinced that tightening the regulation of research won’t fix the crisis. Instead, we need to go back over the past century to a crucial wrong turn in psychology that happened because of a limit in the technology of the time. </p>
<figure class="align-right ">
<img alt="" src="https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=777&fit=crop&dpr=1 600w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=777&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=777&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=976&fit=crop&dpr=1 754w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=976&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/219217/original/file-20180516-155555-1onftyw.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=976&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">William James.</span>
<span class="attribution"><span class="source">Notman Studios</span></span>
</figcaption>
</figure>
<p>In the late 19th century, the American philosopher <a href="https://en.wikipedia.org/wiki/William_James">William James</a> argued that the essence of psychology is hidden purpose. He famously described the purposeful behaviour of a frog held under water in an inverted glass. Despite attempts by the experimenter to stop it, the frog eventually found its way up to the air in surprising ways. James argued that the frog’s purpose was to get to the surface and it did this in different ways each time.</p>
<p>But it isn’t easy to test hidden purpose reliably in humans. Most research in psychology relies on getting large numbers of participants to provide data. The researchers then measure correlations, or the effects of experimental manipulations, in these groups. This research began before the time of the modern computer, when the researcher could simply present a “stimulus” to a participant and measure the response. And this approach <a href="https://www.researchgate.net/profile/Richard_Marken/publication/232499797_You_Say_You_Had_a_Revolution_Methodological_Foundations_of_Closed-Loop_Psychology/links/574da87108ae8bc5d15bce5f/You-Say-You-Had-a-Revolution-Methodological-Foundations-of-Closed-Loop-Psychology.pdf">persists today</a>, making up the vast majority of studies in psychology. </p>
<p>Unfortunately, it’s not reliable. One recent series of “stimulus-response” studies were set up so that participants could respond to an image on a screen by either pushing or pulling a joystick. They were presented with either “negative” or “positive” images or words (stimuli). The researchers proposed that viewing a negative stimulus (such as an angry face) unconsciously <a href="https://pdfs.semanticscholar.org/faf1/b1ebc5b361b56b22e2b29bae5792b0d72dda.pdf">activates the muscles that extend the arm</a>. This is because that’s how we push something away if we are faced with it in real life. The initial studies supported this account – participants were quicker to respond to negative stimuli when the response was to push the lever away from them than when it was to pull it.</p>
<p>However, a huge <a href="https://www.tandfonline.com/doi/full/10.1080/02699931.2014.968096">review of over 68 attempts</a> to test for this effect in more than 3,000 participants showed that this effect was not consistently repeated. Importantly, in tasks that were designed so that pushing the lever actually made the stimulus get closer, the opposite effect was found – negative stimuli were now associated with the response of pulling the lever. </p>
<p>The authors concluded that participants were actually controlling their perceived distance from the negative image through whatever action they could (just like James’s frog). But the traditional experimental design was simply not set up to test this.</p>
<h2>Embracing VR</h2>
<p>In our recent article, we bring together the advances that researchers have made using an approach known as <a href="https://www.pctweb.org">perceptual control theory</a>. It continues where James left off, assuming the hidden purposes of living things, but it tests for them using a sophisticated approach. It typically relies on computing capacity to measure people’s activities in virtual environments, and to build a computer model of the psychological processes within the individual. </p>
<p>The technique is based on creating a situation where the participant can pursue a goal, for example controlling the distance from a negative image on a screen using a joystick. It then measures every change that goes on in the situation continuously (for example by making real-time videos) – including disturbances that get in the way of the person’s goal, such as changes in the experimental set up or physical obstacles. All this data is then used to build a computer model of how each participant is pursuing their goal. </p>
<p>You can then repeat the situation, using the computer model to predict what the individual will do, and constantly compare with what they are doing. If the model fails, you improve it until you’ve got a good match – creating a “<a href="https://link.springer.com/article/10.3758/s13414-017-1398-2">personal profile</a>” for each individual. This can then be tested for replication over repeated sessions. You can also combine data for many individuals to look at mean effects to work out what goals are generally relevant to a given situation.</p>
<h2>Replication … at last?</h2>
<p>The result of this approach is typically a robust model of the psychological processes involved in an activity – such as tracking a target on a screen. These models have been shown to repeat a high level of accuracy over and over again, typically <a href="https://www.ncbi.nlm.nih.gov/pubmed/2943855">showing correlations over 0.98</a> – a perfect correlation is 1.0. A correlation shows the association between two different variables (for example stimuli and response). This is currently virtually unheard of in traditional psychology research, where correlations of as low as 0.3 are regarded as “statistically significant”.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/219216/original/file-20180516-155573-wy926a.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Don’t worry, there’s hope.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/spider-bites-hand-man-afraid-662092573?src=5gS8jB4fJhKhOdOTnux-mw-1-69">Goncharov_Artem/Shutterstock</a></span>
</figcaption>
</figure>
<p>You might think that modelling of this kind is only suitable for simple tasks, but a similar approach has been applied to many areas, including <a href="https://escholarship.org/uc/item/0ch4g2x6">food competition in animals</a>. This used frame-by-frame video analysis to show that a rat holding food in its mouth continually reorients its body to maximise the distance between its food and a competing animal’s mouth. </p>
<p>The same assumptions have informed treatments of spider phobia, helping to build tasks in which the participant can <a href="https://www.sciencedirect.com/science/article/abs/pii/S088761851630408X">control their distance from a spider</a> in a virtual corridor. Facing fears in this way is a treatment known as exposure therapy. However, it was previously unknown what level of control over the exposure works best. The study using this technique showed for the first time that people who have a higher degree of control over the exposure actually ended up avoiding spiders less after the experiment than those who had little control.</p>
<p>There are areas where it will be more challenging to use this technique – such as complex tasks involving memory and reasoning. Nevertheless, it could be easily applied in many areas.</p>
<p>The replication crisis has been the wake up call psychological science needed to think differently – now it is time to embrace the advances in technology that allow us to improve the field.</p><img src="https://counter.theconversation.com/content/94674/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Vyv Huddy is affiliated with the British Association for Behavioural and Cognitive Psychotherapies Control Theory Special Interest Group. </span></em></p><p class="fine-print"><em><span>Warren Mansell does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Studies in psychology cannot be replicated in the same way as in other fields. But technology could change that.Warren Mansell, Reader of Clinical Psychology, University of ManchesterVyv Huddy, Lecturer in Clinical Psychology, UCLLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/902862018-02-21T23:58:58Z2018-02-21T23:58:58ZWhy students are the answer to psychology’s replication crisis<figure><img src="https://images.theconversation.com/files/207382/original/file-20180221-132680-1hql3pa.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Shutterstock</span> </figcaption></figure><p>“Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital,” Aaron Levenstein, a <a href="http://www.nytimes.com/1986/07/05/obituaries/prof-aaron-levenstein.html">business professor at Baruch College</a>, once said. </p>
<p>I first heard a version of this quote in an undergraduate social psychology class in 2003. Nearly a decade and a half later, psychology is having a replication crisis — and the “bikini” is largely to blame. </p>
<p>Recently, more than 270 psychologists set out to <a href="http://science.sciencemag.org/content/349/6251/aac4716">repeat 100 experiments</a> to see if they could generate the same results. They successfully replicated only 39 of the 100 studies. </p>
<p>Over several years, <a href="http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.html">failed attempts to replicate published studies</a> have caused generally accepted bodies of research to be called into question — or rejected outright. </p>
<p>One example is the idea that <a href="https://hbr.org/2016/11/have-we-been-thinking-about-willpower-the-wrong-way-for-30-years">your willpower is a limited resource that, like a muscle, becomes exhausted</a> when it is used. Another is that power posing — <a href="http://journals.sagepub.com/doi/full/10.1177/0956797614553946">standing like a superhero for two minutes</a> — makes you feel bolder, reduces stress hormones and increases testosterone. Both have fallen aside due to failed replications. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=300&fit=crop&dpr=1 600w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=300&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=300&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=377&fit=crop&dpr=1 754w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=377&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/206867/original/file-20180219-75997-bvxnza.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=377&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Psychology was wrong about the power pose.</span>
<span class="attribution"><span class="source">(Shutterstock)</span></span>
</figcaption>
</figure>
<p>These aren’t dusty, arcane findings limited to academic journals; a TED talk by social psychologist Amy Cuddy on the effectiveness of power posing has been viewed over 45 million times and is <a href="https://www.ted.com/playlists/171/the_most_popular_talks_of_all">near the top of the list of the most popular TED talks of all time</a>. </p>
<h2>Bad habits</h2>
<p>The “bikini” at the centre of the crisis refers to the way researchers collect and analyze data and report their results. Many important details and decisions are often concealed.</p>
<p>When carrying out experiments, <a href="http://journals.sagepub.com/doi/abs/10.1177/0956797611417632">researchers make decisions</a> about how much data to collect, whether some observations should be excluded from the analysis and what controls, if any, should be included in analyses. </p>
<p>After the data has been collected, researchers have additional, undisclosed, leeway. </p>
<p>They may “torture the data” <a href="https://theconversation.com/one-reason-so-many-scientific-studies-may-be-wrong-66384">until it reaches statistical significance</a> (a cut-off that suggests the real effect may not be zero), a practice called “p-hacking.” </p>
<p>Or they may <a href="https://www.nature.com/articles/s41562-016-0021">engage in the practice of “HARKing,”</a> short for “hypothesizing after results are known.” Creating a hypothesis to confirm a result that has already been found makes it easier to satisfy journal reviewers and editors who are interested in publishing statistically significant results.</p>
<p>In academia, where researchers are often under pressure to “publish or perish” to advance their careers and win grants, amassing publications is the route to success. </p>
<p>All told, this undisclosed flexibility can lead to extremely high rates of <a href="http://journals.sagepub.com/doi/full/10.1177/0956797616658563">false positive results</a>. A false positive is essentially claiming there is an effect when there isn’t one. An example would be concluding that standing up straight increases testosterone levels, when it doesn’t. </p>
<h2>A new research culture</h2>
<p>Despite all the upheaval, psychology’s replication crisis may have a silver lining. In a few short years, researchers have proposed many ideas and recommendations for <a href="http://science.sciencemag.org/content/sci/348/6242/1422.full.pdf">reforming research with the goal of improvement</a>. </p>
<p>Journals and granting agencies are demanding more from authors with respect to <a href="http://www.science.gc.ca/eic/site/063.nsf/eng/h_415B5097.html">openness</a> and <a href="https://www.psychologicalscience.org/publications/badges">transparency</a>. There are accessible online repositories, such as <a href="http://github.com">Github</a>, the <a href="http://osf.io">Open Science Framework</a> and <a href="http://opendoar.org">OpenDOAR</a>, that allow researchers to share their raw materials, exact protocols, scripts, data, code, etc. with anyone who has an internet connection. The aim is to essentially have nothing concealed in the scientific process.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=338&fit=crop&dpr=1 600w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=338&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=338&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/206869/original/file-20180219-76003-1oake2r.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Researchers who manipulate their data or engage in poor research practices will wind up with results that can’t be replicated.</span>
<span class="attribution"><span class="source">(Shutterstock)</span></span>
</figcaption>
</figure>
<p>Some journals, such as <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456"><em>Psychological Science</em></a>, and recently <a href="http://www.apa.org/news/press/releases/2017/08/open-science.aspx">American Psychological Association journals</a> are encouraging authors to store their data and code in these repositories and to disclose details about data collection decisions before submitting a manuscript for peer review. Researchers can also preregister their hypotheses. But something has been missing. </p>
<h2>The missing link</h2>
<p>While psychological science has been moving toward more open and transparent methods, graduate student training has been largely left out of discussions. </p>
<p>Many of the practices that created the crisis are embedded in our research culture: We do things a certain way because we have always done things this way and other people do too. Much of this culture is assimilated when researchers are in graduate school. </p>
<p>To sustain and maintain the momentum of positive change, it is important for graduate education to keep up with changes in the field. If training fails to keep up, graduate students may leave programs with antiquated ideas and practices. <a href="http://rsos.royalsocietypublishing.org/content/3/9/160384">These ideas and practices can proliferate</a> as students become faculty members, start their own labs and train graduate students in the same manner they were taught.</p>
<p>Part of educating students is ensuring they are aware of the changing cultural landscape, and then explicitly teaching them to follow open and transparent research practices and avoid bad habits. </p>
<h2>Finding the light</h2>
<p>In our department at the University of Guelph, a group of methodologically minded faculty have recognized the importance of tackling this problem head on. Our goal is to create positive change and take steps to avoid history repeating itself with the next generation of researchers. </p>
<p>We created “<a href="https://www.uoguelph.ca/psychology/graduate/thesis-statistics">Statistical methods in theses: Guidelines and explanations</a>” to help students when conducting their thesis research. Students can work through the guidelines with their advisors, allowing them to make better decisions in the planning stages of their research projects.</p>
<p>The document’s rather humble sounding purpose belies an unintended provocative side. The guidelines identify questionable research practices — to provide explanations and advice for students who wish to follow open and transparent research practices. Because some of the questionable practices it identifies may be standard, previously unquestioned — and sometimes taught — procedures, the document has the potential to be viewed, by some, as extreme. </p>
<p>Culture is not something that can be changed overnight. But with explicit efforts to cultivate a new research culture, change can be targeted and purposeful. </p>
<p>This crisis in psychology makes me think about a line in John Milton’s epic poem, <em>Paradise Lost</em>: “Long is the way and hard, that out of Hell leads up to Light.” </p>
<p>By acting on the crisis, psychology has embarked upon its symbolic journey back to “light.” It will be current and future graduate students that will decide how, and where, the journey ends.</p><img src="https://counter.theconversation.com/content/90286/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Jeffrey R. Spence works at the University of Guelph. </span></em></p><p class="fine-print"><em><span>David Stanley is affiliated with University of Guelph. </span></em></p><p class="fine-print"><em><span>Ian Newby-Clark does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Bad research techniques have called into question the results of many psychology studies. Fixing the problem starts with making sure students don’t pick up bad habits.Jeffrey R. Spence, Associate Professor, University of GuelphDavid Stanley, Associate Professor, University of GuelphIan Newby-Clark, Professor, University of GuelphLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/848962017-10-18T23:33:23Z2017-10-18T23:33:23ZA statistical fix for the replication crisis in science<figure><img src="https://images.theconversation.com/files/190686/original/file-20171017-30394-1pcijw3.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Many scientific studies aren't holding up in further tests.</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/closeup-portrait-tired-young-woman-scientistcrashing-677843290?src=LSedwluRRX5MZ_HcDBqbmA-1-0">A and N photography/Shutterstock.com</a></span></figcaption></figure><p>In a trial of a new drug to cure cancer, 44 percent of 50 patients achieved remission after treatment. Without the drug, only 32 percent of previous patients did the same. The new treatment sounds promising, but is it better than the standard?</p>
<p>That question is difficult, so statisticians tend to answer a different question. They look at their results and compute something called a p-value. If the p-value is less than 0.05, the results are “statistically significant” – in other words, unlikely to be caused by just random chance.</p>
<p>The problem is, many statistically significant results <a href="http://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf">aren’t replicating</a>. A treatment that shows promise in one trial doesn’t show any benefit at all when given to the next group of patients. This problem has become so severe that <a href="http://www.nature.com/news/psychology-journal-bans-p-values-1.17001">one psychology journal actually banned p-values</a> altogether. </p>
<p>My colleagues and I have studied this problem, and we think we know what’s causing it. The bar for claiming statistical significance is simply too low. </p>
<h1>Most hypotheses are false</h1>
<p>The Open Science Collaboration, a nonprofit organization focused on scientific research, <a href="http://dx.doi.org/10.1126/science.aac4716">tried to replicate</a> 100 published psychology experiments. While 97 of the initial experiments reported statistically significant findings, only 36 of the replicated studies did. </p>
<p><a href="http://dx.doi.org/10.1080/01621459.2016.1240079">Several graduate students and I</a> used these data to estimate the probability that a randomly chosen psychology experiment tested a real effect. We found that only about 7 percent did. In a similar study, <a href="http://dx.doi.org/10.1073/pnas.1516179112">economist Anna Dreber and colleagues</a> estimated that only 9 percent of experiments would replicate. </p>
<p>Both analyses suggest that only about one in 13 new experimental treatments in psychology – and probably many other social sciences – will turn out to be a success. </p>
<p>This has important implications when interpreting p-values, particularly when they’re close to 0.05. </p>
<h1>The Bayes factor</h1>
<p>P-values close to 0.05 are more likely to be due to random chance than most people realize.</p>
<p>To understand the problem, let’s return to our imaginary drug trial. Remember, 22 out of 50 patients on the new drug went into remission, compared to an average of just 16 out of 50 patients on the old treatment. </p>
<p>The probability of seeing 22 or more successes out of 50 is 0.05 if the new drug is no better than the old. That means the p-value for this experiment is statistically significant. But we want to know whether the new treatment is really an improvement, or if it’s no better than the old way of doing things.</p>
<p>To find out, we need to combine the information contained in the data with the information available before the experiment was conducted, or the “prior odds.” The prior odds reflect factors that are not directly measured in the study. For instance, they might account for the fact that in 10 other trials of similar drugs, none proved to be successful.</p>
<p>If the new drug isn’t any better than the old drug, then statistics tells us that the probability of seeing exactly 22 out of 50 successes in this trial is 0.0235 – relatively low. </p>
<p>What if the new drug actually is better? We don’t actually know the success rate of the new drug, but a good guess is that it’s close to the observed success rate, 22 out of 50. If we assume that, then the probability of observing exactly 22 out of 50 successes is 0.113 – about five times more likely. (Not nearly 20 times more likely, though, as you might guess if you knew the p-value from the experiment was 0.05.)</p>
<p>This ratio of the probabilities is called the Bayes factor. We can use <a href="https://yalebooks.yale.edu/book/9780300188226/theory-would-not-die">Bayes theorem</a> to combine the Bayes factor with the prior odds to compute the probability that the new treatment is better. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=600&fit=crop&dpr=1 600w, https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=600&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=600&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=754&fit=crop&dpr=1 754w, https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=754&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/190907/original/file-20171018-32341-1i70yms.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=754&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">What’s the probability of observing success in 50 trials? The blue curve represents probabilities under the ‘null hypothesis,’ when the new treatment is no better than the old. The red curve represents probabilities when the new treatment is better. The shaded area represents the p-value. In this case, the ratio of the probabilities assigned to 22 successes is A divided by B, or 0.21.</span>
<span class="attribution"><span class="source">Valen Johnson</span>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<p>For the sake of argument, let’s suppose that only 1 in 13 experimental cancer treatments will turn out to be a success. That’s close to the value we estimated for the psychology experiments.</p>
<p>When we combine these prior odds with the Bayes factor, it turns out that the probability the new treatment is no better than the old is at least 0.71. But the statistically significant p-value of 0.05 suggests exactly the opposite!</p>
<h1>A new approach</h1>
<p>This inconsistency is typical of many scientific studies. It’s particularly common for <a href="http://dx.doi.org/10.1073/pnas.1313476110">p-values around 0.05</a>. This explains why such a high proportion of statistically significant results do not replicate. </p>
<p>So how should we evaluate initial claims of a scientific discovery? In September, <a href="https://www.nature.com/articles/s41562-017-0189-z">my colleagues and I</a> proposed a new idea: Only P-values less than 0.005 should be considered statistically significant. P-values between 0.005 and 0.05 should merely be called suggestive.</p>
<p>In our proposal, statistically significant results are more likely to replicate, even after accounting for the small prior odds that typically pertain to studies in the social, biological and medical sciences. </p>
<p>What’s more, we think that statistical significance should not serve as a bright-line threshold for publication. Statistically suggestive results – or even results that are largely inconclusive – might also be published, based on whether or not they reported important preliminary evidence regarding the possibility that a new theory might be true. </p>
<p>On Oct. 11, we presented this idea to a group of statisticians at the ASA Symposium on Statistical Inference in Bethesda, Maryland. Our goal in changing the definition of statistical significance is to restore the intended meaning of this term: that data have provided substantial support for a scientific discovery or treatment effect.</p>
<h1>Criticisms of our idea</h1>
<p>Not everyone agrees with our proposal, including another <a href="https://dx.doi.org/10.17605/OSF.IO/9S3Y6">group of scientists</a> led by psychologist Daniel Lakens. </p>
<p>They argue that the definition of Bayes factors is too subjective, and that researchers can make other assumptions that might change their conclusions. In the clinical trial, for example, Lakens might argue that researchers could report the three-month rather than six-month remission rate, if it provided stronger evidence in favor of the new drug. </p>
<p>Lakens and his group also feel that the estimate that only about one in 13 experiments will replicate is too low. They point out that this estimate does not include effects like <a href="https://doi.org/10.1371/journal.pbio.1002106">p-hacking</a>, a term for when researchers repeatedly analyze their data until they find a strong p-value. </p>
<p>Instead of raising the bar for statistical significance, the Lakens group thinks that researchers should set and justify their own level of statistical significance before they conduct their experiments.</p>
<p>I disagree with many of the Lakens group’s claims – and, from a purely practical perspective, I feel that their proposal is a nonstarter. Most scientific journals don’t provide a mechanism for researchers to record and justify their choice of p-values before they conduct experiments. More importantly, allowing researchers to set their own evidence thresholds doesn’t seem like a good way to improve the reproducibility of scientific research. </p>
<p>Lakens’s proposal would only work if journal editors and funding agencies agreed in advance to publish reports of experiments that haven’t been conducted based on criteria that scientists themselves have imposed. I think this is unlikely to happen anytime in the near future.</p>
<p>Until it does, I recommend that you not trust claims from scientific studies based on p-values near 0.05. Insist on a higher standard.</p><img src="https://counter.theconversation.com/content/84896/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Valen E. Johnson does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Scientists have a big problem: Many psychological studies don’t hold up to scrutiny. Is it time to redefine statistical significance?Valen E. Johnson, University Distinguished Professor and Department Head of Statistics, Texas A&M UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/693462016-12-04T19:08:25Z2016-12-04T19:08:25ZImitation game: how copies can solve our cultural heritage crises<figure><img src="https://images.theconversation.com/files/148209/original/image-20161201-30244-x7zd3v.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The Otsuka Museum of Art in Tokushima features a full-sized replica of the Sistine Chapel. </span> <span class="attribution"><span class="source">Kzaral/Flickr</span>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Visitors to the <a href="http://o-museum.or.jp/english/publics/index/16/0/#page">Otsuka Museum</a> in Japan are offered the chance to see through time. Two life-sized copies of Leonardo da Vinci’s Last Supper are hung on opposing walls, one showing it before the major 1999 restoration, and one as it is today.</p>
<p>Visitors can pivot their view to observe changes in colour on the paintings in front of them. The true-to-scale copies are painted on ceramic tiles, which the Museum claims can maintain their colour and shape for over 2000 years. </p>
<p>The Museum offers visitors the ability to literally walk through the history of Western art’s greatest works. Other recreations include Vincent Van Gogh’s lost Six Sunflowers painting, which was destroyed in 1945 by US airstrikes on Tokyo. Art lovers can view paintings in a manner rendered impossible in real life. </p>
<p>As the world faces ongoing cultural heritage crises – from poverty, to war, to natural disaster – is the creation of copies the answer?</p>
<p>Increasingly sophisticated technology, including 3D printing, offers an alternative to traditional preservation techniques. However, while these new technologies may solve problems of accessibility to precious antiquities they also raise other problems of authenticity and trust.</p>
<p>The New Yorker recently profiled the work undertaken by the <a href="http://www.newyorker.com/magazine/2016/11/28/the-factory-of-fakes">Factum Arte workshop in Madrid</a>, which uses advanced 3D printing technology to recreate ancient artefacts that are being ravaged by time and modern life.</p>
<p>The head of the project, Adam Lowe, describes the new artefacts as “rematerialized” facsimiles. Notable projects include a full sized reproduction of King Tut’s burial chamber, built out of extraordinarily detailed scans. The original tomb is at risk of deterioration due to thousands of tourists breathing on ancient plaster, as well as possible excavations to uncover what could be Nefertiti’s tomb next door.</p>
<p>Despite these successes, there are objections to the practice of creating copies. Critical theorist Walter Benjamin famously argued that art loses its “aura” when it is reproduced: the impact an original artwork creates when it’s <a href="https://ceasefiremagazine.co.uk/walter-benjamin-art-aura-authenticity/">uniquely present in time and space</a> vanishes as soon as copies are made. </p>
<p>Yet ultimately, the transferral of art into a new medium and context allows entire new audiences to have a brand new – and possibly deeper – connection to our greatest treasures.</p>
<p>Anyone who has battled the crowds in front of Rembrandt’s The Night Watch in the Rijksmuseum or the mass of selfie sticks in front of the Mona Lisa in the Louvre, will appreciate how Otsuka Museum affords the visitor the opportunity to experience a painting’s colours, composition and artistic impression.</p>
<p>Of course the experience of these “rematerialized” paintings and artefacts will be different from that of the original pieces. Tutankhamen’s replica tomb, while set near the original in Luxor, is missing the authentic musty smell of the ancient rooms. It also features a digitally restored panel destroyed when the tomb was originally opened. </p>
<h2>Where is the harm?</h2>
<p>But as long as the audience clearly understands that these are replicas, from the perspective of preserving cultural heritage, where is the harm in appreciating these objects in a new medium?</p>
<p>Visitors to the Otsuka Museum and Factum Arte are under no illusion that what they are viewing are originals. These are not fakes, as the attention grabbing headlines claim, but replicas and copies, the distinctive feature being a lack of intent to deceive. Honesty with your audience is of paramount importance. </p>
<p>The issue of restoration and conservation is historically fraught, and intensified now by various economic and cultural tensions. As noted in the New Yorker article, visiting Egypt right now is an unusual experience due to that country’s recent political upheavals. Aside from the chance to visit one of the Seven Wonders of the World without battling hoards of tourists, the issues of preserving of the country’s cultural and archaeological assets are obvious.</p>
<p>The Egyptian Museum in Cairo has limited air-conditioning, with cracked showcases and storage units on display in the main exhibition spaces alongside many priceless relics. They are awaiting the new museum, which has been under construction for many years.</p>
<p>Ironically, the museum collection features a copy of one of the most important Ancient Egyptian artefacts, the Rosetta stone, with the original version found in the British Museum, over 2000 miles away. </p>
<p>In contrast, a different response to cultural heritage concerns can be seen in the vast temples at <a href="http://whc.unesco.org/en/list/88">Abu Simbel</a>. Originally carved into the side of a mountain over the Nile, the temples came under threat with the construction of the Aswan High dam in the 1960s. Under the supervision of UNESCO, the temples were cut out and moved 65m up and 210m northwest.</p>
<p>In this case what has been replicated is not the physical temples of Ramses II but the original location and authenticity of the experience as it was originally intended. </p>
<p>The move meant that the temple’s axis is no longer aligned as it was during Pharaonic Egypt. The structure was created so the sun lit up the statues inside the temple twice a year, on February 21 and October 21. The so-called “miracle of the sun” still occurs, just one day later.</p>
<p>Whilst there is no attempt to conceal the relocation, one cannot help ascribing perceived defects to the move. When did Ramses lose his beard? Was it dropped?</p>
<p>Jonathan Jones recently argued in The Guardian that we should leave the <a href="https://www.theguardian.com/artanddesign/jonathanjonesblog/2016/apr/11/palmyra-isis-syria-restored-3d-printers-vandalism">crumbling remnants</a> of the Isis-ravaged Syrian town of Palmyra alone, and recognise that the destruction of this sacred site forms part of its history and newfound fame.</p>
<p>For Jones, the authenticity of Palmyra is its decay, not the “faked-up approximation” that a <a href="https://theconversation.com/should-we-3d-print-a-new-palmyra-57014">3D printed version</a> might offer visitors.</p>
<p>But we are constantly battling the push and pull of authenticity and heritage. While Jones may deride the inauthentic replication of Syrian archaeological sites, we must confront the issue of preserving our cultural heritage in manner that is accessible in the future. </p>
<p>When these remnants are no more than dust and rubble, would a future generation really rebuff a “rematerialized” 3D printed version? So long as the creation of a replica does no harm to authentic version, where is the problem in creating a coherent copy?</p><img src="https://counter.theconversation.com/content/69346/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Felicity Strong does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Increasingly sophisticated technology allows us to make close-to-perfect copies of everything from paintings to burial chambers. Can a replica bring artefacts to new audiences?Felicity Strong, PhD Candidate - Grimwade Centre for Cultural Materials Conservation, The University of MelbourneLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/528182016-01-11T19:20:17Z2016-01-11T19:20:17ZBusiness journals to tackle publication bias, will publish ‘null’ results<figure><img src="https://images.theconversation.com/files/107491/original/image-20160107-14013-yxo8ff.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The problem of publication bias in academic journals is being tackled.</span> <span class="attribution"><span class="license">Author provided</span></span></figcaption></figure><p>In a <a href="https://drive.google.com/file/d/0B1--xR9XdxCEYTZROWpGRzdHYWs/view?usp=sharing">joint statement, ten editors</a> representing some of the academia’s most prestigious journals for management, organisational behaviour and work psychology research, have vowed to publish research that fails to prove a hypothesis.</p>
<p>Their message: we will now publish good (that is, well-conceived, designed, and conducted) research even if proposed hypotheses are not supported or yield “null” results.</p>
<p>Why publish something that does not prove the hypothesis being put forward? It seems counter-intuitive. But the reasons are more complex than they seem.</p>
<h2>Not all data is equal</h2>
<p>Nobel Prize winning economist Ronald Coase is credited to have said: “If you torture the data long enough, it will confess to anything”. Indeed, <a href="https://www.researchgate.net/publication/268506231_Publication_bias_Understanding_the_myths_concerning_threats_to_the_advancement_of_science">evidence</a> suggests that all disciplines of scientific literature are not free from bias and questionable practices such as selectively reporting hypotheses, excluding data points and variables post-hoc, and <a href="http://www.ncbi.nlm.nih.gov/pubmed/22853650">rounding “p value”</a> - so non-significant results (arising by chance) become significant (that is, has a systematic effect). </p>
<p>While some statistical, analytical tweaks can be completely justified, especially when fully reported, the motivation underpinning more <a href="http://jom.sagepub.com/content/42/1/5.abstract">questionable practices</a> stems from certain beliefs that to get published one requires a “tidy” story, and this demands “clean” results.</p>
<p>Those beliefs are not unfounded. There is evidence of <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">severe replication challenges</a> and that the extent of literature that is publicly available is not representative of completed studies on a particular phenomenon. </p>
<p>This <a href="https://en.wikipedia.org/wiki/Publication_bias">publication bias</a> has long been <a href="http://www.jstor.org/stable/2282137?origin=crossref">recognised</a> and is often <a href="http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off">debated</a> as consequences include overestimating of effect sizes (for instance, relationships appear more important than they are) and proliferation of theory and paradigms (a topic or explanation takes undeserved precedence over alternatives).</p>
<h2>The publishing quandary</h2>
<p>One dominant issue relates to the editorial <a href="https://theconversation.com/explainer-what-is-peer-review-2779">review process</a> and its implications. Journals seek to raise their impact so they become more attractive to future authors seeking to maximise the dissemination of their best research. A journal that publishes important research gets read and cited more, thereby increases its ranking in the competitive market of scientific outlets, and in turn receives more submissions to choose from. </p>
<p>In addition, evidence shows that journal editors and reviewers can be <a href="http://sth.sagepub.com/content/15/1/9.short">biased</a> toward the publication of articles with statistical significant results, such as by <a href="http://archinte.jamanetwork.com/article.aspx?articleid=226270">being less critical</a> of a study’s methodology when the majority of the results are positive. </p>
<p>Meanwhile, the majority of scholarly authors is evaluated against the impact factor of the journals they publish in, in academia this often determines nothing less than reputation, hiring decisions, promotion, tenure, research funding, and pay level. As individuals are highly incentivised to publish in the top journals, some may engage in questionable research practices to achieve that.</p>
<p>Taken together, positive, affirmative and neat research narratives appear more likely to get published and noted. If so, then the research evidence available might not just inform but also distort some of our work practices and organisational policies. </p>
<h2>The new two-stage process</h2>
<p>In the largest initiative to date in the organisational and managerial sciences, the ten journals will introduce and pilot a two-stage review process for empirical contributions. </p>
<p>In the first step, scholarly authors will present journal reviewers with an abbreviated paper comprising the theory, methodology, measurement information, and analysis plan but no results or discussion. </p>
<p>The semi-complete article will then be either rejected, receive a revise and resubmit - called an R&R - or be accepted in principle for publication. The latter decision will ultimately trigger a traditionally formatted manuscript that also includes results and discussion sections (Stage 2).</p>
<p>It’s a small change in editorial and review protocols but a large step for the scientific community, and ultimately everyone. Papers may now be evaluated on the merits, rigour, and quality of the project rather than what is actually found. It is the importance of the research question and the theoretical justification that counts, not whether it holds true.</p>
<h2>Re-establishing trust</h2>
<p>This alleviates a number of pressures. Early-career researchers especially are under considerable pressure to publish in top journals and it is tempting to opt for safer avenues instead of pursuing novel ideas with uncertain outcomes. Now, all researchers may simply discuss the theoretical and conceptual meaning and limitations of what was found and embrace what did and did not pan out. They subsequently can reinstate some authority for the scientific community and what it produces to contribute society.</p>
<p>It will also mean that they can opt to submit what might be called an “extensive research proposal”, explicating what will be researched, why this is important, and how it will realised. This allows reviewers to provide formative feedback about how to potentially enhance the planned study before researchers invest time and money into the collection and analysis of data, a perk typically reserved to PhD candidates through their supervisors. </p>
<p>Often journal reviewers provide very constructive feedback on methodology and making a stronger contribution, and under the traditional model this feedback can mean for researchers to repeat a study, opt for lower ranked journal that publish more limited research, or abandon publication altogether. </p>
<p>The new, two-stage approach thus affords wiser use of research resources, including tax funded grant money and survey respondents’ time.</p>
<p>Not only will the move help re-establish trust in managerial and organisational sciences that can ultimately affect possibly billions of workers, it also means that everyone has to accept that <a href="https://theconversation.com/science-is-imperfect-you-can-be-certain-of-that-4140">scientific inquiry</a> like nature, is complex and messy. </p>
<p>It will be important to see how many researchers and journals indeed opt to publish null-findings, and whether and how that affects their impact factor and ranking over time. </p>
<p>It will be also interesting to see if the <a href="https://theconversation.com/way-off-balance-science-and-the-mainstream-media-4080">media</a> reflects these changes and covers intriguing theory whilst narrating insufficient empirical support. And it will be crucial whether more scientific transparency and neutrality brings about a shift in managers adoption and interpretation of evidence. </p>
<p>For now we are offered an opportunity and we shall embrace and investigate the above – outcome unknown.</p>
<p><em>This article been altered since publication to correct a headline that inaccurately reflected the piece. The error was made by an editor.</em></p><img src="https://counter.theconversation.com/content/52818/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Ramon Wenzel does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Editors of ten authoritative workplace journals amend their publishing criteria to address publication bias.Ramon Wenzel, Assistant Professor of Management & Social Impact, The University of Western AustraliaLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/492022015-11-03T00:31:20Z2015-11-03T00:31:20ZThe replication crisis has engulfed economics<figure><img src="https://images.theconversation.com/files/100277/original/image-20151030-20160-ebvlfq.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">No two alike?</span> <span class="attribution"><span class="source">Image sourced from Shutterstock.com</span></span></figcaption></figure><p>A sense of crisis is developing in economics after <a href="http://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf">two Federal Reserve economists</a> came to the alarming conclusion that economics research is usually not replicable. </p>
<p>The <a href="http://www.businessinsider.com.au/federal-reserve-paper-on-the-replicability-of-economic-studies-2015-10">economists took 67 empirical papers</a> from 13 reputable academic journals. Without assistance from the original researchers they were only able to get the same result in a third of cases. </p>
<p>With the original researchers’ assistance, that percentage increased to about half, suggesting reporting practices and requirements are seriously deficient.</p>
<p>The replication crisis in psychology is well-documented. <em>Science</em> recently published a stunning <a href="http://www.sciencemag.org/content/349/6251/aac4716">report by the Open Science Collaboration.</a> Almost 300 researchers were involved in trying to directly replicate the results of 100 papers published in 2008. This followed earlier exercises involving many labs (such as <a href="https://osf.io/wx7ck/">here,</a> <a href="http://journal.sjdm.org/14/14321/jdm14321.html">here</a> and <a href="http://www.ncbi.nlm.nih.gov/pubmed/24461214">here.</a>)</p>
<p>The researchers did not succeed in the clear majority of cases. On average they found the mean effect size to be only half of what was reported in the original studies. While the report has been questioned (<a href="http://daniellakens.blogspot.nl/2015/08/power-of-replications-in.html">here</a>
and <a href="http://www.spspblog.org/simone-schnall-on-her-experience-with-a-registered-replication-project/">here,</a>) there is growing concern that a cornerstone of the scientific edifice is in serious need of renovation. </p>
<h2>What’s the problem?</h2>
<p>Researchers are too often granted <a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1850704">inappropriate degrees of freedom</a>, and some are just <a href="http://retractionwatch.com/">fraudulent</a>. But that said, some of these distressing replication results are because good science is messy. It involves hard work and reasonable people can reasonably disagree on the various calls that have to be made. </p>
<p>A good illustration is this just-published <a href="http://www.nature.com/news/crowdsourced-research-many-hands-make-tight-work-1.18508">study by Raphael Silberzahn and Eric Uhlmann</a>. The researchers engaged in methodological debates with well-known data sleuth <a href="http://opim.wharton.upenn.edu/%7Euws/">Uri Simohnson</a>. </p>
<p>Simohnson questioned the results of an earlier study from the pair that suggested noble-sounding German names could boost careers. Re-running the analysis with a better analytical approach, Simonsohn did not confirm the effect. Silberzahn and Uhlmann eventually conceded the point in a joint paper with Simonsohn.</p>
<p>In their new study, the researchers provided a data set and asked more than two dozen teams of researchers to contribute. They sought to determine, based on the data set, whether skin colour of soccer players from four major leagues (England, France, Germany, and Spain) influenced how often they were given a red card.</p>
<p>Somewhat shockingly, the answers were rather diverse. Of the 29 teams, 20 found a statistically significant correlation with the median, suggesting dark-skinned players were 1.3 times more likely than light-skinned players to be sent off. </p>
<p>But the researchers reported:</p>
<blockquote>
<p>“Findings varied enormously, from a slight (and non-significant) tendency for referees to give more red cards to light-skinned players to a strong trend of giving more red cards to dark-skinned players.” </p>
</blockquote>
<p>Interestingly, this diversity of results survived even after the researchers debated the methodological approach.</p>
<p>The upshot is that even under the best of circumstances – one data set, what seems like a straightforward question to answer, and an exchange of ideas on the best method – arriving at consensus can be extraordinarily difficult. And it surely becomes even more difficult with multiple data sets and many teams.</p>
<h2>Further scrutiny</h2>
<p>That, of course, is hardly news to most social scientists, who largely accept that any single study is worth only so much. This is why replication efforts and meta-analyses are as important as the <a href="http://www.kurtlewininstituut.nl/teaching/program/content/?cid=289">recent focus on publication bias and underpowered studies</a>. There is tantalising evidence that many experimental economics studies are severely under-powered (although <a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2356018">the evidence so far has been established only for a very simple class of games)</a>.</p>
<p>It will be interesting to see the result of <a href="http://sciencepredictionmarkets.com/repoverview.html">a current collaborative effort by economists to replicate</a> eighteen laboratory economics studies from 2011 to 2014.</p>
<p>It is not just the social sciences that are in the grip of replication crises. The extent and consequences of p-hacking, and publication biases (studies that report no effect not being published) in science, are <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106">well-documented</a> and have been <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124">known for a while</a>.</p>
<p>So, where to from here? With a number of journals (including the <em>Journal of the Economic Science Association</em>, <em>Experimental Economics</em>, <em>Journal of Experimental Social Psychology</em>, <em>Journal of Personality and Social Psychology</em>, <em>Psychological Science</em>, <em>Perspectives on Psychological Science</em>) opening their doors to replication in various guises, we can expect more results to seemingly discredit the social sciences.</p>
<p>Hopefully in the long run it will up the ante on what it takes for a study to be reliable. Replication studies can inflict considerable damage on <a href="http://www.spspblog.org/simone-schnall-on-her-experience-with-a-registered-replication-project/">individuals’ productivity and reputation</a>. There’s a need for minimal reporting standards and acceptable replication etiquette to be clarified, such as whether original authors have to be invited or consulted. Journals should become more serious about their data set collection efforts, when not prevented by confidentiality.</p><img src="https://counter.theconversation.com/content/49202/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Andreas Ortmann does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>A US Federal Reserve paper has come to the alarming conclusion that economics research is usually not replicable.Andreas Ortmann, Professor, UNSW SydneyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/472492015-09-10T10:11:12Z2015-09-10T10:11:12ZReal crisis in psychology isn’t that studies don’t replicate, but that we usually don’t even try<figure><img src="https://images.theconversation.com/files/94294/original/image-20150909-18665-5n51wp.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Run a study again and again – should the results hit the same bull's-eye every time?</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/richardofengland/6788829651">Richard Matthews</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span></figcaption></figure><p>Psychology is still digesting the implications of a large study published last month, in which a team led by University of Virginia’s Brian Nosek <a href="http://doi.org/10.1126/science.aac4716">repeated 100 psychological experiments</a> and found that <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">only 36% of originally “significant”</a> (in the statistical sense) results were replicated.</p>
<p>Commentators are divided over how much to worry about the news. Some psychologists have suggested that the <a href="https://www.psychologytoday.com/blog/the-nature-nurture-nietzsche-blog/201509/quick-guide-the-replication-crisis-in-psychology">field is in “crisis,”</a> a claim that others (such as Northeastern University psychology professor <a href="http://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html">Lisa Feldman Barrett</a>) have flatly denied.</p>
<p>What can we make of such divergence of opinion? Is the discipline in crisis or not? Not in the way that some seemed to suggest, but that doesn’t mean substantial changes aren’t needed.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/94307/original/image-20150909-18672-1sg8wdu.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Does the majority of psych research belong in the trash?</span>
<span class="attribution"><a class="source" href="http://www.shutterstock.com/pic-220895758/stock-photo-a-lot-of-wrinkled-paper-laying-in-and-around-a-wastepaper-basket-picture-is-toned.html">Wastebasket image via www.shutterstock.com</a></span>
</figcaption>
</figure>
<h2>Mixing up what the study really tells us</h2>
<p>Certainly the fact that 64% of the findings were found unstable is surprising and disconcerting. But some of the more sensational press response has been disappointing. </p>
<p>Over at The Guardian, a headline writer implied the study delivered a “<a href="http://www.theguardian.com/science/2015/aug/27/study-delivers-bleak-verdict-on-validity-of-psychology-experiment-results">bleak verdict on validity</a> of psychology experiment results.” Meanwhile an article in The Independent claimed that much of “<a href="http://www.independent.co.uk/news/science/study-reveals-that-a-lot-of-psychology-research-really-is-just-psychobabble-10474646.html">psychology research really is just psycho-babble</a>.”</p>
<p>And everywhere there was the term “failure to replicate,” a subtly sinister phrasing that makes nonreplication sound necessarily like a bad thing, as though “success” in replication were the goal of science. “Psychology can’t be trusted,” runs the implicit narrative here, “the people conducting these experiments have been wasting their time.”</p>
<p>Reactions like this tied themselves up in a logical confusion; to believe that nonreplication demonstrated the failure of psychology is incoherent, as it entails a privileging of this latest set of results over the earlier ones. This can’t be right: it makes no sense to put stock in a new set of experimental results if you think their main lesson is to cast doubt on <em>all</em> experimental findings.</p>
<p>Experiments should be considered in the aggregate, with conclusions most safely drawn from multiple demonstrations of any given finding. </p>
<p>Running experiments is like flipping a coin to establish whether it is biased. Flipping it 20 times, and finding it comes up heads for 17 of them, might start to raise your suspicions. But extreme results like this are actually more likely when the number of flips is lower. You would want to try that coin many more times before feeling confident enough to wager that something funny is going on. Failure to replicate your majority of heads in a sample of 100 flips would indicate just that you hadn’t flipped the coin enough to make a safe conclusion the first time around.</p>
<p>This need for aggregation is the basis of an <a href="http://www.theguardian.com/commentisfree/2015/aug/28/psychology-experiments-failing-replication-test-findings-science">argument advanced</a> by Stanford’s John Ioannidis, a medical researcher who <a href="http://doi.org/10.1371/journal.pmed.0020124">proposed 10 years ago</a> that most published research findings (not just those in psychology) are false. Ioannidis highlights the positive side of facing up to something he and many other people have suspected for a while. He also points out that psychology is almost certainly not alone among scientific disciplines. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=450&fit=crop&dpr=1 600w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=450&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=450&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=566&fit=crop&dpr=1 754w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=566&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/94303/original/image-20150909-18645-jyec1s.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=566&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">You can’t know if the study will produce similar results again and again unless you run it again and again.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/arripay/174695277">Tanya Hart</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<h2>Real crisis is we don’t try to replicate enough</h2>
<p>The fact is, psychology has long been aware that replication is a good idea. Its importance is evident in the longstanding practice of researchers creating systematic literature reviews and meta-analyses (statistical aggregations of existing published findings) to give one another broader understandings of the field. Researchers just haven’t been abiding by best practice. As psychologist Vaughan Bell <a href="http://mindhacks.com/2015/08/28/dont-call-it-a-comeback/">pointed out</a>, a big part of Nosek’s achievement was in the logistical challenge of getting such a huge study done with so many cooperating researchers.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=338&fit=crop&dpr=1 600w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=338&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=338&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=424&fit=crop&dpr=1 754w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=424&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/94305/original/image-20150909-18642-sw2v2w.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=424&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Experiments shouldn’t be one and done – run once, then published as if set in stone.</span>
<span class="attribution"><span class="source">Maggie Villiger</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>This brings us to the <a href="http://nymag.com/scienceofus/2015/08/many-psychology-research-findings-may-be-false.html">actual nature of the crisis</a> revealed by the Science study; what Nosek and his colleagues showed is that psychologists need to be doing more to try to replicate their work if they want a better understanding of how much of it is reliable. Unfortunately, as journalist Ed Yong pointed out in his Atlantic coverage of <a href="http://www.theatlantic.com/health/archive/2015/08/psychology-studies-reliability-reproducability-nosek/402466/">the Nosek study</a> (and in <a href="http://www.theatlantic.com/notes/2015/09/sweeping-psychologys-problems-under-the-rug/403726/">a reply</a> to Barrett’s op-ed) there are several powerful professional disincentives to actually running the same experiments again. In a nutshell, the profession rewards publications and journals publish results which are new and counter-intuitive. The problem is compounded by the media, which tend to disseminate experimental findings as unquestionable “discoveries” or even God-given truths.</p>
<p>So though psychology (and very likely not only psychology) most certainly has something of a crisis on its hands, it is not a crisis of the discipline’s methodology or rules. Two of the study’s authors made some <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">suggestions for improvement</a> on The Conversation, including incentives for more open research practices and even obligatory openness with data and preregistration of experiments. These recommendations reiterate what methods specialists have said for years. Hopefully the discussion stirred up by <a href="https://osf.io/ezcuj/">Nosek and colleagues’ efforts</a> will also inspire others.</p>
<p>In essence, everyone agrees that experimental coin flipping is a reasonable way to proceed. This study exposed a flaw of the discipline’s sociology, of what people <em>actually</em> do and why they do it. Put another way, psychologists have already developed a perfectly effective system for conducting research; the problem is that so few of them really use it.</p><img src="https://counter.theconversation.com/content/47249/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Huw Green does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>The field of psychology is trying to absorb a recent big study that was able to replicate only 36 out of 100 major research papers. That finding is an issue, but maybe not for the reason you think.Huw Green, PhD Student and Trainee Clinical Psychologist at the Graduate Center, City University of New YorkLicensed as Creative Commons – attribution, no derivatives.