tag:theconversation.com,2011:/uk/topics/reproducible-research-40942/articlesReproducible research – The Conversation2022-05-12T18:44:27Ztag:theconversation.com,2011:article/1805412022-05-12T18:44:27Z2022-05-12T18:44:27ZThe idea that power poses boost your confidence fell from favor – but a new review of the research calls for a second look<figure><img src="https://images.theconversation.com/files/462476/original/file-20220511-25-1kzokh.jpg?ixlib=rb-1.1.0&rect=745%2C8%2C5245%2C3727&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">After great popularity, the idea of power poses came under fire.</span> <span class="attribution"><a class="source" href="https://www.gettyimages.com/detail/photo/woman-in-superhero-costume-royalty-free-image/1140379193">Choreograph/iStock via Getty Images Plus</a></span></figcaption></figure><p>If you stand like Wonder Woman or Superman, will you feel stronger? Will you actually be stronger?</p>
<p>Psychology researchers have investigated these questions for decades. After all, mind and body are intertwined. <a href="https://doi.org/10.1007/BF00992249">How you stand or sit can give you feedback</a> on how you feel, and your feelings are often revealed by the way you hold yourself. </p>
<p>One influential study published in 2010 suggested that power poses – body positions like a wide stance with your hands on your hips while standing, or clasping your hands behind your head and putting your feet on a desk while sitting – <a href="https://doi.org/10.1177/0956797610383437">increased levels of the male sex hormone testosterone</a> and decreased levels of cortisol, the main stress hormone. High levels of testosterone and low levels of cortisol are linked to fearlessness, <a href="https://doi.org/10.1016/j.yhbeh.2010.08.020">risk-taking</a> and insensitivity to punishment. From there, scientists assumed that <a href="https://doi.org/10.1177/0956797614566855">power posing could affect how people felt</a>, how they acted and <a href="https://doi.org/10.3389/fpsyg.2016.01463">how others perceived them</a>.</p>
<p>These findings drew enormous attention outside of the lab. <a href="https://www.littlebrownspark.com/titles/amy-cuddy/presence/9780316256551/">Power posing was advertised</a> as a way of improving one’s life, and the idea took off in popular culture. Intentionally adopting the stance of a powerful person could apparently give you the confidence and the appearance of a powerful person.</p>
<p>But in the following years, some researchers could not replicate the original findings when they tried to rerun the experiments. The lead author of the original study <a href="https://faculty.haas.berkeley.edu/dana_carney/pdf_my%20position%20on%20power%20poses.pdf">admitted to mistakes and distanced herself from it</a>. Since then, there’s been a heated debate about whether engaging in power poses really does anything at all.</p>
<p>In an effort to figure out which power pose findings hold up and which do not, <a href="https://psycnet.apa.org/record/2022-61115-003?doi=1">we conducted a meta-analytic review</a> – that is, we combined data from all available research on the topic. Based on dozens of studies, we suggest that there is something to the idea of power poses, even if the research was overhyped in the past.</p>
<h2>Pulling together findings from 88 studies</h2>
<p>We focused on two types of body positions.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="girl seated on couch gets lecture from a woman" src="https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=816&fit=crop&dpr=1 600w, https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=816&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=816&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1026&fit=crop&dpr=1 754w, https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1026&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/462478/original/file-20220511-6370-675uv2.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1026&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">A low-power pose may look similar to a child receiving a reprimand.</span>
<span class="attribution"><a class="source" href="https://www.gettyimages.com/detail/photo/mother-lecturing-daughter-in-living-room-royalty-free-image/107697790">JGI/Jamie Grill/Tetra images via Getty Images</a></span>
</figcaption>
</figure>
<p>The first type included power poses. Examples of high-power poses would be standing or sitting in a very expansive way, taking up a lot of space. A low-power pose would be crossing your legs and folding your arms while standing, or bowing your head and putting your hands on your lap while seated.</p>
<p>The second type included upright postures, like standing erect or sitting up straight in a chair versus bowing your head and slumping. <a href="https://doi.org/10.1111/spc3.12559">Theoretical</a> and <a href="https://doi.org/10.1037/pspi0000181">empirical</a> research have suggested that power poses are nonverbal expressions of dominance, whereas upright postures are displays of prestige. </p>
<p>Following open-science standards, we <a href="https://doi.org/10.17605/OSF.IO/CX2Q3">preregistered our protocol</a> with the Open Science Framework before conducting the analysis. This step is meant to increase transparency. By stating the game plan upfront, you can’t fiddle around with the data to try to find something significant to report.</p>
<p>Then we combed through 12 scientific databases with search terms including “body position” and “power pose.” This hunt turned up over 24,000 potentially relevant studies. We included just the ones that randomly assigned participants to different groups. Only this <a href="https://itfeature.com/design-of-experiment-doe/basic-principles-of-experimental-design">experimental design</a> allows researchers to make inferences about the cause of any effects they identify.</p>
<p>Often if a study doesn’t find a link between the the factors it was investigating, the research doesn’t end up getting published. Because of this phenomenon, called <a href="https://doi.org/10.1371/journal.pone.0215052">publication bias</a>, we sent requests for unpublished data to researchers from six different scientific societies. We also contacted all 21 researchers who had authored at least two articles on body positions to inquire whether they had any unpublished studies. Over one-fourth of the effects we analyzed came from unpublished studies.</p>
<p>In the end, our analysis of high- versus low-power poses and upright versus slumped poses was based on 313 effects from 88 studies that included 9,799 participants. </p>
<h2>What held up and what didn’t</h2>
<p>Our review examined three types of potential effects power poses and upright positions could have.</p>
<p>First there were self-reported effects, such as feeling powerful, confident and positive. These kinds of effects were statistically significant and robust, meaning they were seen again and again across many studies. People told researchers they felt stronger when they engaged in power poses and upright postures.</p>
<p>Then there were behavioral effects, such as how long participants would stick with a task, whether they exhibited antisocial behavior, and how action-oriented they were. Researchers identified these effects in many studies as well, but the findings were less reliable and more subject to publication bias.</p>
<p>Finally there were physiological effects such as hormone levels, heart rate and skin conductance, which often stands in as a way to measure stress in psychology research studies. In our meta-analysis, these effects were not statistically significant across all the studies. It was in this area that the power pose research didn’t hold up. Simply taking expansive body positions does not influence hormones or other physiological indicators as previously believed.</p>
<p>We found these self-reported and behavioral effects in studies from both Western countries like the U.S., Germany and the U.K. that favor the individual and in Eastern countries like China, Japan and Malaysia that favor the collective. Age and gender did not make a difference with respect to the effects. Nor did it matter whether participants were college students or not. From the available data it is not clear, however, how long such effects last after someone moves out of a particular body position.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="woman seated at her desk smiles with her legs open and arms wide on arm rests" src="https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=399&fit=crop&dpr=1 600w, https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=399&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=399&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=502&fit=crop&dpr=1 754w, https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=502&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/462479/original/file-20220511-15-9qbhu7.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=502&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Taking up space can be an expression of dominance.</span>
<span class="attribution"><a class="source" href="https://www.gettyimages.com/detail/photo/woman-posing-by-her-desk-at-home-office-royalty-free-image/499236621">Lucy Lambriex/The Image Bank via Getty Images</a></span>
</figcaption>
</figure>
<h2>What new experiments can explore</h2>
<p>Unfortunately, many experimental studies in our meta-analysis did not include a control group of participants who adopted a neutral body position. That means we can’t tell for sure whether it is high-power poses and upright postures making people feel more positive and powerful, whether it is the low-power and slumped postures making people feel less positive and powerful, or whether it is some combination of the two. Future studies could clarify that question by including control groups that hold neutral body positions for comparison.</p>
<p>Furthermore, most studies included participants from Western, educated, industrialized, rich and democratic societies – characterized as “WEIRD” by psychology researchers. Effects should also be tested in other populations.</p>
<p>To promote and facilitate further insights on the effects of body positions, we also created an <a href="https://metaanalyses.shinyapps.io/bodypositions/">app</a> allowing researchers to enter new data and download the most recent results. Continuing these investigations is important, because science is an ongoing process that usually does not provide definitive final answers. More evidence accumulates with each new study.</p><img src="https://counter.theconversation.com/content/180541/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>For a while it was all the rage to adopt Wonder Woman’s famous stance and other body positions that allegedly pumped up your confidence – until more studies of the phenomenon failed to find the connection.Astrid Schütz, Professor of Psychology, University of BambergBrad Bushman, Professor of Communication and Rinehart Chair of Mass Communication, The Ohio State UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1214722019-08-05T17:05:36Z2019-08-05T17:05:36ZStop blaming video games for mass killings<figure><img src="https://images.theconversation.com/files/286964/original/file-20190805-36381-j2rixr.jpg?ixlib=rb-1.1.0&rect=7%2C7%2C4276%2C3111&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The research doesn't say what some lawmakers suggest every time there's a mass shooting.</span> <span class="attribution"><a class="source" href="https://unsplash.com/photos/m1g4-dczgcc">Fredrick Tendong/Unsplash</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span></figcaption></figure><p>In the wake of the <a href="https://elpasoheraldpost.com/active-shooter-reported-at-cielo-vista-mall/">El Paso shooting on Aug. 3</a> that left 22 dead and dozens injured, a familiar trope has reemerged: Often, when a young man is the shooter, people try to <a href="https://www.vox.com/2019/8/4/20753725/el-paso-dayton-shootings-video-games-gop-mccarthy">blame the tragedy on violent video games</a> and other forms of media.</p>
<p>This time around, Texas Lt. Gov. Dan Patrick placed some of the blame on a video game industry that “<a href="https://twitter.com/existentialfish/status/1158008058930442240">teaches young people to kill</a>.” Republican House Minority Leader Kevin McCarthy of California went on to <a href="https://twitter.com/existentialfish/status/1158021253778874369">condemn video games</a> that “dehumanize individuals” as a “problem for future generations.” And President Trump pointed to society’s “glorification of violence,” including “<a href="https://www.c-span.org/video/?463254-1/president-trump-condemns-racism-bigotry-white-supremacy-mass-shootings">gruesome and grisly video games</a>.”</p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1158021253778874369"}"></div></p>
<p>These are the same connections a Florida lawmaker made after the Parkland shooting in February 2018, suggesting that the gunman in that case “was prepared to <a href="http://www.miamiherald.com/news/local/community/broward/article200279494.html">pick off students like it’s a video game</a>.”</p>
<p>But, as a researcher who has studied violent video games for almost 15 years, I can state that there is no evidence to support these claims that violent media and real-world violence are connected.</p>
<p>As far back as 2011, the U.S. Supreme Court ruled that <a href="https://www.law.cornell.edu/supct/html/08-1448.ZS.html">research did not find a clear connection</a> between violent video games and aggressive behavior. Criminologists who study mass shootings specifically refer to those sorts of connections as a “<a href="https://doi.org/10.1177/1088767913510297">myth</a>.” And in 2017, the Media Psychology and Technology division of the American Psychological Association released a <a href="https://div46amplifier.com/2017/06/12/news-media-public-education-and-public-policy-committee/">statement</a> I helped craft, suggesting reporters and policymakers cease linking mass shootings to violent media, given the lack of evidence for a link.</p>
<h2>A history of a moral panic</h2>
<p>So why are so many policymakers inclined to blame violent video games for violence? There are two main reasons. </p>
<p>The first is the psychological research community’s efforts to <a href="http://www.apa.org/monitor/2011/02/stem.aspx">market itself</a> as strictly scientific. This led to a <a href="http://www.slate.com/articles/health_and_science/science/2016/10/why_the_replication_crisis_seems_worse_in_psychology.html">replication</a> <a href="https://www.theatlantic.com/science/archive/2016/03/psychologys-replication-crisis-cant-be-wished-away/472272/">crisis</a> instead, with researchers often <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">unable to repeat the results</a> of their studies. Now, psychology researchers are reassessing their analyses of a wide range of issues – not just violent video games, but <a href="https://www.thecut.com/2017/01/psychologys-racism-measuring-tool-isnt-up-to-the-job.html">implicit racism</a>, <a href="https://www.npr.org/2016/10/01/496093672/power-poses-co-author-i-do-not-believe-the-effects-are-real">power poses</a> and more.</p>
<p>The other part of the answer lies in the <a href="http://dx.doi.org/10.1037/a0030597">troubled history</a> of violent video game research specifically. </p>
<p>Beginning in the early 2000s, some scholars, anti-media advocates and professional groups like the APA began working to connect a <a href="http://dx.doi.org/10.1037/a0035569">methodologically messy</a> and often contradictory set of results to public health concerns about violence. This echoed historical patterns of moral panic, such as <a href="https://doi.org/10.1007/978-1-349-27458-1_6">1950s concerns about comic books</a> and <a href="https://www.nytimes.com/2015/10/26/us/two-pop-culture-wars-first-over-comics-then-over-music.html">Tipper Gore’s efforts to blame pop and rock music</a> in the 1980s for violence, sex and Satanism.</p>
<p>Particularly in the early 2000s, dubious evidence regarding violent video games was <a href="https://www.apa.org/science/about/psa/2003/10/anderson.aspx">uncritically promoted</a>. But over the years, confidence among scholars that violent video games influence aggression or violence <a href="http://doi.org/10.1111/jcom.12293">has crumbled</a>.</p>
<h2>Reviewing all the scholarly literature</h2>
<p>My own research has examined the degree to which violent video games can – or can’t – predict youth aggression and violence. In a 2015 <a href="http://doi.org/10.1177/1745691615592234">meta-analysis</a>, I examined 101 studies on the subject and found that violent video games had little impact on kids’ aggression, mood, helping behavior or grades.</p>
<p>Two years later, I found <a href="https://doi.org/10.1016/j.ijlp.2017.11.001">evidence that scholarly journals’ editorial biases</a> had distorted the scientific record on violent video games. Experimental studies that found effects were more likely to be published than studies that had found none. This was <a href="http://doi.org/10.1037/bul0000074">consistent with others’ findings</a>. As the Supreme Court noted, any impacts due to video games are <a href="https://www.law.cornell.edu/supct/html/08-1448.ZS.html">nearly impossible to distinguish</a> from the effects of other media, like cartoons and movies. </p>
<p>Any claims that there is consistent evidence that violent video games encourage aggression are simply false.</p>
<p><a href="https://doi.org/10.1111/jcom.12129">Spikes in violent video games’ popularity</a> are well-known to correlate with substantial declines in youth violence – not increases. These correlations are very strong, stronger than most seen in behavioral research. More recent research suggests that the <a href="http://dx.doi.org/10.1037/ppm0000030">releases of highly popular</a> violent video games are associated with <a href="http://doi.org/10.1002/soej.12139">immediate declines</a> in violent crime, hinting that the releases may cause the drop-off.</p>
<h2>The role of professional groups</h2>
<p>With so little evidence, why are <a href="https://www.snopes.com/ap/2018/01/26/kentucky-governor-says-shootings-cultural-problem/">lawmakers still trying to blame</a> violent video games for mass shootings by young men? Can groups like the National Rifle Association seriously blame <a href="https://www.huffingtonpost.com/entry/nra-gun-violence-hollywood-fox-news_us_59d6cb0be4b072637c430e4b">imaginary guns</a> for gun violence?</p>
<p>A key element of that problem is the willingness of professional guild organizations such as the <a href="https://www.apa.org/">APA</a> to promote false beliefs about violent video games. (I’m a fellow of the APA.) These groups mainly exist to promote a profession among news media, the public and policymakers, <a href="https://www.apa.org/advocacy/index.aspx">influencing licensing and insurance laws</a>. They also make it easier to get grants and newspaper headlines. Psychologists and psychology researchers like myself pay them yearly dues to increase the public profile of psychology. But there is a risk the general public may mistake promotional positions for objective science.</p>
<p>In 2005 the APA released its first <a href="https://videogames.procon.org/sourcefiles/resolution-on-violence-in-video-games-and-interactive-media.pdf">policy statement</a> linking violent video games to aggression. However, my <a href="https://doi.org/10.1016/j.ijlp.2018.01.004">recent analysis of internal APA documents</a> with <a href="https://www.scopus.com/authid/detail.uri?authorId=56382474100">criminologist Allen Copenhaver</a> found that the APA ignored inconsistencies and methodological problems in the research data.</p>
<p>The APA <a href="http://www.apa.org/about/policy/violent-video-games.aspx">updated</a> its statement in 2015, but that sparked controversy immediately: More than <a href="https://www.scribd.com/doc/223284732/Scholar-s-Open-Letter-to-the-APA-Task-Force-On-Violent-Media-Opposing-APA-Policy-Statements-on-Violent-Media">230 scholars</a> wrote to the group asking it to stop releasing policy statements altogether. I and others objected to <a href="http://www.newsweek.com/apa-video-games-violence-364394">perceived conflicts of interest and lack of transparency</a> tainting the process.</p>
<p>It’s bad enough that these statements misrepresent the actual scholarly research and misinform the public. But it’s worse when those falsehoods give advocacy groups like the NRA cover to shift blame for violence onto non-issues like video games. The resulting misunderstanding hinders efforts to address mental illness and other issues, such as the need for gun control, that are actually related to gun violence.</p>
<hr>
<p><em>This is an updated version of <a href="https://theconversation.com/its-time-to-end-the-debate-about-video-games-and-violence-91607">an article originally published</a> on Feb. 16, 2018.</em></p>
<p>[ <em><a href="https://theconversation.com/us/newsletters?utm_source=TCUS&utm_medium=inline-link&utm_campaign=newsletter-text&utm_content=expertise">Expertise in your inbox. Sign up for The Conversation’s newsletter and get a digest of academic takes on today’s news, every day.</a></em> ]</p><img src="https://counter.theconversation.com/content/121472/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Christopher J. Ferguson does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>On the whole, results from psychology research studies don’t support a direct connection between playing violent video games and aggressive behavior.Christopher J. Ferguson, Professor of Psychology, Stetson University Licensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/995342018-07-11T08:36:48Z2018-07-11T08:36:48ZDebate: Donald Trump’s war on science and its long-lasting consequences<p>Two years ago, a <a href="https://theconversation.com/science-wars-in-the-age-of-donald-trump-67594">piece</a> in The Conversation predicted that a new breed of science war, not between <a href="https://en.wikipedia.org/wiki/The_Two_Cultures">two cultures</a> this time, but instead between opposed political factions in the United States. In the war, science would be taken hostage by the so-called <a href="https://theconversation.com/science-in-crisis-from-the-sugar-scam-to-brexit-our-faith-in-experts-is-fading-65016">reproducibility crisis</a>, potentially jeopardising its quest for <a href="https://theconversation.com/to-tackle-the-post-truth-world-science-must-reform-itself-70455">solutions</a>.</p>
<p>Sectors of the US conservative establishment have tried for decades to attack regulators such as the US Environmental Protection Agency (EPA) with various forms of <a href="https://www.nytimes.com/2018/03/26/climate/epa-scientific-transparency-honest-act.html">weaponised transparency</a>, e.g., the request that all regulatory science be accessible, from medical records to mathematical models. This strategy has historically been coupled with the standard practice to inflate uncertainty as to make regulations impossible.</p>
<p>Now the same actors take advantage of science reproducibility crisis to say that non-reproducible science cannot be used to regulate, that a failure of scientific peer review implies <a href="https://www.thegwpf.org/content/uploads/2016/10/PeerReview.pdf">discarding the IPCC’s work</a>, and that the crisis is due to the progressive assault on higher education by ideologies such as <a href="https://www.nas.org/images/documents/NAS_irreproducibilityReport.pdf">neo-Marxism and post-modernism</a>.</p>
<p>While this was to be expected, what is discomforting is the reaction of large sectors of the scientific community and institutions: to deny the crisis altogether, and to label those who say otherwise as enemies of science, corruptors of the youth by way of <a href="http://www.pnas.org/content/early/2018/03/08/1708272114">fostering cynicism and indifference</a>, or otherwise <a href="http://www.pnas.org/content/115/11/2620">“partisans”</a>, thus assimilating those who intend to map the crisis and its possible remedies to their political adversaries.</p>
<h2>A bit of history</h2>
<p>The weaponisation of transparency had an important step in the <a href="https://en.wikipedia.org/wiki/Data_Quality_Act">US Data Quality Act</a>, passed in 2001, to “ensuring and maximising the quality, objectivity, utility, and integrity of information”, while in fact being used by industry groups to <a href="https://www.amazon.com/Doubt-Their-Product-Industrys-Threatens/dp/019530067X/ref=sr_1_1?s=books&ie=UTF8&qid=1529596861&sr=1-1&keywords=michaels+doubt+product">undercut scientific reports</a> on everything, from federal report on global warming to the World Health Organisation’s low-sugar dietary guidelines.</p>
<p>The inflation of uncertainty as an anti-regulatory strategy has a likewise <a href="https://www.bloomsbury.com/us/merchants-of-doubt-9781608193943/">long history</a>, famously championed by tobacco companies for whom, in a tobacco industry memo of 1969, “doubt is our product […] also as means of establishing a controversy” on the dangerousness of smoking. Today corporate power blames the dramatic decline in entomofauna – including pollinators, on everything, including climate change, to exculpate <a href="http://iopscience.iop.org/article/10.1088/1748-9326/5/1/014006/meta">pesticides</a>.</p>
<p>Doubt is also spread by generous funding of research, e.g. in the case of tobacco company by funding studies on others cancerogenic agents, in the case of sugar by <a href="https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2548255">funding research on cholesterol</a>, and so on. The older book on the <a href="https://www.amazon.com/Regulation-Games-Strategic-Administrative-Process/dp/0884100669/">“regulation game”</a> instructs lobbyists on how to enroll scientists <a href="http://www.andreasaltelli.eu/file/repository/Chapter1.pdf">“with a modicum of finesse”</a>, lest they “recognise that they have lost their objectivity and freedom of action”.</p>
<p>Today in the <a href="https://global.oup.com/academic/product/the-business-of-america-is-lobbying-9780190215514?cc=gr&lang=en&">US</a> corporate interest can spend on lobbying $34 for each dollar spent by diffuse interest and unions combined, and for scholars in both US and <a href="https://www.routledge.com/Lobbyists-and-Bureaucrats-in-Brussels-Capitalisms-Brokers/Laurens/p/book/9781138289277">Europe</a> a salient aspect of this power is lobbyists’ access to more and better disseminated science, creating a perverse asymmetry in the use of <a href="https://www.sciencedirect.com/science/article/pii/S0016328717300472">evidence for policy</a>. For some observers this would need urgent remedial action to give citizen some <a href="https://global.oup.com/academic/product/the-business-of-america-is-lobbying-9780190215514?cc=gr&lang=en&">structured mechanism</a> of access to independent scientific evidence.</p>
<h2>The contention accelerates</h2>
<p>Since science’s crisis made it to the headlines thanks to a cover of The Economist in <a href="https://www.economist.com/printedition/2013-10-19">October 2013</a> and based on previous works <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124">John P.A. Ioannidis</a>, it was only a matter of time before the crisis become enrolled in this long-standing fight between regulators and the regulated.</p>
<p>Thus, few were surprised when recently the EPA proposed <a href="https://www.epa.gov/newsreleases/epa-administrator-pruitt-proposes-rule-strengthen-science-used-epa-regulations">new rules</a> this April for transparency meant to fight “secret science” and simultaneously the NAS – not the National Academy of Science but the National Association of Scholars [sic] – published a <a href="https://www.nas.org/images/documents/NAS_irreproducibilityReport.pdf">report</a> on science’s “irreproducibility”, urging remedial action including:</p>
<ul>
<li><p>Congress should pass an expanded Secret Science Reform Act to prevent government agencies from making regulations based on irreproducible research.</p></li>
<li><p>Congress should require government agencies to adopt strict reproducibility standards by measures that include strengthening the Information Quality Act.</p></li>
</ul>
<p>The report blames the crisis on the “progressive left” and its attack on higher education with “neo-Marxism, radical feminism, historicism, post-colonialism, deconstructionism, post-modernism, liberation theology, and a host of other ideologies”.</p>
<h2>Science fights back… or not?</h2>
<p>The reticence of science institutions – especially those charged of science for policy, to even mention the existence of a crisis has been already discussed on <a href="https://theconversation.com/to-tackle-the-post-truth-world-science-must-reform-itself-70455">this journal</a>. This new phase though sees science coming out to deny its existence altogether.</p>
<p>For Naomi Oreskes <a href="https://www.nature.com/articles/d41586-018-05207-9">writing on Nature</a> the true science’s crisis is its instrumental use by corporate interests, not reproducibility. For Kathleen Hall Jamieson, <a href="http://www.pnas.org/content/115/11/2620">writing on the Proceedings of the National Academy of Science</a> in March this year, the “science is broken” narrative is a histrionic, unwarranted generalisations. She warns “partisans” against fuelling dangerous narratives, lest these undermine science, and censors the media for their style of reporting of the crisis.</p>
<p>For Daniele Fanelli writing on the <a href="http://www.pnas.org/content/early/2018/03/08/1708272114">same journal</a> science is not facing a reproducibility crisis. For Fanelli, a scholar who is well known for his work on the topic, there is little evidence of an increase in the prevalence of misconduct and growing retractions are mostly <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001563">a good sign</a>, implying better governance and more empowerment of scientists, while in a previous work he concedes that there are important differences in <a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0010068">performance across disciplines</a>. Fanelli also denies the existence of a problem with statistical testing procedures, which is at odds with statisticians’ own <a href="https://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108">concern</a> on <a href="https://www.ncbi.nlm.nih.gov/pubmed/26064558">misuse</a> and <a href="http://journals.sagepub.com/doi/abs/10.1177/2515245918771329">abuse</a> of statistical tests in recent years, and their quest for <a href="https://www.nature.com/articles/d41586-017-07522-z">what to do about it</a>.</p>
<p>In a more recent paper <a href="http://www.pnas.org/content/114/14/3714">written with Ioannidis</a> Fanelli maps the magnitude of the overestimation of effects (an ingredient of the replicability issue) across fields, and identifies factors of risk for producing unreliable results, such as early-career status, isolation, and lack of scientific integrity. Crisis or no crisis then? The last word goes to <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5933781/">Ioannidis</a>:</p>
<blockquote>
<p>“Even well-intentioned academics, perceiving an attack on science, may be tempted to take an unproductive, hand-waving defensive position: ‘We have no problem with reproducibility’, ‘everything is fine’, ‘science is making progress’.”</p>
</blockquote>
<h2>The crux of the matter</h2>
<p>Progressives see the crisis as due to postmodern philosophers, oil friendly politicians, and other enemies of reason. Conservatives blame it on radical feminism, neo-Marxism, liberation theology, and yes, postmodern philosophers again. Scientists caught in the crossfire retrench behind a “Crisis, what crisis?”</p>
<p>Perhaps some reading of postmodern philosophers would benefit all contenders. Scientists in particular could try to feed into today’s more urgent debate, one which sees an <a href="https://www.penguinrandomhouse.com/books/317051/enlightenment-now-by-steven-pinker/9780525427575/">unreflective version of modernity</a> contrasted with more measured understanding of <a href="https://www.jeremylent.com/the-patterning-instinct.html">our dominant narratives</a> and of their dangers for man survival. Many have started to consider a <a href="https://www.tandfonline.com/doi/abs/10.1080/02698595.2017.1370927?journalCode=cisp20">different story</a>, one where the crisis, besides being political and institutional, is also methodological, cultural, ethical, and philosophical. Hopefully, a movement of critical <a href="http://symposium.uoc.edu/17275/detail/post-normal-science-symposium-barcelona.html">resistance</a> is in the making. </p>
<p><strong>Note:</strong> An extended essay version of this work is to appear on the journal <a href="https://www.journals.elsevier.com/futures">FUTURES</a>.</p><img src="https://counter.theconversation.com/content/99534/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Andrea Saltelli ne travaille pas, ne conseille pas, ne possède pas de parts, ne reçoit pas de fonds d'une organisation qui pourrait tirer profit de cet article, et n'a déclaré aucune autre affiliation que son organisme de recherche.</span></em></p>Conservatives have long tried to attack regulators such as the EPA with “weaponized transparency”. Coupled with the inflation of uncertainty, the intent is to make regulations impossible.Andrea Saltelli, Open Evidence Research, Universitat Oberta de Catalunya and Centre for the Study of the Sciences and the Humanities (SVT), University of BergenLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/916072018-02-16T11:41:14Z2018-02-16T11:41:14ZIt’s time to end the debate about video games and violence<figure><img src="https://images.theconversation.com/files/206656/original/file-20180215-131006-1pe88hs.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Playing violent video games doesn't make kids more aggressive.</span> <span class="attribution"><a class="source" href="http://www.apimages.com/metadata/Index/Supreme-Court-Violent-Video-Games/f3c4b3da009842d7b09a18d1829d671f/6/0">AP Photo/Paul Sakuma</a></span></figcaption></figure><p>In the wake of the <a href="http://www.sun-sentinel.com/local/broward/parkland/florida-school-shooting/fl-sb-active-shooter-marjory-stoneman-high-20180214-story.html">Valentine’s Day shooting at a Broward County, Florida high school</a>, a familiar trope has reemerged: Often, when a young man is the shooter, people try to blame the tragedy on violent video games and other forms of media. Florida lawmaker Jared Moskowitz <a href="http://www.miamiherald.com/news/local/community/broward/article200279494.html">made the connection</a> the day after the shooting, saying the gunman “was prepared to pick off students like it’s a video game.”</p>
<p>In January, after <a href="https://www.nbcnews.com/news/us-news/one-dead-after-shooting-kentucky-high-school-multiple-victims-reported-n840171">two students were killed and many others wounded</a> by a 15-year-old shooter in Benton, Kentucky, the state’s governor criticized popular culture, <a href="https://www.yahoo.com/news/kentucky-shooting-suspect-ordered-held-035727084.html">telling reporters</a>, “We can’t celebrate death in video games, celebrate death in TV shows, celebrate death in movies, celebrate death in musical lyrics and remove any sense of morality and sense of higher authority and then expect that things like this are not going to happen.”</p>
<p>But, speaking as a researcher who has studied violent video games for almost 15 years, I can state that there is no evidence to support these claims that violent media and real-world violence are connected. As far back as 2011, the U.S. Supreme Court ruled that <a href="https://www.law.cornell.edu/supct/html/08-1448.ZS.html">research did not find a clear connection</a> between violent video games and aggressive behavior. Criminologists who study mass shootings specifically refer to those sorts of connections as a “<a href="https://doi.org/10.1177/1088767913510297">myth</a>.” And in 2017, the Media Psychology and Technology division of the American Psychological Association released a <a href="https://div46amplifier.com/2017/06/12/news-media-public-education-and-public-policy-committee/">statement</a> I helped craft, suggesting reporters and policymakers cease linking mass shootings to violent media, given the lack of evidence for a link.</p>
<h2>A history of a moral panic</h2>
<p>So why are so many policymakers inclined to blame violent video games for violence? There are two main reasons. </p>
<p>The first is the psychological research community’s efforts to <a href="http://www.apa.org/monitor/2011/02/stem.aspx">market itself</a> as strictly scientific. This led to a <a href="http://www.slate.com/articles/health_and_science/science/2016/10/why_the_replication_crisis_seems_worse_in_psychology.html">replication</a> <a href="https://www.theatlantic.com/science/archive/2016/03/psychologys-replication-crisis-cant-be-wished-away/472272/">crisis</a> instead, with researchers often <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">unable to repeat the results</a> of their studies. Now, psychology researchers are reassessing their analyses of a wide range of issues – not just violent video games, but <a href="https://www.thecut.com/2017/01/psychologys-racism-measuring-tool-isnt-up-to-the-job.html">implicit racism</a>, <a href="https://www.npr.org/2016/10/01/496093672/power-poses-co-author-i-do-not-believe-the-effects-are-real">power poses</a> and more.</p>
<p>The other part of the answer lies in the <a href="http://dx.doi.org/10.1037/a0030597">troubled history</a> of violent video game research specifically. Beginning in the early 2000s, some scholars, anti-media advocates and professional groups like the APA began working to connect a <a href="http://dx.doi.org/10.1037/a0035569">methodologically messy</a> and often contradictory set of results to public health concerns about violence. This echoed historical patterns of moral panic, such as <a href="https://doi.org/10.1007/978-1-349-27458-1_6">1950s concerns about comic books</a> and <a href="https://www.nytimes.com/2015/10/26/us/two-pop-culture-wars-first-over-comics-then-over-music.html">Tipper Gore’s efforts to blame pop and rock music</a> in the 1980s for violence, sex and satanism.</p>
<p>Particularly in the early 2000s, dubious evidence regarding violent video games was <a href="https://www.apa.org/science/about/psa/2003/10/anderson.aspx">uncritically promoted</a>. But over the years, confidence among scholars that violent video games influence aggression or violence <a href="http://doi.org/10.1111/jcom.12293">has crumbled</a>.</p>
<h2>Reviewing all the scholarly literature</h2>
<p>My own research has examined the degree to which violent video games can – or can’t – predict youth aggression and violence. In a 2015 <a href="http://doi.org/10.1177/1745691615592234">meta-analysis</a>, I examined 101 studies on the subject and found that violent video games had little impact on kids’ aggression, mood, helping behavior or grades.</p>
<p>Two years later, I found <a href="https://doi.org/10.1016/j.ijlp.2017.11.001">evidence that scholarly journals’ editorial biases</a> had distorted the scientific record on violent video games. Experimental studies that found effects were more likely to be published than studies that had found none. This was <a href="http://doi.org/10.1037/bul0000074">consistent with others’ findings</a>. As the Supreme Court noted, any impacts due to video games are <a href="https://www.law.cornell.edu/supct/html/08-1448.ZS.html">nearly impossible to distinguish</a> from the effects of other media, like cartoons and movies. </p>
<p>Any claims that there is consistent evidence that violent video games encourage aggression are simply false.</p>
<p><a href="https://doi.org/10.1111/jcom.12129">Spikes in violent video games’ popularity</a> are well-known to correlate with substantial declines in youth violence – not increases. These correlations are very strong, stronger than most seen in behavioral research. More recent research suggests that the <a href="http://dx.doi.org/10.1037/ppm0000030">releases of highly popular</a> violent video games are associated with <a href="http://doi.org/10.1002/soej.12139">immediate declines</a> in violent crime, hinting that the releases may cause the drop-off.</p>
<h2>The role of professional groups</h2>
<p>With so little evidence, why are people like Kentucky Gov. Matt Bevin still trying to blame violent video games for mass shootings by young men? Can groups like the National Rifle Association seriously blame <a href="https://www.huffingtonpost.com/entry/nra-gun-violence-hollywood-fox-news_us_59d6cb0be4b072637c430e4b">imaginary guns</a> for gun violence?</p>
<p>A key element of that problem is the willingness of professional guild organizations such as the <a href="https://www.apa.org/">APA</a> to promote false beliefs about violent video games. (I’m a fellow of the APA.) These groups mainly exist to promote a profession among news media, the public and policymakers, <a href="https://www.apa.org/advocacy/index.aspx">influencing licensing and insurance laws</a>. They also make it easier to get grants and newspaper headlines. Psychologists and psychology researchers like myself pay them yearly dues to increase the public profile of psychology. But there is a risk the general public may mistake promotional positions for objective science.</p>
<p>In 2005 the APA released its first <a href="https://videogames.procon.org/sourcefiles/resolution-on-violence-in-video-games-and-interactive-media.pdf">policy statement</a> linking violent video games to aggression. However, my <a href="https://doi.org/10.1016/j.ijlp.2018.01.004">recent analysis of internal APA documents</a> with <a href="https://www.scopus.com/authid/detail.uri?authorId=56382474100">criminologist Allen Copenhaver</a> found that the APA ignored inconsistencies and methodological problems in the research data.</p>
<p>The APA <a href="http://www.apa.org/about/policy/violent-video-games.aspx">updated</a> its statement in 2015, but that sparked controversy immediately: More than <a href="https://www.scribd.com/doc/223284732/Scholar-s-Open-Letter-to-the-APA-Task-Force-On-Violent-Media-Opposing-APA-Policy-Statements-on-Violent-Media">230 scholars</a> wrote to the group asking it to stop releasing policy statements altogether. I and others objected to <a href="http://www.newsweek.com/apa-video-games-violence-364394">perceived conflicts of interest and lack of transparency</a> tainting the process.</p>
<p>It’s bad enough that these statements misrepresent the actual scholarly research and misinform the public. But it’s worse when those falsehoods give advocacy groups like the NRA cover to shift blame for violence onto nonissues like video games. The resulting misunderstandings delay efforts to address mental illness and other issues that are actually related to gun violence.</p><img src="https://counter.theconversation.com/content/91607/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Christopher J. Ferguson does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>For years, there have been questions about research showing connections between playing violent video games and aggressive behavior.Christopher J. Ferguson, Professor of Psychology, Stetson University Licensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/840322018-01-03T11:20:41Z2018-01-03T11:20:41ZNovelty in science – real necessity or distracting obsession?<figure><img src="https://images.theconversation.com/files/199939/original/file-20171219-5004-1ecssnn.jpg?ixlib=rb-1.1.0&rect=693%2C5%2C2809%2C1943&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">It may take time for a tiny step forward to show its worth.</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/man-grey-suit-hold-light-left-541269598">ellissharp/Shutterstock.com</a></span></figcaption></figure><p>In a <a href="https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970">survey of over 1,500 scientists</a>, more than 70 percent of them reported having been unable to reproduce other scientists’ findings at least once. Roughly half of the surveyed scientists ran into problems trying to reproduce their own results. No wonder people are talking about a “<a href="https://theconversation.com/us/topics/reproducibility-5484">reproducibility crisis</a>” in scientific research – an epidemic of studies that <a href="https://thenextregeneration.wordpress.com/2013/07/23/replicability-of-high-impact-papers-in-stem-cell-research/">don’t hold up</a> when <a href="https://thenextregeneration.wordpress.com/2013/10/26/the-replicability-crisis-in-cancer-research/">run a second time</a>.</p>
<p>Reproducibility of findings is a core foundation of science. If scientific results only hold true in some labs but not in others, then how can researchers feel confident about their discoveries? How can society put evidence-based policies into place if the evidence is unreliable?</p>
<p>Recognition of this “crisis” has prompted calls for reform. Researchers are feeling their way, experimenting with different practices meant to help distinguish solid science from irreproducible results. Some people are even starting to reevaluate how choices are made about what research actually gets tackled. Breaking innovative new ground is flashier than revisiting already published research. Does prioritizing novelty naturally lead to this point?</p>
<h2>Incentivizing the wrong thing?</h2>
<p>One solution to the reproducibility crisis could be simply to conduct lots of replication studies. For instance, the <a href="https://elifesciences.org/collections/9b1e83d1/reproducibility-project-cancer-biology">scientific journal eLife</a> is participating in an initiative to validate and reproduce important recent findings in the field of cancer research. The first set of these “rerun” studies was recently released and <a href="http://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304">yielded mixed results</a>. The results of 2 out of 5 research studies were reproducible, one was not and two additional studies did not provide definitive answers.</p>
<p>There’s no need to restrict these sort of rerun studies to cancer research – reproducibility issues can be spotted across <a href="https://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596">various fields</a> <a href="https://theconversation.com/half-of-biomedical-research-studies-dont-stand-up-to-scrutiny-and-what-we-need-to-do-about-that-45149">of scientific research</a>.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/199940/original/file-20171219-4995-ddcteg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Researchers should be rewarded for carefully shoring up the foundations of the field.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/scientist-working-laboratory-38872966">Alexander Raths/Shutterstock.com</a></span>
</figcaption>
</figure>
<p>But there’s at least one major obstacle to investing time and effort in this endeavor: the quest for novelty. The <a href="https://doi.org/10.3389/fnhum.2013.00291">prestige of an academic journal</a> depends at least partly on how often the research articles it publishes are cited. Thus, research journals often want to publish novel scientific findings which are more likely to be cited, not necessarily the results of newly rerun older research.</p>
<p>A <a href="https://doi.org/10.1016/j.jclinepi.2012.06.009">study of clinical trials published in medical journals</a> found the most prestigious journals prefer publishing studies considered highly novel and not necessarily those that have the most solid numbers backing up the claims. Funding agencies such as the National Institutes of Health ask scientists who review research grant applications to provide an “innovation” score in order to <a href="https://grants.nih.gov/grants/peer/critiques/rpg_D.htm">prioritize funding for the most innovative work</a>. And scientists of course notice these tendencies – one study found the use of positive words like “novel,” “amazing,” “innovative” and “unprecedented” in paper abstracts and titles <a href="https://doi.org/10.1038/nature.2015.19024">increased almost ninefold between 1974 and 2014</a>.</p>
<p>Genetics researcher <a href="http://dbbs.wustl.edu/faculty/Pages/faculty_bio.aspx?SID=5137">Barak Cohen</a> at Washington University in St. Louis <a href="https://doi.org/10.7554/eLife.28699">recently published a commentary</a> analyzing this growing push for novelty. He suggests that progress in science depends on a delicate balance between novelty and checking the work of other scientists. When rewards such as funding of grants or publication in prestigious journals emphasize novelty at the expense of testing previously published results, science risks developing cracks in its foundation.</p>
<h2>Houses of brick, mansions of straw</h2>
<p>Cancer researcher William Kaelin Jr., a recipient of the <a href="http://www.laskerfoundation.org/awards/show/oxygen-sensing-essential-process-survival/">2016 Albert Lasker Award for Basic Medical Research</a>, <a href="http://www.nature.com/news/publish-houses-of-brick-not-mansions-of-straw-1.22029">recently argued</a> for fewer “mansions of straw” and more “houses of brick” in scientific publications.</p>
<p>One of his main concerns is that scientific papers now inflate their claims in order to emphasize their novelty and the relevance of biomedical research for clinical applications. By exchanging depth of research for breadth of claims, researchers may be at risk of compromising the robustness of the work. By claiming excessive novelty and impact, researchers may undermine its actual significance because they may fail to provide solid evidence for each claim. </p>
<p>Kaelin even suggests that some of his <a href="http://www.pnas.org/content/93/20/10595">own work from the 1990s, which transformed cell biology research</a> by discovering how cells can sense oxygen, may have struggled to get published today.</p>
<p>Prestigious journals often now demand complete scientific stories, from basic molecular mechanisms to proving their relevance in various animal models. Unexplained results or unanswered questions are seen as weaknesses. Instead of publishing one exciting novel finding that is robust, and which could spawn a new direction of research conducted by other groups, researchers now spend years gathering a whole string of findings with broad claims about novelty and impact.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=397&fit=crop&dpr=1 600w, https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=397&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=397&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=499&fit=crop&dpr=1 754w, https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=499&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/199942/original/file-20171219-4980-14si8bk.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=499&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">There should be more than one path to a valuable journal publication.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/divergence-paths-forest-crossroads-among-many-681313621">Mehaniq/Shutterstock.com</a></span>
</figcaption>
</figure>
<h2>Balancing fresh findings and robustness</h2>
<p>A challenge for editors and reviewers of scientific manuscripts is assessing the novelty and likely long-term impact of the work they’re assessing. The eventual importance of a new, unique scientific idea is sometimes difficult to recognize even by peers who are grounded in existing knowledge. Many basic research studies form the basis of future practical applications. One recent study found that of basic research articles that received at least one citation, <a href="https://theconversation.com/tracing-the-links-between-basic-research-and-real-world-applications-82198">80 percent were eventually cited by a patent application</a>. But it takes time for practical significance to come to light.</p>
<p>A collaborative team of economics researchers <a href="https://doi.org/10.1016/j.respol.2017.06.006">recently developed an unusual measure of scientific novelty</a> by carefully studying the references of a paper. They ranked a scientific paper as more novel if it cited a diverse combination of journals. For example, a scientific article citing a botany journal, an economics journal and a physics journal would be considered very novel if no other article had cited this combination of varied references before.</p>
<p>This measure of novelty allowed them to identify papers which were more likely to be cited in the long run. But it took roughly four years for these novel papers to start showing their greater impact. One may disagree with this particular indicator of novelty, but the study makes an important point: It takes time to recognize the full impact of novel findings. </p>
<p>Realizing how difficult it is to assess novelty should give funding agencies, journal editors and scientists pause. Progress in science depends on new discoveries and following unexplored paths – but solid, reproducible research requires an equal emphasis on the robustness of the work. By restoring the balance between demands and rewards for novelty and robustness, science will achieve even greater progress.</p><img src="https://counter.theconversation.com/content/84032/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Jalees Rehman receives funding from the National Institutes of Health (NIH). </span></em></p>Scientists are rewarded with funding and publications when they come up with innovative findings. But in the midst of a ‘reproducibility crisis,’ being new isn’t the only thing to value about research.Jalees Rehman, Professor of Medicine, Pharmacology and Bioengineering, University of Illinois ChicagoLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/868652017-11-09T12:39:28Z2017-11-09T12:39:28ZScience’s credibility crisis: why it will get worse before it can get better<figure><img src="https://images.theconversation.com/files/193759/original/file-20171108-26972-17v1ar7.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Science itself needs to be put under the microscope and carefully scrutinised to deal with its flaws.</span> <span class="attribution"><span class="source"> Nattapat Jitrungruengnij/Shutterstock</span></span></figcaption></figure><p>Science’s credibility crisis is making headlines once more thanks to a <a href="http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12461/full">paper</a> from John P. A. Ioannidis and co-authors. Ioannidis, an expert in statistics, medicine and health policy at Stanford University, has done more than anyone else to ring the alarm bells on science’s quality control problems: scientific results are published which other researchers cannot reproduce. </p>
<p>When the crisis erupted in the media in 2013 The Economist devoted it’s <a href="https://www.google.es/search?q=economist++science+goes+wrong&client=firefox-b&dcr=0&source=lnms&tbm=isch&sa=X&ved=0ahUKEwi-uu-D2JrXAhWFOhQKHYcZDF0Q_AUICigB&biw=1920&bih=971#imgrc=6VLfyJMejNEoVM:">cover</a> to “<a href="https://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needs-change-itself-how-science-goes-wrong">Wrong Science</a>”. Ionannidis’s <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124">work</a> was an important part of the background material for the <a href="https://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble">piece</a>.</p>
<p>In previous papers Ioannidis had mapped the troubles of fields such as <a href="https://www.nature.com/nature/journal/v483/n7391/full/483531a.html">pre-clinical</a> and <a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002049">clinical</a> medical studies; commenting how, <a href="http://www.jclinepi.com/article/S0895-4356(16)00147-5/pdf">under market pressure</a>, clinical medicine has been transformed to finance-based medicine.</p>
<p>In this new <a href="http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12461/full">work</a> he and co-authors target empirical economics research. They conclude that the field is diseased, with one fifth of the subfields investigated showing a 90% incidence of under-powered studies – a good indicator of low-quality research – and a widespread bias in favour of positive effects. </p>
<p>The field of psychology had gone through a similar ordeal. Brian Nosek, professor of psychology at the University of Virginia and his co-workers ran a <a href="http://science.sciencemag.org/content/349/6251/aac4716">replication analysis</a> of 100 high-profile psychology studies and reported that only about one third of the studies could be replicated. </p>
<p>Several other instances of bad science have gained attention in the media.
The problems in <a href="https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/comment-page-1/">“priming research”</a>, relevant to marketing and advertising, prompted Nobel Prize winner Daniel Kahneman to issue a publicised statement of <a href="https://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535">concern</a> about the wave of failed replication. </p>
<p>And a study on “power poses”, which claimed that body posture influences a person’s hormones level and “feelings of power” went first viral on <a href="https://www.ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are">TED</a> when it was published – then again when its replication <a href="https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html?_r=0&utm_content=bufferab1e2&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer">failed</a>. </p>
<p>We are observing two new phenomena. On the one hand doubt is shed on the quality of entire scientific fields or sub-fields. On the other this doubt is played out in the open, in the media and blogosphere.</p>
<h2>Fixes</h2>
<p>In his <a href="http://onlinelibrary.wiley.com/doi/10.1111/ecoj.12461/full">newest work</a> Ioannidis sets out a list of remedies that science needs to adopt urgently. These include fostering a culture of replication, data sharing and more collaborative works that pool together larger data sets; along with pre-specification of the protocol including model specifications and the analyses to be conducted.</p>
<p><a href="http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001747">Ioannidis</a> has previously proposed additional remedies to “fix” science, as have <a href="http://orca.cf.ac.uk/97336/">other investigators</a>. The list includes better statistical methods and better teaching of statistics as well as measures to restore the right system of incentives at all stages of the scientific production system – from peer review to academic careers. </p>
<p>Important work is already being done by committed individuals and communities, among them Nosek’s <a href="https://osf.io/ezcuj/">Reproducibility Project</a>, Ioannidis’ <a href="https://metrics.stanford.edu/">Meta-research innovation centre</a>, Ben Goldacre’s <a href="http://www.alltrials.net/">alltrials.net</a> and the activities of <a href="http://retractionwatch.com/">Retraction Watch</a>. These initiatives – which attracted <a href="https://www.wired.com/2017/01/john-arnold-waging-war-on-bad-science/">private funding</a> – are necessary and timely.</p>
<p>But what are the chances that these remedies will work? Will this crisis be solved any time soon?</p>
<h2>Methods, incentives and introspection</h2>
<p>Ioannidis and co-authors are aware of the interplay between methods and incentives. For example, they say they’d refrain from suggesting that underpowered studies go unpublished, “as such a strategy would put pressure on investigators to report unrealistic and inflated power estimates based on spurious assumptions”.</p>
<p>This is a crucial point. Better practices will only be adopted if new incentives gain traction. In turn the incentives will have traction only if they address the right set of science’s problems and contradictions. </p>
<p>Ethics is a crucial issue in this respect. And here is where research effort is lacking. The broader field of economics is aware of its ethical problems after Paul Romer – now chief economist of the World Bank – coined the new term “<a href="https://paulromer.net/wp-content/uploads/2015/05/Mathiness.pdf">Mathiness</a>”, to signify the use of mathematics to veil normative premises. Yet there seem to be some hesitation to join the dots from the methodology to the ethos of the discipline, or of science overall.</p>
<p>The book <a href="https://www.amazon.com/Rightful-Place-Science-Verge/dp/0692596380">Science on the Verge</a> has proposed an analysis of the root causes of the crisis, including its neglected ethical dimension. The formulation of remedial measures depends on understanding <a href="http://www.sciencedirect.com/science/article/pii/S0016328717301969">what happened to science</a> and how this reflects on its <a href="https://theconversation.com/to-tackle-the-post-truth-world-science-must-reform-itself-70455">social role</a>, including when science feeds into <a href="http://www.sciencedirect.com/science/article/pii/S0016328717300472">evidence based policy</a>. </p>
<p>These analyses are indebted to philosophers Silvio O. Funtowicz and Jerome R. Ravetz, who spent several decades studying the <a href="https://en.wikipedia.org/wiki/Uncertainty_and_quality_in_science_for_policy">science’s quality control arrangements</a> and how quality and uncertainty impacted the <a href="http://www.sciencedirect.com/science/article/pii/001632879390022L">use of science for policy</a>.</p>
<p>Ravetz’s book “<a href="https://en.wikipedia.org/wiki/Scientific_Knowledge_and_Its_Social_Problems">Scientific knowledge and its social problems</a>” published in 1971 predicted several relevant features of the present crisis.</p>
<p>For Ravetz it is possible for a field <a href="https://en.wikipedia.org/wiki/Scientific_Knowledge_and_Its_Social_Problems">to be diseased</a>, so that shoddy work is routinely produced and accepted. Yet, he notes, it will be far from easy to come to accept the existence of such a condition – and even more difficult to reform it. </p>
<p>Reforming a diseased field or arresting the incipient decline of another will be delicate tasks, adds Ravetz, which calls for a </p>
<blockquote>
<p>sense of integrity, and a commitment to good work, among a significant section of the members of the field; and committed leaders with scientific ability and political skill. No quantity of published research reports, nor even an apparatus of institutional structures, can do anything to maintain or restore the health of a field in the absence of this essential ethical element operating through the interpersonal channel of communication.</p>
</blockquote>
<p>Ravetz emphasises the loss of this essential ethical element. In later works he notes that the new social and ethical conditions of science are reflected in a set of <a href="http://www.andreasaltelli.eu/file/repository/Maturing_Contradictions_2011_1.pdf">“emerging contradictions”</a>. These concern the cognitive dissonance between the official image of science as enlightened, egalitarian, protective and virtuous, against the current realities of scientific dogmatism, elitism and corruption; of science serving corporate interests and practices; of science used as an ersatz religion. </p>
<p>Echoes of Ravetz’s analysis can be found in many recent works, such as on the <a href="http://www.hup.harvard.edu/catalog.php?isbn=9780674046467">commodification of science</a>, or on the present problems <a href="https://theconversation.com/science-in-crisis-from-the-sugar-scam-to-brexit-our-faith-in-experts-is-fading-65016">with trust in expertise</a>. </p>
<h2>A call to arms?</h2>
<p>Ioannidis and co-authors are careful to stress the importance of a multidisciplinary approach, as both troubles and solutions may spill over from one discipline to the other. This would perhaps be a call to the arms for social scientists in general – and for those who study science itself – to tackle the crisis as a priority. </p>
<p>Here we clash with another of science’s contradictions: at this point in time, to study science as a scholar would mean to criticise its mainstream image and role. We do not see this happening any time soon. Because of the scars of “science wars” – whose spectre is <a href="https://theconversation.com/science-wars-in-the-age-of-donald-trump-67594">periodically resuscitated</a> – social scientists are wary of being seen as attacking science, or worse helping US President Donald Trump. </p>
<p>Scientists overall wish to use their <a href="https://theconversation.com/forcing-consensus-is-bad-for-science-and-society-77079">moral authority</a> and association with Enlightenment values, as seen in the recent <a href="https://theconversation.com/a-scientists-march-on-washington-is-a-bad-idea-heres-why-73305">marches for science</a>. </p>
<p>If these contradictions are real, then we are condemned to see the present crisis becoming worse before it can become better.</p><img src="https://counter.theconversation.com/content/86865/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Andrea Saltelli does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>We are observing two new phenomena. On one hand doubt is shed on the quality of entire scientific fields or sub-fields. On the other this doubt is played out in the open, in the media and blogosphere.Andrea Saltelli, Adjunct Professor Centre for the Study of the Sciences and the Humanities, University of Bergen, University of BergenLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/809972017-07-19T17:01:16Z2017-07-19T17:01:16ZHere’s the three-pronged approach we’re using in our own research to tackle the reproducibility issue<figure><img src="https://images.theconversation.com/files/178674/original/file-20170718-31872-1uv1xdv.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Step one is not being afraid to reexamine a site that's been previously excavated.</span> <span class="attribution"><span class="source">Dominic O'Brien. Gundjeihmi Aboriginal Corporation</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span></figcaption></figure><p>If you keep up with health or science news, you’ve probably been whipsawed between conflicting reports. Just days apart you may hear that “science says” coffee’s good for you, no actually it’s bad for you, actually red wine holds the secret to long life. As <a href="https://www.youtube.com/watch?v=0Rnq1NpHdmw">comedian John Oliver put it</a>:</p>
<blockquote>
<p>“After a certain point, all that ridiculous information can make you wonder: is science bullshit? To which the answer is clearly no. But there is a lot of bullshit currently masquerading as science.”</p>
</blockquote>
<p>A big part of this problem has to do with what’s been called a “<a href="https://theconversation.com/us/topics/reproducibility-5484">reproducibility crisis</a>” in science – many studies if run a second time don’t come up with the same results. <a href="https://doi.org/10.1038/533452a">Scientists are worried</a> about this situation, and <a href="https://www.nature.com/collections/byblhcfwhw">high-profile</a> international <a href="https://doi.org/10.1126/science.aab2374">research journals</a> have raised the alarm, too, calling on researchers to put more effort into ensuring their results can be reproduced, rather than only striving for splashy, one-off outcomes.</p>
<p><a href="https://www.nytimes.com/2016/05/29/opinion/sunday/why-do-so-many-studies-fail-to-replicate.html">Concerns about</a> <a href="https://www.theatlantic.com/science/archive/2016/03/psychologys-replication-crisis-cant-be-wished-away/472272/">irreproducible results</a> <a href="http://www.slate.com/articles/health_and_science/future_tense/2016/04/biomedicine_facing_a_worse_replication_crisis_than_the_one_plaguing_psychology.html">in science resonate</a> <a href="https://fivethirtyeight.com/features/science-isnt-broken/">outside the ivory tower</a>, as well, because a lot of this research translates into information that affects our everyday lives. </p>
<p>For example, it informs what we know about how to stay healthy, how doctors should look after us when we’re sick, how best to educate our children and how to organize our communities. If study results are not reproducible, then we can’t trust them to give good advice on solving our everyday problems – and society-wide challenges. Reproducibility is not just a minor technicality for specialists; it’s a pressing issue that affects the role of modern science in society.</p>
<p>Once we’ve identified that reproducibility is a big problem, the question becomes: How do we tackle it? Part of the answer has to do with changing incentives for researchers. But there are plenty of things we in the research community can do right now in the course of our scientific work.</p>
<p>It might come as a surprise that <a href="https://doi.org/10.1007/s10816-015-9272-9">archaeologists are at the forefront</a> of finding ways to improve the situation. Our <a href="https://doi.org/10.1038/nature22968">recent paper in Nature</a> demonstrates a concrete three-pronged approach to improving the reproducibility of scientific findings.</p>
<h2>Going back to where it all started</h2>
<p>In our new publication we describe recent work at an archaeological site in northern Australia. The results of our excavations and laboratory analyses show that <a href="http://theconversation.com/buried-tools-and-pigments-tell-a-new-history-of-humans-in-australia-for-65-000-years-81021">people arrived in Australia 65,000 years ago</a>, substantially earlier than the previous consensus estimate of 47,000 years ago. <a href="http://theconversation.com/buried-tools-and-pigments-tell-a-new-history-of-humans-in-australia-for-65-000-years-81021">This date has exciting implications</a> for our understandings of human evolution.</p>
<p>A less obvious detail about this study is the care we’ve taken to make our results reproducible. Our reproducibility strategy had three parts: fieldwork, labwork and data analyses.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=906&fit=crop&dpr=1 600w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=906&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=906&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1138&fit=crop&dpr=1 754w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1138&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/178680/original/file-20170718-10320-1sapmfd.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1138&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Ben Marwick and colleagues excavating at Madjedbebe.</span>
<span class="attribution"><span class="source">Dominic O'Brien. Gundjeihmi Aboriginal Corporation</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>Our first step toward reproducibility was our choice of what to investigate. Rather than striking out to someplace new, we reexcavated an archaeological site <a href="https://doi.org/10.1016/j.jhevol.2015.03.014">previously known to have very old artifacts</a>.</p>
<p>The rockshelter site Madjedbebe in Australia’s Northern Territory had been excavated twice before. Famously, excavations there in 1989 indicated that people had <a href="https://doi.org/10.1038/345153a0">arrived in Australia by about 50,000 years ago</a>. But this age was not accepted by many archaeologists, who refused to accept anything older than 47,000 years ago.</p>
<p>This age was controversial from its first publication, and our goal in revisiting the site was to check if it was reliable or not. Could that controversial 50,000-years age be reproduced, or was it just a chance result that didn’t indicate the true time period for human habitation in Australia?</p>
<p>Like many scientists, archaeologists are generally less interested in returning to old discoveries, instead preferring to forge new paths in search of novel results. The problem with this is that it can lead to many unresolved questions, making it difficult to build a solid foundation of knowledge. </p>
<h2>Double-check the lab tests</h2>
<p>The second part of our reproducibility strategy was to verify that our laboratory analyses were reliable.</p>
<p>Our team used <a href="https://www.thoughtco.com/luminescence-dating-cosmic-method-171538">optically stimulated luminescence</a> methods to date the sand grains near the ancient artifacts. This method is complex, and there are only a few places in the world that have the instruments and skills to date these kinds of samples.</p>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=766&fit=crop&dpr=1 600w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=766&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=766&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=963&fit=crop&dpr=1 754w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=963&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/178820/original/file-20170719-27696-r2h9i8.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=963&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Zenobia Jacobs produced the new ages for the Madjebdebe site based on her work in the Luminescence Dating Laboratory at the University of Wollongong, Australia.</span>
<span class="attribution"><span class="source">University of Wollongong</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>We first analyzed our samples in our laboratory at the <a href="http://smah.uow.edu.au/sees/facilities/UOW002889.html">University of Wollongong</a> to find their ages. Then we sent blind duplicate samples to another laboratory at the <a href="https://www.adelaide.edu.au/ipas/facilities/luminescence/">University of Adelaide</a> to analyze, without telling that lab our results. With both sets of analyses in hand, we compared them; it turned out in this case that they got the same ages as we did for the same samples.</p>
<p>This kind of verification is not a common practice in archaeology, but because this site was already controversial, we wanted to make sure the ages we obtained were reproducible.</p>
<p>While this extra work involved some additional cost and time, it’s vital to proving that our dates give the true ages of the sediments surrounding the artifacts. This verification shows that our lab results are not due to chance, or the unique conditions of our laboratory. Other archaeologists, and the public, can be more confident in our findings because we’ve taken these extra steps. This external checking should be standard practice in any science where controversial findings are at stake. </p>
<h2>Don’t let the computer be a black box</h2>
<p>After we completed the excavation and lab analyses, we analyzed the data on our computers. This stage of our research was very similar to what scientists in many other fields do. We loaded the raw data into our computers to visualize it with plots and test hypotheses with statistical methods.</p>
<p>However, while many researchers do this work by pointing and clicking using off-the-shelf software, we tried as much as possible to write scripts in the <a href="https://doi.org/10.1038/517109a">R programming language</a>.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=402&fit=crop&dpr=1 600w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=402&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=402&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=505&fit=crop&dpr=1 754w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=505&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/178686/original/file-20170718-10283-q6g5bg.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=505&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Could be the enemy of reproducibility if it helps obscure the steps in data analysis.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/erinkohlenbergphoto/5353222369">Erin Kohlenberg</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span>
</figcaption>
</figure>
<p>Pointing and clicking generally leaves no traces of important decisions made during data analysis. Mouse-driven analyses leave the researcher with a final result, but none of the steps to get that result is saved. This makes it <a href="https://theconversation.com/how-computers-broke-science-and-what-we-can-do-to-fix-it-49938">difficult to retrace the steps</a> of an analysis, and check the assumptions made by the researcher.</p>
<p>On the other hand, our scripts contain a record of all our data analysis steps and decisions. They’re like an exact recipe to generate our results. Other researchers not using scripts for their data analysis don’t have these recipes, so their results are much harder to reproduce. </p>
<p>Another advantage of our choice to use scripts is that we can share them with the scientific community and the public. We follow <a href="https://doi.org/10.1038/nn.4550">standard practices</a> by making our script files and main data files <a href="https://osf.io/qwfcz/">freely available online</a> so anyone can inspect the details of our analysis, or explore new ideas using our data.</p>
<p>It’s easy to understand why many researchers prefer point-and-click over writing scripts for their data analysis. Often that’s what they were taught as students. It’s hard work and time-consuming to learn new analysis tools among the pressures of teaching, applying for grants, doing fieldwork and writing publications. Despite these challenges, there is an accelerating shift away from point-and-click toward scripted analyses in many areas of science.</p>
<h2>Combating irreproducibility one step at a time</h2>
<p>Our recent paper is part of a new movement emerging in many disciplines to improve the reproducibility of science. Examples of recent papers that have made a commitment to reproducibility similar to ours have come from <a href="https://doi.org/10.1038/nature22975">epidemiology</a>, <a href="https://doi.org/10.1038/s41559-017-0160">oceanography</a> and <a href="https://doi.org/10.7554/eLife.20470">neuroscience</a>.</p>
<p>We hope our example will inspire other scientists to be strategic about improving the reproducibility of their research. Some of these steps can be difficult for researchers: It means learning how to use unfamiliar software, and publicly sharing more of their data and methods than they’re accustomed to. But they’re important for generating reliable results – and for maintaining public confidence in scientific knowledge.</p><img src="https://counter.theconversation.com/content/80997/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Ben Marwick receives funding from the Australian Research Council, the University of Wollongong, and the University of Washington. This work was supported in part by the University of Washington eScience Institute.</span></em></p><p class="fine-print"><em><span>Zenobia Jacobs receives funding from the Australian Research Council. </span></em></p>A team of archaeologists strived to improve the reproducibility of their results, influencing their choices in the field, in the lab and during data analysis.Ben Marwick, Associate Professor of Archaeology, University of WashingtonZenobia Jacobs, Professor, University of WollongongLicensed as Creative Commons – attribution, no derivatives.