tag:theconversation.com,2011:/fr/topics/research-spin-3819/articlesResearch spin – The Conversation2016-03-10T11:12:00Ztag:theconversation.com,2011:article/548752016-03-10T11:12:00Z2016-03-10T11:12:00ZWhen good intentions aren’t supported by social science evidence: diversity research and policy<figure><img src="https://images.theconversation.com/files/114137/original/image-20160307-31275-mlwc4c.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Successful group outcomes aren't guaranteed by the simple recipe of 'Just add diversity.'</span> <span class="attribution"><a class="source" href="http://www.shutterstock.com/pic-273543236/stock-photo-business-people-diversity-talking-communication-concept.html">Talking image via www.shutterstoc.com.</a></span></figcaption></figure><p>You’d be forgiven for assuming a quick and sure way to multiply profits and amplify organizational success is to increase the gender and racial diversity of any group. According to claims in the mainstream media, the effects of gender and racial diversity are universally favorable. News stories tend to mirror this 2014 <a href="https://www.washingtonpost.com/news/on-leadership/wp/2014/09/24/more-women-at-the-top-higher-returns/"><em>Washington Post</em> article’s claim</a> that “researchers have long found <a href="http://www.americanbanker.com/bankthink/women-on-boards-improve-a-banks-performance-1063776-1.html">ties between having women</a> on a <a href="https://www.washingtonpost.com/news/on-leadership/wp/2013/11/27/more-women-on-boards-cheaper-mergers/">company’s board of directors</a> and <a href="http://www.washingtonpost.com/blogs/on-leadership/wp/2014/06/04/an-index-fund-that-bets-on-women/">better financial performance</a>.” </p>
<p>And as <a href="http://www.nytimes.com/2013/10/24/opinion/kristof-twitter-women-power.html">Nicholas Kristoff wrote</a> in <em>The New York Times</em> in 2013:</p>
<blockquote>
<p>Scholarly research suggests that the best problem-solving doesn’t come from a group of the best individual problem-solvers, but from a diverse team whose members complement each other. That’s an argument for leadership that is varied in every way — in gender, race, economic background and ideology.</p>
</blockquote>
<p>The truth is there’s actually no adequate scientific basis for these newsworthy assertions. And this lack of scientific evidence to guide such statements illustrates the troubled relations of science to advocacy and policy, that I have <a href="http://doi.org/10.1111/josi.12163">analyzed in an article</a> in the current Journal of Social Issues.</p>
<h2>A chasm between research findings and advocates’ claims</h2>
<p>I began to think more deeply about these issues during my recent service as president of the <a href="http://www.spssi.org">Society for the Psychological Study of Social Issues</a>. This organization has worked since 1936 to join social science findings to responsible advocacy and effective social policy.</p>
<p>This goal is laudable, but the task is supremely challenging. As I’ve come to realize, different camps have varying goals. Scientists aim to produce valid knowledge. Advocates work to promote their favored causes. Policymakers hope to efficiently deploy resources to attain social and economic ends. And they’re all assuming their claims are supported by the same body of social science research.</p>
<p>In politically sensitive areas, advocates may eagerly invoke social scientific data that support their objectives but ignore nonsupportive findings. They may highlight politically congenial findings that are unrepresentative of the available scientific knowledge. </p>
<p>Researchers, in turn, may fail to communicate their findings effectively. Communication is challenging when study outcomes are more complex and less affirming of advocates’ goals than what they desire and expect.</p>
<p>These issues often arise when research addresses controversial questions of social inequality. That’s where social science myths can and do emerge.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/113898/original/image-20160304-17734-yyab4k.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Does who fills these empty boardroom chairs affect the bottom line?</span>
<span class="attribution"><a class="source" href="http://www.shutterstock.com/pic.mhtml?id=141135247&src=lb-29877982">Boardroom image via www.shutterstock.com</a></span>
</figcaption>
</figure>
<h2>Case study: diversity research</h2>
<p>To illustrate these problems, consider two prominent social science myths about diversity.</p>
<p>One concerns the effects of the gender diversity of corporate boards of directors on firms’ financial performance. The other pertains to the effects of the gender and racial diversity of workgroups on their performance.</p>
<p><a href="https://www.2020wob.com/sites/default/files/2020GDI-2015Report.pdf">Advocates for diversity</a> generally maintain that the addition of women to corporate boards <a href="http://www.catalyst.org/knowledge/companies-behaving-responsibly-gender-diversity-boards">enhances corporate financial success</a>. And they hold that diversity in task groups <a href="http://www.forbes.com/sites/stevedenning/2012/01/16/why-is-diversity-vital-for-innovation/#1cb510fa4e7c">enhances their effectiveness</a>.</p>
<p><a href="http://dx.doi.org/10.1016/j.obhdp.2012.06.003">Abundant findings</a> have accumulated on <a href="http://doi.org/10.5465/amj.2013.0319">both of these questions</a> – more than 140 studies of corporate boards and more than 100 studies of sociodemographic diversity in task groups. Both sets of studies have produced mixed outcomes. Some studies show positive associations of diversity to these outcomes, and some show negative associations.</p>
<p>Social scientists use meta-analyses to integrate such findings across the relevant studies. Meta-analyses represent <em>all</em> the available studies on a particular topic by quantitatively averaging their findings and also examining differences in studies’ results. Cherry-picking is not allowed. </p>
<p>Taking into account all of the <a href="http://dx.doi.org/10.2139/ssrn.2696804">available research on corporate boards</a> and diversity of task groups, the net effects are very close to a null, or zero, average. Also, economists’ studies that carefully evaluate causal relations have <a href="http://dx.doi.org/10.1016/j.jfineco.2008.10.007">typically failed to find that women cause superior corporate performance</a>. The most valid conclusion at this point is that, on average, diversity neither helps nor harms these important outcomes.</p>
<p>Given these overall findings, further studies are needed to identify the conditions under which diversity has positive or negative effects. And there is some progress here. </p>
<p>For example, research suggests that diversity tends to make decision-making groups more effective if their members <a href="http://dx.doi.org/10.5465/amj.2009.0823">create norms that foster personal ties</a> across the races and genders as well as the exchange of ideas. Also, a <a href="http://dx.doi.org/10.1016/j.obhdp.2013.03.003">positive and inclusive mindset about diversity</a> increases the chances of favorable effects on group performance.</p>
<p>But such conditions are often absent. <a href="http://dx.doi.org/10.1037/a0025767">Diversity can create tensions</a> within groups, and the newly introduced female or minority group members may encounter resistance that makes it difficult for them to gain a foothold in decision-making. It’s hardly surprising that the results of empirical studies are inconsistent. These kinds of interpersonal relationships are messy and complicated – it makes sense that upping diversity, on its own, wouldn’t be a magical key to success.</p>
<h2>A worthwhile social outcome</h2>
<p>What’s the harm in journalists announcing false generalizations about diversity if such statements help increase the number of women and minorities in important roles? After all, most people would agree that it would be an egregious violation of equal opportunity and antidiscrimination laws to exclude women and minorities from opportunities merely on the basis of their sex or race. Isn’t any and all support for inclusion valuable? My answer to this question is no.</p>
<p>First of all, social science myths make a mockery of evidence-based advocacy and policy. In fact, an unusually large body of social science evidence has emerged in tests of the effects of diversity on corporate success and group performance. Advocacy and policy should build on this research, not ignore it.</p>
<p>Myths also set people up to expect that corporate financial gains and superior group performance follow easily from diversity. Of course they don’t. That expectation could sideline people from understanding and overcoming diversity’s challenges.</p>
<p>Finally, false generalizations can impede progress toward better science that may disentangle the causes of diversity’s varied effects on group and organizational success.</p>
<p>Social scientists should freely admit that diversity science doesn’t have all the answers. At the same time, they should not silently tolerate distortions of available scientific knowledge to fit advocacy goals. Ideally, researchers are honest brokers who communicate consensus scientific findings to the broader public. Only then can social science make a meaningful contribution to building sound social policy.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=448&fit=crop&dpr=1 600w, https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=448&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=448&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=563&fit=crop&dpr=1 754w, https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=563&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/113901/original/image-20160304-17723-1xsw5gj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=563&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">There are other reasons to value diversity in the group.</span>
<span class="attribution"><span class="source">Table image via www.shutterstockcom</span></span>
</figcaption>
</figure>
<h2>Social justice goals are valid on their own</h2>
<p>Many advocates and policymakers share the admirable goal of producing a more just society. But they’re narrow-minded if they focus only on whether diversity and inclusion foster outcomes such as business profits or effective group problem-solving. The more fundamental gains from diversity pertain to social justice. Diversity and inclusion can serve social justice goals by countering discrimination that may have put women and minorities at a disadvantage.</p>
<p>Beyond countering possible discrimination lies an even more fundamental social justice consideration – that of equitable representation. This principle holds that citizens in democracies should have equal access to influencing the decisions that shape their lives. To the extent that women and minorities are not represented in decision-making groups in proportion to their numbers in the population, they are unlikely to have their interests fairly represented. </p>
<p>As <a href="http://dx.doi.org/10.2307/2647821">political scientists have pointed out</a>, the ideals of democracy are violated if decision-making is dominated by the rich, the white and the male. Then the <a href="http://dx.doi.org/10.1111/j.1540-5907.2011.00569.x">needs of the poor, the minorities</a> <a href="http://dx.doi.org/10.1146/annurev.polisci.11.053106.123839">and the female</a> likely are neglected.</p>
<p>Most advocates, policymakers and social scientists may not be aware of sharp divergence in their claims about diversity. Yet, policy based on sound social science should be a shared goal. Without understanding the causal relations in society that this research helps identify, policymakers lower the odds they’ll reach their targets. Policy based on myths and hunches has little chance of success. To achieve evidence-based policy, all parties should take a close look at what diversity research has produced so far. Rather than selectively featuring congenial results, they should work together to untangle diversity’s complex effects on group and organizational performance.</p><img src="https://counter.theconversation.com/content/54875/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Alice H. Eagly does not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>The relationship between social science research and advocates and policymakers is undermined if they cherry-pick evidence that supports their goals, ignoring the wider field.Alice H. Eagly, Professor of Psychology; Faculty Fellow Institute for Policy Research; Professor of Management and Organizations, Northwestern UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/324312014-10-07T03:45:05Z2014-10-07T03:45:05ZWhat kind of research can we trust?<figure><img src="https://images.theconversation.com/files/60976/original/bpdqn58n-1412643107.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Conflicting recommendations about flu drugs has made it difficult for doctors to decide whether to prescribe them. </span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/stanrandom/3754123623">Andrew Wales/Flickr</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Research involving pharmaceutical company input is notoriously compromised. While not all industry ties lead to biased research, and not all biases are a consequence of industry ties, <a href="http://www.ncbi.nlm.nih.gov/pubmed/23235689">many studies show</a> industry influence can make drugs look safer and more effective than they really are. So where can doctors and indeed the public turn to for reliable information?</p>
<p>One favoured option is research known as systematic reviews, which sift through evidence, evaluate their quality and synthesise conclusions and recommendations for clinical practice. Systematic reviews are considered to be the highest level of medical evidence because they summarise large volumes of evidence and follow strict processes to avoid biases. </p>
<p>Systematic reviews form the basis of evidence-based medicine, but there’s now growing doubt about whether these reviews are as untouched by industry influence as many of us expect them to be.</p>
<h2>A particular case</h2>
<p>Consider the case of a class of drugs known as neuraminidase inhibitors, which has been <a href="https://theconversation.com/the-tamiflu-saga-shows-why-all-research-data-should-be-public-13951">causing controversy</a> in the last few years. These drugs are said to minimise the impact of the flu; you’ll know them by their commercial names Tamiflu and Relenza. </p>
<p>Tens of millions of prescriptions for these drugs have been dispensed and governments worldwide have stockpiled them in preparation for a flu pandemic at the cost of billions of dollars. But there are conflicting views about both their safety and their efficacy – and they’re fuelled by conflicting systematic reviews. </p>
<p>One <a href="http://www.ncbi.nlm.nih.gov/pubmed/24815805">systematic review published this year</a>, for instance, encouraged early use of the drugs in any patient who looks appreciably unwell. <a href="http://www.bmj.com/content/348/bmj.g2545">Another cautioned</a> about their safety and questioned whether they should be used in practice at all.</p>
<p>In an article <a href="http://www.annals.org/article.aspx?doi=10.7326/M14-0933">published today in the Annals of Internal Medicine</a>, we tried to make sense of how such discrepancies arise despite the strict processes that underpin systematic reviews. </p>
<p>Given what we already know about industry influence on research, we suspected the differences might be associated with reviewers’ financial ties to companies that make the drugs. To test our hypothesis, we examined 26 systematic reviews published about neuraminidase inhibitors. </p>
<h2>Sleight of hand?</h2>
<p>We found reviewers with financial ties to drug companies were more likely to present evidence in favourable ways and recommend use of the drugs. In the reviews written by researchers with such ties, 88% of the conclusions were favourable. In the absence of financial links, just 17% were positive.</p>
<p>In other words, reviewers with financial ties to drug manufacturers overwhelmingly decided the drugs were safe and effective while those without ties were considerably more reserved about their value.</p>
<p>So how did the systematic reviews arrive at such different conclusions?</p>
<p>While we were unable to examine the differences statistically, one part of the review process stood out as the point where biases could be more easily introduced: generalising from results to recommendations.</p>
<p>For some systematic reviews, the recommendations made in the discussion sections didn’t match the evidence in the results. That suggests reviewers may have generalised in ways that aligned with predetermined views rather than what the evidence showed.</p>
<h2>What can be done?</h2>
<p>Ours is not the only study that has identified this type of problem. Last year, researchers identified the same association in <a href="http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1001578">systematic reviews of sweetened beverages and weight gain</a>.</p>
<p>While this may make it tempting to ignore all evidence reported by researchers who receive industry funding, we don’t think that’s the answer. There’s much to be gained from collaborations with industry. What we need are <a href="http://www.futuremedicine.com/doi/pdf/10.2217/cer.14.31">better strategies for managing conflicts of interest</a>.</p>
<p>Being able to detect the kind of polarisation in the conclusions of systematic reviews we did is one step towards managing the effects of conflicts of interest. And one way to mitigate these effects may be to ask independent researchers to interpret results and formulate recommendations.</p>
<p>As with other drugs, conflicting recommendations about neuraminidase inhibitors has made it difficult for doctors to decide whether to prescribe them. The most authoritative reviews now show these drugs have small benefits and some risks. These reviews have led to <a href="http://www.theguardian.com/world/2014/apr/10/uk-wasted-560m-stockpiling-flu-drugs">suggestions</a> that <a href="http://www.smh.com.au/federal-politics/political-news/antiviral-drug-stockpile-a-waste-of-money-says-study-20140410-zqt3i.html">stockpiling them</a> may have <a href="http://www.theatlantic.com/magazine/archive/2009/12/the-truth-about-tamiflu/307801/">been unjustified</a>.</p>
<p>To be able to make informed decisions together, doctors and patients need research that’s trustworthy. If systematic reviews are to remain the pinnacle of evidence-based medicine, then the processes underpinning them need to be continually reassessed to ensure they meet the highest of standards.</p><img src="https://counter.theconversation.com/content/32431/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Adam Dunn receives funding from the National Health & Medical Research Council.</span></em></p><p class="fine-print"><em><span>Florence Bourgeois receives funding from the National Institute of Health in the USA.</span></em></p>Research involving pharmaceutical company input is notoriously compromised. While not all industry ties lead to biased research, and not all biases are a consequence of industry ties, many studies show…Adam Dunn, Senior Research Fellow, UNSW SydneyFlorence Bourgeois, Assistant Professor of Pediatrics, Harvard UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/70522012-11-29T00:10:17Z2012-11-29T00:10:17ZPutting psychological research to the test with the Reproducibility Project<figure><img src="https://images.theconversation.com/files/18041/original/8fd3kpy6-1353978927.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Statistical significance doesn't speak directly to the reproducibility of an experimental effect.</span> <span class="attribution"><span class="source">Daniel Leininger</span></span></figcaption></figure><p>An ambitious new project is attempting to replicate every single study published in 2008 in three leading academic psychology journals. It’s called the <a href="http://openscienceframework.org/project/EZcUj/wiki/home">Reproducibility Project.</a> </p>
<p>As the name suggests, the aim is to explore just how reproducible the results of psychological experiments are, and the <a href="http://m.pps.sagepub.com/content/7/6.toc">current issue</a> of Perspectives in Psychological Science is dedicated to it. It’s a laudable goal, but why is it necessary? Surely statistical analysis of experimental data should tell us whether we’re likely to see the same result again.</p>
<h2>What statistics don’t tell us</h2>
<p>There’s a widespread misconception that the statistical analysis typically reported in scientific journals address replication. In particular, many people, including researchers themselves, believe the “statistical significance” of a result speaks directly to the reproducibility of the experimental effect. It does not.</p>
<p>Readers of scientific papers may be familiar with the term “statistical significance”. It’s often expressed as p<.05 after a result, or as an asterisk in a table or figure referring to “significance at the 5% level”. In psychology, statistical significance tests are used to support primary outcomes in <a href="http://pss.sagepub.com/content/18/3/230.extract">97% of empirical articles</a>.</p>
<p>The most common misinterpretation is that p<.05 means there’s a less than 5% probability that the experimental effect is due to chance. (Read this bit carefully — the preceding sentence described a misconception.) From this “due to chance” misconception, we quickly arrive at the false conclusion that the effect is very probably real and that <em>it will replicate</em>.</p>
<p>In fact, a <em>p</em> value is a conditional probability: the probability of observing a particular experimental result, or one more extreme, given that it doesn’t actually exist in the world. The reason that statement doesn’t equate to “due to chance”, “reproducible” or “real” is that it only describes one type of error – the error of finding something that isn’t really there. It doesn’t say anything about the chance of missing what is there, or even indicate how hard we looked! </p>
<p>Statistical significance depends on the size of effect (that is, how much difference the drug or therapy made), the variability in the sample (that’s how much people vary in their reactions to the drug or therapy), as well as other design features of the experiment, including sample size. </p>
<p>In psychology, small sample sizes, modest treatment effects and considerable individual differences (high amounts of variability) work together to create low statistical power in many experiments. Statistical power tells us how likely it is that a given experiment will detect an effect of a certain magnitude as “statistically significant”, if the effect really exists in the world. </p>
<p>Calculations of the average <a href="http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=1991-03239-001">statistical power</a> of published psychology experiments hovers at around 50%. This means that conducting an average psychology experiment is roughly equivalent to flipping a coin, in terms of whether you get a statistically significant result or not. </p>
<p>Many statistically non-significant results are therefore not good evidence of “no effect”, and many statistically significant results that get published are false positives, as we explain below.</p>
<h2>Publication bias and false positives</h2>
<p>An average statistical power of 50%, combined with journals’ biases towards only publishing statistically significant results produces a skewed literature, one that potentially only tells half the story – the statistically significant half. We simply don’t hear about the studies that failed to reach the significance threshold, even though there may be more of them. Those studies stay locked in <a href="http://psycnet.apa.org/journals/bul/86/3/638/">file drawers</a>.</p>
<p>Publication bias pushes the number of false positives in the literature <a href="http://people.psych.cornell.edu/%7Ejec7/pcd%20pubs/simmonsetal11.pdf">far beyond the 5% rate</a> we expect from tests that report p<.05. False positive results detect statistically significant effects when there are no real effects there, like a pregnancy test that reports you are pregnant when you are not. </p>
<p>Add to the mix external pressure to publish — from funding agencies or drug companies – and flexible research designs allowing researchers to stop collecting data when they cross the statistical significance threshold (rather than a predetermining a sample size and sticking to it), and the false positive rate grows even higher.</p>
<p>It might be inconvenient that typical statistics don’t provide us with direct information about the reproducibility of results, but it needn’t be the downfall of the scientific enterprise. The problem lies in falsely believing, <a href="http://www.psycontent.com/content/v1t03r87k1351332/?p=10cf16e5c75748a1b4e2c54c65407c32&pi=9">as many psychology researchers do</a>, that a p<.05 means that it’s really likely you’ll see another low p value next time. </p>
<p>If you think p values tell you what you need to know about replication, you’re much less likely to actually replicate a study. And that means that false positive results can loiter in the literature for a very long time, distorting our understanding of issues, misleading scientific research, and underpinning inappropriate management or policy decisions.</p>
<p>In 2005, John P. Ioannidis made headlines when he claimed that up to 90% of published medical findings <a href="http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124">may be false</a>. Ioannidis described conditions of small sample sizes, small effect sizes, publication bias, pressure to publish and flexible stopping rules — all the problems we identify above. His quantitative conclusions about error rates and false positives were based on simulations, not “real” data.</p>
<p>Unfortunately, looking at real data is just as disheartening. Over the past decade, a group of researchers attempted to replicate 53 <a href="http://www.nature.com.ezp.lib.unimelb.edu.au/nature/journal/v483/n7391/full/483531a.html">“landmark” cancer studies</a>. They were interested in how many would again produce results deemed strong enough to drive a drug-development program (their definition of reproducibility). Of those 53 studies, the results of only six could be robustly reproduced.</p>
<p>It seems reproducible results are pretty hard to come by. Quantifying exactly how hard is what the Reproducibility Project is all about. From it, we’ll learn a lot about which psychological phenomena are real, and which aren’t. We may also learn a lot about how poor statistical practice can delay progress and mislead science.</p><img src="https://counter.theconversation.com/content/7052/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Fiona Fidler has received funding from the ARC.</span></em></p><p class="fine-print"><em><span>Bonnie Claire Wintle does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>An ambitious new project is attempting to replicate every single study published in 2008 in three leading academic psychology journals. It’s called the Reproducibility Project. As the name suggests, the…Fiona Fidler, Senior Research Fellow, The University of MelbourneBonnie Claire Wintle, PhD student, The University of MelbourneLicensed as Creative Commons – attribution, no derivatives.