tag:theconversation.com,2011:/fr/topics/research-impact-3362/articlesResearch impact – The Conversation2023-11-14T19:07:26Ztag:theconversation.com,2011:article/2175412023-11-14T19:07:26Z2023-11-14T19:07:26Z‘You only assess what you care about’: a new report looks at how we assess research in Australia<figure><img src="https://images.theconversation.com/files/559197/original/file-20231113-17-hzq74f.jpg?ixlib=rb-1.1.0&rect=23%2C35%2C7880%2C5214&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><a class="source" href="https://www.pexels.com/photo/photo-of-female-engineer-looking-through-wires-3862623/">ThisIsEngineering/Pexels</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Research plays a pivotal role in society. Through research, we gain new understandings, test theories and make discoveries. </p>
<p>It also has a huge economic value. In 2021, the <a href="https://www.csiro.au/en/work-with-us/services/consultancy-strategic-advice-services/CSIRO-futures/Innovation-Business-Growth/Quantifying-Australias-returns-to-innovation">CSIRO found</a> every A$1 of research and development investment in Australia creates an average of $3.50 in economy-wide benefits. </p>
<p>But how do we know if individual research projects being conducted in Australia are good quality? How is research recognised? The key way this happens is through “research assessment”. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/tumult-and-transformation-the-story-of-australian-universities-over-the-past-30-years-215536">Tumult and transformation: the story of Australian universities over the past 30 years</a>
</strong>
</em>
</p>
<hr>
<h2>What is research assessment?</h2>
<p>Research assessment is not a centralised or necessarily formal process. It can involve various processes and measures to evaluate the performance of individual researchers and research institutions. This includes assessing the quality, excellence and impact of various outputs. </p>
<p>Research assessment can be qualitative or quantitative. It can include publications in journals and the number of people who cite the research, gaining grants to do further research, commercialisation, media engagement and impact on decision-making or public policy, prizes and invitations to speak at conferences. </p>
<p>If research assessment is working fairly and effectively, it should achieve several things. This includes: helping to develop researchers’ careers, making sure innovative research does not get avoided in favour of short-term gains and helping funders and the community have confidence research is providing value for money and adding to the public good. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/we-solve-problems-in-30-days-through-research-sprints-other-academics-can-do-this-too-204373">We solve problems in 30 days through 'research sprints': other academics can do this too</a>
</strong>
</em>
</p>
<hr>
<h2>Our project</h2>
<p>Our new project aimed to provide a better understanding of how research assessment affects research in Australia. </p>
<p>In a <a href="https://acola.org/wp-content/uploads/2023/11/ACOLA_ResearchAssessment_FINAL.pdf">report released today</a>, we surveyed more than 1,000 Australian researchers and more than 50 research organisations. </p>
<p>This included universities, research institutes, industry bodies, government and not-for-profit organisations. The majority of researchers (74%) were in academic roles. Across those research sectors, we also conducted 11 roundtables involving around 120 people and 25 intensive interviews to understand the issues.</p>
<p>This work was commissioned by Chief Scientist Cathy Foley and conducted by the Australian Council of Learned Academies (involving the academies of science, medical science, engineering and technological sciences, social sciences and humanities). </p>
<p>It also comes as the <a href="https://www.education.gov.au/australian-universities-accord">Universities Accord review</a> examines how research is funded and approached within higher education. </p>
<figure class="align-center ">
<img alt="A young man searches the shelves of a library." src="https://images.theconversation.com/files/559200/original/file-20231114-21-9oaq62.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/559200/original/file-20231114-21-9oaq62.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/559200/original/file-20231114-21-9oaq62.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/559200/original/file-20231114-21-9oaq62.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/559200/original/file-20231114-21-9oaq62.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/559200/original/file-20231114-21-9oaq62.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/559200/original/file-20231114-21-9oaq62.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Research assessment should help to develop researchers’ careers.</span>
<span class="attribution"><a class="source" href="https://www.pexels.com/photo/male-student-searching-at-book-shelves-6549376/">Tima Miroshnichenko/Pexels</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span>
</figcaption>
</figure>
<h2>What we found</h2>
<p>We found some difficulties with the current approach to research assessment. </p>
<p>We heard there is a tendency by some researchers to “play it safe” in terms of doing research they believe will score well. We also heard how the assessment process can unintentionally exclude or devalue particular forms of knowledge, particularly in the humanities and the social sciences, where outputs can be less easily quantified or less immediately seen.</p>
<p>As one interviewee said: </p>
<blockquote>
<p>What is assessed and how it is assessed are an indication of what the
organisation values. You only assess what you care about. Values and
culture drive assessment.</p>
</blockquote>
<p>Our roundtables told us senior staff and supervisors are often seen to reinforce the culture of “publish or perish”, with the number of articles being valued more highly than the quality. </p>
<p>We heard early and mid-career researchers and people from underrepresented backgrounds can have difficulties trying to “play the game” to advance their careers. For example, early-career researchers are often expected to produce work that benefits their larger team, at a cost to their own capacity for promotion. </p>
<p>As one interviewee noted: </p>
<blockquote>
<p>Metrics are essential for defining value and comparative difference, but
Australia requires a modern and fair framework for assessing our current
and next generation of researchers.</p>
</blockquote>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1650983340751613952"}"></div></p>
<h2>Survey results</h2>
<p>Our survey found a high level of dissatisfaction with the state of research assessment. This included: </p>
<ul>
<li><p>73% of respondents agreed assessment processes are not consistently or
equitably applied across disciplines, in particular between the humanities and the sciences </p></li>
<li><p>67% said there are not enough opportunities to provide input into research assessment practices</p></li>
<li><p>70% said assessments took up unreasonable time and effort. </p></li>
</ul>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/fieldwork-can-be-challenging-for-female-scientists-here-are-5-ways-to-make-it-better-214215">Fieldwork can be challenging for female scientists. Here are 5 ways to make it better</a>
</strong>
</em>
</p>
<hr>
<h2>The way forward</h2>
<p>In our survey, we asked “What is one specific change you would
recommend to improve current research assessment processes?”.</p>
<p>Respondents wanted to see a shift towards quality over quantity. This means not just a focus on publishing as many papers as possible, but supporting research that may take longer for its value and benefits to emerge. </p>
<p>They wanted interdisciplinary research to be promoted and rewarded, because many of the complex problems of our world – from climate change to domestic violence to housing affordability – require multiple disciplines to be involved in finding solutions. In the same vein, they also wanted collaboration and team work to be rewarded more clearly and transparently. </p>
<p>They wanted less bias towards STEM (science, technology, engineering and maths) research and more promotion of diversity and of early-career researchers. This included better understanding of their personal and cultural situation, more focused career development and better managed teamwork.</p>
<p>To achieve all of this, and more, we will also need to understand that no single measure can assess all research or researchers. So, several tools will be needed, including quantitative indicators as well as qualitative measures and peer review.</p>
<hr>
<p><em>Ana Deletic, Louisa Jorm, Duncan Ivison, Robyn Owens, Jill Blackmore, Adrian Barnett, Kate Thomann, Caroline Hughes, Andrew Peele, Guy Boggs and Raffaella Demichelis were all part of the expert working group supporting this work.</em></p><img src="https://counter.theconversation.com/content/217541/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Kevin McConkey has previously received funding from the Australian Research Council. He is the current chair of the Policy Committee of the Academy of the Social Sciences in Australia. He is the chair of the Expert Working Group of the the Australian Council of Learned Academies, which prepared the report referred to in this article.</span></em></p>The project, spanning researchers across science and the humanities, looks at how ‘research assessment’ affects research in Australia.Kevin McConkey, Emeritus Professor , UNSW SydneyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1815302022-05-26T13:15:14Z2022-05-26T13:15:14ZClimate change isn’t a priority for Kenyan universities. It should be<figure><img src="https://images.theconversation.com/files/463614/original/file-20220517-6205-7bpy4j.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Increased financing will enable universities in Kenya to make climate change activities a central part of curricula and research output.
</span> <span class="attribution"><span class="source">Peter Cade via GettyImages</span></span></figcaption></figure><p>Universities can play a vital role in shaping change in societies. They produce knowledge through research, train future decision makers, and contribute to public awareness of issues. </p>
<p>But not all universities are rising to challenges like this. <a href="https://www.ajol.info/index.php/ajsd/article/view/111117">Studies</a> have established that efforts to integrate sustainable development into African universities’ curricula and their community engagement processes leave something to be desired. Teaching and research don’t always reflect society’s real problems.</p>
<p>Climate change is another area where universities should be identifying and providing solutions. It’s a complex, politicised and global issue which needs informed leadership.</p>
<p>Kenya, for one country, has neglected the integration of climate change into the education system. None of the commissions that have looked into the education system over the years have dealt with this. In his research, Dr Charles Kariuki <a href="https://ir-library.ku.ac.ke/bitstream/handle/123456789/18546/Curriculum%20and%20its%20contribution%20to%20awareness......pdf?sequence=1&isAllowed=y">noted</a> that Kenya’s education policies treat climate change casually. Hardly any learning about it is taking place at any level. </p>
<p>A 2015 <a href="https://www.researchgate.net/publication/279179079_Climate_Change_Knowledge_Gap_in_Education_System_in_Kenya">survey</a> of two public universities in Kenya found that they were yet to incorporate climate-related issues into their programmes. This creates a big gap in the production and dissemination of knowledge on climate change. It also limits climate change mitigation and adaptation within the education sector. </p>
<p>There’s very little in the literature generally on how climate change is represented in Kenyan universities’ curricula, campus activities, institutional governance and community engagement work. Our <a href="https://www.climate-uni.com/_files/ugd/f81108_1fe481d96de94247b5b3a5baab268f33.pdf">working paper</a> sought to fill the gap. </p>
<p>We reviewed the climate change policy environment in the country and the link between national policies and university policies, actions and practices. Our aim was to understand what Kenyan universities were doing to raise awareness and to create capacity to respond to climate change. Responses could include mitigation, adaptation, impact reduction and early warning. </p>
<p>Drawing on national and international policy documents, peer-reviewed journal articles and national climate change reports published between 1999 and 2020, we found a gap. Universities in Kenya aren’t receiving enough guidance from the government in responding to the impacts of climate change. This means they’re also not producing outputs, such as research, that could offer guidance to society more broadly.</p>
<h2>From policy to practice</h2>
<p>Kenya has an elaborate policy framework that addresses climate change matters in specific sectors. These include the <a href="https://cdkn.org/sites/default/files/files/Kenya-National-Climate-Change-Action-Plan.pdf">National Climate Change Action Plan 2013–2017</a> and the <a href="http://www.environment.go.ke/wp-content/uploads/2018/05/National-Adaptation-Plan-NAP.pdf">Kenya National Adaptation Plan 2015–2030</a>. The various policies draw from global declarations like the <a href="https://legal.un.org/avl/ha/ccc/ccc.html">United Nations Framework Convention on Climate Change</a> and the <a href="https://unfccc.int/process-and-meetings/the-paris-agreement/the-paris-agreement">Paris Agreement</a>. They are anchored in the constitution and the Kenya’s <a href="https://vision2030.go.ke/">development vision 2030</a>.</p>
<p>The country’s <a href="http://www.environment.go.ke/wp-content/documents/complete%20nccrs%20executive%20brief.pdf">National Climate Change Response Strategy</a> proposes that schools and colleges include climate change information in their curricula and syllabuses. The adaptation plan 2015-2030 also emphasises the need to integrate climate change content into curricula at all levels. </p>
<p>It’s clear therefore that national policies and strategies recognise the potential of university education to solve climate change issues through teaching, research and community service. But universities in Kenya still lag in coming up with or adopting policies to guide climate action at institutional levels. There’s also inadequate teaching and learning in this area. </p>
<h2>Progress</h2>
<p>Kenyan universities have made some efforts in sustainable development. For instance, in 2014 they established the <a href="https://www.kgun.org">Kenya Green University Network</a>. The aim was to promote greening approaches, such as the use of renewable and clean energy, at university campuses. But the results are few. </p>
<p>The private Strathmore University and the public Kenyatta University have the largest solar installations in the region. They generate <a href="https://strathmore.edu/news/towards-a-sustainable-strathmore-university-600-kw-grid-connected-solar-energy-system/">600KW</a>and <a href="https://www.saurenergy.com/solar-energy-news/kenyatta-university-turns-100kw-solar-power-plant">100KW</a> of solar power respectively.</p>
<p>Our review showed that mostly, universities treat sustainability targets, such as carbon emissions reduction, as more of a government requirement than their own ambition. A number of studies we reviewed confirmed that climate change content was treated casually at all levels of the education system. </p>
<h2>The gaps</h2>
<p>Even though there’s an intricate climate change policy framework at the national level, our review found no evidence that policies are carried through to strategies in higher education. They’re not showing up in curricula or in campus greening activities. </p>
<p>Universities in Kenya should be more sensitive to national policies aimed at addressing the effects of climate change. The national government needs to clarify the role universities should play in the governance of climate change affairs. And it should provide related research funding. </p>
<p>Increased financing, which is currently a barrier to research activities, will enable universities in Kenya to make climate change activities a central part of curricula, campus activities, institutional governance and community engagement work. The universities regulatory authority, the <a href="https://www.cue.or.ke/">Commission for University Education</a>, should also include climate action as a criterion in evaluating university programmes. This would motivate action.</p><img src="https://counter.theconversation.com/content/181530/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Jackline Nyerere does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Universities in Kenya should be more sensitive to national policies aimed at addressing the effects of climate change.Jackline Nyerere, Senior Lecturer of Educational Leadership and Policy, Kenyatta UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1579712021-03-30T13:30:56Z2021-03-30T13:30:56ZTurning findings into policy: six tips for researchers<figure><img src="https://images.theconversation.com/files/392490/original/file-20210330-17-1brjo8g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Scientists and advocates around the world are in support of evidence-based, science-informed public policies.
</span> <span class="attribution"><span class="source">Joseph Gruber/Shutterstock</span></span></figcaption></figure><p>There has been tremendous <a href="https://gh.bmj.com/content/6/2/e004129">growth</a> in the number of studies on sexual and reproductive health in sub-Saharan Africa in the past two decades. Notably, there has been an increase in research documenting <a href="https://www.ghspjournal.org/content/3/3/333">what works in improving adolescents’ health and wellbeing</a>. </p>
<p>However, the use of findings from these studies to inform <a href="https://globalizationandhealth.biomedcentral.com/articles/10.1186/s12992-020-00605-z">the development of policies is low</a>. For example, research shows that <a href="https://www.sciencedirect.com/science/article/abs/pii/S0140197117301471">educating young people</a> about their sexuality and giving them access to contraceptive methods has lifelong benefits. But few sub-Saharan African countries have enacted laws or policies that follow through on the evidence. </p>
<p>As a result of this inaction, adolescents continue to experience <a href="https://reproductive-health-journal.biomedcentral.com/articles/10.1186/s12978-021-01078-y">early unintended pregnancy</a>, <a href="https://www.sciencedirect.com/science/article/pii/S0010782418303883">unsafe abortion</a>, and other poor health outcomes.</p>
<p>There are many obstacles to the use of research in policy making. Researchers and policymakers rarely interact with each other. Policymakers are, therefore, unaware of the latest research findings. Many policymakers in sub-Saharan Africa are not trained to conduct or use research. Researchers often communicate their findings in technical language. They rarely assess the evidence needs of policymakers or involve them in the research processes. And researchers have a limited understanding of the socio-political processes involved in policymaking.</p>
<p>As researchers, we know that studying social problems and publishing findings do not guarantee that policies will draw on the evidence. The changes that society needs, such as preventing adolescent pregnancies, will not happen until researchers can influence policy. To do this, there are some important steps to follow. </p>
<p>The steps described here are based on the model the African Population and Health Research Center is using in its <a href="https://www.tandfonline.com/doi/full/10.1080/26410397.2021.1881207">Challenging the Politics of Social Exclusion</a> project. They also emerge from a review of other models such as <a href="https://includeplatform.net/about/">INCLUDE</a>, a knowledge sharing platform hosted by the <a href="https://www.ascleiden.nl/">African Studies Centre</a>. The INCLUDE platform was created as a place to share knowledge that can lead to evidence-based development policies and programmes in Africa. </p>
<h2>Challenging the politics of social exclusion</h2>
<p>At the African Population and Health Research Center we are implementing an innovative four-year project that began in 2018. The <a href="https://aphrc.org/project/challenging-the-politics-of-social-exclusion-cpse-a-regional-research-and-advocacy-approach-to-contentious-srhr-issues-in-sub-saharan-africa/">project</a> was sparked by shortcomings in sexual and reproductive rights services. </p>
<p>At inception, we brought together policymakers and civil society organisations to define a <a href="https://www.tandfonline.com/doi/full/10.1080/26410397.2021.1881207">research agenda</a> on these issues for the sub-Saharan Africa region. The priorities that were identified guide our ongoing research. They are: abortion incidence, the effect of laws and policies in reducing maternal deaths, and how to get support from religious and traditional leaders for safe abortion and care. </p>
<p>Through our partnerships with regional economic communities, we also train policymakers on the use of research. Further, we have established a <a href="https://aphrc.org/cpse-rapid-response/">rapid response mechanism</a> to provide evidence to policymakers. Policy change in our three focal areas will take time. But we believe our investment in building the culture of using evidence in policy will help bring the change we want.</p>
<p>The project is ongoing, but we have already learnt more about how scientists can use their research outcomes to influence policy. </p>
<h2>Six tips</h2>
<p><strong>Relationship building:</strong> Researchers and academic institutions must establish and nourish relationships with policy actors. This will build an understanding of the policymaking process. Researchers will know more about what policymakers need, why and when. </p>
<p><strong>Needs assessment:</strong> Researchers should engage with policymakers, civil society organisations, funders, government agencies and politicians to identify policy priorities and relevant evidence needs. Together they can come up with research questions to ensure that studies are relevant. But the engagement must continue during and after the data collection. Policy actors can <a href="https://www.tandfonline.com/doi/full/10.1080/26410397.2021.1881207">play a part</a> in analysing and sharing the findings. Researchers must begin with policy impact in mind.</p>
<p><strong>Mutual capacity strengthening:</strong> Policymakers are not experts in interpreting academic literature. Researchers should not assume that they can translate scientific evidence into policy and practice. So researchers have to identify and document how to help them – through regular training, for example. It’s up to researchers to show policymakers the value of evidence-informed policy. </p>
<p>Researchers can also learn from the vast experience of policymakers, especially in understanding the policy landscape and how to engage them. Thus, capacity strengthening must be mutual with both the researchers and the policymakers gaining valuable knowledge and insights. </p>
<p><strong>Communicating to a variety of audiences:</strong> The language of scientific publication is not always clear to policymakers. Researchers must translate knowledge so that a wider audience can understand it. One place to start is writing and publishing a short summary of the research in plain language for the media. Policymakers need summarised documents with less scientific jargon. </p>
<p><strong>Researchers as activists:</strong> In the words of South African academic Linda-Gail Bekker:</p>
<blockquote>
<p><a href="https://bhekisisa.org/article/2021-03-16-trials-tinsel-tango-get-to-know-researcher-linda-gail-bekker/">being a scientist isn’t enough</a> – you have to be an activist too. </p>
</blockquote>
<p>Researchers must use their voice to advocate for evidence uptake, especially in cases where there is an obvious denial of scientific evidence. Scientists who engage with a larger audience can help dispel myths and misconceptions about scientific discovery. </p>
<p><strong>Establishing a rapid response mechanism:</strong> Lastly, establishing a rapid response mechanism is important to provide timely and up-to-date evidence to policymakers as their needs emerge. Through the Challenging the Politics of Social Exclusion project, the African Population and Health Research Center is rapidly using evidence to inform policy debates. More research institutes should do the same.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/covid-19-how-the-conversation-helps-build-bridges-between-research-and-policy-157932">COVID-19: how The Conversation helps build bridges between research and policy</a>
</strong>
</em>
</p>
<hr>
<h2>What success would look like</h2>
<p>We believe that the uptake of evidence in policy and decision making is key for inclusive and sustainable economic growth and development. Evidence will help decision makers to do what is proven to work, reducing wastage of limited public resources. </p>
<p>Effective and efficient use of resources will reassure citizens and help build trust in governments. These are key ingredients for sustainable development. </p>
<p>But achieving this will require researchers to collaborate more with policymakers in evidence generation, translation and use.</p><img src="https://counter.theconversation.com/content/157971/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Caroline Kabiru works for the African Population and Health Research Center (APHRC). Her staff time is partially covered by a grant from the African Regional Office of the Swedish International Development Cooperation Agency for the Challenging the Politics of Social Exclusion project.</span></em></p><p class="fine-print"><em><span>Anthony Idowu Ajayi and Boniface Ushie do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>The changes that society needs, such as preventing adolescent pregnancies, will not happen until researchers can use their findings to influence policy change.Anthony Idowu Ajayi, Associate research scientist, African Population and Health Research CenterBoniface Ushie, Research Scientist, African Population and Health Research CenterCaroline W. Kabiru, Senior Research Scientist, African Population and Health Research CenterLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1513752021-01-10T19:11:50Z2021-01-10T19:11:50ZUnis want research shared widely. So why don’t they properly back academics to do it?<figure><img src="https://images.theconversation.com/files/377666/original/file-20210107-13-v7ugzm.jpg?ixlib=rb-1.1.0&rect=0%2C0%2C2799%2C1873&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/panel-speaker-on-stage-presenting-vision-1223620837">Life and Times/Shutterstock</a></span></figcaption></figure><p>Academics are <a href="https://www.erudit.org/en/journals/cjhe/2017-v47-n3-cjhe04386/1057102ar/">increasingly expected</a> to share their research widely beyond academia. However, <a href="https://onlinelibrary.wiley.com/doi/10.1002/leap.1365">our recent study</a> of academics in Australia and Japan suggests Australian universities are still very much focused on supporting the production of scholarly outputs. They offer <a href="https://onlinelibrary.wiley.com/doi/10.1002/leap.1365">relatively limited support</a> for researchers’ efforts to engage with the many non-academics who can benefit from our research. </p>
<p>One reason engagement is expected is that government, industry and philanthropic sources fund research. And when academics share their research with the public, industry and policymakers, this engagement is good for the university’s reputation. It can also lead to other benefits such as research funding.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/who-cares-about-university-research-the-answer-depends-on-its-impacts-149817">Who cares about university research? The answer depends on its impacts</a>
</strong>
</em>
</p>
<hr>
<p>But the work involved in sharing our ideas beyond academia can be diverse and substantial. For example, when we write for The Conversation, it takes time to find credible sources, adopt an appropriate tone, communicate often complex ideas simply and clearly, and respond to editor feedback. We also need to be able to speak to the media about our findings, and respond to public comments when the piece comes out. </p>
<h2>Unis don’t allow for the time it takes</h2>
<p>However, as one respondent said in explaining why they were <a href="https://www.tandfonline.com/doi/abs/10.1080/0309877X.2020.1807477?journalCode=cjfh20">not sharing research with end users beyond academia</a>:</p>
<blockquote>
<p>It’s not recognised by uni. So, when it is not recognised, it means that I don’t have any workload for that, and obviously I’m work-loaded for other stuff, and that means that I don’t actually have enough time to do this. </p>
</blockquote>
<p>Sharing our findings beyond academia isn’t typically seen as part of our academic workload. This is problematic for academics who are already <a href="https://bmjopen.bmj.com/content/4/3/e004462.short">struggling to find time</a> to do all the things their complex workload requires of them. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/why-the-kpis-on-university-engagement-need-more-thought-78026">Why the KPIs on university engagement need more thought</a>
</strong>
</em>
</p>
<hr>
<figure class="align-center ">
<img alt="woman concentrates as she types on a laptop" src="https://images.theconversation.com/files/377674/original/file-20210107-15-s16his.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/377674/original/file-20210107-15-s16his.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/377674/original/file-20210107-15-s16his.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/377674/original/file-20210107-15-s16his.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/377674/original/file-20210107-15-s16his.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/377674/original/file-20210107-15-s16his.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/377674/original/file-20210107-15-s16his.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">It takes to write an article or engage with non-academics in other ways, but universities typically don’t treat this work as an integral part of academic duties.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/focused-female-customer-working-on-computer-1514779214">Mangostar/Shutterstock</a></span>
</figcaption>
</figure>
<p>In our research, <a href="https://www.tandfonline.com/doi/abs/10.1080/07294360.2020.1815662">time and workload constraints</a> were the most often-cited barriers to sharing research beyond academia. One respondent said they saw lots of opportunities to <a href="https://www.tandfonline.com/doi/abs/10.1080/07294360.2020.1815662">build partnerships</a> with practitioners in their field, but added:</p>
<blockquote>
<p>[I] just cannot do that, because I’m doing other things that, in my work, are a priority. </p>
</blockquote>
<p>When we spend our time sharing our research with academic readers through journal articles, conference papers and academic books, our employers clearly value and expect these scholarly publications. These works, and <a href="https://link.springer.com/article/10.1007/s11192-020-03691-3">how the scholarly community receives them</a>, have more weight in evaluation of our performance. Last year an Australian academic <a href="https://www.smh.com.au/national/nsw/industrial-umpire-lashes-universities-obsessed-with-rankings-and-reputation-20200311-p5495e.html">nearly lost her job</a> for failing to meet a target for scholarly publications. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/what-are-universities-for-if-mainly-teaching-can-they-sack-academics-for-not-meeting-research-targets-143091">What are universities for? If mainly teaching, can they sack academics for not meeting research targets?</a>
</strong>
</em>
</p>
<hr>
<p>Our research found <a href="https://www.researchgate.net/publication/346974034_Perspectives_on_institutional_valuing_and_support_for_academic_and_translational_outputs_in_Japan_and_Australia#read">Japan-based academics feel a greater weight of expectations</a> than their Australian counterparts to engage with diverse audiences beyond academia. Universities clearly expect this engagement. Yet they often don’t back it up with support such as workload recognition, resourcing and training. </p>
<p>Universities need to offer better support if they wish to increase academics’ engagement with diverse audiences. They should also consider both the benefits and risks of this engagement.</p>
<h2>Academics see the benefits of sharing research</h2>
<p>The academics we spoke with valued the <a href="https://www.tandfonline.com/doi/abs/10.1080/07294360.2020.1815662">benefits</a> of engaging with diverse audiences. They were pleased to see others putting their research to use. Sharing research often helped to secure funding. </p>
<p>They also saw engagement as an opportunity to learn from end users. This helped ensure their research was responding to real-world needs.</p>
<p>Even very early in their careers, many researchers look to engage with audiences beyond academia. In previous research, we found doctoral candidates may opt for a <a href="https://www.tandfonline.com/doi/abs/10.1080/07294360.2018.1498461">thesis by publication</a> rather than a traditional thesis approach due to their <a href="https://www.tandfonline.com/doi/abs/10.1080/0309877X.2019.1671964?journalCode=cjfh20">desire to share findings</a>. </p>
<figure class="align-center ">
<img alt="Doctor and researcher chat about findings" src="https://images.theconversation.com/files/377667/original/file-20210107-17-ku8faw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/377667/original/file-20210107-17-ku8faw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/377667/original/file-20210107-17-ku8faw.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/377667/original/file-20210107-17-ku8faw.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/377667/original/file-20210107-17-ku8faw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/377667/original/file-20210107-17-ku8faw.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/377667/original/file-20210107-17-ku8faw.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Engaging with the end users of their research provides valuable feedback for academics.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/doctor-talking-pharmaceutical-sales-representative-1662004078">Halfpoint/Shutterstock</a></span>
</figcaption>
</figure>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/when-measuring-research-we-must-remember-that-engagement-and-impact-are-not-the-same-thing-56745">When measuring research, we must remember that 'engagement' and 'impact' are not the same thing</a>
</strong>
</em>
</p>
<hr>
<h2>What other problems do researchers face?</h2>
<p>The early-career researchers we interviewed noted other barriers and risks in sharing their work with diverse audiences. Universities often did not help with these issues. </p>
<p>They described <a href="https://www.tandfonline.com/doi/abs/10.1080/07294360.2020.1815662">communication skill gaps</a> when seeking to tailor research content for diverse audiences. For example, the way research is communicated to industry experts needs to be different to how it is shared with governments or the general public. </p>
<p>Researchers may need to learn to communicate their ideas in <a href="https://www.onlinelibrary.wiley.com/doi/abs/10.1002/leap.1296?af=R">many different forms</a>. They may have to be skilled in producing industry reports, doing television or radio interviews or presenting their findings in professional forums. </p>
<p>Some encountered frustrations when sharing research via the <a href="https://www.tandfonline.com/doi/abs/10.1080/07294360.2020.1815662">bureaucratic processes</a> of government. For example, a respondent explained:</p>
<blockquote>
<p>There’s still that much back and forth because there’s three or four different government departments that are involved in the process and it goes to different people. Some people don’t want it to be changed because they’re vested in the old way of doing things, and then they’ve got to bring ministers up to speed, and then all of a sudden you’re got a new state government that comes in, so that all changes. </p>
</blockquote>
<p>Many felt unprepared to <a href="https://www.tandfonline.com/doi/abs/10.1080/0309877X.2020.1807477?journalCode=cjfh20">deal with the media</a>. </p>
<p>One respondent described being cautious about overstating the impact of their research. In their field, they saw messages claiming: “This is the be all and end all. This will cure cancer.” They were “wary of accidentally going down that path and making a claim bigger than is true”.</p>
<p>Respondents also described risks in sharing controversial and sensitive research beyond academia. </p>
<h2>What can universities do?</h2>
<p>For respondents in both Australia and Japan, demanding and diverse workloads crowded out opportunities to share findings. Universities cannot just expect engagement responsibilities to be absorbed into an already swollen workload.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/after-coronavirus-universities-must-collaborate-with-communities-to-support-social-transition-140541">After coronavirus, universities must collaborate with communities to support social transition</a>
</strong>
</em>
</p>
<hr>
<p>If universities are serious about supporting the sharing of research beyond academia, they need to recognise these contributions in meaningful ways. For example, Australian academics usually must meet teaching, research and service requirements in their workloads. If sharing research with audiences beyond academia were counted toward service, academics could have this work properly taken into account in performance management and when seeking promotion. </p>
<p>Universities can do better at supporting academics to share their research with the public, industry and government. Improving access to training and mentoring to communicate research findings both in academia and beyond would be an important step forward.</p><img src="https://counter.theconversation.com/content/151375/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Margaret Kristin Merga receives funding from the BUPA Health Foundation, the Ian Potter Foundation, the Copyright Agency Cultural Fund, Edith Cowan University and the Collier Foundation. She is the current inaugural Patron of the Australian School Library Association.</span></em></p><p class="fine-print"><em><span>Shannon Mason receives funding from the Japan Society for the Promotion of Science.</span></em></p>Researchers will struggle to meet universities’ expectations of engagement beyond academia until this work is better recognised as part of their duties.Margaret Kristin Merga, Senior Lecturer in Education, Edith Cowan UniversityShannon Mason, Assistant Professor, Education, Nagasaki UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1498172020-11-22T18:55:47Z2020-11-22T18:55:47ZWho cares about university research? The answer depends on its impacts<figure><img src="https://images.theconversation.com/files/370230/original/file-20201119-13-jbn15o.png?ixlib=rb-1.1.0&rect=0%2C0%2C1066%2C714&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Few researchers will have the impact of Robert Langer, but they can learn from what researchers like him do.</span> <span class="attribution"><a class="source" href="https://en.wikipedia.org/wiki/Robert_S._Langer#/media/File:Robert_Langer_BioTech_Awards_Video_laboratory.png">Science History Institute/Wikipedia</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span></figcaption></figure><p><em>This essay is based on the <a href="https://www.uts.edu.au/partners-and-community/initiatives/impact-studios/podcasts/impact-uts">Impact at UTS</a> podcast series. The audio series examines how a diverse range of researchers embed knowledge exchange and impact in their research strategy.</em></p>
<hr>
<p>Universities are facing great financial challenges and a swathe of <a href="https://theconversation.com/as-universities-face-losing-1-in-10-staff-covid-driven-cuts-create-4-key-risks-147007">redundancy programs</a> is under way. Many senior academics are retiring early. Those that remain are picking up more teaching load. Research and teaching programs are both at risk of being seriously compromised.</p>
<p>Beyond the individual loss for people who have built careers exploring important research challenges, what may be less apparent is our collective loss as a society if academics are deprived of the time to explore tough questions that we need answers for.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/as-universities-face-losing-1-in-10-staff-covid-driven-cuts-create-4-key-risks-147007">As universities face losing 1 in 10 staff, COVID-driven cuts create 4 key risks</a>
</strong>
</em>
</p>
<hr>
<p>It might be hard to look beyond the immediate crisis in higher education, but universities will remain crucial social institutions. Now is the right time to continue the conversation about what they are and who they serve. And what are their impacts?</p>
<p>As part of our <a href="https://www.uts.edu.au/partners-and-community/initiatives/impact-studios/podcasts/impact-uts">Impact at UTS podcast series</a>, we spoke to researchers about how they navigate collaboration, engagement – with communities, industry and government – and impact. The breadth and depth of these impact stories reveal many inter-related insights, which we present later in this article. (You can listen to a full podcast episode at the end.)</p>
<iframe src="https://webplayer.whooshkaa.com/episode/737026?theme=light&visual=true&enable-volume=true&iframe-height=190" height="190" width="100%" scrolling="no" frameborder="0" allow="autoplay"></iframe>
<h2>Why does university research impact matter?</h2>
<p>Universities are uniquely placed to explore complex problems that our collective future depends on. They can do so in a rigorous, ethical, collaborative and enduring way. Peer review regulates subjectivity and biases. </p>
<p>Investing the time to confront complex problems is often beyond the appetite and patience of a corporate agenda driven by other imperatives, including short-term survival. Nationally, Australia continues to lag in <a href="https://data.oecd.org/rd/gross-domestic-spending-on-r-d.htm">OECD rankings for research and development</a>. This is obviously not desirable, and it’s a symptom of bigger problems in the university sector.</p>
<iframe src="https://data.oecd.org/chart/6axG" width="100%" height="645" style="border: 0" mozallowfullscreen="true" webkitallowfullscreen="true" allowfullscreen="true"><a href="https://data.oecd.org/chart/6axG" target="_blank">OECD Chart: Gross domestic spending on R&D, Total, % of GDP, Annual, 2017</a></iframe>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/budgets-1bn-research-boost-is-a-welcome-first-step-billions-more-plus-policy-reforms-will-be-needed-147662">Budget's $1bn research boost is a welcome first step. Billions more, plus policy reforms, will be needed</a>
</strong>
</em>
</p>
<hr>
<p>The sector has rightly begun to question inward-looking measures of success and KPIs, which are largely based on quantifying research grants in and publications out. Only other researchers care about such things. </p>
<p>Here, we ask why is the research worth doing in the first place? What does it contribute beyond the esteem of academic colleagues? </p>
<p>The COVID-19 crisis has intensified the need to revisit the <a href="https://impactstudios.edu.au/podcasts/the-new-social-contract/">relationship universities have with society</a>. Every academic needs to grapple with questions of why or when research should be prioritised over teaching and upskilling job seekers and job keepers.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/universities-have-gone-from-being-a-place-of-privilege-to-a-competitive-market-what-will-they-be-after-coronavirus-137877">Universities have gone from being a place of privilege to a competitive market. What will they be after coronavirus?</a>
</strong>
</em>
</p>
<hr>
<h2>The challenges of assessing impact</h2>
<p>While a shift away from crude input-output metrics towards research impact sounds appealing, assessing impact is much harder to do at scale. </p>
<p>And, perhaps more importantly, many academics are highly specialised. Some are amazing curriculum designers, teachers, grant writers, researchers, report writers, administrators, team managers, stakeholder engagers (if that’s a real word) etc. So, among academics, it’s only natural that some will focus on impact more than others. </p>
<p><a href="https://doi.org/10.5465/AMBPP.2020.18546abstract">Academics create value in myriad ways</a>, and rarely do you find a “<a href="https://en.wikipedia.org/wiki/Purple_squirrel">purple squirrel</a>” – someone who excels at the full spectrum of work to be done. But, if impact is increasingly relevant to all academics, then a shift towards impact opens questions about performing as a team or individually. </p>
<p>There is also understandable <a href="https://doi.org/10.3152/095820211X13118583635693">scepticism and change resistance</a> to the “impact agenda” among academics. They are already time-poor and highly scrutinised. Any additional reporting and accountability requirements, such as the Australian Research Council’s <a href="https://www.arc.gov.au/engagement-and-impact-assessment">engagement and impact assessment</a>, feel like the last straw for academics, especially those who are busy chasing yesterday’s KPIs. </p>
<p>The research engagement and impact agenda needs to be worked through with great care. For a start, measurement of anything indelibly changes it, and new KPIs can introduce perverse responses and behaviours.</p>
<p>Focusing on engagement and impact also reinvigorates important value questions. There is always a risk that fundamental research is viewed as having no foreseeable impact. Yet it has given us so many unexpected and significant societal benefits.</p>
<p>A classic example is radio-astronomy research by <a href="https://theconversation.com/patently-australian-csiro-settles-suits-over-wi-fi-6184">CSIRO</a> and <a href="https://www.mq.edu.au/research/research-expertise/Research-innovation/where-wi-fi-began">Macquarie University</a> leading to wi-fi, which is an enabling technology for further innovations. Similarly, there was no guarantee of success at the start of decades of experimenting involved in innovations like HPV vaccines, cochlear implants and solar panels. Each innovation has directly and indirectly improved millions of lives.</p>
<p>Binary thinking about research versus impact, or applied versus fundamental research, is misguided, as societal benefits rely upon both sides of those coins. The tension of research versus teaching is similarly unproductive.</p>
<h2>Learning from researchers with impact</h2>
<p>We can look to outliers or “purple squirrels” to learn about research excellence with impact. <a href="https://www.natureindex.com/news-blog/coronavirus-vaccine-front-runner-moderna-puts-mit-chemist-entrepreneur-robert-langer-in-the-spotlight">Robert Langer</a> is <a href="https://www.sciencedirect.com/science/article/pii/S0166497218307302?via%3Dihub">an outstanding example</a>. One of his ventures, Moderna, is a <a href="https://theconversation.com/modernas-covid-vaccine-reports-95-efficacy-it-means-we-might-have-multiple-successful-vaccines-150266">leader in developing a COVID-19 vaccine</a>. </p>
<p>Langer’s lab at MIT has generated thousands of articles and patents, raising billions of dollars to spin out over 40 companies. This work includes treating multiple forms of cancer, endometriosis, eczema, vocal cord damage and more, and has <a href="https://news.mit.edu/2016/robert-langer-on-failure-resilience-and-making-an-impact-1031">affected the lives of billions</a>. His papers with industry collaborators are also <a href="https://www.natureindex.com/news-blog/corporate-collaboration-boosts-buzz-on-research">discussed more widely</a> than papers published by academics only.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/CAEllFMi_XA?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Research by Robert Langer and his colleagues is estimated to have affected the lives of 2 billion people.</span></figcaption>
</figure>
<p>Global outliers like Langer are certainly inspiring, but can feel inaccessible for the average researcher.</p>
<p>For our <a href="https://www.uts.edu.au/partners-and-community/initiatives/impact-studios/podcasts/impact-uts">Impact at UTS podcast series</a>, we spoke to highly acclaimed but more accessible researchers about how they navigate collaboration, engagement and impact. We cast a wide net. Their work spans a variety of disciplines and issues, including rebuilding reefs, Indigenous rights and self-determination, beach safety, solving crime through trace detection, access to clean water, autonomous vehicles, and more.</p>
<h2>What did these researchers tell us?</h2>
<p>These impact stories consistently reveal many inter-related insights, including:</p>
<ul>
<li><p>researchers’ desires to effect positive change align with the shift towards valuing benefits</p></li>
<li><p>researchers can be faithful to standards of academic rigour, ethics and independence while having material impact</p></li>
<li><p>complex problems <a href="https://doi.org/10.1057/s41599-019-0380-0">demand multi- or transdisciplinary approaches</a>, which often have engagement built in</p></li>
<li><p>engagement starts before a research project is formalised, and continues during and after it — gone are the days of throwing mono-disciplinary publications behind a paywall in the hope someone will discover it, make sense of the jargon and <a href="https://www.chronicle.com/article/isolated-scholars-making-bricks-not-shaping-policy/">bridge</a> the <a href="https://blogs.lse.ac.uk/impactofsocialsciences/2020/09/22/evidence-based-policy-and-other-myths-what-researchers-need-to-know-to-influence-government/">research-policy gap</a></p></li>
<li><p>engagement is based on shared values, which become shared language and shared understandings</p></li>
<li><p>formal agreements are important, but impactful collaboration is far from being transactional or contractual</p></li>
<li><p>it’s a team effort — there might be one chief investigator, but it’s often a team of researchers and several non-university stakeholders.</p></li>
</ul>
<h2>Fulfilling universities’ public purpose</h2>
<p>These insights reveal a more holistic and integrated picture of research engagement with communities, industry and government. By engaging with research end-users early, researchers get a real understanding of the problem. This helps inform their research, leading to greater impact and adoption.</p>
<p>The lessons learned should resonate with academics from any discipline or stage of career. They are also useful to non-academics as they select which academic or university to reach out to.</p>
<p>Despite the COVID-19 chaos, what endures is that universities are institutions with a public purpose. In Australia, publicly funded agencies employ a significant proportion of the research workforce. University research thus plays a critical role in addressing complex problems and national needs.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/7-6-billion-and-11-of-researchers-our-estimate-of-how-much-australian-university-research-stands-to-lose-by-2024-146672">$7.6 billion and 11% of researchers: our estimate of how much Australian university research stands to lose by 2024</a>
</strong>
</em>
</p>
<hr>
<p>A focus on the benefits that accrue from university research provides an opportunity for universities to enhance public trust and confidence in the value of their research. An engaged and supportive public may just be the most effective pathway towards creating the political will to adopt coherent, evidence-based policy. </p>
<p>For researchers, greater impact contributes to a virtuous research life cycle, including more sustainable funding. Last, but certainly not least, being able to draw on excellent research with impact in the classroom creates cutting-edge education and lifelong learning experiences in a way that more authentically includes the voices of the people impacted by the research.</p>
<iframe src="https://webplayer.whooshkaa.com/episode/743764?theme=light&enable-volume=true&iframe-height=190" height="190" width="100%" scrolling="no" frameborder="0" allow="autoplay"></iframe>
<p><em>Subscribe to the <a href="https://impactstudios.edu.au/podcasts/impact/">Impact at UTS podcast</a> on your favourite podcast app: <a href="https://podcasts.apple.com/au/podcast/impact-at-uts/id1533195116">Apple Podcasts</a>, <a href="https://open.spotify.com/show/1pfok2RN7epljwNiCHOAzJ?si=THFLonqGSC2B8ChCPLMOgg">Spotify</a>, <a href="https://www.stitcher.com/show/impact-at-uts">Stitcher</a>.</em></p>
<p><em><a href="https://impactstudios.edu.au/podcasts/impact/">Impact at UTS</a> was made by <a href="https://impactstudios.edu.au/">Impact Studios</a> at the University of Technology Sydney – an audio production house that combines academic research with audio storytelling for real-world impact.</em></p><img src="https://counter.theconversation.com/content/149817/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Having to do engagement and impact assessments may feel like the last straw for weary and time-poor academics. But thinking about these things can underpin research excellence.Martin Bliemel, Associate Dean of Research; Course Director, Diploma in Innovation, University of Technology SydneyJulian Zipparo, Executive Manager, Research Engagement, UTS; PhD candidate in Higher Education, UNSW SydneyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1416842020-07-08T18:52:50Z2020-07-08T18:52:50ZWhy the h-index is a bogus measure of academic impact<figure><img src="https://images.theconversation.com/files/345665/original/file-20200705-33922-1qdq6zc.jpg?ixlib=rb-1.1.0&rect=41%2C0%2C4625%2C3078&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">A portrait of Albert Einstein on a transformer station in St.Petersburg, Russia.</span> <span class="attribution"><span class="source">(Shutterstock)</span></span></figcaption></figure><p>Earlier this year, <a href="https://www.nytimes.com/2020/05/12/magazine/didier-raoult-hydroxychloroquine.html">French physician and microbiologist Didier Raoult generated a media uproar over his controversial promotion of hydroxychloroquine to treat COVID-19</a>. The researcher has long pointed to his growing list of publications and high number of citations as an indication of his contribution to science, all summarized in his “h-index.” </p>
<p>The controversy over his recent research presents an opportunity to examine the weaknesses of the h-index, a metric that aims to quantify a researcher’s productivity and impact, used by many organizations to evaluate researchers for promotions or research project funding. </p>
<p>Invented in 2005 by the American physicist Jorge Hirsch, the <a href="https://dx.doi.org/10.1073%2Fpnas.0507655102">Hirsch-index</a> or h-index, is an essential reference for many researchers and managers in the academic world. It is particularly promoted and used in the biomedical sciences, a field where the massive number of publications makes any serious qualitative assessment of researchers’ work almost impossible. This alleged indicator of quality has become a mirror in front of which researchers admire themselves or sneer at the pitiful h-index of their colleagues and rivals.</p>
<p>Although experts in bibliometry — a branch of library and information sciences that uses statistical methods to analyze publications — have quickly pointed out <a href="https://mitpress.mit.edu/books/bibliometrics-and-research-evaluation">the dubious nature of this composite indicator</a>, most researchers do not always seem to understand that its properties make it a far-from-valid index to seriously and ethically assess the quality or scientific impact of publications.</p>
<p>Promoters of the h-index commit an elementary error of logic. They assert that because Nobel Prize winners generally having a high h-index, the measure is a valid indicator of the individual quality of researchers. However, if a high h-index can indeed be associated with a Nobel Prize winner, this in no way proves that a low h-index is necessarily associated with a researcher of poor standing. </p>
<p>Indeed, a seemingly low h-index can hide a high scientific impact, at least if one accepts that the usual unit of measure for scientific visibility is reflected in the number of citations received.</p>
<h2>Limits of the h-index</h2>
<p>Defined as the number of articles <em>N</em> by an author that have each received at least <em>N</em> citations, the h-index is limited by the total number of published articles. For instance, if a person has 20 articles that are each cited 100 times, her h-index is 20 — just like a person who also has 20 articles, but each cited only 20 times. But no serious researcher would say that the two are equal because their h-index is the same.</p>
<p>The most ironic in the history of the h-index is that its inventor wanted to counter the claim that the number of published papers represented a researcher’s impact. So, he included the number of citations the articles received. </p>
<p>But it turns out that an author’s h-index is strongly correlated (up to about 0.9) with his total number of publications. In other words, it is the number of publications that drives the index more than the number of citations, an indicator which remains the best measure of the visibility of scientific publications.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=454&fit=crop&dpr=1 600w, https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=454&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=454&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=571&fit=crop&dpr=1 754w, https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=571&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/345699/original/file-20200706-29-ao7mpj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=571&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Raoult Didier made front-page news in France for promoting hydroxychloroquine as a remedy for COVID-19.</span>
<span class="attribution"><span class="source">(Shutterstock)</span></span>
</figcaption>
</figure>
<p>All of this is well known to experts in bibliometrics, but perhaps less to researchers, managers and journalists who allow themselves to be impressed by scientists parading their h-index. </p>
<h2>Raoult vs. Einstein</h2>
<p>In a recent investigation into Raoult’s research activities by the French newspaper <em>Médiapart</em>, a researcher who had been a member of the evaluation committee of Raoult’s laboratory said: “<a href="https://www.mediapart.fr/journal/france/070420/chloroquine-pourquoi-le-passe-de-didier-raoult-joue-contre-lui">What struck her was Didier Raoult’s obsession with his publications. A few minutes before the evaluation of his unit began, the first thing he showed her on his computer was his h-index</a>.” Raoult had also said in <em>Le Point</em> magazine in 2015 that “<a href="https://www.lepoint.fr/invites-du-point/didier_raoult/raoult-evaluer-la-recherche-mesurer-ou-tricher-04-10-2015-1970477_445.php">it was necessary to count the number and impact of researchers’ publications to assess the quality of their work</a>.” </p>
<p>So let’s take a look at Raoult’s h-index and see how it compares to, say, that of a researcher who is considered the greatest scientist of the last century: Albert Einstein.</p>
<p>In the <a href="https://www.webofknowledge.com/">Web of Science database</a>, Raoult has 2,053 articles published between 1979 and 2018, having received a total of 72,847 citations. His h-index calculated from these two numbers is 120. We know, however, that the value of this index can be artificially inflated through author self-citations — when an author cites his own previous papers. The database indicates that among the total citations attributed to the articles co-authored by Raoult, 18,145 come from articles of which he is a co-author. These self-citations amount to a total of 25 per cent. Subtracting these, Raoult’s h-index drops 13 per cent to a value of 104.</p>
<p>Now, let’s examine the case of Einstein, who has 147 articles listed in the Web of Science database between 1901 and 1955, the year of his death. For his 147 articles, Einstein has received 1,564 citations during his lifetime. Of this total number of citations, only 27, or a meagre 1.7 per cent, are self-citations. Now, if we add the citations made to his articles after his death, Einstein has received a total of 28,404 citations between 1901 and 2019, which earns him an h-index of 56.</p>
<p>If we have to rely on the so-called “objective” measurement provided by the h-index, we are then forced to conclude that the work of Raoult has twice the scientific impact of that of Einstein, the father of the photon, restricted and general relativities, the Bose-Einstein condensation and of the phenomenon of the stimulated emission at the origin of lasers. </p>
<p>Or maybe is it simpler (and better) to conclude, as already suggested, that this indicator is bogus?</p>
<p>One should note the significant difference in the number of total citations received by each of these researchers during their careers. They have obviously been active at very different times, and the size of scientific communities, and therefore the number of potential citing authors, have grown considerably over the past half-century. </p>
<p>Disciplinary differences and collaboration patterns must also be taken into account. For example, theoretical physics has far fewer contributors than microbiology, and the number of co-authors per article is smaller, which affects the measure of the productivity and impact of researchers.</p>
<p>Finally, it is important to note that the statement: “The h-index of person P is X,” has no meaning, because the value of the index depends on the content of the database used for its calculation. One should rather say: “The h-index of person P is X, in database Z.” Hence, according to the Web of Science database, which only contains journals considered to be serious and fairly visible in the scientific field, the h-index of Raoult is 120. On the other hand, in the free and therefore easily accessible database of Google Scholar, his h-index — the one most often repeated in the media — goes up to 179.</p>
<h2>Number fetishism</h2>
<p>Many scientific communities worship the h-index and this fetishism can have harmful consequences for scientific research. France, for instance, uses a <a href="https://www.sigaps.fr/">Système d’interrogation, de gestion et d’analyse des publications scientifiques</a> to grant research funds to its biomedical science laboratories. It is based on the number of articles they publish in so-called high impact factor journals. As reported by the newspaper <em>Le Parisien</em>, the frantic pace of Raoult’s publications <a href="https://www.leparisien.fr/societe/didier-raoult-une-frenesie-de-publications-et-des-pratiques-en-question-12-06-2020-8334405.php">allows his home institution to earn between 3,600 and 14,400 euros annually for each article published by his team</a>.</p>
<p>Common sense should teach us to be wary of simplistic and one-dimensional indicators. Slowing the maddening pace of scientific publications would certainly lead researchers to lose interest in the h-index. More importantly, abandoning it would contribute to producing scientific papers that will be fewer in number, but certainly more robust.</p>
<p><em>This is a corrected version of a story originally published on July 8, 2020. The earlier story said John Hirsch had invented the h-index instead of Jorge Hirsch.</em></p><img src="https://counter.theconversation.com/content/141684/count.gif" alt="La Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Yves Gingras a reçu des financements du CRSH et du FQRSC. </span></em></p><p class="fine-print"><em><span>Mahdi Khelfaoui a reçu des financements du CRSH</span></em></p>The h-index has become an indicator of quality for many researchers and may influence the allocation of research funds. But some question its value.Yves Gingras, Professeur d’histoire et de sociologie des sciences, Université du Québec à Montréal (UQAM)Mahdi Khelfaoui, Professeur associé, Université du Québec à Montréal (UQAM)Licensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1363902020-04-21T04:49:44Z2020-04-21T04:49:44ZAntarctic endeavours, primary health-care research and dark matter exploration – the coronavirus casualties you haven’t heard of<figure><img src="https://images.theconversation.com/files/329327/original/file-20200421-126488-1sx9hsq.jpg?ixlib=rb-1.1.0&rect=100%2C22%2C7348%2C4947&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>The year 2020 came with big expectations for researchers, myself included. Last year I was successful in the first round of the <a href="https://www.nhmrc.gov.au/funding/data-research/outcomes-funding-rounds#download">National Health and Medical Research Council Investigator Grants scheme</a>. Six years since completing my PhD, I managed to launch my <a href="https://healthyprimarycare.com">Healthy Primary Care</a> research team. </p>
<p>We investigate how principles of wellness such as healthy eating and exercise are incorporated into health care, particularly in general practice. I spent the summer planning how to support my team for the next five years, focusing on impact and research translation into real-world settings. </p>
<p>Big things were in the works. It was an exciting time. But as it turns out, wellness in health care isn’t a priority during the COVID-19 crisis. </p>
<p>As the pandemic lingers, big players (<a href="https://www.fiercebiotech.com/biotech/big-pharma-companies-join-forces-for-fightback-against-covid-19">especially pharmaceutical companies</a>) around the world have understandably dropped everything, <a href="https://www.gisaid.org/">joining forces to give the virus their undivided attention</a>.</p>
<h2>A sudden loss</h2>
<p>Many of my team’s projects relied on doctors, nurses and other health professionals to collect or provide data. With the strain placed on health care by the pandemic, continuing was no longer viable. <a href="https://www.nhmrc.gov.au/funding/find-funding/synergy-grants">Grant applications</a>, domestic and international travel, conferences and meetings have all been cancelled or postponed indefinitely. </p>
<p>As a supervisor, the hardest part was withdrawing research students and interns I’d lined up to start projects in clinics. This pandemic has challenged the relevance, impact and productivity of our work. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/for-most-universities-theres-little-point-to-the-governments-covid-19-assistance-package-136244">For most universities, there's little point to the government's COVID-19 assistance package</a>
</strong>
</em>
</p>
<hr>
<p>This shock comes shortly after a summer of devastating bushfires which hindered research progress by forcing experts out of fire-affected regions, destroying expanses of equipment and <a href="https://www.nature.com/articles/d41586-020-00130-4">reportedly setting some studies “back months or years”</a>.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=450&fit=crop&dpr=1 600w, https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=450&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=450&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=566&fit=crop&dpr=1 754w, https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=566&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/329341/original/file-20200421-126515-2sr1f6.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=566&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">This photo was taken in Junee, New South Wales, in January. According to reports, the total tangible cost estimate of the summer bushfires was close to A$100 billion.</span>
<span class="attribution"><span class="source">Shutterstock</span></span>
</figcaption>
</figure>
<h2>Stoppages across the field</h2>
<p>Social distancing, travel bans and quarantine restrictions mean scientific fieldwork across the world has almost completely stopped. </p>
<p>The Australian Antarctic Program, led by the federal Department of Agriculture, Water and the Environment has been <a href="http://www.antarctica.gov.au/news/2020/impacts-of-covid-19-on-the-australian-antarctic-program">reduced to essential staff only</a> to keep the Antarctic continent COVID-19-free. Instead of sending 500 expeditioners in the next summer season, the Australian Antarctic Division will only send about 150. </p>
<p>Social distancing measures are also preventing climate scientists from <a href="https://www.theguardian.com/science/2020/apr/03/climate-monitoring-research-coronavirus-scientists">being able to visit their laboratories</a>. If the pandemic continues, this could hamper important weather and climate surveillance practices. In some cases, labs have been reduced to one essential worker <a href="https://www.sciencemag.org/news/2020/03/updated-labs-go-quiet-researchers-brace-long-term-coronavirus-disruptions">whose sole job is to keep laboratory animals alive</a> for when research resumes. </p>
<p>Delays have also impacted one of the world’s largest efforts to investigate the nature of dark matter. The <a href="https://science.purdue.edu/xenon1t/?cat=3">XENON experiment</a> based in Italy is worth more than US$30 million, <a href="https://www.nytimes.com/2020/04/07/science/dark-matter-elena-aprile-coronavirus.html">according to the New York Times</a>. It faced a multitude of roadblocks when the country was forced into lockdown earlier this year. </p>
<h2>Young research stars missing opportunities</h2>
<p>For young researchers, social distancing and event cancellations are especially damaging to professional development. Scientific conferences and meetings foster collaboration and can also lead to <a href="https://www.scientificamerican.com/article/how-the-coronavirus-is-hampering-science/">employment opportunities</a>.</p>
<p>Although funding cancellations and <a href="https://www.nhmrc.gov.au/about-us/news-centre/update-changes-nhmrc-2020-funding-schemes">grant scheme delays</a> mostly impact established researchers, other schemes supporting early career and postdoctoral researchers have also been postponed, such as the <a href="https://research.flinders.edu.au/rp/Blog/14981-A/new-opportunities-rebecca-l-cooper-medical-research-foundation-project-grant-round-delayed">Rebecca L Cooper Medical Research scheme</a> and the <a href="https://www.griffith.edu.au/research/research-services/research-grants/funding-opportunities/internal-grants">Griffith University Postdoctoral Fellowship</a> scheme. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/coronavirus-social-distancing-is-delaying-vital-scientific-research-133689">Coronavirus: Social distancing is delaying vital scientific research</a>
</strong>
</em>
</p>
<hr>
<p>This crisis has left the next generation of researchers unsupported, and have negative flow-on effects for all research areas. In health and disease prevention, research efforts apart from vaccinations are still vital, as the onset of COVID-19 hasn’t stopped the rise of <a href="https://www.ncbi.nlm.nih.gov/pubmed/23245609">chronic disease</a>.</p>
<h2>There are positives</h2>
<p>Australia boasts a robust and passionate research workforce, which means we can divert resources to a united cause such as the coronavirus crisis. As the race for a vaccine continues, the value of research has never been more apparent to the non-scientific community. This may help <a href="https://www.timeshighereducation.com/news/how-could-coronavirus-crisis-change-research-spending">weaken anti-science messages</a>.</p>
<p>The pandemic is also providing opportunity for future university leaders to understand university management, funding and governance decisions. Never before has information been so accessible on where <a href="https://theconversation.com/australian-unis-may-need-to-cut-staff-and-research-if-government-extends-coronavirus-travel-ban-132175">funding comes from</a>. </p>
<p>Online conferencing and collaboration related to research has also made participation more accessible and affordable. This increases inclusively by removing barriers for people who may not be able to attend in-person gatherings, such as people <a href="https://www.weforum.org/agenda/2020/04/covid-19-isolation-disabilities/">living with a physical disability</a>, full-time carers and people experiencing financial hardship. Less domestic and international travel is also helping <a href="https://www.nature.com/articles/d41586-020-00786-y">reduce carbon footprints</a>. </p>
<h2>Charging forward</h2>
<p>The health system isn’t working normally, which means my team’s research isn’t working normally. Nonetheless, we’re pivoting well in this uncertain time. We’re helping plan the <a href="https://aaapc.org.au/event-3649286">first online conference for Australian primary care</a> to improve access to relevant research across the country. </p>
<p>New grant opportunities are aligning COVID-19 to our research focus, such as the Royal Australian College of General Practitioners’s and the Hospitals Contribution Fund’s <a href="https://foundation.racgp.org.au/grants">special call for projects on COVID-19 in general practice</a>. </p>
<p>Some may think non-COVID-19 research isn’t currently necessary, but it will be once we combat this disease. And when that happens, we’ll be ready to continue right where we left off.</p><img src="https://counter.theconversation.com/content/136390/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Lauren Ball works for Griffith University. She receives external funding from the National Health and Medical Research Council. She is on the Executive Committee of the Australasian Association of Academic Primary Care and is a member of the Dietitians Association of Australia.</span></em></p>With a threatening virus sweeping the world, research efforts across sectors have ground to a halt. But one thing is clear: the non-scientific community has never valued research more.Lauren Ball, Associate Professor/ Principal Research Fellow, Griffith UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/942192018-05-07T10:39:12Z2018-05-07T10:39:12ZRedefining ‘impact’ so research can help real people right away, even before becoming a journal article<figure><img src="https://images.theconversation.com/files/217524/original/file-20180503-182160-1osxwoj.jpg?ixlib=rb-1.1.0&rect=128%2C85%2C2394%2C1637&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Park guards view maps and photos of high-altitude glaciers -- information that can be shared with local communities dealing with changing water levels. </span> <span class="attribution"><span class="source">Anne Toomey</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span></figcaption></figure><p>Scientists are increasingly expected to produce research with impact that goes beyond the confines of academia. When funding organizations such as the National Science Foundation consider grants to researchers, they ask about “broader impacts.” They want to support science that directly contributes to the “<a href="https://www.nsf.gov/od/oia/special/broaderimpacts/">achievement of specific, desired societal outcomes</a>.” It’s not enough for researchers to call it a day, after they publish their results in journal articles read by a handful of colleagues and few, if any, people outside the ivory tower.</p>
<p>Perhaps nowhere is impact of greater importance than in <a href="https://onlinelibrary.wiley.com/doi/full/10.1111/j.1523-1739.2006.00434.x">my own fields of ecology and conservation science</a>. Researchers often conduct this work with the explicit goal of contributing to the restoration and long-term survival of the species or ecosystem in question. For instance, research on an endangered plant can help to address the threats facing it. </p>
<p>But scientific impact is a very tricky concept. Science is a process of inquiry; it’s often impossible to know what the outcomes will be at the start. Researchers are asked to imagine potential impacts of their work. And people who live and work in the places where the research is conducted may have different ideas about what impact means. </p>
<p>In collaboration with several Bolivian colleagues, <a href="https://doi.org/10.1007/s13280-018-1056-5">I studied perceptions of research and its impact</a> in a highly biodiverse area in the Bolivian Amazon. We found that researchers – both foreign-based and Bolivian – and people living and working in the area had different hopes and expectations about what ecological research could help them accomplish.</p>
<h2>Surveying the researchers</h2>
<p>My colleagues and I focused on research conducted in Bolivia’s Madidi National Park and Natural Area for Integrated Management.</p>
<p><iframe id="tc-infographic-261" class="tc-infographic" height="400px" src="https://cdn.theconversation.com/infographics/261/0788cc1758c2628bcc75755907ba82cbf7e1e414/site/index.html" width="100%" style="border: none" frameborder="0"></iframe></p>
<p>Due to its impressive size (approximately 19,000 square kilometers) and <a href="https://www.redorbit.com/news/science/1112693995/wcs-biodiversity-madidi-national-park-091412/">diversity of species</a> – including endangered mammals such as the spectacled bear and the giant otter – Madidi attracts large numbers of ecologists and conservation scientists from around the world. The park is also notable for its cultural diversity. Four indigenous territories overlap Madidi, and there are 31 communities located within its boundaries. </p>
<p>Between 2012 and 2015, we carried out interviews and workshops with people living and working in the region, including park guards, indigenous community members and other researchers. We also surveyed scientists who had worked in the area during the previous 10 years. Our goal was to better understand whether they considered their research to have implications for conservation and ecological management, and how and with whom they shared the results of their work.</p>
<p>Eighty-three percent of researchers queried told us their work had implications for management at community, regional and national levels rather than at the international level. For example, knowing the approximate populations of local primate species can be important for communities who rely on the animals for food and ecotourism. </p>
<p><iframe id="VW2iO" class="tc-infographic-datawrapper" src="https://datawrapper.dwcdn.net/VW2iO/4/" height="400px" width="100%" style="border: none" frameborder="0"></iframe></p>
<p>But the scale of relevance didn’t necessarily dictate how researchers actually disseminated the results of their work. Rather, we found that the strongest predictor of how and with whom a researcher shared their work was whether they were based at a foreign or national institution. Foreign-based researchers had extremely low levels of local, regional or even national dissemination. However, they were more likely than national researchers to publish their findings in the international literature.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=456&fit=crop&dpr=1 600w, https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=456&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=456&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=574&fit=crop&dpr=1 754w, https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=574&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/217525/original/file-20180503-153881-f8tblv.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=574&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Celín Quenevo and other leaders of the Takana indigenous nation raised money in the 1990s to translate a 1950s book written about the Takana people by a German anthropologist into Spanish.</span>
<span class="attribution"><span class="source">Anne Toomey</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<h2>Ongoing scientific colonialism?</h2>
<p>This disparity raises concerns about whether foreign-led research in tropical nations such as Bolivia is perpetuating colonial-era legacies of <a href="https://www.smithsonianmag.com/science-nature/science-bears-fingerprints-colonialism-180968709/">scientific extractivism</a>. </p>
<p>Along with its South American neighbors, <a href="http://www.alternautas.net/blog/2017/4/1/part-2-histories-of-biological-science-in-bolivia">Bolivia was subject to centuries of European explorations</a>, during which collectors gathered interesting specimens of flora and fauna to ship back to the country financing the expedition. As late as the 1990s, <a href="http://www.scielo.org.ar/scielo.php?script=sci_arttext&pid=S0327-93832005000200001">more than 90 percent of 37,000 zoological specimens</a> from Bolivia were in collections beyond its borders. The expatriation of biological samples has become increasingly restricted under a <a href="https://libya360.wordpress.com/2015/03/16/bolivias-decolonization-mission/">national political climate of “decolonization.”</a></p>
<p>But many locals in the Madidi region still expressed to us perceptions that “research is only for the researcher” and “researchers leave nothing behind.” In interviews and workshops, they lamented opportunities missed because they didn’t know about the results of research conducted on their lands. For example, when the park staff learned about previous research done on mercury levels in the Tuichi river that runs through the park, they talked about the importance of sharing this information with local communities for whom fish is a main sources of protein. </p>
<p>Our results suggest that foreign researchers should be wary of a modern form of scientific colonialism – conducting fieldwork in a far-off land and then taking their data and knowledge home with them.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=900&fit=crop&dpr=1 600w, https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=900&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=900&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1131&fit=crop&dpr=1 754w, https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1131&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/217529/original/file-20180503-153888-1f0zsrd.JPG?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1131&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">One solution: Colorful banners with information in the local language about past ecological research conducted in Madidi are displayed at the park offices.</span>
<span class="attribution"><span class="source">Anne Toomey</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>Our study also revealed that in some cases, the question of whether or not research had been disseminated was a matter of perspective. Park offices, indigenous council headquarters and government institutions all held dusty libraries full of articles and books that were in many cases the final products of scientific studies. But very few people had actually read these reports, in part because many were written in English. Also, people in the Madidi region are more accustomed to obtaining knowledge orally rather than through written texts. So finding new ways to communicate across cultural and language barriers is key.</p>
<h2>Collaboration beyond publication</h2>
<p>Perhaps one way forward is to <a href="https://doi.org/10.1111/conl.12315">think differently about what is meant by impact</a> and when it takes place. Although it’s typically understood to occur after the results have been written up, our research found that the most meaningful forms of impact often took place prior to that.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=450&fit=crop&dpr=1 600w, https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=450&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=450&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=566&fit=crop&dpr=1 754w, https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=566&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/217538/original/file-20180503-153881-oiwcki.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=566&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Research often depends on collaboration across groups. Here, Madidi National Park guards and Bolivian scientists work together in the protected area.</span>
<span class="attribution"><span class="source">Marcos Uzquiano</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>In ecological and conservation science research, locals are hired as guides or porters, and researchers often stay for days or weeks in communities while they are collecting data. This fieldwork period is filled with potential for knowledge exchange, where both parties can learn from one another. Indigenous communities in the Madidi region are directly dependent on local biodiversity. Not only does it provide food and other resources, but it’s vital for the continuation of their cultures. They possess unique knowledge about the place, and they have a vested interest in ensuring that the local biodiversity will continue to exist for many generations to come. </p>
<p>Rather than impact being addressed at the end of research, societal impacts can be part of the first stages of a study. For example, people living in the region where data is to be collected might have insight into the research questions being investigated; scientists need to build in time and plan ways to ask them. Ecological fieldwork <a href="https://www.ecologyandsociety.org/vol21/iss2/art28/">presents many opportunities</a> for knowledge exchange, new ideas and even friendships between different groups. Researchers can take steps to engage more directly with community life, such as by taking a few hours to teach local school kids about their research. </p>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=630&fit=crop&dpr=1 600w, https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=630&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=630&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=792&fit=crop&dpr=1 754w, https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=792&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/217539/original/file-20180503-182160-pw9ipm.JPG?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=792&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">The author worked with two indigenous communities to develop ideas for how local leaders could negotiate future relationships with researchers.</span>
<span class="attribution"><span class="source">Anne Toomey</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<p>Of course, such activities do not make disseminating the results of research at multiple levels less important. But engaging additional stakeholders earlier in the process could make for a more interested audience when findings are available. </p>
<p>Whether studying hive decline with beekeepers in the United Kingdom or evaluating human-elephant conflicts in India, those affected have the right to know about the results of research. If “broader impacts” are to become more than an afterthought in the research process, non-academics need a bigger voice in the process of determining what those impacts may be.</p><img src="https://counter.theconversation.com/content/94219/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Anne Toomey received partial funding for this research from Lancaster University and additional financial support from the Rufford Foundation and the Royal Geographical Society.</span></em></p>Science can’t just stay in the ivory tower. But what does impact really mean and how does it happen? A study of more than a decade of ecological fieldwork projects in Bolivia suggests a better way.Anne Toomey, Assistant Professor of Environmental Studies and Science, Pace University Licensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/885812017-12-06T13:35:21Z2017-12-06T13:35:21ZEngineering research in Africa is growing but it’s still a patchy picture<figure><img src="https://images.theconversation.com/files/197773/original/file-20171205-22967-swwkhw.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Engineering can greatly bolster any country's development and growth.</span> <span class="attribution"><span class="source">Gorodenkoff/Shutterstock</span></span></figcaption></figure><p>Africa’s vast land mass and rich natural and mineral resources make it strategically important and an increasingly significant <a href="https://na.unep.net/atlas/africa/downloads/chapters/Africa_Atlas_English_Intro.pdf">global player</a>. It is also a dynamic young continent: about 60% of its residents are aged <a href="http://www.bbc.com/news/world-africa-34188248">below 25</a>.</p>
<p>The African Union is trying to harness this enormous potential through its <a href="https://au.int/en/agenda2063">Agenda 2063</a>, which includes elevating Africa through improved education and application of science and technology in development. </p>
<p>Engineering is an important branch of science and technology. It has a significant impact on the overall development of any nation, region or continent. It is, as Professor Calestous Juma <a href="http://www.nation.co.ke/oped/opinion/Engineering-is-the-engine-that-will-power-Africa-s-growth-/440808-2309528-pq151w/index.html">has written</a>, an engine to power growth – especially in Africa.</p>
<p>The World Bank predicts that Africa needs to spend about <a href="http://www.nation.co.ke/oped/opinion/Engineering-is-the-engine-that-will-power-Africa-s-growth-/440808-2309528-pq151w/index.html">USD$93 billion per year</a> in the coming years to improve its infrastructure. Part of this investment must be in world class engineering education and research.</p>
<p>Given the discipline’s importance, I wanted to understand how Africa is performing in terms of engineering research. How much are the continent’s researchers contributing to new ideas and thinking around engineering? To find out, I <a href="http://www.tandfonline.com/doi/full/10.1080/20421338.2017.1341732">searched, downloaded and analysed</a> scholarly publication data from academic publisher Elsevier’s citation and abstracting service, <a href="https://www.scopus.com/">Scopus</a>®. It’s a huge index of articles, covering 22,800 journals belonging to more than 5,000 international publishers across disciplines. </p>
<p>I also examined how many times articles from Africa were being cited, which is crucial to map the relevance and impact of any research. For instance, one of the criteria for winning a <a href="http://www.lindau-nobel.org/blog-on-fundamental-science/">Nobel Prize</a> in science is how frequently a researcher’s work has been cited.</p>
<p>The data I analysed shows that scholarly research output in terms of journal articles, conference papers and so on in engineering fields from Africa has increased over the past two decades. The number remains small in comparison to other, more developed continents and countries. But the continent’s contribution to global thinking and understanding about engineering is growing, and this should be celebrated.</p>
<h2>Analysing data</h2>
<p>My analysis reveals that Africa has recorded a tremendous growth in its output of academic engineering research over the past 20 years. In total, 75,157 scholarly articles about engineering subjects emerged from Africa between 1996 and 2016. About 1,500 of these were published in the first seven years under review. In the past three years, about 9,000 engineering articles from Africa were published annually. That’s a significant percentage increase.</p>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=397&fit=crop&dpr=1 600w, https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=397&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=397&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=498&fit=crop&dpr=1 754w, https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=498&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/197855/original/file-20171205-23009-1tqh4ia.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=498&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Africa’s engineering research output over 20 years.</span>
<span class="attribution"><a class="source" href="http://www.scimagojr.com/countryrank.php">Scimago</a></span>
</figcaption>
</figure>
<p>The problem is that African countries’ outputs are not uniform. South Africa leads the pack, with 22,156 articles over 20 years. This puts it at 41st in the world for output in engineering research. It is followed by Algeria (16,617 articles) and Tunisia (14,805 articles). Some countries have barely contributed to engineering research: Cape Verde produced only nine articles in 20 years; the Central African Republic just seven and Somalia only six.</p>
<p>The continent is also not producing nearly as much engineering research as others and other regions.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=480&fit=crop&dpr=1 600w, https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=480&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=480&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=603&fit=crop&dpr=1 754w, https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=603&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/197935/original/file-20171206-910-pqurdu.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=603&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Africa’s engineering research output is still lower than other continents and regions, but its growth over 20 years has been encouraging.</span>
<span class="attribution"><span class="source">Scimago</span></span>
</figcaption>
</figure>
<p>I also wanted to know how often African researchers’ work was being cited by others. This is a good way to understand the impact a piece of research has, and is called citation analysis. It counts the number of times an author’s article is cited in other scholarly works. And <a href="https://medium.com/@write4research/why-are-citations-important-in-research-writing-97fb6d854b47">citations are important</a> because they reveal that a piece of research is being used by others.</p>
<p><a href="http://www.tandfonline.com/doi/full/10.1080/20421338.2017.1341732">The results</a> are encouraging. The average citation for academic engineering papers from Africa is 5.48 per paper. This is almost equal to the performance of papers from Asia, and is well above the average citations received by papers from Eastern Europe. This suggests that African engineering research is influencing others’ thinking and contributing to global knowledge about the discipline.</p>
<p>So how can Africa improve its engineering research output, especially with an eye to meeting the goals of Agenda 2063? Collaboration will be crucial.</p>
<h2>Collaborative work</h2>
<p>South Africa does well with collaboration. Articles from the country tend to involve more than one research organisation or institution. Co-authored articles are common. Its researchers work with others on the continent and with global partners. Countries in North Africa, however, are less active when it comes to collaboration. </p>
<p>Africa-Africa collaboration, involving institutions and individuals across the continent, needs to be strengthened. This is because only African countries can truly understand the continent’s pressing needs, and develop appropriate solutions. Countries like South Africa that perform well collaboratively can offer support and advice to others. </p>
<p>It may also be time to set up an exclusively African citation database. Even Scopus®, the world’s largest indexing and abstracting database, offers very limited coverage of African science. By developing a resource that focuses only on African engineering research, the continent will be able to get a more complete, clear picture of its output and respond accordingly. The Council for the Development of Social Science Research in Africa is creating an <a href="http://africancitationindex.org/">African Citation database</a>, but it will be some time before this is a fully fledged searchable database.</p><img src="https://counter.theconversation.com/content/88581/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Swapan Kumar Patra receives funding from National Research Foundation, Republic of South Africa, Post doctoral research fellowship, through Tshwane University of Technology</span></em></p>Africa has recorded a tremendous growth in its output of academic engineering research over the past 20 years. Greater collaboration can increase this growth even more.Swapan Kumar Patra, Tshwane University of TechnologyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/872522017-11-28T03:53:16Z2017-11-28T03:53:16ZStarting next year, universities have to prove their research has real-world impact<figure><img src="https://images.theconversation.com/files/194918/original/file-20171116-19768-8scze5.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">For research to have an impact, it needs to be used or applied in some way.</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>Starting in 2018, Australian universities will be required to prove their research provides concrete benefits for taxpayers and the government, who fund it.</p>
<p>Education Minister Simon Birmingham recently <a href="https://ministers.education.gov.au/birmingham/focusing-research-make-difference">announced</a> the Australian Research Council (<a href="http://www.arc.gov.au/">ARC</a>) will introduce an <a href="http://www.arc.gov.au/engagement-and-impact-assessment">Engagement and Impact Assessment</a>. It will run alongside the current Excellence in Research Australia <a href="http://www.arc.gov.au/excellence-research-australia">ERA</a> assessment exercise. This follows a <a href="http://www.arc.gov.au/ei-pilot-overview">pilot</a> of the Engagement and Impact Assessment, run in 2017.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/pilot-study-on-why-academics-should-engage-with-others-in-the-community-76707">Pilot study on why academics should engage with others in the community</a>
</strong>
</em>
</p>
<hr>
<p>Until now, research performance assessment has mostly been focused on the number of publications, citations and competitive grants won. This new metric changes the focus from inputs and outputs to outcomes. This is part of a continuing shift from quantity to quality, which began in <a href="http://www.ams.org/journals/notices/201103/rtx110300434p.pdf">earlier iterations</a> of the ERA. The Engagement and Impact assessment reflects a significant change in thinking about the types of research impact we value and why. </p>
<p>For research to have an impact, it needs to be used or applied in some way. For example, health research aims to have an impact on health outcomes. For that to happen doctors, nurses and people working in health policy would need to use that research evidence in their practice or policy decision-making. </p>
<p>Despite the <a href="https://www.education.gov.au/review-research-policy-and-funding-arrangements-0">initial focus on commercial outcomes</a>, the Engagement and Impact Assessment has evolved to include a range of impact types. It provides an important incentive for researchers in all fields to think about how to engage those outside of academia who can translate their research into real-world impacts. It also enables researchers who were already engaging with research end-users and delivering positive impact to have these outcomes formally recognised for the first time at a national level. </p>
<h2>Community input</h2>
<p>Including an engagement component recognises researchers are not in direct control of whether their research will actually be used. Industry, government and the community also have an important role in making sure the potential benefits of research are achieved. </p>
<p>The engagement metrics allow universities to demonstrate and be rewarded for engaging industry, government and others in research, even if it doesn’t directly or immediately lead to impact. Case studies were chosen to demonstrate impact because they let researchers describe the important impacts they are achieving that metrics can’t capture.</p>
<p>The case studies will need to include the impact achieved, the beneficiaries and timeframe of the research impact and countries where the impact occurred. They’ll also include what strategies were employed to enable translation of research into real world benefits.</p>
<p>The results will be assessed by a panel of experts for each field of research who will provide a rating of engagement and impact as low, medium or high.</p>
<h2>Cultural impacts</h2>
<p>The ARC has defined engagement as: </p>
<blockquote>
<p>the interaction between researchers and research end-users outside of academia, for the mutually beneficial transfer of knowledge, technologies, methods or resources. </p>
</blockquote>
<p>Impact has been defined as: </p>
<blockquote>
<p>the contribution that research makes to economy, society and environment and culture beyond the contribution to academic research.</p>
</blockquote>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/when-measuring-research-we-must-remember-that-engagement-and-impact-are-not-the-same-thing-56745">When measuring research, we must remember that 'engagement' and 'impact' are not the same thing</a>
</strong>
</em>
</p>
<hr>
<p>The definition of impact has been amended to include “culture”, which was not part of the definition applied in the pilot. This amendment speaks to concerns raised by the academic community around quantifying and qualifying impacts that vary significantly across different academic fields. It’s hard to compare, for example, the impact of an historic exhibition to the impact of astrophysics research on gravitational waves.</p>
<p>It’s also difficult to compare more basic or experimental research with applied research, such as health and well-being programs that can be directly applied in the community. Basic or experimental research can take a long time to lead to a measurable impact.</p>
<p>Classic examples of experimental research that had significant economic, health and social impacts that it didn’t specifically set out to achieve are the discovery of <a href="https://www.acs.org/content/acs/en/education/whatischemistry/landmarks/flemingpenicillin.html#alexander-fleming-penicillin">penicillin</a>, and <a href="http://www.sbs.com.au/news/explainer/wifi-australian-invention-helping-world-connect">WiFi</a>.</p>
<h2>An addition, not a replacement</h2>
<p>The <a href="http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/">traditional research metrics</a> of grants, publication and citation, which work for basic, experimental and longer-time-to-impact research, are still in play. The Engagement and Impact Assessment has <a href="https://www.timeshighereducation.com/news/australian-universities-unimpressed-impact-assessment-plans">not been tied to funding decisions</a> at this stage.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/explainer-how-and-why-is-research-assessed-36895">Explainer: how and why is research assessed?</a>
</strong>
</em>
</p>
<hr>
<p>A <a href="https://search-proquest-com.ezp01.library.qut.edu.au/docview/1850750875?pq-origsite=summon">study</a> of the impact case studies submitted to the UK’s <a href="http://www.ref.ac.uk/">Research Excellence Framework</a> found high-impact scores were correlated to high quality scores. They concluded “impact was not being achieved at the expense of research excellence”. <a href="https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-14-2">Previous research</a> has shown research quality is an important enabler of the use of research.</p>
<p>Engagement and impact outcomes for a specific field of research at one university will be assessed against the same field at another university. This is also the case with traditional metrics and grants assessment.</p>
<p>Engagement will be assessed on four key metrics and an engagement narrative. These metrics are focused on funding provided by end-users of research such as businesses or individuals outside the world of academia who directly use or benefit from the the research.</p>
<p>The four metrics are: cash support (against <a href="https://www.education.gov.au/higher-education-research-data-collection">Higher Education Research Data Collection</a> categories) or sponsored grants from end-users, research commercialisation income and how much income is made per researcher.</p>
<p>The engagement narrative will enable universities to provide detail about how they are engaging with end-users. There is also a <a href="http://www.arc.gov.au/sites/default/files/filedepot/Public/EI/Engagement_and_Impact_Assessment_Pilot_2017_Report.pdf">list of other engagement indicators</a> universities can draw on to describe their engagement activity.</p>
<p>At times, the value of research <a href="http://www.dailytelegraph.com.au/news/nsw/taxpayer-dollars-wasted-on-absurd-studies-that-do-nothing-to-advance-australian-research/news-story/c0c20e651da84b3f249f6e77405cfc7c">has been publicly questioned</a>. The Engagement and Impact Assessment will help the general public better understand the value of the research they fund.</p><img src="https://counter.theconversation.com/content/87252/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Pauline Zardo does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Engagement and impact will be part of research performance assessment starting in 2018, signalling a shift in what kind of research we value and why.Pauline Zardo, Data & Policy Research Fellow, Queensland University of TechnologyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/428242017-09-08T04:59:45Z2017-09-08T04:59:45ZResearch does solve real-world problems: experts must work together to make it happen<figure><img src="https://images.theconversation.com/files/185015/original/file-20170907-8344-1286buo.JPG?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Working within and across disciplines allows blue sky research to deliver real world impact. </span> <span class="attribution"><span class="source">Arry Tanusondjaja</span>, <span class="license">Author provided</span></span></figcaption></figure><p>Generating knowledge is one of the most exciting aspects of being human. The inventiveness required to apply this knowledge to solve practical problems is perhaps our most distinctive attribute.</p>
<p>But right now we have before us some hairy challenges – whether that be figuring our how to save our <a href="https://theconversation.com/the-worlds-coral-reefs-are-in-trouble-but-dont-give-up-on-them-yet-78588">coral reefs from warmer water</a>, landing a <a href="https://theconversation.com/the-new-space-race-why-we-need-a-human-mission-to-mars-73757">human on Mars</a>, eliminating the <a href="https://theconversation.com/closing-the-gap-is-failing-and-needs-a-radical-overhaul-72961">gap in life expectancy</a> between the “haves” and “have-nots” or delivering reliable <a href="https://theconversation.com/pace-of-renewable-energy-shift-leaves-city-planners-struggling-to-keep-up-82206">carbon-free energy</a>.</p>
<p>It’s commonly said that an interdisciplinary approach is vital if we are to tackle such real world challenges. But what does this really mean? </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/it-takes-a-community-to-raise-a-startup-65324">It takes a community to raise a startup</a>
</strong>
</em>
</p>
<hr>
<p>Listen and read with care and you’ll start to notice that the words crossdisciplinary, multidisciplinary, interdisciplinary and transdisciplinary are used interchangeably. These words describe distinctly different ways of harnessing the power of disciplinary expertise to chart a course into the unknown. </p>
<p>In navigation, the tools and methods matter – choose differently and you’ll end up in a different spot. How we go about creating knowledge and solving problems really matters – it changes not only what questions can be asked and answered but fundamentally shapes what’s possible.</p>
<h2>What is a discipline?</h2>
<p>For centuries we have organised research within disciplines, and this has delivered extraordinary depths of knowledge. </p>
<p>But what is a discipline? It’s a shared language, an environment in which there’s no need to explain the motivation for one’s work, and in which people have a shared sense of what’s valuable. </p>
<p>For example, my background discipline is optical physics. I know what it’s like to be able to skip down the corridor and say, </p>
<blockquote>
<p>“I’ve figured out how we can get broadband flat dispersion - we just need to tailor the radial profile!” </p>
</blockquote>
<p>…and have people instantly not just know what I mean, but be able to add their own ideas and drive the work forward.</p>
<p>In long-established disciplines it’s often necessary to focus in a narrow area to be able to extend the limits of knowledge within the time-frame of a PhD. And while it’s rarely obvious at the time what benefits will flow from digging a little deeper, our way of life has been transformed over and over as result.</p>
<p>Disciplines focus talent and so can be amazingly efficient ways of generating knowledge. But they can also be extraordinarily difficult to penetrate from the outside without understanding that discipline’s particular language and shared values. </p>
<p>The current emphasis on real-world impact has sharpened awareness on the need to translate knowledge into outcomes. It has also brought attention to the critical role partnerships with industry and other end-users of research play in this process. </p>
<h2>Creating impact across disciplines</h2>
<p>Try to solve a problem with the tools of a single discipline alone, and it’s as if you have a hammer - everything starts to look like a nail. It’s usually obvious when expertise from more than one discipline is needed.</p>
<p>Consider a panel of experts drawn from different fields to each apply the tools of their field to a problem that’s been externally framed. This has traditionally been how expertise from the social sciences is brought to bear on challenges in public health or the environment.</p>
<p>This is a <em>crossdisciplinary approach</em>, which can produce powerful outcomes provided that those who posed the question are positioned to make decisions based on the knowledge generated. But the research fields themselves are rarely influenced by this glancing encounter with different approaches to knowledge generation.</p>
<p><em>Multidisciplinary research</em> involves the application of tools from one discipline to questions from other fields. An example is the application of crystallography, discovered by the Braggs, to <a href="https://www.umass.edu/microbio/rasmol/1st_xtls.htm">unravel the structure of proteins</a>. This requires concepts to transfer across domains, sometimes in real time but usually with a lag of years or decades.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/if-we-really-want-an-ideas-boom-we-need-more-women-at-the-top-tiers-of-science-56999">If we really want an ideas boom, we need more women at the top tiers of science</a>
</strong>
</em>
</p>
<hr>
<p><em>Interdisciplinary research</em> happens when researchers from different fields come together to pose a challenge that wouldn’t be possible in isolation. One example is the highly transparent optical fibres that underpin intercontinental telecommunication networks.</p>
<p>The knowledge creation that made this possible involved glass chemists, optical physicists and communication engineers coming together to articulate the possible, and develop the shared language required to make it a reality. When fields go on this journey together over decades, new fields are born.</p>
<p>In this example the question itself was clear – how can we harness the transparency of silica glass to create optical transmission systems that can transport large volumes of data over long distances?</p>
<p>But what about the questions we don’t know how to pose because without knowledge of another field we don’t know what’s possible? This line of reasoning leads us into the domain of <em>transdisciplinary research</em>. </p>
<p>Transdisciplinary research requires a willingness to craft new questions – whether because they were considered intractable or because without the inspiration from left field they simply didn’t arise. An example of this is applying <a href="http://www.adelaide.edu.au/news/news94522.html">photonics to IVF incubators</a> - the idea that it could be possible to “listen” to how embryos experience their environment is unlikely to have arisen without bringing these fields together.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/national-science-statement-a-positive-gesture-but-lacks-policy-solutions-experts-74987">National Science Statement a positive gesture but lacks policy solutions: experts</a>
</strong>
</em>
</p>
<hr>
<p>In my own field, physics, I discovered that when talking to people from other areas the simple question “what would you like to measure?” quickly led to uncharted territory. </p>
<p>Before long we were usually, together, posing fundamentally new questions and establishing teams to tackle them. This can be scary territory but it’s tremendously rewarding and creates space for creativity and the emergence of disruptive technologies.</p>
<h2>Excellence, communication, co-location, funding</h2>
<p>One of the best ways of getting out of a disciplinary silo is to take every opportunity to talk to others outside your field. Disciplinary excellence is the starting point to get to the table. </p>
<p>And while disciplinary collaborations can flourish over large distances because they share a language and values, it’s usually true that once you mix disciplines co-location becomes a real asset. Then of course there are the questions of how we fund and organise research concentrations to allow inter- and transdisciplinary research to flourish. </p>
<p>With the increased emphasis on impact, these questions are becoming ever more pressing. Organisations that get this right will thrive.</p><img src="https://counter.theconversation.com/content/42824/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Tanya Monro receives funding from the Australian Research Council. She is Deputy Vice Chancellor of the University of South Australia, a member of the Commonwealth Science Council, the CSIRO board, the SA Economic Development Board and Defence SA. </span></em></p>Ensuring knowledge creates impact involves disciplinary excellence, communication, co-location and funding.Tanya Monro, Deputy Vice Chancellor Research & Innovation, University of South AustraliaLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/753412017-06-19T10:57:51Z2017-06-19T10:57:51ZAcademics fear the value of knowledge for its own sake is diminishing<figure><img src="https://images.theconversation.com/files/174025/original/file-20170615-23518-1d10ljr.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Is“useful” knowledge the only knowledge worth knowing?</span> <span class="attribution"><span class="source">shutterstock</span></span></figcaption></figure><p>A climate of “<a href="https://www.theguardian.com/environment/2017/apr/18/donald-trumps-march-for-science-washington-climate-change">anti-intellectualism</a>”, faltering levels of trust in “<a href="https://www.theguardian.com/tv-and-radio/2016/jul/02/professor-brian-cox-interview-forces-of-nature">experts</a>” and an era of “<a href="https://theconversation.com/navigating-the-post-truth-debate-some-key-co-ordinates-77000">post-truth</a>” provides a rather dreary depiction of the state of academia today. </p>
<p>Compound this with the reorganisation of higher education – where universities are run more like <a href="https://theconversation.com/in-the-race-to-turn-higher-education-into-a-market-were-ignoring-lessons-from-history-35792">businesses</a> – along with the politics of austerity, and it may be little surprise that the sector is said to be in <a href="https://www.theguardian.com/news/2017/jan/08/from-universities-to-schools-the-system-is-in-crisis">crisis</a>.</p>
<p>This is all coming at a time when there is an <a href="https://theconversation.com/what-ref-case-studies-reveal-on-measuring-research-impact-39349">increased expectation</a> for academics to be more accountable for their research by evidencing its economic and societal benefits – known as impact. </p>
<p>This expectation has received mixed responses from many people working in universities. At first, some academics crudely dismissed impact as a nasty government idea. Many researchers could not see how their work could align with it and, fearing a loss of freedom <a href="http://www.nature.com/news/science-funding-duel-to-the-death-1.11073">some claimed</a> “science is dead”. This was even accompanied by the <a href="http://www.nature.com/news/science-funding-duel-to-the-death-1.11073">arrival of a hearse</a> outside the offices of the Engineering and Physical Sciences Research Council in the UK – sending out the message loud and clear that the <a href="https://theconversation.com/chasing-the-money-in-science-funding-will-lead-to-fools-gold-14310">impact agenda</a> was problematic and unwelcome. All of which reflected deep emotional and moral concerns within academia about the over management and politicisation of knowledge.</p>
<p>But on the flip side, impact has been welcomed by others for the <a href="https://theconversation.com/scientists-need-to-prove-their-research-is-worth-it-14618">opportunity</a> it provides academics to make their work more visible and accessible.</p>
<h2>The impact agenda</h2>
<p>To find out more, <a href="http://www.tandfonline.com/doi/abs/10.1080/07294360.2017.1288709?journalCode=cher20">our research</a> looked at academics’ emotions in response to the impact agenda – both in the UK and Australia. As part of this, we carried out interviews with 51 professors and senior career-level academics. </p>
<p>Our findings confirm that while pockets of the academic community are deeply concerned about an impact agenda – both in terms of funding and assessment – these reactions do not reflect a lack of willing or sense of duty. Rather academics want to see disciplinary diversity respected and this reflected in terms of research policy.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/174003/original/file-20170615-23518-walbd8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/174003/original/file-20170615-23518-walbd8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/174003/original/file-20170615-23518-walbd8.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/174003/original/file-20170615-23518-walbd8.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/174003/original/file-20170615-23518-walbd8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/174003/original/file-20170615-23518-walbd8.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/174003/original/file-20170615-23518-walbd8.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">What makes research valuable?</span>
<span class="attribution"><span class="source">Pexels</span></span>
</figcaption>
</figure>
<p>The academics we spoke to expressed a range of emotions regarding this increased focus on impact. These ranged from distrust to acceptance, and from excitement, to love and hate. For every academic who spoke of despair, a balance of commitment and even love for their work (and its potential for impact) was also expressed.</p>
<p>As one politics lecturer said: </p>
<blockquote>
<p>It’s sort of where my heart lies – quite deliberately and specifically working to apply the research that you are doing to real world political and social challenges across domains of theory and practice.</p>
</blockquote>
<p>An archaeology professor also expressed similar sentiments:</p>
<blockquote>
<p>We are paid from the public purse and we should be doing research – we are ridiculously privileged to work on whatever we like and it’s wonderful. </p>
<p>To bend your mind a little bit to the fact that some of the stuff you do does have benefits outside the academy, and to put measures in place to make that happen, it’s a minor tax.</p>
</blockquote>
<h2>Justifying your job</h2>
<p>But despite these positive sentiments, academics we spoke to also expressed concerns over their workloads and career security. </p>
<p>Others feared losing credibility and worried about being “exposed” or losing control of their work through public engagement. Though this is perhaps indicative of a lack of skill and confidence in this area. As well as a greater need for academics to understand how to communicate their research appropriately as opposed to “tokenistically”.</p>
<p>Academics also felt the impact agenda challenged them to justify their existence and their academic freedom – something which was felt on a very personal level. A music professor explained:</p>
<blockquote>
<p>I don’t feel happy with it, and do I need to justify my job? How many levels do I have to justify it?</p>
</blockquote>
<h2>Useful knowledge</h2>
<p>During the interviews, words such as “scary”, “threat”, “nervousness” and “worry” were used, as many spoke of their “frustrations”, “suspiciousness” and even “resentment” of the focus on impact. </p>
<p>Academics reported feeling sad, unhappy, jealous, anxious, demoralised and disillusioned by the impact agenda. And this sense of vulnerability seemed to be further exacerbated by risks of <a href="https://www.theguardian.com/higher-education-network/2016/oct/28/i-couldnt-get-funding-for-my-research-so-i-sold-out-and-i-dont-regret-it">professional penalisation from their academic peers</a>. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/174004/original/file-20170615-23518-1t4sl9a.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/174004/original/file-20170615-23518-1t4sl9a.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/174004/original/file-20170615-23518-1t4sl9a.jpeg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/174004/original/file-20170615-23518-1t4sl9a.jpeg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/174004/original/file-20170615-23518-1t4sl9a.jpeg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=502&fit=crop&dpr=1 754w, https://images.theconversation.com/files/174004/original/file-20170615-23518-1t4sl9a.jpeg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=502&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/174004/original/file-20170615-23518-1t4sl9a.jpeg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=502&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Who gets to decides what’s useful knowledge and what isn’t?</span>
<span class="attribution"><span class="source">Pexels</span></span>
</figcaption>
</figure>
<p>But it was clear from talking with these academics that these criticisms do not come from a place of entitlement or frustration at having to account for their work. Instead, this was in response to fears about the changing nature of their role and concerns for those whose work does not naturally align with what’s considered to be “useful”. And this increasing pressure to focus on impact at all costs could well damage academic morale, as one theatre professor described:</p>
<blockquote>
<p>This agenda reinforces the notion that the only valuable thing in life is money. That is deeply worrying.</p>
</blockquote>
<p>Ultimately, our research shows that most academics feel a duty to share their work, they want to make a difference, and they want <a href="http://www.humansandnature.org/the-moral-obligations-of-scientists">to communicate their findings</a> to wider audiences. But many are still uncomfortable with this idea of having to “sell” their work, as well as the preoccupation with what is “useful” – because after all how do you really decide what is or isn’t useful to society?</p><img src="https://counter.theconversation.com/content/75341/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Jennifer Chubb consults for FastTrack Impact</span></em></p>Are we in danger of losing academic freedom?Jennifer Chubb, Doctoral Researcher, University of YorkLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/712232017-01-16T15:19:18Z2017-01-16T15:19:18ZHow plugging into well-connected colleagues can help research fly<figure><img src="https://images.theconversation.com/files/152622/original/image-20170113-11191-hv7zq4.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Getting input from well-connected academics and researchers is crucial to a paper's scientific impact.</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>Informal intellectual collaboration <a href="http://jfe.rochester.edu/jointed.pdf">is crucial</a> for good social science research. This includes interactions with colleagues to improve a paper before it is sent to a journal.</p>
<p>Our <a href="https://ssrn.com/abstract=2877586">new research</a> explored the value of informal intellectual collaboration. It highlights the importance of social networks in academia. </p>
<p>What we uncovered suggests the scientific impact of a research paper increases with every additional commenter who provides feedback. This impact is measured by the number of citations over a paper’s lifespan. The same holds when we look at the probability of publishing a paper in top journals.</p>
<p>But here’s the true novelty of our paper: it found that the feedback of more central or connected people is more valuable than less central, less connected ones when it comes to impact. And no, it’s not as simple as just asking your most senior colleague for their input. Seniority isn’t what matters. It’s all about how well connected an academic or researcher is.</p>
<p>This is important information. Our results should encourage university management to actively encourage collaboration among scientists, across departments, and across universities – and to make networking and seeking feedback part of PhD training.</p>
<h2>Connectedness is key</h2>
<p>So how do you define “well connected” in this case?</p>
<p>A researcher is well connected in a social network because they are connected to other well-connected researchers. We used what might sound like a tautological idea in our research: the so-called <em>eigenvector</em> centrality, which posits that if you know important people you are probably important in that field, too. It is the <a href="http://www.math.cornell.edu/%7Emec/Winter2009/RalucaRemus/Lecture3/lecture3.html">same idea</a> that allows Google’s search algorithm to identify relevant websites.</p>
<p>But, as we’ve already pointed out, our findings weren’t about “importance”, or status. These eigenvector central academics are not necessarily the most well known or most senior. And they aren’t always affiliated to the most prestigious universities. Yet in the social network they occupy influential positions. It’s about connections. Think of them as opinion leaders.</p>
<p>Feedback from eigenvector central academics has a much larger impact on a paper’s publication success than feedback from isolated loners. Highly connected commenters may point authors to emerging <a href="http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00430">new topics</a> or the most rewarding avenues for future research. </p>
<p>To reach these conclusions, we built the first and most comprehensive view of the social network structure among financial economists. It connects authors and acknowledged commenters from published papers. This is a novel approach because it captures all those that have contributed to a paper, not only authors.</p>
<p>Our innovative approach was to use acknowledgements as a primary source of data. In financial economics, authors often acknowledge from which colleagues they have received helpful feedback. We collected more than 5,800 research papers from six major financial economics journals. About 90% of these acknowledge helpful input by colleagues.</p>
<p>After consolidation we create the network. Two researchers are connected when they have co-authored a paper or one acknowledges the other. This network connects about 7,500 researchers and indicates information flows between them. Then we computed the network positions and ranked individuals according to their eigenvector centrality. </p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=600&fit=crop&dpr=1 600w, https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=600&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=600&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=754&fit=crop&dpr=1 754w, https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=754&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/152626/original/image-20170113-11183-1x3zl1k.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=754&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">A visualisation of the largest connected component of the network of intellectual collaboration in financial economics using publications from 2009 to 20011. Red links connect researchers when they have co-authored a paper, blue links indicate that one acknowledged the other, and purple links indicate the both happened. The darker the node the more important the researcher is in terms of eigenvector centrality.</span>
</figcaption>
</figure>
<p>Such an analysis helps uncover patterns and structures that remain hidden when looking at individual researchers only. </p>
<p>We then used a quasi-natural experiment – the assignment of discussants at top conferences – to show our main argument: getting feedback from a colleague increases the scientific impact of a paper more if the colleague is more eigenvector central in the social network of their profession.</p>
<p>On our <a href="http://www.central-places.net/index">website</a>, we have developed an interactive tool where financial economists can find themselves on our database.</p>
<p>In a <a href="https://ssrn.com/abstract=2709107">companion paper</a> we explore the determinants of the most eigenvector central financial economists. That is, we contrasted their eigenvector centrality rank with their individual characteristics. We found that traditional author metrics such as citation counts or their number of published papers cannot explain which researchers are eigenvector central. </p>
<p>One part of the answer is certainly that citation counts have many <a href="http://www.nature.com/nature/journal/v465/n7300/full/465870a.html">problems</a> and poorly capture true academic strength. Another one is that is generally difficult to identify the opinion leaders unless you know all the network.</p>
<h2>New insights in the sociology of economics</h2>
<p>Our analysis is not exhaustive and research is ongoing. But it is clear that understanding knowledge flows helps in understanding <a href="http://voxeu.org/article/nine-facts-about-top-journals-economics">productivity differentials</a> among scientists. </p>
<p>Hopefully these results will inspire university managers to actively encourage collaboration among scientists, across department and across universities. </p>
<p>Our results also support calls to measure scientific impact <a href="http://blogs.lse.ac.uk/impactofsocialsciences/2015/04/20/scholarly-behaviour-evaluation-criteria-citations/">broader</a>, and not just based on citations. </p>
<p>Finally, our findings highlight the importance of sufficient travel funding for academics, given the crucial role of <a href="http://blogs.lse.ac.uk/impactofsocialsciences/2016/08/16/the-last-great-unknown-the-impact-of-academic-conferences/">academic conferences</a> as a networking opportunity.</p>
<hr>
<p><em>Authors’ note: This article is based on a <a href="http://blogs.lse.ac.uk/impactofsocialsciences/2017/01/10/feedback-helps-increase-the-impact-of-academic-research-even-more-so-when-coming-from-well-connected-colleagues/">post</a> written for the London School of Economics’ blog.</em></p><img src="https://counter.theconversation.com/content/71223/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>The scientific impact of a research paper increases with every additional commenter who provides feedback – particularly if the comment came from a well-connected academic.Michael E. Rose, PhD Candidate in Economics, University of Cape TownCo-Pierre Georg, Senior Lecturer, African Institute for Financial Markets and Risk Management, University of Cape TownLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/710782017-01-12T19:27:23Z2017-01-12T19:27:23ZFive things to consider when designing a policy to measure research impact<figure><img src="https://images.theconversation.com/files/152344/original/image-20170111-29019-qdkzu6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">What's the best way to measure research impact?</span> <span class="attribution"><span class="source">from www.shutterstock.com</span></span></figcaption></figure><p>This year will see the Australian government <a href="http://www.arc.gov.au/news-media/media-releases/2017-pilot-test-impact-business-engagement-researchers">pilot</a> new ways to measure the impact of university research. </p>
<p>As recommended by the <a href="https://theconversation.com/watt-report-suggests-financial-incentives-for-measuring-research-impact-51815">Watt Review</a>, the Engagement and Impact Assessment will encourage universities to ensure academic research produces wider economic and social benefits. </p>
<p>This fits into the <a href="https://theconversation.com/will-the-national-innovation-and-science-agenda-deliver-australia-a-world-class-national-innovation-system-52081">National Innovation and Science Agenda</a>, in which taxpayer funds are targeted at research that will have a beneficial future impact on society. </p>
<p>Education Minister <a href="https://ministers.education.gov.au/birmingham/2017-pilot-test-impact-business-engagement-researchers">Simon Birmingham said</a> the pilots will test</p>
<blockquote>
<p>“how to measure the value of research against things that mean something, rather than only allocating funding to researchers who spend their time trying to get published in journals”. </p>
</blockquote>
<p>This move to measure the non-academic impact of research introduces many new challenges that were not previously relevant when evaluation focused solely on academic merit. <a href="http://dx.doi.org/10.1080/1360080X.2016.1254429">New research</a> highlights some of the key issues that need to be addressed when deciding how to measure impact.</p>
<h2>1. What should be the object of measurement?</h2>
<p>Research impact evaluations needs to trace out a connection between academic research and “real world” impact beyond the university campus. These connections are enormously diverse and specific to a given context. They are therefore best captured through case studies. </p>
<p>When analysing a case study the main issues are: what counts as impact, and what evidence is needed to prove it? When considering this, Australian policymakers can use recent <a href="http://dx.doi.org/10.1080/10564934.2016.1237703">European examples</a> as a benchmark. </p>
<p>For instance, in the UK’s <a href="http://www.ref.ac.uk/">Research Excellence Framework</a> (REF) – which assesses the quality of academic research – the only impacts that can be counted are those directly flowing from academic research submitted to the same REF exercise. </p>
<p>To confirm the impact, the beneficiaries of research (such as policymakers and practitioners) are required to provide written evidence. This creates a narrow definition of impact because those that cannot be verified, or are not based on submitted research outputs, do not count. </p>
<p>This has been a cause of frustration for some UK researchers, but the high threshold does ensure the impacts are genuine and flow from high quality research.</p>
<h2>2. What should be the timeframe?</h2>
<p>There are unpredictable time lapses between academic work being undertaken and it having impact. Some research may be quickly absorbed and applied, whereas other impacts, particularly those from basic research, can take decades to emerge. </p>
<p>For example, <a href="http://journals.sagepub.com/doi/full/10.1258/jrsm.2011.110180">a study looking at time lags in health research</a> found the time lag from research to practice to be on average 17 years. It should be noted, though, that time lapses vary considerably by discipline.</p>
<p>Only in hindsight can the value of some research be fully appreciated. Research impact assessment exercises therefore need to be set to a particular timeframe. </p>
<p>Here, policymakers can learn from previous trials such as one conducted by <a href="https://www.go8.edu.au/sites/default/files/docs/eia_trial_guidelines_final_mrb.pdf">Australian Technology Network and Group of Eight in 2012</a>. This exercise allowed impacts related to research that occurred during the previous 15 years.</p>
<h2>3. Who should be the assessors?</h2>
<p>It is a long established convention that academic excellence is decided by academic peers. Evaluations of research are typically undertaken by panels of academics. </p>
<p>However, if these evaluations are extended to include non-academic impact, does this mean there is now a need to include the views of end-users of research?
This may mean the voices of people outside of academia need to be involved in the evaluation of academic research. </p>
<p>In the 2014 UK REF, over 250 “<a href="http://www.ref.ac.uk/about/users/">research users</a>” (individuals from the private, public or charitable sectors) were recruited to take part in the evaluation process. However, their involvement was restricted to assessing the impact component of the exercise. </p>
<p>This option is an effective compromise between maintaining the principle of academic peer review of research quality while also including end-users in the assessment of impact.</p>
<h2>4. What about controversial impacts?</h2>
<p>In many instances the impact of academic research on the wider world is a positive one. But there are some impacts that are controversial - such as fracking, genetically modified crops, nanotechnologies in food, and stem cell research - and need to be carefully considered. </p>
<p>Such research may have considerable impact, but in ways that make it difficult to establish a consensus on how scientific progress impacts “the public good”. Research such as this can trigger societal tensions and ethical questions. </p>
<p>This means that impact evaluation needs to also consider non-economic factors, such as: quality of life, environmental change, and public health. Even though it is difficult placing dollar values on these things.</p>
<h2>5. When should impact evaluation occur?</h2>
<p>Impact evaluation can occur at various stages in the research process. For example, a funder may invite research proposals where the submissions are assessed based on their potential to produce an impact in the future. </p>
<p>An example of this is the European Research Council <a href="https://erc.europa.eu/proof-concept">Proof of Concept Grants</a>, where researchers who have already completed an ERC grant can bid for follow-on funding to turn their new knowledge into impacts.</p>
<p>Alternatively, impacts flowing from research can be assessed in a retrospective evaluation. This approach identifies impacts where they already exist and rewards the universities that have achieved them. </p>
<p>An example of this is the <a href="http://www.vsnu.nl/en_GB/sep-eng.html">Standard Evaluation Protocol</a> (SEP) used in the Netherlands, which assesses both the quality of research and its societal relevance.</p>
<p>A novel feature of the proposed Australian system is the assessment of both <a href="https://theconversation.com/when-measuring-research-we-must-remember-that-engagement-and-impact-are-not-the-same-thing-56745">engagement and impact</a>, as two distinctive things. This means there isn’t one international example to simply replicate. </p>
<p>Although Australia can learn from some aspects of evaluation in other counties, the Engagement and Impact Assessment pilot is a necessary stage to trial the proposed model as a whole. </p>
<p>The pilot - which will test the suitability of a wide range of indicators and methods of assessment for both research engagement and impact - means the assessment can be refined before a planned national rollout in 2018.</p><img src="https://counter.theconversation.com/content/71078/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Andrew Gunn receives funding from Worldwide Universities Network, the British Council (administering the Newton Fund), the UK Higher Education Academy, the United Kingdom Political Studies Association, the New Zealand Political Studies Association and the UK Quality Assurance Agency. Andrew Gunn concurrently holds visiting academic positions internationally.
</span></em></p><p class="fine-print"><em><span>Michael Mintrom does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>This move to measure the impact of university research on society introduces many new challenges that were not previously relevant when evaluation focused solely on academic merit.Andrew Gunn, Researcher in Higher Education Policy, University of LeedsMichael Mintrom, Professor of Public Sector Management, Monash UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/667692016-10-18T06:45:58Z2016-10-18T06:45:58ZWikipedia is already the world’s ‘Dr Google’ – it’s time for doctors and researchers to make it better<figure><img src="https://images.theconversation.com/files/142101/original/image-20161018-12440-1f759xc.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Research shows that Wikipedia is one of the most read sources of medical information by the general public across the world.</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/67272961@N03/6123892769/">jfcherry/Flickr</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Health professionals have a duty to improve the accuracy of medical entries in Wikipedia, according to a letter published today in <a href="http://www.thelancet.com/journals/langlo/article/PIIS2214-109X(16)30254-6/fulltext">Lancet Global Health</a>, because it’s the first port of call for people all over the world seeking medical information.</p>
<p>In our correspondence, a group of international colleagues and I call on medical journals to do more to help experts make Wikipedia more accurate, and for the medical community to make improving its content a top priority. </p>
<h2>Use around the world</h2>
<p>Ranked the <a href="http://www.alexa.com/topsites">fifth most-visited website</a> in the world, Wikipedia is one of the <a href="http://jamia.oxfordjournals.org/content/16/4/471.full">most-read sources of medical information</a> by the general public. It’s also frequently the first port of call for <a href="http://www.jmir.org/2015/3/e62/?trendmd-shared=1">doctors</a>, <a href="http://www.tandfonline.com/doi/full/10.3109/0142159X.2012.737064?scroll=top&needAccess=true">medical students</a>, <a href="http://heinonline.org/hol-cgi-bin/get_pdf.cgi?handle=hein.journals/yjolt12&section=3">lawmakers</a>, and <a href="http://www.pewinternet.org/2013/02/28/how-teachers-are-using-technology-at-home-and-in-their-classrooms/">educators</a>.</p>
<p>Access is provided free of charge on mobile phones in many countries, under the <a href="https://blog.wikimedia.org/c/wikipedia-zero/">Wikipedia Zero scheme</a>. In developing nations, this has helped the site become the main source of information on medical topics. During the <a href="http://www.nytimes.com/2014/10/27/business/media/wikipedia-is-emerging-as-trusted-internet-source-for-information-on-ebola-.html?_r=0">2014 Ebola outbreak</a>, for instance, page views of the <a href="https://en.wikipedia.org/wiki/Ebola_virus_disease">Ebola virus disease</a> peaked at more than <a href="http://www.wikipediatrends.com/Ebola_virus_disease.html">2.5 million per day</a>.</p>
<p>Earlier this year, the site launched the free <a href="https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomwikimed&hl=en">Medical Wikipedia Offline app</a> in seven languages. The Android app has had nearly 100,000 downloads in its first few months of release. It’s particularly useful in low and middle-income countries, where internet access is typically slow and expensive.</p>
<p>All this makes Wikipedia’s accuracy vital because every medical entry on the collaborative online encyclopedia has the potential for immediate <a href="http://www.who.int/bulletin/volumes/87/4/08-056713/en/">real-world health consequences</a>.</p>
<h2>A question of priorities</h2>
<p>Given its model of allowing anyone to edit entries, Wikipedia is already <a href="http://www.nature.com/nature/journal/v438/n7070/full/438900a.html">surprisingly accurate</a>, famously rivalling Encyclopedia Britannica. But even as the online encyclopedia matures, the accuracy of its medical content remains inconsistent.</p>
<p>The platform has historically <a href="https://www.theguardian.com/education/2011/mar/29/wikipedia-survey-academic-contributions">struggled to attract expert contributions</a> from researchers. Improving Wikipedia entries tends to be low on the list of priorities for doctors and other health professionals. </p>
<p>Finding time to write unpaid content in an unfamiliar format can easily lose out to more immediate career concerns. Doctors consistently work <a href="http://jamanetwork.com/journals/jama/fullarticle/1475198">long hours</a> with patients, and researchers tend to be busy applying for grants and <a href="http://jmi.sagepub.com/content/14/4/321.short">publishing in academic journals</a>. </p>
<p>The <a href="https://en.wikipedia.org/wiki/Stillbirth">entry on stillbirth</a> illustrates well why Wikipedia needs to attract more expert contributors. Every day, there are <a href="http://www.who.int/reproductivehealth/topics/maternal_perinatal/stillbirth/en/">7,000 stillbirths worldwide</a>, but before my colleagues and I updated the Wikipedia page, it was was missing crucial information.</p>
<p>It didn’t mention key causes, such as malaria, and common complications, such as depression. Having the full picture of a medical condition is extremely <a href="https://www.jmir.org/2003/3/e17/?trendmd-shared=1">important for effective health care</a>. And it’s vital for patients as well. Knowing that depression is a normal side effect of stillbirth, for instance, can help women <a href="http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2524.2008.00814.x/full">cope with the emotional fallout</a>. </p>
<p>Similarly, <a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0106930">accurate information on medication</a> affects what doctors prescribe, what patients request, and what students learn.</p>
<p>Such important topics quite simply demand accuracy.</p>
<h2>Possible solutions</h2>
<p>While spotting the shortfall is easy, solving it will require the concerted efforts of multiple communities with unique strengths.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/142102/original/image-20161018-12440-13bev85.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/142102/original/image-20161018-12440-13bev85.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=398&fit=crop&dpr=1 600w, https://images.theconversation.com/files/142102/original/image-20161018-12440-13bev85.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=398&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/142102/original/image-20161018-12440-13bev85.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=398&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/142102/original/image-20161018-12440-13bev85.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=500&fit=crop&dpr=1 754w, https://images.theconversation.com/files/142102/original/image-20161018-12440-13bev85.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=500&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/142102/original/image-20161018-12440-13bev85.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=500&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Wikipedia has historically struggled to attract expert contributions from time-poor researchers and doctors.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/youraccount/7839794684/">Garnet/Flickr</a>, <a class="license" href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC BY-NC-ND</a></span>
</figcaption>
</figure>
<p>Doctors and researchers can provide expert knowledge about complex topics; medical journals can leverage their infrastructure for robust peer review and indexing; Wikipedians can provide their experience in encyclopedic writing and technical expertise; and medical schools can encourage student involvement. </p>
<p>Simultaneously publishing peer-reviewed work in academic journals and in Wikipedia could benefit all participants. This would include both putting existing entries through academic peer review, and converting suitable journal articles into Wikipedia entries. Official recognition of authors’ efforts through their citeable publications by scholarly journals is an important reward for time-pressed contributors.</p>
<p>Peer review would ensure the quality of content, and for journals wanting to <a href="http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(16)31771-8/fulltext">have an impact on public health</a>, Wikipedia is the among the best outreach tools available.</p>
<p>Several scholarly journals have been exploring academic peer review of Wikipedia entries and more look to soon join them. Examples of joint-publishing include the Wikipedia articles for Dengue fever and the cerebellum, which have been reviewed and published by the medical journals <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4242787">Open Medicine</a> and the <a href="https://en.wikiversity.org/wiki/WikiJournal_of_Medicine/The_Cerebellum">WikiJournal of Medicine</a> respectively.</p>
<p>PLOS Computational Biology similarly <a href="http://collections.plos.org/topic-pages">joint-publishes review articles</a> in its journal and in Wikipedia for maximum impact. And, the journal RNA Biology requires researchers describing a <a href="http://www.nature.com/news/2008/081216/full/news.2008.1312.html">new RNA family</a> to also write a Wikipedia entry for it.</p>
<h2>Embedding the new approach</h2>
<p>Progress has been slow, but several independent ventures show how the attitudes of major players in the biomedical ecosystem are beginning to shift further, and take Wikipedia more seriously.</p>
<p>Cochrane, which creates medical guidelines after reviewing research data, now finds <a href="http://www.cochranelibrary.com/editorial/10.1002/14651858.ED000069">Wikipedian partners for its Review Groups</a> to help disseminate their information through Wikipedia.</p>
<p>Medical schools are also <a href="http://journals.lww.com/academicmedicine/Abstract/publishahead/Why_Medical_Schools_Should_Embrace_Wikipedia__.98408.aspx">getting involved</a> in improving Wikipedia entries. Medical students at University of California, San Francisco, can <a href="http://www.nytimes.com/2013/09/30/business/media/editing-wikipedia-pages-for-med-school-credit.html?_r=0">gain course credit</a> for supervised editing of Wikipedia articles in need of attention.</p>
<p>These and similar schemes can hopefully normalise Wikipedia editing within future medical community. And patients will ultimately be the winners. When it comes to health content, the deadline is now.</p><img src="https://counter.theconversation.com/content/66769/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Thomas Shafee is an active Wikipedian and is on the WikiJournal of Medicine editorial board. </span></em></p>Medical entries on Wikipedia are widely consulted across the world. Doctors and medical researchers need to make efforts to ensure the content on the online collaborative encyclopedia is accurate.Thomas Shafee, Research Fellow in Biochemistry and Evolution, La Trobe UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/616842016-06-27T19:08:59Z2016-06-27T19:08:59ZHow a deep dive into people’s lives helps to separate fact from fiction<figure><img src="https://images.theconversation.com/files/128264/original/image-20160627-28362-9ow7gb.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">In-depth surveys allow governments to drill their understanding down to street level.</span> <span class="attribution"><span class="source">Image courtesy GCRO/Clive Hassall</span></span></figcaption></figure><p>How do you decide whether a society is making progress or not, and on what basis are these judgements made? In an election year – which is the case in nearly <a href="https://eisa.org.za/index.php/election-calendar/">a third</a> of Africa’s 54 countries during 2016 – political rhetoric spikes. But the sweeping statements that politicians make are seldom supported by evidence that can be trusted. </p>
<p>The <a href="http://gcro1.wits.ac.za/qolviewer/">Quality of Life survey</a>, conducted every two years over the past eight, has worked to address this. It is produced by a university-based research institute, South Africa’s <a href="http://www.gcro.ac.za/">Gauteng City-Region Observatory</a>. This was established in partnership with the Gauteng government. Its purpose is to inform provincial and local government so that they can make better policy decisions. The institute’s independent character underpins the reliability of the survey’s findings. </p>
<p>The survey is distinctive because it takes objective, subjective, economic and non-economic factors into account to arrive at a composite view of the quality of life in the region. As has become evident, personal human fulfilment is not reliant on the delivery of basic services alone, as important as these are.</p>
<p>One of the key things we’ve learned from these successive in-depth community surveys during the past eight years is that South Africa is a society characterised by relatively low levels of trust. This can be seen in how people relate to their fellow citizens, people from other countries and the government. A dearth of data can contribute to this deeply rooted mistrust. And the data that does exist may either be viewed as irrelevant – it addresses the wrong questions – or partisan.</p>
<p>By understanding the drivers of this mistrust, the government is better able to address its manifestations. For instance, South Africa has been beset by <a href="http://www.sahistory.org.za/article/xenophobic-violence-democratic-south-africa">xenophobic violence</a> in recent years. <a href="http://www.gcro.ac.za/media/reports/GCRO_Vignette_Informal_cross_border_trade_spending_in_Gauteng_FA_XD9dcZI.pdf">Data</a> shows politicians just how much migrants from elsewhere in Africa <a href="http://www.gcro.ac.za/media/reports/September_Map_of_the_Month_WrWTRBo.pdf">contribute</a> to Gauteng’s fortunes. They can then share this empirical, accurate information with the general public to hopefully start shifting xenophobic attitudes.</p>
<h2>Unpacking attitudes</h2>
<p>The most recent survey polls the views of 30 000 respondents across all quarters of Gauteng, South Africa’s wealthiest and most populous province, and its economic heartland. The province attracts people from around the country and the continent. </p>
<p>Respondents from all walks of life are probed about their material circumstances. This includes their dwelling, the basic services they can count on, their employment, income and availability of transport. They’re also asked to discuss the psycho-social dimensions of “headspace”: their perceptions around race relations, neighbourhood, xenophobia and gender relations. Some of the questions allow respondents to reflect on their attitudes to the various spheres of government, as well as the likelihood that they’ll vote or participate in civic affairs.</p>
<p>Such surveys are an example of scholarly research that’s purpose-designed to provide rigorous and disinterested data and insight. These can then be used by the government and broader civil society. Our research occupies the hybrid space between the worlds of academia and government, with the intention to inform the complex business of urban futures. Researchers can gain an increasingly nuanced insight into the challenges of governance, then shape their work to address these questions. In this way, surveys like ours become a reliable source of frank information for government to use in its decision-making processes. </p>
<p>This relationship requires a high degree of trust between the parties. The researchers commit to pursuing their scholarly work in good faith for the public good. The government guarantees that it won’t try to exert political control over the results – even when they reveal discomforting insights. </p>
<p>Primary data-sets from the Quality of Life survey are made available to civil society organisations, other academics and political parties, for their own analysis. This guarantees the independence of our work. In other words, this is publicly-funded research, made publicly available, to inform the processes of democracy.</p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/128335/original/image-20160627-28373-1cb6o6g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/128335/original/image-20160627-28373-1cb6o6g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=134&fit=crop&dpr=1 600w, https://images.theconversation.com/files/128335/original/image-20160627-28373-1cb6o6g.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=134&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/128335/original/image-20160627-28373-1cb6o6g.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=134&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/128335/original/image-20160627-28373-1cb6o6g.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=168&fit=crop&dpr=1 754w, https://images.theconversation.com/files/128335/original/image-20160627-28373-1cb6o6g.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=168&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/128335/original/image-20160627-28373-1cb6o6g.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=168&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">What better way to understand a complex city like Johannesburg than by talking to its residents?</span>
<span class="attribution"><span class="source">Paul Saad/Flickr</span>, <a class="license" href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC BY-NC-ND</a></span>
</figcaption>
</figure>
<h2>A thirst for evidence</h2>
<p>The key question, of course, is whether this research makes a difference to how government does its work. The call for evidence-based policy-making is gathering <a href="https://theconversation.com/the-science-of-using-research-why-it-starts-with-the-policymaker-59265">increasing credence</a> in government circles. It is clear that both academics and governments need to understand more clearly how such work can be done.</p>
<p>In the first instance, research that seeks to inform governance – and much research does not have this intention – needs to be configured to ask the right questions. Its outcomes need to framed in ways that make them digestible and usable in government decision-making. Academia in general is not known for these qualities. It is predicated on advancing scholarly fields of knowledge rather than the practical business of government. </p>
<p>On the other hand government agencies need to have the capability to receive research insights and integrate these into their regulative systems. Again, this is not a natural feature of many government processes. These are inclined rather to respond to political imperatives and the contingencies of the day. Good quality research takes time; government often demands nimble responsiveness. Finding the means to bridge these incommensurate cultures is not straightforward and there’s clearly still a long way to go.</p>
<h2>Changing, informing governance</h2>
<p>So how does one judge the impact so far of this kind of work on government decision-making? One indicator is the growing appetite for the information provided from this survey: Gauteng’s government has continued to expand its funding for the research and the province’s <a href="http://www.localgovernment.co.za/provinces/view/3/gauteng">three metropolitan municipalities</a> have now added to this. They want increasingly nuanced data about their respective wards and are moving towards linking senior managers’ performances with the survey’s outcomes.</p>
<p>Gauteng’s Premier <a href="http://whoswho.co.za/david-makhura-7560">David Makhura</a> has provided strong public leadership on key issues, for example on <a href="http://ewn.co.za/2016/05/25/Makhura-Africa-must-unite-and-reject-xenophobia">xenophobic violence</a>. This is informed at least in part by insights from previous iterations of the Quality of Life survey. </p>
<p>However, increasingly evidence-based and inclusionary processes for governance and policy-making are necessarily ongoing objectives. They require sustained political will (beyond the inevitable term-limit churn), increasingly astute approaches to policy-related research, and the capabilities within government to invest in and make use of the insights from systematic scholarly work. </p>
<p>Strenghtening this knowledge-based dialogue between sectors of society will ultimately enable South Africans to make decisive improvements in the quality of life for all its citizens.</p><img src="https://counter.theconversation.com/content/61684/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Dr. Robin Moore is the Executive Director of the Gauteng City Region Observatory (GCRO), which is a partnership between the Universities of Johannesburg (UJ) and the Witwatersrand (Wits) and the Gauteng Provincial Government (GPG) and organised local government. The GCRO receives funding from the GPG, the Metropolitan governments within Gauteng and in-kind support from the two Universities. Rob Moore is the Chair of the Board of The Conversation Africa.</span></em></p>Without data, people don’t know what to believe or whom to trust. Empirical, thorough data collected by academics can help to fill important governance gaps.Robin Moore, Executive Director: Gauteng City-Region Observatory, University of the WitwatersrandLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/597082016-05-24T12:04:59Z2016-05-24T12:04:59ZWhy getting medical information from Wikipedia isn’t always a bad idea<figure><img src="https://images.theconversation.com/files/123554/original/image-20160523-11000-6mwr04.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">More medical experts should contribute to Wikipedia to ensure its health pages are accurate.</span> <span class="attribution"><span class="source">Gary Cameron/Reuters</span></span></figcaption></figure><p>Wikipedia’s detractors will tell you that the site is inaccurate, incomplete and unreliable. Many universities <a href="http://isites.harvard.edu/icb/icb.do?keyword=k70847&pageid=icb.page346376">won’t allow</a> students to use Wikipedia as a reference in essays or assignments. So it may come as a surprise to learn that it’s the <a href="http://www.ncbi.nlm.nih.gov/pubmed/25739399">most commonly used</a> source for obtaining medical information online – even among medical students and doctors.</p>
<p>In fact, research has found that Wikipedia is <a href="http://www.ncbi.nlm.nih.gov/pubmed/25739399">more popular</a> for this kind of information than reputable bodies’ websites – including those belonging to the World Health Organisation and the US’s Centres for Disease Control. In some settings, researchers <a href="http://www.ncbi.nlm.nih.gov/pubmed/25739399">have discovered</a>, more than 90% of medical students and 50% of doctors turn to Wikipedia at some point.</p>
<p>But the academic medical community largely views Wikipedia <a href="http://www.ncbi.nlm.nih.gov/pubmed/26443650">with suspicion</a>. This appears to be because the site doesn’t adhere to traditional peer-review mechanisms. There’s also no reward for a busy academic or medical practitioner who takes the time to improve existing Wikipedia pages and ensure that medical information is accurate. Some traditional journals and medical schools are starting to take Wikipedia more seriously, but we wanted to take things a step further by marrying Wikipedia and a traditional journal model. That’s how the <a href="https://en.wikiversity.org/wiki/Wikiversity_Journal_of_Medicine">Wikiversity Journal of Medicine</a> was born.</p>
<h2>Attitudes starting to shift</h2>
<p>Most journals are expensive, hard to access and considered quite elite. They also <a href="http://www.straitstimes.com/opinion/prof-no-one-is-reading-you">aren’t read</a> by very many people beyond academia and research houses. Research has suggested that medical journals need to increase their <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3221335/">social impact</a> by actively promoting knowledge sharing on sites like Wikipedia. This offers scope for people all over the world and from a variety of language groups to get more reliable information about health and medicine.</p>
<p>Some journals have heeded this call. <a href="http://journals.plos.org/ploscompbiol/">PLOS’s Computational Biology</a>, for instance, requires any author it publishes to also write a <a href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002446">Wikipedia page</a> on the topic. The journal article is static, referenced and unchanging. The Wikipedia page is changeable and invites contributions. Another journal, RNA Biology, <a href="http://dx.doi.org/10.1038/news.2008.1312">requires</a> the same approach. </p>
<p>There have also been <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4242788/">experiments</a> that have seen a Wikipedia article put through traditional medical journal quality control processes. It is then formally published in the journal and the original Wikipedia article is updated.</p>
<p>A few medical schools are embracing this new approach, too. The University of California San Francisco has <a href="http://www.nytimes.com/2013/09/30/business/media/editing-wikipedia-pages-for-med-school-credit.html?_r=0">introduced a course</a> into its curriculum that teaches medical students how to contribute to Wikipedia.</p>
<p>These are all laudable efforts that point to a growing <a href="https://arxiv.org/ftp/arxiv/papers/1506/1506.07608.pdf">open-access</a> movement in the world of <a href="http://dx.doi.org/10.1161/CIRCOUTCOMES.115.002415">scholarly communication</a>. </p>
<h2>Challenges and successes</h2>
<p>The Wikiversity Journal of Medicine, which was launched in 2014, is hosted directly by the <a href="https://wikimediafoundation.org/wiki/Home">Wikimedia Foundation</a>, the same organisation that hosts Wikipedia. It uses the same <a href="https://en.wikipedia.org/wiki/MediaWiki">software</a>, MediaWiki, which makes editing and processing very easy.</p>
<p>The whole service is free to authors and readers; as with Wikipedia our operating costs are covered by donations from around the world. The Wikiversity Journal of Medicine follows standard international best-practice guidelines for medical journals, drawing from such reputable bodies as the <a href="http://www.icmje.org/icmje-recommendations.pdf">International Committee of Medical Journal Editors</a>. </p>
<p>Submission and acceptance follow the traditional medical journal processes, including peer-review by experts on the topic that’s being written about. One important difference is that authors have the option of submitting their article directly onto the journal’s site. This option was designed to enhance transparency and has been taken up by some authors. Others have been more hesitant, as other journals may consider a paper that’s on Wikiversity to be already publicly available and may reject it as a result.</p>
<p>The editorial board includes people from three continents: Africa, North America and Europe. Among them are the editor-in-chief, Sweden’s Dr Mikael Häggström. He’s made extensive <a href="https://upload.wikimedia.org/wikiversity/en/7/7b/Medical_gallery_of_Mikael_H%C3%A4ggstr%C3%B6m_2014.pdf">image contributions</a> to Wikipedia – for example, the site’s <a href="https://en.wikipedia.org/wiki/Ebola_virus_disease">Ebola</a> page features <a href="https://upload.wikimedia.org/wikipedia/commons/4/4a/Symptoms_of_ebola.png">his images</a>.</p>
<p>Dr James Heilman is another board member. He’s arguably the world’s leading expert on <a href="http://www.ncbi.nlm.nih.gov/pubmed/?term=James+Heilman">Wikipedia and medicine</a>.</p>
<p>So far the journal has published 16 articles about diverse medical topics. We believe that the journal’s association with Wikipedia has created the false notion that anyone can edit an accepted journal manuscript. There are two versions of each published article. One is a PDF that cannot be edited and stands as the version of record. The second is a wiki and can be edited by anyone. The board monitors these edits. </p>
<p>The journal’s model has potential, though. A US physics professor, Guy Vandegrift, has established <a href="https://en.wikiversity.org/wiki/Second_Journal_of_Science">a second</a> wiki-based journal. This, along with the broader debate around open access to medical information, suggests that the Wikiversity Journal of Medicine provides a feasible, scalable and sustainable model. Of course, it should not be the only source of information – in the same way that no single article in any format should ever be one’s only source. We hope that even if medical experts and researchers don’t contribute to the journal, they will start to take Wikipedia more seriously and, where necessary, to improve it so that people have access to more reliable information.</p>
<p>Such initiatives can, we believe, help to further address the <a href="https://theconversation.com/its-time-to-redraw-the-worlds-very-unequal-knowledge-map-44206">profound inequities</a> in the global knowledge economy that greatly hamper public health.</p>
<p><em>This article was co-authored with Dr Mikael Häggström, a medical doctor in Sweden who is also the editor-in-chief of the Wikiversity Journal of Medicine.</em></p><img src="https://counter.theconversation.com/content/59708/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Gwinyai Masukume is the assistant to the editor-in-chief of the Wikiversity Journal of Medicine and is a Wikipedian.</span></em></p><p class="fine-print"><em><span>James Heilman is affiliated with the Wikiversity Journal of Medicine, Wikipedia, Wiki Project Med Foundation, and the Wikimedia Foundation.</span></em></p>The academic medical community largely views Wikipedia with suspicion. But some traditional journals are starting to take the site more seriously – and some journals work very closely with it.Gwinyai Masukume, Medical Doctor, Epidemiologist and Biostatistician: University College Cork, University of the WitwatersrandJames Heilman, Clinical Associate Professor, Department of Emergency Medicine, University of British ColumbiaLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/592652016-05-19T17:57:52Z2016-05-19T17:57:52ZThe science of using research: why it starts with the policymaker<figure><img src="https://images.theconversation.com/files/123171/original/image-20160519-30717-1j5e4w6.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Piles of evidence don't make any difference if they're not being used to develop policy.</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>Governments all over the world <a href="http://data.worldbank.org/indicator/GB.XPD.RSDV.GD.ZS">invest</a> large sums of public money into producing knowledge that helps them understand their countries’ complex socioeconomic issues. This knowledge, in the form of research, can be used to formulate potential solutions through public policies and programmes. </p>
<p>But it’s not enough just to produce research. It must also be considered and drawn from when policies are being created. However, a range of barriers might prevent policymakers from accessing and using evidence in their work. To understand the use of evidence, then, it’s important to understand the policymaker. Who is she? What are her incentives and biases? What is her professional and institutional context? </p>
<p>This is important for two reasons. The first is that it’s wasteful for governments to fund research – with taxpayers’ money – that’s just going to gather dust. The second is that governments may implement programmes and policies that have no impact or are actually detrimental to the very people they’re supposed to help. This isn’t unprecedented: a programme run in the US to <a href="http://www.jstor.org/stable/3658560?seq=1#page_scan_tab_contents">scare juvenile delinquents “straight”</a> was implemented even though researchers had shown that it had, on average, previously caused more harm than merely leaving these young people be.</p>
<p>A new global <a href="http://eppi.ioe.ac.uk/cms/Portals/0/PDF%20reviews%20and%20summaries/Science%202016%20Langer%20report.pdf?ver=2016-04-23-122500-213">systematic review</a> conducted by the <a href="http://eppi.ioe.ac.uk/cms/">Evidence for Policy and Practice Information and Co-ordinating Centre</a> has shed some light on the important issue of getting more scientific about the practice and study of research use. Our report, “Science of using science: researching the use of research evidence in decision-making”, combined insights from 36 existing systemic reviews that reported on 91 different research-use interventions. It identifies the most effective strategies for increasing and strengthening how research is used to build public policy.</p>
<h2>What the review found</h2>
<p>A golden thread throughout the review findings is the importance of getting serious about approaching research use from a policymaker’s perspective. For example, we found that programmes supporting practical research-use skills, such as appraising the quality of a study, were effective. Likewise, targeting and tailoring the communication of research findings to policymakers’ preferences yielded positive results. This could be achieved by framing research findings according to policymakers’ mode of decision-making – such as being risk or loss averse. </p>
<p>We also found that policymakers place an opportunity cost on every interaction. They’ll forgo and sacrifice other commitments or work to engage with researchers. If those interactions don’t come with tangible benefits, the policymaker is unlikely to bother making time for researchers again.</p>
<p>Each policymaker will have her own networks of people with whom she engages and shares information. So if researchers engage with the same group of policymakers again and again, there is a risk that the research they share never spreads through the system. Researchers need to target policymakers who can act as bridges between, for instance, different government departments. This creates more effective networks through which evidence can flow.</p>
<p>Lastly, the review identified how important it is to facilitate evidence use through organisational processes. This could involve supervising how the evidence is used and giving policymakers the tools they need to apply evidence effectively.</p>
<h2>The South African context</h2>
<p>We were particularly interested in how these findings can be applied in South Africa, as this is where we conduct our work through the <a href="http://www.africaevidencenetwork.org/">Africa Evidence Network</a>.</p>
<p>South Africa is one of only a handful of countries that has created government structures that institutionalise the use of research evidence in policymaking. Government <a href="http://www.dpme.gov.za/publications/Strategic%20Plan%20and%20Annual%20Reports/DPME%20Strategic%20Plan%202015-2020.pdf">policy</a> is organised according to a framework of 14 key outcomes that all departments must work towards. A range of evidence is used to assess government’s progress and the effects of its policies and programmes on contributing to the national key outcomes. </p>
<p>South Africa is in a rare position: there’s a high-level demand for evidence-informed decision-making. The country’s cabinet meetings often <a href="https://pmg.org.za/committee-meeting/16821/">discuss</a> impact evaluation reports. Organisational structures and processes are being put in place to nurture this demand.</p>
<p>This approach is yielding results. A number of national policies have already been systematically informed by the best available research evidence. These include the <a href="http://www.gov.za/services/child-care-social-benefits/child-support-grant">child support grant</a> and the <a href="http://www.treasury.gov.za/documents/national%20budget/2011/Confronting%20youth%20unemployment%20-%20Policy%20options.pdf">youth wage subsidy</a>.</p>
<p>But, as <a href="http://www.aejonline.org/index.php/aej/article/view/145">a survey</a> has shown, the use of evidence is still far from common decision-making practice. Government demand for evidence also relies on a research supply of policy-relevant evidence, which can be <a href="http://www.ksi-indonesia.org/index.php/publications/2016/03/23/82/using-evidence-to-reflect-on-south-africa-s-20-years-of-democracy.html">a challenge</a> at times. There is still a lot of work to be done. Our review offers some ideas and suggestions that South Africa and other countries could adopt.</p>
<h2>Effective strategies</h2>
<p>It is crucial to invest in policymakers’ skills to use evidence. If they have the capacity and tools they need, there’s a greater chance they’ll use evidence. In South Africa, a number of different organisations and bodies offer capacity building around research use. But their activities are not homogeneous.</p>
<p>A more systematic approach to capacity building would mean that public servants and policymakers are exposed to similar support, particularly at provincial government level.</p>
<p>Also, if policymaking is to be more frequently informed by scientific evidence, researchers need to understand policymaking. Sadly most researchers don’t often leave their natural habitat at universities to engage and collaborate directly with policymakers. Researchers and policymakers could establish mentoring relationships – an effective strategy for exchanging knowledge.</p>
<p>One thing lies at the heart of all these suggestions: the use of evidence as a salient feature in decision- and policymaking. </p>
<h2>Engagement and dialogue</h2>
<p>Direct engagement between researchers and policymakers is crucial. There are growing opportunities for this, such as at the <a href="https://confsa.eventsair.com/QuickEventWebsitePortal/evidence-2016-conference/aen-2016">2016 Africa Evidence Network conference</a>. The conference will focus on three themes: engage, understand and impact. These also feature in the discussion around a <a href="http://www.ksi-indonesia.org/index.php/publications/2016/03/23/82/using-evidence-to-reflect-on-south-africa-s-20-years-of-democracy.html">new landmark report</a>, “Using evidence to reflect on South Africa’s 20 years of democracy”, which was published in March 2016.</p>
<p>This and similar initiatives mean that the time is ripe for South Africa’s research-use community to interrogate how effective its strategies are in supporting evidence-informed decision-making. It’s time to become more scientific about the use of research evidence.</p>
<p><em>Authors’ note: The Science of Using Science project was led by the Alliance for Useful Evidence, with generous funding and support from the Wellcome Trust and the What Works Centre for Wellbeing. The research was undertaken by Laurenz Langer, Janice Tripney and David Gough of the EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London.</em></p><img src="https://counter.theconversation.com/content/59265/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Laurenz Langer receives funding from from the UK government's Department for International Development for a programme supporting capacity to use research evidence in southern Africa.</span></em></p><p class="fine-print"><em><span>Ruth Stewart receives funding from the UK government's Department for International Development for a programme supporting capacity to use research evidence in southern Africa.</span></em></p>Researchers and policymakers need to talk to each other. If they don’t, important research will merely gather dust and policies might do more harm than good.Laurenz Langer, Senior Researcher, University of JohannesburgRuth Stewart, Professor: Evidence-Informed Decision-Making, University of JohannesburgLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/567452016-04-10T20:02:33Z2016-04-10T20:02:33ZWhen measuring research, we must remember that ‘engagement’ and ‘impact’ are not the same thing<figure><img src="https://images.theconversation.com/files/117931/original/image-20160408-23914-15ysns8.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">What is the purpose of measuring engagement, impact or quality?</span> <span class="attribution"><span class="source">from www.shutterstock.com</span></span></figcaption></figure><p>In the <a href="https://theconversation.com/turnbull-seeks-ideas-boom-with-innovation-agenda-experts-react-51892">Innovation Statement</a> late last year, the federal government indicated a strong belief that<a href="https://theconversation.com/ten-rules-for-successful-research-collaboration-53826"> more collaboration</a> should occur between industry and university researchers. </p>
<p>At the same time, <a href="https://docs.education.gov.au/system/files/doc/other/20151203_main_report1.pdf">government</a>, <a href="https://www.go8.edu.au/programs-and-fellowships/excellence-innovation-australia-eia-trial">education</a> and industry groupings have made numerous recommendations for the “impact” of university research to be assessed alongside or in addition to the existing assessment of the quality of research. </p>
<h2>How should we measure research?</h2>
<p>But what should we measure and, more importantly, why should we measure it?</p>
<p>In accounting, we stress that the measurement basis of something inevitably reflects the purpose for which that measure is to be used. </p>
<p>So what is the purpose of measuring engagement, <a href="https://theconversation.com/there-is-no-easy-way-to-measure-the-impact-of-university-research-on-society-50856">impact</a> or, for that matter, quality? </p>
<p>The primary reason for measuring quality seems fairly self-evident – as a major stakeholder in terms of funding (especially dedicated research-only funding), the government wants an assessment of just “how good” by academic standards such research really is. </p>
<p>Looking ahead, measures of quality such as the Excellence in Research for Australia (ERA) rankings have been speculated to potentially influence future funding via prestigious competitive schemes (such as the Australian Research Council), block funding for infrastructure and the availability of government support for doctoral students via Australian Postgraduate Awards. </p>
<p>So the demand for a measure of research quality and the potential uses of such a measure are pretty clear.</p>
<p>But what valid reasons are there for investing significant resources in the measurement of research impact or engagement? </p>
<p>If high-quality research addresses important practical problems (large or small), surely we would expect impact would follow? </p>
<p>In this sense, the extent of impact is really a joint product of the quality (or robustness) of research and the choice of topic (ie, practical versus more esoteric).</p>
<h2>Research impact needs time</h2>
<p>But over what period should impact be measured? </p>
<p><a href="https://www.go8.edu.au/programs-and-fellowships/excellence-innovation-australia-eia-trial">Recent exercises</a> such as that conducted by the Australian Technology Network and Group of Eight have a relatively short-term focus, as would any “impact assessment” tied to the corresponding period covered by the existing ERA time frame (say the last six years). </p>
<p>I and many others maintain that impact can only be assessed over much longer periods, and that in many cases short-term impact is potentially misleading. </p>
<p>How often have supposedly impactful results subsequently been rejected or overturned? </p>
<p>Such examples inevitably turn out to reflect low quality (and in some cases outright fraudulent) research.</p>
<h2>Ranking impact</h2>
<p>Finally, how can impact be ranked? Is there a viable measure that can distinguish between high and low impact? Existing case-study approaches are unlikely to yield any form of quantifiable measurement of research impact.</p>
<p>Equally puzzling is the call to measure research engagement. What is the purpose of such an exercise? Surely in a financially constrained research environment, universities readily recognise the importance of such engagement and pursue it constantly. </p>
<p>We don’t need a national assessment of engagement to encourage universities to engage. </p>
<p>Motive aside, one approach canvassed is the quantum of non-government investment in research (ie, non-government research income). </p>
<p>This is arguably one rather limited way to measure engagement, and is focused on input rather than output. If the purpose of any measurement is to capture outcomes, does it make sense to focus exclusively on inputs? The logic of this escapes me.</p>
<h2>Engagement and impact are not the same thing</h2>
<p>Even more worryingly, some use the terms engagement and impact interchangeably. </p>
<p>They would have us believe that a simple (but useful) measure of impact is the extent to which university researchers receive industry funding. Surely this is, at best, a measure of engagement, not impact.</p>
<p>Although the two are likely correlated, the extent will vary greatly across discipline areas. </p>
<p>Further, in business disciplines, much of the “knowledge transfer” that occurs via education (including areas such as executive programs) reflects the impact of the constant process of researching better business practices across areas such as accounting, finance, economics, marketing and so on.</p>
<p>Discretionary expenditure on such programs by business is surely an indication of the extent to which business schools and industry are engaged, yet this would be ignored if we focused on research income alone.</p>
<p>We must not lose sight that quality (ie, rigour and innovativeness) is a necessary but not sufficient condition for broader research impact.</p>
<p>Engagement is not impact, and simple measures such as non-government research income tell us very little about genuine external engagement between universities and industry.</p>
<p>As accountants know, performance measurement reflects its purpose. What we need before any further national assessment of attributes such as impact or engagement is clear understanding of the purpose of such an exercise. </p>
<p>Only when the purpose is clearly specified can we have a sensible debate about measurement principles.</p><img src="https://counter.theconversation.com/content/56745/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Stephen Taylor is affiliated with the Australian Business Dean's Council as the 2016 ABDC Research Scholar </span></em></p>Engagement is not impact, and simple measures such as non-government research income tell us very little about genuine external engagement between universities and industry.Stephen Taylor, Professor of Accounting, University of Technology SydneyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/552612016-04-04T09:37:47Z2016-04-04T09:37:47ZAn education in irony: why academics need to be funny<figure><img src="https://images.theconversation.com/files/115859/original/image-20160321-30908-1740j1o.jpg?ixlib=rb-1.1.0&rect=16%2C13%2C949%2C639&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Education should be a laughing matter.</span> <span class="attribution"><span class="source">Ollyy/www.shutterstock.com</span></span></figcaption></figure><p>Despite what some people might think, academics are not all humourless boffins out of touch with the real world. They can also be funny, and some are turning to humour to help get their messages across. </p>
<p>Last November, the Times Higher Education <a href="https://www.timeshighereducation.com/news/why-did-chicken-cross-road-public-engagement-course">reported that</a> zooarchaeologists from the University of Nottingham had hired a stand-up comedian to disseminate the findings of their research on “human-chicken interactions” to the public. While the report took the opportunity to poke a bit of fun at this, the purpose was to make the entirely serious point that human-chicken interactions are no joke. </p>
<p>Using comedy to create awareness of research is less ridiculous than it might at first appear. Public engagement is the new black in academia, with pressure growing on academics around the world to <a href="https://theconversation.com/the-impact-of-impact-on-the-ref-35636">show the “impact”</a> of their research. A growing number of academics are now starting to realise the potential of comedy to deliver hard-hitting social insights. </p>
<p><audio preload="metadata" controls="controls" data-duration="806" data-image="" data-title="Extract from The Anthill 16 podcast: Humour Me, on humour and academia." data-size="12378050" data-source="The Conversation" data-source-url="https://theconversation.com/anthill-16-humour-me-82845" data-license="CC BY-ND" data-license-url="http://creativecommons.org/licenses/by-nd/4.0/">
<source src="https://cdn.theconversation.com/audio/879/humour-academia.mp3" type="audio/mpeg">
</audio>
<div class="audio-player-caption">
Extract from The Anthill 16 podcast: Humour Me, on humour and academia.
<span class="attribution"><a class="source" rel="nofollow" href="https://theconversation.com/anthill-16-humour-me-82845">The Conversation</a>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a><span class="download"><span>11.8 MB</span> <a target="_blank" href="https://cdn.theconversation.com/audio/879/humour-academia.mp3">(download)</a></span></span>
</div></p>
<h2>Academics take to the stage</h2>
<p>In 2010, <a href="https://www.theguardian.com/science/blog/2010/nov/04/laugh-learn-science-bright-club">two science communicators</a> in public engagement at University College London created the <a href="http://www.brightclub.org/">Bright Club</a>, where “researchers become comedians for just one night”. The success of the format – each performance features a mix of both comedians and researchers – has resulted in chapters sprouting up across the UK. </p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/tEBfCyJdG9A?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Bright Club in action in Dundee.</span></figcaption>
</figure>
<p>One of the Bright Club founders, Steve Cross, <a href="http://www.scienceshowoff.org/">created and now comperes Science Showoff</a> a “chaotic cabaret for science lovers”. He welcomes any sort of performance, including stand-up comedy, and is keen to have academics present. </p>
<p>It would be easy to dismiss the performers at the Bright Club as wannabe stand-ups intent on escaping the drudgery of academia if only for one night. But good stand-up comedians are capable of achieving what social scientists often crave: getting an audience to critically engage with a subject. </p>
<h2>Humour as a tool</h2>
<p>And as anthropologist Kate Fox <a href="https://books.google.co.uk/books/about/Watching_the_English_The_International_B.html?id=fKtvAgAAQBAJ&source=kp_cover&redir_esc=y">has argued</a>: “at its best … social science can sometimes be almost as insightful as good stand-up comedy”. </p>
<p>Comedy can be used to challenge social norms, or speak out against social injustices. For instance, what better way to begin explaining what a class system is than to say it is “what you use to discriminate against people who look like you” – as comedian <a href="http://www.boreme.com/posting.php?id=32551#.Vq-AOdKLS72">Reginald D Hunter has done</a>.</p>
<p>But comedy in the social sciences is not just about public engagement and brightening up the otherwise dull existence of the average academic. Humour also has an important role to play in the social sciences as a tool for analysis and its neglect should be a cause for concern, given its essential place in the experience of being human. </p>
<p>Humour hasn’t just been neglected by social scientists, it has been actively rejected. To be a humorous academic appears to be an unacceptable oxymoron and those who use humour in their work run the risk of being seen as non-serious, and therefore trivial. Even <a href="http://sociology.about.com/od/Profiles/p/Erving-Goffman.htm">Erving Goffman</a>, one of the greatest social scientists of the 20th century, is regarded in some quarters with suspicion for his “sparkling” humorous prose. There are many who would agree with his own apparently self-denigrating epithet of being an “elegant bullshitter”.</p>
<h2>It doesn’t get better than irony</h2>
<p>Goffman’s technique depends on irony. By overturning received ideas and the logical contradictions that shape our most deeply held convictions, irony has the potential to provide new theoretical insights. It can function as an analytical tool. </p>
<p>An appreciation of irony is arguably an essential attribute for the social scientist. It underlies <a href="http://www.econlib.org/library/Enc/UnintendedConsequences.html">the law of unanticipated consequences</a> which says that “actions of people – and especially of government – always have effects that are unanticipated or unintended”. It has been claimed by some very eminent thinkers (among them Adam Smith, Karl Marx and Machiavelli) that this is the most important phenomenon the social sciences have to deal with. </p>
<p>Irony also underpins the concept of <a href="http://kbjournal.org/long_tending">“planned incongruity”</a> in which, through a deliberately induced incongruity, the apparently rational is undermined, re-emerging as irony. For example, the American economist <a href="http://www.britannica.com/biography/Thorstein-Veblen">Thorstein Veblen’s</a> idea of “trained incapacity” – that a person’s narrow specialisation in a particular subject can actually widen their field of ignorance. </p>
<p>Irony certainly produces comic pleasure and the extent of this pleasure can be directly related to two key principles: the principle of “high contrast” and the “law of irony”. The principle of high contrast says that the more incongruous the ideas put together, the more pleasurable the result will be for the reader or listener. The law of irony, meanwhile, asserts that when an ironic statement combines high contrast and high certainty of outcome, the resulting statement is of greatest theoretical value. For example, the Utopian paradox which states that utopia, if achieved, would <a href="https://books.google.co.uk/books?id=RCXxEumXFhUC&pg=PA174&lpg=PA174&dq=richard+harvey+brown+law+of+irony&source=bl&ots=M81CP3p6-u&sig=pLmvYr1gq33exJ7V-pwiWWlZrPA&hl=en&sa=X&ved=0ahUKEwj0hInM19jKAhVFXRQKHfQlCaoQ6AEIHzAA#v=snippet&q=%22law%20of%20irony%22&f=false">result in disaster</a>. </p>
<p>By combining these two ideas, we can see that the most pleasure is derived from the highest incongruity, which also gives rise to the greatest theoretical insight. This produces the somewhat unexpected – hence ironic – finding that the most important sociological insights are also likely to be the ones which produce the greatest comic effect. In other words, the funniest.</p><img src="https://counter.theconversation.com/content/55261/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Irony can provide new theoretical insights. Social scientists should embrace it.Cate Watson, Professor in Professional Education and Leadership, School of Social Sciences, University of Stirling., University of StirlingIoannis Costas Batlle, PhD Researcher in Education, University of BathLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/553332016-02-24T17:03:47Z2016-02-24T17:03:47ZAcademics’ ability to lobby government under threat from new funding clause<figure><img src="https://images.theconversation.com/files/112739/original/image-20160224-15614-qbk2v5.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">No go zone for academics?</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/garryknight/4404488882/sizes/l">garryknight/flickr.com</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span></figcaption></figure><p>There are growing worries in universities that restrictions on the use of public money to lobby the government, initially focused on charities, may have a chilling effect on the independence of publicly funded research. Whether this is ministers’ intention is an open question. </p>
<p>In mid-February, the <a href="https://www.gov.uk/government/news/government-announces-new-clause-to-be-inserted-into-grant-agreements">government announced</a> that a new clause will be inserted into new and renewed grant agreements from May 1, forbidding recipients from “using taxpayer funds to lobby government and parliament”. As well as covering grants to <a href="https://theconversation.com/why-new-anti-lobbying-rules-leave-small-charities-out-in-the-cold-54509">charities</a> to carry out services, this <a href="https://www.gov.uk/government/publications/interim-guidance-on-applying-a-new-clause-in-government-grant-agreements">also covers</a> those “funding research and development”.</p>
<p>Pessimists, and conspiracy theorists, discern a pattern of the government displaying alarming authoritarian instincts. Ministers do not seem to recognise the need for pluralism in an open and democratic society. So “short” money, to help fund political parties (and a “loyal” opposition that is duty-bound to attack the government), <a href="http://www.independent.co.uk/news/uk/politics/george-osborne-quietly-cuts-short-money-funding-to-opposition-parties-a6748696.html">has been cut</a>. Meanwhile, local councils <a href="http://www.theguardian.com/society/2016/feb/15/councils-and-nhs-trusts-to-be-blocked-from-boycotting-israeli-products">have been instructed</a> not to use procurement procedures to enforce boycotts of dubious suppliers. It is not just campaigning charities that are in the government’s sights.</p>
<h2>Public interest put aside</h2>
<p><a href="https://www.gov.uk/government/news/government-announces-new-clause-to-be-inserted-into-grant-agreements">Comments by Matt Hancock</a>, the minister for the Cabinet Office, when announcing the new clause, were not reassuring. He justified the restrictive clause on the grounds it would put a stop to “the farce of government lobbying government”. Leaving aside the large-scale lobbying of the Treasury, or his own Cabinet Office, by other Whitehall departments, and by local government or the Scottish and Welsh Governments (all publicly funded, by definition), this is a chilling as well as a naïve view. </p>
<p>He does not seem to recognise that it is in the wider public interest, if not that of the ruling party, that the government is held to account and vigorous public debate is encouraged. These are basic principles of democracy.</p>
<p>Optimists point out that the government’s target is narrower, radical charities especially those on the fringes of Islamic fundamentalism – so many political initiatives undermining human and political rights can be traced back to the toxin of “the war on terror”. No doubt, direct-action environmental campaigners are in there, too. They add, to reassure nervous scientists and scholars, that there are no explicit references to publicly funded research. </p>
<p>They point to the <a href="https://www.gov.uk/government/publications/interim-guidance-on-applying-a-new-clause-in-government-grant-agreements">actual phrasing</a> of the new regulations for charities: </p>
<blockquote>
<p>The following costs are not eligible expenditure: payments that support activity intended to influence or attempt to influence parliament, government or political parties, or attempting to influence the awarding or renewal of contracts and grants, or attempting to influence legislative or regulatory action. </p>
</blockquote>
<p>No – direct – threat to the independence of research. And yet … the bulk of research in UK universities is public funding for projects by research councils and core funding through the Higher Education Funding Councils. Much of the rest comes from charities (which arguably could get caught in the government’s anti-lobbying net), and <a href="https://theconversation.com/britain-should-stay-in-the-eu-for-science-18129">successive European programmes</a> (although the much-maligned Brussels bureaucracy has no plans to impose similar restrictions on lobbying). </p>
<figure class="align-right ">
<img alt="" src="https://images.theconversation.com/files/112740/original/image-20160224-16455-w273p1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/112740/original/image-20160224-16455-w273p1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=600&fit=crop&dpr=1 600w, https://images.theconversation.com/files/112740/original/image-20160224-16455-w273p1.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=600&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/112740/original/image-20160224-16455-w273p1.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=600&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/112740/original/image-20160224-16455-w273p1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=754&fit=crop&dpr=1 754w, https://images.theconversation.com/files/112740/original/image-20160224-16455-w273p1.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=754&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/112740/original/image-20160224-16455-w273p1.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=754&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Shut out?</span>
<span class="attribution"><span class="source">Agenturfotografin/www.shutterstock.com</span></span>
</figcaption>
</figure>
<p>If the government genuinely believes its new anti-lobbying rules do not apply to publicly funded research, it should consider explicitly exempting it – to remove all possible doubt.</p>
<p>This is not so much an argument for special treatment as for consistency across government, because we do not yet have – thank goodness – a monolithic Jacobin or Stalinist State. For a start there is a range of regulation, and even some <a href="http://www.legislation.gov.uk/ukpga/1988/40/section/202">legislation</a>, that protects academic freedom, and <a href="http://www.universitiesuk.ac.uk/highereducation/Documents/2011/FreedomOfSpeechOnCampus.pdf">places a duty</a> on universities to uphold it. </p>
<h2>Conflicting demands</h2>
<p>Yet academic freedom without the right to active dissent, and effective action in support of that dissent, is not worth much. There is now <a href="http://www.rcuk.ac.uk/RCUK-prod/assets/documents/documents/RCUKOpenAccessPolicy.pdf">also a requirement</a> that the findings of all publicly funded research is made available freely by being placed in open-source repositories, on the grounds that what the public has paid for the public has a right to access. The public not the government, please note, although this is a distinction ministers seem to be struggling to understand.</p>
<p>And it was the government, admittedly the previous Labour one not the present government, that <a href="https://theconversation.com/the-impact-of-impact-on-the-ref-35636">insisted the “impact” of research</a> should be a key criterion in measuring its quality – not the university and research community which remained sceptical. Yet again, the “impact” of much social scientific research cannot be reduced to producing “evidence” to support pre-selected government agendas. Nor is it easy to draw a line between the vigorous dissemination of research findings (a “good thing”) and advocacy or lobbying (“bad things”). </p>
<p>The risk that the anti-lobbying rules will have a worrying effect on university research cannot be dismissed. The government has form here – not just the cuts in the “short” money and the restrictions on local authority procurement but also, much closer to home, the restrictions its <a href="https://theconversation.com/beware-the-security-creep-into-british-universities-37867">Prevent anti-radicalisation strategy</a> have placed on freedom of speech in universities. Its determination to propagate <a href="https://theconversation.com/uk/topics/british-values">“British values”</a> has the potential to actively undermine these very values.</p>
<p>The danger is probably not so much that mad ministers will succeed in occasionally censoring, or neutering, pieces of research but that habits of caution and timidity will spread glacially through Whitehall and quango-land. The current situation is a salutary reminder of the perverse effects of Margaret Thatcher’s belief that “there is no such thing as society”. We should not reject the idea that there is a public sphere distinct from, bigger than (and, yes, occasionally even antagonistic to) the State.</p><img src="https://counter.theconversation.com/content/55333/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Peter Scott has received funding from the Department of Business, Innovation and Skills for research into higher education in further education colleges and the Economic and Social Research Council and Higher Education Funding Council for England as a member of the Centre for Global Higher Education which they jointly fund. He is a member of the Labour Party and a trustee of the Council for the Defence of British Universities. </span></em></p>New rules could have a chilling effect on academic freedom of speech.Peter Scott, Professor of Higher Education Studies, UCLLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/521522015-12-14T19:26:24Z2015-12-14T19:26:24ZWill the impact framework fix the problems the research audit found?<figure><img src="https://images.theconversation.com/files/105367/original/image-20151211-8291-8gyfh3.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">How useful is ERA for measuring research quality?</span> <span class="attribution"><span class="source">from www.shutterstock.com</span></span></figcaption></figure><p>The results from the latest university research audit indicate that <a href="https://theconversation.com/are-australian-universities-getting-better-at-research-or-at-gaming-the-system-51895">research in Australia is improving</a>.</p>
<p>This suggests that the Excellence in Research for Australia (ERA) exercise is working: ERA has achieved its main aim of boosting the quality of Australian research.</p>
<p>However, this headline statement masks a plethora of concerns.</p>
<p>Under the <a href="https://theconversation.com/turnbull-seeks-ideas-boom-with-innovation-agenda-experts-react-51892">government’s latest reform</a> of research funding, academics will be assessed not only on their quality of research through the ERA, but also on the economic, social and environmental impacts of their research through <a href="https://theconversation.com/watt-report-suggests-financial-incentives-for-measuring-research-impact-51815">a new impact framework</a></p>
<p>The impact and engagement measures herald a new era that rewards researchers for collaborating beyond their institutions.</p>
<p>It is timely, then, to reassess ERA’s utility. Is it fit for purpose? Will these two assessment systems complement or contradict one another?</p>
<h2>What has gone well in ERA?</h2>
<p>The ERA processes have recognised peer review alongside metrics. </p>
<p>Research efforts at universities are arguably now more focused towards areas of strength. There is a clearer (though contested and arguably narrower) understanding of scholarly research, particularly that which is non-traditional. </p>
<p>On paper, ERA has established a system whereby research can be compared nationally and against international benchmarks.</p>
<h2>What isn’t working?</h2>
<p>Individual researchers are not assessed by ERA per se. However, they are assessed in line with ERA at the institutional level — in a system that awards a single score for an entire discipline cohort.</p>
<p>Inter-disciplinary research has been disadvantaged. ERA’s 1,238 fields of research (FoR) codes make it problematic for researchers to publish outside their discipline or academic unit. </p>
<p>Publishing, performing or exhibiting internationally is perceived to be more prestigious than in Australia. This unjustified exoticism diminishes the importance of Australian research and puts local and Australian publication outlets at risk. </p>
<p>A lack of transparency and accountability remains a critical problem. </p>
<p>The process by which final rankings are calculated remains opaque. It is unclear how the peer review of evaluation units is moderated and benchmarked globally. The rationale for inclusion, exclusion and change in the list of journals recognised by ERA has not been made public. </p>
<p>Whole disciplines ranked “below world average” are reliant on empirical research to fathom what went wrong. There is no feedback other than the score.</p>
<p>Esteem measures are narrow. The category “prestigious work of reference”, for example, is strikingly limited. It has never been opened to public discussion. Why have some publications been chosen and others omitted?</p>
<p>The ERA journal rankings were <a href="https://theconversation.com/why-the-era-had-to-change-and-what-we-should-do-next-1874">abolished in 2011</a>. However, their ghost influences decisions from journal selection to academic recruitment and promotion. </p>
<p>Universities still reward publication in high-ranking journals from the list; some institutions recognise only research published in A or A<sup>*</sup> journals, or those marked “quality” in the current list. </p>
<p><a href="http://link.springer.com/chapter/10.1007/978-94-007-7085-0_22">As predicted</a>, the editorial boards of these journals are struggling to cope with the influx of submissions. Lower-ranked journals and those with lower impact factors are struggling to survive. Many <a href="http://www.australianhumanitiesreview.org/archive/Issue-May-2009/genoni&haddow.htm">Australian journals</a> are disadvantaged by the bias towards international journals. </p>
<p>The audit culture most affects early career academics. They and others <a href="http://files.eric.ed.gov/fulltext/EJ926450.pdf">struggle</a> to negotiate the system, juggle heavy <a href="http://www-tandfonline-com.dbgw.lis.curtin.edu.au/doi/pdf/10.1080/07294360.2013.864616">teaching loads</a> and manage the <a href="https://minerva-access.unimelb.edu.au/bitstream/handle/11343/28917/264644_GoedegebuureAustraliasCasual.pdf?sequence=1">precarity</a> of casual academic employment. </p>
<p>The <a href="http://www.lhmartininstitute.edu.au/userfiles/files/research/attractiveness_ac_prof_res_brief.pdf">international mobility of Australian academics</a> is high and early career academics are the most likely to <a href="http://www.cshe.unimelb.edu.au/people/bexley_docs/The_Academic_Profession_in_Transition_Sept2011.pdf">move overseas or leave higher education</a>.</p>
<p>The loss of young academics from an ageing academic workforce risks Australia’s ability to meet future demand. Moreover, it impairs capacity for innovation.</p>
<h2>What are the concerns?</h2>
<p>Measuring engagement according to <a href="https://theconversation.com/australias-innovation-agenda-embracing-risk-or-gambling-with-public-health-52003">research income from industry</a> is concerning. </p>
<p>How, for example, will <a href="https://theconversation.com/in-universities-obsessed-with-research-heres-what-falls-between-the-cracks-938">collaborative research with not-for-profits</a> and innovative start-up companies be measured? How will the new measures account for these organisations’ exemptions from a cash contribution for Australian Research Council Linkage proposals? </p>
<p>There is a contradiction between a new impact measure that encourages a culture of risk-taking and ERA, which promotes risk-avoidance behaviours and impacts upon <a href="http://www.victoria.ac.nz/law/research/publications/vuwlr/prev-issues/volume-44,-issue-34/07-Butler.pdf">academic freedom</a> by directing research behaviour. This is particularly problematic for new researchers, blue-sky research and research with benefits that emerge only in the long term. </p>
<p>Both systems place professional service outside academic workloads. This raises new questions. Who will edit the journals, convene the conferences, become officers of professional associations, or write the handbooks and textbooks?</p>
<p>These activities are essential to the health of all disciplines. Increasingly, they are unrecognised and unrewarded. This has long-term ramifications for both research quality and impact. </p>
<p>Neither system recognises investments in partner communities that are critical to social licence to operate in many disciplines. </p>
<h2>Improving ERA</h2>
<p>Has ERA run its course? Perhaps. It certainly needs improvement. </p>
<p>The ERA process should be subject to external review. We need greater transparency about the criteria that inform assessment categories. We need discussion of categories not yet opened to consultation. </p>
<p>Given concerns <a href="https://theconversation.com/are-australian-universities-getting-better-at-research-or-at-gaming-the-system-51895">over gaming the system</a>, we need an audit of data that has been excluded from ERA submissions. There should be a review of disciplinary membership of the committees in terms of institutional representation through time.</p>
<p>We need ERA to cease peer reviews of outputs already subject to double-blind peer review.</p>
<p>There is a dire need to review the real cost of each ERA exercise, which runs approximately every three years. We need to consider whether the costs of assessing research excellence <a href="http://rev.oxfordjournals.org/content/20/3/247.short">exceed the benefits</a>. </p>
<p>While the ARC’s <a href="http://www.arc.gov.au/sites/default/files/filedepot/Public/ARC/Budget/PDF/2015_16%20ARC_Budget_Statements.pdf">administrative and departmental costs</a> are low, we also need to assess <a href="https://theconversation.com/the-era-assessed-cost-not-rated-and-league-tables-is-there-a-better-way-to-do-it-51865">the costs of university compliance</a> and of playing an effective strategic assessment game.</p>
<p>The new impact and engagement measures redress some of ERA’s deficiencies, but the challenges of cost, transparency, audit culture and external oversight remain. And teaching remains out in the cold.</p><img src="https://counter.theconversation.com/content/52152/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>In 2007 Claire Smith was a member of the Humanities Assessment Panel for the Australian Government's short-lived Research Quality Framework. From 2009-2011 she was a member of the Humanities and Creative Arts Panel, College of Experts, Australian Research Council.</span></em></p><p class="fine-print"><em><span>Dawn Bennett is an ARC Peer Reviewer and an ARC Assessor.</span></em></p>The new impact framework will improve some of the problems arising out of the ERA’s university research audit, but major challenges will remain.Claire Smith, Professor of Archaeology, Flinders UniversityDawn Bennett, Research Professor, Curtin UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/515802015-12-03T18:52:25Z2015-12-03T18:52:25ZShift away from ‘publish or perish’ puts the public back into publication<figure><img src="https://images.theconversation.com/files/103839/original/image-20151201-26546-1ygiihy.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Too many academic careers are shaped around writing journal articles nobody reads and planning twice-weekly lectures to a diminishing class of students.</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/sixthlie/4463280095/in/photolist-7NptF6-6exZWd-853ie9-6j1nCw-5uHok4-5uHo6M-5uHq7R-5uHq2a-oKtd57-532VZB-gCYnE-5uHpVi-5uHoVa-5uMNfw-5uHpxZ-5uHpHP-5uMKwQ-5uMMxd-5uMMMW-5uHopR-5uMKVd-5uHoQv-5uMLYd-5uMLmf-5uMM9y-5uHqoZ-5uMMC7-5uMLAL-iLW5F-5uHoMn-3KjAhq-4QX6U6-bmxRwg-cY9gH-igeuAm-3cqpFK-5BjEq5-EZenw-6eHX3K-6eHNe2-6eHZmn-6eJ3nR-9MuMhR-6eEoE7-6eAdfD-6eHXUr-64ysXy-nnHih-7GivPP-zLRm1">flickr/Sixth Lie</a>, <a class="license" href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC BY-NC-ND</a></span></figcaption></figure><p>Earlier this year, I visited the library at the Australian National University with my son so he could borrow some books for an essay on Chinese history. Wandering past shelf after shelf, he asked me, “How does it feel to be writing another book that no-one will read?”</p>
<p>It was just another teenage jibe, but in policy terms it was a prescient analysis.</p>
<p>In recent weeks there have been <a href="http://www.smh.com.au/federal-politics/political-news/academic-publications-to-become-less-important-when-funding-university-research-20151112-gkxkgl.html">reports</a> that the government is considering making publication output much less important in the formulae that allocate research funding to universities. </p>
<p>Prime Minister Turnbull has signalled a desire to move away from a “publish or perish” culture to a new set of academic incentives that prioritises engagement and impact.</p>
<p>With more than A$1 billion per year in research grants on the table, even a marginal change in allocation methods could see big changes in the dollars flowing to some fields of study. </p>
<p>There is real concern among some academics that the changes will be unfair: scientists will be able to demonstrate impact in the form of patents, commercial spin-offs and industry engagement much more readily than their colleagues in the social sciences or humanities. </p>
<p>When the new Chief Scientist, Alan Finkel, <a href="http://www.smh.com.au/federal-politics/political-news/academic-publications-to-become-less-important-when-funding-university-research-20151112-gkxkgl.html">talks about</a> the importance of demonstrating “a measurable return on investment,” historians, anthropologists, philosophers and linguists are understandably anxious.</p>
<p>But is a defensive reaction necessary?</p>
<h2>Change on the way</h2>
<p>Those of us who work in the social sciences and humanities place a great value on the persuasiveness of our words. We can write; not perfectly, but better than most. New and genuinely public forms of publication, rather than the semi-private domain of journals and monographs, provide us with powerful platforms for our academic passions. </p>
<p>We don’t need to be afraid of funding formulae that focus on the quality of societal engagement rather than the quantity of journal articles or monographs.</p>
<p>But it will take a change of attitude and of academic practice. </p>
<p>If we continue to shape our careers around the twice-weekly lecture (to a diminishing class of students) and two journal articles per year (in good quality journals, so our peers can praise them without reading them) our future will be much narrower than it could be.</p>
<p>Academic websites would be a good place for reform to start. Most departmental webpages are online ghost towns, attracting negligible traffic despite the effort and angst put into producing and, intermittently, maintaining them. They do very little to generate broader societal impact via outreach or engagement. </p>
<p>Rather, they exist primarily to reassure academic units of their own existence. They are like sacred totemic objects that symbolise the unity of the academic clan – they are brought out from seclusion in times of social crisis (such as a managerial attempt to rationalise unread online content), briefly venerated, and then forgotten. And one of the ironies of university life is that the managers of websites regularly complain that they struggle to receive content.</p>
<p>Effective engagement and outreach will require a much more nimble academic posture. We need to diversify the way we write. It’s time to stop looking down our nose at public commentary as a second rate form of academic communication. We can rediscover the power of images and sounds. </p>
<p>An ability to operate effectively in the online world should gradually become a baseline academic selection criteria; just as important as the ability to give a lecture or write a chapter. </p>
<h2>Rising to the challenge</h2>
<p>In no way should this diminish the importance of basic, speculative and even eccentric research. I am an anthropologist and, as my son kindly pointed out, I know what it’s like to write books and articles that don’t exactly fly off the shelves. But I have spent the past decade combining formal promotion-friendly publication with blogging, opinion pieces and media interviews. </p>
<p>The ideas, inspiration and energy flow two ways: from formal research to public outreach and back again. Some of my research has been rather esoteric (<a href="http://onlinelibrary.wiley.com/doi/10.1111/j.1835-9310.2006.tb00057.x/abstract">spirit beliefs in northern Thailand</a>, anyone?) but I have always enjoyed using insights from that work in public discussions about power, politics and democracy.</p>
<p>Have I been able to demonstrate, or even measure, the impact of my public outreach? To some extent, but certainly not perfectly. Working on this will be challenging and, at times, frustrating.</p>
<p>But engaging in the debate will be more productive than retreating behind a “nobody understands our worth” barricade. There are many qualitative and quantitative tools that we can use to demonstrate our engagement and impact. It will seem like sacrilege to many, but perhaps re-tweets could become an academic metric that sits alongside citation rates? </p>
<p>The challenge laid down by the government is not to abandon pure research or scholarly writing, but to put the public back into publication. </p>
<p>It’s a challenge we should embrace.</p><img src="https://counter.theconversation.com/content/51580/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Andrew Walker co-founded the website New Mandala.</span></em></p>Prime Minister Turnbull has signalled a desire to move away from a ‘publish or perish’ academic culture toward one that prioritises public impact and engagement. It’s a challenge scholars should embrace.Andrew Walker, Professor of Southeast Asian Studies, Australian National UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/508562015-12-02T19:25:37Z2015-12-02T19:25:37ZThere is no easy way to measure the impact of university research on society<figure><img src="https://images.theconversation.com/files/102942/original/image-20151124-18261-lw9how.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The iPhone is a good example of an entire industry built on the back of publicly funded research outcomes. The 'iPhone fish' is designed to teach people healthy eating through portion size control.</span> <span class="attribution"><a class="source" href="http://pictures.reuters.com/C.aspx?VP3=SearchResult&VBID=2C0BXZJPXACK2&SMLS=1&RW=1871&RH=1265#/SearchResult&VBID=2C0BXZJPXACK2&SMLS=1&RW=1871&RH=1265&POPUPPN=2&POPUPIID=2C0FQECOSD3P">Lucy Nicholson/Reuters</a></span></figcaption></figure><p>Prime Minister Malcolm Turnbull <a href="http://www.theage.com.au/federal-politics/political-news/academic-publications-to-become-less-important-when-funding-university-research-20151112-gkxkgl.html">is rumoured</a> to announce major changes to the way academic research is funded in his forthcoming innovation statement this month. </p>
<p>Media reports suggest the government wants academics to spend less time writing for journals and more time working with industry to ensure research has a commercial and community impact. It is suggested the aim is to strengthen and build ties between universities and industry and make sure all publicly funded research is contributing to society in some way.</p>
<p>Discussions about how to measure university research have been around for a long time. </p>
<p>Back in 2005, the then minister for education, Brendan Nelson, announced the Research Quality Framework (RQF) aimed at improving the assessment, quality and social impact of university research. </p>
<p>In 2007, the Labor government abandoned the RQF – which was based on using case studies – and focused on evaluating research quality under the banner of Excellence in Research for Australia (ERA). </p>
<p>While case studies have valid uses – such as showcasing real-world benefits of research – an independent body found that there are <a href="http://trove.nla.gov.au/work/27396497?selectedversion=NBD41572224">significant issues</a> in using case studies to measure the social and economic benefits across the research system.</p>
<p>Case studies can demonstrate impact, but <a href="https://www.escholar.manchester.ac.uk/uk-ac-man-scw:23935">can’t be used to measure impact</a> for these reasons:</p>
<ul>
<li><strong>Timing</strong> – research often has impacts long after the research is completed.</li>
<li><strong>Attribution</strong> – innovation is a team sport involving multiple projects, actors and inputs, often in isolation from each other.</li>
<li><strong>Appropriability</strong> – it can be difficult to know who benefits from research and account for the diverse impacts that can arise.</li>
<li><strong>Inequality</strong> – it is difficult to compare across different types of impacts. For example, how do you compare the <a href="https://en.wikipedia.org/wiki/Gardasil">development of Gardasil</a> with research on the effects of climate change in the <a href="http://csiropedia.csiro.au/display/CSIROpedia/Murray-Darling+Basin+Sustainable+Yields+Project">Murray-Darling Basin</a>?</li>
</ul>
<p>Without a broad sample of case studies from all universities it’s also difficult to use these as evidence of who should get funding, especially if the research we are talking about can <a href="http://jrs.sagepub.com/content/104/12/510.full">go back 10 or 20 years</a>. </p>
<p>Last year in the UK, almost 7,000 case studies were used in the latest round of the Research Excellence Framework (REF). REF 2014 included an impact assessment <a href="http://www.ref.ac.uk/panels/assessmentcriteriaandleveldefinitions/">worth 20% of the final rating</a>. </p>
<p>A <a href="http://www.hefce.ac.uk/pubs/rereports/Year/2015/analysisREFimpact/">recent report</a> has found that while case studies can show the benefits of research, based on the UK experience, it is unlikely that we will be able to develop specific metrics of impact. After all, like research itself, impact is a complex beast. </p>
<p>And so here we are, a decade on, still grappling with questions of why and how we should measure publicly funded research.</p>
<p>The government is now urging researchers to work more closely with the users of research – across the public, private and not-for-profit sectors. The idea is to help them innovate and so improve their competitiveness and enhance their products, processes and services. </p>
<p>At the same time, it is hoped that publicly funded research will stimulate innovative industries – and jobs – for the future. </p>
<p>The iPhone is one <a href="http://marianamazzucato.com/the-entrepreneurial-state/">example</a> of product built almost entirely on the back of publicly funded research. </p>
<h2>A good idea for Australian higher education?</h2>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/102946/original/image-20151124-18233-1i5q39h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/102946/original/image-20151124-18233-1i5q39h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=458&fit=crop&dpr=1 600w, https://images.theconversation.com/files/102946/original/image-20151124-18233-1i5q39h.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=458&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/102946/original/image-20151124-18233-1i5q39h.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=458&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/102946/original/image-20151124-18233-1i5q39h.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=576&fit=crop&dpr=1 754w, https://images.theconversation.com/files/102946/original/image-20151124-18233-1i5q39h.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=576&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/102946/original/image-20151124-18233-1i5q39h.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=576&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">The government wants academics to balance their effort writing for journals with time spent on research activities that deliver economic and societal impacts.</span>
<span class="attribution"><span class="source">www.shutterstock.com</span></span>
</figcaption>
</figure>
<p>At the moment, researchers are measured and recognised for the number of journal articles they have published and where they have published them. This has led to Australia being among the best research nations in the world. </p>
<p>But it does mean that while our universities are <a href="https://www.comlaw.gov.au/Details/C2013C00029/Html/Text#_Toc345666977">supposed to be doing a range of valuable tasks</a> – like educating and training students, creating and advancing knowledge and applying knowledge to solve the most pressing issues of our times – academic practices and institutional incentives support them for only one activity: getting grants to undertake research that can be published in journals, to get more grants, and so on.</p>
<p>If we want our universities to work with the community, governments and businesses to deliver greater prosperity and wellbeing, then measuring their research by publications alone is unlikely to fit the bill.</p>
<p>Something like the <a href="https://www.atse.org.au/atse/content/publications/reports/industry-innovation/research-engagement-for-australia.aspx">plan</a> by the Academy of Technological Science and Engineering (ATSE) is a good addition. What this does is measure where academics are working on research with groups outside the university sector based on their sources of research income. </p>
<h2>The main challenges</h2>
<p>But if such a plan is put in place, it’s important to be aware of a few things:</p>
<ol>
<li><p>We don’t want to focus all our university research on having an impact. We still need to do “basic research” – that is research that pushes knowledge forward for its own sake. This generates the insights that later get applied to specific problems. </p></li>
<li><p>Research and innovation involve multiple actors, often working in isolation from each other and across long time frames. It involves a lot of chance and trial and error; it is almost never a linear processes. There are <a href="http://www.csiro.au/en/About/Strategy-structure/Our-impact-framework/Ensuring-CSIRO-delivers-impact">activities that improve the likelihood of impact</a> occurring – such as engaging the public and making research widely available. Understanding the context that a researcher wants to deliver impact in (and how it is changing over time) is key to delivering profound and lasting impacts. </p></li>
<li><p>There is no silver bullet when it comes to measuring research, only partial measures and proxies. Done well, proxy measures relate directly to the underlying processes they are measuring – for example, on average, researchers who get more of their income from industry sources are doing research that is most relevant to those industries, and so income can be a proxy for industry relevance.
But a range of activities will be missed, such as research training or in-kind support. And so <a href="http://www.theaustralian.com.au/higher-education/atse-proposes-expert-panels-to-measure-research-impact/story-e6frgcjx-1227630202210">academic judgement is also needed</a> to make sense of the numbers.
As long as we understand the limits of proxy measures, we can anticipate and <a href="http://www.researchgate.net/publication/222687327_Explaining_Australia's_Increased_Share_of_ISI_PublicationsThe_Effects_of_a_Funding_Formula_Based_on_Publication_Counts">minimise perverse outcomes</a>. </p></li>
</ol>
<h2>How should we measure research?</h2>
<p>Given all of this, what we should be looking for is a multi-dimensional set of measures.</p>
<p>This means a robust measure of research excellence (such as ERA); measures of activities that increase the chances of research impact such as engagement with the public; and more than likely some measure of research training.</p>
<p>Missing any one of these means that we are missing out on the important benefits of university research:</p>
<ul>
<li>Research excellence without engaging with the public, industry and government means that we will not apply our world-class research to improving the prosperity of the public that funds it; </li>
<li>Engagement in the absence of excellence means that we would not be delivering the best outcomes; </li>
<li>Research training builds a pool of qualified researchers who can work within the public and private sectors, and who can receive and apply the lessons of research to maximise its benefits. </li>
</ul>
<p>It is our investment in research as a society that will give us the best chance of solving the pressing issues of our times – inequality, climate change, food security, global conflict – so there is a lot at stake. </p>
<p>A narrow focus on academic publishing has helped to create a world-leading university sector. </p>
<p>The pressing issue now is how to develop a system that can apply this research to improve the lives of the public.</p>
<p><em>This article is part of our series <strong><a href="https://theconversation.com/au/topics/why-innovation-matters">Why innovation matters</a></strong>. Look out for more articles on the topic in the coming days.</em></p><img src="https://counter.theconversation.com/content/50856/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Tim Cahill consults for the Academy of Technological Sciences and Engineering on the Research Engagement for Australia project. He is currently consulting to The Conversation on research engagement metrics.</span></em></p><p class="fine-print"><em><span>Mark Bazzacco is affiliated with CSIRO. </span></em></p>Publicly-funded research should contribute to society in some way. But we need to think carefully about how we create a system that allows us to measure the impact of research.Tim Cahill, Adjunct research fellow, Swinburne University of TechnologyMark Bazzacco, Executive Manager, Planning & Performance, CSIROLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/484372015-10-23T03:13:18Z2015-10-23T03:13:18ZAcademics must still ‘publish or perish’ under revamped research funding policy<figure><img src="https://images.theconversation.com/files/98844/original/image-20151019-23264-1gofhqg.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Academics are under enormous pressure to publish prolifically because this generates subsidies for their universities.</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>The South African government’s research funding policy has long been <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3721321/">criticised</a> by academics. The policy has three major weaknesses:</p>
<ol>
<li><p>it offers incentives for output quantity and productivity rather than research quality;</p></li>
<li><p>it promotes the practice of dividing research outcomes between articles, thus diluting the impact of the research; and</p></li>
<li><p>it penalises work done in large collaborative projects.</p></li>
</ol>
<p>Now South Africa’s Department of Higher Education and Training has approved a revised version of the contentious <a href="http://www.dhet.gov.za/Policy%20and%20Development%20Support/Policy%20and%20procedures%20for%20measurement%20of%20Research%20output%20of%20Public%20Higher%20Education%20Institutions.pdf">2003 policy</a>. The <a href="http://www.dhet.gov.za/Policy%20and%20Development%20Support/Research%20Outputs%20policy%20gazette%202015.pdf">Research Outputs Policy 2015</a> comes into effect from January 1, 2016. It has been <a href="https://theconversation.com/financial-reward-for-research-output-under-the-spotlight-in-south-africa-45567">welcomed</a> by some academics. They believe it has the potential to introduce considerable changes in how research output funds are awarded.</p>
<p>But will the apparently “new” policy actually just be more of the same?</p>
<h2>The status quo</h2>
<p>At the moment, a considerable portion of government funding to South African universities is allocated to an individual institution based on its academics’ annual research outputs. These outputs, collected under the umbrella of “publication units”, include peer-reviewed articles, books, book chapters and conference proceedings. </p>
<p>Academics have complained that university management exploits the policy and uses it as a tool for accumulating funds. Some universities use the number of publication units as a <a href="http://www.saip.org.za/images/stories/documents/PhysicsComment/pcmarch2015.pdf">key indicator</a> when measuring academic staff members’ performance.</p>
<p>This drives a dangerous culture of “publish or perish” that makes the quantity of research output far more important to an individual academic’s career track than the quality of their work.</p>
<p>The existing policy also doesn’t encourage collaboration. That’s a big problem for scientists who participate in “big science” projects like the European Organisation for Nuclear Research (<a href="http://international-relations.web.cern.ch/International-Relations/nms/southafrica.html">SA-CERN</a>) or the Square Kilometre Array (<a href="http://www.ska.ac.za/">SKA</a>). Hundreds of authors feature on the publications that emerge from these collaborations. But these outputs earn zero subsidies under the government’s existing metric.</p>
<h2>Positive steps</h2>
<p>The updated research policy introduces some important improvements. For instance, it changes the way that scholarly books are subsidised. These books counted for five publication units under the 2003 policy; they are now worth ten units. The humanities and social sciences will mostly benefit from this.</p>
<p>A particularly positive aspect of the 2015 policy is that it highlights research integrity. This is a long overdue response to the “publish or perish” culture, not only in South Africa but elsewhere in the world. <a href="http://www.theguardian.com/science/2011/sep/05/publish-perish-peer-review-science">Academic dishonesty</a> and <a href="http://www.theguardian.com/science/2011/sep/05/publish-perish-peer-review-science">plagiarism</a> increase when quantity is more profitable than quality.</p>
<p>In an important step, universities will be expected to take full ownership when it comes to protecting research integrity. Institutions must establish a Research Integrity Committee to ensure they become compliant with the integrity issues raised by the policy. The practice of dividing research outcomes between articles is also strongly discouraged, for the reason that it undermines the integrity of scholarship.</p>
<p>The policy, in its own words:</p>
<blockquote>
<p>… aims to support and encourage scholarship. Institutions and academics must remember the importance of research integrity when submitting their claims and are urged to focus on quality research and not maximum accrual of subsidy funds.</p>
</blockquote>
<p>In all, the word “quality” features 12 times in the 2015 policy – a big jump from the 3 mentions it received in the 2003 version.</p>
<p>So, the policy talks the “quality” talk. But does it walk the walk? Sadly not.</p>
<h2>Big changes are absent</h2>
<p>In October 2013, a committee set up by the higher education and training minister released its list of <a href="http://www.dhet.gov.za/SiteAssets/Latest%20News/Report%20of%20the%20Ministerial%20Committee%20for%20the%20Review%20of%20the%20Funding%20of%20Universities.pdf">recommendations</a> about university funding. </p>
<p>How many of the committee’s major recommendations have been implemented in this new policy? Unfortunately almost none.</p>
<p>The committee suggested that a new formula ought to be introduced for calculating accredited publication units. This, it said, should take into account the scientific impact of a publication in terms of citation indexes, journal impact factors and publishers’ rankings. The new policy mentions this recommendation in passing – but as a throwaway line, something that should be discussed some time in the future. </p>
<p>There is also no mention of special rules aimed at incentivising academics’ involvement in international collaborations. For instance, if 15 South African authors are involved in a paper about a big science project, why not split the subsidy between these 15 and their institutions? That they are working with international academics shouldn’t stop local authors or their institutions from being rewarded by the South African government.</p>
<p>Most academics across a range of disciplines choose to disseminate their results through subsidised journal articles. The new policy has made no changes to the funding schemes for such articles, so it’s very likely that academics will feel forced to continue choosing quantity over quality.</p>
<h2>Aligning policies</h2>
<p>The Department of Higher Education and Training’s policy needs a far wider revision. This must be carried out in consultation with all the relevant stakeholders.</p>
<p>The focus must be drifted away from the publication units metric. Instead, it must reward quality and encourage South African academics to get involved in more international collaborations, knowing that they will be subsidised for this sort of “big science”.</p>
<p>Finally, the government policy needs to be aligned with the country’s National Research Foundation <a href="http://www.nrf.ac.za/rating">metrics</a>. These are based entirely on the quality and impact of research. This will grant South African scholars the full and rounded support they need as they strive for quality, impactful research to support a knowledge-based economy.</p><img src="https://counter.theconversation.com/content/48437/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Emanuela Carleschi receives funding from the National Research Foundation of South Africa and the University of Johannesburg. </span></em></p>A new policy on research outputs and funding will be introduced in South Africa in 2016. But it leaves too much unchanged from the old policy.Emanuela Carleschi, Senior Lecturer in Condensed Matter Physics, University of JohannesburgLicensed as Creative Commons – attribution, no derivatives.