tag:theconversation.com,2011:/fr/topics/student-surveys-14375/articlesStudent surveys – The Conversation2022-07-18T20:06:23Ztag:theconversation.com,2011:article/1850412022-07-18T20:06:23Z2022-07-18T20:06:23ZWe have developed a way to screen student feedback to ensure it’s useful, not abusive (and academics don’t have to burn it)<figure><img src="https://images.theconversation.com/files/474471/original/file-20220718-14-lunsaw.jpg?ixlib=rb-1.1.0&rect=10%2C26%2C3540%2C2338&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Holland Taylor as Professor Joan Hambling in The Chair.</span> <span class="attribution"><span class="source">Elize Morse/Netflix</span></span></figcaption></figure><p>This week, many Australian universities will be sending academics the results of the first semester student evaluation surveys. </p>
<p>For some this will be a worrying and unpleasant time. The comments university students make anonymously in their teaching evaluations can leave academics feeling <a href="https://theconversation.com/lose-some-weight-stupid-old-hag-universities-should-no-longer-ask-students-for-anonymous-feedback-on-their-teachers-173911">fearful</a>, <a href="https://theconversation.com/read-the-student-survey-responses-shared-by-academics-and-youll-see-why-professor-hambling-is-justified-in-burning-hers-167897">distressed</a> and <a href="https://theconversation.com/our-uni-teachers-were-already-among-the-worlds-most-stressed-covid-and-student-feedback-have-just-made-things-worse-162612">demoralised</a>. </p>
<p>And with good reason. As a 2021 <a href="https://www.tandfonline.com/doi/abs/10.1080/02602938.2021.2012643?journalCode=caeh20">survey</a> of Australian academics and their experiences of student feedback found: </p>
<blockquote>
<p>Personally destructive, defamatory, abusive and hurtful comments were commonly reported.</p>
</blockquote>
<p>Hurtful or abusive comments can remain permanently on record as a <a href="https://www.tandfonline.com/doi/full/10.1080/02602938.2021.1888075">measure of performance</a>. These records can affect applications for promotion or for secure continued employment. </p>
<p>The authors of the 2021 survey, led by Richard Lakeman at Southern Cross University have been among those <a href="https://theconversation.com/lose-some-weight-stupid-old-hag-universities-should-no-longer-ask-students-for-anonymous-feedback-on-their-teachers-173911">calling for</a> anonymous online surveys to be scrapped. Some academics, burned by their experience of student feedback, say they no longer <a href="https://www.tandfonline.com/doi/full/10.1080/02602938.2020.1805409">open or engage</a> with student evaluation reports. They said the risk of harm outweighed any benefits. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/lose-some-weight-stupid-old-hag-universities-should-no-longer-ask-students-for-anonymous-feedback-on-their-teachers-173911">'Lose some weight', 'stupid old hag': universities should no longer ask students for anonymous feedback on their teachers</a>
</strong>
</em>
</p>
<hr>
<p>In the Netflix show The Chair, a memorable scene sees the character Professor Joan Hambling <a href="https://theconversation.com/read-the-student-survey-responses-shared-by-academics-and-youll-see-why-professor-hambling-is-justified-in-burning-hers-167897">burn</a> her student evaluations. Clearly, a different solution is needed. </p>
<p>Feedback from students can still be valuable for lifting teaching standards and it’s important students have their say. </p>
<p>We have developed a <a href="https://www.tandfonline.com/doi/full/10.1080/02602938.2022.2081668">screening system</a> using <a href="https://theconversation.com/machine-learning-is-changing-our-culture-try-this-text-altering-tool-to-see-how-159430">machine learning</a> (where software changes its behaviour by “learning” from user input) that allows students to talk about their experiences while protecting academics from unacceptable comments.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/read-the-student-survey-responses-shared-by-academics-and-youll-see-why-professor-hambling-is-justified-in-burning-hers-167897">Read the student survey responses shared by academics and you'll see why Professor Hambling is justified in burning hers</a>
</strong>
</em>
</p>
<hr>
<h2>Why a new approach is needed</h2>
<p>University codes of conduct remind students of their general obligation to refrain from abusive or discriminatory behaviour, but not specifically in regard to student evaluations.</p>
<p>Instead, universities rely on self-regulation or on others to report incidents. Some institutions use profanity blockers to screen comments. Even then, these often fail to detect emerging terms of abuse in online speech. </p>
<p>So, in setting up our screening system, we wanted to:</p>
<ul>
<li>promote staff and student well-being</li>
<li>enhance the reliability and validity of student feedback</li>
<li>improve confidence in the integrity of survey results.</li>
</ul>
<p>We developed a method using machine learning and a dictionary of terms to screen for unacceptable student comments. The dictionary was created by QUT drawing on historically identified unacceptable comments and incorporating prior research into abusive and discriminatory terms. </p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1480282414760955904"}"></div></p>
<h2>Our ‘Screenomatic’ solution</h2>
<p>There is not a lot of published work on the detection of unacceptable or abusive comments in student evaluation surveys. So our team adapted <a href="https://link.springer.com/article/10.1007/s10115-020-01481-0">earlier research</a> on detecting misogynistic tweets. This worked because often the student comments we looked at were similar in length to a tweet’s 280-character limit. </p>
<p>Our approach, which we call “<a href="https://www.tandfonline.com/doi/full/10.1080/02602938.2022.2081668">Screenomatic</a>”, automatically reviewed more than 100,000 student comments during 2021 and identified those that appeared to be abuse. Trained evaluation staff members manually reviewed about 7,000 flagged comments, updating the machine-learning model after each semester. Each update improves the accuracy of auto-detection.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/gender-bias-in-student-surveys-on-teaching-increased-with-remote-learning-what-can-unis-do-to-ensure-a-fair-go-for-female-staff-178418">Gender bias in student surveys on teaching increased with remote learning. What can unis do to ensure a fair go for female staff?</a>
</strong>
</em>
</p>
<hr>
<p>Ultimately, 100 comments were removed before the results were released to educators and supervisors. University policy enables comments to be re-identified in cases of potential misconduct. The central evaluations team contacted these students and reminded them of their obligations under the code of conduct.</p>
<p>The <a href="https://www.tandfonline.com/doi/full/10.1080/02602938.2022.2081668">Screenomatic</a> model can help protect both educators and students. Staff are safeguarded from abuse, and students at risk – who make comments that indicate they need mental health help, include allegations of bullying or harassment, or that threaten staff or other students – can be offered support. Universities can share data to train the model and maintain currency. </p>
<p>Importantly, the process enables universities to act morally to harness student voices while protecting people’s well-being. </p>
<h2>Useful feedback, not abuse</h2>
<p>The number of educators who receive abusive feedback may be relatively <a href="https://link.springer.com/article/10.1007/s10734-014-9716-2">small</a>. However, it’s still unacceptable for universities to continue to expose their staff to offensive comments in the full knowledge of their potential impact. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/our-uni-teachers-were-already-among-the-worlds-most-stressed-covid-and-student-feedback-have-just-made-things-worse-162612">Our uni teachers were already among the world's most stressed. COVID and student feedback have just made things worse</a>
</strong>
</em>
</p>
<hr>
<p>With last year’s High Court ruling on <a href="https://www.nfplaw.org.au/news/new-high-court-decision-affects-all-not-for-profit-organisations-that-use-social-media">liability for defamatory posts</a>, and attempts to <a href="https://www.ag.gov.au/legal-system/social-media-anti-trolling-bill">improve online safety</a>, there is a growing acknowledgement that people should not be able to post anonymous, harmful messages.</p>
<p>After all, the cost of screening responses is nothing compared to the cost to individuals (including mental health or career consequences). And that’s ignoring the potential costs of litigation and legal damages. </p>
<p>At the end of the day, the anonymous comments are read by real people. As a tweet in response to the <a href="https://doi.org/10.1080/02602938.2021.2012643">Lakeman findings</a> noted:</p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1475521673407389701"}"></div></p>
<p>The <a href="https://www.tandfonline.com/doi/full/10.1080/02602938.2022.2081668">Screenomatic model</a> goes a long way towards enabling the “tons of useful feedback” to serve its intended purpose while ensuring people aren’t harmed in the process.</p><img src="https://counter.theconversation.com/content/185041/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Abby Cathcart leads the QUT Evaluation Strategy</span></em></p><p class="fine-print"><em><span>Melinda Laundon and Sam Cunningham do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>The new ‘Screenomatic’ model can protect students and academics, while still providing useful feedback.Abby Cathcart, Professor of Higher Education & Governance, Queensland University of TechnologyMelinda Laundon, Postdoctoral Research Fellow, School of Management, Queensland University of TechnologySam Cunningham, Lecturer, QUT Academy of Learning & Teaching, Queensland University of TechnologyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1771552022-02-22T19:09:06Z2022-02-22T19:09:06ZYes, uni students say some awful things in teaching surveys, so how can we use them to improve?<figure><img src="https://images.theconversation.com/files/447169/original/file-20220217-17-1hadj1i.jpg?ixlib=rb-1.1.0&rect=1745%2C1358%2C4407%2C2774&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>Imagine some of the key evidence for promotions at work being <a href="https://theconversation.com/lose-some-weight-stupid-old-hag-universities-should-no-longer-ask-students-for-anonymous-feedback-on-their-teachers-173911">anonymous responses</a> from coworkers who just received a bad performance evaluation from you. Something similar happens in higher education, with teachers rated by students grateful for good grades or disgruntled by low grades. That’s a bitter pill to swallow for <a href="https://theconversation.com/read-the-student-survey-responses-shared-by-academics-and-youll-see-why-professor-hambling-is-justified-in-burning-hers-167897">some academics</a>. </p>
<p><a href="https://doi.org/10.1080/02602938.2017.1416456">Evidence</a> tells us students take their feedback personally. <a href="https://doi.org/10.1080/20445911.2021.1923723">Jurors’ decision-making</a> is similarly affected by their emotional state. People make worse decisions when they are <a href="https://psycnet.apa.org/doi/10.1037/0022-3514.81.6.973">uncertain</a> or <a href="https://doi.org/10.1016/j.bbr.2013.04.046">stressed</a>, which are two common states for students.</p>
<p>So how unreliable are student evaluations? And what can we do about it? <a href="https://doi.org/10.53761/1.18.8.1">Our work</a> indicates there is still much to be done in this space, but we can set some rules to make it easier.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/lose-some-weight-stupid-old-hag-universities-should-no-longer-ask-students-for-anonymous-feedback-on-their-teachers-173911">'Lose some weight', 'stupid old hag': universities should no longer ask students for anonymous feedback on their teachers</a>
</strong>
</em>
</p>
<hr>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1493742260957368320"}"></div></p>
<h2>All surveys are not equal</h2>
<p>Australia’s national <a href="https://www.qilt.edu.au/docs/default-source/default-document-library/2020-ses-national-report.pdf?sfvrsn=d1237953_5">Student Experience Survey</a> is considered “the pulse” on student satisfaction rather than a device to enable teacher growth, with the data being easily skewable by circumstances at the time. Unsurprisingly, during 2020, universities that already had an online presence saw the smallest decline in student experience scores. </p>
<p>So the question becomes: did the quality of learning crash in Group of Eight universities, which had the greatest declines in student experience? Unlikely. Instead, students’ ratings reflected their difficulties engaging with new forms of teaching and learning, plus the inertia of COVID-19 lockdowns. </p>
<p>Maybe they should have <a href="https://doi.org/10.1080%2F00986280701700318">given students chocolate</a>?</p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"452161822330593280"}"></div></p>
<p>The reality is these surveys do not tell us how students learn, but instead how students perceive their learning. Yet students <a href="https://doi.org/10.1016/j.jarmac.2019.12.009">aren’t experts at what learning is</a>. And when students don’t receive effective training in evaluation, it’s hardly a surprise that teacher <a href="https://doi.org/10.1080/02602938.2021.1888075">gender, race and attractiveness</a> change scores. As the popular allegory puts it: </p>
<blockquote>
<p>“Everyone is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid.” </p>
</blockquote>
<p>Instead, let’s ask students to share the most enjoyable content, the most rewarding educational technologies, and where improvement was needed. Include ethics and feedback training for bonus credit.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/our-uni-teachers-were-already-among-the-worlds-most-stressed-covid-and-student-feedback-have-just-made-things-worse-162612">Our uni teachers were already among the world's most stressed. COVID and student feedback have just made things worse</a>
</strong>
</em>
</p>
<hr>
<h2>Making survey tools that work</h2>
<p>Psychometrics is the study of measurements. Interestingly, many academics have specialist knowledge in developing surveys that are designed to be valid and reliable. But it’s unclear if universities use them as a resource to develop their surveys, with some academics wondering if they should. The <a href="https://www.qilt.edu.au/docs/default-source/default-document-library/2021-ess-methodological-report.pdf?sfvrsn=db6efd6b_0">2021 Employer Satisfaction Survey Methodological Report</a>, for example, does not refer explicitly to the words validity or reliability once across its 140 pages.</p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1493334209141821440"}"></div></p>
<p>Valid surveys exist when the questions align to what we think they are measuring. Using a stopwatch to measure time is easy. When we try to decide how we feel about intangible concepts, it’s harder. </p>
<p>The national <a href="https://www.qilt.edu.au/docs/default-source/default-document-library/2020-ses-national-report.pdf?sfvrsn=d1237953_5">Student Experience Survey</a> asks students whether they have developed a sense of belonging to their institution. Yet the <a href="https://doi.org/10.1037/0033-2909.117.3.497">evidence on belonging</a> indicates it is typically developed through interpersonal relationships, not institutions, and not <a href="https://doi.org/10.53761/1.18.4.2">through universities</a>. </p>
<p>Reliable surveys exist when the questions generate consistent results over time and over different participants. It’s analogous to when we bake a cake and we assume the scales will always accurately measure 40 grams of butter. </p>
<p>Speaking of sweets, scores in student surveys are <a href="https://doi.org/10.1080%2F00986280701700318">easy to game</a>. <a href="https://doi.org/10.1080/01973533.2020.1756817">Inflating student grades</a> does the trick. </p>
<p>In contrast, as an example, the Australian <a href="https://www.qilt.edu.au/docs/default-source/default-document-library/2020-ses-national-report.pdf?sfvrsn=d1237953_5">Student Experience Survey</a> asks whether students have developed their critical thinking skills during their course. How accurately can a person with low critical thinking skills answer this question? </p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1447106074046308356"}"></div></p>
<h2>5 rules for surveys to help teachers improve</h2>
<p>There are ways that surveys can be used for good. To actually help teachers be better educators and improve student learning. But it requires a reset. </p>
<p>Here are five rules institutions could consider when developing their surveys.</p>
<p><strong>1. Find psychometric specialists to create quality tools</strong></p>
<p>We go to dentists to have our teeth fixed. The same rule applies here. Find individuals who can take the <a href="https://search.informit.org/doi/abs/10.3316/INFORMIT.491551710186460">theory</a> of <a href="https://doi.org/10.1016/j.leaqua.2018.07.001">scale development</a> (producing reliable and valid measures to assess an attribute of interest) into the practices of learning and teaching.</p>
<p><strong>2. Change when the survey is done</strong></p>
<p>Lots of <a href="https://doi.org/10.1002/hrdq.1099">evaluations</a> are done before, during and after a program. In higher education, they are completed only after the class has ended. </p>
<p>A change to evaluations at multiple points will help identify if the learner makes progress during the class. This would also help control for cohort problems (one year, for example, students are smarter). </p>
<p>For student experience, contrasting how the same student rates different classes each semester may serve as a stable measure to see which classes need review. </p>
<p><strong>3. Use more than just numbers</strong></p>
<p>The numbers explain how we are tracking, and this is not inherently bad. The qualitative comments (<a href="https://theconversation.com/lose-some-weight-stupid-old-hag-universities-should-no-longer-ask-students-for-anonymous-feedback-on-their-teachers-173911">mostly</a>) help us explore what those mean. <a href="https://doi.org/10.1016/j.evalprogplan.2016.01.001">Mixed methods</a> approaches can help. </p>
<p><strong>4. Control for bias</strong></p>
<p>It’s not always possible to eliminate bias and emotion. We can seek to understand them and use the measures as a case-by-case conversation about improving teaching. Developing reliable and valid tools will help, but if the aim is for these to help teachers improve, then we need to focus on that, not cross-institutional comparisons. </p>
<p>Better yet, let’s actively recognise teachers’ professional growth, call decline into question, and report on averages. </p>
<p>We can also train students to be better evaluators. </p>
<p><strong>5. Create a growth community</strong></p>
<p>Teaching quality surveys do not necessarily increase <a href="https://doi.org/10.1080/0260293022000009294">teaching quality</a>, but they can. </p>
<p>The surveys offer an opportunity to raise awareness of differences. If students rate seven items at 90% but one is 84%, this should prompt research into the reasons. It could be a great opportunity to create more meaningful content; it could also just be an <a href="https://en.wikipedia.org/wiki/Outlier">outlier</a>. </p>
<p>Use these findings as <a href="https://doi.org/10.53761/1.18.8.1">publishing opportunities</a> to share what was learned.</p>
<hr>
<p><em>Correction: This article has been updated to remove an erroneous attribution of the allegory. It is often attributed to Albert Einstein, but there is <a href="https://quoteinvestigator.com/2013/04/06/fish-climb/">no evidence</a> that he said it.</em></p><img src="https://counter.theconversation.com/content/177155/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Students’ evaluations can be very personal and many factors can skew survey responses. But surveys can be designed to produce more valid and reliable results that could be used to improve teaching.Joseph Crawford, Senior Lecturer, Educational Innovation, University of TasmaniaKristyn Harman, Associate Professor in History, University of TasmaniaMichael Cowling, Associate Professor – Information & Communication Technology (ICT), CQUniversity AustraliaLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1678972021-10-07T19:06:19Z2021-10-07T19:06:19ZRead the student survey responses shared by academics and you’ll see why Professor Hambling is justified in burning hers<figure><img src="https://images.theconversation.com/files/424617/original/file-20211005-15-1l9r1ul.jpeg?ixlib=rb-1.1.0&rect=0%2C8%2C5897%2C3916&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><span class="source">Eliza Morse/Netflix</span></span></figcaption></figure><p>If you’ve watched the Netflix sitcom <a href="https://www.netflix.com/au/title/81206259">The Chair</a> you’ll remember the scene in which Professor Joan Hambling burns her student evaluations, after admitting she hadn’t read any of them since the 1980s. Many of us in academia whooped in delight when Professor Hambling lit that match. </p>
<p>We know exactly how she feels. For LGBTIQ+ people in particular, student experience or satisfaction surveys can be a source of distress as they provide students with an anonymous means to discriminate against and harass queer academics. At times, these surveys are little better than university-facilitated hate speech. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/our-uni-teachers-were-already-among-the-worlds-most-stressed-covid-and-student-feedback-have-just-made-things-worse-162612">Our uni teachers were already among the world's most stressed. COVID and student feedback have just made things worse</a>
</strong>
</em>
</p>
<hr>
<h2>An unreliable guide to teaching quality</h2>
<p>Adding salt to the wound is that universities then use these surveys to assess academics’ teaching performance, despite growing evidence they are not fit for this purpose. The University of New South Wales has even proposed to <a href="https://www.smh.com.au/national/what-goes-wrong-when-uni-students-mark-their-teachers-20210831-p58nk0.html">publish these survey results</a>.</p>
<p><a href="https://www.tandfonline.com/doi/full/10.1080/2331186X.2017.1304016">Research</a> <a href="https://www.tandfonline.com/doi/full/10.1080/2331186X.2017.1304016">shows</a> student evaluations of teaching are not accurate measures of teaching effectiveness. Other <a href="https://www.tandfonline.com/doi/abs/10.1080/0260293022000009294">research</a> shows these surveys do not lead to higher teaching quality or better learning outcomes and are not trusted by students as a means of giving them a voice. In contrast, such surveys are linked to <a href="https://www.tandfonline.com/doi/full/10.1080/01973533.2020.1756817">poorer teaching, grade inflation</a> and to <a href="https://www.researchgate.net/publication/349864729_Sexism_racism_prejudice_and_bias_a_literature_review_and_synthesis_of_research_surrounding_student_evaluations_of_courses_and_teaching">racism, sexism and homophobia</a>. </p>
<p>A <a href="https://www.tandfonline.com/doi/full/10.1080/01973533.2020.1756817">number</a> of <a href="https://www.tandfonline.com/doi/abs/10.1080/13562510601102131">studies</a> have shown grade satisfaction is a major factor in survey results – the higher the student’s grades the better the feedback they give. Students at prestigious universities are also <a href="https://www.researchgate.net/publication/283298711_Designing_a_predictive_model_of_student_satisfaction_in_online_learning">more likely to positively rate their lecturers</a> because the university and its courses are seen as “world class”. Most damningly, student evaluations are often <a href="https://www.researchgate.net/publication/349864729_Sexism_racism_prejudice_and_bias_a_literature_review_and_synthesis_of_research_surrounding_student_evaluations_of_courses_and_teaching">little more than veiled bias</a> about their lecturer’s personal traits, especially gender, race and sexuality.</p>
<figure>
<iframe width="440" height="260" src="https://www.youtube.com/embed/qF1JPTdS5pA?wmode=transparent&start=0" frameborder="0" allowfullscreen=""></iframe>
<figcaption><span class="caption">Despite what she tells the chair of the department, Professor Joan Hambling has resisted reading her student evaluations.</span></figcaption>
</figure>
<h2>Sharing the best and worst feedback</h2>
<p>I recently asked a dozen academics from universities across Australia to share their worst and best student feedback stories. A common thread in these stories was students using the surveys to voice homophobic and transphobic sentiment. These are real student responses to questions about teaching quality:</p>
<blockquote>
<p>I couldn’t concentrate because I couldn’t tell if the teacher was a man or woman.</p>
<p>I found it extremely frustrating that a lot of examples and theories all revolved around sexuality/gender/identification and how it affects him. Speaking to a number of students in this topic, a lot of us felt like it was over the top.</p>
<p>This lecturer has no empathy for students not supporting the LGBTQ ideology.</p>
<p>She looks like a man professor not a woman one.</p>
<p>He made me uncomfortable because gays and lesbianism are against my religion.</p>
<p>There are only two genders, men and women!</p>
</blockquote>
<p>Some other comments were so offensive they were unpublishable. </p>
<p>There was also a strong thread of sexism. <a href="https://www.ofce.sciences-po.fr/pdf/dtravail/WP2015-13.pdf">Research</a> shows women receive lower ratings than male academics for doing the same thing. Women academics were judged harshly for being feminist or not conforming to stereotypical gender norms. One academic copped abuse for both in a single comment:</p>
<blockquote>
<p>Question: Do you have any other comments to add about this teacher in this unit? Answer: You look like 13 year old boy but the brain of a woman power bullshit and your (sic) a germ.</p>
</blockquote>
<p>The academic in question had a short, Pixie-style haircut at the time. Here we have the student’s perception of her gender non-conformity negatively impacting the academic’s teaching quality score.</p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1095775692996763649"}"></div></p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/male-teachers-are-most-likely-to-rate-highly-in-university-student-feedback-111741">Male teachers are most likely to rate highly in university student feedback</a>
</strong>
</em>
</p>
<hr>
<p>These surveys provide two forms of so-called data, a numeric score and qualitative data in the form of student comments. To assess teaching performance, or to decide if an academic will be appointed or promoted, the numeric score alone is normally used. This means an academic given a poor score accompanied by a discriminatory comment is being evaluated without proper context. </p>
<p>That being said, neither the numeric score nor the comments necessarily reveal the student’s true motivation for the feedback. Students are discouraged from openly venting their racism, homophobia and sexism but this does not mean their attitudes change. They are just cleverer about how they express it. Anonymous surveys enable them to rate an academic harshly without having to justify the rating or say why.</p>
<h2>Many responses have nothing to do with teaching</h2>
<p><a href="https://www.tandfonline.com/doi/abs/10.1080/02602938.2019.1640863">Research</a> also shows students are often not even answering the question they are asked, as the comments above show. They often base their scores and comments – both positive and negative – on things outside the classroom and beyond the academic’s control. Here are some examples:</p>
<blockquote>
<p>It would’ve been nice not to have to miss so many classes due to public holidays due to the classes being on a Monday.</p>
<p>Library access sometimes confusing - not everything available online.</p>
<p>IT help at this university is terrible, nothing ever works how it should and they never fix it.</p>
</blockquote>
<p>One academic I contacted received a positive score and comment because of her wardrobe:</p>
<blockquote>
<p>Question: What was good about the course?
Student comment: I like your shirts 😊</p>
</blockquote>
<p>Another academic received a low teaching quality score because the classroom did not have a nice view and the student found that depressing. </p>
<p><div data-react-class="Tweet" data-react-props="{"tweetId":"1381510610790285312"}"></div></p>
<p>Although academics generally value and respect their students, it would be foolish to pretend that as a group they will give objective feedback with the sole aim of improving teaching. About <a href="https://www.tandfonline.com/doi/full/10.1080/03075079.2021.1972093">one in ten students</a> routinely cheats on their assessments. <a href="https://www.emerald.com/insight/content/doi/10.1108/HE-05-2019-0023/full/html">Half of British university students</a> experience assault and harassment on campus from other students. Another <a href="https://link.springer.com/article/10.1007/s10734-008-9172-y">UK study</a> showed close to a quarter of LGBTIQ students had been a victim of homophobic harassment or discrimination, including threats of physical violence, at university.</p>
<p>Most students are good people, but enough harbour sexist, racist and homophobic views to distort survey outcomes.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/sexual-abuse-harassment-and-discrimination-rife-among-australian-academics-97856">Sexual abuse, harassment and discrimination 'rife' among Australian academics</a>
</strong>
</em>
</p>
<hr>
<h2>What are the impacts on academics?</h2>
<p>Having positions of relative authority in the university system does not make LGBTIQ academics immune to <a href="https://theconversation.com/sexual-abuse-harassment-and-discrimination-rife-among-australian-academics-97856">homophobia on campus</a>. If anything, they may feel like they have targets on their backs that force some back into the closet. Giving students an anonymous means to vent their bias and purposely harm academics’ careers and well-being just makes things worse.</p>
<p>Foregrounding student evaluations of teaching over other ways of assessing teaching performance — such as peer review and actual student learning outcomes — also leads some academics from vulnerable communities to self-censor in classes. Some queer academics, especially those on precarious casual contracts, try to be “less queer”. One non-binary academic adopted a “cisgender-friendly way of dressing” for the classroom after student comments. Having to wear more normative clothing made the academic feel they were “in a form of prison, wearing an inmate’s uniform”. </p>
<p>Obviously, having to hide who we are is not conducive to a productive teaching environment nor to our well-being.</p>
<p>Furthermore, for surveys to be statistically relevant and represent the majority attitudes of any given class the response rates need to be at 60% or higher – a benchmark routinely expected of survey data. Often students participate in these surveys at much lower rates. These low rates give a louder voice to those who wish to use the surveys to punish academics for their non-conformity to hetero-patriarchal values.</p>
<p>We already have better ways of assessing teaching quality and student learning, and ensuring those processes are authentic and fair. They’re called assessment outcomes. </p>
<p>In contrast, student evaluations of teaching are not fit for purpose and commonly discriminate against LGBTIQ+ and women academics. Perhaps Professor Hambling had valid reasons for burning her student feedback evaluations.</p><img src="https://counter.theconversation.com/content/167897/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Pema Düddul does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Student experience or satisfaction surveys are not a reliable guide to teaching performance. Even worse, anonymous survey responses are at times little better than university-facilitated hate speech.Pema Düddul, Associate Professor in Writing and Publishing, University of Southern QueenslandLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1626122021-07-18T20:04:48Z2021-07-18T20:04:48ZOur uni teachers were already among the world’s most stressed. COVID and student feedback have just made things worse<figure><img src="https://images.theconversation.com/files/411127/original/file-20210713-15-2odvfa.jpeg?ixlib=rb-1.1.0&rect=0%2C47%2C4560%2C3233&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/young-tired-female-professor-having-headache-1950069691">Shutterstock</a></span></figcaption></figure><p>Australia’s higher education workforce has literally <a href="https://www.theguardian.com/australia-news/2021/feb/03/more-than-17000-jobs-lost-at-australian-universities-during-covid-pandemic">been decimated</a> during the COVID-19 pandemic. Mass forced <a href="https://theconversation.com/big-spending-recovery-budget-leaves-universities-out-in-the-cold-160439">redundancies and non-renewal of casual contracts</a> were highly stressful. And now some disciplines and academics who committed their lives to teaching feel publicly invalidated as unnecessary in the reconstruction of the sector to produce what the government deems to be “<a href="https://theconversation.com/3-flaws-in-job-ready-graduates-package-will-add-to-the-turmoil-in-australian-higher-education-147740">job-ready graduates</a>”.</p>
<p>Our <a href="https://www.tandfonline.com/doi/abs/10.1080/1360080X.2021.1934246?journalCode=cjhe20">recent review</a> finds academics in Australia and New Zealand were suffering high levels of occupational stress well before COVID-19. Recent upheavals only added to existing problems. This is likely to jeopardise recruitment and retention of staff even in the very areas, such as <a href="https://www.abc.net.au/news/2020-07-18/healthcare-social-services-stable-jobs-after-coronavirus/12462670">health, teaching and medicine</a>, where the government expects high future demand.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/the-2021-22-budget-has-added-salt-to-universities-covid-wounds-160862">The 2021-22 budget has added salt to universities' COVID wounds</a>
</strong>
</em>
</p>
<hr>
<p>Our research team members are now turning their attention to the <a href="https://scuau.qualtrics.com/jfe/form/SV_6XWZoavOraIaZts">impacts of anonymous student feedback</a> on academics’ well-being. Preliminary findings suggest it’s having extreme impacts on the mental health of some of the workforce that remains, especially early career academics. We are also investigating their perceptions of the impacts of this feedback on teaching quality and academic standards. </p>
<h2>What are the main sources of stress?</h2>
<p>The review of university teaching staff over the past 20 years found five key factors contributed to stress and distress: </p>
<ol>
<li><p>balancing teaching and research workloads</p></li>
<li><p>lack of job security in an <a href="https://theconversation.com/more-than-70-of-academics-at-some-universities-are-casuals-theyre-losing-work-and-are-cut-out-of-jobkeeper-137778">increasingly casualised workforce</a></p></li>
<li><p>the role transition from professional to academic practice in applied disciplines — for example, a shift-working nurse moving from a hospital setting to teaching in a university</p></li>
<li><p>role differences for academics compared to other university staff such as administrative and IT staff as most academics have to work after hours and on weekends to manage their workload and meet performance indicators for research and teaching (including student feedback scores)</p></li>
<li><p>the overarching impacts on the sector of “<a href="https://onlinelibrary.wiley.com/doi/10.1111/j.1467-9299.1991.tb00779.x">new public managerialism</a>”.</p></li>
</ol>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/more-than-70-of-academics-at-some-universities-are-casuals-theyre-losing-work-and-are-cut-out-of-jobkeeper-137778">More than 70% of academics at some universities are casuals. They're losing work and are cut out of JobKeeper</a>
</strong>
</em>
</p>
<hr>
<p>Since the 1990s, managerialism has become firmly <a href="https://www.tandfonline.com/doi/abs/10.1080/713696158">embedded in university culture</a>. This <a href="https://www.researchgate.net/publication/287312918_Hierarchy_and_Organisation_Toward_a_General_Theory_of_Hierarchical_Social_Systems">managerialism</a> reflects beliefs about management’s power and tight control over staff. </p>
<p>Academics are facing tighter managerial control and greater surveillance. Every facet of their role is subject to oversight and regulation. </p>
<p>The great changes in technology have contributed to this situation. While technology may enable and enhance the educational experience online, it’s also increasingly used to monitor and manage performance. </p>
<p>Universities that have embraced performance management, reduced the professional autonomy of teaching staff and demanded increased productivity have the <a href="https://link.springer.com/article/10.1007/s10734-013-9668-y">lowest rates of job satisfaction</a>. Australian academics’ satisfaction with their jobs and their institutions’ management is <a href="https://www.tandfonline.com/doi/full/10.1080/1360080X.2010.491111">very low</a> compared to other countries. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/journal-papers-grants-jobs-as-rejections-pile-up-its-not-enough-to-tell-academics-to-suck-it-up-153886">Journal papers, grants, jobs ... as rejections pile up, it's not enough to tell academics to 'suck it up'</a>
</strong>
</em>
</p>
<hr>
<h2>What about the students?</h2>
<p>Ultimately, overworking and micro-managing teaching staff may <a href="https://www.nature.com/articles/d41586-021-00663-2">lead to burnout</a> and reduced enthusiasm for teaching. Additionally, an overemphasis on student retention and happiness may contribute to an erosion of academic standards. </p>
<p>Increasingly, though, the performance, promotion and continuing tenure of academics are directly <a href="https://srhe.tandfonline.com/doi/full/10.1080/02602938.2016.1224997?casa_token=LcbiXoxQKs4AAAAA%3AU3CHGTJo8innyzXX9q22rR3o76FXYXSGpvFvo8W1ybdV5Rm6RNhZeD0aQqJIL6Xa9J0UuycoMMwLQQ#.YMf0MKhLjIU">aligned with measures of student satisfaction and success</a>. The number of students who pass is one such measure. </p>
<p>This means many academics must struggle to balance keeping students happy, ensuring they succeed, while trying to maintain professional and academic standards. Many must also find the time to produce “quality” research outputs in an increasingly competitive environment. </p>
<p>Student satisfaction is now almost universally gauged through online surveys. These include <a href="https://idp.springer.com/authorize/casa?redirect_uri=https://link.springer.com/content/pdf/10.1007/s10734-014-9716-2.pdf&casa_token=wvbqjhCYPacAAAAA:WnQ1smQOsk66t0RGgqEgiULaLgGK5ou3TULwgAhgD40b9R6zguhLzZ43qvTV5zChh6pHgaM0K37cdv0okQ">anonymous verbatim student comments</a>. </p>
<figure class="align-center ">
<img alt="Man reads from laptop" src="https://images.theconversation.com/files/411130/original/file-20210713-13-118ak6i.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/411130/original/file-20210713-13-118ak6i.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/411130/original/file-20210713-13-118ak6i.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/411130/original/file-20210713-13-118ak6i.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/411130/original/file-20210713-13-118ak6i.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/411130/original/file-20210713-13-118ak6i.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/411130/original/file-20210713-13-118ak6i.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Academics report PTSD-like responses to the unfiltered anonymous feedback from student surveys.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/young-handsome-man-working-on-laptop-1744217480">Shutterstock</a></span>
</figcaption>
</figure>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/stressed-out-dropping-out-covid-has-taken-its-toll-on-uni-students-152004">Stressed out, dropping out: COVID has taken its toll on uni students</a>
</strong>
</em>
</p>
<hr>
<p>So far, several hundred academics have completed our research team’s voluntary <a href="https://scuau.qualtrics.com/jfe/form/SV_6XWZoavOraIaZts">survey</a>. The majority report receiving comments that were distressing, offensive or disrespectful. Even though these student comments are personally hurtful, many report that such comments are not redacted before being distributed, sometimes widely, within the university. </p>
<p>Universities appear to neglect the impacts of this feedback on academic well-being and reputation. One respondent wrote:</p>
<blockquote>
<p>“I have watched colleagues go through a post-traumatic stress disorder (PTSD) of sorts when evaluation swings around. They have a physiological response: sweaty palms and rapid heart rate.”</p>
</blockquote>
<p>It remains to be seen how extensive this experience is and how the problem can be managed so an experienced, qualified and enthusiastic workforce is maintained. </p>
<p>If you are an Australian health academic who would like to be involved in this research on the influence of anonymous narrative student feedback, please consider completing this ten-minute <a href="https://scuau.qualtrics.com/jfe/form/SV_6XWZoavOraIaZts">survey</a>.</p><img src="https://counter.theconversation.com/content/162612/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Workplace stress among academics has long been higher in Australia and New Zealand than overseas, and research suggests the flow-on impacts on students could fuel a vicious cycle of negative feedback.Megan Lee, Academic Tutor and PhD Candidate, Faculty of Health, Southern Cross UniversityDima Nasrawi, Lecturer in Nursing, Southern Cross UniversityMarie Hutchinson, Professor of Nursing, Southern Cross UniversityRichard Lakeman, Senior Lecturer, Health & Human Sciences, Southern Cross UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1447992020-09-15T06:08:54Z2020-09-15T06:08:54ZPhD students need support at the best of the times. How can you help in a pandemic?<figure><img src="https://images.theconversation.com/files/357630/original/file-20200911-20-k1bbqo.jpg?ixlib=rb-1.1.0&rect=0%2C8%2C6000%2C3979&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The loss of in-person contact with supervisors and peers has added to the challenges graduate students face.</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>A typical Australian PhD often involves a focused research project at one university, with one to two supervisors, and often far from your home or home country. It can be a quite isolating experience. </p>
<p>PhD students are also at great risk of mental health problems. A pre-COVID study from <a href="https://www.sciencedirect.com/science/article/abs/pii/S0048733317300422">Belgium</a> found one in two PhD students experiences psychological distress. One in three is at risk of psychiatric disorder. </p>
<p>And this year a pandemic has been thrown into the mix.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/covid-19-increases-risk-to-international-students-mental-health-australia-urgently-needs-to-step-up-137596">COVID-19 increases risk to international students' mental health. Australia urgently needs to step up</a>
</strong>
</em>
</p>
<hr>
<p>The cross-institutional, collaborative nature of the <a href="https://www.arc.gov.au/grants/linkage-program/arc-centres-excellence">Australian Research Council (ARC) Centres of Excellence</a> has allowed a re-imagining of the Australian PhD experience. Here we outline the steps the <a href="https://climateextremes.org.au/about-us/">ARC Centre of Excellence for Climate Extremes</a> has taken to support our graduate students during COVID. There’s a checklist later in this article.</p>
<p>The graduate director, a dedicated academic position, leads the program and also acts as an advocate and mentor to our students. At any one time the centre has 100 graduate students, 60% of them international, enrolled in one of five universities across four cities.</p>
<h2>COVID added to existing needs</h2>
<p>The cross-institutional nature of our centre meant our students went into the pandemic already used to videoconference meetings, seminars and training. What was lost, however, was the important and often ad-hoc supportive and collaborative conversations with peers and colleagues. These chats could happen at conferences and workshops, or simply with office mates in the work kitchen. </p>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="Weary man leaning against shelf of books" src="https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=920&fit=crop&dpr=1 600w, https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=920&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=920&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1156&fit=crop&dpr=1 754w, https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1156&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/357628/original/file-20200911-14-12yox89.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1156&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Many PhD students feel the strain of juggling multiple challenges, often a long way from home.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/college-student-who-has-run-out-1209186">Shutterstock</a></span>
</figcaption>
</figure>
<p>Beyond the challenges of continuing their research under new, even more isolating, conditions, our students also have a range of individual factors to deal with in the current circumstances. These may include: caring responsibilities, challenging work-from-home environments, being far from family, or simply the mental load of undertaking a PhD during a pandemic.</p>
<p>Early on in the centre’s life we set up mental health initiatives to ensure centre-wide support and well-being. These initiatives range from weekly hump-day tips focused on mental well-being through to ensuring we had staff and students at each university trained in <a href="https://mhfa.com.au/">mental health first aid</a>. This sort of support has become increasingly important during the calamity 2020 has become. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/no-one-would-even-know-if-i-had-died-in-my-room-coronavirus-leaves-international-students-in-dire-straits-144128">'No one would even know if I had died in my room': coronavirus leaves international students in dire straits</a>
</strong>
</em>
</p>
<hr>
<h2>Help where you can, or call on others</h2>
<p>Importantly, we recognise as supervisors we cannot resolve all issues. While we can provide an empathetic ear, in many cases students need to be redirected to the mental health support our universities’ counselling services provide.</p>
<p>We realised early on in the pandemic leaders in the centre can only offer support when they themselves are supported. Therefore, members of our centre executive received guidance, and have undertaken training, on managing others in a time of crisis. This was offered via their universities’ employee assistance program.</p>
<p>Our researchers were specifically tasked with checking in on all their students. We asked questions about working-from-home environments to ensure students had the computing and internet resources needed to continue their research. We also made note of any circumstances that might affect progress, such as caring responsibilities. </p>
<p>Any issues the centre could resolve we did. Those we could not resolve we reported in student progress review documents. We ensured students were aware of extra support available such as university counselling services.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/3-ways-the-coronavirus-outbreak-will-affect-international-students-and-how-unis-can-help-131195">3 ways the coronavirus outbreak will affect international students and how unis can help</a>
</strong>
</em>
</p>
<hr>
<h2>Financial hardship increased</h2>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=600&fit=crop&dpr=1 600w, https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=600&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=600&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=754&fit=crop&dpr=1 754w, https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=754&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/357631/original/file-20200911-18-h0lj48.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=754&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Three-quarters of PhD students surveyed expected the pandemic to cause them financial hardship.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/young-student-worried-over-unpaid-bills-257244583">Shutterstock</a></span>
</figcaption>
</figure>
<p>A <a href="https://www.researchsquare.com/article/rs-36330/v2">recent study</a> of Australian PhD students showed 75% expected to experience financial hardship as a result of the pandemic, and we have seen that. Students close to completion were hit hardest. They were faced with the end of their scholarships at a time of employment uncertainty, often with no access to government support, and closed borders that prevented them returning home. </p>
<p>The centre quickly offered scholarships to provide bridging funds after thesis submission. These scholarships were offered once individual university support was exhausted and required a tangible outcome at the end, such as writing up a thesis chapter for submission to a journal.</p>
<h2>Working to stay connected</h2>
<p>At the same time as we dealt with the practical realities of student finances and research, we made every effort to keep everyone socially connected. </p>
<p>Our annual winter school, a cornerstone event of the graduate program, shifted online. To avoid Zoom fatigue we replaced the week-long face-to-face schedule with a winter school offered in two-hour slots. Sessions were recorded and breakout rooms used to focus student engagement with each other. When we saw how delighted students were to see each other in small groups and catch up, it was a sign for us to step out and let them engage with their peers.</p>
<p>We introduced additional Slack channels, held virtual morning and afternoon teas and put together a weekly centre-wide lunch for all early career researchers. The numbers in these virtual meet-ups declined as students settled into new routines. In contrast, our research meeting and seminar attendances went through the roof. We found if an event has a purpose people attend, even while social check-ups became less successful.</p>
<p>Finally, we continued to celebrate successes and PhD submissions in a long-running weekly email update. In these updates, we made it clear we understood the impacts of these uncertain times on students’ progress.</p>
<figure class="align-center zoomable">
<a href="https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="Checklist of ways to support grad students" src="https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=849&fit=crop&dpr=1 600w, https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=849&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=849&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1067&fit=crop&dpr=1 754w, https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1067&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/357632/original/file-20200911-22-fbgj2t.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1067&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption"></span>
<span class="attribution"><span class="license">Author provided</span></span>
</figcaption>
</figure>
<p>While not everything worked perfectly, our centre-wide relationship with students put us in an excellent position to respond quickly, transparently and with student input as the pandemic unfolded.</p>
<p>We do not know how long this pandemic will last. What we do know is all current and any incoming PhD candidates will feel the impacts in some way. With PhD students producing <a href="https://go8.edu.au/files/docs/the-changing-phd_final.pdf">more than half</a> of university research in Australia, this crisis illustrates the importance of ongoing development and support of higher-degree research students.</p><img src="https://counter.theconversation.com/content/144799/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Melissa Hart does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Life for graduate students can be hard work and often isolating, and COVID-19 piled on the pressures. That’s when having an academic leader and program dedicated to supporting them proved its worth.Melissa Hart, Graduate Director, ARC Centre of Excellence for Climate Extremes, UNSW SydneyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1411342020-08-13T20:09:18Z2020-08-13T20:09:18ZA 3-decade ‘moving picture’ of young Australians’ study, work and life, thanks to LSAY<figure><img src="https://images.theconversation.com/files/350985/original/file-20200804-22-1reruft.jpg?ixlib=rb-1.1.0&rect=9%2C56%2C6253%2C4638&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">
</span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/teenage-students-using-laptops-library-143880715">Shutterstock</a></span></figcaption></figure><p>The Longitudinal Surveys of Australian Youth (<a href="https://www.lsay.edu.au/">LSAY</a>) unpack the lives of young Australians as they leave school, enter further study or the workforce and make the transition into adulthood. </p>
<p>The latest findings are now available for the group of young people who completed their first questionnaire back in 2009 at age 15. This group’s <a href="https://www.lsay.edu.au/data/access">11th and final survey</a> shows young people are completing university at higher rates than ever before, while participation in apprenticeships and traineeships is taking a dive.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/most-young-people-who-do-vet-after-school-are-in-full-time-work-by-the-age-of-25-133060">Most young people who do VET after school are in full-time work by the age of 25</a>
</strong>
</em>
</p>
<hr>
<p>The information collected from these groups of students, or “cohorts”, is used to better understand what helps or hinders this transition. This includes things like the effect of schools on year 12 completion, whether government benefits like <a href="https://www.servicesaustralia.gov.au/individuals/services/centrelink/youth-allowance">Youth Allowance</a> help students complete their studies, and the factors that help a young person find full-time work sooner. </p>
<p>Each cohort starts with about 14,000 students in the first survey, or “wave”. From the age of 15 to 25, they complete a 20-minute survey once a year to share what’s been happening in their lives. LSAY asks about their experiences at school, their post-school study and work, as well as their health and home life. </p>
<p>Six cohorts have taken part so far. The recent release of findings from the fifth cohort’s final survey is a milestone, with LSAY <a href="https://www.lsay.edu.au/data/access">data now available across three decades</a>. This means we can study generational changes in transition patterns.</p>
<p>To capture the many changing events or factors that affect young peoples’ transition, the survey has added questions about caring responsibilities, volunteering activities, participation in the gig economy, their personality traits and whether they have access to social support. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/students-own-low-expectations-can-reinforce-their-disadvantage-23501">Students' own low expectations can reinforce their disadvantage</a>
</strong>
</em>
</p>
<hr>
<h2>Data dating back to the ’70s</h2>
<p>LSAY is one of Australia’s biggest and longest-running panel surveys. More than 60,000 young people have been surveyed since 1995. It’s recognised as one of eight <a href="https://thesource.dss.gov.au/final-report-review-of-australias-longitudinal-data-system">core longitudinal data assets in Australia</a>.</p>
<p>The surveys grew out of the Youth in Transition (<a href="https://dataverse.ada.edu.au/dataverse/yit">YIT</a>) studies in the 1970s.
The decade’s <a href="https://www.jstor.org/stable/10.7591/j.ctt207g7cv.4?seq=1#metadata_info_tab_contents">oil price shocks</a> caused unemployment to soar, with young people hit the hardest. This created a need to better understand their school-to-work transition in the face of global technological and economic change. </p>
<p>Then came the Australian Longitudinal Surveys (<a href="https://www.aph.gov.au/binaries/library/pubs/bp/1990/90bp21.pdf">ALS</a>) and Australian Youth Surveys (<a href="https://dataverse.ada.edu.au/dataverse/australian-youth-survey?q=&types=dataverses%3Adatasets&page=1&sort=nameSort&order=asc">AYS</a>) in the 1980s. One of the more prominent pieces of research using these data found the aptitude of new teachers fell substantially as teacher pay declined compared to other salaries.</p>
<p>These three longitudinal studies were combined to create the LSAY program. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/if-you-have-a-low-atar-you-could-earn-more-doing-a-vet-course-than-a-uni-degree-if-youre-a-man-121624">If you have a low ATAR, you could earn more doing a VET course than a uni degree – if you're a man</a>
</strong>
</em>
</p>
<hr>
<h2>Researchers mine LSAY for insights</h2>
<p>More than <a href="https://www.lsay.edu.au/research">300 published research papers</a> have used LSAY data. The report <a href="https://www.lsay.edu.au/publications/search-for-lsay-publications/25-years-of-lsay-research-from-the-longitudinal-surveys-of-australian-youth">25 years of LSAY: Research from the Longitudinal Surveys of Australian Youth</a> showcases some of the highlights.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="McDonald's worker hands over order at a drive-through counter." src="https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=600&fit=crop&dpr=1 600w, https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=600&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=600&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=754&fit=crop&dpr=1 754w, https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=754&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/350998/original/file-20200804-18-18hi7ae.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=754&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">LSAY shows working a few hours a week while at school helps get a full-time job later.</span>
<span class="attribution"><span class="source">Shutterstock</span></span>
</figcaption>
</figure>
<p>LSAY research has shown working just a few hours a week while at school <a href="https://www.lsay.edu.au/publications/search-for-lsay-publications/2398">improves prospects of getting a full-time job</a>. But working long hours has a slightly negative effect on school completion. The research also found females are better at balancing school and work than their male peers. </p>
<p>Research has also shown that students participating in school-based vocational education and training (VET) had <a href="https://melbourneinstitute.unimelb.edu.au/publications/research-insights/search/result?paper=3194369">higher rates of school completion</a>, full-time employment and incomes in their first year after school than non-VET students with similar characteristics. Ex-VET students were also more likely to be in a job they liked as a career. These benefits were associated with school-based VET programs with a workplace learning component.</p>
<p>The Productivity Commission used LSAY data to <a href="https://www.pc.gov.au/research/completed/university-report-card">investigate the demand-driven university system</a>. Many disadvantaged students successfully attended university as a result of the expansion of the system. However, those with lower literacy and numeracy were more likely to drop out. The study recognised schools and universities need to do more to prepare and support students, and that university might not always be the best option. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/more-students-are-going-to-university-than-before-but-those-at-risk-of-dropping-out-need-more-help-118764">More students are going to university than before, but those at risk of dropping out need more help</a>
</strong>
</em>
</p>
<hr>
<p>LSAY has been an important source of evidence for policy. National reviews and inquiries informed by LSAY data include the <a href="https://docs.education.gov.au/documents/national-partnership-youth-attainment-and-transitions-second-interim-evaluation-report">COAG Reform Council’s reporting on youth transitions</a> (2009), the <a href="https://www.voced.edu.au/content/ngv%3A32134">Bradley Review of Higher Education</a> (2008) and the <a href="https://www.aph.gov.au/Parliamentary_Business/Committees/House_of_Representatives_Committees?url=edt/schoolandwork/report.htm">House of Representatives inquiry into combining school and work</a> (2008-2009).</p>
<p>The recent Education Council <a href="https://www.pathwaysreview.edu.au/">Review of Senior Secondary Pathways</a>, released in July, draws heavily on LSAY to establish how students can choose the best pathway for their transition from school. </p>
<p>LSAY has a high degree of comparability with international youth surveys. These include the Transition from Education to Employment (<a href="https://www.tree.unibe.ch/index_eng.html">TREE</a>) study in Switzerland, the Youth in Transition Survey (<a href="https://open.canada.ca/en/suggested-datasets/youth-transition-survey">YITS</a>) in Canada, the Education Longitudinal Study (<a href="https://nces.ed.gov/statprog/handbook/els2002.asp">ELS</a>) and National Longitudinal Survey of Youth (<a href="https://www.nlsinfo.org/content/cohorts/nlsy79">NLSY</a>) in the United States, and <a href="https://esrc.ukri.org/research/our-research/next-steps-formerly-longitudinal-study-of-young-people-lsype/">Next Steps</a> in the UK. </p>
<p>Most of these have a starting sample of about 9,000 individuals. Next Steps has 16,000. LSAY’s starting sample of 14,000 young Australians makes it one of the largest surveys of its kind in the world. </p>
<h2>Tracking lives through the GFC and COVID-19</h2>
<p>These datasets enable us to transform a snapshot of a person’s life into a moving picture. Compared with cross-sectional studies, these longitudinal datasets provide a much clearer picture by accounting for personalities, life events and pathways. </p>
<figure class="align-center ">
<img alt="Four fingers representing people with different personalities" src="https://images.theconversation.com/files/352616/original/file-20200812-16-1ovbxt3.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/352616/original/file-20200812-16-1ovbxt3.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/352616/original/file-20200812-16-1ovbxt3.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/352616/original/file-20200812-16-1ovbxt3.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/352616/original/file-20200812-16-1ovbxt3.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/352616/original/file-20200812-16-1ovbxt3.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/352616/original/file-20200812-16-1ovbxt3.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">The longitudinal dataset helps account for different personalities.</span>
<span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/finger-art-concept-group-people-different-324484541">Shutterstock</a></span>
</figcaption>
</figure>
<p>Combining a longitudinal study with cohort studies sheds more light on this picture by controlling for inter-generational differences, or crises such as wars, financial downturns or natural disasters. </p>
<p>For example, using data from four LSAY cohorts, <a href="https://content.apa.org/fulltext/2016-06225-001.html">one study</a> found the well-being of those whose transitions occurred during the global financial crisis (<a href="https://www.rba.gov.au/education/resources/explainers/the-global-financial-crisis.html">GFC</a>) was much worse on several measures, including standard of living, home life, career prospects, social life and independence. </p>
<p>The extraordinary challenges Australian youth face as a result of the coronavirus pandemic will be documented when the sixth LSAY cohort, now aged 20, complete their sixth survey in 2020 and further surveys in the years thereafter. </p>
<p>By providing a valuable resource to explore the longer-term effects of this crisis, LSAY continues to stand the test of time.</p><img src="https://counter.theconversation.com/content/141134/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Somayeh Parvazian works for National Center for Vocational Education Research. </span></em></p><p class="fine-print"><em><span>Ronnie Semo works for the National Centre for Vocational Education Research.</span></em></p>LSAY is one of the biggest and oldest surveys of its kind in the world. It follows young Australians from the age of 15 to 25 to find out what helps and hinders them along the way.Somayeh Parvazian, Survey Methodologist, National Centre for Vocational Education Research (NCVER)Ronnie Semo, Senior Research Officer, National Centre for Vocational Education Research (NCVER)Licensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/1117412019-02-13T19:18:39Z2019-02-13T19:18:39ZMale teachers are most likely to rate highly in university student feedback<figure><img src="https://images.theconversation.com/files/258659/original/file-20190213-90491-1i6pkbe.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">Students are invited to give feedback on teachers performance at Australian universities. </span> <span class="attribution"><a class="source" href="https://www.shutterstock.com/image-photo/back-view-man-presenting-students-lecture-478521652">from www.shutterstock.com</a></span></figcaption></figure><p>University students, like many in <a href="https://theconversation.com/to-achieve-gender-equality-we-must-first-tackle-our-unconscious-biases-92848">society</a>, demonstrate bias against women and particularly women from non-English speaking backgrounds. </p>
<p>That’s the take home message from a <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209749">new and comprehensive analysis</a> of student experience surveys.</p>
<p>The study examined a large dataset consisting of more than 500,000 student responses collected over 2010 to 2016. It involved more than 3,000 teachers and 2,000 courses across five faculties at the University of New South Wales (UNSW), Sydney.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/study-of-1-6-million-grades-shows-little-gender-difference-in-maths-and-science-at-school-101242">Study of 1.6 million grades shows little gender difference in maths and science at school</a>
</strong>
</em>
</p>
<hr>
<h2>Most bias in science and business</h2>
<p>Interestingly, the bias varies. </p>
<p>In parts of science and business the effects are clear. In the science and business faculties, a male teacher from an English-speaking background was more than twice as likely to get a higher score on a student evaluation than a female teacher from a non-English speaking background.</p>
<p>But in other areas, such as arts and social science, the effects are almost marginal. In engineering, effects were only detected for non-English speakers.</p>
<p>When one looks at the probability of scoring very high ratings, and dissects the categories into genders and cultural background, the results are clear. The disparities occur mostly at the very top end: this is where bias creeps in.</p>
<p>Previously the university had looked at just the average (mean) ratings of teachers of different genders, and found that they are more or less indistinguishable (unpublished data). But this new study goes further and provides information that is not evident in superficial analyses.</p>
<h2>Should we abandon student feedback?</h2>
<p>Student feedback can be a useful mechanism to understand the varied experiences of students. But student feedback is sometimes used inappropriately in staff performance evaluations, and that’s where the existence of bias creates serious problems.</p>
<p>One can make the case for abandoning student feedback – and many <a href="https://www.universityaffairs.ca/news/news-article/arbitration-decision-on-student-evaluations-of-teaching-applauded-by-faculty/">have</a>.</p>
<p>But it’s problematic to turn a deaf ear to the student voice, and that is not what national approaches such as the Quality in Learning and Teaching processes (<a href="https://www.qilt.edu.au/">QILT</a>) are doing.</p>
<p>This is because feedback can often be helpful. It can make things better. In addition, it is often positive. Sometimes the feedback is actually the way students say thanks. </p>
<p>However, sometimes it can be very hurtful and damaging, particularly if it is motivated by prejudice. We have to be aware of that and the barriers it can create. </p>
<p>We know that minority groups already suffer from <a href="https://theconversation.com/how-men-and-women-can-help-reduce-gender-bias-in-the-workplace-62041">reduced confidence and visibility</a>, so biased teacher evaluations may exaggerate existing inequities.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/how-understanding-animal-behaviour-can-liberate-us-from-gender-inequality-102981">How understanding animal behaviour can liberate us from gender inequality</a>
</strong>
</em>
</p>
<hr>
<h2>What do the numbers mean?</h2>
<p>It is very important to be cautious when looking at the raw numbers.</p>
<p>Firstly, let’s consider what the numbers mean. Students are not evaluating teaching and learning in these surveys. They are telling us about their experiences – that’s why we call them MyExperience surveys at UNSW. We resist the idea that they are student evaluations of teaching, as are used in some settings.</p>
<p>Peer review can make contributions to evaluating teaching while assessments can help evaluate learning – however they may not be enough to overcome bias. When considering professional performance at UNSW, we do not exclude the feedback that students provide on their experience, but we look at a basket of indicators.</p>
<p>Secondly, one has to be serious about the biases that emerge, acknowledge them and confront the issues. Most universities pride themselves on being diverse and inclusive, and students support this. </p>
<p>But this study reminds us that we have work to do. Biases exist. The message is strong. You are more likely to score top ratings if you come from the category of white male: that is, if you are from the prevailing establishment.</p>
<h2>The influence of history</h2>
<p>These results may be surprising given the diversity of the student and staff body at Australian universities.</p>
<p>But our cultural milieu has been historically saturated by white males, and continuing biases exist. The important thing is to be aware of them, and when looking at the numbers to realise that the ratings are provided in the context of a particular society at a particular moment in time. </p>
<p>The scores should not be blindly accepted at face value.</p>
<p>Most universities, including ours, are working on being more inclusive. At UNSW a new Deputy Vice-Chancellor Equity, Diversity and Inclusion – <a href="https://www.unsw.edu.au/about-us/governance/management-board/deputy-vice-chancellor-equity-diversity-and-inclusion">Eileen Baldry</a> – was recently appointed, and we are working hard to combat bias and to introduce new strategies aimed at supporting diversity. For example, the university will introduce new training for members of promotion panels, explaining the biases detected in our new study. By understanding the problem, we can begin to address it.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/walking-into-a-headwind-what-it-feels-like-for-women-building-science-careers-102259">'Walking into a headwind' – what it feels like for women building science careers</a>
</strong>
</em>
</p>
<hr>
<p>All staff across all of our universities can benefit from becoming more aware of issues around bias – especially those in powerful positions, such as members of promotion committees.</p>
<p>Reducing bias will have great benefits for society as university students represent a large proportion of future leaders in government and industry.</p>
<p>It is clear that negative stereotypes will contribute to the partiality that exists within our student community. Encouraging more women and cultural minorities at all levels in higher education, in leadership positions and in membership of key committees will help shrink these biases. </p>
<h2>Training in values</h2>
<p>Training students is challenging, especially at large modern universities such as UNSW, which has a cohort of over 50,000 coming from more than 100 countries. But our study found similar levels of bias in local students, as we did in international students.</p>
<p>In training students we have to remember that we provide knowledge, but also communicate values via our words and our behaviours. </p>
<p>If we are to continue to listen to the student experience, we need to be careful with the results. Rigorous statistical analyses such as this study, can help us recognise bias and work to address it. If our students graduate with less bias than when they entered their degree, we will be contributing to creating a more equitable and inclusive society in the future. </p>
<p>It is not easy to uproot prejudices but the data are clear. We expect people will be on board and be pleased to contribute to moving things in the right direction.</p><img src="https://counter.theconversation.com/content/111741/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Merlin Crossley is the Deputy Vice-Chancellor Academic at UNSW. He receives funding from the ARC and NHMRC. He is on the Editorial Board of The Conversation, the Board of the Australian Science Media Centre, and the Trust of the Australian Museum.</span></em></p><p class="fine-print"><em><span>Emma Johnston is Dean of Science at UNSW and President of Science and Technology Australia (STA), STA is a member organisation representing >70,000 scientists and technologists across Australia. </span></em></p><p class="fine-print"><em><span>Yanan Fan is Associate Professor at the School of Mathematics and Statistics at UNSW, and an Associate Investigator for the ARC Centre of Excellence for Mathematical and Statistical Frontiers (ACEMS).</span></em></p>A study or more than 500,000 surveys shows university students demonstrate bias against women teachers, and particularly women from non-English speaking backgrounds.Merlin Crossley, Deputy Vice-Chancellor Academic and Professor of Molecular Biology, UNSW SydneyEmma Johnston, Professor and Dean of Science, UNSW SydneyYanan Fan, Associate Professor of Statistics, UNSW SydneyLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/901662018-01-17T18:44:14Z2018-01-17T18:44:14ZSurveys are not the best way to measure the performance of Australian universities<figure><img src="https://images.theconversation.com/files/202231/original/file-20180117-53302-7x64gn.jpg?ixlib=rb-1.1.0&rect=7%2C170%2C991%2C453&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">The surveyors start out with almost 100,000 graduate contacts, of whom less than 10% provide their supervisor's details and of those supervisors, less than half participate in the survey. </span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>Depending on who’s doing the reading, the <a href="https://www.qilt.edu.au/about-this-site/employer-satisfaction">2017 Employer Satisfaction Survey</a> shows the overwhelming majority of employers <a href="https://www.universitiesaustralia.edu.au/Media-and-Events/media-releases/Grads-well-prepared-for-jobs--employer-satisfaction-remains-high--survey#.Wl1NX66Wa70">think graduates are well-prepared for jobs</a>. Or it might show universities <a href="https://www.senatorbirmingham.com.au/improving-the-employment-potential-of-graduates/">need to do more to address the needs of graduates and employers</a>. Or even that universities are <a href="http://www.theaustralian.com.au/higher-education/university-degrees-are-failing-to-deliver-for-business/news-story/554c200326874314f8b14ff161296503">not delivering to businesses</a>.</p>
<p>Concerns about whether universities are doing enough to make graduates job-ready are not new. As higher education policy expert Andrew Norton observed <a href="http://andrewnorton.net.au/2018/01/08/over-qualification-hard-to-measure-harder-to-avoid/">recently</a>, this issue was alive in the early <a href="http://www.voced.edu.au/content/ngv%3A29728">70s</a>. </p>
<p>In fact, you can go back in time even further. In the 1950s, the Report of the Committee on Australian Universities <a href="http://www.voced.edu.au/content/ngv%3A53782">called</a> upon both universities and government to do more to work with industry to identify future labour demand and offer courses accordingly. </p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/five-myths-about-australian-university-graduate-outcomes-87074">Five myths about Australian university graduate outcomes</a>
</strong>
</em>
</p>
<hr>
<p>The release of the latest <a href="https://www.qilt.edu.au/docs/default-source/gos-reports/2017/2017_gos_national_report_final_accessiblea45d8791b1e86477b58fff00006709da.pdf?sfvrsn=ceb5e33c_4&_cldee=c29waGllLmhlaXplckB0aGVjb252ZXJzYXRpb24uZWR1LmF1&recipientid=contact-09dc957fbd87e7118120e0071b66a691-0930c8235d7d4a10bd63a0fdd8972b65&esid=f8ef3ba5-1cf7-e711-8134-e0071b68f7c1">Graduate Ouctomes Survey</a> had a similar effect in focussing attention on higher education performance. </p>
<p>A focus on graduate employability is not surprising. What is surprising is we are using mostly the same ways of measuring university performance we have for decades, when more accurate means exist. </p>
<h2>Why the way we use the surveys is flawed</h2>
<p>Like all surveys, the <a href="https://www.qilt.edu.au/about-this-site/employer-satisfaction">Employer Satisfaction Survey</a> has to account for and overcome a number of elements that can affect the validity of the results. For example, 4,348 survey responses sounds like a lot, but this represents only 9.3% of all possible employer contacts. </p>
<p>Also, the way employers are contacted is a problem. It’s the graduate who’s contacted and invited to provide their supervisor’s details to the survey team. So the surveyors start out with almost 100,000 graduate contacts, of whom less than 10% provide their supervisor’s details and of those supervisors, less than half participate in the survey. </p>
<p>Another issue is the survey relies largely on subjective measurements of perception. For example, data <a href="http://andrewnorton.net.au/2018/01/08/over-qualification-hard-to-measure-harder-to-avoid/">shows</a> the supervisors of graduates are more likely than the graduates themselves to think the graduate’s qualification is important. Two perceptions of the same qualification in the same context - which one, if either, is right? </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/202236/original/file-20180117-53328-12peyri.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/202236/original/file-20180117-53328-12peyri.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=400&fit=crop&dpr=1 600w, https://images.theconversation.com/files/202236/original/file-20180117-53328-12peyri.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=400&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/202236/original/file-20180117-53328-12peyri.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=400&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/202236/original/file-20180117-53328-12peyri.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=503&fit=crop&dpr=1 754w, https://images.theconversation.com/files/202236/original/file-20180117-53328-12peyri.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=503&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/202236/original/file-20180117-53328-12peyri.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=503&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Employers and graduates had different ideas of how important the graduate’s qualification was.</span>
<span class="attribution"><span class="source">Shutterstock</span></span>
</figcaption>
</figure>
<p>The <a href="https://www.qilt.edu.au/docs/default-source/gos-reports/2017/2017_gos_national_report_final_accessiblea45d8791b1e86477b58fff00006709da.pdf?sfvrsn=ceb5e33c_4&_cldee=c29waGllLmhlaXplckB0aGVjb252ZXJzYXRpb24uZWR1LmF1&recipientid=contact-09dc957fbd87e7118120e0071b66a691-0930c8235d7d4a10bd63a0fdd8972b65&esid=f8ef3ba5-1cf7-e711-8134-e0071b68f7c1">Graduate Outcomes Survey</a> also relies on graduates being willing to complete the survey. The latest survey had a response rate of 45%, which is very good for surveys. But the survey is sent out only four months after graduation. It does not, then, necessarily reflect the short - let alone medium or long-term - employment prospects for the individual. </p>
<p>This is not to say the methodologies underpinning these surveys are not robust, or the <a href="http://www.srcentre.com.au/">Social Research Centre</a>, who deliver the surveys, are not experts in their field. It is and they are. When a survey is the best option for gathering data, then these types of survey should be run. But we shouldn’t be using findings such as these to measure university performance, when there are better options available. </p>
<h2>The missing link</h2>
<p>For decades now, there has been an administrative link between a graduate’s education and taxation records. If domestic students have ever wondered why they are issued with a<a href="http://studyassist.gov.au/sites/studyassist/helppayingmyfees/pages/track_debt"> Commonwealth Higher Education Student Support Number</a> (CHESSN), and why they need to provide their Tax File Number (TFN) to the university, this is the reason. </p>
<p>The CHESSN tracks their educational history, even when they change courses or institutions. Consequently it keeps track of their
<a href="http://studyassist.gov.au/sites/studyassist/helppayingmyfees/hecs-help/pages/hecs-help-welcome">HECS-HELP</a> debt. By linking the CHESSN to the TFN, a record of the debt can be provided to the Australian Taxation Office, for future collection. </p>
<p>This administrative link could also be used to provide accurate and detailed longitudinal analyses of which jobs all graduates end up in, not just those motivated to respond to a survey. As time passes - or by going back further into the records - detailed pictures can be provided about how graduates perform over time, which organisations recruit and retain the most graduates, which courses show evidence of greater graduate mobility, actual lifetime earnings (as opposed to predicted), and so on. </p>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/202237/original/file-20180117-53289-1ke8jld.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/202237/original/file-20180117-53289-1ke8jld.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=381&fit=crop&dpr=1 600w, https://images.theconversation.com/files/202237/original/file-20180117-53289-1ke8jld.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=381&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/202237/original/file-20180117-53289-1ke8jld.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=381&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/202237/original/file-20180117-53289-1ke8jld.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=479&fit=crop&dpr=1 754w, https://images.theconversation.com/files/202237/original/file-20180117-53289-1ke8jld.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=479&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/202237/original/file-20180117-53289-1ke8jld.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=479&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Students already have to provide their TFN to their place of study and work so that debt can be tracked.</span>
<span class="attribution"><span class="source">Shutterstock</span></span>
</figcaption>
</figure>
<p>Graduate outcomes would also be better contextualised against non-graduate outcomes, as well as national and international labour market trends. One-off, or purpose-specific analyses could be more easily provided to address specific government or community concerns as and when they arise. </p>
<p>If the government were to make key findings of these analyses publicly-available on a regular basis, students, politicians and policymakers would be able to make much more informed decisions regarding future study requirements. The current surveys would still be important, as they can provide additional information government records cannot. But when it comes to measuring university performance, hard data is the key.</p>
<hr>
<p>
<em>
<strong>
Read more:
<a href="https://theconversation.com/the-problem-isnt-unskilled-graduates-its-a-lack-of-full-time-job-opportunities-90104">The problem isn't unskilled graduates, it's a lack of full-time job opportunities</a>
</strong>
</em>
</p>
<hr>
<p>Linking government records in this way is a sensitive issue. There would need to be a significant investment in the right infrastructure and systems to ensure the data was protected and analysed appropriately. Given the centrality of graduate employability to government higher education policy, now might be the time for this investment.</p>
<hr>
<p><em>The final paragraph of this article has been amended since publication to reflect that there is some existing legislation that could facilitate this data collection.</em></p>
<hr><img src="https://counter.theconversation.com/content/90166/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Tim Pitman does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>An administrative link between a graduate’s education and taxation records already exists, and it could be used to give us more accurate and detailed longitudinal analyses of graduate outcomes.Tim Pitman, Senior Researcher Fellow, Curtin UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/854512017-10-24T19:05:09Z2017-10-24T19:05:09ZWhat makes a good university? Academics and students have different ideas<figure><img src="https://images.theconversation.com/files/190312/original/file-20171016-27711-zyhel0.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">There is no good reason the views of students should be disregarded in what defines quality higher education.</span> <span class="attribution"><span class="source">Shutterstock</span></span></figcaption></figure><p>New analysis of the Federal Government’s <a href="https://www.qilt.edu.au/">Quality Indicators of Learning and Teaching</a> (QILT) and the <a href="https://www.timeshighereducation.com/world-university-rankings">Times Higher Education</a> (THE) reveals a divide between what academics think defines quality and what students actually experience. This is demonstrated by world university rankings and student satisfaction surveys.</p>
<p>Quality teaching is an essential component of the QILT program. Generally, satisfaction levels among Australian university students are high at around 80%. However, there is often a significant gap between what students consider a quality teaching experience and what academics consider quality teaching.</p>
<p>World university rankings continue to attract attention, with almost all Australian universities now participating. Millions of potential students from around the world use ranking information to make choices about their future studies. Rankings guide policy, investment, jobs and partnerships across the globe.</p>
<p>The THE university ranking, operated by <a href="https://www.tesglobal.com/">TES Global</a>, has a media reach of almost 700 million. It ranks 2,150 institutions worldwide, 35 in Australia. Overall, Australia performs well in the rankings.</p>
<h2>Student perspectives absent from reputation surveys</h2>
<p>Half of the Times’ measure for quality teaching is based on an <a href="https://www.timeshighereducation.com/world-university-rankings/academic-reputation-survey-explained">academic reputation survey</a>. The reputation survey claims to be the largest of its kind in the world with more than 10,320 respondents, and underpins ranking results. </p>
<p>The academic reputation survey asks respondents which universities are “the best”, based on their experience in a particular field. QILT on the other hand, asks a series of targeted questions about the quality of teaching. These include overall experience, learning engagement, explanations on assessment, course structure and focus.</p>
<p>The ranking is a major source of information for students, but the Times are entirely devoid of student experience data. Despite this, universities are drawn to using the ranking to promote a quality education experience, based on information that provides only part of this assurance, and none from a student experience perspective.</p>
<p>Analysis indicates the academic reputation survey conducted by the Times has almost no bearing on actual experience recalled by students. In many cases, there is a significant divide between what academic peers and students consider to be quality teaching.</p>
<hr>
<p><iframe id="3masC" class="tc-infographic-datawrapper" src="https://datawrapper.dwcdn.net/3masC/2/" height="1020px" width="100%" style="border: none" frameborder="0"></iframe></p>
<hr>
<h2>Size and ranking correlate, but ranking and satisfaction do not</h2>
<p>Interestingly, universities whose academic peers are critical of their institution actually tend to rate better with students.</p>
<p>The typical profile of an Australian university with very optimistic academic peers is: 106 years old, has over 55,000 students, performs 71% better in quality teaching according to the rankings – but is below or average according to students. Examples are Monash University, The University of Melbourne, the University of New South Wales, the University of Queensland and Sydney University.</p>
<p>On the other hand, Australian universities with very pessimistic academic peers are on average: 29 years old, with about 27,000 students, perform - 36% worse in quality teaching according to the rankings, but up to 13% better according to students. Examples include Australian Catholic University, Bond University, Edith Cowan, Murdoch University, University of Sunshine Coast and Western Sydney University.</p>
<hr>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/191306/original/file-20171023-13966-pfq6w4.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/191306/original/file-20171023-13966-pfq6w4.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=517&fit=crop&dpr=1 600w, https://images.theconversation.com/files/191306/original/file-20171023-13966-pfq6w4.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=517&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/191306/original/file-20171023-13966-pfq6w4.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=517&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/191306/original/file-20171023-13966-pfq6w4.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=650&fit=crop&dpr=1 754w, https://images.theconversation.com/files/191306/original/file-20171023-13966-pfq6w4.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=650&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/191306/original/file-20171023-13966-pfq6w4.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=650&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption"></span>
<span class="attribution"><span class="source">Author provided/Times Higher Education rankings</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<hr>
<p>In Australia, there is a relationship between size and ranking. Highly ranked institutions are usually big. The expansion of the higher education sector coincides with Australia’s comprehensive ranking results on the world stage. The catch is that when it comes to student satisfaction, being big doesn’t seem to help.</p>
<hr>
<figure class="align-center ">
<img alt="" src="https://images.theconversation.com/files/191503/original/file-20171024-1748-1cm8t9r.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&fit=clip" srcset="https://images.theconversation.com/files/191503/original/file-20171024-1748-1cm8t9r.png?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=556&fit=crop&dpr=1 600w, https://images.theconversation.com/files/191503/original/file-20171024-1748-1cm8t9r.png?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=556&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/191503/original/file-20171024-1748-1cm8t9r.png?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=556&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/191503/original/file-20171024-1748-1cm8t9r.png?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=699&fit=crop&dpr=1 754w, https://images.theconversation.com/files/191503/original/file-20171024-1748-1cm8t9r.png?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=699&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/191503/original/file-20171024-1748-1cm8t9r.png?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=699&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption"></span>
<span class="attribution"><span class="source">Author provided/Times Higher Education rankings</span>, <a class="license" href="http://creativecommons.org/licenses/by-nd/4.0/">CC BY-ND</a></span>
</figcaption>
</figure>
<hr>
<h2>Elation & peril</h2>
<p>There is also a circular logic to quality teaching in the rankings. High rankings build reputation, while reputation is required to achieve a high ranking. So it makes sense that the rankings are generally stable at the top and volatile in the middle, with a long tail.</p>
<p>The Times does attempt to gather data on student experience, but only 1,000 students are included and all are based in the UK. In Australia, the Commonwealth supported QILT program <a href="https://www.qilt.edu.au/about-this-site/student-experience">administers a survey</a> to more than 123,000 students each year and is overwhelmingly more comprehensive.</p>
<p>While rankings can be a source for elation, a fall in the rankings is the dread of almost all universities. However, rankings are a mainstay of the global higher education landscape and are useful. Policy makers, like students, should consult the rankings with careful deliberation. If expectations are high among students, those expectations should be respected.</p>
<p>All this confirms that lower-ranked universities are not simply an assembly of “easy-to-please” students. Rather, the ranking of quality teaching using reputation indicators is incomplete.</p>
<p>Notions of quality are a subjective matter with many facets and competing perspectives. That said, the views of students should be respected not ignored in what defines quality higher education.</p><img src="https://counter.theconversation.com/content/85451/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Omer Yezdani does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.</span></em></p>Lower ranked universities often outperform their highly ranked peers in student satisfaction.Omer Yezdani, Director, Office of Planning and Strategic Management, Australian Catholic UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/400712015-05-25T20:06:45Z2015-05-25T20:06:45ZUniversities should change the way they measure success<figure><img src="https://images.theconversation.com/files/78684/original/image-20150421-12015-9nf9a1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">How do universities measure their success? How should they?</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/benbeiske/4099931431">Ben Beiske\Flickr</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Universities have a vast array of measures to gauge how successful they are. Most of the measures have a lot to do with prestige and not much to do with the outcomes of their graduates or the quality of the education their students receive. </p>
<h2>Cut-off scores</h2>
<p>Probably the most pervasive measure of universities’ success in Australia and in many other countries is their selectivity of undergraduate student entry, measured in Australia by cut-off scores. A program’s <a href="http://www.uac.edu.au/undergraduate/atar/cut-offs.shtml">cut-off score</a> is the lowest score needed to gain entry to the program. </p>
<p>Cut-off scores are not a good indicator of the quality or even of the prior attainment of entering students because most students have a score above the cut-off. Often programs with a cut-off score around 60, for example, admit most students with a score above 80. A far better measure of the attainment of students entering a program, inasmuch as it is a good measure of anything, is the median or middle entry score of students.</p>
<p>Nonetheless, cut-off scores are widely viewed as an indicator of a program and university’s quality by students, parents, schools, governments, members of the public and even by university staff who should know better. A good example is NSW Education Minister Adrian Piccoli and the Group of Eight elite universities <a href="http://www.theaustralian.com.au/higher-education/unis-vow-to-fight-teaching-cut-off/story-e6frgcjx-1226581416720">who sought to require minimum entry scores</a> for admission to teacher education programs or to all higher education bachelor programs.</p>
<h2>Student surveys</h2>
<p>Australia, with many other countries, relies heavily on student surveys to evaluate universities’ performance. The <a href="http://www.graduatecareers.com.au/research/researchreports/">course experience questionnaire</a> is administered to graduates some four months after they complete their program. </p>
<p>It has been <a href="http://www.tandfonline.com/doi/abs/10.1080/03075079112331382944?journalCode=cshe20#.VTOf12RViko">rigorously evaluated</a> and several studies have refuted the common criticisms that it favours shallow teachers or soft markers. There is now over 20 years of course experience questionnaire results, providing a valuable sequence of data on university teaching. </p>
<figure class="align-right ">
<img alt="" src="https://images.theconversation.com/files/78689/original/image-20150421-12025-1wszkpj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/78689/original/image-20150421-12025-1wszkpj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=360&fit=crop&dpr=1 600w, https://images.theconversation.com/files/78689/original/image-20150421-12025-1wszkpj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=360&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/78689/original/image-20150421-12025-1wszkpj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=360&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/78689/original/image-20150421-12025-1wszkpj.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=452&fit=crop&dpr=1 754w, https://images.theconversation.com/files/78689/original/image-20150421-12025-1wszkpj.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=452&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/78689/original/image-20150421-12025-1wszkpj.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=452&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px">
<figcaption>
<span class="caption">Students complete satisfaction surveys.</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/yourdon/2681686220">Flickr/Ed Yourdon</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span>
</figcaption>
</figure>
<p>Administered with the course experience questionnaire is the <a href="http://www.graduatecareers.com.au/Research/ResearchReports/GraduateDestinations">graduate destination survey</a>. This reports graduates’ employment, study or other activity four months after graduation.</p>
<p>The <a href="http://www.graduatecareers.com.au/research/surveys/beyondgraduationsurvey/">beyond graduation survey</a> reports on graduates’ activities, outcomes and experiences three years after graduation. </p>
<p>The Australian government also funds a <a href="https://education.gov.au/university-experience-survey">university experience survey</a>. This surveys current students during second semester. </p>
<h2>Statistics</h2>
<p>The Australian government reports numerous <a href="https://education.gov.au/selected-higher-education-statistics-2013-student-data">student statistics</a> to assess universities’ performance on the admission and performance of students from equity groups and on students’ attrition, success and retention.</p>
<p>The Commonwealth also reports data on universities’ <a href="https://www.education.gov.au/selected-higher-education-statistics-2014-staff-data">staff</a>, <a href="https://docs.education.gov.au/category/program-139">financial results</a> and performance. </p>
<h2>My University web site</h2>
<p>Much of this data from student surveys and student and staff statistics are reported for each university program on the government’s <a href="http://myuniversity.gov.au/">My University</a> website. </p>
<p>While the website is reasonably comprehensive and has a function allowing users to compare university programs, there is no evidence that it informs prospective students’ choice of programs extensively. </p>
<p>The site is therefore being replaced by <a href="https://education.gov.au/upholding-quality-quality-indicators-learning-and-teaching">quality indicators for learning and teaching</a>. These will include a new <a href="https://education.gov.au/employer-satisfaction-survey">employer satisfaction survey</a> if grave methodological difficulties can be overcome.</p>
<h2>Excellence in research for Australia</h2>
<p>The <a href="http://www.arc.gov.au/era/era_2012/outcomes_2012.htm">excellence in research for Australia</a> (ERA) assessments are strong evaluations of each university’s research performance in each of 157 detailed fields aggregated into 22 broad fields and eight discipline clusters. </p>
<p>The <a href="http://www.arc.gov.au/era/era_2015/era_2015.htm">2015 assessments</a> are currently being conducted. While the ERA assessments are reasonably rigorous and accordingly strongly influence university research policy, they need some time and expertise to interpret informatively.</p>
<h2>World rankings</h2>
<p>Well-funded universities’ reputations and their demand from international students are influenced strongly by world ranks. There are dozens of world ranks of very variable rigour, coverage and subject matter. There is, for example, a <a href="http://www.gottshall.com/squirrels/campsq.htm">campus squirrel listing</a>, which rates some USA and Canadian campuses by the size, health and behaviour of their squirrels.</p>
<p>Of the prominent world ranks, the most rigorous is Shanghai Jiao Tong University’s <a href="http://www.arwu.org/">academic ranking of world universities</a>. The rank’s relative rigour is achieved at the expense of a narrow focus on research, particularly empirical research. </p>
<figure class="align-left zoomable">
<a href="https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=401&fit=crop&dpr=1 600w, https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=401&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=401&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=504&fit=crop&dpr=1 754w, https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=504&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/78862/original/image-20150422-23627-ricmr1.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=504&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">Which uni ranks best?</span>
<span class="attribution"><a class="source" href="https://www.flickr.com/photos/sidneiensis/12263332156/in/photolist-jFEMpj-6HAE5q-jFEcMC-jFECXh-kFYN4E-pDCGUW-kFUgHg-kFYrLA-kFWyyh-kFYCBJ-kFWaCZ-kFUNBR-jWkJaa-kFVUSr-kFWXhe-jWkscg-fKdiPa-7w9gdp-7w9fbK-aCHyoX-766wgJ-jFCFLM-kFZezu-jWmaoK-4VYVGU-jFEdoC-kFXkQZ-jWmosv-jHLCfF-kFZjPt-aCLfYu-8t59nR-pTVpeL-766BrX-ngXZV1-8ZE79P-9Yiw94-7w9ixi-7842Yd-78438w-76azc1-77Z6gx-783Z87-77Z6WX-7wd6kN-77Z7BM-766Cn2-766D1B-9La8JH-766zPc">Jason James/Flickr</a>, <a class="license" href="http://creativecommons.org/licenses/by/4.0/">CC BY</a></span>
</figcaption>
</figure>
<p>The other two prominent ranks rely on reputation surveys. The rankers publish little information about their surveys, but from what information is available it is clear that they <a href="http://www.insidehighered.com/news/2013/05/29/methodology-qs-rankings-comes-under-scrutiny">fail basic undergraduate social sciences methods</a>.</p>
<p>Half of QS Quacquarelli Symonds’ rank is based on reputation surveys, so it is particularly vulnerable to the <a href="http://link.springer.com/article/10.1007%2Fs10734-010-9339-1#page-1">bias introduced and amplified by reputation surveys</a>.</p>
<p>The Times Higher Education (THE) rank is <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2927316/">too volatile</a>, which is the result of <a href="http://higheredstrategy.com/times-higher-rankings-weak-methodologies-and-the-vastly-overblown-rise-of-asia/">poor methods</a>. Times Higher Education <a href="http://rankingwatch.blogspot.co.uk/2014/11/the-times-higher-education-rankings.html">changed its method and data supplier</a> again last year, so changes from year to year are largely uninformative. Unless THE improves on its previous practice, its 2015 ranks shouldn’t be compared with previous years.</p>
<h2>Evidence of university standards</h2>
<p>Much as universities rail against the current league tables and indicators of quality, they boast about their results in shoddy league tables and invest very little in developing alternatives. Universities should provide publicly verifiable evidence of the standards of their graduates. Instead there are numerous indications that <a href="http://www.universityworldnews.com/article.php?story=20100226130940456">standards vary markedly between years, qualifications and institutions</a>. </p>
<p>One possible measure of the standards of university graduates is the OECD’s <a href="http://www.oecd.org/education/skills-beyond-school/testingstudentanduniversityperformancegloballyoecdsahelo.htm">assessment of higher education learning outcomes</a> (Ahelo), which trialed exams for graduates in economics and engineering. The trial involved 23,000 students from 248 higher education institutions in 17 countries, including Australia. </p>
<p>The results from the Ahelo trial were mixed and the project is controversial. That might be expected from a project that was initially presented as higher education’s version of the international school-aged test <a href="http://www.oecd.org/pisa/aboutpisa/">PISA</a> (Program for International Student Assessment). </p>
<p>Nonetheless, Ahelo still has potential and offers the best prospect of improving on current methods for measuring universities’ success. It would be the only internationally comparative measure of teaching or learning, and it would assess graduates’ attainment, not just report their experiences or satisfaction.</p>
<h2>Trust us - we are the experts</h2>
<p>Australia is probably more advanced than most in measuring its universities’ success. However, it suffers with many other countries in relying excessively on numerical measures and their associated ranks. </p>
<p>There is a tension between peoples’ understandable interest in a simple indicator of quality and even the most rigorous indicator’s distortion by “<a href="https://en.wikipedia.org/wiki/Campbell%27s_law">Campbell’s law</a>”, which means the more an indicator becomes a target, the more it measures targeting rather than performance and the more it distorts performance. </p>
<p>In the absence of an array of better indicators, particularly of teaching and learning, it would be better to rely less on simple indicators and more on expert judgement.</p>
<hr>
<p><strong><em>The Conversation is running a series on “What are universities for?” looking at the place of universities in Australia, why they exist, who they serve, and how this is changing over time. Read other articles in the series <a href="https://theconversation.com/au/topics/what-are-universities-for">here</a>.</em></strong></p><img src="https://counter.theconversation.com/content/40071/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Gavin Moodie is an adjunct of RMIT which is ranked by the various measures discussed in this piece.</span></em></p>Currently universities have a vast array of measures they use to gauge how successful they are. Most of the measures have a lot to do with prestige and not much to do with the outcomes of their graduates or the quality of the education their students receive.Gavin Moodie, Adjunct professor, RMIT UniversityLicensed as Creative Commons – attribution, no derivatives.tag:theconversation.com,2011:article/361292015-01-19T19:26:07Z2015-01-19T19:26:07ZRating your professor: five myths about university teaching quality<figure><img src="https://images.theconversation.com/files/68789/original/image-20150113-23804-8aye9k.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=496&fit=clip" /><figcaption><span class="caption">People are happy to say university teachers are not good teachers, but the students seem to think otherwise.</span> <span class="attribution"><a class="source" href="https://www.flickr.com/photos/kaptainkobold/98688066">Alan/Flickr</a>, <a class="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a></span></figcaption></figure><p>Prospective students, parents of prospective students, and taxpayers deserve to know about the quality of teaching in our universities. But how do you measure teaching quality? Based on student results? But results depend on the cohort and how hard the subject is, so that’s hardly fair.</p>
<p>One measure that does exist is student feedback, but the relevance of the data is often hotly debated. A recent article in <a href="https://theconversation.com/students-dont-know-whats-best-for-their-own-learning-33835">The Conversation</a> argued that students don’t know what teaching is good for them and questioned whether such surveys should be conducted at all.</p>
<p>While I agree that student evaluations should be used and interpreted with caution, I firmly believe that universities are best served by collecting feedback from students and making it as freely available as possible. The benefits outweigh the downsides.</p>
<figure class="align-right zoomable">
<a href="https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip"><img alt="" src="https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=237&fit=clip" srcset="https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=600&h=840&fit=crop&dpr=1 600w, https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=600&h=840&fit=crop&dpr=2 1200w, https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=600&h=840&fit=crop&dpr=3 1800w, https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=754&h=1056&fit=crop&dpr=1 754w, https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=30&auto=format&w=754&h=1056&fit=crop&dpr=2 1508w, https://images.theconversation.com/files/69326/original/image-20150119-2749-z8vmub.jpg?ixlib=rb-1.1.0&q=15&auto=format&w=754&h=1056&fit=crop&dpr=3 2262w" sizes="(min-width: 1466px) 754px, (max-width: 599px) 100vw, (min-width: 600px) 600px, 237px"></a>
<figcaption>
<span class="caption">University teaching is not poor, nor do the students think so.</span>
<span class="attribution"><span class="source">Shutterstock</span></span>
</figcaption>
</figure>
<p>The data could help us to recognise and reward good teaching in the same way we reward research. Releasing the data may also increase public and government trust in higher education. Now that social media allow informal student feedback to be <a href="http://www.ratemyprofessors.com/">published</a> it will be important that authoritative sites also exist where the formal methodologies as well as the raw data are provided. </p>
<p>Personally, I have found student comments helpful in shaping my own lectures and in my role as Dean of Science at UNSW I see the student evaluation surveys for all science courses (for subjects like first-year maths, second-year organic chemistry, or third-year molecular genetics). The students provide this feedback via anonymous online surveys and they do this before they get their grades.</p>
<p>While I can’t publish the results of the individual surveys, I can share some of the aggregated results with you, results that bust several myths about university teaching. </p>
<h2>Myth 1: University teaching is poor</h2>
<p>While not everyone thinks university teaching is poor, when past graduates look back on their experiences they tend to remember the extremes, the very bad or very good lecturers, and present-day student chatrooms aren’t shy about highlighting poor performances. <a href="http://www.theage.com.au/national/education/research-needed-on-whether-university-lecturers-could-do-with-teaching-lessons-20140731-zytv9.html">Newspapers</a> and <a href="http://www.hackcollege.com/blog/2012/01/30/classroom-confidential-when-lectures-go-bad.html">online media</a> have long pronounced the lecture is dead because teachers can’t hold the attention of students for any prolonged period <a href="http://www.adelaide.edu.au/professions/pedagogical-possibilities/sgde/strategy/experience/lectures/">due to poor teaching</a>.</p>
<p>Nevertheless, looking at the student feedback data in its entirety shows things are more often good than bad. In response to the question,</p>
<blockquote>
<p>overall were you satisfied with the quality of the course?</p>
</blockquote>
<p>we found that 91% of students expressed agreement. Things are not perfect since a handful of courses had less than 50% approval, but this means we can focus on improving these courses and things should get better still.</p>
<h2>Myth 2: Good researchers make poor teachers and vice versa</h2>
<p>The stereotype of the gifted but socially inept professor, unable to communicate, is <a href="https://cdn.auckland.ac.nz/assets/education/hattie/docs/relationship-between-research-and-teaching-(2004).pdf">nearly as widespread</a> as the opposite view that teachers are unable to do research, encapsulated in the cliché - those who can do, do, and those who can’t do, teach.</p>
<p>Our university data, however, fails to show any link, positive or negative, between ability in research and in teaching. On the one hand many good researchers are also good communicators - using their skills not only in teaching but in convincing grant funding bodies to support their research. On the other hand some staff who are less active in research dedicate a lot of their time to teaching, and it shows in their achievements.</p>
<h2>Myth 3: Poor teaching is tolerated because only research counts</h2>
<p>As government research assessment exercises have become dominant especially in the UK, New Zealand and Australia, the perception that <a href="http://insocrateswake.blogspot.com.au/2010/05/teaching-counts-for-nothing.html">only research counts</a> has spread. This together with the fact that universities don’t name and shame individual staff who have received poor student feedback leads some to believe <a href="http://universitydiary.wordpress.com/2014/01/28/do-we-recognise-good-teaching-in-our-universities/">universities care little about teaching</a>, for either <a href="https://www.insidehighered.com/news/2009/07/20/polisci">appointments or promotions</a>. </p>
<p>In terms of appointments, fierce competition for academic jobs means that those with promise in both teaching and research usually secure jobs. Promotion policies now routinely emphasise strong performance in all areas of academic life for promotion. Nevertheless the difficulty in agreeing on measures of teaching excellence and the secrecy that surrounds student evaluations mean that it is difficult to bust the myth that teaching does not count.</p>
<h2>Myth 4: Student evaluations of teaching only encourage showmanship</h2>
<p>There are many who will tell you student evaluations are little more than a <a href="http://www.jstor.org/discover/10.2307/40374335?sid=21105602542903&uid=2&uid=4&uid=3737536">popularity contest</a>, and therefore it is the <a href="http://teachingcenter.wustl.edu/Journal/Reviews/Pages/Research-Course-Evaluations.aspx#.VLRramSUdOx">charisma of the teacher and not the quality of the lesson</a> that is being scrutinised.</p>
<p>But our student feedback data shows strong student evaluations correlate well with other indicators of teaching excellence, such as peer evaluations and institutional or government teaching awards. Students are good arbiters of quality. Perhaps unsurprisingly students care about the same things administrators and academics value – clarity, organisation, timeliness in feedback and general professionalism in teaching.</p>
<h2>Myth 5: Responding to student surveys will lead to a dumbing down of the curriculum</h2>
<p><a href="http://www.owl232.net/sef.htm">Fear of declining standards</a> is used to argue against student satisfaction surveys lest academics avoid challenging material and courses collapse into cheap popularity contests where every student is given high marks in order to ensure positive feedback.</p>
<p>But provided the surveys are done properly - anonymously and before the students know their marks - the risk that easy courses and soft marking will be offered to boost student evaluations is diminished. Nor have I seen any clear declines in standards. On the contrary, as international competition increases and more people have access to university, the top students and top courses may well be getting better.</p>
<p>So is university teaching perfect? Of course not – all universities are striving to improve. But it is unfortunate if people believe university teaching is poor, that it has taken second place to research, or that it is not a priority in academic life.</p>
<p>Such myths have arisen partly because most universities don’t make student feedback widely available. In some cases this is a consequence of employee contracts that may set limits on how the data is used and some people believe the whole topic is best avoided.</p>
<p>The tertiary sector in Australia plays a critical role in society, providing not only research but also teaching for the next generation of students. No sector can expect to be exempt from the scrutiny of its stakeholders, and as we academics grade our students, we should also look forward to the ratings they give us for our performances.</p><img src="https://counter.theconversation.com/content/36129/count.gif" alt="The Conversation" width="1" height="1" />
<p class="fine-print"><em><span>Merlin Crossley works for the University of New South Wales. He receives research funding from the Australian Research Council and the National Health and Medical Research Council.</span></em></p>Prospective students, parents of prospective students, and taxpayers deserve to know about the quality of teaching in our universities. But how do you measure teaching quality? Based on student results…Merlin Crossley, Dean of Science and Professor of Molecular Biology, UNSW SydneyLicensed as Creative Commons – attribution, no derivatives.