The headlines
The Los Angeles Times: People trust eyes - not ears - when judging musicians
Classic FM: Classical singers judged by actions not voice
Nature: Musicians’ appearances matter more than their sound
The story
If you wanted to pick out the musician who won a prestigious classical music competition would you listen to a clip of them playing or watch a silent video of them performing the same piece of music?
Most of us would go for an audio clip rather than video, and we’d be wrong. In a series of experiments, Chia-Jung Tsay from University College London, showed that both novices and expert musicians were better able to pick out the winners when they watched rather than listened to them.
The moral, we’re told, is that how you look is more important than how you sound, even in elite classical music competitions.
What they actually did
Dr Tsay, herself a classically trained musician, used footage from real international classical music competitions. She took the top three finalists and asked volunteers to pick out the real winner - with a cash incentive - by looking at video without sound, sound without video, or both.
Over a series of experiments she showed that people think that audio will be more informative than video, but actually people are able to pick the real winner when watching video clips. But they aren’t able to do this when listening to audio clips (these test subjects only perform at the level of chance). The shocking thing is that when people get sound and video clips, which notionally contain more information, they still perform at chance. The implication being that they would do better if they could block their ears and ignore the sound.
Follow up experiments suggested that people’s ability to pick winners depended on their being able to pick out things associated with “stage presence”. A video reduced to line drawings, designed to remove details and emphasise motion, still allowed people to pick out winners at an above chance rate. Another experiment showed that asking people to identify the “most confident, creative, involved, motivated, passionate, and unique performer” tallied with the real winners.
How plausible is this?
We’re a visual species. How things look really matters, as everyone who has dressed up for an interview knows. It’s also not uncommon for us to be misled into believing that how something looks isn’t as important as it really is (here’s an example: judging wine by the labels rather than the taste).
What is less plausible is the spin put on the story by the headlines. We all know that looks are important, but how can they really be more important than sound in a classical music competition? The most important thing really is the sound, but this research resonates with a popular cliché about how irrational we are.
Tom’s take
The secret to why these experiments give the results they do is in this detail: the judgement that people were asked to make was between the top three finalists in prestigious international competitions. In other words, each of these musicians is among the best in the world at what they do. The best of the best even.
In all probability there is a minute difference between their performances on any scale of quality. The paper itself admits that the judges themselves often disagree about who the winner is in these competitions.
The experimental participants were not scored according to some abstract ability to measure playing quality, but according to how well they were able to match real-world competition outcome.
The experiments show that matching the judges in these competitions can be done based on sight but not on sound. This isn’t because sight reveals playing quality, but because sight gives the experimental participants similar biases to the real judges. The real expert judges are biased by how the performers look - and why not, since there is probably so little to choose between them in terms of how they sound?
This is why the conclusion, spelt out in the original paper, is profoundly misleading: “The findings demonstrate that people actually depend primarily on visual information when making judgements about music performance”. It remains completely plausible that most of us, most of the time, judge music on how it sounds, just like we assumed before this research came out.
In ambiguous cases we might rely on looks over sounds - even the experts among us. This is a blow to musicians who thought it was always just about sound - but isn’t a revelation to the rest of us who knew that when choices are hard, whether during the job interview or the music competition, looks matter.
Read more
The original paper: Sight over sound in the judgment of music performance. Tsay, C-J (2013), Proceedings of the National Academy of Sciences
Special mention for the BBC and reporter Melissa Hogenboom who were the only people, as far as I know, who managed to report this story with an accurate headline: Sight dominates sound in music competition judging
The interaction between the senses is an active and fascinating research area. Read more from the Crossmodal Research Laboratory at the Univeristy of Oxford and Cross-modal perception of music network at the University of Sheffield