UK United Kingdom

What eye tracking tells us about the way we watch films

In Blade Runner (1982), Roy Batty, the leader of an outlaw group of Nexus-6 replicants, undertakes a quest to “meet his maker”. When he encounters Chew, the genetic designer of his eyes, he mocks: “Chew…

If only you could see what I’ve seen with your eyes … Jodi Sita

In Blade Runner (1982), Roy Batty, the leader of an outlaw group of Nexus-6 replicants, undertakes a quest to “meet his maker”. When he encounters Chew, the genetic designer of his eyes, he mocks: “Chew, if only you could see what I’ve seen with your eyes.”

The exciting news is that eye-tracking technology now allows us to objectively track and record what viewers are looking at, for how long, and in what ocular sequence.

Eye trackers work by reflecting infrared light onto the eye, which is then captured by a sensor. The sensor allows the captured movements to be tracked and specialised software overlays these movements over what was being viewed.

The eye fixates when gathering information and move between points in saccades (fast eye movements) (see image 1 below).

Image 1. Jodie Sita

Where we attend to, and what draws our attention, is of interest as this process is driven both by our conscious intentions and by unconscious reactions to visual stimuli such as objects, textures, and movements.

Looking at gaze behaviour can hint at the cognitive drives behind them. Even more interesting is the ability to look at the gaze behaviour of groups of people in relation to similar stimuli.

Image 2. Jodie Sita

This is made possible through the use of heat maps (see image 2 above), which effectively show the weighting of all the viewing occurring and gaze plots, which show the location of the fixations made as well as the sequence in which they were made (see image 3 below).

Image 3. Jodie Sita

What we’ve done

In our recent studies we used a Tobii x120 eye tracker to record 12 subjects watching scenes from three films and analysed their fixation data using Tobii Studio Software.

We are part of a group of multidisciplinary researchers that together form the Melbourne-based Eye Tracking and the Moving Image group. This group has recently been analyzing data collected from three films:

Data for this study was collected and early analysis of fixations completed with a focus on narrative development, characterisation, and sound in relation to gaze patterns; further in-depth analysis is planned over the next three months.

The three films were chosen because they had very strong and directed storylines, interesting soundscapes, and allowed us to consider whether there were differences in the way live action and animation might be viewed.

The fact that two of the films were made by the same director is also of real interest, although not the focus of the research presently.

In terms of UP we are looking at what the eyes track and focus upon as viewers watch the film’s opening montage sequence:

This is where Carl and Ellie, the two central characters, get married, fall deeper in love, and live out their lives together, until Ellie eventually dies. It is a short but emotive sequence, set to a subtle musical score, but without a word of dialogue spoken.

We are finding that hot spots emerge around the character’s mouths, as if our viewers are conditioned to look for identification through the way a film’s central characters “speak”, and are searching for narrative clarification through dialogue exchanges that actually never emerge.

We are also finding that objects and motifs that actively move the story along and define character mood are picked out of the mise-en-scène and gazed at even when the scene is fluid and action is taking place across the filmic space (see image 3 above and 4 below).

Image 4. Jodie Sita

Our eyes, then, seem to be active in finding emotive objects even when they are found in a busy scene. There is a caveat here of course: the “extra” attention to significant objects may well have been because the scenes were free of dialogue.

Image 5. Jodie Sita

In terms of both Monsters Inc. and Saving Private Ryan, we created a control condition in which our viewers watched the two scenes with and without sound. Both scenes were “action” sequences, although the Ryan Beach Landing scene is hyper-realistic, infused with photo-realism and documentary-style aesthetics:

Warning: this clip contains graphic images.

So far our results have been a little inconclusive with only slight variance in where viewers look when the scenes are played with and without sound. But, with regards to Monsters Inc. we have found when the sound is off, viewers actually migrate slightly away from the character’s mouths to focus on their eyes, and to objects that are pregnant with narrative information, such as the red warning light on the door (see images 5 and 6 below).

This, we suggest, is because the viewer is searching for non-verbal signs to confirm what might be taking place and because the eyes, culturally speaking, are where “truth” is to be revealed.

Image 6. Avni Pepe

On one level this contests our results for UP, where the mouth was gazed at; on another, it confirms the way that impressionable objects are intensely gazed at.

Briefly, with Saving Private Ryan we found that when the scene was played without sound, gaze patterns slightly migrated to objects that could be predicted to be key to what might happen next.

That is, there was a greater level of prediction when the sound was off, as viewers searched for visual clues to where the action might “explode” next.

For both Monsters Inc. and Saving Private Ryan we measured pupil dilation, and we found that there was a close correspondence between the presence of sound and larger pupil sizes.

That is, when various elements of sound design were present, pupil enlargement indicated that the viewer was experiencing an increased emotional response, giving weight to the importance of sound in the affecting power of cinema.

So, what have we seen with your eyes? The power of your insight, the way you search for meaning and follow narrative and aesthetic clues, the importance of sound – of hearing and seeing to not only where you look but also to how you feel when you do so.

Join the conversation

8 Comments sorted by

  1. Ken Taylor

    Research Scientist at CSIRO

    I was expecting an interesting story but was disappointed to get to "Our study, which will take place over a three-month period...".

    1. Sean Redmond

      logged in via Facebook

      In reply to Ken Taylor

      Hi Ken, apologies for any confusion here. The 3 month study is with regards to the initial population sample and in relation to some core, targeted AOIs we are presently working with. Our research is on-going, started last year with work on Sherlock, and will continue to develop around new texts, areas of investigation. If you are interested in the wider parameters of our work, drop Jodi or I an email and we would be happy to talk more - best, Sean

    2. Ken Taylor

      Research Scientist at CSIRO

      In reply to Sean Redmond

      I used eye trackers a few years ago on user interface research. I saw the Tobbii tracker at a trade show and it was an unforgettable ambush that has had me thinking ever since they were ideal for the research you describe, so I was curious to see what has been learned.

      The demo started with a dry discussion of the technology while there was an attractive and provocatively dressed couple on the screen. After a few minutes surreptitiously acquiring data they turned on the heat map.

    3. Jodi Sita

      Senior Lecturer, Faculty of Health Sciences at Australian Catholic University

      In reply to Ken Taylor

      Hi Ken, I think I know the image you are referring to- their were male and female difference to where each respectively gazed at on the male and females bodies- interestingly if I am remembering correctly- females checked out the ring finger of the male, their face and chest but went no lower yet when looking at the female they looked at the females like they were comparing them to themselves (sizing the competition up was how it was put), where the men checked out the female physique, without looking to see if she was married, but equally sized up the males. Wonder if that was the same one you saw.

    4. Jodi Sita

      Senior Lecturer, Faculty of Health Sciences at Australian Catholic University

      In reply to Ken Taylor

      Just to qualify- the data for the study has been collected and first run analysis completed- we will be working over the coming few months on more detailed and deeper examination of these findings.

    5. Ken Taylor

      Research Scientist at CSIRO

      In reply to Jodi Sita

      I saw only my own heat map, and wondered how it compared to others, rather than an analysis of the data. The salesman claimed he could easily tell whether the viewer was gay. I also felt the technique required deception to be effective because gaze is socially and unconsciously mediated. That is, people behave differently without even realising it, when they know they are being watched. In my case I hadn't been consciously aware of looking at the pictures at all.

      On the basis of the analysis you mention, an interesting approach might be to group the data by sex, age etc. but I wonder how you can avoid the influence of social mediation.

  2. Jeff Payne

    PhD in Political Science and Masters in Public Policy

    Interesting article for us ignorant of this kind of research. Reading this fascinating area of work I can't help but to be concerned how this kind of technology will be deployed in the future. Advertisers will use it to make us buy more and political parties will use it to make us vote for them. It will be used as part of the ongoing attempt to sidestep our discerning, conscious areas of the brain and work on our 'subconscious' (if this idea is still in use) to make us do things. I guess we need to know what is happening and try to develop strategies to deal with it all. Enjoyable article.

    1. Sean Redmond

      logged in via Facebook

      In reply to Jeff Payne

      Thanks Jeff, we enjoyed writing it very much. There has and there is work being carried out using eye tracking technology to discover how consumers scan and gaze at adverts! Jodi is also doing some fascinating work with eye trackers and sports coaching. One of my own interests is how viewers look at 'spectacle' in science fiction film.