“Over the last decade, machine learning, which is part of artificial intelligence (AI), has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome.” (Lee Bell, Wired, 2016)
With machine learning, we can also now teach our gestures to a machine using the microsensor and their data are already everywhere. Movements can be recognised, memorised, interpreted and shared through networks. There are many applications in health, sports and especially education in the digital age. A recent article in Le Monde stated:
“Driven by advances in artificial intelligence and natural language processing, voice is slowly establishing itself as the new interface to reach the digital universe.”
Gestures are thus another emerging interface to reach the digital universe.
Billions of sensors
During the last 10 years, microsensors have proliferated and invaded our lives. They constantly detect and measure our movements. We know, but often we do not even notice, that they constantly count our daily steps. Up to 100 measurements per second for each of these billions of sensors in our lives. What is the meaning of all these data? What is their presence in our lives? The answer might be “important, but not that much”, as digital technology is today primarily built on omnipresent screens. They put such a pressure on our daily lives that they substitute images for the real world, and actually stop the body.
With the data from motion microsensors, processed by machine learning in real time, we try to get back to the real world: can we imagine a digital world without screens and keyboards, which would lead us to love to communicate through movements and gestures using our mobile phones? Can we be all dancers in the digital age? Can this amazing image be made real by the alliance of sensors and of machine learning to process their data in real time and put them in our hands?
A new tool for home rehabilitation
Rehabilitation was our first experience. Entering these questions through this application was the obvious thing to do. How can you best accompany a patient who returns home after a knee- or hip-joint replacement? With the help of an increasingly individualised technological environment, built up through close collaboration with the nursing staff. This is the heart of collaboration with the trauma department of the Lariboisière Hospital in Paris. The application that was developed is based on the link between sensors, intelligent data processing and the design of the patient interface. In practice, a small electronic plate on which the smartphone sensors and a wi-fi are put together is sewn into the sock or the garment.
Witnessed by a physiotherapist, the patient teaches the machine the limit of his or her postures, beyond which a warning signal is given. This can thus practice in the patient’s daily life, under the watchful gaze of this technology. Individualisation is an essential aspect of this e-coach. Show me how you move…
Movement measurement and data processing have a dual benefit for the patient. The real-time control of the signal increases his or her confidence zone. The patient also knows that the data are transmitted and analysed by caregivers if the situation requires it.
Enhancing our gestures to learn them better?
The video above shows how it is possible to associate any type of sign or digital information with gestures and movements. This example is based on the work of Frédéric Bevilacqua group at IRCAM (Institut de Recherche et Coordination Acoustique Musique) in Paris. It is no surprise that IRCAM, funded by composer Pierre Boulez, is a key player in this research. In the list of our finest and most difficult gestures to acquire, the ones imposed by musical instruments are among those at the top. Can you learn to make music by moving a smartphone?
Here you can try the dialogue with the learning machine. This is the web application COMO produced by IRCAM. Learn your own gestures to your smartphone and play with the sounds. From IRCAM, this technology has spread to the Center for Research and Interdisciplinarity (CRI) at the Université Paris Descartes, a hub for experimentation and pedagogical innovation. In a “motion lab”, CRI Paris is tackling “the question of gesture as an emerging interface to reach the digital universe” from the point of view of learning and education.
How do we learn today? In fact, how can we learn to learn as the world is changing so fast and profoundly? How to build this learning society? The answer from Francois Taddei, director of the CRI: by working together with kindness and by keeping a sense of astonishment when looking at the world. Using these technologies for lifelong learning in a world that has become massively digital is exciting but difficult. In fact, no one is here an expert, as this technological breakthrough takes us beyond the usual patterns of thought. But there is good news: as we all move, everyone is welcome on the boat…
From scientists to contemporary dance
It took me some time to understand why researchers and students coming from contemporary dance, computer science and of course, robotics, design, pedagogy, textile, physics, mechanics, anthropology, health, sports, music, sound and more are gathering around IRCAM, CRI Paris and the ENSCI Les Ateliers design school. In the workshops there are all these people we explore scenarios to use these fascinating but also confusing tools. For example, I once learned a gesture to my smartphone, when I tried to replicate it the machine didn’t recognise it. A dancer who was with us took my smartphone and mimicked my gesture. They both were quickly able to identify “my gesture”.
If the gesture can become an interface to reach the digital universe, then a new way of moving appears. It leads us to pay close attention to movements that we make every day without thinking about them. In Le Geste et la Parole (1965), written by André Leroi-Gourhan in a then non-digital world:
“The acrobatics, the balancing exercises, the dance materialize to a large extent the effort of subtraction from normal operating chains, the search for a creation that breaks the daily cycle of positions in space.”
I couldn’t have said it any better. And really, it’s not a surprise that a collaboration with contemporary dance has quickly established itself in this research on gesture and movement coming from technology and science.