Auracle: how are the salient cues situated in audiovisual content?

Project leaders:
Christian Frisson, Nicolas Riche

Participants:
Antoine Coutrot, Charles-Alexandre Delestage, Stephane Dupont, Onur Ferhat, Sidi Mahmoudi, Matei Mancas, Parag Kumar Mital, François Rocca, Alexis Rochette, Willy Yvart

Abstract:
This project aims at studying the mutual influence of audio and visual cues on spectators of audiovisual media content: feature films, animation movies, video games recordings by undertaking an eye tracking user experiment on a database of such content. Based on the results, models and algorithms of the relation between sound and image may be refined and their predictions compared with actual gaze fixations recorded from participants. The media and fixations database will be released for free. The results of such analyses may serve to breed new crossover media genres featuring sound and moving images. (More info)

Results (new)

A character animation authoring platform for the 99%

Project leaders:
Alexis Heloir, Fabrizio Nunnari

Participants:
Roberto Pulisci, David Tavarez

Abstract:
Animated characters are a paramount element in video games, where movement quality and gesture repertoire has now become an intrinsic characteristic of the essence of a character, at the same level as appearance, story background and skills. Animated characters might also be a privileged support of the narrative: in recent years, some media organizations have begun to use animations in news reports. Animated characters also have the capability to support the expression of deaf individuals on the internet, by allowing them to express opinions anonymously in their primary language: Sign Language. All these applications: character customization, animated narrative and sign language generation – share a common requirement: they all need a convenient and straightforward character animation authoring system. In this project, we would like to explore the capabilities of the new Kinect device (v.2), combined with the Leap motion for capturing hand animation and improving our existing character animation authoring framework. Secondly, we would like to explore new interaction metaphors enabling Novices and Deaf users to author animations. (More info)

Results (new)

Exploring stylistics of expert gestures and singing voice with motion capture and reactive statistical mapping

Project leaders:
Nicolas d'Alessandro, Joëlle Tilmanne, Sotiris Manitsaris

Participants:
Maria Astrinaki, Radhwan Ben Madhkour, Huseyin Cakmak, Eva Coupete, Kevin El Haddad, Alina Glushkova, Edgar Hemery, Sohaib Laraba, Alexis Moinet, Gueorgui Pironkov, Thierry Ravet, Yasemin Timar, Jerome Urbain, Christina Volioti

Abstract:
For several years, the need for accurate and real-time gesture following and recognition techniques is increased in the context of natural user interaction. Indeed the development of affordable markerless motion capture techniques – such as depth cameras – has allowed to design new kinds of applications where the continuous tracking and understanding of gestural behaviors and styles may greatly improve usability and user experience. Particularly the fields of intangible heritage preservation and performing arts nowadays require going beyond the simple ideas of recording and explicit mapping. In this project, we aim at prototyping a playing booth for the use case of piano playing. An instrumented and motion-captured keyboard will allow the analysis of expert piano playing in real-time and the recognition of specific gestures, but also of the ongoing playing style, i.e. an distance measure with existing teaching schools and emotions. Our real-time stylistic gesture recognition relies on advanced HMM-based techniques that we have exported from the speech processing research. When the player can be positioned in a stylistic space, we aim at using this information to drive a singing synthesizer that also exposes stylistic information, in the form of singing techniques. Our platform will allow the testing of various stylistic mappings. (More info)

MILLA - Multimodal interactive language learning agent

Project leaders:
Nick Campbell, Joao Cabral, Eamonn Kenny, Emer Gilmartin

Participants:
Fasih Haider, Mina Kheirkhah, Andrew Murphy, Neasa Ni Chiarain, Thomas Pellegrini, Odei Rey, Sree Ganesh Thottempudi

Abstract:
Learning a language involves the acquisition and integration of a range of skills. A human tutor aids learners by (i) providing a framework of tasks suitable to the learner's needs; (ii) continuously monitoring learner progress and adapting task content and delivery style; (iii) providing a source of speaking practice and motivation. The goal of this project is to create a multimodal dialogue system which will provide some of the advantages of a human tutor which are not normally encountered in self-study material and systems. We envisage a system which combines language learning tasks with a chat module (chatbot) providing fun practice in the target language. Speech recognition will be used to monitor learner pronunciation and provide error detection. We would incorporate multimodal sensors in this system to aid in monitoring learner interest and affect. The information gathered by these sensors along with ASR (automatic speech recognition) results can then be used to inform the virtual tutor's output and behavior, tailoring the activities to the individual learner's needs. We will provide a dialogue platform incorporating TTS (Text-To-Speech) synthesis, ASR (automatic speech recognition), cameras, and biosensors. We can also provide sample language activities but welcome participants' own ideas. The participants will work together on design, implementation, and testing of the system. (More info)

Results (new)

Online platform for obtaining personalized synthetic voices

Project leaders:
Daniel Erro, Inma Hernaez, Eva Navas

Participants:
Agustin Alonso, Haritz Arzelus, Nguyen Quy Hy, Igor Jauk, Carmen Marariños, Ruben Perez-Ramon, Martin Sulir, Xiaohai Tian, Xin Wang, Jianpei Ye

Abstract:
Speech synthesis technologies have evolved during the last decade from selection and concatenation paradigms to statistical parametric ones. The main advantage of hidden Markov model (HMM) based speech synthesis is its enormous flexibility for speaker/style adaptation, though it exhibits also some others: low footprint, smooth synthetic signals without annoying discontinuities, etc. The availability of an open source statistical parametric speech synthesis system, HTS, has played a key role in this recent technological evolution. Statistical parametric speech synthesis has enabled lots of new applications that were not possible in the previous technological states: voice reconstruction for people with several speech impairments, personalized speech-to-speech translation, noise-robust speech synthesis, etc. The primary goal of this project is the design and development of a web interface that allows non-expert users to get their own personalized HTS-compatible synthetic voice. This interface could be used, for instance, by people suffering from degenerative pathologies before the symptoms are visible or before surgery to create a "backup" of their voice. To reach a broad audience, the system is intended to be multilingual as far as possible. (More info)

Results (new)

Quantification of the alteration in the resting state networks for measuring brain pathologies and diseases [CANCELLED]

Project leaders:
Ibai Diez, Asier Erramuzpe, Sebastiano Stramaglia, Jesus M. Cortes

Abstract:
Resting State Networks (RSNs) are the result from the coherence across brain regions in the high-amplitude ultra-slow (0.1 Hz) fluctuations in the Blood-Oxygen-Level-Dependent (BOLD) signal acquired with resting state functional magnetic resonance imaging (rs-fMRI). The RSNs are universal, meaning that, beyond individual brain differences, they emerge at rest in all healthy subjects. This universal behavior in healthy brains has pushed forward the strategy for quantification of the RSNs alterations in brain pathological conditions to find for new neuroimaging markers for both mental and cognitive disorders. Indeed, alterations in the RSNs have been found in patients with deficit of consciousness after brain traumatic injury, in Alzheimer disease and schizophrenia. Based on these results, it is expected that RSNs alterations occur as well for other brain disorders and diseases. This is the main motivation of this project, given an input rs-fMRI data of a specific patient, to develop new and efficient methods capable to quantify the precise amount of alteration (respect to control) for each of the resting networks. (More info)

The smart attentive room - Can we measure in a noninvasive way, the attention behavior that a user has in a room? [CANCELLED]

Project leaders:
Julien Leroy, Matei Mancas

Abstract:
The objective of this project is the design and implementation of a lowcost tracking system to simplify the implementation of smart room and behavioral analysis environments. To achieve these objectives the proposed methodology is to design an efficient, easy to install, inexpensive, multicamera 3D system and allowing in realtime behavioral information extraction. One of the features of the proposed system is to be able to measure in real time the attention caracteristics of a subject: measuring the elements that attract the attention of a person in a room through the scene analysis but also the reactions of users' attention, for example the phenomena of joint attention. This project will address two issues related to the analysis of attention: (i) modeling human attention, through computational attention mechanisms for the detection and measurement of protruding elements in an environment; (ii) Detection and analysis of attentive behavior in subjects. The ultimate goal would be to provide measurement solutions to these two problems and determine how these information can be correlated. (More info)