Volltext-Downloads (blau) und Frontdoor-Views (grau)

Multimodal space representation driven by self-evaluation of predictability

  • PROPRE is a generic and modular neural learning paradigm that autonomously extracts meaningful concepts of multimodal data flows driven by predictability across modalities in an unsupervised, incremental and online way. For that purpose, PROPRE consists of the combination of projection and prediction. Firstly, each data flow is topologically projected with a self-organizing map, largely inspired from the Kohonen model. Secondly, each projection is predicted by each other map activities, by mean of linear regressions. The main originality of PROPRE is the use of a simple and generic predictability measure that compares predicted and real activities for each modal stream. This measure drives the corresponding projection learning to favor the mapping of predictable stimuli across modalities at the system level (i.e. that their predictability measure overcomes some threshold). This predictability measure acts as a self-evaluation module that tends to bias the representations extracted by the system so that to improve their correlations across modalities. We already showed that this modulation mechanism is able to bootstrap representation extraction from previously learned representations with artificial multimodal data related to basic robotic behaviors [1] and improves performance of the system for classification of visual data within a supervised learning context [2]. In this article, we improve the self-evaluation module of PROPRE, by introducing a sliding threshold, and apply it to the unsupervised classification of gestures caught from two time-of-flight (ToF) cameras. In this context, we illustrate that the modulation mechanism is still useful although less efficient than purely supervised learning.

Export metadata

Additional Services

Share in Twitter Search Google Scholar


Author:Mathieu Lefort, Thomas Kopinski, Alexander Gepperth
Parent Title (English):4th International Conference on Development and Learning and on Epigenetic Robotics
Document Type:Conference Proceeding
Year of Completion:2014
Release Date:2019/06/27
Institutes:Fachbereich 1 - Institut Informatik
DDC class:000 Allgemeines, Informatik, Informationswissenschaft / 000 Allgemeines, Wissenschaft
Licence (German):License LogoNo Creative Commons