Multimodal Deep Learning (Jiquan Ngiam et al., 2011 ICML)
Motivation - Learn features over multiple modalities (ex. audio + video). - Human understands speech with audio+visual information. McGurk effect: visual /ga/ with a voiced /ba/ is perceived as /da/ by most subjects. (McGurk, H. and MacDonald, J. Hearing lips and seeing voices. Nature, 264(5588):746–748, 1976.) Learning architecture - Input: Audio + Visual (lip motion) of single letter & digit. - Find correlation between viseme and phoneme, not raw visual and audio. 1. Sparse RBM (Figure 2-a, b) Baseline. Also use as pretraining models. Informally, these models transform raw data into viseme and phoneme. 2. Bimodal DBN (Figure 2-d) By using RBM pretraining models, find correlation between viseme and phoneme. 3. Single Deep Autoencoder (Figure 3-a) For Bimodal DBN, there is no objective and it is hard to train when only single modality appears. Single Deep Autoencoder solved these two issues, focused on training with single modality. This model uses a RBM pretrain...