EE Seminar: Variational Maximum Mutual Information for Reducing Mixture Models
Speaker: Yossi Bar-Yosef
Ph.D. student under the supervision of Prof. Yuval Bistritz
Wednesday, June 14th, 2017 at 15:00
Room 011, Kitot Bldg., Faculty of Engineering
Variational Maximum Mutual Information for Reducing Mixture Models
Abstract
The Gaussian mixture model (GMM) is a very powerful parametric modeling tool for representing complex data distributions. Consequently, GMMs are widely used in various statistical machine learning applications. In many cases, high-order mixtures (mixtures with large number of components) are used to get an adequate representation of the data, leading to heavy computational and storage demands. This often raises a need to approximate the high-order models by models with fewer components.
We propose a novel approach to this problem based on a parametric realization of the maximum mutual information (MMI) criterion and its approximation by an analytical expression named variational-MMI (VMMI). The suggested VMMI objective can then be maximized by analytically tractable algorithms that optimize the parameters of the requested reduced models. Differently from previous methods that produce a reduced model for each class independently, without considering the relations between the classes, the VMMI optimization produces models with improved discrimination ability - a desirable goal for any classification problem.
The use of VMMI optimization for GMM reduction was evaluated by experiments carried out with two speech related classification tasks: phone recognition and language recognition. For each task, the VMMI-based parametric model reduction was compared to other state-of-the-art reduction methods. Experimental results show that the new discriminative VMMI optimization produces models that significantly outperform the classification ability of comparable reduced models obtained by standard non-discriminative methods.
