EE Seminar: RNN Fisher Vectors for action recognition

~~
Speaker: Gil Sadeh
M.Sc. student under the supervision of Prof. Lior Wolf and Dr. Benny Applebaum 

Wednesday, May 18th, 2016 at 15:30
Room 011, Kitot Bldg., Faculty of Engineering

RNN Fisher Vectors for action recognition
Abstract

Recently, Recurrent Neural Networks (RNNs) have had considerable success in classifying and predicting sequences. Additionally, the Fisher Vector (FV) encoding has been widely used for pooling local features. We present the RNN-FV, which is a novel pooling method designed especially for sequential features. The methodology we use is based on FVs, where the RNNs are the generative probabilistic models, instead of the commonly used Gaussian Mixture Model (GMM), and the partial derivatives are computed using backpropagation. This proposed method is applied on sequential feature representation of videos, to achieve a new, fixed-length, discriminative video representation. We also explore different sequential feature representations of videos on which we apply our proposed method. Using our new RNN-FV based video representations, state of the art results are obtained in the task of video action recognition on two challenging datasets, UCF101 and HMDB51. We also demonstrate how to exploit the fact that the RNN is trained in an unsupervised manner in terms of the action labels, and show that training the RNN on one dataset and testing on another does not reduce performance significantly, and state-of-the-art results are achieved while using this transfer learning approach as well. We also show another surprising transfer learning result, from the task of image annotation to the task of video action recognition, which additionally improved our results.