EE Seminar: On the Generalization Properties of Deep Neural Networks
(The talk will be given in English)
Speaker: Prof. Miguel Rodrigues
University College London (UCL)
Sunday, November 12th, 2017
15:00 - 16:00
Room 011, Kitot Bldg., Faculty of Engineering
On the Generalization Properties of Deep Neural Networks
Abstract
Deep neural networks (DNNs) – which consist of a series of non-linear transformations whose parameters are learned from training data – have been achieving state-of-the-art results in a wide range of applications such as computer vision, automatic speech recognition, automatic speech translation, natural language processing, and more. However, these remarkable practical successes have not been accompanied by foundational contributions that provide a rationale for the performance of this class of algorithms.
This talk concentrates on the characterization of the generalization properties of deep neural network architectures. In particular, the key ingredient of our analysis is the so-called Jacobian matrix of the deep neural network that defines how distances are preserved between points at the input and output of the network.
Our analysis – which applies to a wide range of network architectures – shows how the properties of the Jacobian matrix affect the generalization properties of deep neural network; it also inspires new regularization strategies for deep neural networks. Finally, our contributions also bypass some of the limitations of other characterizations of the generalization error of deep neural networks in the literature.
Our insights are also supported by a number of experimental results on the MNIST, CIFAR-10, LaRED, and ImageNet datasets.
This represent joint work with Jure Sokolic (UCL), Raja Giryes (TAU), and Guillermo Sapiro (Duke U.).
BIO
Dr. Rodrigues is a Reader in Information Theory and Processing with the Department of Electronic and Electrical Engineering, University College London. He was previously with the Department of Computer Science, University of Porto, Portugal, rising through the ranks from Assistant to Associate Professor, where he also led the Information Theory and Communications Research Group at Instituto de Telecomunicações – Porto.
He obtained his Licenciatura degree in Electrical and Computer Engineering from the University of Porto, Portugal and the PhD in Electronic and Electrical Engineering from University College London, UK. He then carried out postdoctoral research work both at Cambridge University, UK and Princeton University, USA. He also held visiting research and academic appointments at Cambridge University, Princeton University and Duke University from 2007-2016.
Dr. Rodrigues’ most relevant contributions have ranged from the information-theoretic analysis and design of communications systems, information-theoretic security, information-theoretic analysis and design of sensing systems, and, more recently, foundations of machine learning and deep learning problems. His work, which has led to
over 150 papers in the leading journals and conferences in the field, has also been honoured with the IEEE Communications and Information Theory Societies Joint Paper Award 2011.
Dr. Rodrigues has served on the organizing and technical committees of various international conferences. He is currently Co-Chairing the Conference on "Mathematics of Data: Structured Representations for Sensing, Approximation, and Learning" organized under the auspices the Isaac Newton Institute for Mathematical Sciences programme on "Approximation, Sampling and Compression in Data Science". He has also co-organized the Workshop on Sensing and Analysis of High-Dimensional Data Workshop in 2015 and 2014. He has also co-chaired the Technical Programme Committee of the IEEE Information Theory Workshop 2016, Cambridge, UK.
He serves as Associate Editor to the IEEE Communications Letters and lead guest editor of the Special Issue on "Information-Theoretic Methods in Data Acquisition, Analysis, and Processing" of the IEEE J. Selected Topics in Signal Processing. He is also co-editing a book on "Information-Theoretic Methods in Data Science" to be published by Cambridge University Press (with Y. C. Eldar).
Dr. Rodrigues has been recipient of the Prize Eng. Antonio de Almeida, Prize Eng. Cristiano Spratley, the Merit Prize from the University of Porto, Portugal, and fellowships from the Portuguese Foundation for Science and Technology as well as the Foundation Calouste Gulbenkian.
