EE Seminar: Universal Loss and Gaussian Learning Bounds
(The talk will be given in English)
Speaker: Dr. Amichai Paisky
Massachusetts Institute of Technology
Monday, December 25th, 2017
15:00 - 16:00
Room 011, Kitot Bldg., Faculty of Engineering
Universal Loss and Gaussian Learning Bounds
Abstract
In this talk I address two fundamental predictive modeling problems: choosing a universal loss function, and how to approach non-linear learning problems with linear means.
A loss function quantifies the difference between the true values and the estimated fits, for a given instance of data. Different loss functions correspond to a variety of merits, and the choice of a "correct" loss could sometimes be questionable. Here, I show that for binary classification problems, the Bernoulli log-likelihood loss (log-loss) is universal with respect to practical alternatives. In other words, I show that by minimizing the log-loss we minimize an upper bound to any smooth, convex and unbiased binary loss function. This property justifies the broad use of log-loss in regression, in decision trees, as an InfoMax criterion (cross-entropy minimization) and in many other applications.
I then address a Gaussian representation problem which utilizes the log-loss. In this problem we look for an embedding of an arbitrary data which maximizes its "Gaussian part" while preserving the original dependence between the variables and the target. This embedding provides an efficient (and practical) representation as it allows us to consider the favorable properties of a Gaussian distribution. I introduce different methods and show that the optimal Gaussian embedding is governed by the non-linear canonical correlations of the data. This result provides a primary limit for our ability to Gaussianize arbitrary data-sets and solve complex problems by linear means.
Bio
Amichai Painsky is a Post-Doctoral Fellow, co-affiliated with the Israeli Center for Research Excellence in Algorithms (I-CORE ALGO) at the Hebrew University, and the Signals, Information and Algorithms (SIA) Laboratory at MIT. Amichai received his B.Sc. in Electrical Engineering from Tel Aviv University (2007), his M.Eng. in Electrical Engineering from Princeton University (2009) and his Ph.D. in Statistics from Tel Aviv University (2016). He is a recipient an outstanding Ph.D. students award from the school of Mathematical Sciences, the Weinstein Institute of Signal Processing and the Marejn Foundation. Previously, he received a Brain Return Ph.D. Scholarship from the Israeli Center for Returning Scientists.
