Speaker:  Mor Goren

M.Sc. student under the supervision of Prof. Ram Zamir

Wednesday, January 2^th, 2018 at 15:00

Room 011, Kitot Bldg., Faculty of Engineering

Combating Packet Loss in Image Coding using Oversampling and Irregular Interpolation
 

Abstract

            In the age of the Internet, where communication lies deeply on networks that suffer from packet loss, there is a great need in a system that can provide a reliable communication of information in presence of erasures and quantization noise (an outcome of compression). Using source channel separation principle, we suffer of cliff phenomenon, with no improvement in case the channel behaves better than the usual.

            Diversity "multiple description" (MD) source coding promises graceful degradation in the presence of an unknown number of erasures in the channel. A simple solution in the case of two descriptions consists of oversampling the source by a factor of two and delta-sigma quantization. This approach was applied successfully to JPEG-based image coding over a lossy packet network, where the interpolation into two packets is done in the discrete cosine transform (DCT) domain.

            The extension to a larger number of descriptions, however, suffers from noise amplification whenever the received descriptions form a nonuniform sampling pattern. Previous work shows that using frames for oversampling, irregular spectrum is the best possibility for nonuniform sampling pattern, among tested frames, in a way that it minimizes the excess rate.

            In this work we show how noise amplification can be reduced by optimizing the interpolation filter. We propose two interpolation methods which, for a given total coding rate, minimize the average distortion over all (K out of N) patterns of received packets, known as side receivers in the MD literature. We provide experimental results comparing low-pass (LP) and irregular interpolation filters for the side receivers, and examine the effect of noise shaping on the trade-off between the all-N central receiver and the side receivers.

סטודנטים לתארים מתקדמים וסגל, זה הזמן לחשוף אתכם למחקרים הכי עכשווים ופורצי דרך של המחלקה להנדסת תעשיה. 

קישור לתכנית

(The talk will be given in English)

Speaker:     Dr. Ziv Goldfeld

Laboratory for Information and Decision Systems (LIDS) at MIT

SUNDAY, December 23^rd, 2018
15:00 - 16:00

Room 011, Kitot Bldg., Faculty of Engineering

Estimating the Information Flow in Deep Neural Networks

Abstract

This talk will discuss the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the Information Bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information I(X;T) between the input X and internal representations T decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true I(X;T) over these networks is provably either constant (discrete X) or infinite (continuous X). We will explain this discrepancy between theory and experiments, and clarify what the estimated mutual information curves from past works were actually tracking.

To this end, an auxiliary (noisy) DNN framework will be introduced, in which I(X;T) is a meaningful quantity that depends on the network's parameters. We will show that this noisy framework is a good proxy for the original (deterministic) system both in terms of performance and the learned representations. To accurately track I(X;T) over noisy DNNs, a differential entropy estimator tailored to exploit the DNN's layered structure will be proposed and theoretical guarantees on the associated minimax risk will be provided. Using this estimator along with a certain analogy to an information-theoretic communication problem, we will unveil the geometric mechanism that drives compression of I(X;T) in noisy DNNs. Based on these findings, we will circle back to deterministic networks and demonstrate that the past observations of compression were in fact tracking the same geometric phenomenon. Future research directions inspired by this study aiming to facilitate a comprehensive information-theoretic understanding of deep learning will also be discussed. 

Bio:

Dr. Ziv Goldfeld is currently a postdoctoral fellow at the Laboratory for Information and Decision Systems (LIDS) at MIT. He graduated with a B.Sc. summa cum laude, an M.Sc. summa cum laude and a Ph.D. in Electrical and Computer Engineering from Ben-Gurion University, Israel, in 2012, 2014 and 2017, respectively. His research interest include theoretical machine learning, information theory, complex systems, high-dimensional and nonparametric statistics and applied probability. Honors include the Rothschild postdoctoral fellowship, the Feder Award, a best student paper award in the IEEE 28th Convention of Electrical and Electronics Engineers in Israel, B.Sc. and M.Sc. Dean's Honors, the Basor fellowship, the Lev-Zion fellowship and the Minerva Short-Term Research Grant (MRG).