Prof. Kfir Y. Levy - Do Stochastic, Feel Noiseless: Stable Optimization via a Double Momentum Mechanism

Electrical Engineering Systems Seminar

(The talk will be given in English)

Speaker:     Prof. Kfir Y. Levy

Electrical and Computer Engineering Department, Technion

011 hall, Electrical Engineering-Kitot Building

Monday, June 10^th, 2024

12:00-13:00

Do Stochastic, Feel Noiseless: Stable Optimization via a Double Momentum Mechanism

Abstract

The tremendous success of the Machine Learning paradigm heavily relies on the development of powerful optimization methods, and the canonical algorithm for training learning models is SGD (Stochastic Gradient Descent). Nevertheless, the latter is quite different from Gradient Descent (GD) which is its noiseless counterpart. Concretely, SGD requires a careful choice of the learning rate, which relies on the properties of the noise as well as the quality of initialization. It further requires the use of a test set to estimate the generalization error throughout its run. In this talk, we will present a new SGD variant that obtains the same optimal rates as SGD, while using noiseless machinery as in GD. Concretely, it enables to use the same fixed learning rate as GD and does not require to employ a test/validation set. Curiously, our results rely on a novel gradient estimate that combines two recent mechanisms which are related to the notion of momentum. Finally, as much as time permits, I will discuss several applications where our method can be extended.

Short Bio

Kfir Y. Levy is an Assistant Professor in the Electrical and Computer Engineering Department at Technion – Israel Institute of Technology. Kfir’s research is focused on Machine Learning, AI, and Optimization, with a special interest in designing universal methods that apply to a wide class of learning scenarios. Kfir did his postdoc in the Institute for Machine Learning at ETH Zurich. He is a recipient of the Alon fellowship, the ETH Zurich Postdoctoral fellowship, as well as the Irwin and Joan Jacobs fellowship. He received all of his degrees from the Technion.

השתתפות בסמינר תיתן קרדיט שמיעה = עפ"י רישום שם מלא + מספר ת.ז. בדף הנוכחות שיועבר באולם במהלך הסמינר