A Comparative Analysis of Discrete Entropy Estimators for Large Alphabet Problems

Electrical Engineering Systems Seminar

Speaker: Assaf Pinchas

M.Sc. student under the supervision of Prof. Irad Ben-Gal & Dr. Amichai Painsky

Wednesday, 8^th May 2024, at 14:00

Room 011, Kitot Building, Faculty of Engineering

A Comparative Analysis of Discrete Entropy Estimators for Large Alphabet Problems

Abstract

This paper provides a comparative study of entropy estimation in a large alphabet regime. A variety of entropy estimators have been proposed over the years, where each estimator is designed to a different setup with its own strengths and caveats. As a consequence, no estimator is known to be universally better than others. This work addresses this gap by comparing twenty one entropy estimators in the studied regime, starting with the simplest plug-in estimator and leading up to the most recent neural network-based and polynomial approximate estimators. Our findings how that the estimators' performance highly depends on the underlying distribution. Specifically, we distinguish between three types of distributions, ranging from uniform to degenerate distributions. For each class of distribution we recommend the most suitable estimator. Further, we propose a sample dependent approach, which again considers three classes of distribution and reports the top performing estimators in each class. This approach provides a data-dependent merit for choosing the desired estimator in practical setups.

השתתפות בסמינר תיתן קרדיט שמיעה = עפ"י רישום שם מלא + מספר ת.ז. בדף הנוכחות שיועבר באולם במהלך הסמינר