Electrical Engineering Systems Seminar

Speaker: Michael Fiman

M.Sc. student under the supervision of Prof. Hadar Averbuch-Elor

Sunday, 13^th October 2024, at 15:00

Room 011, Kitot Building, Faculty of Engineering

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

Abstract

Most humans use visual imagination to understand and reason about language, but models such as BERT reason about language using knowledge acquired during text-only pretraining. In this work, we investigate whether vision-and-language pretraining can improve performance on text-only tasks that involve implicit visual reasoning, focusing primarily on zero-shot probing methods. We propose a suite of visual language understanding (VLU) tasks for probing the visual reasoning abilities of text encoder models, as well as various non-visual natural language understanding (NLU) tasks for comparison. We also contribute a novel zero-shot knowledge probing method, Stroop probing, for applying models such as CLIP to text-only tasks without needing a prediction head such as the masked language modeling head of models like BERT. We show that SOTA multimodally trained text encoders outperform unimodally trained text encoders on the VLU tasks while being underperformed by them on the NLU tasks, lending new context to previously mixed results regarding the NLU capabilities of multimodal models. We conclude that exposure to images during pretraining affords inherent visual reasoning knowledge that is reflected in language-only tasks that require implicit visual reasoning. Our findings bear importance in the broader context of multimodal learning, providing principled guidelines for the choice of text encoders used in such contexts.

השתתפות בסמינר תיתן קרדיט שמיעה = עפ"י רישום שם מלא + מספר ת.ז. בדף הנוכחות שיועבר באולם במהלך הסמינר

Electrical Engineering Systems ZOOM Seminar

Speaker: Daniel Shalev

M.Sc. student under the supervision of Prof. Yuval Shavitt

Wednesday, 25^th September 2024, at 15:00

FlowPicClip: Language Supervision Improves Network Traffic Classification

Abstract

Traffic classification has gained much attention in the past decade, and using deep learning proved to exhibit good performance.  However, the lack of large labeled datasets pushed research to explore few-shots learning approaches where only a few labeled samples per class are available, and usage of augmentation techniques tailored to the domain of network traffic was proved as a viable solution.

In this paper, we demonstrate that it is possible to improve the performance of augmentations using a different approach. Our solution simplifies preprocessing and reduces training time, while effectively utilizing small amounts of training data. Furthermore, it proves to be highly effective in few-shot scenarios, demonstrating robust results when tested on disjoint datasets, specifically the UCDAVIS19 and ISCX datasets. Inspired by the recent breakthroughs in integrating image and text data, particularly the CLIP model by OpenAI, we introduce FlowPicClip. This model harnesses the power of contrastive learning with FlowPics and their labels as text sentences. By leveraging Large Language Model (LLM) encoders FlowPicClip aligns network traffic representations with their textual descriptions.  We demonstrate 2.75% and 1.4% improvements over the best published results on the UCDavis19-Human and ISCX datasets for classification tasks, along with 7.7% and 6.2% improvements in few-shot classification on these datasets.

השתתפות בסמינר תיתן קרדיט שמיעה = עפ"י רישום בצ'ט של שם מלא + מספר ת.ז.

חברת NVIDIA מזמינה אתכן ואתכם לצפות בפאנל מיוחד של בעלי מקצוע שונים בתחומי AI בו יספרו על מסלול הקריירה שלהם, הלימודים וההסמכות שעשו בדרך.

בין הפאנליסטים - יזם סטארטאפ, פרופסור באקדמיה (וגם יזם סטארטאפ), מגייסת באנבידיה, ראש מחלקת למידת מכונה ומדעי נתונים. 
בואו לשמוע את סיפורי הקריירה הייחודיים של הפאנליסטים שלנו, וקבלו תובנות יקרות ערך, טיפים וטריקים להצלחה בתחום.

גלו אלו קורסים והסמכות  AI של NVIDIA  חשובים בעיצוב הקריירה וההשתלבות שלכם בתעשייה.
ההשתתפות פתוחה לכולם: סטודנטים, בוגרים טריים ובעלי ניסיון כאחד.

הוובינר יתקיים אונליין בשפה האנגלית ביום חמישי, 26 לספטמבר, בשעה 17:00 זמן ישראל.

להרשמה