EE Seminar: Salient Scene Graph
Speaker: Ran Badanes
M.Sc. student under the supervision of Prof. Ariel Shamir and Prof. Shai Avidan
Wednesday, October 24th, 2018 at 15:00
Room 011, Kitot Bldg., Faculty of Engineering
Salient Scene Graph
Abstract
This research focuses on the question of what do people perceive as important in images. We examine what people report when they look at an image, and ask if it can be represented accurately by an algorithm. Our assumption was that the most important information--or as well call it here, captions--would be consistent across annotators and would answer the question posed by this thesis. We gathered simple, single sentence annotations about a database of images and called this database the “Salient Scene Graph” (SSG), and used it as Ground Truth (GT) to develop algorithms that mimic human behavior.
We used two approaches in this research. The first assumes that when asked what is important, people will mention only the visually salient parts of the image.
The second approach assumes that people mention the most visually salient object in the image together with a related contextual object that may not be as visually salient.
Results show that in terms of accuracy, the second approach gives better results than the first. This implies that people tend to describe images using the most visually salient objects and then add a description of the context in the image. Still, both proposed approaches did not produce highly accurate results in several sets of experiments and analyses that were conducted. This indicates that what people perceive as important in an image is not always the most salient object, or at least not the most important object according to the chosen visual saliency methods, and according to the labeled data that was collected (SSG).
