Visual Element Property Graphs For Bridging The Symbol Description-Recognition Gap

This thesis addresses the semantic gap between visual perception and functional significance of symbols used in road vehicles. It presents a novel approach that enables users to identify and understand automotive symbols by describing what they visually perceive, translating visual descriptions into practical implications. A system combining a property graph representation of visual components and semantic relationships with a language model-powered natural language interface is developed. This method explicitly models relationships between visual elements and interpretations, differing from end-to-end vision-language models. Evaluations, using automated metrics and human assessment, demonstrate performance exceeding baseline large language models, with a BERTscore F1 of 0.765, compared to the best baseline's 0.597. Analysis of visual symbol queries reveals human description tendencies, favoring intuitive analogies and basic shapes. Contributions include a symbol decomposition methodology, an advanced property graph schema, natural language query processing, and evidence supporting structured knowledge representation for symbol description-recognition, applicable beyond automotive interfaces.