YorkSpace has migrated to a new version of its software. Access our Help Resources to learn how to use the refreshed site. Contact diginit@yorku.ca if you have any questions about the migration.
 

Chart Question Answering with Visual and Logical Reasoning

Loading...
Thumbnail Image

Date

2022-12-14

Authors

Masry, Ahmed

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

Charts are very popular for analyzing data. When exploring charts, people often ask complex reasoning questions that involve several logical and arithmetic operations. They also commonly refer to visual features of a chart in their questions. However, most existing datasets do not focus on such complex reasoning questions as their questions are template-based and answers come from a fixed-vocabulary. In this thesis work, we present a large-scale benchmark covering 9.6K human-written questions and 23.1K questions generated from human-written chart summaries. To address the unique challenges in our benchmark involving visual and logical reasoning, we present transformer-based models that combine visual features and the data table of the chart. Moreover, we propose chart-specific pretraining tasks that improve the visual and logical reasoning skills of our models. While our models achieve the state-of-the-art results on the previous datasets and our benchmark, the evaluation also reveals several challenges in answering complex reasoning questions.

Description

Keywords

Computer science

Citation

Collections