GPT-4-based Visualization Reasoning Dataset

Description: GPT-4 is a recently released LLM that takes multimodal inputs (image, text), and produce text outputs. It has a strong competence to handle complex vision-language tasks. On the other hand, how human understands and solves questions on information visualisations is poorly understood. Current chart question-answering datasets (e.g. PlotQA) only provide question answer pairs, rather than user's reasoning process. However, annotating user's reasoning process of visualisations is usually laborious and expensive. To this end, we aim to use GPT-4 to generate a large-scale visualisation reasoning dataset.

Goal:
* Collect a large-scale visualisation reasoning dataset by GPT-4's api
* Data analysis: how good is the outputs from GPT-4 compared to real human data.

Supervisor: Yao Wang

Distribution: 30% Implementation, 50% Data collection & processing, 20% Evaluation & analysis

Requirements: Strong programming skills (Django / Node.js (for processing GPT-4's api), python libraries, e.g. Numpy, Pandas). Basic knowledge of data analysis.

Literature:

Openai. 2023. GPT-4 Technical Report. [Openai](https://cdn.openai.com/papers/gpt-4.pdf).

PlotQA: Reasoning over Scientific Plots [WACV](https://arxiv.org/pdf/1909.00997.pdf) </p>