VisDoT
VisDoT: Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
My role: proposed grounding the framework in graphical perception theory, designed DoT prompting, and built the VisDoTQA data-generation pipeline; fine-tuned and evaluated the models.
Problem
LVLMs answering questions over charts fail when the question refers to visual attributes: "the leftmost bar," "the dark red segment." The root cause is a grounding failure: models cannot align visual primitives (position, color, length) with what they actually denote in the chart. Chain-of-thought prompting offers limited improvement here, because longer reasoning cannot fix perception: if the model is looking at the wrong element, more logic only elaborates on a wrong premise. We diagnosed the underlying issue as prior methods treating perception and logic as a single undifferentiated reasoning step.
Approach
VisDoT restructures reasoning so that perception happens explicitly, and before logic.
Grounded task design. We formalized four perception tasks (Position, Length, Pattern, and Extract), selected not arbitrarily but from graphical perception theory (Cleveland & McGill, 1984), which ranks how accurately humans decode visual encodings: position is decoded most accurately, length next, while angle and area are distortion-prone and were excluded.
DoT (Decomposition-of-Thought) prompting. Adapting the least-to-most decomposition strategy from my earlier table reasoning work (KICS 2025) to the visual domain, DoT decomposes questions into perception sub-questions and logic sub-questions, with perception forced first. For "What is the difference between the leftmost bar and the blue bar?": (1) value of the leftmost bar → 39, (2) value of the blue bar → 24, (3) difference → 15. Unlike CoT, decomposition is triggered only when perceptual cues require it, preventing over-decomposition, and each sub-question localizes the visual element before reasoning over it, cutting off grounding errors at the source.
Using this framework, we built VisDoTQA, a 331K-sample perception-following dataset, and fine-tuned open-source LVLMs on it.
Results
+11.2% on ChartQA and +33.2% on VisDoTQA, surpassing GPT-4o (also on ChartQAPro). Most notably, 7.4K training samples outperformed methods trained on 163K–320K (ChartGemma, ECD), evidence that supervision design, not data volume, is what drives efficiency. Ablations confirmed the perceptual tasks and DoT are complementary (3.2× over CoT on VisDoTQA when combined). Zero-shot gains on POPE and MMMU show the perception-first principle generalizes beyond charts.