Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding

1. 1Evaluate the performance of large vision language models on visual document understanding benchmarks
2. 2Analyze the generated responses to identify potential gaps between internal representations and responses
3. 3Investigate the internal workings of the models to understand how they process and represent visual documents
4. 4Develop new evaluation metrics that go beyond generated responses to assess the models' true understanding

AI researchers and engineers working on visual document understanding tasks can benefit from understanding the gap between internal representations and responses, as it can inform the development of more accurate models

💡 The performance of large vision language models on visual document understanding tasks may be overestimated due to the reliance on generated responses as evaluation metrics

💡 Large vision language models' responses may not reflect their internal understanding of visual documents #AI #VDU