CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

1. 1Identify the limitations of existing evaluations for analogical reasoning in MLLMs
2. 2Develop a novel task and dataset that tests compositional analogical reasoning
3. 3Evaluate MLLMs using the CARV benchmark to assess their ability to compose rules from multiple sources
4. 4Analyze the results to improve the models' higher-order intelligence capabilities

AI researchers and engineers working on multimodal LLMs can benefit from CARV to evaluate and improve their models' compositional analogical reasoning capabilities

💡 CARV addresses the gap in existing evaluations by testing the ability to compose rules from multiple sources

🤖 Introducing CARV: a diagnostic benchmark for compositional analogical reasoning in multimodal LLMs #AI #LLMs