SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions

arXiv:2307.01139v2 Announce Type: replace-cross Abstract: Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present \textit{SciTune} as a tuning framework to improve the ability of LLMs to follow multimodal instructions generated from scientific publications. To test our method