Zhiwen Fan - VLM 3R Vision Language Models Augmented with Instruction Aligned 3D Reconstruction

3D foundation models are a key step toward spatial and physical intelligence, but their impact is often limited by heavy compute and scarce, well-curated 3D data. In this talk, I will present a set of 3D methods that make reconstruction real-time and integrate it with generative VLMs for real workloads, along with a scalable 3D data engine spanning 3D/4D environments and human avatars. Together, these components give VLMs strong spatial awareness and the ability to track temporal 3D changes. The result is higher accuracy, and better scalability. Taken together, these advances move us toward AI systems that interact with the physical world with genuine spatial understanding and real-time performance. Zhiwen (“Aaron”) Fan is an Assistant Professor in the Department of Electrical and Computer Engineering at Texas A&M University. He received his Ph.D. from The University of Texas at Austin. He was awarded the 2022 Qualcomm Innovation Fellowship and a Best Paper award at the CVPR AI4CC Workshop. He has served as an Area Chair for multiple AI/ML conferences and has completed research internships at Meta, NVIDIA, and Google. This session is brought to you by the Cohere Labs Open Science Community - a space where ML researchers, engineers, linguists, social scientists, and lifelong learners connect and collaborate with each other. We'd like to extend a special thank you to Benedict Emoekabu and Mayank Bhaskar, Leads of our Computer Vision group for their dedication in organizing this event. If you’re interested in sharing your work, we welcome you to join us! Simply fill out the form at https://forms.gle/ALND9i6KouEEpCnz6 to express your interest in becoming a speaker. Join the Cohere Labs Open Science Community to see a full list of upcoming events (https://tinyurl.com/CohereLabsCommunityApp).