Getting Started with Multi-Modal LLMs

LangChain · Beginner ·🧠 Large Language Models ·2y ago

Key Takeaways

The video introduces multi-modal LLMs, provides background information, and showcases 5 new templates for GPT-4V, Gemini, and open source models, highlighting trade-offs between architectures for multi-modal RAG and how to use these templates to get started.

Original Description

Visual assistants will be an important theme in 2024 as multi-modal LLMs gain wider adoption and capabilities. We’ve released 5 new templates as an entry points to GPT-4V, Gemini, and open source models. In this video, we provide some background on multi-modal LLMs, show results from our internal evaluations using LangSmith, highlight the trade-offs between architectures for multi-modal RAG, and introduce how to use these templates to get started. Important Links (1) Open source multi-modal LLMs for private visual search over your photos https://templates.langchain.com/?integration_name=rag-multi-modal-local https://templates.langchain.com/?integration_name=rag-multi-modal-mv-local (2) GPT-4V or Gemini for visual RAG over slide decks https://templates.langchain.com/?integration_name=rag-gemini-multi-modal https://templates.langchain.com/?integration_name=rag-chroma-multi-modal https://templates.langchain.com/?integration_name=rag-chroma-multi-modal-multi-vector Slides https://docs.google.com/presentation/d/19x0dvHGhbJOOUWqvPKrECPi1yI3makcoc-8tFLj9Sos/edit#slide=id.p
☆ Save to playlist Watch on YouTube ↗
AI explanation not available for this lesson yet
This lesson is still being prepared for the AI tutor. In the meantime, explore lessons that are ready.
Browse explainer-ready lessons →

Playlist

Uploads from LangChain · LangChain · 34 of 60

← Previous Next →
1 Chat With Your Documents Using LangChain + JavaScript
Chat With Your Documents Using LangChain + JavaScript
LangChain
 2 LangChain SQL Webinar
LangChain SQL Webinar
LangChain
 3 LangChain "OpenAI functions" Webinar
LangChain "OpenAI functions" Webinar
LangChain
 4 LangSmith Launch
LangSmith Launch
LangChain
 5 LangChain x Pinecone: Supercharging Llama-2 with RAG
LangChain x Pinecone: Supercharging Llama-2 with RAG
LangChain
 6 LangChain Expression Language
LangChain Expression Language
LangChain
 7 Building LLM applications with LangChain with Lance
Building LLM applications with LangChain with Lance
LangChain
 8 Benchmarking Question/Answering Over CSV Data
Benchmarking Question/Answering Over CSV Data
LangChain
 9 LangChain "RAG Evaluation" Webinar
LangChain "RAG Evaluation" Webinar
LangChain
 10 Fine-tuning in Your Voice Webinar
Fine-tuning in Your Voice Webinar
LangChain
 11 Tabular Data Retrieval
Tabular Data Retrieval
LangChain
 12 Building an LLM Application with Audio by AssemblyAI
Building an LLM Application with Audio by AssemblyAI
LangChain
 13 Superagent Deepdive Webinar
Superagent Deepdive Webinar
LangChain
 14 Lessons from Deploying LLMs with LangSmith
Lessons from Deploying LLMs with LangSmith
LangChain
 15 Shortwave Assistant Deepdive Webinar
Shortwave Assistant Deepdive Webinar
LangChain
 16 Cognitive Architectures for Language Agents
Cognitive Architectures for Language Agents
LangChain
 17 Effectively Building with LLMs in the Browser with Jacob
Effectively Building with LLMs in the Browser with Jacob
LangChain
 18 Data Privacy for LLMs
Data Privacy for LLMs
LangChain
 19 "Theory of Mind" Webinar with Plastic Labs
"Theory of Mind" Webinar with Plastic Labs
LangChain
 20 LangChain Templates
LangChain Templates
LangChain
 21 Using Natural Language to Query Postgres with Jacob
Using Natural Language to Query Postgres with Jacob
LangChain
 22 Building a Research Assistant from Scratch
Building a Research Assistant from Scratch
LangChain
 23 Benchmarking RAG over LangChain Docs
Benchmarking RAG over LangChain Docs
LangChain
 24 Skeleton-of-Thought: Building a New Template from Scratch
Skeleton-of-Thought: Building a New Template from Scratch
LangChain
 25 Benchmarking Methods for Semi-Structured RAG
Benchmarking Methods for Semi-Structured RAG
LangChain
 26 LangSmith Highlights: Getting Started
LangSmith Highlights: Getting Started
LangChain
 27 LangSmith Highlights: Debugging
LangSmith Highlights: Debugging
LangChain
 28 LangSmith Highlights: Datasets
LangSmith Highlights: Datasets
LangChain
 29 LangSmith Highlights: Evaluation
LangSmith Highlights: Evaluation
LangChain
 30 LangSmith Highlights: Human Annotation
LangSmith Highlights: Human Annotation
LangChain
 31 LangSmith Highlights: Monitoring
LangSmith Highlights: Monitoring
LangChain
 32 LangSmith Highlights: Hub
LangSmith Highlights: Hub
LangChain
 33 SQL Research Assistant
SQL Research Assistant
LangChain
 ▶ Getting Started with Multi-Modal LLMs
Getting Started with Multi-Modal LLMs
LangChain
 35 Build a Full Stack RAG App With TypeScript
Build a Full Stack RAG App With TypeScript
LangChain
 36 Auto-Prompt Builder (with Hosted LangServe)
Auto-Prompt Builder (with Hosted LangServe)
LangChain
 37 LangChain v0.1.0 Launch: Introduction
LangChain v0.1.0 Launch: Introduction
LangChain
 38 LangChain v0.1.0 Launch: Observability
LangChain v0.1.0 Launch: Observability
LangChain
 39 LangChain v0.1.0 Launch: Integrations
LangChain v0.1.0 Launch: Integrations
LangChain
 40 LangChain v0.1.0 Launch: Composability
LangChain v0.1.0 Launch: Composability
LangChain
 41 LangChain v0.1.0 Launch: Streaming
LangChain v0.1.0 Launch: Streaming
LangChain
 42 LangChain v0.1.0 Launch: Output Parsing
LangChain v0.1.0 Launch: Output Parsing
LangChain
 43 LangChain v0.1.0 Launch: Retrieval
LangChain v0.1.0 Launch: Retrieval
LangChain
 44 LangChain v0.1.0 Launch: Agents
LangChain v0.1.0 Launch: Agents
LangChain
 45 Build and Deploy a RAG app with Pinecone Serverless
Build and Deploy a RAG app with Pinecone Serverless
LangChain
 46 Hosted LangServe + LangChain Templates
Hosted LangServe + LangChain Templates
LangChain
 47 LangGraph: Intro
LangGraph: Intro
LangChain
 48 LangGraph: Agent Executor
LangGraph: Agent Executor
LangChain
 49 LangGraph: Chat Agent Executor
LangGraph: Chat Agent Executor
LangChain
 50 LangGraph: Human-in-the-Loop
LangGraph: Human-in-the-Loop
LangChain
 51 LangGraph: Dynamically Returning a Tool Output Directly
LangGraph: Dynamically Returning a Tool Output Directly
LangChain
 52 LangGraph: Respond in a Specific Format
LangGraph: Respond in a Specific Format
LangChain
 53 LangGraph: Managing Agent Steps
LangGraph: Managing Agent Steps
LangChain
 54 LangGraph: Force-Calling a Tool
LangGraph: Force-Calling a Tool
LangChain
 55 LangGraph: Multi-Agent Workflows
LangGraph: Multi-Agent Workflows
LangChain
 56 Streaming Events: Introducing a new `stream_events` method
Streaming Events: Introducing a new `stream_events` method
LangChain
 57 Building a web RAG chatbot: using LangChain, Exa (prev. Metaphor), LangSmith, and Hosted Langserve
Building a web RAG chatbot: using LangChain, Exa (prev. Metaphor), LangSmith, and Hosted Langserve
LangChain
 58 OpenGPTs
OpenGPTs
LangChain
 59 Open Source RAG with Nomic's New Embedding Model (and ChromaDB and Ollama)
Open Source RAG with Nomic's New Embedding Model (and ChromaDB and Ollama)
LangChain
 60 LangGraph: Persistence
LangGraph: Persistence
LangChain

The video provides an introduction to multi-modal LLMs, their applications, and how to get started with 5 new templates for GPT-4V, Gemini, and open source models. It highlights the trade-offs between architectures for multi-modal RAG and showcases results from internal evaluations using LangSmith.

Key Takeaways
  1. Understand the basics of multi-modal LLMs
  2. Explore the 5 new templates for GPT-4V, Gemini, and open source models
  3. Learn about the trade-offs between architectures for multi-modal RAG
  4. Use the templates to get started with multi-modal LLMs
  5. Evaluate the results using LangSmith
💡 Multi-modal LLMs have the potential to revolutionize visual search and other applications, and the 5 new templates provide an easy entry point for developers to get started.

Related Reads

Up next
Claude AI SEO Skill: 0 to 325 Clicks a Day
Julian Goldie SEO
Watch →