Why do we need Vector Databases? Why not just use relational database?

Ajay Gupta · Intermediate ·🧠 Large Language Models ·2y ago

About this lesson

Why Do We Need Vector Databases? Why Not Just Use Traditional Databases? In this video, we dive into the need for vector databases and how they differ from traditional databases. Vector databases efficiently store and manage vector representations or embeddings, crucial for similarity searches and recommendations. Here's what you'll learn: Vector Databases Overview: Understand what vector databases are and their role in storing and managing data as vectors for efficient similarity searches. Traditional vs. Vector Databases: Discover the limitations of traditional databases for similarity searches and why vector databases are better suited for these tasks. Real-World Analogy: Compare traditional databases to filing cabinets for music sheets, storing specific details like song titles and artists. Vector databases are like recommendation systems, finding songs similar to the ones you like based on their "musical essence." Applications: See how vector databases excel in image and video recommendation systems by capturing and searching for content based on user preferences and interactions. Technical Implementation: LangChain Integration: Use the LangChain library to integrate OpenAI's GPT-4 model and manage embeddings with Chroma. Creating and Using Embeddings: Generate and store embeddings from document loaders like PyPDFLoader, and query these embeddings for similarity searches. Code Walkthrough: Set up the environment, load documents, generate embeddings, and perform similarity searches using LangChain and Chroma. Follow along to improve your data management and recommendation systems with vector databases today! Connect with me over LinkedIn - https://www.linkedin.com/in/ajaygupta200/ Post your questions and suggestions in the comments section below. #genai #openai #openaichat #openaiapi #openaichatgpt #langchain #vector #database #vectors #embeddings #similaritysearch #recommendationsystem #python #pythonprogramming #llm #llms #chatgpt #chatbot

Full Transcript

in the last video we tried and understood what are embeddings what are vectors in this particular video we try and understand what are vector databases and why do we need them as the name might indicate Vector databases are databases which help us store and manage vectors efficiently right as the name indicates uh why do we need Vector databases uh let's try and understand that with a simple example imagine you're trying to organize a giant music collection right uh a regular database might help you store information like song title artist and genre this is great if you want to find a specific song by name but not as helpful if you're trying to look for similar songs to the one that you like so here's where Vector databases come in instead of just storing text it can actually capture the music's Essence using vectors now with this Vector database you can search for songs that sound similar to the ones that you like even if they have different titles artists albums it doesn't matter Vector databases can help you find other songs which are like in the same neighborhood regular databases uh are like filing cabinets uh for music sheets right they they help you find specific details but Vector databases can act like recommendation systems suggesting you new music as per you as per what you already like so uh in this particular example the only thing that's different is uh this is a rag application the only thing that's different is instead of Facebook uh Vector DB we are using chroma Vector DB why are we using chroma because chroma allows us to store the database on disk now to import chroma I've used Lang chore chroma if I show you here I first installed it and then I've imported it here in code right and the last thing that is different is instead of uh loading the document first and then chunking it I've I've used a document loader API that is provided by langore community so what we get is uh the pages the the chunks of that particular document and the embeddings and we pass that into chroma do from documents what it will do is it will create a vector database using the text that I've provided using the embeddings function embeddings object that I've created and it will persist it into my chroma DB directory because I've named it that way right so that's stored in db2 but next what I'm doing is for my actual similarity search if you see I've used db3 so what I'm doing is is I'm retrieving that particular database from disk so you can store it and and open it up later that's that's how it will work so I've loaded it up up again in a different object I ran similarity search I ran QA chain and I'm getting my response uh what are the skills these are the skills for this particular candidate let me show you what chrom chrom ADB looks like on database so there are two files in total the first one being this bin file which stores the actual vectors uh in the database but this file is unreadable what we can though read is this sqlite file this will store the actual uh database tables indexes how they are stored right if you can see some part of it is readable where it creates tables it create indexes so this part is readable to us and that's how CHR RB stores the vectors so that was it about what are vector databases and why do we need them uh thank you for watching see you in the next one

Original Description

Why Do We Need Vector Databases? Why Not Just Use Traditional Databases? In this video, we dive into the need for vector databases and how they differ from traditional databases. Vector databases efficiently store and manage vector representations or embeddings, crucial for similarity searches and recommendations. Here's what you'll learn: Vector Databases Overview: Understand what vector databases are and their role in storing and managing data as vectors for efficient similarity searches. Traditional vs. Vector Databases: Discover the limitations of traditional databases for similarity searches and why vector databases are better suited for these tasks. Real-World Analogy: Compare traditional databases to filing cabinets for music sheets, storing specific details like song titles and artists. Vector databases are like recommendation systems, finding songs similar to the ones you like based on their "musical essence." Applications: See how vector databases excel in image and video recommendation systems by capturing and searching for content based on user preferences and interactions. Technical Implementation: LangChain Integration: Use the LangChain library to integrate OpenAI's GPT-4 model and manage embeddings with Chroma. Creating and Using Embeddings: Generate and store embeddings from document loaders like PyPDFLoader, and query these embeddings for similarity searches. Code Walkthrough: Set up the environment, load documents, generate embeddings, and perform similarity searches using LangChain and Chroma. Follow along to improve your data management and recommendation systems with vector databases today! Connect with me over LinkedIn - https://www.linkedin.com/in/ajaygupta200/ Post your questions and suggestions in the comments section below. #genai #openai #openaichat #openaiapi #openaichatgpt #langchain #vector #database #vectors #embeddings #similaritysearch #recommendationsystem #python #pythonprogramming #llm #llms #chatgpt #chatbot
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
I Added Observability to My AI PR Agent and It Was Silently Failing the Whole Time
Add observability to AI-powered tools to detect silent failures, as seen in an AI PR agent using Groq's Llama 3.3
Medium · Python
📰
The Machine Accent Travels: AI Text Is Rhythmically Monotone in 70/70 Cells Across 3 Languages
Discover how AI-generated text has a rhythmically monotone pattern across languages, and why this matters for natural language processing
Dev.to AI
📰
SynapseAI: What Happened When I Stopped Trusting Just One LLM
Learn how to stop relying on a single LLM and explore alternative AI models for more accurate results
Medium · LLM
📰
Claude make Fable 5 permanent
Claude makes Fable 5 a permanent feature for Max and Team Premium plans, expanding access to AI-powered tools
Simon Willison's Blog
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →