Introduction to Retrieval

Key Takeaways

Welcome back. In this video, we will go through an introduction to building retrieval systems, which will be the key theme of this entire module. The key topics we'll be covering include understanding a standard retrieval system workflow, document loaders, document splitters and chunkers, embedding models, vector databases, and building retrievers on top of vector databases. Let's get started. So this is a standard retrieval workflow which is typically the first step of building a retrieval augmented generation a rag system. Here we will be focusing on the retrieval aspect of the rag system. Now in a typical retrieval workflow you typically connect to a certain data source that could include the internet, a database. You source the data which could be PDFs, text documents, JSON documents and so on. You use a relevant data loader to load the data from these data sources and convert it into relevant text data. Now once you have these really large text documents or could be small documents depending on your source, we transform them into small chunks or paragraphs and once we transform by breaking them into smaller chunks, we will typically pass them through an embedding model to convert it into embeddings. convert each document chunk into relevant embeddings and then we would store these small chunks and embeddings into a vector database and then build a retrieval strategy on top of this vector database so that we can retrieve similar documents or relevant documents based on an input question by the user and this entire workflow consists of these key components which is what we will be covering in detail throughout this module. So let's understand what are document loaders. Document loaders are typically used to import or load up data from various sources by langchain as document objects. Now typically that means if you have PDFs, word documents, PowerPoint presentations, the idea is to use document loaders to extract typically the text content from these documents and load them up as a line chain document object. A document object typically includes the text content along with relevant metadata which could be like the name of the file from which we got the data, a page number, the author and so on. Now, Langchain is integrated with over 100 document loader tools and libraries which includes third party tools and libraries to load documents of various formats. Now, what do we do after we extract and load up the documents as text? The next step is typically we might need to transform them for optimal use with large language models. So a common transformation step includes splitting a large document into smaller segments or chunks as you see here so that we can fit the large language models context window because in many cases let's say if your LLM's context window is maybe 4,000 tokens or 32,000 tokens and your documents are larger than that context window size then you will easily end up getting a token limit exceeded error. So the idea is to chunk the documents into smaller, more manageable chunks so that you can retrieve the most relevant chunks based on your question, your task and so on. Langen provides several built-in document transformers to typically facilitate the splitting and chunking so that you can split these larger documents into smaller chunks. The next step after breaking your documents into more manageable chunks is to convert these chunks into some kind of a numeric representation because at the end of the day machines understand only numbers and we can use something like a document vector similarity to find out how close are certain documents, how relevant is a document to a question and so on. So typically what we do is we use embedding models to create a vector representation for any document which is known as an embedding because the meaning of the document chunk is embedded into a fixedsized vector of numbers. Typically these are floatingoint numbers as you can see here. Now in lang chain there is a base embeddings class which provides two methods. One is for embedding documents and one is for embedding a question or a query and lang has integrations with all popular embedding model providers which includes open AI as well as open source or open models from hugging face. So the key idea of using embedding models is to transform these document chunks the text into an embedding vector which can then be used to look up and see okay which documents are similar to the embedding of a given question by the user. What do we do once we have the embeddings? So once we have the embeddings the idea is to store these document chunks and embeddings into a vector database so that we can do very fast querying and lookup in the future. So that's what a vector database tries to do. It manages document storage embeddings and helps in performing semantic vector search. It supports common operations including creating, updating, deleting and querying. And lang has integrations with all popular vector database providers like Chroma, Pine Cone, VV8, Milvas and many more. So the key functionality of a vector database is to store your document chunks and its corresponding embeddings so that in the future whenever you're asking questions it can retrieve the most relevant documents to your given question based on embedding similarity. Next step is typically retrievers which is where you build a retrieval strategy on top of your vector database. So the idea is a retriever is basically an interface which accepts a string query as an input and it returns a list of relevant documents as output which are closest to the question in terms of embedding similarity. So what happens as you can see in this workflow is you will have uh some kind of a question. The question will go into the vector database. The vector database will use its embedding model to transform the question into an embedding vector. It will then look up and see which embedding vectors are most close to that questions embedding and then it will return the corresponding document chunks whose embeddings were closest to the questions embedding back to the user. So the idea is to query the vector store and retrieve the most similar documents based on your input question. Now retrievers are typically created from vector databases or other search APIs. Langchain enables us to use retrievers with a variety of retrieval strategies like cosign similarity, maximal marginal relevance, multi-query retrieval and other kind of strategies which includes using things like reancher models and so on. We will see all of this in the relevant units in the future videos. Thank you. And that's it for this video.