Vector Search RAG Tutorial – Combine Your Data with LLMs with Advanced Search

Key Takeaways

in this course I'll teach you how to use Vector search and embeddings to easily combine your data with large language models like GPT 4 first I'll teach you about the concepts and then I'll guide you through developing three projects in the first project we'll build a semantic search feature to find movies using natural language queries for this we'll use Python machine learning models and at list Vector search next we'll create a simple question answering app that uses the rag architecture and Alis Vector search to answer questions using your own data and in the final project will modify a chat GPT clone so it answers questions about contributing to the freeco camp.org curriculum based on the official documentation and if you like you can use your own data or documentation the first two examples use Python and the third uses JavaScript but you should be able to follow along with just a basic knowledge of either mongodb provided a grant that made this course possible their Atlas Vector search allows you to perform semantic similarity searches on your data which can be integrated with llns to build AI powered applications so let's start by talking about Vector embeddings imagine you have a lot of different objects like fruits and you want to organize them in a way that shows how similar or different they are are in the real world you might sort them by color size or taste in the digital world we can do something similar with data and that's where Vector embeddings come in Vector embeddings are like a digital way of sorting or describing things each item like a word image or anything else you can think of is turned into a list of numbers this list is called a vector the cool part is that similar items will have similar vectors by turning items into vectors or lists of numbers we can use math to understand and process them for example we can measure how close two vectors are to see how similar the items they represent are words can be turned into vectors and words with similar meanings have vectors that are close together this helps in tasks like searching for information translating languages or even shatting with AI creating these embeddings usually involves a lot of data and some complex math the computer looks at many examples like how words are used in sentences and learns the best way to turn them into vectors Vector search is a method used to find and retrieve information that is most similar or relevant to a given query but instead of looking for exact matches like traditional search engines Vector search tries to understand the meaning or context of the query Vector search is a way to to implement semantic search which means using the meaning of words to find relevant results Vector search uses vector embeddings by transforming both the search query and the items of the database like documents images or products into vectors and then comparing these vectors to find the best matches in essence Vector search leverages Vector embeddings to understand the content and context of both the query and the database item by comparing these vectors it efficiently finds and ranks the most relevant results providing a powerful tool for searching through large and complex data sets MB Atlas Vector search allows you to perform semantic similarity searches on your data which can be integrated with LMS to build AI powered applications data from various sources and in different formats can be represented numerically as Vector embeddings Atlas Vector search allows you to to store Vector embeddings alongside your Source data and metadata leveraging the power of the document model these Vector embeddings can then be queried using an aggregation pipeline to perform fast semantic similarity search on the data using an approximate nearest neighbors algorithm in this course I'll be demonstrating how to use Atlas Vector search in your applications Atlas has a basic free tier m0 cluster that you can use that is free forever in this first project we'll Implement semantic search for movie recommendations we'll be using a sample movie data set containing over 20,000 documents in mongodb and will'll be using the All mini LM L6 V2 model from hugging face for generating the vector embedding during the index time as well as query time but you can apply the same concepts by using a data set and model of your choice and our code we'll find movies with semantic search using the process I described earlier about Vector search all the data including our embeddings will be stored in a mongodb instance so we'll be connecting our mongodb instance but first we have to create that and before we can create that we need an account so let's create a mongodb atlas account if you don't already have one if you have one you can just click sign in so you're basically going to be clicking trree or sign in I already have one so I'm just going to click sign in and if if you're making account for the first time it will probably help you have you create a project but uh since I already have an account I'm going to go to new project and then I'm just going to call this gen Ai and then next and create project okay now I need to create a deployment and when we create an deployment I if you are just creating your account for the first time you'll see this but I'm just going to create the select the free options so we're going to it doesn't matter really which one of these you choose you don't even have to have an account it's going to set that all up for you so I'm just going to set pretty much everything to default and then I'll just click create so you can see it's provisioning right down here in the corner so here we can set up the authentication and I'm just going to create this user it automatically gives us a username and password we are going to be working on our local environment and since we're using the local environment it already has autofilled my IP address so I can click finish and close okay we are going to be using some sample data pertaining to movies now you could use your own data for this project but mongodb offers some sample data so I'm going to click load sample data and you can see it say we are loading your new data set to Cluster zero and this is going to load quite a few different sample data databases basically and we are just going to be using the one related to movies which is called sample inflix we will look through the sample data in a minute but it's still loading so let's go over to visual studio code or you can also do this on Jupiter notebook or any other IDE but I'm going to be using visual studio code you can see I already have a file created movie rex. py and first we need to install the P package so I'm going to go to the terminal and do pip or pip 3 install P okay now we see the sample data set is deployed so I'll go to browse collections you you can see we have all these different collections here we're just going to be using the sample inflix collection and it has a bunch of movies if I go to uh movies here you can see here's the first one The Great Train Robbery and then it has a plot it has the runtime the directors the the rating and there a bunch of stuff about the movie uh 1903 and there's a bunch of these you can see if we look up here there is over 21,000 movies in this database so really old ones to to newer movies so let's get connected to this database from our local environment so I'm in this file I'm going to paste in some code we're going to import P and then we are going to connect to our database it says your mongodb U well I'm going to have to go and get that right now so to get that we're going to go over to here I'm going to click I'm go back to this gen here's the cluster Zer deployment I'm going go to view deployment and then I can go to connect over here and there's a few different ways you can connect your database I'm going to go to just the first one drivers and uh you can go through and select all these like um it's kind of optional to to do that because we are installed Pongo we really just need this connection string right here so I'm going to get the connection string and then I am going to uh go over here and for this example I'm going to hardcode it into here but it's not best practice you really want to use an environmental environment variable but since I'm just going to be running this locally it's not going on to production or anything I'm not putting on the GitHub I can I'm just going to put this right into this file here and then for the password now I'm just going to show it to you um you may have already seen it earlier I'm going to delete this before I post this video so I just have to go get my password now when I was setting everything up I could have used a password I remember or copy my password but I forgot to do that but that's no problem we can just go to the security database access and then here's the account going to edit and then edit password and then we're just going to create a new password it's just a bunch of random letters I'll copy this and then I will update user now I can go over here and I can put in the password now let's just do a quick test to see if this is working I'm going to do print col collection find. liit five so we'll find the first five items okay I'm going to save that and we're going to try it on the terminal okay we got the cursor object that should be what we want now let's try that one more time okay good yes that's we went and we got all that the first five items in the database so we successfully connected to our database the next step is that we're going to set up the embedding creation function there are many options for creating embeddings like calling a managed API or hosting your own model or having a model run locally you can create embeddings using the open AI API that costs some money but there is a free way to do it we're going to be using the hugging face inference API and we're going to use this model all- mini lm- lx- V2 hugging face is an open source platform that provides tools for building training and deploying machine learning models and they make it easy to use machine learning models via apis and sdks so once we're on hugging face you're going to make sure you create an account or log in I already am logged in to my account here and then we are going to go to settings access tokens and then we are going to create a new token and I'll just call this gen Ai and you want to make sure it's set to read and then we can get the token to authenticate to the hugging face inference API so do is generate token and now we're going to define a function that will be able to generate embeddings now this we're just going to set this up and we're going to run in a minute so I'm going to just get rid of this and paste this we're going to move input request to the top and here we are going to have to put in our token now again you don't want to put it in your file if you're going to be putting this into production or or if you're going to be putting this on GitHub you want to use an e environment variable but to make it simple and since this is only local I'm going to put in here and I am going to delete this token before I publish the video so don't worry about it being revealed or anything so here's my token and then we have this embedding URL and we're going to generate the embedding so you can see the function is going to take in one argument to text and it's expected to be a string and this indicates that it's going to return a list of floats this is a typin suggesting that the output will be a list where each element is a floating Point number and then we are going to make the request request. poost to the embedding URL and we have to pass in the token and then we have to pass in the text and then it's just going to return the embeddings which is just going to be a list of floats so we're just going to add this final line to test this generate embedding of free C Camp is awesome so let's try this out we'll just run it it says the model is currently loading so let's just try that again okay we didn't get the air but we forgot to print it so let's print that okay so that first thing I guess the was just temporarily unavailable the error but here is the embedding so you can see this is just a list of floating Point numbers which is the embedding Vector generated for free C Camp is awesome and it was derived using the hugging face uh Transformer model now the API we're using the hugging face inference API is free to begin with and it's meant for quick prototyping with strict rate limits it is possible to set up a paid plan a paid hugging face inference inpoint and this will create a private deployment of the model for you just in case you're running into rate limits but but we shouldn't need need that right now okay the next step is that we want to create and store some em embeddings based on our database so we're going to execute an operation to create a vector embedding for the data in the the plot field in our movie documents and store it in the database so the plot field is just the plot of the movies in the database um for instance the the sap head this simple-minded son of a Min a rich financier must find his own way in the in the world you can see the popup will tell you the the rest of the plot so there's a simple There's a summary of the plot of each movie in the database and we are going to create an embedding based on the plot field that will allow us to search and find movies that have some similarities in their plots and will be able to use natural language to do that creating Vector embeddings using a machine learning model is necessary for performing a similarity search based on intent so let's update our code again uh I added a new section to the code and this is going to create Vector embeddings for 50 documents in our data set that have the field plot so for dock in the collection find where there's a plot that exists and just get 50 and then we are going to generate the embedding based on the plot and you see we're basically adding a new field right to the database called plot embedding HF and then the replacing uh the collection. replace one is going to update the document in the database the the movie information for that movie with the new information which is basically all the same except it has this new field here now in this code we are storing the vector embedding in the original collection meaning alongside the application data this could also be done in a separate collection instead of just replacing the document in the same collection with the new one that has the embedding we could create a separate collection for the embeddings as we have the original information it just depends on your use case and how you want to do that so I'm going to run this code and we're not printing anything so we shouldn't see anything in the terminal but there will be a way to find out if it worked successfully let's just wait till it ends okay so it finished running now I can go back to the database and I'm going to just refresh that so there's a refresh button right here and if I scroll down let's see if we have the new field okay so this now has the plot embedding HF field and you can see these are the vector embeddings that we just created with the hugging face API and there's quite a few so there's 383 embeddings for this one and we did it for the first 50 so this one will have them and then we can just if we scroll down lot so this is the first 1 through 20 okay now I'm at results 61 through 80 so if we look that we know we don't have that field here because we've only done it for the first 50 results now ideally for this project you would do it for every single movie in the database but because of the rate limits and because of the time it would take to run through all what was it 20 thou over 20,000 records I'm just doing the the first 50 but if you want you can take off the limit 50 and then run it for every single document in the collection that's going to give the the best results when you do the search but if you're going to do all the documents you may need to create a paid inference endpoint that I was talking about earlier okay the next step is to create a vector search index so I'm going to go back over to Mong B and we want to go to the search tab here so now we're we're at Atlas search and we we're going to create an index so let's just click create search index and then we will use the Json editor and next so there's a few steps we have to do on this section first we need to select the database and the collection on the left so the infix database and then we need the movies collection okay now we're going to enter the index name where just going to call this plot semantic search and then I'm just going to put in some Json here let's look at this Dynamic equal true this indicates that mang Tob should automatically index all fields in the documents added to this collection so if you have a varing or unknown schema you may want to set to false and then only Fields explicitly defined and the mappings will be index index and then for the fields so this is the name of the field in the mongodb documents that we want to index so remember we already looked at that and when we were looking at the example in the database we saw that there were 384 numbers in each of under each of the field remember it started at zero so it all it went Zer to 383 this specifies the dimensionality of the vectors stored in the field the dimensionality is important for the indexing and search algorithms to to function correctly and then similarity dotproduct this defines this setting defines the similarity metric used when performing searches against this Vector field the dotproduct similarity means that the dot product between vectors will be used to measure how similar they are a higher dotproduct indicates greater similarity this is a common choice for normalized vectors and then we have the type KN andn Vector this indicates the field is a K near nearest neighbor Vector type Ken Vector Fields allow you to perform efficient similarity searches to find documents with vectors vectors similar to a query Vector then we can just click next and then create search index and you can see the the search index build is now in process so it takes a little bit of time to process it okay it's done the the index is built we are going to test it but not with this button I'm going to go back over to our code and then I'm going to add a new code snippet that will explain first of all I did comment all of this we could have made two separate python files one to generate the embeddings and one to use the embeddings but we already generated them we don't need to generate them again so I commented all this out and now basically this is what we're going to be this is the code to try out and and use the embeddings so this code will perform a vector search in the mongodb collection using the vector search aggregation pipeline stage this is used to find documents in the collections who where the plot embeddings HF field is semantically similar to the provided query so we got the query imaginary characters from outer space at War which you can try changing it to something else and this sets the the the query that we're going to use here when see where it says vector query generate embedding query so oh we actually do need this part of the code I forgot we just didn't need to generate all the embeddings again but we are going to generate the the embedding for the query string that's important so we're going to generate the the embedding basically the list of numbers for what we're searching for and uh kind of to backtrack when it says collection. agregate that executes an aggregation Pipeline on the collection and the aggregation pipeline is a power field is a powerful feature in mongodb that allows you to process data and return computed results so like I said this is going to this is a specific stage that is going to perform a vector search and we already talked about create getting the embedding and the path or the the field basically that we're using and then the num candidat 100 this is an optimization parameter that tells mangu to be how many candidate matches to consider internally before returning the final results a higher number can improve the accuracy of the results but may increase the computation size and then the limit four that's just going to limit the number of results return to the top four matches uh you may want to return a lot more than four like 10 20 but we're just going to limit it to four and the index plot semantic search well this specifies the name of the index used for the search which is the one that we created so now we're going and so then the final part of the code is going to go through every item in the result and give us the movies now we're just about to do it but remember we only actually ran it on 50 movies if we had created embeddings for every instead of limit 50 if we did it all did it all of them the results would be a little better but let's see what we get when we run on 50 [Music] movies well I guess the first thing is to fix the code and then we run on fthy movies those curly braces and brackets okay so remember the the query was imaginary characters from outer space at War so we basically got four different ones so this one you can see it's about going to war so that's kind of similar this one's about going to war all these have to do with war none of them seem to have to do with outer space but that's probably just because the first 50 movies didn't have very many outer space ones pulo garal already created embeddings for all 20,000 plus movie plots in the database and ran the same search term and came up with these movies so we only created 50 embeddings but if we had created embeddings for all of them this these are the results that we would get when it's looking at all the movies in the database with with the embeddings it would come up with movies that more closely matched the query about the imaginary characters from outer space at War so this project was with the free hugging face API to get the embeddings if you want to use the open AI API to get the embeddings for the search term it is a paay API but the sample database makes that a lot easier so we actually added the embeddings to this movies database but you can see there's already an embedded movies database where they've already and this is just part of the sample data that you get and there's already a plot embedding for each for each item for each document all 3,483 documents and it's created with the open AI API and you when you're getting embeddings you basically have to use the same API so we can't use the hugging face API to do a search based on these embeddings that were created with the open AI API but let me just show you really quick how we could use the open ai ai embedding API to query based on these embeddings that are already in here and then we won't just be doing a query based on a few of the embeddings that we created for for this database but we would be quering against every document in the whole database so first we're going to have to create a search index pretty much just like before so create new index Json editor so first we'll choose the database embed movies then index name same as before we'll call this plot Mantic search and then this will be a little different so the number of Dimensions is different the path instead of plot and Bings _ HF for hugging face it's just plot and Bings that's the field and for S similarity this time we're going to use ukian so I'll go to next and create the search index now I already went ahead and updated the code we have movie Rex 2 to just use the open AI API so we let me just show you a few of the differences we input open AI API we have this API key again you never want to put it right in in the file but this is just for demonstration purposes and I will be deleting this API key before this video goes live and this is all the same but now when we generate the API we we have this generate embedding function uh but now the in what's in the function is different so this is basically all that's different so now we're using open ai. embeddings that create and we use this embedding model and now we are going to get the embedding so basically we already had this function generate embedding and now we just change what's inside that function to use the open AI API instead of the hugging face API and then the only other thing we changed is the path is now plot embedding instead of plot embedding underscore hugging face so let's try running the function let's try running this code okay remember the query is imaginary character from outer space and now these are actually related to outer space extraterrestrials who dominate Earth we have uh aliens and imaginary characters and they're going they're at War we have Bender game which is again uh Planet Express so that's related to um planets and at War Guardians of the Galaxy which is now another uh some more imaginary characters outer space at War so now that we are looking through the entire database we get results that more closely match our query you can see there's no way we could get results like this with just normal word searching but when we search using Vector embeddings we can have more semantic search we can use more natural language to figure out the movies that we actually want or if you're not using a movie database whatever is in your database it's going to give a lot more power to your searching so we just demonstrated how to use hugging face inference apis how to generate embeddings how to use Atlas Vector search and we learned how to build a semantic Search application to find movies whose plots most closely match the intent behind the natural language query rather than searching based on the existing keywords in the data set we also demonstrated how efficient it is to bring the power of machine learning models to your data using the analyst developer data platform in the next project we'll look again at using Atlas Vector search as a vector store and how to deal with some limitations of large language models we'll also learn about retrieval augmented generation or rag I'll demonstrate how to develop a real world project that uses these Technologies and Concepts along with the Lang chain framework open AI models and gradio let's discuss some of the limitations of llms that will be able to overcome using Vector search llms sometimes generate factually inaccurate or ungrounded information a phenomenon known as hallucinations llms are trained on a static data set that was current only up to a certain point in time this means they might not have the information about events or developments that occurred after the training data was collected llms don't have access to a user's local data or personal databases they can only generate responses based on the knowledge they were trained on which can limit their ability to provide personalized or context specific responses llns have a maximum limit on the number of tokens or pieces of text they can process in a single interaction the retrieval augmented generation or rag architecture was developed to address these issues rag uses Vector search to retrieve relevant documents based on the input query it then provides these retrieved documents as context to the LM to help generate a more informed and accurate response that is instead of generating responses purely from patterns learned during training rag uses those relevant retriev documents to help generate a more informed and accurate response this helps address the limitations in llms specifically Rags minimize hallucinations by grounding the model's responses and factual information by retrieving information from upto-date sources rag ensures that the model's responses reflect the most current and accurate information available while rag does not directly give llms access to a user's local data it does allow them to utilize external databases or knowledge bases which can be updated with user specific information also while rag does not increase an llms token limit it does make the model's use of tokens more efficient by retrieving only the most relevant documents for Genera a response this section of the tutorial will demonstrate how the rag architecture can be leveraged with Atlas Vector search to build a question answering application against your own data let me quickly tell you about some of the other technologies will use for this project Lang chain is a framework designed to simplify the creation of llm applications it provides a standard interface for chains lots of Integrations with other tools and into and chains for common applications this allows AI developers to build llm applications that leverage external sources of data in Lang chain a chain refers to a sequence of components that are linked together to process language tasks each component in the chain performs a specific function like understand a query or generating response and passes its output to the next component in the sequence this modular approach of chaining different components or modules simplify complex application development debugging and maintenance this is not a link chain tutorial so I'll only be briefly describing these sections of our code gadio is an open source python Library that's used to build a web interface for machine learning and data science applications we'll be using the open AI API to access two different models the first is open ai's embedding model which we use to create embeddings the second is a large language model which we'll use to generate text now now let's go to the code editor so now we're going to see how all these Concepts work by creating a question answering app that can answer questions from our custom data we're going to start by installing a bunch of packages so um a lot of these are ones that we have talked about linkchain Pongo open Ai gradio and you'll see how all the we use all these as we develop our app application you are going to need an open AI account and you're going to have to have enough credits which basically means you're probably going to have to have a paid account so I'm going to go over to API keys and then I'm going to create a new key let's call it answer and I will be revoking this key before this tutorial goes out so don't worry about it being on screen or anything I'm going I already have three files made and I'm going to create save the open AI key in this file so open [Music] AI API key and in this file we are also going to save our URI and the URI we're just going to use the same mango URI that we used before we can go to connect and it's just going to be this and we have to make sure to update the password and then I'll just save that file we shouldn't need that one anymore so now we're going to use these two scripts uh load data. Pi is going to be used to load the documents and ingest the text and Vector embeddings in a mongodb collection and then extract information will generate the user interface and will allow us to perform question answering against our data our data using Atlas Vector search and open aai we're going to import the same libraries into each file so P this open AI embeddings mongodb Atlas Vector search directory loader open AI retrieval QA gradio base and then import key pram which is basically just this other file now we are going to get our sample documents in this tutorial we'll be loading three text files from a directory using the directory loader that's this one these files should be saved in a directory named Sample files so I'm going to create a new directory called sample files and then inside this directory we'll have three files we'll have log example.txt we'll have chat conversation. txt and aerodynamics txt and I'm just going to paste in some text into each of these files if you look in the description you can get all the code and text that we're using in this tutorial so in the log example just going to go bam so we have just a bunch this log here and then for the chat conversation we're going to put a sample chat conversation and we can kind of go through it see it says Alfred hi can you explain to me how compression works and Mong be Bruce sure mongod B supports compression of data at rest it uses either ZB or Snappy compression algorithms at the collection level and then it kind of just kind of explains how that works and then with the conversation continues Alfred interesting that's helpful to know can you also tell me how index indexes are stored in Mong b Bruce Mong B indexes are stored in B trees so so basically it's giving information about how mongodb works so the last thing we're going to uh the last file is aerodynamics so this is giving information about aerodynamics boundary layer control achieved using suction or blowing methods can significantly reduce the aerodynamic lag drag on aircraft Wing surface the ya Angle an aircraft indicative of its side by-side motion is crucial and so on so this is all information that we're going to be able to access within our our chatbot okay we don't need these files anymore so we're going to do a little bit more of code that we're going to use in both the load data and extract information files but then they're going to start being different so this is how we're going to access our database so first we're going to load the Mong Tob client from our URL our database name is linkchain demo collection name collection of text block loobs and then we we're going to get access to the collection so both both files we need to be able to get access to the collection now we're going to start focusing on the load data file and the next thing we're going to do is initialize the directory loader so I'm going to put a little more code here and you can see it's going to load the files it's going to load the sample files directory and we're going to load this data and next this line here we're going to define the open AI embedding model we want to use for the source data the embedding model is different from the language generation model so we're using the open AI embeddings model before we showed how to get embeddings with the hugging face API model but you can also get embeddings from the open AI model okay this next line we're going to initialize the vector store we're going to vectorize the text from the documents using the specified embedding model and insert them into the specified mongodb collection so that's what this whole thing is doing we see we're passing in the data the embeddings model and then the collection we're going to put them into and it's just using this Mong Alis Vector search which we are importing here so now that we have this this code is actually done now and let's run it to create all the embeddings and put them into the mongodb collection so we'll just do python load data okay it it should be finished but let's check so we'll go over here we're going to go to our Collections and look we have our Lang chain demo and then here's the collection of text blobs we have the embeddings based on the first document the chat conversation. txt the beddings based on the aerodynamics document and the log examples document and then we can see the array so here's actually the the array of all the floating Point numbers and you can see let's see how many there are this time 1,535 actually 36 because it starts at zero so we've create we successfully created our embeddings next step create our atas search index so I'm going to go to search and we have this one from before but I'm going to create a new search index use the Json editor and then we have to choose the right collections and now we're going to update the Json this time we'll just use the default index name and everything's pretty similar except our field is called embedding and the dimensions are different from the one from the example from before and now we're using the similarity of cosine which we can see it it tells what that means it measures the similarity based on the angle between vectors allowing for similarity measurements not scaled by magnitude so depending on what you're trying to do you're going to use different similarity and we use the same type of cane and Vector okay I will create the search index and the build is now in process while that build is happening we are going to now fill out the extract information script and so all this is going to be basically the same we're going to connect to the client the same uh collection but now it's going to start to get a little different here and we are going to obviously being we're going to extract information so first let's get access to the embed we're basically defining the open AI embedding model we want to use so that's basically just like before we also want to get access to our Vector store and we're going to pass in the collection and the embeddings model so now we're getting access to the vector store that's stored on the atlas basically this is the Atlas Vector store that's stored on mongodb okay now here's our query data function it's going to take a query as its input basically the person the user is going to send a query and the function is designed to process the query and retrieve relevant information using some different tools so the first step in the function have this one here it it invol this is going to convert the input query into a vector using open AI embeddings this process is crucial for preparing the query for similarity searches in a vector space it's going to perform an atlas Vector search using Lang Chain's Vector store so the similarity search passing the query in K1 means we're going to retrieve the most similar document from B based on the query Vector so then this line here is going to extract the page content from the top document in the return list which is the most relevant information in our original query so next we are going to define a llm or a large language model using open AI API and if not specified the default model use it you the default model used is the Open Eyes GPT 3.5 turbo here we see the temperature set to zero which generally means the model's response will be more deterministic and less creative then we do a retriever here we're going to initialize a retriever for the mongod Tob Vector store this retriever is responsible for fetching documents that are relevant to our query then here we're going to this uh retrieval QA uh this chain is then executed with original query this process involves the llm generating responses based on the documents retrieved and the nature of the query you see we're going to load the stuff documents chain stuff documents chain takes a list of documents inserts them all into a prompt and passes that prompt to an llm and then finally we're going to execute the chain and then the function is going to return two outputs the as output which is the content of the most similar document from the atlas Vector search and the retriever output which is the output generated by the retrieval QA chain using the rag architecture so this stri script basically showcases is the integration of of an advanced Ai and database technology to create a powerful tool for information retrieval and processing we combine the capabilities of open ai's language models open mongodb's Vector search and Lang chain to efficiently process and answer complex queries and next we have this basically we're just creating a web interface for the app using gradio now I'm not going to go into a lot of details about this CU that's kind of not really what this tutorial is about but you can see it's it's pretty simple it's just using gradio to create a simple web interface there's going to be some a text box where we enter your question there's going to be a submit button but one thing I want to point out is that there's two outputs and outputs correspond to these two outputs from the function the first output is just what it would normally output with only using Atlas Vector search the second output is gener generated by chaining Atlas Vector search to L change retrieval QA and the open AI llm so we can kind of see what it would what it look what the output is like just with normal Atlas Vector search and with all these other tools so let's test it out so we'll do python EXT ract information. and we just have to go to this URL so I'm just going to enter a question did any error current August 16th if yes what was the error caused by the apple with just Atlas Vector search it Returns the text field as is so basically that's the entire error it doesn't do anything with it but when we chain the search to L chain retrieval QA and open AI the open AI LM yes an aor occurred on August 16th eror was caused by resume of chain stream was not possible the resume Point May no longer be in the oplog it's much easier to read that way okay let's try one more what questions did Alfred ask what were Bruce's answers please summarize in bullet points here we're basically the entire conversation the entire field but down here it's actually able to do it uh Alfred ask what does compression work how does compression work m b how are indexes stored MB and so on and then we get the answers Mong supports compression of data at rest using either ZB or Snappy compression algorithms at the collection level so now it's using so now it's actually giving the answer something that's in a way that's easier to understand we can also use this for sentiment analysis so I'm going to say what was the overall sentiment of Alfred's chat with Bruce what was the likely cat again we're just getting the entire chat right in here but down here Alfred's chat with Bruce was likely very positive the conversation was focused on technical details and Bruce was able to provide helpful explanations the likely SE cat would be high we can also use this to get precise answer retrieval like there's a lot of information how can we get like an exact answer so how about what's y okay we have something that has some information but it gives us a gives us a lot more information it basically gives us the whole text but here we have yah is Angle an Al crafts an aircraft side to side motion and it's controlled primarily by the rudder and you can kind of see where it comes from the angle of aircraft side to side motion uh control primary by the rotor but then we don't get we don't get all the other information that we don't need okay well so in this section of the tutorial we've seen how to build a question answering app to converse with your private data using atas Vector search as a vector sore while leveraging the retrieval augmented generation architecture with Lang chain and open AI Vector stores or vector databases play a crucial role in building llm applications and rag is a significant advancement in the field of AI particularly in natural language processing by pairing these together it's possible to build powerful AI powered applications for various use cases and this final project will modify a chat gbt clone so it answers questions about contributing to the FOC cap.org curriculum based on the official documentation you can use these Concepts to create your own chatbot that connects with your own data to answer questions this project has some similarities to the last one but takes things to another level so let's look at a chat gptt type application that gets information from your own documentation we'll be using the free code Camp documentation so a lot of this code is already developed and I'll be walking you through it and we'll be adding some things so if you're following along at home start by cloning the Repository I have a link in the description and then doing mpm install and then you're going to have to configure the application we have this emv. example you're going to have to create a new file that's just aemv and then you're just going to fill in your open AI key and your Mong Tob Atlas URI we I already showed you how to get that URI in the the previous part of the tutorial you can even use the same URI that we already used for the first project for all these projects you can use the same mongodb Atlas URI with the password and the username right in there after you do that we can test the application let's just do mpm run Dev so here's what the application is going to look like right now it's basically just like chat GPT and it does not use our own documents so I can say write me a poem about being a web developer and it's using the open AI API to do this but it's not accessing any custom information so let's now see how we can make it access it can draw data from the free Camp documentation so first of all I want to show you the documentation I'm talking about this is specifically the documentation on how to contribute to free code Camp to help out with free code Camp so there's a lot of sections there's the getting started how to help with translation and then the biggest section is how to contribute to free Cod Camp free C Camp freecodecamp.org is an an open source project and you can help out with the codebase so it talks about how to set up locally the best practices how to work on the codebase and so on there's a lot of documentation on how to get it set up locally the freec camp.org and how to work on it and then there's also some additional guides down here and what we are going to do is we are going to put all of this documentation into our chat app so if you clone the repo it will be all the documentation will be right in assets FCC do FCC docs and then we have the markdown file of all the markdown files of all the documents mentation on that page here and so we are going to basically put all this documentation into our chat app we're going to do that by creating embeddings for all this information and then creating a vector search index it's going to be kind of similar to what we did in our first project but now we're doing it with more data and more of a real world type project so there's already some code here to create the embeddings so if we go to create embeddings at MJS let's look at this code here so some of this is going to look similar to code that we've already seen so first off we're importing the necessary modules so FS for the file system operations then we have the recursive character text splitter for splitting text into chunks then this one mongodb Atlas Vector search is for storing vectors in mongodb Atlas and the open AI embeddings from Lang chain is for creating the embeddings and the client for connecting to the uh database and then EMV for the environment variables and here kind of like what you've already seen we're just creating a m Tob client using the connecting URI from the environment variable and the script connects to the docs database and the embeddings collection next we specify the directory containing our documents and read the file name using the fsp do reader directory so these are all our markdown files from the free C Camp documentation and then we're going to start processing each document we're going to read the content and log the file name and then the next steps will involve vectorizing the document so this section right here we're going to create a recursive character text splitter configured for markdown langu language and we're going to split the document into CH chunks of 500 characters with a 50 character overlap and then we are going to create the embeddings we're going to use Mong be Alis Vector search to create the embeddings for each chunk using the open open AI embeddings class and we're going to store them in the mongodb atlas collection and then finally we are going to close the connection so basically the script efficiently processes documents creating embeddings and stores them in the mongod to be Alis using the linkchain library so now we're just going to run this I'll run in the terminal down here and just so you know how I'm going to do it if I go into the package.json we have an embed script that's just going to call that file so I'll just do mpm run embed and we can see it's going to create embedding from all these documents and it's vectorizing all of them okay we're done now let's view the embeddings in the database so I'm back at mongod be atas and I am going to go to browse Collections and then here is the new one docs embedding these are the embeddings we just added so if we scroll down we can see that we have the embeddings for each document so if we look at this one we can see the embeddings array 1,535 or 1,536 uh entries in the array or documents so now we've created the embeddings for everything you can see some of the text right in here and it's bro broken that text into different segments and create an embedding for each one so at this point we need to create the vector search indexes this is very similar to what we did before so I will scroll up to the top of here not those if embeddings but I'll go to search and then I'm going to put create search index then we'll go to the Json editor next and then we'll select the database and connection so so embeddings and we'll leave the default index name and then we can see we're dealing with the fields the field embeddings the dimensions uh the similarity cosine this measures the similar similarity based on the angle between vectors allowing for similarity measurements not scaled by magnitude and then the type is K and and vector and then next then create search index okay now we just have to wait for the build to to complete now we're going to update the API routes to use these embeddings so let's look at the current API route again we're not building this app from scratch the code is already given so let's see what the API route already looks like so this is the default API route for chats and basically ju