Decoding RAG: Algorithms That Power Vector Databases

Key Takeaways

hello everyone welcome to AI anytime channel in this video we are going to look at different algorithms that helps you uh perform search in a vector database or a vector store uh while you are building a rack pipeline you will be using this under the hood of your vector databases or stores like chroma F skrant milver viate and pine cone and so on and so forth right we'll go deep and see how we how these products or the open source libraries uses these algorithms like hnw lab uh Google scan you know Spotify anoy fast clustering and some of the other algorithms as well now I already have created a video uh where I have shown that how you can create your own Vector store from scratch and that was the first part of this video this is the second part that I'm going to explain you know more in depth that how this happens when you are performing the retrieval task right how these algorithms basically you know create indexes then perform some kind of s semantic search with different metrices that you can you know configure now this going to be an important video for you to understand uh this how this technology Works how rag pipelines Works in a in a uh mainly for the retrieval part because generation is more out of the box where you just feed your retrieved context to a large language model of course with different uh methods like reranking long context reorder so on and so forth have been used recently now I will recommend you to watch this video till the end if you want to uh learn and derive lot of findings if you are not interested to watch the video and directly want to jump into the code you can just get it from my GitHub repository that's in the video description but this video will help you understand that how this Vector databases and stores basically Works under the hood and how langen have been utilizing them you know as a high level abstract class so let's start our experimentation with these different algorith ithms to see that how we can you know build our own Vector store from scratch all right uh let's start our experimentation with the different Ann algorithms and for that experimentation you can see I am in my Jupiter notebook and we will have a look at few algorithms and techniques uh let me write it over here I want to explain that in detail so you understand the you know uh intricate details about this algorithms and you know uh and this algo as in techniques rather than just focusing on Lang chain or Lama index abstract classes right that will definitely not make you good uh not make you not make you an expert in generative AI or a large language model if you don't understand how this algorithm works now uh let me first write out uh different Ann alos okay and here I'll first write for example we're going to look at uh we going look at uh first for example we'll start with H&W okay or let me write it fully hns live this is more of proximity graph and don't worry I will explain everything once I'm once I start covering this one by one second will be uh Google is isnn or scan n that's for that's mainly a very that's very intuitive because it's not graph based this is more of a vector comparation so let me just write that and then we have at number three we'll look at Spotify n o okay which is really interesting as well okay uh it's a tree based so let me just write trees here okay and uh the last we'll also probably look at you know fast you know you already know fast but I'll just cover because because it it it is just clustering in a way so I'll just cover this okay so these are the different alos that we'll look in this video you know hns Li Google scan Spotify ano and fast if you want to build your own Vector database or a vector store like W8 Pine con cant milver so on and so forth right how can you uh create a Noel architecture because if you create the same way that they are doing it you will not succeed in your startups right it will not work out because there are already thousands of vector databases startups that are working maybe you can have an ensemble you know architecture you can have an ensemble approach of different Ann algorithm nearest neighbors algorithms know approximate nearest neighbors by the way or something very new right which is not right right there and people are trying to solve that and that's why this this tutorial is important for you if you don't want to go deeper dive if you don't want to dive deeper into this technology you can skip this video not have to watch it but if you really want want to understand how these things works then I think this will probably will be very helpful for you okay now all of these are being used by chroma DB in some aspect like for example hnsw fast as clustering some of the others like they use Vector compression and all right so this these organizations products startups whatever you call them depending on which Vector database you use they have been using this algorithms in their product development so you can also do that of course you can come up with different approaches as well we'll start with uh as I said we'll start with hnsw live first okay let's start with that now hnsw live and I also write keep on writing here as a documentation because this will help you so hnsw for example is uh I hope I spelled it right it's an hierarchal navigable small world that's what the uh that's what the acronym is for I don't know if I spelled it right okay r a r c i hopefully uh navigable uh a small world okay now this is what it stand for it's a graph based algo okay it's a graph based algo you know used for efficient approximate nearest neighor okay that's what we do in a uh in a nend dimensional space when we work with the vector store of vector database once you create and store the embeddings right now approximate uh nearest ners or that's what basically your Ann is okay now this search in a very high dimensional space so I'm just write like hid space okay or rather it will be spaces 10 now hnsw is part of the uh is a part of library of hnsw Library hnsw live it's a library that we have it's a part of that and you know is very popular for its performance uh per performance in terms of speed and accuracy for Ann task now and that's what we're going to do here so probably let me just one more more enter okay now this is fine this is a very high level intro of hnw lib and this is what we're going to try here guys and for that what we're going to do is we'll have a sample text some paragraphs or some sentences we'll use an uh embedding model to First create some embeddings and then we'll use this alos to perform the retrieval task so let's do that the first part of the video that I created how to create a Vector store from scratch where I covered more uh more of the crud uh crud application where you have create read update delete application in the vector store how to do those task that has been already covered now this is the next step of it how to do the retrieval with different alos now the first thing that we have to do is install so let me just do pip install we need an embedding model for that I'm going to use an open source model for from sentence Transformers so let me just do that so sentence Transformers and then hnsw leave of course the library that we have so let me just install this P install sentence Transformer and hnsw live okay you can do this exercise in collab notebook as well or wherever notebook that you want to you know try it out you can do that now once we do that we'll use uh let me just write the code also so from sentence Transformers so let me just write it down I'm going to use uh sentence Transformer class so this is how we import that sentence Transformer and then we need numpy okay so import numpy as NP this is what we need and I will add few more sales after that uh what else we need is to model and I'm going to use mini language model L6 V2 so let me just do that so for that I'm going to use this sentence Transformer class that I have defined above I'm still not running it because okay I think this has been okay this been run successfully now we can get this in and now I want to have sentence Transformer class so let's just do that now here I'm going to use this model which is mini LM excuse me mini LM L6 and then V2 this is the model that I'm going to use and you can see it's completely importing I'll just add few more sales and after that we need some paragraph so let me just get the paragraphs in okay and let me get this in as well the model and it will take a bit of time to get the model in but let's get the paragraph So I already have created a set of paragraphs for us let me just get it from here let me just do raw and I will just copy this so let me copy it over here and come back okay and you can see now imagine if you use Lang chain know P PDF loader or directory loader or document loader whatever and you get the text right these are all as you can see it's a python list and it all have an element uh if you do zero it will come this then this then this right so similarly you can have you can replace this with your PDF uh uh content as well your extracted text now I have paragraphs here I'm going to Define this paragraph so let me just do that now on top of this paragraphs what I have to do is I have to basically create uh the the next thing which is the generate the embedding so let me just Define something called generate embeddings so let me just do that and here in embeddings going to encode that so model. incode sentence Transformer has a class uh sorry excuse me sentence Transformer class as a method in code and here I'm going to pass my paragraphs now this is a paragraph that I'm passing and it should generate the embeddings for me you can see it took around 1 seconds to generate the embeddings and once I then again print embeddings you will see an array of numerical representations and you can find it out over here now your embeddings are done here I will then import hnsw lab okay so import hnsw lab that we have installed then we need the dimensions of the embedding and for that uh let's have a variable called di dimensions and going to use the sap method sap not a method sap and then one so dimension of the embeddings and now here we'll initialize okay so initialize the hnhh W lib index so index equals dot index you can see it has two BF index and index and index I'm going to have some space so let's just do some give some space and then I'm going to keep Dimension equal Dimensions excuse me sorry I wrote 80 it's I don't know it's L2 okay so you can see it's a suppose it's space must be one of the L2 IP or cosine depends on which similarity space that we're going to use okay so what kind of uh basically it's again an algorithm within an algo that we are using so cosine similarity you would have heard you have about ukian jackard lanstein you know dot so on and so forth right so this is what it is now in space I'm going to use L2 for this so let's just uh do L2 and this is something you can also read Because this is important that you should have the understanding of how are we you know initializing this different dimensions and all uh Etc now L2 is done for ukian distance and the dimension that you have are for the on the top in sale Number Eight is the dimension of the data points okay now you can also further make it more complex you know if you want to make it more complex but know it's fine so let's uh move further and let me just run this now and you can see it's now takes your index now shape one if you have if want to know more it determines the dimensionality of the embeddings embeddings do shape one gives the number of columns in the embeddings array which corresp corresponds to the size of of each embedding Vector okay now now this L2 creates an instance of the hnw index now the space parameter is set for the equilibrium distance that will help us measure the distance between points now you'll have n number of points the vectors that you'll have in a high dimensional space how can you measure the distances between these you know vectors that that's why we are using the ukian and the dimensional parameter is specifi the D dimensionality of the data point that we have on top now moving on next what I'm going to do is once we do that uh let's initialize the index now so for that I going to have a number of elements so number elements let's first get the length of paragraphs okay and excuse me num elements and you can see it's 50 okay now there are some parameters that you know we can consider let me first write it then I will explain okay these are some uh parameters that we can tune as well so I'm going to initialize the index you can see it has init index method and inside this I'm going to pass something called Max elements and for that I'm going to use the number of elements because and it's a better way of doing it so you should know what's the uh number of elements in your paragraphs that you have and for this I want to call this num elements and then I have something called I'll explain that uh how should we why it's not showing me EF construction and then I will have something called M and this is very very interesting uh to understand that why are we using this now EF construction and Mr parameters that control the construction of the index and that affects your speed and accuracy that how fast your retriever will be you know uh in a way because these are very helpful once we uh tweak these uh parameters okay now the EV that we have defined is Max elements equal to number of elements that basically sets the maximum number of elements or embeddings that the index can hold and this is what it is now let me just Define these uh numbers here so for that what I'm going to do is uh EF construction equals let's keep 200 and then m equal to 16 okay and now we're going to add the embeddings to the index so let's add embeddings into the index and here index. add items okay and add items I'm going to add embeddings now we are done with the add embeddings part now we have to query the index so how can we query the index is something that we let me just after here we'll add one more I'll make it markdown so let me just do this markdown thing and let me say qu in the index okay now for that uh let me add uh let me first have a query sentence let's have that so query sentence and let's call something like programming languages like python have revolutionized software development and I believe that's true revolutionized software development now this is my this is my query that I have or let me just make it a doc string so you can I can just divide it in software uh revolutionized software development and then just end the doc string this is my query sentence now again we have to create the embeddings that's what we do right once you build a rag pipeline retrieval augmented generation what we do there guys okay once we get the query from end user again we have to create the embeddings of it and that's why we do right we again use the embeddings to create embeddings and then we use that uh embeddings to go inside the vector datab and find out the relevant chunk or the similar chunks for that exactly that's what we are doing now not through an high level abstract class but here we are going a bit deeper to understand how it really does now query embeddings equals model. incode and in this incode what I'm going to do is pass this a python list and you can pass m n number of thingy here so for that query sentence and let's do that now let me also print query embedding for you so you can just see there again a numerical representation inside an array a and that's what we have let me just add a couple of more sales now we'll query the index so query index so query the index Let's uh how many number of nearest neighbors so for now let's keep two and see what it does or let's keep two for now and we'll see now what I'm saying is labels and distances equals index dot knnn query okay this is the method that I'm using k NN query and you can see it takes uh index so it takes your index and it takes your query embeddings it also takes K which is the integer value that we already have defined so let me just pass query embedding and then let me just pass k equals K and let's run this now okay we have run it and now what I'm going to do is let's print this and see some result so print query and inside this query I'm going to pass query sentence and then okay let's me first get this it says query was programming software like I can just write it here only so what I'm going to do next is for label and distance we have to jip that so in jip and inside that I'm going to pass my labels the first one of the labels and the first one of the dist H excuse me the distances so let's do that okay so labels and distances and then this is okay and then just print so print and paragraph and this will hold your paragraph thingy so paragraphs label paragraphs label and then come out of this key value pair and then then you write distance so distance will again be very similar ah excuse me distance will again be distance so let's do distance okay so what we are doing here is we are saying okay paragraph and then we are giving paragraphs label and then we are saying distance and then just giving the distance let's print it out and now you can see it says programming languages like python have revolutionized software development this was your query now that these are the two result that we have fetched okay and this is important to understand I'll explain that uh why and let me just explain this part also k equal to 2 that you see it sets the number of nearest neighbors to find and labels and distance that we are using performs a k nearest neighbor in know very traditional machine learning algorithms that we have used n number of times once we used to you know use with K and algorithms that we used to work uh I don't know after after the rise of chat GPT and T people are still using it but I think they are there is still a need of it now you can see performs a k nearest nwor query using the hnsw index now it Returns the indices which are basically nothing but the labels of the nearest Neighbors in the index data and their respective distances from the query embeddings and that's what it does now to understand the results uh you can see that we have got two responses two paragraphs basically this are your result now the distance score that you see in the right hand side basically it it represents how close or far the embeddings of each paragraph is from the embedding of your query sentence now the in the context of hnsw lab the distance is calculated based on the metric we specified so metric that we have specified is ukian distance this is what we did with L2 here that you see L2 is what where we do ukian distance in your case and the space equal to L2 now lower score equals to closer match so let me just write it over here I'll make this mark down okay so results interpretation of course we can sort this out and that's that's how we sort this so lower score equals closer match that's what we to do here so closer match so lower distance that you see a 0.73 63 that we have on the top for the history of pip python dates back to the late 908s now Clos the match to the query the closure the score is zero the more similar the paragraph is to the query sentence that we have and that's what it is okay so you can find out now this is our experimentation with hnw uh and I hope you uh I hope you understood the first one hnsw leave the proximity graph which is an acronym for hierarchical navigable small world you know that that's been used uh for efficient Ann SE in high dimensional spaces like vector database and Vector store now what we will do guys we'll move to the next one that we have in our list to you know explore which is Google scan and Vector comparation we'll look at that uh the next and so uh you can if you want to skip the video if if you have already familiar with Google scan you can just skip to the Spotify anoy now but let's start our experimentation with Google scan here so let me just add a few uh let me just add a few sales first and then we'll move to our second algo for the day which is Google scnn okay this is what we're going to look at in this one now Google scnn is is really really interesting and it's there an acronym for let me just write it first okay what was that scalable nearest Neighbors so this is the this is the acronym for this and this has been designed for you know this has been designed for efficient efficient Vector similarity uh Vector similarity uh search at scale so this will help you at scale and it's suited for handling very large data sets Okay very large data sets so this is not a graph based but this is this leverages techniques like and that's we we look at the advantages and disadvantages of course because you know sometimes you sometimes there'll be tradeoffs for the speed sometimes the trade-off will be for the accuracy and a lot of other parameters that we'll talk about in the end of the videos okay now techniques like vector compression and quantization to speed up the search of course there be some performance degradation that might happen because you are compressing it and that's there's no doubt about it but uh so you know it achieves its efficiency through a few let me just have some features I just write it like how do it works okay so the feature is partitioning the first is the way it create the partitions the partitioning so dividing the data sets dividing the data sets into smaller more manageable clusters that's the first one the second one is quantization so approximating the high dimensional spaces or not high dimensional SPAC that should be vectors High dimensional vectors with compact lower dimensional representation that that's how basically quantization will also happen isn't it now dimensional representations and the third one and the third one is scoring the way it is scores you just uh and I will we will cover this as well so let me just write we'll cover this once we are scoring it so using efficient alos to compute a Ann okay approximate nearest neighbor this is very high level about Google scannon and this again a library available for that so we have to install that so let me just do that so what I'm going to do is PIP install scnn and I'm going to install this Library it says could not find a version and this is uh that requirements oh okay this is surprising so let me just do because I think there's a problem if I just do here pip install it should work in collab I understand that the library might not support the latest version of python let me see that fine so we'll move from there to here for at least this one let me just make it uh a bit bigger so you can see it and we don't need that okay so let me just okay we'll for at least with Google will work in this one uh because you know it does not supports my local notebook due to the python version conflicts I'm sure about that and know it's it's okay it's an acronym for scalable nearest neighbor it has in its name guys it's used for scalability okay if you want to scale this you scale your vector databases further that is that's is what going to help you now let's start our code here so for that we'll have a few things again let's let's just get it uh let me see so from I'm not sure if sentence Transformers are part of collab by default you know they have made lot of changes recently but I don't think that it's a part so but anyway we'll try it out if it not a part of that okay it's not a part of that so let me just add after here anyway it seems meanwhile I will get the paragraph so let me just get the paragraph here so paragraphs and this is our going to be our paragraph and after this we'll have the will load the model okay it says no module let me just install that if there's no module pip install sentence Transformer thingy so sentence Transformers and this will install sentence Transformer and after that let's bring the model here so let's initialize the mini LM L6 version 2 model you can see it suggest already and Google collab is work generative is working fine in Goog Google collab the recommendation that it gives it's pretty impressive now uh okay again at least till embedding generation it will be the same so let me just run this quickly after I run this I will run this part as well because it has to download the model weights of mini LM L6 B2 around some 500 100 plus MB okay of the model wids now you can see that is done let me just get the this thing in and after that let me add a couple of sales and first thing that I'm going to do is embeddings and embeddings is going to be numpy do array so numpy array and I'm going to do model in code paragraphs this is what I'm going to do it says no nump it seems which is fine so for that let's get it here only so for that let me just add to import numi as NP okay and now this NP has been used here model. incode so here what we're doing is we're generating the embeddings guys so let's do that and you can also print it says name paragraphs is not defined so let Define it and once it is done we'll go to the next that you will create your embeddings and here I'm going to write the same thing pre print number of embeddings number of embeddings ings and then it will basically create your embeddings do SAP sap and then goes your first one it should just print the embeddings and if you want to see that the numerical representation of your vectors you can also find it out over here now let's import scnn this is what I'm going to do here and you can see Ive imported successfully hopefully okay yeah now now here here we going to assume a few things and you know we have to do on top like that so let me let me first write the code and then I will explain that what are we doing here because uh we need an uh scnn Builder so we need uh scalable approximate nearest number Builder there's basically a builder class with the embeddings the number of neighbors to return and the similarity metric you know that can be dot that can be any others as well so let's let's do that so for that what I'm going to do is let me just first write it down here okay so let me first write the code and then I will explain so let's call it Searcher and you can you can see it suggest something but I'm not interested totally probably in that okay so uh partially I need the part of code but I will just write okay Ops and I need the py bind as well but I don't need load Searcher for now what I need is Builder I need the Builder one so P Bend Builder yeah this is what I need perfectly fine now you can see what we are doing here we have an embeddings and then after that embeddings we have some uh parameter that we are tweaking basically the that value that 10 that you see is basically the number of neighbors to return because I have 51 here the paragraph 50 paragraphs and I want 10 to uh 10 number of neighbors to return and then I am using a similarity metric which is nothing but the dot product in this case and then I'm saying okay I'm using a tree I don't know why why it's uh showing me that thingy here but anyway let me just get that okay this is bad okay tree and then I will close this here now if you look at what we are doing we have a tree and inside that tree we have a number of lips because it's if you can see number of lips it say 2,000 I don't need 2,000 as number of lips that's too much for me what I'm going to do I'll keep it very less I'll keep it 10 as a reduced number of lifts and then says number of number of lifs to search and I'll just say five just search in five lifs just to save some time but you can you know you can tweak around these numbers okay now the next thing that I'm going to do is have something called training uh yeah sample size perfect and let's keep that 50 if you have your large Corpus of text then you have to this number will be higher and then basically this adjusts to match the number of embeddings right we have 50 that's why I'm keeping it 50 now the tree basically configure the tree part that you see this the tree part this configures the partitioning of the data set okay basically create the smaller data sets right and number of lifts is the total number of partitions that it has and number of lifts to search is the number of partitions to search in for a query that's what basically it's doing now we have something called ah which is really intuitive and that's why data structure guys are really important right ah is nothing but the asymmetric hashing if you have studied hash then you will understand hash trees and all hash Etc now score ah sets up the scoring configuration using aetric hashing ah and anisotropic quantization that's really you know uh that will be too technical to cover probably in this video maybe we can have a separate if if you guys can share your thoughts in comment box if you want to understand that how en Tropic quantization works for uh SC scanon I I think I can cover that as well now here what we are doing is just understand that this is a scoring configuration using hash mechanism that we are doing so let me just do that so score uncore ah and I'm going to set this H2 so let me just do that and then I have something called anisotropic so let me just do anisotropic okay uh anisotropic quantization thol and let's keep 0.2 and then reorder I don't want 100 it's too big and then just build that now let me just run and see perfect now you can see that we have our uh this part is done our Builder has done its job now I just going to query the uh uh this whatever we have created so far so basically the build is nothing but builds the scannon Searcher that we have uh scalable approximate nearest neighbor the 100 that you see or the 10 that you see here the 10 that we have it's basically nothing but it's for reorders the top 10 candidates from the ah scoring for accuracy that's what we are we are doing that okay so now let's go in and let's have our query so query sentence equals and I will have the same query where is that where I was using for software development thingy okay this this is what it is okay so let me just get it ah excuse me this has to be a string so let me just get that cool this is again our query sentence and we're going to use the same embedding thingy so query embeddings equals model. incode query sentence now we have let me just print query embedding as well so you can just see it ah excuse me this is not Capital let me add few more code sales here now we have our query embeddings now we have to resap guys we have to resap the query embedding to an one dimens array okay so let me just do that so query uncore M I don't know why it's wrting Capital maybe probably I just doing underscore that's why query embeddings equals query embedding do resap so query uncore embedding Dot resap and just I'm going to use for the resap let's just do a resing of this now we're going to use the recept query embeddings for searching so for that I'm going to have neighbors so neighbors and then distances so neighbors distances and then Searcher do search Searcher do search and here I'm going to just put query embedding and final number of neighbors that's what we going to return final number of neighbors five okay now let's print uh so print query and inside this query I'm going to have my query sentence query sentence and then print uh nearest neighbor for what wrong I doing that it says it's okay I forgot to give comma here excuse me okay uh what the heck ah this should be inside this so bad okay and then what I'm doing is for idx okay the indices of neighbors so let's get the neighor and distance uh this looks good so for idx neighbor distance in in numerator then we jipping it the neighbors and distances then saying saying after that let's have paragraphs also Soh okay so the indent is also wrong see paragraph and then paragraph equals extended paragraph equals paragraph paragraphs and then I'm going to pass my neighor and then I'm going to print what's wrong here then what saying okay this has to come back okay print and then if ID x + one which is right ID x + 1 and then we have our paragraph let just uh write paragraph and paragraph This is not required so paragraph and then we have our let's then do for distance for the for the distance what I'm going to do is distance and let me just remove everything from here and this should not be inside perfect and now you can see so now we got our result here guys okay and if you look at uh the result that we have now the distance metric that we it depends right if you're using dot product if you using you know ukian distance or whatever okay so that that depends on on that but if you look at the response that we got the history of python dates back to the late 98s let's let's look at the hnw lib what was we we also getting that right for this particular query so you can find it out over here uh and let's look at the diversity if it gives you the artificial intelligence as been yeah fine so the both of it kind of gives you the same similar responses on the top two that you see here you can find it out over here now here as we if you go up uh you can see we have used dot product now but in that case we were using L2 now here we are using dot product it means higher the score is better okay so you can see it higher the score it means it's more closer to the similarity search that's what it is but in the case of ukan it's the reciprocal of it it's the reverse okay in that case now you can find out you know it kind of we got what we wanted to do with this uh algorithm as well the scannon okay now this is fantastic by the way this we're going bit deeper into to understand how the things Works basically so now distance score for idum lower score equals to closer match higher score equal to closer match for DOT product and it depends what kind of thing that you are using the the relevance is very important the results basically suggest here you can see the model finds the paragraph about the history of python okay uh it it understands basically a bit also the contextual uh a meaning behind it you know it it gets this python like about the history of python most relevant to your query followed by other related to technology and science that's what it's able to get it now that these results can be used in applications basically like you know in our rag the information retrieval also in recommendation systems or know clustering the documents we're understanding the semantic similarity between different text is crucial that's what it is guys now I think we're also done with uh scan okay now well let me just write code here also and I will give the notebook in my GitHub repository in a very different file format so don't worry about you know there'll be different file for all these algorithms let me just add a few sales here and here let me make a text and call in now we're going to look at Spotify Spotify anoy let's look at that now this is this is the name is very interesting uh let me just go inside this now uh ano is like approximate nearest neighbors oh yeah I just love the name so approximate nearest neighbor oh yeah and you can find out is a cc++ library by the way okay it's very fast but we're going to use this in Python so we're going to use the python binding of it okay so python findings of this and basically it's to basically to search for points in space that are close to a given query Point again the underlying Concepts remain the same that how does a vector database or a store performs the retrieval task you know using this Advanced algorithms using Lang chain Lama index you just uh run one line of command and you think that uh because it's a black box right if you use the high level abstract classes my agenda of this video is to to give you an enough concept so you can go deeper and understand how this retrieves that's basically what I'm trying to do now uh basically this Spotify build this by the way okay if you can go through the research paper Spotify build this for music recommendations that's what they did and now we also use it so music recommendation system they used it for uh that's not be recommendation by the way these guys will say that okay not writing it correct it's used for large scale nearest neighbor search due to the efficiency and ability to work with large data sets just like Google scan and now let's uh excuse me let's also install this this Al a library pip install ano and if you are building your vector database probably you have to write different classes for all of these algorithms and depending on your uh type of data if it's a text Data if it's know it's like have multimodalities nature then depending on that you need on Sample uh way of architectures to handle those uh data types that's what you can do okay so we do not have to worry a lot about this because we have already embeddings that we can use now we'll directly build the okay let me have to first get the anoy thingy so from anoi let me know if my videos are annoying by the way okay this we I'll figure it out for that something uh from anoi import ano index and here what I'm going to do is see we already have paragraphs we already have embeddings for the above ones we just going to use that now ano requires let me just write building the Ani index now we using different techniques to build indexes guys this the important now building the NY index what I'm doing is nooy requires you to specify the number of dimensions for the vectors and the metric let me just write it over here let me make this as a text and here I'm going to write ano requires you to specify hopefully I'm typing it right specify the number of dimensions for the vector store for the vectors rather and the metric like and it uses a different metric I'll cover something called angular I don't know if you aware about angular you know it also has ukian of course for your uh ukian and you know so on and so forth you know you can cover that the whole day and let's do that so the first thing I'm going to do is I'm going to look at how we're going to create uh this uh index let's build the index for this so let's have variable called f equals embeddings we already have the embeddings embeddings dop and let me just first get the shap of embeddings basically it's a length of itom vector that will be index that's what we are doing here the next thing is a t basically it's and we're going to use angular for this that's what they recommend so Ani index F and angular so angular is one of the distance metrics that we using it's not the JavaScript framework that you are thinking it's a distance Matrix now the next thing that I'm going to do is I'm going to say okay for I in or not I in for I and Vector so I Vector in enumerate so I'm going to uh enumerate so enumerate and I'm going to pass the embeddings and then add the item I think this is right so T do add item let's see that okay this is correct now we're going to build 10 trees okay so here this is what we're going to do next okay and let's do that okay so let's build a 10 trees with index with the 10 trees more trees gives higher Precision when quering but of course you can again I'm not going to create really a vector database that you can start uh selling it off or at least make it open source on GitHub right you have to do a lot of tweaking and uh trial and error to make that work so here what I'm going to do is okay uh to make it more uh tal to 10 let's do that and so how should we do T dot this should be a build that should be a build yes T build 10 more trees will lead of course lead to more accurate results but take more time to build and memory because we building a t here it's a time consuming thing okay more time to build and more memory and that's why if you have worked with decision trees and all that if we say that it's not good for larger data sets because sometimes it might make it very slow okay and it also takes a lot of memory and that's also an issue with this so T do build and T do save so we're going to save that and it will return something like pass or something I'm going to call this test. n angular let's do that you can see it says true sorry I said pass okay basically saves the built index to a f name I don't know if it's showing it here or something okay uh how can I check that out you can see that we have we have persisted on this so it's persist that we are persisting on this basically it saves the build index to a file name test and and this allows the index to be reload later and that's what we are doing right because we will not do every time on a run time we have to save and persist this on disk in our memory and then we have to load that uh the next time that we are doing so now let's load that okay so for that what I'm going to do let's query this first now let me just write it querying the index quering the index so let's have no OA index and we have f and angular and I'm loading the test. and it should also say true or something and you can say it says true we are able to load it now now even if the index is already in memory these lines demonstrate how to load the saved index by the way guys the. load this is useful in scenarios when you build the index once and need to query it multiple times possibly you know different sessions or application that you'll be working with let me have a query sentence over here so query sentence again and why I'm writing again let me just get the query sentence so where is our query sentence okay we already have defined query sentence so I'm not going to write it again why I'm writing it so we don't do to query sentence we already have the query embedding as well so what I'm going to do here is I'm going to write n neighbors n neighbors equals let's keep it five it specifies the number of nearest neighbors to find out okay now in this I'm going to have a nearest neighbors so nearest neighbors equals and then U do get n or nns by the way get nns and by Vector okay so y Vector get n by vector and here I'm going to pass my query embedding so query embedding n NE so you pass your query embeddings you know n numbers and then include distances as well so for that include distances equals to this is what I'm doing here so I'm saying near neighbors or rather if you include distances then you have to separate that with distances as well right we need distances as well now n neighbors equals 5 nearest neighbors or let me just make it more explanatory this will be indices so this will be explanatory n neighbors five nearest neighbors uh indices then distances U dog NS by Vector query embedding and neighbors and then include distances true fantastic let's run this fine now I'm going to just going to print that so for that uh let me just print print query query sentence and I'm saying print nearest NE which is fine what I'm saying is uh for I oh let me just I don't have to write that much of code for this so what I'm going to do is for I and neor index so let's have neighor index and distance in in numerate again jip nearest neor indices and distances this looks good and then paragraph equals paragraphs neighor ID X the indices and then I'm saying print I + one the paragraph and then Sol the paragraph and then Sol the distance and now you can see it gives you the history of python dates back to the late 90 t8s fantastic uh and you can see that score that we got here again the score that we are getting the scoring mechanism again can and just to we'll come back to this but this question can be about angular let me just explain a bit so the angular distance that you have between two vectors is the angle in radians angular is the distance between the vectors that we have in the angle the the angle in radians between the two vectors irrespective of their magnitude because a vector has both a direction and a magnitude that's what we have studied in physics if you remember in your higher secondary days right and you should know the difference between a scalar and a vector and Tor you should know that now it's a way to measure how two vectors are oriented relative to each other this is very very intuitive and Spotify has done a great job you know to create this beautiful library now it is often calculated using the cosine of the angle between the two vectors which ranges from minus one which is completely opposite and of course to one exactly the same is communally used measure in this case now the angular distance can be derived from cosine as well now there's one more step involved which is normalization before Computing the angular distance vectors are typically normalized to unit unit length which their magnitude is skills to one now this normalization ensure that the distance measure reflects the direction of the vectors not their length and that's why it is important in case of you know embeddings and rag you know uh metric is very particularly suitable for high dimensional data like text or image embedding this can also be used for image embeddings where the orientation of the vectors like representing the semantic or visual similarity is more important than their magnitude so just to understand that you can use this in your if you are building a multimodal vector database a vector database which can index both text and images data you can use Spotify anoid to build a retrieval there okay that's what you can do now see if you want to compare with ukian which considers both magnitude and Direction angular distance focuses only on the direction it only looks at the direction this makes it more suitable for cases where the scale of the vectors is not relevant or is normalized that's that's where it's important okay and yeah I mean that's why you know Spotify have used it for a longer period of time suddenly I don't know what if they are using but this is fine now the last one that I'm going to cover is is very quickly uh is fast everybody's favorite and I know you you guys would have been already working with fast that I also do uh fast is is does not require any introduction because it has been created by Facebook Facebook has created it okay and yeah so let's uh jump into now fast so let me just write it down write it down here I'll make a text and this is going to be our last and and I'll go deeper into some of the other algorithms if you guys like this video I want to create uh our own open source Vector store at least not the vector database which might be really difficult but a vector store which is like really a no code kind of a thing okay so let me just do that fast or let me just write the code so for the fast what I'm going to do is uh it's of course it's it's an acronym for Facebook AI similarity search an very efficient Library library for clustering and similarity search on the last data set developed by Facebook the fair team of Facebook Facebook AI research team you know Yan leun has been the director of that if I'm not wrong uh from his NYU days few years back I think 2013 they created that lab okay now the basic step to install to install fast what going to do is PIP install fast CPU let's get the CPU because we are dealing with very less data here you can also get GPU if you have n number of documents you know having uh multiple Pages you can do that as well now after that what I'm going to do is I'm going to import fast so let me just do import fast so I'm going to have import fast here and then import numpy as NP okay we already have numpy we don't need that so let's jump in directly let's just create the uh or let me just do a bit so embeddings because we have to do a bit differently we need flow 32 yes correct uh so np. array and then we have model. incode paragraphs this is fine and then I'm going to write H type h type Flo 32 so let's let's just do that uh this is this is important let just do it once we do that I'm going to have a dimension so Dimension embedding saap one that is right and then let's create the index so I'm going to use a flat index for Simplicity fast supports many type of indexes okay it's a very vast Library you know there having a lot of developments with fast as well and of course after the rise of rag they have been making a lot of changes with fast you know they have bm25 retriever they they have they deals with the sparse vectors they deal with the hybrid search there a lot of things that you can do with fast now I'm going to create a uh flat index here so let me just do that so for that what you going to do is index equals you can see it's a index flat l2d and then I'm going to add the embeddings into that so index. add and I'm just going to add my embeddings into that okay uh so let's do that that's it let's have our query sentence uh then I will have my query embedding the same way so let me just copy this here so just copy to come here and paste and call this as something like query embeddings and in this not embeddings by the way this will be embedding query embedding and I'm saying np. array model. incode and this is not paragraphs this has to be changed and this needs to be done as uh query sentence model. incode and then query sentence ah excuse me that has to be query sentence and then as type FL 32 perfect now the number of nearest never to retrieve I'm going bit fast for fast because I'm sure that you guys know about fast already if you have worked with any kind of I'm just saying index. search you know query embedding K let's do that and then just do for or let's just print query first so print and query for I ah not required not required we can just do I in range okay so for I in range and in the range of K in just the range of K you just do print uh of course F and I + one H excuse me paragraph paragraph and then you just do the paragraph inside a paragraph and here I'm going to pass my I of zero and then again I two and then distance then pass the distance the same way so D zero and passing the same way I let's do that mhm surprising programming languages like python have revolutionized paragraph T a e TF h embeddings np. aray f add embeddings query [Music] sentence so for fast clustering you getting programming languages like python have paragraph of index searge so I'm saying for I in range K print f i + 1 and then paragraph and then parag ah okay not paragraph paragraph so bad so bad paragraphs that's what I was saying why I'm getting these characters here you know not the complete tokens distance let's see now hopefully should work yes and you can see we got it okay uh for this as been yeah this has been a long video guys you know I covered all the different type of algorithms that I should cover uh the uh all this Vector store and DBS like chroma fast V8 pine cone milus skrant Etc are using this kind of algorithms okay to build their Vector databases and stores I hope you will find some uh you'll find something from here you know you like to you would like to create your own maybe now after this some Vector databases and stores and please let me know if you are uh getting instruct somewhere I'll be more than happy to help you this code will be available on my GI repository separately for each of these algorithms yeah and that's all with the experimentation guys that's all uh for this experimentation guys you know I hope you understood the concepts in detail and that might help you in your journey of building a vector store or a database both open source or a product close Source or whatever now if you have any doubts uh thoughts feedbacks please let me know in the comment box you can also reach out to me through my social media channels please find those details in my channel uh about us and also on the YouTube channel Banner okay if you like the content I'm creating please hit the like icon for this video and if you haven't subscrib the channel yet please do subscribe the channel uh that will help me create more videos in near future thank you so much for watching see you in the next one