How modern search engines work – Vector databases explained! | Weaviate open-source

Key Takeaways

hi there i've been active in what you might call the field of artificial intelligence for a while now of course miss coffee bean knew everything about the technology behind ai-based search engines but i honestly didn't have any deeper thoughts about what it meant to actually implement a search engine ever but i was approached by semi technologies that made an open source vector search engine called we v8 a vector what yes exactly in this video we will explain the general mechanism of modern search engines it's especially important to know if i'm ever going to make my private custom search engine for my super secret miss coffee bean data starting from the data we will go through and explain the most important steps transforming the database into a vector database with a machine learning model like fast text or transformers then querying the database thanks to semi technologies for sponsoring this educational video miss coffee bean are you ready to do the explaining bits okay let's go search engines are the algorithms helping you search over large sets of documents or other kinds of data and do it fast these documents can be either from the internet or the documents a company or organization has on their private servers when search engines first appeared they were not the clever ai-powered beasts of today we'll get to those in a minute the first search engines were relatively simple implementations of full text based search this is very similar to how we look something up in the dictionary if you searched for computer the algorithm behind the search engine went and matched strings to find occurrences of the word computer on the internet or text database there were clever tricks for example to generalize from computers to computer word inflections got removed or simple word distances were employed that you also see in spell checkers like the lievenstein metric that measures the minimum number of single character edits needed to change one word into the other but you see there are many reasons to let these search engines evolve into something smarter what if with computer i do not search for a pc or mac or a compute cluster but what was originally meant by the word computer a human person usually a woman that used to do computations before digital computers were invented in this ambiguous case how should the search algorithm know which meaning of the word i intended taking the most frequent meaning would make the search engine insensitive to rarer use cases of words even more think of what happens if i search something that is not even retrievable by simple string matching like when i ask a question who is the one who has the most money how should the string matching algorithm know that most money means the richest or that the one refers to coffee beans or even worse what if we try to do an image search these two images might look the same but if we look closer bit by bit they are different because the same content is encoded differently so if we search for an image in a database or on the internet we might quickly fail if we just match strings we need a search engine that is more than just a dictionary lookup and is aware of content and context so let's bring in the heavy artillery of today's ai powered search engines well ai is a little of a buzzword here calling them for what they really are modern search engines are vector search engines over vector databases vector databases are a way of organizing the documents in the database in meaningful vectors to be able to search through them with queries that are vectorized beforehand too in contrast things in a dictionary are organized alphabetically which makes sense when searching by string matching but vector databases allow us to organize things even better based on the context in which information finds itself so we have a new system for organizing our database or the index internet based on the principle of context words that occur in the same context belong together we have here two examples of how words that are different live in the same context it becomes clear that the computer which i turned off is different from the computer as a job description that miss coffeebean's grandmother had of course one example is often not representative so we'd look at more and more examples like these and we can set up a graph where computer is close to pc and mac and the jobs are close to computer 2 but pc and the jobs should be farther away from each other if we take in all the other words used on all web pages or in our database we can grow a big graph a dictionary was in a sense one-dimensional as things follow one another only alphabetically the first character was our only feature but now we organize our database entries in this graph where things have more than one feature one feature might be describing how much of a job something is the other feature might describe how human something is from zero human to very human with each feature or dimension that we add the more characteristics we can assign for a data point therefore to what we have crammed here into a two-dimensional visualization we can actually give more space where all the features possibly hundreds each occupy a dimension in a vector space to really flesh out the complex contextual relationships hereby each word becomes a vector encoding the exact position of the word in this high dimensional space to explain this on even higher levels imagine you are going to the supermarket in the classical search engine case it's like the supermarket was a long row of products all organized after the first letter in the alphabet but the problem there is that if we go and find the apples next to the apples there are bongos and not other fruit like pears or strawberries modern vector databases are just giving things more space for modeling relationships better in each vector we can store the position of each data point in this vector space encoding complex relationships in between so after we encoded everything in a meaningful vector we now have a vector database and of course we do not have to do this encoding of words into vectors by hand and we also do not have to come up with features or vectors entries ourselves because there are already available machine learning models for this proprietary search engines of course do not let you choose the vectorizer but customizable search engines like vav88 do give you the choice you could take their custom one based on fast text called the contextionary or the transformers module or a resnet for images or you could add your own machine learning models to vectorize with but for this explanation we will resort to miss coffee beans favorite vectorization method transformers that are state-of-the-art at this they are reading in lots of documents like your whole database the whole wikipedia or the whole internet not kidding after they have read in all sentences in the input data they corrupt their input by deleting words and then they try to reconstruct these words the words are all represented by random vectors at first so initially the output looks random too but example after example these transformers start to learn something about language by deleting reconstructing words and updating their vectors to be better many many times until they start to encode patterns in context dependent vectors these patterns reflect which words occur together in these many many sentences and which context words have a kind of self-organization occurs where vectors in this vector space start to cluster together based on how their respective words behaved in the input data such a model has to be trained once and we can now feed any kinds of text into it without the need of retraining and can encode our whole database now with all the words indexed in our database in vector form we can take in our query pass it through the same model to produce us vectors for the query too and now we can compare the vectors if we search for female computer the word for computer in the query is also informed by its context female therefore we will retrieve the database entries that also figure the word computer in the same job context through the same mechanism we can answer questions too the cool thing is that what we have explained here with text is also possible with images videos speech or other kinds of data too especially because these vectorizing models the transformers can handle all these kinds of data to compute content and context aware vectors for more about this watch some of our videos on multimodality well this is what happens on a high level in ai-powered search engines yeah it's cool especially because it does not need human supervision that would be impossible at these large scales but admittedly it's less magical when one really knows what happens right we are not really telling you that the google or bing or any other search engines work exactly this way the algorithms behind them are proprietary and not disclosed but judging by their capabilities and especially their failure modes there is definitely a vector database and the vector search engine behind them the real lessons about these modern vector search engines we can know from open source projects vv8 is such an open source vector search engine you know open where one can inspect the code open the documentation this is important to have not only because you know what's going in there but also in the situation where you are a company or an institution with millions of documents and want to search through them for example what if you're a hospital with millions of clinical data samples or a factory with product data or a shop with lots of customer data post all your data on the internet to be searchable by google search of course that's a no-go it's estimated that more than 99 of data is not publicly available so if you possess a part of that data you would either have to implement a search engine yourself or take and contribute to the open source work that was already done having control about the search engine on your data is important also concerning the vectorizing model we talked about just before that takes in data and transforms it into vectors because that model is good with types of data it has already seen if it was trained on web pages it is good at judging internet forum text normal websites but what if you have technical engineering terms or medical data in your database in this case you need to take this model and find unit on that medical data meaning that you train it further by showing it medical text it takes it in and self trains by following the same procedure as before it deletes words and tries to reconstruct a text based on the corrupted version only this then lets it extract patterns on this new domain of data like medical texts or other domains we v8 even has a simpler solution where for every undefined word you can provide the definition of that word and use this as a context and you do not need to do any further training so to recapitulate what we need for a vector search engine first we need the data then we need the machine learning model like the transformer but doesn't need to which we use to vectorize the data so our database is now a vector database and we are ready to search through it all we need is a query that we should also vectorize with the same machine learning model because we now want to ask which vector in the database is closest to our query but miss coffee bean is not the best coder when it comes to handling tons of data without running out of memory and waiting for ages for the search to complete searching through a lot of data fast is not an easy thing admittedly so here is where she wants to rely on vv8 that has already figured it out how to scale the vector search for millions of vectors in milliseconds okay miss coffee bean you are allowed to check it out i mean why not it's open source after all so now miss coffee bean are you ready to find out the truth we need to ask where can i get the best coffee for my ai coffee break ah around the globe exactly that's the spirit we hope you liked our explanation of modern search engines in general if you're interested in a more technical presentation about vd8 in particular check out the video in the upper right corner also linked in the description below you