Search Systems & Why Keyword Search Falls Short | Vector Databases for Beginners | Part 11

Key Takeaways

We'll be talking about how vector search works from a concept perspective, but then also going a little bit deeper and talking about how it works in vector databases and why you actually need a vector database. Um, yeah. So, without further ado, let's get into it. Uh if you're looking for a um sort of text version of a lot of the stuff that's covered in this webinar, I would definitely recommend checking out this blog post. Um I wrote it back in November of last year, but it'll go through a lot of the concepts we're going to cover today. Um along with it has the code example that we'll cover for the how to implement vector search from scratch in Python. Um yeah. Um you can also find these slides on slides.lo.com. locom.com as long as as well as my other slides as well. So, first thing I want to start with is search systems kind of run the world. Um, the way we interact with almost anything on the internet is search. And as much as they've gotten more sophisticated in the last year, the one thing that still stands true is that a good search system is sort of just normal. A bad search system is really, really annoying, right? If you're searching something on Google and you're not finding the right results, if you're trying to debug something um and you're still using Stack Overflow or Google to find results for your bug and you're not getting anything, it's really annoying. If you're searching for some sort of fashion item on an e-commerce site and you're just not getting the right results, really annoying. Um, so good search systems are great. bad search systems really really suck. Um, and everything we do on the internet sort of starts with a search, whether it's debugging code, shopping for things on Amazon or H&M or even searching through our own internal documents in our company. So, search systems are super important and the way they've kind of been built in the past is with traditional search or keyword search. So what keyword search does is it matches the exact terms between a query. So whatever the user is inputting the search term whatever you're searching in Google or in Zara or e-commerce sites or your internal company search system and it matches exact terms from that query to all the documents in the database. Um, so in this example over here, I'll have two documents. And basically what happens behind the scenes in a lot of keyword search systems is that these documents are tokenized and assigned a document ID, organized, stuff like that. And then when a query comes in, it's going to look through all the tokens in this table and it's going to match the exact tokens and return the related documents. So this is great. I mean, this is what has powered search for a really long time. But the problem, the main problem with this is that you're not capturing the meaning of whatever you're searching for, right? You're just doing exact matches. And so if I search for Alaskan fish, it may not return the right results. Um, so this can really struggle with conversational language. You need to have the exact right terms in your query in order to turn the right results. Um, this also can struggle with typos and synonyms. There have been workarounds on this. So, a lot of traditional search systems, you'll make like synonyms lists. So, for a word, you have like a bunch of synonyms manually written up in a document that then you either do query um modification or document modification with those. But still, any sort of typos and synonyms, this can really struggle. Sorry about that. So, what's the solution here? Um, vector search.