Mastering Embedding Stores & Vector Databases in LLM Apps with Anton Troynikov: Chapter 15

Key Takeaways

[Music] hi folks it's Anton here from chroma um here to talk to you a little bit about embedding stores and Vector databases and how they work in the context of llm in the loop applications like the One You're Building in this course so by now you've learned a little bit about embeddings and nearest neighbor and you know how to use them for similarity search um I want to talk to you a little bit about some of the software packages and tools available for you that make that a bit easier especially if you're working with large amounts of data or you're ready to build a production grade application so let's get right into it what is an embedding store an embedding store is basically a software package that abstracts away a lot of the operations that you've already learned about so first you take a set of documents that represent your knowledge base and you can use the embedding store to embed them using its embedding function and then when a query comes in the same embedding function gets called it generates an embedding uh as the query embedding and then the embedding store itself performs the nearest neighbor search for you and Returns the relevant documents to the llm context window thus basically simplifying a lot of the operations that you would have to implement yourself um so the next question is when should you be using one well the first answer is when your data gets sufficiently large Computing distances to each embedding for each query is is pretty slow and expensive especially u under certain distance functions A good rule of thumb is that when you're are over about 10,000 embeddings using some of the commercial or open source embedding models it's probably a good time to switch to a vector database or embedding store and we'll get to a little bit about why that is in a minute the other important thing is not to underestimate the convenience that actually using an off-the-shelf package gives you here llm powerered applications need to support many users across many indices um you need to handle data and scaling automatically and basically you want it to just work so rather than getting into all the nitty-gritty implementation as long long as you understand what's going on often times it's easier to just use an off-the-shelf solution like chroma so how exactly do embedding stars in Vector databases deal with large amounts of data without incurring the costs of computing distances between the query and every single embedding that it stores well as we've seen exact's neighbor basically requires us to scan over the entire list of embeddings and compute the distances to each one um that takes o n operations commonly called linear time where n is the number of embeddings you already have embedding stores typically use an approximate nearest neighbor algorithm and basically what they do is they exploit the underlying structure of the data um that you have stored and they take Only log n operations in other words they're sublinear they um it's a lot less expensive to do an OLN lookup than it is to scan the entire list and the way that they do that is because they trade recoil in other words sometimes they might miss um the truly nth nearest neighbor and and grab something else in exchange for Speed and computational complexity now that might sound a little worrying but the reality is that the trade off between speed and recall can be tuned depending on the algorithm used and there's a lot of flexibility in how these things are implemented so going into a little bit more depth about approximate nearest neighbors a&n algorithms are an active area of research and there's a lot of different ones um commonly used ones include things called inverted file indexes locality sensitive hashing and hierarchical navigable small world graphs and if you want to learn more details about each of these there's plenty of detail available online different algorithms work best in different settings um a lot of the approximate nearest neighbor algorithms used in many solutions today are set up for the case where the index doesn't change very much and gets updated only infrequently however if we're building an llm in the loop application it's very likely that our data will mutate online quite frequently and so a graph-based algorithm like hnsw works well in these types ofli because we can iteratively construct and iterate on the graph so what's next I mean this all sounds great but the issue is that there is quite a few fundamental limitations even when you have a software package that's supporting you in finding nearest Neighbors in this way because it doesn't answer certain questions it doesn't answer for you which embedding model is best for my task or your data um it doesn't tell you how you should be chunking up your data to ensure good results and just because you've gotten the n nearest neighbors doesn't actually tell you whether those neighbors are relevant or how relevant they actually are so what should you do um with the information that they're not sometimes and finally I think it's really important and one of the core principles of AI is that we can easily incorporate human feedback into the data that we're returning to the model so providing opportunities for and affordances to actually do that for developers is something that's really important these are all features that we're looking into building into chroma and as I said this is a still growing uh this is still a field with a lot of product experimentation going on and we're hoping to support uh users like you in doing all of these things in the near future thank you