[Exclusive] Zilliz Roundtable // Why Purpose-built Vector Databases Matter for Your Use Case

Key Takeaways

[Music] hey Y what is happening everyone welcome welcome welcome to another mobs Community round table today we got something special in the works talking all about Vector databases and hopefully going to Demis a bit of what you're looking at when you're thinking and you're testing out your vector databases I figured you know what as we like to do around here I might as well bring out our guest of honors with a little diddly and so I brought my guitar and if you will permit me in this moment I'm going to start playing that guitar because we've got three incredible speakers or panelists if you want to call them that and it's only right that we give them a warm welcome so while I am jumping in here and making some noise or what some people might call music I would love to hear in the chat where you all are calling in from let's get this started huh [Music] [Applause] today we're going to talk about these Vector databases we've got three of the best out there coming and giving us faces Frank Eugene and Yen are going to be schooling us lots of head of AI platforms and ecosystems at zillis head of AI n l at zillis we've also got a developer Advocate it's time to start the show and if anybody knows what what's going on with these damn Vector databases drop a question in the chat we want to hear from you heyo let's bring out our guests of honor now where they at Mr Frank there he is for everybody that do not know head of AI and ml at zillas I see you laughing you were not expecting that kind of intro were you I I was not expecting that kind of intro but I'm very pleasantly surprised as I am always when someone serenat you we've also got you Jen where you at man hey there you are the developer Advocate at zillas and last but not least Jen where you at there you are head of AI platform and ecosystem so today folks we're going to be talking all about why purpose built Vector databases matter and and if you should be thinking about them for your use case we really want to get into a deep dive on the vector database scene because right now it feels like just about every database out there has bolted on a vector offering and so I think there is this huge question in people's eyes as far as when do I just use the vector database that I've got and use these Vector offerings from whatever my favorite Vector database or favorite database flavor is versus when do I just totally look at going Allin on a vector database and why I would want to what are the trade-offs that I'm thinking about and so I'm excited I can't believe we got all three of you on here on this call at the same time it's an absolute priv privilege and an honor and so I think what we should start off with and Frank because I brought you up first I'm gonna direct this first question at you sh when it comes to Vector databases there are always the questions of if you need it or not because these llm context windows are increasing right and most specifically when we talk about Rags I think a lot of people as Google put out their their paper like oh do we need Vector databases anymore because now the LM it can throw the Bible at it and you can throw the Quran at it and you can throw the tietan Book of the Dead at it whatever you want to throw you can throw it all and more at it so where you fitting in this how do you look at it well yeah you could throw the Bible at it you know you could throw the Quran at it whatever you could throw the you but can you I think the real question is there's a couple of key key factors here I think to think about first is do you really want to pay if you're going to query over let's say the Bible do you really want to pay a dollar per query right you do you really want to say okay I'm gonna have all this stuff into the context window for my large language model and I'm gonna pay a dollar to ask a question right now if I have a 100,000 questions I have to pay $100,000 just for for you know just for all of that so that's one question so it's as the context as you have more and more context the the price you know what you're charged uh becomes more and more expensive and not just from a dollar perspective it also from a computer perspective as well so large language models um you know they're transform based at the end of the day so they have uh they really are quadratic when it comes to token L uh in terms of compute complexity right in terms of runtime complexity so it's both factor of time as well as cost I think the other thing to think about is okay you know you have a million tokens that you can feed your large language model but if I'm trying to index let's say you know the San Francisco Public Library um or if I have a ton of books I want to index it's not all going to fit into the context window at the day still I think it's still you know there's these two factors irrespective of how long your context window is irrespective of whether or not you have a a language model that you how large it is uh you will still need Vector databases to to to to Really retrieve uh to store and retrieve all context that you really want yeah I see eugin I I see that you were like thinking maybe oh should I go should I step up to the mic I'll just uh the I mean like the other thing that I would say with this is just like you know they still they still have this kind of like lost middle problem and like um you know I don't know what like uh claud's sorry Claud 3's new cost is like $75 for a million tokens and that's that's a lot you know so just to kind of add on to what Frank was saying like a dollar per per question might actually be underestimating it if you're G to be talking to the Bible you know yeah and it does feel like I I often wonder about the performance on if that is the best way to do it so they're is a clear piece around speed not being up to par and then the other side is like you're potentially going to burn a hole in your pocket yeah but like is this actually going to give you the most relevant answer is probably the most important thing because maybe people will pay at the end of the day and they will wait but if it also doesn't give you that the most relevant answer or the best answer that you could get I think then it just makes no sense so that's another piece that I I would love I haven't seen much around that I don't know if any any of you guys have yeah I think that's an interesting question I think a lot of the a lot of the research at least I've seen today when it comes to retrieval is is very much uh what people like to call needle the Hast stack retrieval so they'll have you know you'll have I think one of the earliest examples is you have Harry Potter you know you have the Harry Potter series they add it whatever Harry Potter in the gimber of Secrets and you take this entire book all of its tokens and you fit that into the token window and then right in the middle you have a sentence like um and eug Jan is a developer Advocate as Zill is or something silly like that right and Eugen loves machine learning something like that but to me it's a really really unfair comparison first because how many times is the phrase machine learning appear in the Harry Potter series right how many times does the word eug Jan appear in the Harry Potter series probably no time like it doesn't appear at all so it's very very easy you know it's very very easy for it's very easy for a large language model for the these attention mechanisms that are in LMS to pick out that that that subset those phrases and the other thing is I think there's also a recent research I was reading a paper recently it talks about the Holy Grail of sort of large language models is re and generalization right across you know different uh you know across different t asks and if you try to do reasoning across you know if you try to you know you have long context and you add these sort of bits and pieces and you try to do reasoning in Cross law context the performance becomes much worse as you increase the the LA of your as you increase like the amount of tokens larage so at the end of the day it's still relevant you know retrieval is still very very relevant my eyes and I think it will continue to be now maybe one day I think one day really really far down the road it doesn't really make sense to use rag I I do see that potentially happening but I think in the short term doesn't matter how long these Contex very yeah the other thing that I often think about too is like how much goes on outside of just the prompt and the context window and how many other pieces you need to be thinking about that is not just like oh cool I'm going to put all this data into a context window and then it's going to make my life easier or and really it reminds me of that famous Google paper back in the day where it had the model and that was like just one part of the entire system and so it still feels like this like the context window is just one part like how clean is the data how are you ingesting that data how are you quality controlling like if you're getting tables from PDFs how do you make sure that those tables are actually what are in the PDF and there's there's just so many other question questions that are beyond the the context window and so that that's another huge piece yeah absolutely I think I I wna I want to give J Jang some time to speak about this as well because he he does uh he does Zill pipelines and he's he's got tons of experience you mentioned PDFs tables all these kinds of different modalities of data but the the analogy that I like to use is that the context window I saw this somewhere I don't remember where I saw this but I thought it was a great analogy right the context Windows like your is like um or it's like it's like a it's like an L1 L2 L3 cache and your database is is like this now can you store everything in your database probably not unless you want your cost to be astronomical but it is meant to give you it's meant to give you different different layers of storage so to speak right and and you don't want to just as you don't want everything to be in your RAM all the time and continue to ask questions over that you don't necessarily you want to offload some of those capabilities to disk um to your vector database so anyway that's just my two sents I think I I I think perhaps maybe we haven't done as great of a job of educating the community as why Vector search is so relevant even with these long context models that can do needle Hast Ty retrieval but um that's what we're doing here right and um so uh so you know hopefully in the future you know things will things will change talk to us talk to us about pipelines yeah yeah so come coming from a background of search indexing I I do have a lot to say about this this kind of um um I would say a hype of long contacts will take over everything um to me that doesn't doesn't really make sense from a production perspective because um well there are just so many things to consider where are serving user queries in in a real world or considering the latency or considering the user experience you're considering the cost like all sorts of um of aspects are not considered when when doing this more abstractive like academic research on needle in HC um uh experiment it's more of a stress testing of the capability of large langage model rather than promoting while you should use this in your production setup um I think that that makes no sense but why um so I can I can give some examples or some throw in some some ideas here so one thing is um we like in in the in the search world we always have the analogy that the the serving time like serving user query is just tip of the asper there's a whole amount of efforts being spent in the offline indexing time uh doing so is because um a lot of the analysis of the understanding of the content are like really really expensive um they are like um they involves Running Money machine learning models and they involve like um doing very lengthy analysis on those content and you cannot simply cannot afford doing that when the user is sending a query to you right you are supposed to give back an answer within like 10 milliseconds something like that so that there's a there's a a huge principle of doing all those kind of heavy lifting stuff offline and that's I think that's that's one of the main um principle of rack as well right for for R you you do retrieval um to augment or to enhance the large language models and a huge part of retrieval is that you need to understand the content and prepare all those efficient index for those content before the user query comes um so that's in that that pattern is a great advantage or value of of right um there are also other considerations because um by doing this offline you are saving time time not only time but also cost because if you were not doing that then every time the user quiry comes you need to do that once more like for for that particular part and that's not affordable so even though like say understanding this whole Bible takes say $3 right like spending $3 once is is whole different story that's spending in for millions of times uh when the user query comes so all those sorts of things were not considered in this experiment I'm I'm like personally excited about this experiment because that's really extending the boundary of the capability of large language models but like looking back um into what kind of production system we want today I think rag is still relevant it's it's never more relevant than than right now I really enjoy this idea that you're talking about about like what the needle and the hay stack should be looked at is more of a stress test not like a daily driver type thing it's not a a best practice that people are advocating for it's just like how far can we push this and at least not today and we shouldn't be thinking that's how we we want to architect our systems and we want to like Leverage that it's just that okay this is another tool in our Arsenal I also don't want to let anybody think that I did not let it um go over my head we are using the Bible as our reference here which some people on YouTube have said I bear striking resemblance to a person in the Bible and so we use a different book now if you want we could talk about maybe like encyclopedia for yeah oh the encyclopedia works as well yeah the encycl or Harry Potter was great yeah Harry Potter's good Harry Potter is probably more culturally relevant than the encyclopedia so then now that we got that out the way I uh Frank were you able to find what the name of that paper was that you you referenced a minute ago and if not I'll let you look for it yeah I unfortunately have not yet I'm need a little bit more time there and in the chat when there are so many papers coming out that it just kind of drives me crazy and you know I I I I try to go through this these days I only read the the intro and the conclusion smart man you are a smart man yes I try and uh understand it and then if it's really interesting then I'll try and dive into it but all right uh there is one thing that I want to talk about so we talked about this rag idea and we talked about hey uh is is there still room for Vector databases there but let's move on a little bit more because as we are talking in this the general studing of Vector database a purpose-built vector database versus like just Vector database bolted on the normal database that I'm using whatever my flavor is and before we get there maybe can we set a bit of a scene because are there other things so if you have another database that is specialized in certain um areas and then it throws on the vector database aspect to it that's one thing but with like vector databases are there other types of data that you would think about storing in a vector database and why if yes like why would you do that I guess I can I can take this one uh to start with and then uh Frank and John can give their opinions on this um so I've been saying this for a while because I I heard this from James and I was kind of curious to look into this um but Vector databases are really compute engines and the whole like name Vector database is actually just a Mis nowhere uh it's just that it's easier to tell people it's a database because you're storing something but the reality is you're actually storing something in um you are storing vectors in memory but most of the data that you store is actually held into permanent storage so like S3 or minio or something like that um so you can store basically any types of data that you want in the VOR database uh just like you could treat it just like a no SQL database and um you would store things like text for rag uh or if you're going to be working with multimodal rag you're going to be storing images perhap you know probably links to images um videos things like that and you should store this stuff and you should store the metadata that maybe tells you like hey this is when this was published this is who wrote it this is uh you know XYZ about the um the information that you're storing uh and this will basically let you use something that was originally built as the compute engine to do Vector search on top of you know uh uh a um a permanent storage layer as if it were a real database and this is basically kind of how let's say no seal databases work um and the difference between the way that a purpose-built vector database works and something that you just tack on to let's say no SQL database or seal database is uh in the way that it's designed and um you know we all kind of have this intuition that uh things are best used for the purposes that they are designed for and so Vector databases are designed to be able to do this high amount of compute efficiently and effectively at scale I mean let's just take open source for example if you just look at some of the open source projects you can go in and you can look at you know how much work was really put into this and um I'm not going to say that that is the only indicator like how many lies of code maybe is not the only indicator of how robust uh or feature-rich a system is but um it is definitely something where you can kind of look at and be like okay like you know there's a lot of um you know features here that kind of help you do things perhaps at an Enterprise scale or at a most or at a more robust uh level yeah I think um if you if you think of maror database in the E ecosystem of search or information retrieval um it is supposed to be the most efficient way of retrieval some information that has been offline index and and organized however when you organize the information you probably don't only don't only want to organize the say the IDS or the the the index of it you also want to kind of put the original content as close as possible to it because that's what you really want to retrieve and return back to your customers or clients right so with this mindset we we do design ways of storing not only Vector data but also other kind of AD structure data which is the the source of the information where those the vectors are generated from we by PL placing them closely with vectors and return them at the same time or together we we can help the developers to build more efficient applications because they do not need to retrieve this um say raw data from somewhere else again and they they can achieve the the the best efficiency from this paret yeah I was sorry I was sort of busy searching for that paper I have not found it yet so I I I apologize to the uh uh I search through a couple different places I will find it if I don't find it by the sear if I don't find it by the end of the session I will I will build a new search on Twitter I'll build a new search yeah I'll post it on Twitter I'll tag the yeah archive you're gonna build a whole new archive search function so that you can find it next time yeah I'm G all these Pap you know because cuz I mean because the way the way I'm I'm searching for right now which you know it sounds really it sounds really it's pure keyword based on I'm searching like you know reasoning capabilities as context window increases um and it's just not popping up there's actually there's actually some diagrams in there which I actually remember very very fondly uh and if I find those images I'll I'll sort of share them share them with you as well Demetrios maybe you can post them but yes please we'll send an email out to everyone to the uh the other thing that I was going to mention unless did you have something you wanted to tack on there I I imagine you were I think I think Jang and Jang and you J said it really really well right at the end of the day you know this a vector database is a compute engine but the the one thing that the one thing that I do want to say this is my personal opinion um I do think Jang disagrees with me here so uh it would be interesting to just ch about chat about a little bit more if you look at look you got guys like Monger to be coming in and um data sex and and you know elastic or whatever I'm not I'm not trying to throw shade on them I think these are really gray systems but what what does their marketing team say right they say hey you should just come use us because you can store structured or semi-structured data in in in here you you can store Json documents in mongod DB and you can store Vector data too right K kill two birds with one stone right just just use us and I think they really missed the point point in that these databases at the end of the day you know Vector databases were built for for different things Vector databases are built for the ground up to support Vector search and filtered search and and you know hybrid Spar Den search it doesn't mean that Vector databases don't support Json data right you could I've I I have a blog post that's coming up that's all about just storing using your vector database as a pure no sequal data store right you can just store Json data in it without vectors you can do that if you want it's like hey if mongodb wants to play like game we can play that game too it's just that we don't because I think we've always been at least in my eyes we've always been very much about accelerating the adoption of vector search and accelerating the adoption of of embeddings and embedding based retrieval that is my two cents um I know there I know there are differing opinions out there um and I'm not saying that you should store Json data and Vector database I'm not saying that all saying is that if you are if you don't want to manage multiple databases if you don't want to manage mul sources of data just start with a vector database right future proof yourself right don't don't you know don't listen you know don't listen to the you know don't drink the Kool-Aid so to speak or what is what what is the phrase that folks use for this kind of stuff these days what are all the cool kids saying these days what are all the cool kids saying you know I doesn't want to throw shade but I'll throw shade try to store a million try to store a million vectors in mongod DB and retrieve them and come back to me you know yeah I mean I think I think mongod DB is great it's excellent right but at the end of the day it's it's it's you're doing different things U it doesn't mean that we don't support Json which don't support it as well as mongodb does just like mongodb doesn't support Vector data as well as we do that's and that's exactly it it's like why are why is there a whole Market of so many different databases because each database has their specific flavor that they do things really well at and for use cases you need these different flavors because it is what you're optimizing for yeah um as a matter of fact we are actually building this data connectivity with um all sorts of connectors so even if some of the data while they there are they more meant to be um stored and managed by another purpose build uh uh database or data management system um we do have a wallid pass for those data to uh become vectors like being generated as vectors and then um being um being sent to Vector database for efficient and timely retrieval so there there's some questions coming through in the chat that I want to ask and then I want to get into a few ways to Future prooof yourself and how to like use specific features in the vector databases that can help you uh but we've got one question coming through since we are talking about like pros and cons of different Vector databases uh someone is asking about milis vers pine cone I think one piece is I'll start milis is open source uh pine cone is not right I think that's probably the the biggest thing in my mind but you all have fielded this question many more times than I have so is there anything else that you can say I'm looking at Eugene licking his licking his lips he like let me add it hold on let me turn myself off mute I got other stuff to say I was already talking about now pine cone this is the best day of my life you got Center Stage dude I I don't wantan to I don't want to be too inflammatory here uh but uh go on Pine's website see how big you can how many how many how much data you can really store we've got some we've got some benchmarks for this they're open source as is the ethos here as zillis uh and you can really take a look at the data sets we use and the way we test the data and Benchmark the data and just look at just you can just text the benchmarks yourself like I don't think I need to say too much about this I'm just going to say look at the benchmarks it's all open source you can see exactly how it's done all the data is out there you can bring your own data just Benchmark it yourself yeah I think on top of that um other than the open source me we Al we also have um managed me which is C cloud and in zillis Cloud we added tens of features on top that we have a um even more performance even though like m is already performant we have even more performant um core or like indexing engine um on Z cloud and we have this um this feature called Z Cloud pipelines which U provides the streamlined fashion of um generating V Bings and then storing and retrieve them Vector database and we also have the data connectivity that we mentioned we are building data connectors with all sorts of data sources in this uh ecosystem and all all those features I think um those provide some differentiation between us and and pine con um if if open source is not only the the differentiator excellent the the next question coming through in the chat is asking I'm Vector DB curious but very ignorant on the topic how flexible or feasible would it be to use the same Vector DB of images with annotations for data loading during training as part of the inference deployment um so you're looking to use Vector databases in your training Loop and I think that's it's it's actually not a common use case that we see today just across the board but it will be very very common for and we've already got uh you know I won't necessarily name who but we already have folks that actually use buis um in the training Loop not just for large language models but for other models as well and one of the things that they use it for is for example the duplication of your training data and one really interesting way that you can use it you can actually just say hey I'm going to I have this model that I've trained I'm going to use it to fetch items that are uh make sure that I fetch items for training my new model that I know are are not duplicates of what I've already used uh and there's also other ways other really really interesting ways to use your back database in the training loop as well which I won't get too much into some of the details but specifically as it comes to your use case when it comes to in images with annotations um if you have let's say you're train you're trying to train let's let's say a very very large multimodel model uh you can take some of these smaller embedding models that you have and you can especially if you have these multimodel models and you can say hey I want to maybe for this particular batch or I want to train this particular expert on some you know one particular cluster of images or one particular um one particular set of images right and that's one of the really really interesting just one of the large subset of ways I think you can use milis you can use not just milis right other Vector databases as well inside of your TR I think we're going to see more and more of these sort of use cases for Vector search in the future um really ties in very very nicely with mlops yeah yeah we did see some use cases of using VOR database as retrieval and for D duplication of images during the fining phase um I don't think we have seen like kind of combining this workload with the online serving workload because um one is like one comes in bu and um it'll probably take over some of the uh computer resource of the of your collection so even though both of the use cases are um are being seen your in your uh uh client use cases um it kind of makes sense to to split them into two different collections or or even two different physical setups um so that you don't hurt the performance of the online serving with your like offline training use cases yeah there's also like we can kind of actually see some people are already doing something similar to um uh uh the the the question askers question um so go look at the question askers question right but uh yeah if you go look at you know like arise and V Soul 51 and Galileo and triera like some of these companies are doing this kind of thing where they're using um a vector database to kind of show you the data quality and the way that your data is clustered and uh I know I just worked on something with uh Jacob from uh BOS 51 where um you know we use milis to store some data and uh to load some similar data and and to to observe some of the vector embeddings uh and and how they look like and so I think this is definitely something that people are looking at doing and and in fact already being done by um a bunch of startups great question beran I really appreciate that one because I did not think of that at all and so it's almost like yeah that beginner's mind came through and who would have thought that this is actually a pattern that you're starting to see emerge and probably will be something as we move forward so now before we jump into embeddings because I feel like we can't talk about Vector databases without talking about embeddings I do want to talk for a minute about the different like ways to Future prooof your AI applications and especially when you are using a purpose built Vector database like what are things that we should be thinking about as we are moving through our life cycle and like how we're setting up our system and when it comes to Mo most specifically like the vector database are there features that we need to be keeping in mind is there metadata that we need to be thinking about all of that fun stuff there's a lot of new features coming in milis 24 um like hybrid search and by the way I'm going to Define hybrid search here hybrid search is when you do metadata filtering on your vector on top of your vector search search pre-filtering on your metadata for your VE for your vector search okay that's what hybrid search is anybody who says otherwise is wrong uh and then there's also a multi Vector search coming so that's when you're going to be able to search multiple vectors and you're going to be able to uh rerank based on how the vectors are so for example uh a use case for this might be that you have both um a semantic similarity aspect that you would like as well as maybe a visual similarity aspect that you would like in which case you would want that ve vors to do the semantic similarity search to get your traditional ve or what we call Vector embeddings right now to do your uh semantic search and then you're going to want maybe something like pixel embeddings to do your visual search so uh there's definitely these use cases that we start seeing come up for multi Vector search and you know uh maybe even you have text and you want to compare it and rerank based on the text or uh things like that so there's a lot of these kinds of uh different new use cases that are a little bit more let's say Advanced that are coming um in milus 24 yeah This this term of hyri resarch is really funny it's like everybody is using it for different meanings um yeah to my understanding well initially I I was thinking like H resarch is probably you're you're searching the same thing through different modality or like different thought different ways of representing things well to to avoid the confusion I will just I will just describe the way of search directly rather than using this fancy terms um so yes gen side in the upcoming 2.4 release where supportting uh we're supporting the um well not hybrid the mix of ser with with st and sparse iding because um there are definitely some great features or properties of sparse iding they they grabs the they grabs the the individual concepts of a l of a lengthy documents um a bit better or like in a more understandable way than the dening Bing so that um there are definitely some Dev for feing um sparse search if not in stand alone but also um like combined with the the the dens iding retrieval so that yeah we we toally Echo that requirement we we build that feature um really really fast um and we are releasing it uh very shortly there's also um while we're still enhancing the other aspects of the uh of the me was offering um just internally we're working on providing a uh a set of like how uh easy to use um utility functions to generate the embeddings from the client side so that you don't need to play with all sorts of Frameworks and and get confused in between them uh if you have a relatively simpler um requirements regarding which kind of factory eding you want to generate for example if you want to generate the sparse embedding from splade model or bm25 which is more like old school way uh of representing text or um definitely the hugging face sentence Transformer and open ey and other like providing services so you bring up a very good point and I think it is the perfect segue into embeddings in general and there was a question that came up in the MLS Community slack and it was kind of around the lines of so what are some good practices around generating beddings what are the best models and the consensus was it's kind of just like throwing spaghetti at the wall right now uh you see what works and hopefully you get lucky go on to the leaderboard grab a few of those models and then does it work for your use case cool you're lucky it does it not keep searching you know maybe you guys have seen better uh practices than that because it feels like there's got to be a different way right well first I I want you to say there's whole story of marchine learning and artificial intelligence is like throwing strategy onto the war and see what works all thisal system of Technology are are being developed but we don't want every single developer to do that again and again we we do want to provide some insights on on on that yeah I would like Frank to to speak on that Frank is definely wild with work in this area okay my my my personal opinion is so I see I see a lot of folks I talked to a lot of devs out there and you know when they when they're building out their application that leverages llms or leverages you know generative AI they they just say I'm using open AI API or I'm using open AI at an end point which is great like I don't want to you know I don't want to throw on them I think it's a great embedding model but like there's not one like it there's no way that there's one embedding model that fits your use case right if you look at I have this I I wrote this I wrote this blog and there's this great example on there where um I I don't remember where I got this example from again I I get a lot of these from different locations different locations around the web from different people and I sort of repurpose them so you know this is a great example um where one sentence right is let's eat I'm gonna say let's eat Eugen here let's eat let's eat right and the other sentence is let's eat comma Eugen and these mean to very very different things right now for some applications let's say you know for some applications let's say you're you're talking in a legal context or you're talking in you know um that type of uh you're talking in in things that that are a littleit fuzzier you probably want the embeddings for these two sentences to be very very close to each other right they're very very related they have almost the same tokens with the exception of some punctuation um and the people the person that I'm talking about for both these in this case it's a hug in it right and in other you know if you if you want to if you want to talk about these two sentences from a the perspective of what does it actually mean these mean very very two different things two very very different things in one sentence I'm saying let's actually you know physically eat eug Jen which you know let's not do that and in the other sentence I'm saying hey e Jen let's go get let's go get lunch or let's let's go out to eat right these mean two very very different things so for some you know it depends on what you want to do you doesn't there's no one size fits all embedding embedding model and and if you look at a lot of the metrics out there of how oh you know embeddings are't as good as B of 25 it's it's because you're not using something that is tailored to your application right uh that is tailored you know that is semantically relevant to your application and I think try my recommendation if you're looking to do if you're looking to get Ser Vector databases or vector search try different models I know it's a an you know hugging face and sense Transformers and you know there's Voyage as well they make it you know all these they make it very very easy for you to try different things try different models and see what what works best for your application uh use the lgtm 10 test right um you know if try try down 10 R examples let's see which one looks the best um or try 20 different examples and see which one looks the best at the end of the day there's uh you know humans are still the best evaluators right uh and you should depending on what it is that you're trying to build evaluate your embedding model according I'll give a slightly different answer here um he it you should F tune your embedding models you should find tun your B models with of what you're going to be doing um and actually I this was I talked about this in my paper with uh boxel 51 as well um we took some examples with uh clip vit and the C4 10 data set and we search three words uh Ferrari Pony and Mustang and so you know you can see Ferrari is going to give you cars back Pony is Gonna Give You animals back but Mustang is going to give you both horses and cars back and so depending on what you're working on let's let's say for example you're working on cars you don't really want pictures of horses back and so you have to find Tor embeding models to kind of get back the right uh um context yeah um I think there are just just so many embedding models available on the market and um unfortunately most of the developers just choose say open ey in beding not because of its quality but because of the the name of open a right so that that's definitely not the way I I I would recommend um I think surprisingly just with a very small data set you can probably test the effectiveness of the inviting model on your particular use case um so that my top recommendation is that if you have this um this capacity of doing some testing and evaluation definitely build your own data side which can speak um best for your use use case than any other open this such as Ms macro or beer um and also um this this um this Public public leaderboard from MDB right um this Mt leaderboard has been overfitted by many many models already so it's really hard to Simply judge I mean that's still instructive but we we shouldn't simply judge by by their ranking on this leaderboard however that being said if you don't have the capacity of doing any evaluation then choosing from some big name is to a practical way of making the drop down so there there's something fascinating that you said there like creating your own data set and I think I just saw some post on uh one of the social medias talking about like how a big question an open question is how much data is enough to be able to say like yeah this is good and especially if we're doing for taking your Jan's recommendation and saying all right we'll find tunar embedding model have you seen best practices around that like what is a good amount of data look like and how can you assure that quality of that also like how can you make sure that it is nice and diverse robust all the fun stuff in there too so there was a recent paper from 2023 um or maybe it was 2022 I think it was 2023 it's called neural priming and it comes out of uh um oh man I don't know which school it comes out of but it's it's uh Ali farhadi is the the lead researcher for that he's currently the CEO of AI 2 and basically what show is that if you prime a model uh with some images so this isn't even fine- tuning this is I mean this is like kind of fine tuning it's like very you know it's very small amount of fine tuning uh with the right amount of with the right context you will get um some decent percentage uh Improvement in results and of course this will vary from data set to data set uh but for Te for images you're really only looking at 2025 images to kind of skew the um to skew the model towards what you're looking for and forch next you're looking at maybe like 100 to 120 sentences um from my just my experience of working with evaluation I think the bare minimum is definely tens to hundreds of of of examples to give you a meaningful indication of how well this model performs um if you have thousands of examples you are probably in a good shape as a general rule of St so basically it to try and summarize this so that it's not throwing spaghetti at the wall right what what we're going to do is we're going to go on the leaderboard we're going to grab a few models test those out with maybe a 100 if you can 100 different types of um of examples and then if you see that there's one that's performing well go ahead add that extra tuning fine-tune it with as much data as you can but you probably don't need to be thinking more than like a thousand examples examples you probably should be thinking more in the hundreds of examples and may be able to get away with the tens of examples the data has to be really high quality though right so you don't want to what large language models are really or what large langage models are really good at is you know they have just a huge wealth of pre-t trending data and if you want to if you want to use very very little data that's okay but you just have to make sure that it's really really relevant to your data set it's you know very you know it make sure that it's a good representation for what things are actually going to look like when you go into production yeah that's so true I and that was another great paper the Lima paper like less is more it's alignment I I remember that one I'll drop that one in the chat that one I can definitely find because it's an easy name right uh Eugen if you have the link to the paper that you've referenced throw it over so before we jump folks I want to give a few cool maybe hot takes I don't know we'll see uh mild takes to but I think there's there's something that we could talk about which has been taking Twitter by storm right which is the colar model and it feels like these late interaction models maybe eug Jan can you talk to us about what those are and why that's interesting and what uh what you think the reason for it to become more popular right now is yeah so this this C bear um paper and the late interaction model behind it that's that's really interesting and that's catching the eyes be because right now um people kind of realize that this Paradigm of iding model plus Vector retrieval is a perfect thing for like um for for retrieving a bunch of like candidates of the final result but if you add another layer which called rerun C model on top of this to kind of refine the results and and select the the word best from the the best candidates then you can probably achieve better results we did some Quant uh quantifiable analysis on this yes you you can improve the um like the record numbers or uh um all those quality metrics by a few perc so that's that's uh that's meaningful difference however this reine model is super expensive um you are probably doing something that you should do in offline indexing at online query serving time just because you're using this runker because they they use the cross encoder model uh which needs to basically just look at the the the whole uh uh query and all the candidate documents rather than just Computing the cosine similarity score which is quite cheap computation wise right so to solve this problem there's uh this this late interaction model being proposed which kind of combines the the good things about um embeding Vector retrieval and this Fring coder ranking it generates a whole set of embeddings from one single query or document and then do the late interaction by Computing the um I'll say the the similarity score between them at query time so that's cheaper than reer but that represents more raw information than the single dense Factory bed um so I think that's that's a great idea um but again putting it into production that's that's questionable uh to I mean at least per se um because it definitely use more like space to encode this information um so that cre unique challenge during the online retrieval time and moreover like a lot of the efforts from those Community has been put on single Vector dat embedding so that the embedding model is being refined and being like uh being trained over and over with better data so that it does achieve better performance than the co bear embedding which has been only trained by the uh Academia Community however we are seeing like more and more effort on on this so it's hard to say what will happen in the future um we are looking into this and we are also thinking of um where we start to think about how to how to integrate this pattern into the vector retrieval uh facilities that we are Bing so that in the future if this becomes the I'll say a popular choice among developers will have a solution that pairs it here's your hot take for this okay for get coar the only coar you needs to know is Stephen Kar uh what you should really be doing is you should be using mil's multivator search with reranking to produce the same results at the at at a better at a better price and it's easier we all love cheaper that is very very useful to know the cheaper the better I think and that's cool that you all have been inspired and taken that into account I know we talked a little bit about I I want to take the next like two minutes just to shine a light on some of the cool stuff you all are doing at milis and zillis because we talked about the pipelines we also talked about the um the ranker what other features out there I know that we said that there's in the pipeline we've got a few features that are coming out like the is it sparse embeddings and um and a few others but like the re is coming out um so the we have just supported the reuner model in the bu Cloud pipelines and for the open source M us uh we are about to support the spars and den Vector hybrid retrieval I cannot avoid using the word hybrid um in the next release Frank loves it Frank's like oh we got to get a better naming convention around this well fellas this has been awesome I really appreciate you doing this one thing as we end I would love to ask you a question that I tend to ask when it comes to Vector databases it's not like doesn't really fit into the specific is this a specific Vector database or like uh kind of catchall database that has vectors support theme but I do I I was talking to to some data Engineers two weeks ago from Quantum black and they were saying how hard it is to make sure that when you put information into a vector database for your Rags how do you ensure that that information if it gets updated it is the one that is being retrieved when you want to go and ask answer questions so they gave me an example of like an HR policy being updated and now all of a sudden you have less vacation time and so if you have a chat bot that gives the old vacation time that's not good because then the company all of a sudden is like hey no actually you only have 10 days not uh 20 and you could get some pissed off employees so how have you all seen best practices around that yeah should treat your vector data datase as like a living entity right as you as you you know create these uh and this is one of the great things about you know what some of the stuff Jen has does had Jen and his team have done which is as you have as you you have all these all these data you have all these edings inside re VOR database and create these uh these real time sources these real time syncs and make sure that it's up to date at all times I know that's I know that sounds I know that's like a really really simplistic answer um which it is but I think at the end of the day that's really your vector database you'd want to treat it almost as other ve other as you do other databases as well as a single source of Truth for semantic information for semantic representations and that shouldn't change um just beca