Navigating the Vector Database Landscape

Key Takeaways

you're listening to gradient descent a show about making machine learning work in the real world and I'm your host Lucas bald EO Liberty is the founder and CEO of pine cone which is the leading Vector database for machine learning applications previously he was leading research at Amazon AI labs and Yahoo's New York lab he's got to be one of the world's experts on Vector databases and we talk about everything from the tradeoffs building Vector databases to the applications that are built on top of them to what motivates him personally I hope you enjoyed this episode all right man well thanks for taking the time to to do this um you know obviously you're the founder and CEO of pine code so I feel like we better start with an explanation of what that is and generally what a affector database is uh sure uh glad uh we glad we're chatting uh so yeah I mean um uh pine cone is the uh the facto standard in uh Cloud Victor databases now and Victor databases are the uh the way that large language models access information and so people try to make their AI knowledgeable whether it's semantic search or chat or what's called rag your trival augmented generation the tool of choice is a vector database and within that the tool of choice is almost always spine nice well let's get a little more um specific can you can you describe how they work sure so vcal databases really are uh a new kind of database that is actually used more like a search engine than a database often uh where object are represented by what's called a vector which is a an array of numbers so it's like literally a float array uh uh and that is the numeric representation that uh models foundational models large Lang large language models multimodal models give to any object whether it's an image or an audio snippet or a piece of text um that encode the meaning of that object in some sense what does it say what does it stand for what does it contain and so on okay and the the role of the vector database is to be able to retrieve similar objects things that mean the same that contain the same topics that contain the same content that are relevant to some question that align or associate with some task um and because of this numeric representation because of this like fuzzy matching uh you can now retrieve over very large amounts of data uh relevant information so you can build question answering for example and make it very knowledgeable over your own company data uh extremely easily and efficiently which you couldn't do before and so the the challenge here is you're trying to find the set of numbers that are kind of closest to another set numbers is is that right like you s yep that's pretty much it you nailed it that's everything we do uh no I mean it's think about that as uh so this is the basic function you're right I mean in the sense that uh you can think about alignment of these arrays think of arrays is vectors sequences numbers you can ask yourself how aligned they are or if you look at them as vectors in high dimensional space what's the angle with them you say Hey or what's the distance between them and say hey fine retrieve everything from some period some area of space or some small anle around what I'm looking for but first of all that alone becomes exceedingly hard when you have hundreds of millions or billions of of uh of these vectors and you want to do this efficiently and quickly and at at Cost but you uh add to that the need to filter by metadata the need to boost by keywords and other like sparse signals so you want to now change the score uh the ability to do that you know uh at bursts of queries with low latency and sometimes do this very infrequently and so on so you really have to build a very very complex system they can organize data correctly index it correctly access it correctly and so on can you describe like for that kind of fundamental task of you know kind of finding the closest vector by some Metric to another metric like how do you how do you even get started with that like what's the most basic way that you would Implement a vector database I I'll start by saying that uh I have in my career alone built quite a few different systems that needed to do something like this and so add serving in Google and feed ranking at Facebook and shopping recommendation at Amazon and you name it all use uh Vector search behind the scenes to some capacity okay and and of course thousands of other applications as well not just those right um there isn't one way uh to uh implement it and in some some sense that's what's so exciting about building a company like Pine count we can automate we can experiment we can Implement hundreds of different techniques and and figure out what works best and and optimize and so on it's it's a very interesting engineering problem but if you want to just simulate what's this what's the basic thing you can literally compute the literally take a numpy array as your Matrix of vectors and literally compute the say cosine similarity ukian distance Lally the sum of distances Square from your query and just take a threshold in fact my LinkedIn profile for a while my title was like a oneliner in Python that did Vector search nice it's like like that was like after a while people told me what the hell is this nonsense I'm like maybe I'm just confusing people I thought he was I thought he was clever but maybe maybe you know maybe the people thought he was either obnoxious or or enigmatic so it kind of I stop so you can do it in a naive way it's just extremely slow and inefficient extremely memory consuming and of course you know when you run these things at scale of course you can't do it on your laptop not you know and of course you can't do it in Python either so like what are the first you know optimizations that you would think of and what are like the the big tradeoffs of the different ways you could approach the problem so there are there are many different um optimization functions you might think about optimization function I don't mean mathematically I mean as as an operate right are you minimizing latency are you mis minimizing cost are you minimizing operational you know complexity and and overhead right uh are you maximizing throughput I mean this like again as as every engine you really have to figure out what it is that you're trying to do the vast majority of uh libraries for Vector search out there really only optimize F sometimes really minimize latency but but not always like really maximizing throughput is in some sense became I don't know why the main competition right uh uh I can tell you that we had pine cone even though we ran a competition called Big Ann in which we Benchmark these and we open it as an academic competition ran it we actually participated as well and we we actually dominated of course all the you know all the tracks which is maybe not surprising because we're professionals this is an academic competition but this was just to make the point that so what is interesting is an academic exercise it's not what most people care about the vast majority of our customers don't really care if uh they uh get a th000 queries per second or 2,000 queries per second on a single core or like a whatever on a single machine uh a because they don't have that flut they might have five quers per second okay or 10 for each index for each small index and B because they don't run the hardware they usually manage service so they really don't care how much Hardware we use behind the scenes it just care about how much it costs so we figured hey okay fine we get it so we optimize for two things we optimize for cost and you optimize for ease of use and what we launched last month almost exactly a month ago is a cous offering for pine cone which seems like okay you just took your engine and made it serverless and but it's it's not that at all we actually had to rebuild almost everything from scratch to be completely uh supported uh in in a cous way okay and we had to I I I'd love to tell you how we what we did differently but the outome come is that this is 10 to sometimes 50 times cheaper for most workloads and it's completely handsfree so you don't have to choose Hardware or algorithms or anything you've started index without saying anything and you just push data to it and it says great and it doesn't matter if it's 100 vectors or 100 million or 100 billion I mean it matters to us it doesn't matter to you as a customary for us it matters a lot well look I mean so that's that's amazing and I want to get to that but I feel like I'm like at a little bit more of like a basic level than you like I'm you know I I vaguely remember my like databases class and I can kind of understand for you know search how you can create an index and then the latency like won't scale linearly with the amount of data that you put in but if you have to compute like the dotproduct of your vector with every single Vector in your database how do you keep the database from getting slower when you have like you know 100 million and then you go up to a billion How do you keep it from getting 10 times slower each time you scale the size of your database by a factor of 10 so uh one of the things we're doing and this is one of the hardest thing one of the hardest architectural Feats in cist was the fact that as zaa comes in we in fact process it with a very highly specialized right path that actually arguments uh tiny not Tiny But small files in Blob storage that contain of uh vectors that are likely to be queried together if it's uking distance that may be close if it's if it's angular distance they might be within some cone um and it's in fact very hard to do at Large Scale so it's easy to do on a thousand vectors it's very hard to do when you have 10 billion or 100 billion to keep all of that in check and organized in blop storage and keeping a very good uh keeping keeping track of what is where is actually incredibly complex right so that's one part of it and the second part of it is a query planner the given a query now has in some sense like a model for each file and says oh out of the 10,000 files that you have each one of them might contain a few thousand vectors you can you you can even just look at these 100 right and with all likelihood they'll contain the top results you right so they might not uh all the time but as a as a usually they would and that's when we start talking about like recall and the accuracy so on so you have to trade off a little bit of accuracy for your speed but if you do this well you can look at a ttin fraction of the data and and get uh almost all the result so you want some usually literally all of them uh and as a system you need to also have the whole caching hierarchy done really well so the vast majority of the data is actually dormant and in Blob storage and you never touch it and never costs you anything and the small amount of data that is hot actually gets propagated to ssds and into memory and so on so you can actually be very very does you a question let me see if I understood it I'll try to repeat it back to you um it sounds like you kind of break up the space into some like regions and then each region kind of gets it own um kind of mini file of of the stuff in that region and then when you know something comes in that you want to match you like look for what region it is and and just focus on that particular file that contains the the vectors that are sort of in that area is that that is correct in spirit but it's actually a lot more complicated than that because some some vectors might actually appear in multiple places in multiple files uh because they have to for performance reasons uh and some regions might overlap some regions might be contained in other regions and be bigger or smaller and so on so it's not it's not as simple I'll just divide the space in each part of the space like these points go there and these points go there it's not quite as simple as that but conceptually that's a right way to think about it and I guess if you have like a stream of um new uh points coming in do you have to then at some point stop and sort of like um reallocate everything or kind of create a new index across this is that an issue so that's the beauty of of and why I said it was very complicated to build that because you can I mean that you can always say hey I'll take some amount of data and then I'll just after some period of time I'll just Rec crunch everything and and that's it and that's that's the that's what uh some are doing in some sense that's that's the trivial thing to do right uh that ends up being incredibly expensive and incredibly slow because that people have like these recompute like events when the data is bigger and every time it gets bigger it gets more expensive and so in the beginning it doesn't look very bad and so when you have 100 million vectors that's just okay it's annoying but all right but then you get to a billion and that thing takes a day and costs you $10,000 I'm like holy this is not acceptable and then it goes to 10 billion and this thing costs $100,000 okay this is okay we have to rethink this thing um well look I mean you've obviously been working on Vector databases for a long time and I was you know having breakfast this morning with a friend who was talking about how uh he's like yeah you just been doing these Vector databases way before um it was cool and and we were wondering like Did you sort of see these rag applications coming in some sense like I I remember like there were embeddings before you know GPT but at the same time it does seem like you know llms and GPT have really made an explosion of interest in it like how did you think about that so I started saying that Vector databas is really search like used like search engines uh they search for making AI more knowledgeable and they bar use this information retrieval uh tools an information retrieval existed for a long time before rag in fact how rag uses vical datab basis is a little bit like a search engine you say hey give me everything that all the documents are relevant for this question that's a search query in some sense right and that is put into the context of the LM and then the like answering the question becomes easier because you don't only have the question you also have the reference documents from which you can take the answer right and so the answer is no I didn't know rag would be a thing and I didn't know llms would become so good so fast we did know that embeddings are becoming much better we already saw in 2019 when I started pine cone uh language embeddings were getting dramatically better very quickly ganss were already starting to be a lot better um generation uh started happening uh it wasn't great yet but it was it was already surprisingly good compared to what existed before right and so it was clear that something tramatic is happening I didn't know it would happen so fast and so dramatic so this is so explosively uh uh and I think I mean somebody maybe a handful of people that Google an open the eye and others like had known because they were working on it but I think the rest of the market could know do you um do you have opinions on different embedding strategies we have a lot of questions from our users that wanted to ask you about U different things that have happened like it uptake an interest on Co Colbert embeddings or um open AI um new embeddings model that replac the long-standing OT 2 um I don't know what are you watching this and and do you have like recommendations for someone that um was wondering what embeddings they should use so um I do and I don't so let let me unpack that for a second uh there was a flurry of activity from 2017 is to 20 20 something um on creating of these like two Tower embeddings and like language embeddings and so on uh Bert was making the rounds everything had to be with bird like Roberta and I actually try to egg my research team to launch a uh a bird model Bird model called lead bird te nice genius and I F I failed they didn't want to do it uh I mean just for the pun it wasn't it wasn't it wasn't uh motivating enough um uh by and large uh these efforts have died down for two reasons a because the embedding models that are provided by the companies who do that became good enough and B because the efforts associated with training those models from scratch have gotten higher so you your ability to some two sides of the same coin you know those those models have gotten good and you know uh good enough that that you don't actually get uh a ton by un by by uh one uping them in some sense and also one uping them is is in terribly expensive and complicated this right so either people take it as a big mission to retrain those models uh or they should just use what's what's given so the one answer is uh for the vast majority of people don't train their own okay uh which one of uh my many collaborators in the industry should use uh I will make a lot of people angry if I said anything I will say that uh uh people use maybe rather than um rather than you know name a specific embeding I understand why you you don't want to do that but it's but there isn't one answer I mean different tasks require different embeddings but usually it's experimentation it's easy enough for people go and switch out different models to see what works best for the application that in some sense that question is like a hackathon away hackathon level of effort away from you knowing the best answer for your product right because they're all API driven just go just try five and see which one is best for you right uh and lln sorry rag we see is actually a huge equalizer so this is It's Not only because it gives models uh the right context the power of the model itself is less important it's really more about summarization and and creating a narrative around context they already have which most foundational models are already very good at and so it actually equalizes the quality almost uh completely and so now it really becomes a question of speed and cost and maybe you know a few percents worth of of of difference in in accuracy but it sounds like you also see people fine-tuning their own embeddings model that that's kind of interesting is that actually useful it is uh it's not very common because it's not very easy uh but it does happen and people do it uh and in fact we see that it it does help uh the main problem with it is actually operational and Regulatory uh not regulatory but governance related data governance related uh and companies who do that uh either for some reason don't have that problem maybe they use public data or something and they don't have that issue or uh they actually spend a lot of energy resolving it but uh when you find tun model that data should be uh uh conceptually thought as being in the model in the model being sort of like uh assuming the model has access to it right and there is no delete mechanism you can't train a model and then say hey wait a second please forget example 5,712 that you saw 5 hours ago that that function doesn't exist right and so well I didn't see how that's like a huge problem with generating content but is is it as much of a problem with making an embedding that seems like a little bit lower risk doesn't it not really right I mean uh these models are these numeric representations are you know they're opaque to us is as humans because we're not very good at looking at a thousand floating white numbers and and and that making sense to us right uh but that's us we're Limited in that way the models use that for for example for machine translation you can take a sentence in in French encode it and then unencoded in English right clearly those numbers meant the same thing because you just took the numbers without knowing the original French sentence and translated it into English right and so the meaning is is there quote unquote somewhere I mean how how exactly it's there we can talk and and we can discuss what What mechanisms encoded but clear it's not information free in fact the whole point of of using vectal space and vectal search and Vector dat basis is because those representations are highly semantically valuable they they're they're the uh their information reach you know and so you know in fact we think they'll more information Rich than than the original sentence right because you know that might know really not contain the meaning of the words or the synonyms or the context around it and so on does that uh yeah totally I mean if I guess if if I was fine-tuning my own embedding model then every time I add data do I need to recompute all the embeddings of of everything that I want to search over or or is it change exactly so this is this is another problem right if you change the model that generates the embeddings then yes you have to go back and regenerate all the embeddings for all the data that you have so that's another reason why people usually stick with pre-trained models and don't change them at all okay well I have the um most softball question I've ever asked a guest on this uh on this show Bring it on I am really curious you know obviously um you know you guys have really stood out as the um you know the most used Vector database in a really crowded space including like every database company on the planet sort of saying that they're going to add or have added uh Vector database capabilities like you know when you you attribute to to your success um so I'll start by saying that uh we've been at it for a very long time right and and this like surveillance offering that we put out uh is we internally call it V4 because it's the fourth time we have completely rethought how everything works and optimize the crap out of every possible part of of our system so that we can reduce cost and make it easier to operate and roll both of them into customers right so that Relentless you know uh pursuit of scale you higher scale lower cost eal to operate higher scale lower cost eal to operate uh is takes a long time and a ton of effort right and um and we're delighted to do it I mean we're geeks we love it we we live live us to our own devices we will spend a week with the rust compiler and have a blast okay um uh but as companies grow around us right they view victos as an add-on they view it as like a type in their document or in their keyword search engine or in their data lake or in whatever M and that's okay I mean yes if you have embeddings next to some other data great uh fine keep it wherever you want post or whatnot right but if you actually want to use that you need something that's optimized for it that's actually convenient for it and it's plugged into the right tools and and so on right and so uh that's that's where we shine and we have uh teams that come to us all the time and say hey I I thought that using you know database you know blah I won't name any one of them is going to be good enough cuz I tried it on you know a million vectors and it worked fine and I built some POC that my whatever director of engineering was excited about but now we're trying to scale it up and the whole thing just goes belly up right or just becomes super expensive or just latencies go completely Haywire whatever and I'm like yeah what what did you think I it's not what it's a no skial database it's designed for key value and for metadata it's not designed for vectos right it's it's a key value it's what it's a keyword search engine it's designed to search for keywords not for vectors I mean it's not it's a complete different thing so uh boiling all of it down is really on the one hand the fact that this is a completely new architecture it's a completely new kind of tool built from the bottom up to do exactly this and and B is just doing exactly maniacally focusing on this uh in only this infrastructure for now five years uh you know has results I guess when I my understanding of the history of databases which I'd be the first to admit I don't know super well is that yeah they were kind of better close Source uh relational databases and then over time the open source ones kind of you know one out just because like there's so many people working on it like do do you worry with all the enthusiasm around Vector databases that you know there's going to be tons of contributions to open source and they'll kind of get better and better and of course like you know most of the people in the world aren't working um at Pine Cone so you might expect them to sort of overtake you eventually um so we can look at the history first of all we thought about this problem a lot and I'm sure strategically deciding whether you want to be open source or close source is a huge decision it's a company defining decision and in in fact it's almost impossible to reverse once you've gone one path it's really like essentially a one-way door okay we have chosen to go close source for many reasons so just for the history of databases if you go back five not five maybe 10 to 15 years you really didn't have the adoption of cloud services uh where it is today and so the vast majority of comput in the world and pretty much the whole target audience of most databases was on print or literally on like single you know literally compute clouds for large customers or just like server racks in their office right um there you really don't have the only way to reach that audience is open source and so you really want if you want to put your product in in the hands of Engineers that know what they do without involving sales the only way to do that is open source so that became the the the wave of plg this is a product that grows in in the world before Cloud okay basically give your software to Engineers they'll figure out what to do with it come and visit them 6 months later after this is already in massive amount of usage in the company and sell them on your products sell them on some what of services or whatnot right that completely changed with Cloud because now it's actually much easier to consume a service than it is to consume open source if I gave pine cone the code base to somebody they would think I'm nuts like it's tens of services and cicd pipelines and and metering and and and and whatever like it's it's a crazy machine right nobody would even want the open C so like are you nuts you run this for me why why should I run this right uh and as a result the the the manag service ends up being so much easier to consume and because all the AI H companies and all the tech companies are forward leaning and and and Technology Building which is everybody in AI today nobody builds the next Cutting Edge of AI on Prem right they build it all in a cloud right pretty much all of them prefer a managed service and so that for me is is the question what do engineers want and how do we give it to them as fast as possible and make it as easy as possible to consume the best possible level of service right now you're right that there is a community aspect of Open Source that is very appealing and there are other reasons why open source is fun and exciting and there are other reasons why services are fun and exciting I think they are secondary questions what do you mean they're secondary questions they're secondary questions in terms of the quality of the product right uh you know there's some people are open source Fanatics I fanatic sounds bad they're enthusiasts I should say I shouldn't it's not there's no moral judgment it's fine to love I mean I myself contribute to apachi uh projects and I'm very happy with open source I have nothing against it uh um but if my team told me hey we're not going to consume S3 we're going to run the S3 code base my ourselves um I I would tell them you you you've gone completely insane like this is what's what's wrong with you why would you do that right uh I think so that is when I say secondary question I mean what is the right thing for the product and for the customer and in some sense also for the company and for us we understand it's being a managed service and once you do that being open source doesn't actually add a lot of value I'm saying it's whether it's open or not it's a secondary issue because it does have secondary effects on community on marketing on on positioning on whatever like on different channels and and so on again I'm not saying it's the same but it's not it's not significant I don't think definitely not as significant as it was 15 years ago this okay so one question a lot of our our team had um was what have been the kind of optimization challenges and important technical trade-offs that you've made while developing pine cone and maybe I would add how has that you know changed it sounds like the four versions of your product especially um the serverless one that you you just launched so just like technical trade-offs literally like in the weeks like Lally algorithmic or engineering trade-offs yeah I guess like you know it sounds like there's always sort of like a latency versus throughput trade-off I imagine I sort of imagine that there's a maybe a trade-off between you know in databases kind of like the speed of writing and the speed of reading could be wrong there's maybe a trade-off between sort of getting the exact right answer and sort of like the speed of returning something or maybe those are just levers that people could pull you know on your database are there any sort of I guess is that the right mental model to have of your product it is it is so if you I'll just kind of maybe maybe uh list to the tenets of of us like developing how we think about what needs to guide our engineering decisions I'm happy to dive into examples of what I mean by that but but but you're right there there are tradeoffs between accuracy and and speed or latency between uh ease of use and custom customizability between you know so on so we can kind of list all that um we have decided to be very explicit and opinionated on one side of those issues we want to be extremely easy to use right and so that means taking away levers right you don't get to choose the H you don't get to choose the algorithms you don't get to choose your 17 parameters they control how VL search works right now you can say hey don't you want to choose the algorithm for optimizing you know I can squeeze out 5% more accuracy on this or that and the answer is no it's our job to figure that out you need to not worry about it you need to you need to worry about developing your application right it's our job to Monitor and and view that figure out hey something else needs to be optimized we'll go and do that without you being in the loop so EAS of use scale okay scaling is incredibly difficult okay incredibly difficult to be done right and if you don't plan for it whatever if you know you're going to run at very small scale you can make a lot of really easy assumptions right I'm going to always run on one machine I can stick all my data in memory you can you can make a lot of really easy uh assumptions we are always optimized for scale okay we don't when you create an index we don't know and you shouldn't care if you're going to put a million items in there or a billion items in there in fact it might be 10 million today and tomorrow you're going to add you're going to start adding half a million every day and you're going to do that for five years and you shouldn't care right right and so ease of use scale right um um and accuracy okay we understand that people people need the latency and throughput and so on it's very hard to control for accuracy so we wanted to make sure the results are good enough and the accuracy is high enough okay that you shouldn't worry about it and latency is something that is very easy to measure accuracy is actually hard to measure it's hard to measure requires a lot of trust and come coming back to what we said in the beginning doing this hard takes many years to actually perfect it and so on right one of the hardest things to do is actually to earn trust uh and trust is you know built over years and lost in minutes or uh immediately you know we do everything we can to earn trust and make sure that people get good results on a consistent basis and then everything else is in some sense secondary right um uh oh and I forgot one of the most important ones is cost obviously right I mean so this is again you m using a maned service you're trying to optimize for costs right uh and so some customers of far sometimes come to me and say hey I I you know I really wish I could you squeeze the lemon a little bit or I want the ability to tell you hey for this index do something slightly different and it'll be a little bit better uh I said yeah I mean that that'd be nice but would you want to do that like for it to cost like three times more and they say no but by way not three times all because I will charge them or just because whatever uses more memory or something like that changes the the skew of the workload right and 99% of the time they say no so uh we're very explicit on our choices do you I guess does any or or how much of your success would you attribute to sort of fundamental insights in algorithms you know versus just like kind of you know getting a million details right it's a fantastic question um I'll attribute it to three things really but you're right algorithmic insights and acceleration and high performance kind of the core of the engine is incredibly complex and hard to do right okay um but that's just one uh one area right the second thing is really the the the cloud architecture and how you put everything together this is not again talking about open source this is not pine cone is not code it's a live system it's a multi-tenant massive service running okay that is incredibly uh and it's incredibly complex to put it's like you know people say cloud native uh you know throw this around as a term as like I think this cheapens what it means I mean it's incredibly like the cost of networking of every subsystem and the you know everything is is modeled into this so that is again took us took us years to perfect and it's silver lless actually takes it to a whole new level it's really new architecture that I wish I could talk for hours about it it's it's fascinating uh and I think the third which is the most in some sense the most boring but also the most valuable is just operations half of my engineering team spends time on monitoring on billing on on on on call support on on uh viability on on uh on region deployments on you know uh you name it right there's a huge amount of effort in just op operationalizing such a system that that supports thousands of customers in uh live the people on the one hand uh fail to appreciate on the other I love they fail to appreciate it because it means that it just looks easy and that means we're doing our job you know one of the trends that we've really felt in 2024 you know at at weights and biases is this shift towards multimodality you know we I think we we were kind of overwhelmed by language applications for a few years and and now we see all kinds of things right like you know videos um you know images audio uh all kinds of interesting like lar and medical uh yeah does that affect the way you think about your business at all or is that kind of Upstream from you and you just sort of like ingest embeddings I mean it's mostly Upstream for us uh we try to run a an unopinionated engine and in fact we have people use audio and video uh people use uh uses for Pharma for drug discovery on theal molecule structures and protein structure and binding sites and so on um uh on complete documents on parts of documents on different kinds of text in different subdomains with embeddings of size 100 and embeddings of size uh 15,000 right and so on the one hand we're agnostic on the other hand uh those applications end up having different workload demands that we need to meet like what what would change about the the workload so one of the thing is a lot of people build customer facing products with us right and they need latency to be you know 100 Milli 200 mli 300 Milli right if this was an analytical workload for just like drug Discovery say right and you really don't care if your analyst sits there for a second or five seconds then yeah I mean that's a different requirement in the system than somebody like notion who uses us for tens of hundreds of thousands of their own customers for Q&A they're like hey when I when a user asks a question the result come needs to come back and in subse second right we don't we you can't wait 5 Seconds it needs to come back immediately so we have to build a system they can deploy on you know that where it can come in you know their own user could be out of the system for two days and come back and ask one question and their data might be completely dormant and still this thing needs to be in in front of them in less than a second you we might have to deploy hundreds of CPUs to answer that question in a subsec right and and F a lot of data Maybe for object storage and so on so for example that customer fa requirement is incredibly important customers have flut issues and and and volume issues update you know some some data sets are updated all the time if it's customer if it's used to generated content often times if it's not used to generated content you might have the whole data upfront and it never changes and so on so those kinds of workload dependencies are very very domain specific while the v don't change are those places where you can actually um give customers uh a lever to say hey I have a a latency Max that I'm willing to tolerate or do you somehow like architect your system in a way that customer really doesn't have to think about that and they'll just sort of get high latency for any um type of workload so we try to do the best that we can under the uh our pricing uh that this feature that I don't know if you see the the thumb comes yeah I speak with my hands I I keep getting G zoom on can I do that I don't know doesn't work for me yeah I don't know oh no so so we try to minimize latency under the the the cost constraints that we operate on that are very strict right uh in what we see customers are very happy with fresh data so this is data that's that's quered constantly uh uh having uh queries having latencies of like 50 100 milliseconds and something that's like cold something that you know like again for if you're like a rag provider I'll call you whatever if you're a company building some application with rag a vector database for maybe a thousand of your own customers M uh if one of those customers haven't used the system for a day and comes back in right you might be saying okay I'm fine with this like first query taking whatever like two seconds because okay fine it's not a big problem and then after that it should be fast right so that's kind of the model we operating on got it um okay are there there's honestly like the a space is changing a lot right now are there trends that you're really watching because it sort of affects the way you operate um yeah 100% so I mean uh again we we we started pine cone in 200 I started bying 2019 before rag was a thing you know are are the the patterns of use change all the time um one thing that stays constant is people's desire to make machines more intelligent and more knowledgeable uh and uh if you want to make your AI knowledgeable uh today that means using rag in this way and we're already seeing trends let's say you know there's a lot more you can do like reranking like chunking different like you know like you asked about different models the models that now get specialized so the models are specialized in legal or medical domains so on and should we use those or should we not use those uh and so on so there's there's a there's a whole uh list of of improvements and ideas and I think industry trends that will come to happen this year year uh in improving quality and at the same time there are trends that have to do with governance and security and trustworthiness and the ability to reduce hallucinations and the requirements sometimes regulatory requirements to reduce hallucinations and so on we'll see both of those and both of those change our our business significantly because the demands on our system become higher what are the most um successful applications that you see right now built on top of pine con and and what do you sort of expect to see in the next year or two so that it it all comes down to being truly uh for your AI to be truly knowledgeable again we talked about that bit right uh companies use it for legal documents for their own you know notion use it for their own uh uh for their own for their own customers uh data you know for them I mean not for notion their own customers for their own uh company documents gong uses it for uh transcribed sales calls and getting insights out of those and semantic is searching through those um uh I can go on and on you people transcribing and and searching and getting chat Bots to be in guess Wasing like like bucket them in your mind and tell me what the biggest buckets are like obviously like like like I've heard of these but if I just show up as a customer what are like the top two or three things you would expect me to be doing so look I mean rag is is a huge it's probably one of the biggest use cases right now uh but like rag applied to what like like d you seeing like this is where I I think this splinters off and in some sense why this is so interesting I think rag is a catchall syn to get some relevant text and put it in front of a of of an llm right and uh you know I've been saying for a long time that text doesn't exist uh there's no such thing as text because people like because I'm trying to like shock the system a little bit like because people tell me oh there's text and audio and video I'm like what's text text is emails it's slack messages it's Twitter posts it's it's j tickets it's legal documents it's medical transcriptions it's sales calls I mean the content is different what you want to do with it is different the products you built with this are different the insights are different like everything is different right there's the the the the fact that we save them as txt files means absolutely nothing uh to the application Builder right and so uh for me rag is a very general framework in fact I spoke with a an image search company that does image generation like a generative like a it's a found generative it's a generative AI for image generation okay and they're building a model for it and I try to describe to them how retrieval should be in my opinion used in their framework right and I basically described this rack R right they're like oh this it's not rag it's images I'm like what so what I mean if you can retrieve 5,000 relevant images and out of those build an image why is this not rag because it's not text right it's all like oh yeah that makes sense you know and so you know I I think rag is is a very general thing and it's it's it's it's what's happening predominantly now we see semantic search as still being a very big use case uh anomaly detection being you know anomaly detection recommendation other as sort of like taking I think a kind of a passenger seat uh uh for for a little bit is there um any generic advice that you would give to somebody kind of start wants to start a startup um yeah doing um some kind of AI application like is there like a place that you would Point them that seems extra interesting right now um I'm open I'm always uh tempted to give the same advice to Startup uh people who want to start startup which is don't it's almost always a bad idea um but uh I I will say that uh I can tell you what uh what not to do and again this is you know I have a this my opinion okay my two cents on what what not uh there is a there is a a pattern of scene that people think that oh I can use this and this model 2 XYZ and it's going to be easy okay oh I can do this in this whatever like I'll take this and I'll hook it in here I'll call an API call put in front of a customer and then I I'll build the company uh it's wrong in two ways a it's going to be much harder than you think not building the POC building the company Building Company needs takes many years and you need to care deeply about your customers and you about your product and you about your your problem if you don't kill deeply and become a world expert in it the be your product would suck and B if you're so lucky and somehow building that product was easy then you you would also be completely disrupted like five minutes later by somebody else that does exactly the gut same goddamn right and so in some sense you choose to open a company because it's hard because it's something you're so passionate about that you're willing to invest a decade of your life doing even though you know it's going to be like a Monumental amount of effort right if and if if that's the case and and an llm is a part of it go for it that's the right thing to do if you're think doing it because you think it's going to be easy is going to be Al you know the this the likelihood of that succeeding is is is uh very low in my opinion well that's a good segue into my second last question here which is on a personal level what's driving you at this point what what gets you out of bed to keep working out in Pine Cone um the fact that we barely got started man I mean we have stuff on the road map that I have been wanting to do since before we started okay there is so much to be done yet and give me give me some examples let's what's uh um so I'll give you an example we we we loaded all of uh good chunk of the internet all of common call so we basically but scraped chunk and vectorized the internet and put it into a pineco index and use that as rag support uh different llms on fat show questions from dat okay we saw that that reduced hallucination on all models including gp4 by roughly 50% okay and we need to be very careful when I state that and exactly how we measured and so on and there's a lot of information in the blog post that we put online and how we did all of that and so on and so I don't I I I hope people don't you know latch on to this detail all that so all the details are out there okay and sorry what exactly did you do here how how did this work so vectorized uh uh web pages from common craw you know Common craw of course yeah I get that part but then how does you use that database correct so this is the interesting thing right when you now get a query for say a factual question right you can ask yourself okay now I know where that factual question what what where the answer was because well we know what the answer is because we generated the answer from some page from comma so we know what we should answer so now you can give it to an llm and ask that question without rag without context and see how of how accurate the result is or you can add the llm and add R sorry add the vector datab base with Rag and give in some sense the llm access to the internet through veal search right that the difference ends up being reducing hallucinations for uh uh gd4 by roughly 50% which is again I should I shouldn't say hallucinations this is where Details Matter you increase usefulness from 70% to something like 85% or reduce unuseful from 30% to roughly 15% and something for something to be useful it's not enough for it to not hallucinate it also shouldn't say I don't know because I don't know is not the hallucination but it's also not help totally and so you want to see that you get a factual answer right that is that required a lot of work and a lot of innovation on exactly how the the search is done and exactly way you do with the results and so on uh and and in fact the ability to really drive more knowledgeable Ai and the ability that do the understanding that hey wait a second you can actually make uh agents and support Bots and then you know all of the all the rag applications people build with us make them a hell of a lot better by supporting the ecosystem better by by providing more features that is incredibly exciting right so you want to like provide your own database that anyone can use where I could just sort of query the general web or the common craw am I am I understanding what you're doing we're not supporting that uh we might that's not a bad idea uh but we not supporting that all all I'm saying is the unlocked V so the the the power of foundational models next to Vector databases as a knowledge base is very far from being tapped okay yeah see our ability to build both the vector base and the foundational models and the intersection between them the interaction between them right the ability to do that well and to understand all the you know the the tradeoffs we as a society are very far from from understanding and we already know because we already have experiments in the lab that show that there is a huge gap between where the world is at right now and what our customers do right now and what we can already see is possible right so the question is how do we get people from here to here and hopefully in the same period of time our research is going to go from here to to there and we can already chart like two years into the future again okay that's actually a great seg to my last question for you um which is what's the hardest part about getting these rag powered applications into production because I think where we sit at weights and biases we see a ton of people that have kind of amazing prototypes and they're sort of like you know getting ready to do this but I'm sure there's more issues than just you know kind of getting the the vector database right and making sure the latency is low enough lik