STOP using RAG for Everything, Do this (ft. Tiger Data)

Key Takeaways

stop using rag for everything. Most of the time when people say they need rag for their AI agents, they don't need rag. Actually, all they need is highly scalable semantic search and implementing a highly scalable semantic search is generally very difficult. It's a tough engineering problem until now. Tiger data has launched agentic posgress on tiger cloud. This is their fully managed posgress cloud platform and they are offering a free tier. Thanks to tiger data for sponsoring this video. So in the next couple of minutes I'm going to teach you how to implement a hybrid search which is a combination of a keyword search using BM25 and also a semantic search where we going to use TF. You can always replace this with your own embering like open a embering. Before we move forward important things that you need to know about a tiger data agentic posgress transforms vanilla posgress into a database that is required for the AI agent era. It has got four new primitives. One is focable infrastructure, interfaced control agents, integrated search and persistent memory. First of all, all you have to do is link it in the YouTube description. You can just straight away click the link and then get started. Once you do that, like you have to sign up with your Google. Once you do that, once you log in, you're going to see something like this. You're going to see all your projects. In your case, probably you wouldn't have any projects. You might have the database name, but you wouldn't have anything else. The first step that you have to do is you have to click new service. Once you click new service, you can see what are the things that you're going to do with the new service. The most important thing you have to understand is you're going to get 750 MB per month for free and it is going to be hosted in the US. So you don't have to be worry about your data security. So this is a posgress SQL with time scale DB and there are some vector extensions that are available. So I'm going to click create a service. Once you click create a service, you have two options. One to download the text credentials. The second one is to download the environment file. So if you're going to use it uh for any cloud-based project, I would strongly encourage you to download environment file so that env file so that you don't leak your API keys. But in our case, in this case, I want to do everything within the web service, their own service. So I can just straight away simply click skip this step and then go to the next step. Once I click skip this step, then finally you have the option to connect it with any AI tool. For example, if you're using cursor claude, you want MCP support, you can just simply use this and then do it. You can see my service is currently ready. That means I can go ahead and then see here. So if I click services, you can see I've got two services. Both the services are currently running. In one of the service, I've already created all this information. So you can see DB3085 875. I've got uh the table, I've got the data, everything. Let me quickly show you a demo before we actually go through the code. So let's say I've this is a solution for pharmaceuticals or an e-commerce website that sells pharmaceuticals. Pharmaceuticals is very critical because you're not just going to search by the tablet's name or the medicine's name. You are also going to have the value um you know the dosage of it and sometimes you like customers might come and search just based on the symptoms that they have got. So all these things have to be kept in mind when you're designing a highly scalable search solution and it could be like feeding directly into an AA agent. So first thing I'm going to do is I'm going to just go here simply search for ibuprofen 200. I'm going to run the code in front of you and you can see I've just ran the code and less than a second we have got ibuprofen 200 as the top item. So we have got a hybrid score. I'll come back to the score later but you can see I proof 200 is the top one. Then I proof 400 we have got. So I'm going to go back and then search something else. So I'm going to search for Adill 200. So now Advil is also Irroofen. Okay, it's just like the brand name. And when I search for Adill 200, you can still see that I've got Ibuprofen 200. So that indicates that we have successfully implemented both the keyword search which means like the 200 comes up at the start and also when I search for Adill 200 I still get a booofen in fact I can go here and then say the powerful painkiller and once I search for this most likely I should get yeah I've I've gotten the powerful IU proof in our database which is ibuprofen 800. So how did we arrive at this particular place? So in your case like when you create a new service all you have to do is go here and click the service that you want and after you click the service that you want you can just click the SQL editor and get started. So the SQL editor I have is exactly the same. I'll share the entire SQL in the YouTube description so you can get started if you want to get started with my example that I've provided here. First thing that we have to make sure is we have to enable the extension pg search pg tick search then we have to enable the extension vector which is for PG vector. See again u this is very important for you to understand because Postgress has been like there for a very very long time and one of the most beloved RDBM systems people have been using and what we are trying to do here is we are trying to use the agentic posgress or like the newest version of Postgress that can help us build AI agents on top of it. So now we have got the both PG tech search and vector enabled and the next thing that we have to do is we just like sanity we are going to drop the table if it doesn't exist. Now we're going to add some sample data. So I'm going to create very simple table schema here. I've got like bunch of things. Most importantly I've got an ID which is highly irrelevant at this particular point. But you can see that I've got generic names, brand names, form strength and then you know adverse effects um um like other informations including an embedding. This is where if you're using open a embedding a Gemini embedding you will insert your embedding here which is like a 384 dimension embedding. In this tutorial, I'm going to show you TF based embedding which is just to make it easier for you to understand. The next thing is we're going to insert sample data for different tablets like we have got midformin uh which is like a diabetes uh tablet. We have got ibuprofen. So pain, fever, OTC um it's it's an OTC tablet. So we have got all the other tablets available here. So we are just inserting these rows in the table that we have created. Once we have that, we're going to normalize the data in such a way that it is easier for us to use with BM25 search. And honestly, like I said, like BM25 search is one of the most preferred searches these days. And you can literally implement it for any client that you want. So the next thing is we're going to create TF based embeddings. TF stands for term frequency. IDF stands for inverse document frequency. It's one of the most popular technique in NLP natural language processing. So we're going to create a TF based 384 dimensional vectors here. So we have created that once like this is this is the part where we have created that once we have created that we have literally added that as a vector within the embedding field. So we have done all those things. So in fact I can probably show you that. So I can say select a star from drug strength um documents drug strength documents. I'm going to run this run the selection. So you can see here we've got ID, we have got the generic name, we've got the brand name, we have got the form tablet or whatever it is, we've got strength and we have got a bunch of other information along with the embedding that we have created. So we have our initial data ready. Probably in your company you don't have to do do this. You already have data. All you're going to do is implement an optimize search at scale. So once we have this, we are going to create two indices. We're going to create the first index which is a BM25 index. And one important thing here is we are going to keep the text configuration simple so that the numerals like 500 are kept plainly as tokens. Second thing is is we're going to create an index. So we're going to create the index drug strength_x and index based on the embedding that we have created. So we have got two index. One is for uh the keyword search. The second one is for semantic search. So now I'm going to just simply show you an example about just to show you that the keyword search actually works. So we've got metformin 500 mg GI diarrhea. I'm going to run this. When I run this, you're going to see that the metformin 500 exactly as we expected it worked. So the next thing is what we're going to do is uh we know at this point the BM25 search has successfully worked. So the next thing is we're going to create a hybrid search. It's a combination of BM25 and the vector similarity search. And the method that we're going to use is called reciprocal rank fusion RRF. Again, one of the most popular methods currently to combine two different methods. People also use it in rag. But in this case, even with semantic search, this is like a very powerful method to combine two different rankings. One is the ranking from vector search. The other one is the ranking from keyword based search. So we have got BM25 with semantic similarity search. After we have all these things um very simple code that combines two one important factor here is that you would see like there is a weightage element here. So the weightage element is what you have to go change whatever that you want to do. But again like these are like few things that you can always optimize for. Uh I'm going to run this. When I run this uh you can see it probably giving me yeah so you can see it has given me ibuprofen 800 which is um which is the most powerful painkiller here. So I'm going to say powerful metformin. Okay, let's see if it gives us metformin thousand. Yeah, it gives us metformin,000. Now this entire thing is running on agentic posgress. This is like a database that has been built by tiger data for the agentic world currently running on the free tier of tiger cloud which you can just access for free. Thanks to tiger data for the partnership. I hope you learned something that is very critical in production. You can do a lot of consultation simply using this code and then make money. But if you're in an organization where you're implementing search, I think this could be really helpful. Let me know in the comment section what do you feel about it. Just try agentic posgress free on tigercloud today. Link is in the description. You don't need any credit card to get started. See you another video. Happy prompting.