Vectorless RAG - Local Financial RAG Without Vector Database | Tree-Based Indexing with Ollama

Key Takeaways

is what partnerships did Nvidia announce for AI infrastructure? The question asks about partnerships announced for AI infrastructure. Looking at the note summaries, the most relevant information is found in note 0005 data center. So this is the note and here it says that it announced a strategic partnership with OpenAI to deploy. What you put in your context window is the most important part when you're building a rack system. But can you do it without a vector database or you really need one? Hey everyone, my name is Venoline and in this video we're going to be building a completely vectorless rack or a rack system that doesn't use a vector database. We're going to be using the structure of the document and we're going to be allowing our OM to use that structure in order to create the retrieval step of ARC system. Let's get started. If you want to get access to the complete source code that I'm going to show you in this video, go and subscribe to M Expert Pro. There you're going to find a complete AI engineering academy that starts from setting up your WCO environment and shows you how to run WOC OS. Then it goes through the complete pipeline that you need to build in order to create a classical arc system. And then it goes into aentic systems, how to evaluate them and how to deploy them in production. So if you want to become a better AI engineer, go and subscribe to MX Pro. Thank you. The difference between the classical rack implementation and our vectorless rack is going to be the way that we're going to be indexing the input document and then allow our OM to reason over the resulting structure. In the classical example that I have shown you within the previous videos, we've seen that you need to get an document as an input and usually this document is an a markdown or another format. Then you're going to be chunking this document text into chunks. Then you're going to be using embeddings for each chunks and you're going to be storing that within the vector database. And we have used Postgress SQL with the PG vector extension. Then you are going to be passed in a query from the user. This query you're going to be doing the embedding using the same embedding model. And then you're going to be essentially looking for the most similar chunks that are close to this query. Then you're going to get the K chunks and use that is an input for the OM and provide and generate answer with that. Of course, we have added hybrid search on top of that which allow us to use keywords in order to search the text inside and create a even better retrieval chunks. For the vectorless implementation, this is going to be a bit simpler. We again are going to start with a document which is going to be a markdown text in our case. Then we're going to be creating a three structure from this document that I'm going to show you how we're going to do that. Then we're going to be creating summaries for each leaves of the document. That is the most bottom document chunks and we're going to be creating a JSON tree with that. Then we're going to be passing in a query from the user. And then we're going to be passing also the chunk summaries to our LOM. And the LM itself is going to choose which chunks are going to be used in order to answer the query. Then we're going to be creating the answer using the selected nodes of our tree. Provide that to the M and finally answer the original query. The core idea of this implementation is to use the structure of the document and of course if your document has such structure such as most of the 10Q 10K etc documents that are within the financial space also within the legal space those documents are usually well structured well you can use that structure in order to create such threebased indexes that I'm going to show you here. So what are the pros and cons of this approach? Of course, at first you don't need to have a vector database. So you don't have to have additional dependencies or a database at all. Also, you're going to be able to use the structure of the document. If your documents are structured, you're going to be using this approach in order to parse that structure. Within that structure, you're going to be very well positioned in order to site the sources and use that as a grounding information for your LOM. And you're going to be able to use multiple levels within the tree in order to create the responses that is this decided from the OM itself. Of course, it is a bit more expensive to do the retrieval since you're not using an embedding system, but you're using an OM with an additional call or multiple calls depending on the amount of documents and depending on the amount of context that you really have that you want to work with. It is also costly to index the whole document since you have a single query to the OM for each summarization that you're going to be doing and your bills are going to be go up as well. And this doesn't scale very well if your model is with not a good context window. Of course, you can do summaries of summaries and continue to scale this in some other ways, but at least within this basic implementation, this wouldn't scale well with a model that has for say 30k context window. And the summarization quality is very important. So here you can't use just some 1 billion parameter local model. You probably have to use a model that is a bit better. I would say four or eight or even more billion parameter models that is able to create good summaries in particular for your domain of documents that you're going to be using. When to use each approach? This depends highly on the documents that you're going to be building and the cost of the OM that you're going to be using as well as the cost for managing your vector database. But in simple local rack systems, I would say that vectorless rack when you're building something with a small amount of documents is going to be performing quite well. that is if you have the structure for these documents and you're going to be using a wo that is not going to be a bit on the sw side in order to do the inference for your ARC system. Another important thing here is that the OM can reason about why each of the chunks is being selected based on the reasoning from the tree that you have created. So this is really important in order to understand why the M is giving you this and probably do some prompt tuning on top of that. So you're going to be able to more easily get better results. On the other hand, if you're using embeddings, those are a bit harder to tune. And if you're using even a rear ranker or something like that that supports instructions, again, there it is a bit harder to do the debugging and more importantly do any fixes on top of the original embedding structure that you have. So when you have much more documents in the realm of thousands plus documents, in this case, classical rack is still going to be probably much better since you're going to be able to do vector search much faster compared to the vectorless rack. And in those cases, you also might want to put more document chunks within the rack system. So you can essentially use that as a trade-off versus the vectorless arc approach. Of course, you can do some enhancements as we have done for the hybrid search within our classical rack approach and those enhancements are working quite well in practice. The idea presented here is based on the Raptor paper that was recursive abstractive processing for three organized retrieval. This is the original paper at least as far as I know that is presenting this idea and allowing you to increase and look at it a bit further. Then there is page index which is a popular oss library or open source library that is also now having a commercial product. I haven't used it but I say that a lot of people are pretty happy with it. And then wma index has a three index. So if you want to experiment with it and using already wma index you're going to be able to use that in order to create your three indexes that are going to be very similar to what we have here. So the key takeaway that you would like to take here is that vectors are finding similar chunks while the tree that we're going to be building the tree index that we're going to be building can allow the to reason about the structure of the document and of course there are pros and cons as we have discussed two of them. Now let's try to implement this and see how it works in practice. We're going to be working with one of the latest Nvidia financial statements. Here on the left you can see the original PDF file that we're going to be using. It contains a lot of structured information including these tables, the outlook, the highlights, the data center, etc. Revenue etc etc. Here on the right you can see the extraction from this document using docklink and a VM. You can see that we have the preserved table structures, the outlook and pretty much all of the available structure for this document. And this is going to be the input source that we're going to be using for our vectorless track. This is the vectorless rack implementation that we're going to be working with. And once again, this code is available for MLX Pro subscribers. So if you want to get access, go and subscribe. Here you can see that we have the initialization of our model. We're going to be building this entirely locally. And I'm going to be using geometry, the 4 billion parameter version of this model. Of course, probably using a larger model is going to increase the accuracy and the performance of this type of rack. In this case, I'm going to stick with this smaller model just to show you that it is very effective even within this size. Then we have a three node which essentially is going to be the structure that is going to support our threebased index. Each note is going to have a title, a note ID, a content, which is going to be the text, and a summary that our LM is going to provide for us, as well as children, which is going to be these notes. And we're going to have a method inside here to dict. This is going to essentially convert this note within a dictionary. And we're going to be having this note count, which is going to be our way to create newer note IDs. We have a three search result. This is going to be for the second stage of our implementation. We're going to be using this in order to run the reasoning over the three structure that we have already created. And then you see that we have the three different prompts. First one is going to be to create the summary for a given section. I am asking for a two to three sentence summary here. And if you have a larger and more powerful LM, you can increase this even more. We have the three search prompt. Here we're going to be giving the notes with their ids, title, and summary. And our is going to need to give us the notes that are relevant to the current user query and might contain the answer. And then we have the answer prompt. Answer the question based on the context below. You can see that these prompts are very generic and probably you're going to be able to tune this for your domain. For example, since we're using financial documents, we might try to tune this information for the financial context and we're going to be probably increase the accuracy of the tree search prompt and even on the answer prompt if we try to. The built tree function is going to be responsible for creating our threebased index. And here we're going to be using the markdown header text splitter from lankchain. We're going to be splitting through the three different sections. Of course, if your document has more depth, you can increase this in order to parse it. And then we're going to be using our three node and create for each of the sections within the deepest level. But here we're going to be implementing an iterative solution in order to traverse R3 that is the structure of the document. Note that we're using a stack here and we're going to be starting with the root note. And if you're familiar with lead code, this should probably make you quite happy to see that these types of implementations are still relevant when we're building AI systems. Then we're going to be having this summarized tree. This is going to be using the parts of the note that we're going to be using in order to create the summary for the different chunks. And this is going to create the chunk summaries bottom up. It is going to start with the leaf notes at the bottom. then it is going to go up and use the children summaries in order to create a summary for the current node. And in this way, we're going to be having a summary for each of the nodes of the tree in a bottom up fashion. But let me show you actually how this all works. And this will w pretty much everything. And then we're going to start with the first step which is going to creating the tree structure. And note that we have wed the document. It is a relatively large document as I have shown you and you see that we're creating the note summaries for the different children that we have started with here and you can see that we are in real time creating the summaries and this is the tree structure that we have created by using this information from the document. You can nicely see the actual structure that our model is going to be working with. We have the titles, the summaries for the different subsections. So now that we have this tree structure, let us continue with the tree. This is the actual source of information that we're going to be using in order to pass in the OM the context that it needs in order to search for what is happening under the hood. So this is going to be the root the first child and then this child is going to have 15 children again we have uh one note on a higher level and then a lot of children at least for this particular document and let's see how all of this works by running this we are testing with three different queries what was NVDS total revenue and earnings per share in Q3 here you can see the reasoning that we have for this particular query and the notes that it has retrieved the 2026 fiscal summary and condensed consolidated statements of income. And here you can see the final answer. And let's try to find this information within the fiscal summary. Here you can see within that one you can see that Q3 2026 we had this number here 57006 which is exactly this number that we found here for the revenue which again is correct and then we have a diluted per share price of $13. The next question is what partnerships did Nvidia announce for AI infrastructure? The question asks about partnerships announced for AI infrastructure. Looking at the note summaries, the most relevant information is found in note 00005 data center. So this is the note and here it says that it announced a strategic partnership with OpenAI to deploy at least 10 GW of Nvidia systems. Other partnerships with quotequote Microsoft, Oracle and XAI. Here you can see those names cited here. strategic partnership with open air to deploy at least 10 gigabytes of Nvidia AI systems. The final question is what is the outlook for Q4 revenue and gross margins? Here it says that this is coming from the title of the outlook note that we have created. So it retrieved that this is the note. It's very short and sweet. Revenue is expected to be 65 billion plus or minus 2%. GAP and non-GAAP gross margins are expected to be 74.8 and 75. So you can see here the exact same numbers that we have been retrieved for this document plus or minus 50 basis points. Again this is retrieved from here. So this is it for this video. We've seen how you can build a complete vectorless ARC system without using vector databases or semantic search and do it all with a wo. We've seen that we can use these types of systems on financial documents such as the Nvidia filing that we had. And on this relatively complex document, we've seen that this type of system has proven to be able to answer our queries. Of course, you might test it on your own and see whether or not this is going to cover your use cases. Thank you for watching, guys. Please like, share, and subscribe. Also, join the Discord channel that I'm going to link down into description of this video. And if you want to get access to the complete source code that I have shown you in this video, go and subscribe to Moxer