Goodbye RAG - Smarter CAG w/ KV Cache Optimization

Key Takeaways

hello Community it is the end of 2024 and we will delete rag out of our systems because we have a new system called CAG let's have a look now you remember T from October 2024 and was a novel rack system and they used Here For the First Time The pre-computing and the storing here of the key value caches for the documents and they told us hey we are 10 times faster than any classical rack system now at December 2024 Just 4 days before Christmas we have now the new publication don't do rag anymore when cash augmented generation C A is all you need for your knowledge task by our research colleagues from type taian so beautiful let's have a look what this is here all about first there's a message that goes here our finding suggest that as long as all documents fit within the extended context length of our llm the traditional rack system can be replaced by systems like o o1 or gerini and you might say unbelievable how they dare to claim this but you know what it is so easy because even Gemini 1.5 Pro the old gerini ey already had a token count of 2 million token contact length so easy and you know the other argument is here yeah but we have a quadratic complexity with the transform architecture for long yeah Al 38 months ago I showed you the solution ring attention for real long context length or if you want this video also eight months ago infin attention here where we already ask here hey is this substituting not a r system so we have no external data anymore can we integrate this and also show you other Alternatives but there's a second message in this beautiful new paper here from our colleag from type and they say with the llms featuring significant extended context windows and maybe you are developed that is already working with a four or a six million token context length this paper proposes you an alternative to rack the cash augmented generation CAG that bypasses you completely the real-term retrieval of our classical rag system and the idea is simple we just preload all the relevant external resources into our model and then we cach its runtime parameters so we have an original internal representation of the data from our self attention mechanism in the decoder structure of a transformer but I know what you're going to say you say hey wait a minute and there are two arguments I can't anticipate you say but hey rack has access to an extreme huge Vector store you know this is of course they are commercial stores and you have to pay them and they is an established industry that we face and this industry makes commercial and it tries to convince you that Vector stores is all you need but as you my viewers noticed in my last video and this was so amazing that I saw that on this video where I showed you here we don't need rag we can have another component ecl here in Ina you showed me here my viewers responded that nobody would put personal Med or personal financial data in a vector store anymore where we have Vector emings of our personal data nobody would do this and there were 10 20 responses by you my viewers said hey this this nobody would do this into rag anymore know and you know this is beautiful because now suddenly what used to be our old classical rag we have now if you want a vector store and we have now the idea that all the private detail personal data go now in a secure database and it is not anymore like it was at the beginning of this year that all everything has to be indexed and everything has to be embedded and everything has to be a vector now that we break with this idea we can move forward so you told me my viewers told me hey you put here the primary knowledge let's say how to solve a new task in a vector store in a vector EMB addding and we build this mathematical spaces we are expert because we can build our own domain Vector um mathematical spaces and then we have here this detailed uh secure database so if you want we have kind of a separation of the knowhow how to perform a new task this is an instruction based methodology from the real top down individual financial data of Mr Green and Mrs blue so you see we have here now a new idea how to handle this and this is great because if we now deny the vors to access to the private personal data because we don't need them embedded in a vector representation this is a huge step forward because you know what's happening now I show you this in my last video when we went beyond rag the Pirates of rag rag is not secure rag introduces challenges such as the retrieval latency potential errors into document selection and increase system complexity and it is not secure and if you can get rid of all of that that would be an interesting step especially at the end of this year now so you might say hey wait a minute is this kind of an aggregated view all is something is this similar to this abstraction that meter introduced when it moved from the large language M with the next token prediction to the large concept model is this concept somehow related reled yeah a little bit but I want to show you here more detail did you understand better that we are working here on a more General approach than met with its concept yeah and the second argument of course is hey but rag is established and a lot of you write to me hey this is an established method you show us new methods new technologies hey and I understand you say hey I see 10 YouTubers 10 20 YouTube channels that are promoting here Vector stores commercial Vector stores and all those YouTubers tell me how great this is in their commercial ad for their commercial Vector stores where the vector stores pay the YouTuber for the commercial ad how beautiful this rag system is and yes I understand it but you know if you were into science and if you have an open mind you might ask hey now what the hell is this CAG in data so let's start here with new methodology that will replace your old rack system it is a new framework that leverages you extended context capabilities of the long context llms remember Gemini over 2 million maybe you work with stronger miles to enable here a retrieval free knowledge integration of external knowledge a retrieval free so in our rag we don't have a retriever anymore we don't have Vector edings anymore we don't need it anymore and you want it comes out of the already what is implemented in Turbo rag turbo rag going one step further it just diminishes here that we need rag the development of turo rag shows you that they already applied here the next Generation here of knowledge integration and it was done here by pre-computing the key value tensor structures and to Cache them and therefore they could address you the computational challenges in the inefficiency that were inherent to the real-time retrieval in the traditional rack systems so if you want from inside of rag it destroys rag itself and you have a new Phoenix arising here From the Ashes of the rack system so if you love rack hey this is the new CAG I hope I motivate you here with this video okay back to science and you say hey wait a minute key value I know key value know this are here in the in the transform architecture if we have your Transformer block or a multihead self attention mechanism here that we have here in detail and we calculate here the tensor structures we have to learned weight parameters and we calculate here with the soft Max here the specific attention for the specific attention heads and then we have our feet forward Network are those key value exactly those that we calculate here yes you are absolutely spoton we're really deep into the self attention mechanism of a transform of the layers in the multi hats and this is what we going to use here for the next step so key value cach in the Transformer model you remember transform as a decode only we have an auto regressive generation powerful mechanisms yes you noce but key values need to be recomputed if you do it classically for each token if we have the auto regressive generation so that for every next token all the key values need to be recomputed for every other token and years ago they found hey if you cach those key values tzo structures so have a key value cach optimization storing the previously computed key values and reusing them for the subsequent token generation thereby reducing here the Redundant computations and improving your inference efficiency significantly so key value cash is an old friend of us but we never used it in the way that we explore in this video so you know classically I would have told you in the Transformer you know why why we do the caching of key value pairs to accelerate the inference of our llms no because if you look now in absolute detail you see at each decoding step the autogenerating model performs your self attention over the entire sequence of our token to generate the next token X at the time T +1 so this process requires here the computation of the keys and the value matrixes for all previously processed token because we aut to regress if we look here at the complete left side for the time T So notably when generating here the token for the time T the llm has already computed the key on the value for the token for the complete interval T minus one so the key value cash optimizes now this process by storing the pre ly computed key value tensor structures matrices for T minus one and reusing them for the followup calculation thereby only requiring the computation of key values for the next token X at a time period T and this of course significantly improves here the computational processing here of our matrix multiplication of our tens operation and if you want to see this in detail this year is the cach key value Matrix that we use here in detail this CAG methodology this is here the if you want GitHub repo it is available to you just days ago it was published 17 hours ago it was updated but nobody is noticing it because nobody is telling you about it because yes I understand if you're a YouTuber and you have a lot of revenues for doing commercial ads here for for rec system and Vector stores I do understand that there is here maybe some conflict of interest for you but you know in this repo you have everything that you need to try it out yourself so maybe you are interested and maybe you give it a try let's have a very short summary so what is this cash augmented generation why we don't have a retriever anymore why we don't build here extensive here Vector spaces extensive new mathematical spaces why can we go here with a cached internal representation of the transform a tensor product itself because it is so much faster it's so much better it is so much more performant so norvel approach enhance the performance of llms and unlike the traditional oldfashioned retrieval augmented generation we preload now all the relevant documents here into the model into the context during a pre-processing phase so while we do here that we build up the vector spaces with the retrieval what we do now is let's say you are company and you have your Handbook of company processes and let's say I don't know 2,000 Pages or whatever 2 million tokens is about 20 books so I think even with two M the classic the mod it's free ger free to use here you already have 2 million context length here in tokens so you know now if you want to upload here all the I don't know quality management of your company whatever you have it is already enough so you pre-process this in a phase you preload this and you LM already ingest here all this external information and stores it in an internal representation for for a future tza calculation in the self attention mechanism itself we don't have to construct a workaround over a vector space over a vector store we directly stay within the operational mathematical operation of the LM itself we do not find other representation we work here with this new self attention from the external data and we just store it then externally to the working directory of the llm and just wait till a new query for this particular domain comes in and then we upload here this pre-processed system so how does it work step one document got all relevant documents all your quem documentation all your process documentation or all you I don't know how does mathematics work how does theoretical physics work whatever you have upload it and this documents are domain specific and cor it here beautiful then you encode it here in this precalculation so the documents are fed into the llm nothing else happen no no training nothing just fit it in you have now the key value cash here from the llm encoder you capture now the tensor structure because they are now the essential self attention information here of the content of the semantic content of all these documents in a form that you can immediately feed into your llm and then you have a cash reset if you'll upload the next thing of data so whatever you store now the key value cache of physics externally or the key value cache of your mathematical s or the key value cash of your operational SN or the key value cach of your quality management you get the idea so whenever a new query comes into the llm the llm knows hey I don't have just to rely on my original parametric knowledge I have encoded it from the pre-training and the fine tuning and the alignment I have external here a pre-calculated key value cache where all the quality management operation for this particular company are detailed I have a complete process layout so I need now to upload this knowledge so I need to upload now this external key value cache into my operational memory and I can immediately answer the query and there's no rag involved at all isn't this nice because in the oldfashioned rag you know what we had to do we had the vector embedding yeah retrieve the relevant documents in real time the quaries high dimensional Vector space and then we had to reduce the vector space and we had to find similar vectors here and the with the query Vector with the encoding of the query vector and we had our coine similarities and then we had the Retriever and then we had a ranker and then we had a ranker to the ranker where the first did 200 elements and the next did only 100 elements and my goodness and the system became complex and complex and you paid more and more for the commercial Solutions and you had you introduced more latency more potential errors and the system was hallucinating like great and now this new CAG says I have advantages because everything is preloaded to your normal system you don't have to find your your system you just up upload it in advance you don't have vector embeddings and you are much more efficient because everything is already waiting in the inference phase and if you have the inference all these pre-computed key value cache memory that is stored which contains now the lm's internal representation of all the relevant external knowledge of the complete Process Management of your company of the complete quality management of your company or whatever is now loaded into the llms working memory given you have a specific query so we have an inference computation where we just upload in the working memory some additional precomputed key value cat already in the internal representation of the llm it is such a beautiful methodology but a lot of ideas how to optimize this further operational advantages reduced latency error mitigation and simplified architecture now I know that the central argument for this new cache augmented generation is that with the increased context Windows capabilities of the llms 2 million plus preloading now the relevant knowledge directly in the M context and cashing get runtime parameters the internal llm runtime parameters offers a more efficient robust solution than any classical rack and now I know what you're going to say because it's exactly what I S hey and if I if I work on an experimental 8 million token context length llm but is the max content length really a limiting factor for this methodology as it turns out it is not and here now I'm trying to to a little teaser to show show you what is happening and I've never seen this in any other YouTube channel I don't know why but look what just in the week of Christmas what is happening here we have a new um publication on our here December 27 just yesterday a survey on large language M acceleration based on key value cash management oh look at this from chingua University in Fudan University they look at a multimatrix factorization attention mechanism where they optimize the under string and key value cash constraint or look at this one here we have from from Christmas December 24 2024 here we have a pre attention key value cash paper for University of Maryland Yale University and so on so you see look at this for the day before Christmas December 23rd 2024 Google deep mine goes here for cash augmentation that has a very specific offer line co-processor that operates here only on the mall's key value cash and this co-processor augments you the cash with a set of letter embeddings for a better decoding and they demonstrate that with a frozen llm they can be augmented with an offline co-processor that operates y on the m key value cach and has further optimization of the performance of the llm especially for reasoning intensive task to think more so this is definitely a Hot Topic look at this this is Purdue University December 18 we already looking here for key value cash tensors with a 4bit quantization so the researchers are all trying to reduce this to bring this here to maybe a consumer GPU system well look at this this is from December 19 we have here an task ER adaptive key value cash compression methodologies for the long context llm I just hope that I show you because you don't see something on YouTube or don't see something on medium or you don't have a Reddit sub whatever it is there this is happening this is what is coming what look at this here December 18 optimizing key value cash compression in Long context generation with a beautiful idea maybe I do a separate video on this those are the papers I currently read and I think they're indicating here the path to the future but if you think this key value caches here just separated here in this week around Christmas with this eight publication no let's have an overview of the methodologies we can optimize this on different levels we have a token level optimization this Focus here on the F control of the key value cache we have a model level optimization on the architecture ual Innovations how we do this with the attention grouping in the non- Transformer design and we have a system level optimization especially if you go here for the resource utilization and here the especially the memory optimization and this is this is real hard topics the advanced memory management how you handle this with this llms that you are not restricted to 2 million token 8 million tokens or whatever imagine a system where we would have an almost unlimited memory management so let's have a look at this start with the token level optimization for the key value cash management and here from this literature from this overview of the survey look this are just papers to optimize the token level optimization of the key value cache the selection the budget allocation the merging of different key value cache if you do chunking of different topics different domains in multiple key value caches if you do get a quantization if you have a low rank decomposition yes Laura you have an idea static everything these are all papers research papers that examine exactly this or we go here key value cash management for the model level look token level optimization model level optimization system level optimization data set and Benchmark how they interrelated or let's have a look if you want a system level optimization for the key value cach management the pure memory management this is real something for specialist for expert the scaling task the hardware a very design for which GPU are you which are the possibilities of the new gpus how you can integrate here the hardware software optimization for this key value cash management for this Advanced reasoning where you do not need a r system anymore the research is simply amazing you go from flash attention to all the other things there topics I don't even know I've never read about this but I think it's amazing what is currently between the years happening now on the key value cash optimization for our llms so therefore this is it I hope this was just a teaser for you to understand maybe with 2024 we all come together and say okay we don't need a r system anymore we further evolve and now we have new technological possibilities we have new ideas of the optimization we don't need to build Vector spaces we don't need to build retriever that go on a vector embedding structure we can have this in the internal tensor representation of the self attention of our particular Transformer architecture if you go with a decoder only or whatever and we have here a pre-computed cash augmented generation of external knowledge with the inclusion of all the possibilities that the key value cash optimization on each and every level provides for our new AI systems in 2025 so I hope there was some new ideas I hope you are a little bit here out of your comfort zone I definitely enjoy doing this video maybe you want to subscribe maybe you want to see my next video because I can tell you this wasn't not the last video we're going to talk about here better reasoning for our llms better performance for our llms and what are the new technology that you can apply hopefully real soon for your particular EI applications would be great to see you in my next video