NVIDIA NIM RAG Optimization: QuietSTAR (Stanford)

Key Takeaways

hello Community wasn't this a beautiful keynote we were told what to expect from the future and we will have our Blackwell 200 our b200 here a foundation of your EI Center of Excellence but more important for COD is that we will have Nemo for customizing our llms and now Nim for deploying this models and for both here Nvidia Enterprises here to Gateway now if you're not familiar with nvidia's Nemo framework simple an open source tool quit you create you train your F tune your EI model should be easier for developer build Advanced EI application and it's done in pytorch the key features are it's modeler you have a collection of pre-trained models supports fine-tuning and and and and in particular you can upload custom data set this is interesting and you know with the new press release now we have it here Nvidia inference microservices isn't this beautiful and if you look at the very last line here you can see microservices for retrieval augmented generation and you say hey wait a second are you sure so let's check microservices for retrieval augmented generation so if this is the focus now of nim by Nvidia we say okay now we we know what they're looking for here and yeah just 7 hours after the keynote I had some ideas I published here in my community Tab and I said yeah Nim is part of the Nvidia enterprise software where a strong use case for Nims will be the support of the rack deployment models so this is Nim for rack but you know what's interesting is is the partnership and if you look here at Snowflake and this is from 2023 and they said hey we want to run here our llms in snowflakes to use them in conjunction with snowflake data and snowflake container services beautiful and now you guess it here now in March 2024 we have no extension of this partnership from Snowflake plus Nim so if you buy the Nvidia Hardware you get the Nvidia software suit the Nvidia Enterprise Edition eii Edition with your Hardware machines Now isn't this beautiful and then you can run all of this within for example Snowflake and as mentioned here you know NE here the framework Cal for fine-tuning pre-existing llm or developing new llms from scratch using here the vast amount of Enterprise data stored in Snowflake so snowflake says hey you can stay stay within our boundaries protected by our cyber security you do not have to reach out somewhere else here to do the computation we in our data centers we have Nvidia we have here the Blackwell 200 we have the Super Chips and we have here the Nvidia inference microservices to fine tune everything and you can do whatever you want you don't have to leave snowflake for example so if you're a company that can afford to buy the Nvidia product you clearly see here the value proposition from Nvidia to all the companies in the world for example here you stay within the secure environment of snowflakes data Cloud now since this is not a sponsored video please exchange snowflake with whatever data provider you like but this is just about the idea what is happening hardware and software become standardized and if you buy this you can integrate it here within your Enterprise so Nvidia EI Enterprise is the Gateway for every developer for every coder for everybody who wants here to customize large language model with their company data and then deploy this large language models here with microservices from Nvidia Nim so you see a beautiful new world everything is standardized everything is available from Nvidia from the Nvidia Hardware machine the Nvidia software packages and the Nvidia technology to make it run on a node or multiple node of Nvidia black worlds Now isn't this beautiful yeah and maybe there were some independent coder some independent developer but let's talk about the beauty first now of course you might say if you want to be a professional Maybe you go to this Nvidia developer page where you have the tutorials you learn about everything that's going to happen and maybe this standardized software this standardized Hardware that goes together with Nvidia is the way forward if you want to be an Enterprise AI engineer coder developer whatever you want to be so you better get up to terms with Nvidia and just to show you this even yes yesterday March 18 we have here from hugging phas now you can easily train your mod your models that you have on hugging phas with the h100 gpus on Nvidia cloud services Now isn't this fantastic so even now here hugging phase gives you if you have the Enterprise hugging phase Edition then you have your access and you can pay here for Nvidia cloud and the h100 GPU St directly from hugging face so I think you see the signs of the time and I just would like to recommend this article here March 18 hugging face easily train your mods with h100 gpus on the Nvidia cloud so you have your cooporation Nvidia with hugging face so the GPU poor like I am they are no more according Hing face beautiful so Enterprise Hub organization can give their team in instant access so you have to have an Enterprise account with here our beautiful hugging face and then you're allowed to pay and you get access to here our cloud services let's have a look how it works you have a hugging face Enterprise Edition beautiful and you pay for this and you go to hugging face and you have your llama 7B and here you see you have your train your Auto Train your Amazon Sage maker or your Nvidia cloud and you go of course you to the Nvidia cloud you select your organization with your organization account they go of course here being from hugging face with their hugging face account they click here on create new space that gives you exactly what you have to do and once you are in the Auto Train space you can create your training job here configure your Hardware your base model your task all your training parameters you can do everything here according here to the presets that are allowed that you are using here and then you have here for example your base base model mral 7B beautiful your started training you can have one or eight h100 gpus and it just depends here and you can start the training by clicking start training now this is amazing now and you can monitor the training by opening here the Lo box off the space and you see exactly where you are pricing is I would say yeah a little bit expensive but if you have an Enterprise Hub organization account and you have one Nvidia h100 with 80 gb you pay about 8 and a half bucks per hour okay and the hugging face says we are just getting started collaborate with Nvidia so you see everything is moving to Nvidia cloud and this is it the Nvidia machines are there and if you are a developer maybe you should get familiar with the Nvidia ecosystem because maybe a lot of companies are gone use those standardized approaches of a combination of hardware and software for more information why not go directly to hugging face and check I'm sure in the next days they will have some further information for you so let's come back and says now Nim is part of Nvidia Enterprise a software site beautiful and they are in support for the rag deployment model so let's talk about Nim Rag and Everything is Beautiful Everything is standardized we have envidia and it's a beautiful day however and you might ask yes Professor what are you saying what is on your mind and I might say you know there was a tiny little bit of information because on Tuesday's Nvidia Q&A session Jensen was asked what you do about AI hallucination you know this strange little scientific thing here the tendency for AIS for llms for re system to make up answer that sound plausible but on based on fact of pure hallucination pure nonsense wrong factual data and Jensen appear to be visibly frustrated by this question how could you dare to ask here this in an official Q session but you know what thanks to take crunch here we got this from Johnson he said you know what add a rule for every single answer you have to look up the answer examine the source and the content compare the facts containing in the source to known truths and you know hey this is easy for us humans because known truths I have my known truths if you watch television in the US you know you have CNN and this is your known truth and if you are in the US and you watch Fox Entertainment you know this is the known truth so you see we as humans we have no problem of at all to have known truth but this poor little AI system here does it know what are our known truths our individual known truths and J G if the answer is factually inaccurate even partially discard the whole source and move on to the next one don't answer this question and now comes something from Jensen that I love the Y so our llm our re system our microservices shouldn't just answer and say but this is the topic of a large language well no I have a query and I get an answer no now today we learn Jensen tell us thei should do research first to do determine which of the answers are the best I say okay Chan has to know it he has millions of GPU so he's the person to go to so AI should do research first and then we have a lot of answers a set of answer maybe infinite amount of answers and then the research will tell us which answer is the best one and you know somewhere in my brain there's this little voice and says are you sure the llm should do research first I mean research haven't we not trained the LM and the r system and everything from a vision language Transformer system that it learns something so it has the knowledge and it can apply the knowledge and now we tell hey we don't trust the system the system should do research first and this from Johnson so let's do this and you say hey no problem at all you know February 2024 we have the corrective retrial augmented generation and I showed you this here in my video simple ideas to improve Rea from Stanford and Google and I went through this CRA and how you can apply it and what are my insights into CRA but of course you have seen here my video from four months ago about selfrag a self-reflective almost a r retrieval augmented generation that constantly checks its own processes you know but self R yeah it's four months old so is it doing research or is it just controlling supervising argumenting itself and then comes something beautiful that I really love and Jensen says Hey for Mission critical answers so if it really depends on this such as health advice so if you're bi technology bio engineering or just in medicine there Nvidia CEO suggested perhaps checking multiple resources and known sources of Truth is the way forward so ladies and gentlemen there you have it this is the way forward no question asked so you check multiple resources CNN plus Fox and known sources of Truth uh this would be um this would be yeah okay and known sources of Truth don't over think this and this is the way forward perhaps so here's our solution congratulations and you know this little voice here way back here in my mind says where is Stanford if you needs them and you know what Stanford is here Stanford University on the very same day March 18 2024 published here desolution and you're missed I knew you were Amed because it's a quiet star this is a new methodology that language model our llms can teach themselves to think before speaking and you would say hey disqualifies here as doing research no this is of course something here happening at the exact same day to our GTC 2024 so here we have it and you say I hope it is really research you say you know I hope this is really something complicated for research for Science and not just some simple algorithm and if you look at the algorithm you say okay yeah this looks yeah it's yeah the hidden States the mixing yeah the log to reinforce and re yeah this looks here as a Nao operation here in a mathematical space there okay let's say this is research can somebody explain this to me because if I look at this I have no idea what this is all about but hey it's stand for it so come on let's chill and relax and you know since it's a quiet star we have to look back in history to the original Star the self-thought Reasoner bootstrapping reasoning with reasoning and sometimes you feel a little bit strange when you do EI videos on YouTube but never mind here we go so May 20th 2022 we had here beautiful Department of computer science Stanford University and Google research hey what a coincidence and here and you have a simplified star the original Star algorithm and you look at this and say hey now I immediately understand what they are talking about but if you are not that advanced I have here some slides for you because I had to go the long path so star simply improving the ability of llms to perform complex reason in task and this is achieved by generating stepbystep rationals and you say we know this of course it's from 2022 so llms when trained to generate explicit rationals or maybe you notice under the name chain of sord processes before arriving at a final answer exhibit better performance and a better generalization ability especially in complex reasoning task and we know this the nice thing is that there's a bootstrapping so it usces a novel bootstrapping mechanism that iterally improves the rational Generation by learning from a small initial set of examples where you have exactly the data this is the task and here I show you how you can argue what is here the stepbystep guidance and from this small initial set of examples the system the LM improves and every run gets better every time it ex executes this EXA ly task so it goes over the limitations of learning only from correctly solved problems from in context learning now it is able to learn itself includes a backward reasoning approach enriching the training data set under overal model performance and it is something like a little bit of a self-improving LM and this self-improving mechanism opens new Pathways for developing more auton ious and efficient system isn't this beautiful so what we have a small set of example with given rational in a given Topic in a given structure and this will be the patient zero to teach the model to self-generate high quality rationals going through this process multiple times beautiful you might ask one example here simple example question what can be used to carry a small dog and the choices are given here a pool swimming pool a basket a dog show a backyard and the own home and now you want the model to be transparent and says hey show me your rational before you answer and the Ral of our llm is to carry a small dog the object must be portable and designed to hold s securely a swimming pool a dog show a backyard and one's own home are not designed for transporting objects among the object give him a basket is specifically designed to hold and carry items including potentially a small little dog so baskets are portable and come into various sizes making them suitable for carrying a small dog comfortably then given this rational in this argumentation the system answers therefore the answer is B in the basket this is what we mean with rational and if the system learns more and more complex rationals this is the Way Forward it is important to have this because it generates insight into the decision making we can see here with the debugging clearly the errors that are happening in the model reasoning structure we have more trust if we understand how Ani system builds up its rational for the decision and if it is a self- learning system it can better generalize for more advanced reasoning abilities that might come up in the future and you say hey beautiful so star why is star not everywhere let's go back to quiet star and let's see how Stanford University improved on Star in two years let's look at this can you see it yes you know what I mean now what they did is massive paralyzation so I have a token wise parallel sampling algorithm using learnable tokens indicating the start and end of a sort of a rational and an extended teacher forcing technique and you might say what is this let me show you in a second but before tell me the performance of this system you might have read here in some publication that quiet star is a Improvement of 100% And it is true but no the data so deos found here Ser shot Improvement on a particular Benchmark set from 5.9% and it increases with quiet start to 10.9% and for some common sense question and answer data set it moved up from 36% to 47% now I don't know about you but if I have a mathematical Benchmark with a rate of 10 or 11% or a common sense Q&A with less than 50% I would be a little bit careful to say that this is an EI breakthrough in particular they have some interesting weight to structure this so at first understand that they generate rationals after every token and I really mean after every single token in the input sequence they generate a rational to EXP play here the future text this is what they call the thinking process then they mix here the future text prediction with and without the rationals this is what they call here the talking and then learning to generate better rationals using here the reinforce mechanism that learn phase and if you look for the reinforce mechanism you find it here in 1992 and yes it was a reinforcement learning for a statistical gradient folling algorithm this is here the the real the original idea here from 1992 beautiful how does quiet star work and it is no joke it is really taking every input token and generating a sword and you see here the LM generates a sword and LM generates a sword for each and every think think talk learn but let me give you a better explanation of the three phases those three main steps you have at first a parallel rational generation so in parallel across n tokens XI in an input sequence x from 0 to n we generate R rationals of a specific length T resulting now n * r candidates then we insert some particular tokens the start of ass sort token and the end of a sort token to Mark here where it ends and starts and then from the hidden State output after each rational for each token we have a mixing hat this is just a shallow multi-layer perceptron producing a weight this weight determines how much of the post rational next token predicted logits should be incorporated compared to the base llm predicted logits easy and then you have to sech step you optimize here rational generation parameters think about H parameters to increase the likelihood of rational that make future text more probable so this is an interesting aspect it just does not look to the next token but it goes beyond the next token it looks more into the future to generate here rational generating parameters with an extended reach if you want and then the use here this reinforced learning this reinforced to provide here a learning signal to rational based on their impact on future token prediction this is a nice way so for green grasshoppers there's three phases here in a more simplified version so coming back we can state that Stanford generalized the classical star methodology to learn now reasoning self reasoning from some diverse unstructured Text data so you do not have to provide learning examples you just give it I don't know the content of the internet there you have it and then you decide how to start reasoning having un structured factual data this is an interesting way so as you remember the first step was parallel generation here and this is efficiently generate rationals at each token position in the input sequence and they do this with a little bit of an optimation so they allow a highly parallel Generation by first absorbing that an inference SP of an llm produces a probability distribution over the next tokens for all input tokens and this allows them to sample one next token from each token in the input and they cachier the forward pause and concatenate a diagonal attention mask to the previous attention mask so each generat token now attends to all of the tokens that were used to generate it as well as add to itself so have a look at the original literature this is a nice methodology doing this with attention masking for the future tokens but let's have a look at the results what is the what is the end result of after applying all of this beautiful extreme complicated research that an LM should do given chance and advice now because this argumentation or chain of swords at generated by simpler chain of Swords one might refer to this as a net of Swords so if you see net of Swords you know chain of Swords generate more complicated chain of swords and they have a Rous effect on the other chain of swords and my goodness so they compare here five solution from the model with a fixed random question from this particular Benchmark and the llms that they use is a simple mistal 7B and yes you guessed it Stanford visibly cannot afford to use a more expensive model so we go with a 7B for causal reasoning and self trining my goodness and now you understand why we go from 5% to 10% % but anyway the methodology is interesting and the output of The Missle F tune for the same number and steps on the open web map data and beautiful beautiful so here we have it now the task is easy we have Janet's ducks lay 16 eggs per day she eats three eggs for breakfast every morning and bakes muffins for her friends every day with four EXs she sells the remainder at the former Market daily for two bucks per fresh duck egg how much in dollars does she make every day at the farmer's market and we apply now all what I just told you a complexity of a system that has been pre-trained and fine tuned and quiet star trained here on mistro this C star and this here is the first response so out of the box we got from a Mist 7B and you know I like a Mist 7B okay 16 x per day three x are eaten for breakfast four are taken for muffin so 16 - 3 - 4 9 nine of the eggs is sold at the former Market at two bucks so 18 bucks right out of the box and then you go to the fifth response we have the duck lays 16 a per day this is the total number of Acts then we have an operation 3 + 4 + 16 = 23 and it's a little bit surprising because if you think about this what is this and then cor statement 3x are eaten yeah 4X for muffin and 16 eggs are sold at the farmer market for two bucks each every day no this is wrong because the duck lay 16 eggs per day and three are eaten and four eggs are gone for the muffins so this is nonsense and why we have to add here the number this is nonsense too and then we have a statement the 16 X that are sold at the farmer's market are the remaining a but that's nonsense because the duck lays 16 x per day this is wrong this is wrong this is wrong but then somehow the system it's coming back and says hey maybe I do here another mathematical operation so 16 - 3 - 4 is 9 9 * 2 bucks so 18 bucks so you see seeing here now the way that this l am on this rack system here the G and the rack thinks about deducting here causal argumentation I know I would not trust here the fifth response of the system because 16 a are sold at the farmer market for two bucks every day is simply wrong and this is also wrong but luckily sometimes it jumps through the correct path but this should not be an ano answer so like Chon told us choose from the multiple set of answer the right answer given that you know the truth my goodness what a genius so you might say hey is overthinking that the system learns to get more and more complicated it caal reasoning for llm or for the generating interact system or is it just because the mistal is in itself not the right llm for doing causal argumentation and I think it's kind of both 7B is current state of Technology not the right tool not the right llm to do this complex self teing selflearning causal argumentation paway and the over thinking here is just wrong just nonsense yeah and then you know my good old chat gp4 turbo super beautiful and I just put in here the query here in gbd4 and it comes up here with equation tells me everything tb4 just starts here the python environment calculates here for this equation here with the number here the correct result and the result is Chan makes 18 bucks every day at the farm market by selling the remainder of her duck XS beautiful so here you have it again from Nvidia CEO the person who knows everything for Mission critical ARs here of your G of your rack system your treal augmented generation or if you just use an llm as an agent or you have multi llm agents working and lying to each other if you Mission critical perhaps check multiple resource and the known sources of Truth and this is the way forward to do Nim and Nim racks ladies and gentlemen as always it was amazing I hope to see you in my next video and in my next video can I show you that we can't forget supervised fine cuing we have something better to train our llms