Self Learning AI: Accelerate w/ new RL

Key Takeaways

Oh, community. So great that you are back. We have a new reinforcement learning algorithm for artificial intelligence. Doesn't it look beautiful? Come on, you're going to love it. Welcome to my channel, Discovery. We have a look at the latest AI research paper. Now, you know there is a secret of our current reasoning research in AI. Everybody can make an AI model improve for a few steps, but keeping this beast stable for a long training without a ground truth data manifold is incredible hard. Now, you know, the classical reinforcement learning is reinforcement learning with verifiable rewards. And the holy grail of course is selfsupervised reinforcement learning. No humans at all inside the model generates its own questions. It answers the own question. it grade itself and it has a selfsupervised learning experience. But until now this approach has been fundamentally unstable. And today today we look at a new method that finally stabilizes here the self-improvement loop. And you might say, "Oh, this will be heavy on the mathematical side." No, nothing at all. It will be so easy you're going to laugh about it. Now you know SRT selfrefined training here works on a kind of a little bit of a dangerous premise. No, the Maldi model acts both as the student and its own teacher. So you do not have here a huge I don't know proprietary GPD6 system here as the teacher and a little local student. No, it is the same model. Now you know we have a interesting phenomenon. No, we have a policy collapse in those models. And you say, "Yes, of course." No, it's trying to be the student and the teacher at the same time. What do you expect? Well, if we have a little closer look at SOT, the self-refined training, you see, the more steps we have in a learning process, the accuracy reward sometimes just drops significantly and the validation accuracy goes yeah down to zero. As you can see from different models and we have here a QN3 model, a four B 4 billion base model. And yeah, now why is it happening? Because this I model learns to game its own reward policy. It generates high confidence garbage. The validation accuracy. This means the real performance of the system is absolutely now decoupled from the training reward. What the models think is good. So how do we treat these policy collapse? No. Welcome to this brand new study here of today. MGRPO stabilizing self-supervised reinforcement learning for large language model with a momentum anchor policy optimization plus an additional filtering otherwise it would be a little bit boring no and you might say hey I know something no with the momentum there was something with a GPO optimization yes but now we have the next step because as I told you there is something with filtering so let's have a look the orers here beautifully are from Shanghai innovation institute they call of future information technology in Fudan, the Shanghai AI laboratory and the Chinese University of Hong Kong. Beautiful. Published December 15, 2025. Now they look at this and they can replicate exactly that we have these collapse. Now this is the primary crush graph here where you see everything falls apart. This SRT is just a catastrophe. It doesn't selflearn. Yeah. Now I know if you just look at the gpo part you know this is the typical trick that or advice that you get just increase the sample size. So we say okay let's do this no so if we generate now more rollouts g should we get a better gradient estimate no everybody says yes of course that's the way to do it now just a little flashback you remember this no this was the main paper here from case man can lodge reasoning models self train this was here in this version the latest uh October 825 and they handle exactly here self-train meaning paradigms and the idea is so simple. Normally with reinforcement learning with a bioherbal reward, you have here a prompt. Then you have here your little LLM. It gives a response. You just have a verification with the ground proof and depending on if it is correct or incorrect, you give them a specific reward. Now the self-rewarded training says, okay, I understand my little LLM is just a a pure chaos machine. It is a probability engine. It's a machine. So, you know what? I think that if it is, if we're working with statistic, let's generate here three answers, 50 answers, 500 answers, multiple responses from the identical LRM. And then since we have 500 different answers with the statistical machine, then we do a majority voting by the same machine that generated this answer. And you might think at this moment, hm, there's something wrong here. No, this sounds not really in a clear mathematical process, but hey, this is SRT. And then you have a verification with an estimated ground truth and then you give them a reward. So you see SRT compared to RLVR. This is the difference. No, this is here our rollouts and here we have three rollouts. We can have 10 rollouts or 128. Let's have a look at this. Now the orders of today's papers if you want tested here the scaling rollouts from 16 to 128 and the result is the higher rollouts increase here the peak score slightly but they do not prevent the complete crash of the model they only delay it partially so our conclusion we are facing here still a structural instability in SRT in self-arning AI not just a high variance so what We need we need yes you know it we need a better algorithmic approach not just more compute of the same we have to have a new idea let me give you the data you say okay so we have here the MAT benchmark and here the reasoning benchmark and you go at whatever you like let's start at the beginning so original if you go with a Q13 4 billion base model we have here an accuracy of 61.5% and original model for meta 500. And then you see if we have our SRT and we go now with 16 rollouts. Wow. We jump to 75%. If we go here to a different temperature here T is our temperature. You see we do have here some variation. But let's go now to 32 rollouts. So we jump from 75% down to 74%. Interesting. Look at 64 rule outs. We reach our peak with 79.2%. At a temperature of 1.1. Interesting. Now, yeah, you can have a look at this and you see it has quite some cotic system although 64 seems here for the Q13 4bit model here quite here as the peak. So, we manually go now and we stop the system here after 64 rollouts. to say hey we have empirical data we have a feeling that this is here the peak performance of the system so let's go with this now for a different benchmark you see this would be incorrect because here at 16 we have a much better performance here for a different task of benchmark so you see it is pure chaos it is cherrypicked it is handpicked I'm spending hours just to find here the peak performance of the systems so what it proves throwing more compute just scaling up and say yeah go higher and higher does not solve the stability problem at all scaling does not work but you know now if you watch my channel we have a second failure mode here in self-arning of course we have an entropy collapse for those who are new to my channel yes of course I always give you here the most important paper and would say this is it here end of May 2025 here the entropy mechanism of reinforcement learning for reasoning large language models Beautiful. You have everything GitHub and everything. Shanghai AI laboratory, Chingua, Chingua University, Piging University, Ning University and beautiful. You remember when we talked about this, we said does reinforcement learning for LLM just trade the entropy for the performance? You remember how we looked at this and we said hm interesting. So over 95% of the entropy drop gains take place at the very early stage of reinforcement learning and then we have a plateau. Now this is now a second problem we have to trade wave and you remember we already were talking about your samples. here beautiful and the optimization you know this and if you're not familiar with the mathematical notation I have a particular video here there also I show you here the token entropy we took we look here at high entropy tokens here for different things and I showed you here even a duro reinforcement learning mechanisms so accompanying now the performance crash of our self-arning AI now we also have to deal with a collapse in the policy entropy is okay now it gets It's interesting. But what is this policy entropy collapse? How can we get a feeling for it? No. It is simply the AI system becomes arrogant. No, it converges rapidly into a single mode of reasoning and stays there. It effectively stops exploring other solution. It found one solution and says that's it. That's that's done for today. I'm not going to look anymore around. No. So the model becomes now overconfident in an suboptimal solution manifold. Once the entropy hits the floor, you see it really went down to almost 5%. The model now loses its own ability to self-correct, to selfexplore because it stops generating more diverse candidates to learn from. This is what we called here the collapse of policy entropy in self-arning artificial intelligence system. And yeah, if you want to see this for different B 4B base models here with different G at different temperatures, here you have it. But I think it's clear now. Everything just drops here. Beautiful. So what is the goal? So I would say okay now we understand what is all not working. Now we have to come up with a solution. So now we need a new way to force them. Force them well force our little eye to motivate our little eye to stay curious being the high entropy while still optimizing here for the perfect accuracy. I told you you're going to laugh because this is all that they built here with this new GRPO optimization and you say what we just have a new momentum policy model that is now combined and this is it. Yes, this is it. It is sometimes just a simple idea and it even gets even a little bit more simpler if we have a closer look. So what do we call it? We call it the momentum anchored group relative policy optimization. And we have two stabilizing forces. Now in addition to the classical GRPO, we have now an additional momentum model. We have a strategy pi and theta k. So instead of calculating the truth, the real ground truth based of only the current which means unstable AI model, we now maintain in parallel a teacher model. And this teacher AI is simply an exponential moving average of the student AI. It changes slowly. This is an old grandfather AI walking 10 steps behind the student. The student is running around exploring everything. And 10 steps behind this student we have our good old teacher EI system that is just averaging all the input and everything that is that student learns and changes only real slowly. What is the benefit? If the student goes crazy with one idea or hallucinate one idea, the teacher will be the anchor that says okay this is not within our parameters. You're not looking for the solution. Let's stay with our old solutions. So the hybrid ruler pool is self-explanatory. No, we generate m answers from the current policy. Told you fast, crazy, unstable little fellow. Then we generate n answers from the momentum policy or slow stable grandfather. And then we have a majority voting here where the grandfather has of course also the wood and we combine this to determine here the sido ground truth of a self-arning AI. And you might say, hey, this sounds absolutely perfect. No, we have a sido ground truth of a self-arning AI. What could go wrong? No. And you say, well, yeah, but at first, why does it work? This momentum while this grandfather AI acts as an anchor. Now, this is a slowly sinking grandfather says, "Yeah, run around. I'm see I'm going my stable path forward." So even if the student tries to drift off a cliff, the teacher who remembers how the model behaved the last 100 steps here pulls the truth back to the center onto the manifold that is proven. Inclusion of a stable momentum model rolls out in the voting pool is crucial for mitigating yet the noise and the instability of labels generated purely from the rapidly changing current policy model or little student model. And with the PO ground truth established, we can now calculate the reward scores of the current policy model M rollout based on the PO ground truth. The reward is binary. And you know here we have the normalized advantage function that you were familiar with. And if you want to see the final learning objective that we optimize, this is it. And if you're not familiar with the mathematical nomencleical tour, I have a particular video where I explain EI mathematic the easy way in 10 in 50 in 52 minutes here. I explain everything that you need to know about what does those symbols mean. Great. Now coming back here to our actual paper. Yeah. So we have here low reasoning bad reasoning here. Data flow coming in. Beautiful. And now we understood we have here some momentum prisms. No, we have here some glass structure here that kind of optimize now direct here the flow and I told you we have some additional idea here for MRGBO. This is a filtering system and you might say why do we need this? Remember, we have to still do to care about the entropy collapse and our filters will help us here that we have just a perfect laser beam, the perfect reasoning coming out here of our engine. How do we filter? You're going to love it's so simple. We have an IQR trajectory pruning methodology. IQR the interquartile range. the simplest statistical idea to solve here now the entropy collapse. Now in the good old days we used to um apply your static thresholds no somewhere if you read here in the instruction then you had here the line and now you throw away the bottom 10%. And you thought well that's an innovative approach for artificial intelligence. No just throw away here everything at the bottom. Great. Now in this new methodology we are much more scientific. No. So we calculate the entropy of every trajectory in the batch. We calculate now the statistical spread Q1 and Q3 of the interquartral range and then we throw away those. So you see now we are much more professional. Yeah. So we discard here the trajectories that are statistically of a low entropy outlier function. Yeah. So more or less we throw away the bottom 10%. Great. Now what is the impact? The impact is what we hope for is to get rid of the laser overconfident low entropy reasoning path in the self-arning AI structure here. No self-referencing uh or so if you want we starve your eye model of the data that causes you the mode collapse and we force it to learn here for more diverse more high entropy sorts. So we just have a pre-selection what we say okay we just want you the high entropy sort to be here in the new training data set. Now what about the performance? Now it gets absolutely fascinating. Does it work and we encounter a problem? An absolute unforeseen problem. So let's go. So here you have if you want the result table now December 15, 2025. Beautiful. We have it here for the benchmarks in mathematic in reasoning and in code. Great. Of course they go with a Q13 4 billion base model. Beautiful. The original performance of this base model is as you see 61.5%. Aim 25 here 5% reasoning 34% and code about 10%. Great. So now what do we have now? You remember we have here our SRT the classical one and we have here two. We have the best and the final. So what is the best? I told you this is the high score. This is the absolute peak value here that we will cherrypick or did we use to cherrypick here from the SRT model. Now this was the absolute best performance somewhere happening between 200 and 400 or 800 steps and I just had to test everything and then I said ah it is at step 612. Okay this is the best possible way the SRT best in SRT final. This is not a low score. This is not the model at the end of the training run. So if you want this here we see the policy collapse. Now you see the original model was at 61% and now the SIT final is at 47%. In the particular case of Matt 500. So if you look at some other cases here at AIM25 we go from 5% now to 8% but there's somewhere on the way to this 500 steps the peak of 11.67%. So you see this was our cherry picking. This was really we had no idea where the model would land. No, this was really expensive. Now unfortunately as you see there is a typo. There's a mistake here because we have a second SRT final. Now this is impossible. Look at just the values. No, but since those values are so close now to this value, I have um idea. So I think here the SOT final should be simply the new MG grpo final because the very next one is here the MGRPO plus the filtering final. So it makes sense to show what is it without the filter and then with the filter but it cannot be an SRT final because you can't have two different rows with different values and the same headline. So unfortunately this is not correct here published here. This is why it's a preprint and not here a peer reviewed paper but okay. So I will continue with my assumption that this here line number three is not SRT final but MGPO final without filters. And this would explain that we have just a little tiny jump here in the performance. Great. Can we find somewhere else in the paper some verification for my assumption of this typo? Yeah. Look, the magic if you want of this MGRPO methodology is now that you don't need to cherrypick anything because look at this. The accuracy reward just goes up and the validation accuracy is not crushing down here at 500, but it is going up and plateauing nicely. So I know this is a much more stable model. So I will do now my GRPO is now with MGRPO. It will stabilize the training so that the final model is also the best model or at least a plateauing model. So you can let it run as long as you want. I mean as long as you can pay for it without the fear that you will have here an immediate crash at I don't know step 500 or step 600. Real nice. Therefore, I think this is just a typo and I go with my assumption. But you see just one typo here in a mathematical paper and it's yeah, we are back to guessing. Okay, so step back. What have we solved? We have a problem. We had a problem. from a self-supervised reasoning suffers here from a policy collapse due to unstable targets and an entropy loss that as I showed you the fix of this new paper was twofold a momentum anchor stabilizes here the teacher [snorts] and a filtering mechanism that simply preserves here the exploration or the entropy we want a higher entropy reasoning values and the result we achieved state-of-the-art on our benchmarks without human labels and we eliminated the training instability which is great. So what is the final takeaway? Yeah, we are moving towards a world where the eye models improve themselves but of course hey within their own limits. No, not that you think they just go from an idiot to a genius. And this MJPO provides you the brakes and the steering that is needed to make the self-improvement safe and sustainable over longer training horizons. Beautiful. Now we can reframe this a little bit. At the very end of the video, if you're still with me, you know I have a little bit fun sometimes. I say, "Okay, can I see this from any other perspective?" So just think about the student. This is our current policy of the eye. No, the student is a little bit wild. No, it's an explorer. It's a little bit cotic. But the student can also hallucinate complete wrong patterns and it gets excited about this incorrect patterns. No, the student the eye might run off into a chaos dimension where it might think, hey, this bad mathematic is actually a good mathematic. No, and if you leave it alone, it just goes crazy and the result is what we all know as a policy collapse here with a self-arning SRT. Now what was the main idea? The main idea is simply you have a slow memory that you attach to it and you have here a voting system with a slow memory attached. So you have now a mean average teacher ERI system. This is here the solid momentum. No, this is the stabilizer. This is the anchor. The behavior of this momentum EI is simply it does not explore at all. It is just an average of the student's brain over the last 10,00 step. So whatever we survived together here the student and our and the teacher AI great it worked. We have not fallen off a cliff. No we are still alive. But the job here of the teacher is to evolve very very slowly. If I would be a teacher I see my student is hallucinating. I say okay yeah just you're hallucinating but we will not integrate your hallucination data into our reasoning stream into our reasoning trace. So you see it's such a simple idea and filtering then out that we are left with the high exploration rate with the higher entropy reasoning paths that are still within if you say the limitations of the AI itself. It's not just some crazy high entropy. No, it is just a little bit above here. The lower entropy is already enough to stabilize here the complete system. So if you want here I see here the mean teacher AI or this momentum AI as a kind of institutional memory that is allowed to vote on the next steps. So this AI remembers what was true an hour ago, thousand steps ago and is now a little can say very skeptic about any new crazy ideas. So therefore the authors try to get rid of real crazy hallucinations. The disadvantage is we go baby steps forward. No tiny little steps. Tiny little steps. So like if you want a coolback label uh term if we have here probability distribution. So the idea is in most cases always the same simple idea in artificial intelligence. Okay this is it for today. I hope you had a little bit of fun. I hope you achieved here some new insights. Anyway, hey, why not leave a like, subscribe, become a member of my channel. And anyway, I hope to see you in my next