POPE RL Curriculum Learning (CMU)

Key Takeaways

Hello community. So great that you are back. Today we talk about a new methodology in curriculum learning for artificial intelligence. And here we have Cornic Mlan University telling us hey we have a brand new methodology as published January 26 2026 how we can deal with reinforcement learning to make our AI models more intelligent. Now you know we in general have a core problem. We have the valley of death in reinforcement learning. So you know this no on a hard problem the probability that an AI model maybe a small model randomly sampling here a correct chain of sort can be effectively zero. This means the model generates 100 wrong rollouts. The reward is exactly zero for all of those rollouts. The gradient is therefore zero and the model learns absolutely nothing. In reinforcement learning, we can encounter those learning plateaus anywhere in the manifold here for reinforcement learning. And the consequence is clear. No, we are currently limited to any eye training methodology on particular problems where the IM model can already almost solve these topics. But this also means we cannot teach our AI truly new order reasoning capabilities via a pure reinforcement learning. This is here also what the orus here of conl tells us this is a real problem. Yeah. Now we have two standard solutions. The first is the supervised fine tuning. It turns out it's a trap. The second of course is curriculum learning. Turns out it's a trap too. So let's have a closer look. SFT this simple know you have a hard problem and you have a human oracle solution. So you clone them all on a human solution to teach it here this new data. Why does it fail? Simple. They give you the reason because the human reasoning is structured and represented here as an off policy. This means it is statistically very different from how the AI model internally thinks. Supervised fine-tuning forces model to memorize here more or less only the human specific token paths. And this causes an entropy collapse in our supervised fine-tuning AI models. This is not what you want to achieve with learning AI. Now the supervised fine-tuning model can now recite a specific human answer. And you might think, hey, this CI learned exactly what I wanted it to learn. But guess what? On the other side, and you do not notice this, this model loses its ability to explore or self-correct. And you know, we always have this the hard between exploration and exploitation here. So we are either searching or we have a deep dive in known solution. And this paper by Carnegie Melon shows us that supervised fine-tuning actually hurts in particular then the downstream reinforcement learning performance and you would say yeah of course if the human traces are off policy so what we do the eye uses now reasoning leaps or stylistic patterns that are alien to the eye mal's internal latent structure and we do have a decoherence process here in the learning you are familiar with this I've given you here multiple videos here in the last month like this here for example Google invented a new training methodology or I've showed you here if you want a hypergeeometric edition where I tried to explain here a more unified theory of AI reasoning integrating here based on the work of Berkeley and Nvidia here supervised finetuning and reinforcement learning in the next generation we had look at AI phase transition and a quantum or the reasoning process and of course we showed you here that there are different low cores subspaces if we go then with verifiable reward structures we know that SFT is not working now let's have a look at curriculum because curriculum was the general opinion this is working fine no we have some easy data and then it becomes more and more our training data more complex And so like a curriculum, we guide you our AI from some simple data to the more hardcore knowology on your domain specific topics. Now start out with a 50% hard problem and 50% easy problems. No, and you just hope for the skill transfer. And the authors show us in this paper by Carl why this fails. The authors identify after tests and tests a phenomenon they call ray interference. Careful, it is not interference from computer science. This is interference from physics. So what is happening? It is easy. I explain it in simple terms. You can have it on a pure mathematical level or if you want here the gradient that you see from the easy problems in the training data is strong and directional. Yeah. I I easily identifies, hey, I know exactly how to solve this task query. No, there's a high signal. If the eye encounters now real hard complex problem, the gradient is almost zero. The eye has no idea where to go to, how to solve it, what is the next step. So the result is the optimizer simply follows here a loud signal. And this is here the easy problems. So if you want it kind of sharpens now the eye model on the easy problems pushing here the weight distribution into a local optimum that makes it even harder now to explore the higher entropy path that are needed for the hard problems. So we steer the complete AI model away from solving hard problems. Hey look it is so much nicer, so much more fun, so much easier to go only for the simple problems. So this inhibits learning on hard data. The authors show this in some beautiful details. Now they have a new idea say okay if we have these two problems what we can do we have now a third paradigm that we like to introduce in reinforcement learning and these are helicopter drops. Now the idea is simple. If you have to solve this maze here and at the beginning here at the start of the maze, you have a little bit of help. You have a trace here of tokens that guide you in the right direction. This is it. So instead of forcing now the agent to walk from the start or carrying it to the end here, this new pope methodology drops the agent if you want halfway through the correct path or shows it here. the path here with the golden tokens here. Anyway, it uses a prefix of the Oracle solution and just tells it, hey, listen, I dropped you off here, so you are on your own, my little AI agent, but just follow here for the first three steps here the prefix I in addition provide to you for finding the solution and then find the rest of the way yourself. So you provide here some startup help to the reasoning process. Now it is easy to miss here the point because if the agent itself generates here the rest of the path itself the learning is now an on policy. The agent now uses its own reasoning style and maybe it will succeed and maybe it will fall. But since it's a probabilistic system, there is still a chance, a nonzero chance that it might find the correct solution. Now, the idea is now simple. Over time, through here what the authors call a phenomenon like stitching, the agent hopefully learns to link its unguided starting attempts to the successful intermediate paths eventually solving here the hard problem from scratch without now the helicopter drop. So you see we move now here from I just give you a 50/50% distribution from easy and hard tropics. I give you here a hard topic but I give you here the first three correct step to find your solution yourself. So you will be on policy and of course if it's a complex maze if it's a complex path a complex manifolded the eye has to explore you hope that over time statistically this will come together and find via stitching here a complete path. I would say this is here a nice idea but does it work in reality? Now the authors extract here a prefix such that our base policy pi has a nonzero probability of completing this. Now this is here quite interesting. So you have to really carefully design now a certain base policy a strategy in the reasoning process of our EI that you know has a nonzero probability of completing the solution. So you have to know exactly what this AI model is able to solve and just give it a little bit of a hint that it is not running into the zero gradient plateaus. Careful, this pope methodology does not treat the prefix as a target to be cloned like in supervised fine-tuning. It uses this prefix from the idea to transport the agent now to a different region of the state space where now some rewards are theoretically maybe attainable. The policy gradient is calculated on the completion generated by the CIO. Okay. So the inside is if the main idea is stitching by exploring here from intermediate state you drop off with the from the helicopter here your agent somewhere after three steps. No hope that the I model learns sub trajectories going forward that with the time will overlap with states reachable from the unguided start state. So this hopefully allows you the learned behaviors to transfer back to the unguided problem. And as you can hear, I have a lot of theoretically and maybe and hopefully. So you see that this is here an interesting statistical phenomenon. Now the question is how much energy would you have to not waste but build up to allow the mall here to find its own nonzero probability? Yeah, this is here from the publication itself. You have your standard reinforcement learning. Then here if you want the optimization pathologies that you if you look a little bit closer into the results you see here for bay the reinference this orange line. Yeah. And you see here if you have here the hard here on the x-axis and the easier on the y- axis, this orange enters now into this hard problem territory rather soon. And you see it tries here to learn also a little bit of the harder stuff. If you look now at C and D, the success rate on those hard problems here and you have on wider the success rate from zero to 100%. You see that the orange line our new methodology pop he is here faster here at the beginning and has also if you have a hard problem roll out here sooner a good solution. So I try to formulate this for you and you see I'm a little bit careful in my wordings now indicating an acceleration in the solvability of the hard problems. I did not write down indicating here that suddenly the eye was able to solve a complete new unseen art problems in coding and augumentation and reasoning and logic whatever no it is indicating an acceleration in the already available solvability of the hard problem in this reinforcement learning. So you might ask hey wait a minute so we are back to have limitations in reinforcement learning even with this new methodology in my understanding of this paper I would say yes the authors show us here another visualization to say okay so here we have the problem if you want here this is the drop off zone and then with this new methodology we show here the eye hey do not go here in incorrect regions We give you a guidance here of the first three step. You have to go in this direction because you know the green dot is here and here. So go over there and then yeah maybe you come back and circle around but this is reasoning with a guidance. However remember we in a real highdimensional space. So you have carefully to design to give the right amount of guidance, the right amount of steps into the right direction even for a particular complexity topology that you might be unaware of. So it is not that simple. And as I told you, I think the authors make this here beautiful. There are here some let's call it a plateau where suddenly in the reasoning process of reinforcement learning you get no reward back. The mall is on a plateau. So what we get back is zero and the mall has no idea where is my gradient going to drive me. What is my next direction? Direction is zero. So we are sometimes stuck on this reasoning plateaus and the hope is now okay and if I give it now here a little bit of a startup help no and I say okay where is here the correct solution we give it a little bit of help so an interesting idea but does it work out is it really a solution let's have a look at the result now Here we have it. Here just focus here on the blue frame. So if you look at the hard problems and complex problems here we have the classical hard problems posit one let's go with 13.5 whatever it is with this new methodology pope it increases here to 15 H you might say okay so the improvement is yeah okay what about here a real benchmark that we have feeling about aim 25. So we go here from the classical 49.58 percentage to 53. So the orers tell us look we are 7% better now with this new methodology. But if you look at the posit 16 we just go from 81.4 to 82.6 with the complete new methodology here of pope. So there's a lot of exercise we have to do for this and we have an improvement of plus one H. Now let's look here and I have here a title specifically on curriculum learn. Let's go here. If we have a hard task and an easy task and we have a curriculum learning, what is the difference? Now, if you go hard and easy without this new methodology, let's say at a pass one at a 25, we at 57.19. If I activate now this new methodology, pope, I go to 58.7. So, I have plus three H. So you decide now what you think is this carefully designed hope methodology where say I give it the first three solution steps in the right direction you have to design this you have to train this you have to provide solution for it this is here the performance improvement that we can see and I know what you see you say but wait there's another line and let's look here at this orange box here. So what about we have a lot of hard problems uh and some easy problems here. So what is it? Now here you clearly see what they found out about this continuum continuous learning problem here with the curriculum learning that there's indeed something because look the easy gradient the one drowns out the hard gradient. So on the hard problems that pass at one, we have a performance of two, which is almost zero. So as you see, if we have 1,000 easy problems and 256 hard problems, yeah, the model really follows now in the curriculum learning, it has a tendency to be redefined here to follow the easy route and not look here at learning the hard problems at all. Performance is two If you combine this with this new methodology pope now here you see we have an improvement of 524% and great but in absolute terms is we just go from a 2% performance to a 13.98% performance of an artificial intelligence system. So you might say okay yes there is an improvement and wow 500%. But yeah always check out this the real data and this is the beauty in this study. They give you the real data. This is really so beautiful in science. You don't have to rely on some marketing slogan. You can check here their data. Therefore I highly recommend this study. Now the authors call this particular behavior expanding the coverage of reachable states for DCIS. Now what is the underlying assumption I would like to to really pinpoint to you? The idea is that those AI models have already the knowledge to solve this more complex task. Just those AI models have not yet discovered the right path forward towards those knowledge manifold subspaces or the subspaces where the right solution is stored. Now you know you forgot this model here was really interesting by Chingua here. model whisper steering vectors unlock your LLM's potential in the test time compute. So, and they told us here in December 2025, we steer our IM model toward an internal state of a higher confidence activating its inherent abilities most relevant to the current task in the test time compute. Looking at this study and at the current study, I have a feeling that I have to tell you h I see an isomorphism between the token and the vector representation of those studies. And I think the connection now to this new study pope is matically profound because if you think about it, both methods are doing more or less the exact same thing. They use state space jumping methodology to improve the performance of their systems. And I think one of the reason Pope works and please prove me wrong. The reason probe works is the oracle prefix is just a textual steering vector that we apply here. Think about it. Stal state space is high dimensional. Let's go in a simple llama 3 4k dimens. Now we do have the easy plateaus. No, the model normally explores a small manifold simple easy dimension up to 500. And then there are those hard plateaus. The advanced reasoning capabilities live on a different orthogonal subspace or in a different mathematical subspace from the dimension 2 to 2,500 which are rarely activated by our standard prompts and have therefore a low probability of ever being thoroughly learned by this AI. So we do have this if you want EI shortcut. Hey, let's just look at the simple facts at the at the easy plateau. You know, we know how to move around there, solve the problems there, stay away from the hard plate, but with a textual steering vector enforced into the system, you really now put if you want here the eye really on this hard plateau and you kind of enable now here or you uncover here the solution in this order. I hope you see where I want to go with this because in my next video tomorrow I try to show you another point of view, another reframing. I will take another study that also was published just days ago and I will show you how we can combine now the insight from all the research that is done globally on this topic and how they can come together by piece. I hope you enjoyed this video. We had a little bit fun, maybe some new information for you. Why not subscribe, become a member? I hope to see you in my next video.