Explaining OpenAI's o1 Reasoning Models

Key Takeaways

okay so open AI has released two new models and these models are actually quite different than anything we've seen from them before so let me just start off by saying these models are not GPT 5 I repeat these models are not GPT 5 these models are called 01 and 01 mini and we also know that these models are not a substitute for gp5 in fact at the end of the blog post they specifically say we also plan to continue to veloping and releasing our models in our GPT Series so what are these 01 models so specifically these models are reasoning models and the idea here is that they're going to basically take your prompt or your sort of question and they're going to think about it more they're going to break down that thinking and far more than what we've seen in the past with standard things like Chain of Thought and stuff like that these models will actually take time and that amount of time seems to vary based on how complicated the questions are that really hints that perhaps these things are even doing multiple passes as they go through so if we think about the standard sort of chat GPT and the GPT 4 models Etc you basically pass in one prompt you get one response back from that these models are fundamentally working in a different way to this and one of the cool things is that it does seem that with this release suddenly a lot of the researchers that open and AI are talking publicly and privately to people about how these things actually work what they've been working on for a long time and some of the key benefits that you can actually see out of these models so let's start off by talking about how these models are different okay so let's look at what makes this different than the standard GPT models that are out there so they talk about this as being something that's trained with reinforcement learning and here it looks like that we're not just talking about reinforce M learning from Human feedback but actually sort of rolling out trajectories or trees and then using those trajectories to work out which one is the best one and then using that to basically improve the model in here so they talk about the whole concept of the large scale reinforcement learning algorithm teaches the model how to think productively using its Chain of Thought in a highly data efficient training process then on top of that the thing that makes this seem to work even more is that not only are they doing that at the train time but when they come to the inference time they're actually doing this as well here and we can see from this graph that not only are they using large amounts of compute in the train time but they're also using that test time compute as well and they even have the statement here that the constraints on scaling this approach differ substantially from those of llm pre-training and it sounds like the major reason for that is that they're rolling out these really long Chain of Thought trajectories as they go along so if we look at a sort of standard llm for training you would have sort of standard pre-training where you would train different kinds of data and stuff like that you have a certain structure of how you input the data and the Order of the data then at the end of the pre-training you do like an analing stage to wind it down and then you come to your post training up until now post training has generally meant doing some kind of supervised fine tuning or instruction tuning and then also some kind of rlf reinforce learning from Human feedback or reinforcement learning from AI feedback as in RL aif now these can sometimes be replaced by DPO and stuff like that but the concept is kind of key here that you're basically aligning the model and guiding it to the kind of outputs that you want to have once that's done basically doing inference out of this model is just a single pass through and it requires way less compute than anything in the training itself it does seem possibly with the 01 model that what they're doing here is in their posttraining while they have perhaps reasonably similar kind of pre-training in their posttraining they're also adding in a step and we don't know for sure where this is happening my guess is it's probably happening after the supervised fine tuning or the instruction tuning and this step is basically predicting out lots of reasoning traces or sort of self-play trees and then having some kind of Checker or some kind of system to see which one worked out the best and then reinforcing that in the model itself through updating the weights so really what they're doing is they're actually training in the chain of thought into the actual model and this is one of the things that Jason wayy points out in his tweet that came out about this that one of the things that they've learned is don't do Chain of Thought purely via prompting train models to do better of Chain of Thought using RL and once you've got those chain of thoughts you can see that they mentioned through reinforcement learning they're able tone these chain of thoughts and refine the strategies that they use so then ideally the model then learns to recognize and correct mistakes perhaps even do things like backtracking where it starts to predict something realizes that's wrong it erases that comes back to a certain point and then predicts again from that point and this is where we see one of the biggest differences in the model for inference is that it's using a lot more compute for inference because now it's generating these really long Chain of Thought sort of reasoning traces that unfortunately it looks like open AI is not sharing with us and we don't know if those traces are all done in one shot if they're done in sort of multiple shots and then combined together the details of how they're doing that are kind of being kept secret at the moment but what we know is sampling out so much more via inference here is improving the results of the model and you can imagine that as these things get better and better at sort of long form Chain of Thought reasoning traces that those things alone become a whole new data set for training and for improving the model so the model is then able to predict out these multiple trajectories of Chain of Thought have some way of working out which one is the best one and then use that as the data to be able to improve itself for these reasoning SL Chain of Thought traces going forward now you imagine that you're doing this Millions if not hundreds of millions perhaps even more than that times and suddenly now you're getting something that's very good at breaking down the instructions in a prompt being able to sort of translate that to the problem and then tracing through this sequence of instructions to perhaps even do sort of fin uh chain of thoughts on individual parts of it or assemble kind of logic to it to be able to come up with something that's kind of like checking itself as it goes along before it comes to the final result out in here now we' got a number of sort of key tweets from people who worked on it that kind of reinforce some of these ideas so you can see this tweet here from lukash Kaiser talking about that models that train hidden chains of thoughts more powerful than raw Transformers they learn from less data and they generalize better so it certainly does seem that this whole idea of improved chain of thoughts seems to be one of the fundamental things behind the 01 model's ability to be able to get better answers out out so when we're looking at the avows that they've done it certainly seems that the data sets that they've benchmarked against are things that benefit from this long form Chain of Thought reasoning kind of thing here so that's going to be things like math like code like various sort of high level thinking tasks Etc perhaps not as much to be seen there with things like creative writing and that kind of thing well I don't want to go into their evals too much one thing I will say that's really interesting is that they compare the GPT 40 the 01 preview and the 01 so the 01 is a model that's not out yet it's probably not set up for inference in sort of production yet but one of the things that they point out here which is really interesting that unless otherwise specifi we evaluated 01 on the maximal test time compute setting meaning that we know that each result out of this is not necessarily perhaps just one pass through it it takes time for it to generate the output and you could imagine that's doing either multiple passes through of passing the chain of thoughts that it creates back into itself or perhaps it's backtracking on some of them or perhaps both of them in here compared to when we see the 01 preview compared to the 01 it may be that this is just taking a lot longer time to be able to do this and they do mention in their evils that when they give it the longest amount of time it's able to perform you know much better perhaps than what what we're seeing in the 01 preview and I guess the idea of thinking here is that they're not going for the fastest model with this and they're not really worrying if it's going to take 20 seconds a minute a few minutes to basically come up with an answer or perhaps if it's going to take hours or even days if that answer is going to be a cure for cancer or some new drug Discovery or some new breakthrough in science or medicine or mathematics so the idea here is that yes you're in resting a lot in this test time compute or this sort of inference compute that's going on but you're only going to do it for really difficult problems or things where it's worth it to basically let it take its time to come up with an answer so they mentioned that the models have done really well on code on math on a whole variety of these kinds of tasks interestingly though this model doesn't always get the best result out for things where humans are evaluating them where it's you know kind of like subjective evaluation so things like personal writing it doesn't seem to do as well as GPT 4 but with things like data analysis computer programming math humans actually evaluate the outputs of this as much better than GPT 40 now it would be really interesting to had them compare this against the original GPT 4 and see a lot of people believe that gb4 has taken a hit in some of these domains already so one of the things that I think everyone is getting excited about is these hidden chains of thought what actually is in them what you know they actually doing that it but unfortunately when we come into to actually look at the details in here we can see the open AI has actually decided that they're not going to show you those hidden chains of thought and they claim multiple factors for this including the user experience obviously their competitive advantage another thing is that they claim that those hidden chains of thought one way that they can monitor the model to see for any signs of the model trying to manipulate users trying to do anything underhand in there so another thing that's really interesting is the second model that they've released the 01 mini and this model they claim seems to be optimized for stem reasoning so again we see things where it's doing really well on mathematics on coding and you can see when compar to the 01 preview it's very close in many things and actually rates higher for things like you know data analysis in here so this sort of raises the question of is this technique that they're doing really reliant on the size of the model or is it just that creating this ability to have really long chain of thoughts is something that's going to improve every model which really is exciting from an open source perspective that you could imagine if we can make models that just generate these kinds of long form reasoning traces you could get a substantial boost so then the question then becomes is this a new form of scaling can you scale out this way and interestingly in regards to this we have a tweet from gome Brown who's one of the people who worked on the team also a well-known researcher but even before he joined open AI for creating some of the poker playing systems that were using reinforcement learning ET and he talks about some of the results that they've got that show that this is not just a one-off Improvement that you get for each model this is a new scaling Paradigm here and again he reinforces that this whole new dimension of scaling is inference time scaling so that we're no longer bottlenecked by pre-training we can now scale inference compute too so overall while there's no in-depth paper about exactly how they're doing this and many of us would love to know the details of how they're actually doing it and also many of us would love to see those hidden chain of thoughts in there this still gives us a clear sense of the direction that they're going with these reasoning models and perhaps a taster of what we're going to see even more in the future when this gets combined with a new GPT model so you could imagine that the next GPT model will be using this kind of thing as well as improving its pre-training and its sort of standard post-training before adding this kind of thing in there all right let's jump in and have an actual play with the models so the standard question that people seem to basically start off asking the 01 models is all about strawberries and how many RS and strawberries how many T's and strawberries Etc but if you look carefully this is all getting it correct already with gbt 40 so I kind of feel like in many ways clearly the models have either been trained on this they already know some of this and even gome Brown basically said that question wasn't the reason why the model had been called strawberry in the first place and honestly this sort of reinforces a lot of what I've seen trying things out is that a lot of things if you look at the latest models they're actually quite good at these prompts already and often get them right okay so if we ask it some questions to basically get it thinking my mother was 24 when I was born how old is she when I was oneir her age 1 half her age and 3/4 her age so you can see that the model's determining ages it's doing a whole bunch of this sort of Chain of Thought in here as it's actually working this stuff out so it's calculating it worked out that it needs the ratios to be able to do this so it is very interesting looking how it kind of works out the various Chain of Thought Parts as we go through this so it's got calculating age ratios determining ages figuring out variables determining family ages so for me it's interesting here that it sort of decides like okay it needs to have some variables to actually do this and you could imagine that this is what's improving this model so much over comparable models of the same size is just that it's got this ability to be able to do these chain of thoughts out as it goes along okay so this one is an example of a cryptic crossword clue normally models totally suck at this I will be surprised if we can get it right let's see how it actually goes again though look at the chain of thoughts coming out as it's doing this I think in some ways this is the kind of interesting thing of basically analyzing possible meanings breaking the clue down this idea of breaking down the prompt into its component parts seems to be a really big thing with this model piecing things together and we can see that it's taken quite a long time to basically try and work this out of what this should be doing now this basically the nine means that it's going to be nine characters long so my guess is it will probably understand that okay so we can see that it ended up taking 58 seconds to do this and it didn't get it right it failed quite miserably it should actually be Bara was the answer here but let's look at the chain of thought that it did as it went through it it's like breaking down this clue analyzing possible meanings so it's worked out that supporter down under is an Australian slang in here and you can see that it started to guess at certain things but it's then realized that those things are clearly wrong and so perhaps it's backtracking on some of these as in generating answers and basically trying these things out as it goes through weighing the various options it doesn't look like got close to the answer at all in there it is interesting that it's understood that okay that certain anagrams don't fit like that it needs to be a n letter word in here so I think this is kind of interesting looking at its reasoning as we go through it but it doesn't come to the right answer in the end for this one okay if we give it these kind of relationship questions of where you're asking it about brothers and sisters and like that I think there've been a lot of these around it's actually quite good at being able to work out these kind of answers quite quickly and we can see that this kind of thinking doesn't require as many Chain of Thought traces or certainly as long traces coming out of this lastly I just come in here quickly and I want to see if we can get it to give us some of its reasoning of how to break down problems okay so this time I'm asking can you explain to me how to break down a math problem and give me the exact steps needed to problem solve for situations like this I'm really curious to see okay can it actually give it some of its own Chain of Thought thinking it's interesting how it's got analyzing The Prompt breaking down things again it seems to really look at the different things and it's kind of interesting also that these things change right it doesn't have the same kind of analyzing The Prompt every single time so we can see that it's given us almost a prompt right of breaking down a math problem understand the problem read carefully identify what's given determine what's asked visual representation devise a plan carry out the plan review and check and in many ways you could say this is kind of what it is doing in here so to get at that it was like analyzing The Prompt assessing the context breaking down math problems withholding assumption that's kind of interesting breaking down the task mapping out logical steps verifying the equation so it's gone through and actually sort of done these in a way that's sort of giving us something out that we can use for this my guess is that by asking it a number of questions like this you may be able to spot some kind of common certainly there's some interesting patterns in GPT 40 when you ask these kind of things you start to see some of its reasoning traces and you start to see some of the things repeated in there but clearly this is taking it to a whole new level in here right let's jump in and have a quick look at doing a few of them with code to see what we actually get back in the responses okay so I've just got the open AI package in here I've got a key that I'm bringing in and then I'm just taking one of their examples and generating it out in here to see okay how this actually goes so the big thing here is to just look out that we get back code Etc but also we getting back this explanation for this and in this case the explanation is actually quite short even though it seemed to be longer in time for generating this okay so this time I'm asking it what is the value of the last five numbers in the first 100 digits of pi added together so let's see what the answer we're going to get out of here is and I'm also more curious to see okay how does it actually show us the tokens that we're paying for but we're not actually getting out of this okay so it's turning away we can see the time that it took there was 28 seconds of going through this what we can also see is I've just got and got pi and we can see that these are the last five digits in there so let's look at the an and see what we got back okay so just printing it out and looking at it here we can see that sure enough it worked out that it needed the first 100 digits of pi and it looks like it got those right from there so it's basically come out and that it's got 76 7 9 and added them up and got 29 which fits what we've got in here as we're looking at it okay so just coming in and looking at the response that we get back we can actually look at the usage tokens and we we can see how many completion tokens what the prompt tokens were for our input what the total number of tokens were and we can see this number of reasoning tokens in here so that you can see sure enough we really only got a couple hundred tokens out of the model we only put in 27 tokens but we used 3,000 plus reasoning tokens in there so calculating that at the the sort of output rate we can see that we spent about 18 cents on actually doing the reasoning for this task so this really starts on the track to wonder whether this is something that know you really want to spend this money on it's also going to make you wonder a little bit that okay why am I paying for tokens that I'm actually never getting to see also it would have been really interesting to see what is actually in those 3,000 tokens that's a lot of tokens for doing Chain of Thought in there and even looking at the verions on the chat GPT interface they're not outputting that many tokens when we look at some of those things okay so if we take one of the questions we asked earlier on I was curious just to see what we got out in and you can see here this the whole question about brothers and sisters Etc we're getting out a very small number of tokens here and you can see the output tokens only 100 plus tokens but the reasoning tokens are you know a lot more than that now okay we only spent two cents bit over two cents on the reasoning this time but we can see that the reasoning tokens are just so much more than the actual output tokens so this is again really interesting to see okay if we could actually get our hands on those reasoning tokens my guess is that you could probably train up an open- Source model to replicate a lot of this stuff quite well they probably have some you know interesting ways in how they're rolling out the trajectories with different Chain of Thought but overall it seems like the the trick of all of this is just to get really good Coots and then maybe have some backtracking and stuff in there as well so when we come in and look at the pricing for this we can see that these are definitely much more expensive than the most recent 4 models that are out there whether that's the full-size one or the mini one so for the 01 preview and remember this is just a preview we don't know if this is actually going to be the same pricing for the full 01 model but currently this is going to be six times what the cheapest GPT 40 model is so that's basically going to be $15 per million input tokens and $60 per million output tokens if you compare that to the most recent 40 model that's only $10 per million output tokens that we've got there so this is fundamentally different in here and when we look at the mini models comparing the 01 mini against the GPT 40 mini the difference gets even more we're now looking at a sort of 20x of where the GPT 40 mini is only 15 cents per million tokens in and 60 cents out as opposed to the 01 mini which is $3 per million tokens in and $12 per million tokens out added on top of these like I've already talked about you're also paying for tokens that you don't actually get back right you're paying for all that extra Chain of Thought tokens reasoning tokens as they call them in the API but unfortunately you don't get those to use you only get back what open AI allows you to get back as the sort of final answer with perhaps some explanation thrown in there as well so definitely pricing wise these are quite different from what's been out there now you could imagine that open AI May sweeten this deal in the near future by having some kind of automated router that routes the easiest prompts to the cheap models and only uses the 01 for prompts that require a lot of reasoning require a lot of thinking as it goes through Etc but clearly pricing is something that you've got to think about as you're going to start using these models in your organization for things like agents Etc so just to wrap up what does this all mean for LM apps in many ways you can actually look at the 01 model as being like an agent in many ways we really have to trust open AI to actually know whether this is just a single model we really don't know if it's doing multiple calls we suspect that's what's going on we don't know for example if it is using like some kind of open interpreter like a running code that it's writing and then returning that into the Chain of Thought as well and so in some ways you may want to think of this as being an example of where things are going with agents and I certainly do think this model is going to be very useful in helping agents to do planning and perhaps getting your agents to stay on track with lots of small little plans as they go along for this and this is certainly something I'm going to be testing out over the next few days and if I find some interesting stuff I'll probably make a video about it but I must say I do like the idea of that we're now in this realm of reasoning models so this is something I've talked about before and even if this kind of system is really having just multiple bytes getting it right it still makes it really interesting of how we can use this for various kinds of llm apps you're probably not going to notice any difference if you were just using this in a chat app I think we're well beyond at the level now where most of the frontier models are actually beyond what most people use when they're doing some kind of chat or something like that but this certainly is something where you can get more in-depth thinking and perhaps more in-depth reasoning with this model than what we've seen before anyway as always if you found the video useful please click like And subscribe I'd love to hear other people's comments below if you've got any questions feel free to put them there as well and as always I will talk to you in the next video bye for now