OpenAI's Q*?: Reinforcement Learning, Model-Based vs. Model-Free Methods, and Q-Learning

Key Takeaways

hi I'm Harper I'm head of AIML at breev dodev and today I'm going to talk about reinforcement learning modelbased and model free methods and Q learning because there's been a lot of interest in Q learning ever since open aai released information barely any information about their qar project and a lot of people are assuming it has to do with Q learning however I just want to say that this is all conjecture we don't know what qstar is about opena hasn't released anything about it and my guess is that it may not necessarily be Q learning um but it's probably a way of um estimating Q the value um function which of which Q learning is one way to do so and you'll learn about that today but Q learning is a type of temporal difference learning where you estimate q and the maximum value is qar but you'll learn about that today this video should be accessible to everyone not just people who have ml backgrounds um I try to make this easy to understand for people who do not have a computer science or machine learning background and if you do this video is a good way to brush up on um what you know about reinforcement learning and Q learning in reinforcement learning you have States and actions you can take from those States and rewards that you get from taking those actions from that given state so that's kind of the core of reinforcement learning so for example say you're a baby and you see a stove a hot stove you could stand there and touch the stove and get some reward for that action you could stand there and not touch the stove and get some reward for that action or you could walk away and so say touching the stove you get a reward of negative five or1 say not touching the stove and standing there you get a reward of plus one or plus two because your mom Praises you or you just walk away and get a reward of zero so those are the actions and those are the associated Rewards and in reinforcement learning you have this concept of exploration versus exploitation and exploration is trying new things to learn the rewards that are associated with those actions so for example you are the baby and you have never touched the stove before but you have tried not touching it you know your reward is one but you think hm there's this action I haven't tried what happens if I touch the stove and so you touch the stove and you learn that your reward is-10 and and you know from there and on out not to do it again because the reward is lower than the reward of not touching the stove and you also have this concept of exploitation where you exploit what you have learned already so you use your policy which is the best action to take in any state as in it gives the highest reward so once you've tried all the actions or you know if you've only tried a subset of the actions you could exploit by using your current policy by seeing based on what I've learned before or based on what I in some cases based on what I um my prior belief on what my reward is going to be I'm going to take this action because I believe it'll yield the highest reward that is exploiting and so this also applies to you know if you go to a restaurant that you really like will you order your favorite meal will you exploit or will you try something new explore and see if you actually like it better um to update your policy then so that you would order the new thing that you just tried because you had a higher reward so the concept of exploring versus exploit is represented in a transition function so you have some probability of exploring and some probability of exploiting and that is held by the transition function you also have a reward function which says given this state and this action this is the reward that you get and and in modelbased methods those functions transition and reward functions are held by the model and so you you actually know what those values are um however this can be pretty unwieldy for a model because you know you're holding every state and action combination and the reward so as the dimensionality grows as you have more States and more actions that's again unwieldy for this model to hold because there's so much there so model free reinforcement learning is when you estimate R and T rather than having the values explicitly you are estimating them through experience so as you go about the world you are estimating the reward function so the total reward into Infinity that you can have by navigating this environment and this Q function is being updated iteratively as you get new samples from navigating the environment we can incorporate time and we can have a Decay factor and a learning rate in the equation so that we are prioritizing State action and rewards that we see recently that we actually experience more recently and Decay ones that are older and Decay ones that are in the future because again this equation actually incorporates an estimation of what our future reward is and we'll see that in the equation so this provides some robustness to the model evolving over time time if you have some experience with machine learning you might know about the loss function which is the OB the objective of the model is to minimize that loss function and you do so usually using gradient descent you can think of the reward function in reinforcement learning and in this case Q as analogous to that loss function because you are aiming to rather than minimize maximize the reward function over time and so what you want to do is at every step maximize the total reward into Infinity that you will get so they're pretty similar they're both objectives of the model and one is minimizing and one is maximizing so Q is an estimate of the state action value function and so again it takes in the state in action and it estimates the discounted return over time and so we have this disc count value gamma we also have a learning rate which if it is a constant it will decrease past values and so we have these basically two discount values where gamma is for future so this is the future estimation over time so if you take the Max the action that maximizes this Q function so this is estimating in the future and as you go further into the future this gamma value increases so it iteratively increases as you go further into the future and then this is you know the current and the past and if we change this Alpha value this will reduce past values and then this is the next state and this is a diff this is temporal difference learning where you take where the increase is proportional to the difference between the present and past estimates and so again Q learning is one type of temporal difference learning qar or Q asterisk is the optimal q and it is possible when you are able to sample from all possible outcomes forever over time so continuously updating your reward for all possible outcomes and this is the name of the open AI project qar and again Q learning is one type of temporal difference learning um one way of estimating q but it's not the only one so we don't really know if they're talking about Q learning or even temporal difference learning but we maybe see someday so I hope you understand reinforcement learning exploration and exploitation model based and model free methods and Q learning and temporal difference learning a little bit more after watching this video so let me know if it was helpful and if you have any remaining questions and I will see you next time