Reinforcement Learning (RL) explained (LLM, Vision, Robot)

Key Takeaways

hello Community welcome to AI robotics 4. now in this fourth video what are we talking about we are talking about using the intelligence of our jet GPD of our large language model combine it with a real-time Vision Transformer system and put the whole system into a robot so we can interact with our environment a real interaction not just the words that you see on your screen but machines that do something in our world so we are talking here about learning robot policies to solve large language condition tasks from Vision input and I thought hey why not call Simply this new chapter here policy it is robotics policy if you want to be more specific here you see a definition what a policy is but based on our other models I just wanted to tell you hey the rt2 model is fine-tuned to act as a generalizable and semantically aware robotic policy and I wanted to talk with you about multi-agent non-markovian rewards in AI robotics but something happened in one of my last video this was the video I showed you here a specific research paper and in this research paper you see here that right next here to our different llm system we have the base model like gbt 3.5 or Jubilee 4 and then we always have here a reinforcement learning from Human feedback and I received a lot of question hey do we always have this reinforcement learning what is it and why is it almost with any model that we see now so before we talk about robotic policies we have to talk here about this reinforcement learning based here so that we can use it and for our new robotic policy so first question is this for all fine-tuned AI system what is the function of it so therefore I call now ai robotics4 reinforcement learning so what is it it allows our large language model on our vision model and our robotics model to interact with a real world environment you could also have a simulation you can have a game and it learns from the feedback it gets in the form of Rewards or penalties so our language model Vision model Transformer model learns to make decisions that maximize a certain reward so let's start from the very beginning what do we have we have a empty Transformer model and we have a huge data set for pre-training let's say we have here the complete internet so you know in the pre-training phase here we combine here we pre-train here our Transformer model with this data set and we create here our pre-trained model our pre-trained model costs quite a lot of money it is to give you an idea for huge domains like chatubity minimum one million dollar if you have a smaller domain like medicine or physics or Finance you could pick up and running for about one hundred thousand dollars if you do the pre-training and you have the data available if you decide you want a better performance system that is fine-tuned for a particular Downstream task you find yourself a data set for this particular task to fine tune it further you do Define tuning and you end up with a fine-tuned model that knows everything about the internet has been fine-tuned on your specific task and now we have a fine-tuned model and you might say okay what about reinforcement learning well imagine you want to further optimize now your fine tune model for a certain behavior of your fine tune model now think about jet GPT you have heard that at the very beginning of that GPT it had also positive and negative statements in its answer to the query if you want now to train a certain behavior that the system is only nice and friendly when chit GPT answers you have to train your system here you'll find your model for a certain behavior and this is exactly what reinforcement learning you want to reinforce a certain behavior on your system so what you need is here you have your model you need a second Model A different model this model has one function it gives you a reward it tells you if it gets an input from the fine tune model let's say a sentence tells you hey this is a positive sentence or this is a negative sentence so the reward is either you can have a binary plus minus good bad or whatever you like so you have here two models and of course those two models have to interact in a certain way so let's look at the reward model context of reinforcement learning a model that provides feedback or rewards to the agent this is our fine tune model that we want to further train based on the action it takes I explain what is in action in a second the purpose of the reward model is to guide the learning process of the complete agent so in reinforcement learning and age and interacts with an environment by taking action and for each action it receives a reward from the environment good bad the goal of the agent is now to learn a specific policy which is a strategy of choosing specific actions that maximize the sum of the rewards over time this is it so if you're training a language model to generate text the next word it is not obvious how to assign a reward to each generated word or each generated sentence this is here where our reward Model come in let's have a look at this now a reward model is trained to predict the reward for a given State action pair in the case of a language model the state might be the current text and the action might be the next word in the text that the system is generating and the reward might be a measure how good the generated new text fits to the other part of the text so once the reward model is trained can also be a Transformer mod it can be used to provide rewards to the agent during well by the reinforcement learning process and then the agent our base model uses this reward to update its policy and improve its overall system performance on a specific task beautiful now we know what is a reward model abstract think about it in you want in a mathematical way it is a function a reward function for the system Behavior you want to achieve and you get an evaluation positive or negative in the simplest way easy example look at this dog if you train him to watch here when your llms is training and to bark if the training finishes the reward would be a little piece of meat or something so your reward is always specific to the task that you ask for another example of a reward model the mathematical function here for NVIDIA is maximize the corporate profit for the global player and now we might say we have two options either my action is I have now a further optimization of my gaming gpus like RTX 1490 where my profit function is about 90 percent or my second option my second action would be I optimize or I focus my company on data center AI gpus the h100 GPU for example why my profit margin is about 1200 so given you overall mathematical reward function that you refined that you want the system to behave you want Nvidia to be a for-profit company and you have two actions that the system can act on you say to maximize my profit should I act on option number one or an option number two so you see the reward function and the reward model Define the complete behavior of the system and defines also the next action of the system if we think about a sequence and you know then we have time series multi-agent and so on of course you know that we do not go simply with one action but we want to have a normal distribution of a multitude of action but you can see here if you want to maximize your profit and you have here for example the option to go for a data center AI GPU with a very high profit margin you know where is your highest probability so we have our fine tune model we have our reward model and now somehow some miraculously we have to bring the reward back so that the model learns the model optimizes our fine tune model goes to stage two with reinforcement learning how does it do this well it's easy it is a Transformer model so you know what's happening we modify we further adapt we optimize here the weight tensor of our Transformer architecture of our model of our fine tune model for a certain behavior and this certain Behavior we have the numerical values from our reward model if we know this is the positive trajectory and this is the negative trajectory so this is all there is to it so once we have the base model which has been fine-tuned using supervised learning as I showed you and a second model a reward model could be a bird model the next step is to use now the reinforcement learning algorithm to further optimize the base model you notice this is PPO the proximal policy optimization comes into play here in the context here for example of Transformer reinforcement learning and you have seen T or l so two terms I would like to explain policy rollout I was asked what is a policy rollout now the base model the fine-tuned model also known as the policy as reinforcement learning terms it generates an action as I showed you it generates the next word in a sentence for example so in the context of a language model action generator word generate the next sentence or whatever the reward calculation now you know what is a rewarding you can imagine how we calculate this the generated actions are then passed from our base model to the reward model and the reward model calculates now on this a specific reward function a numerical value for a binary for the reward for each action the reward simply is a measure of how well the action aligns with the desired outcome if I showed you Nvidia wants to optimize its profit you know which action it will support what is a policy optimization the rewards are then used as I showed you to update the base model so the goal is to adjust the models parameter in a way that increases the expected sum of rewards the overall reward over a certain time period and PPO as an algorithm can be used for this optimization so you know statistics optimization steps you know where we are you repeat it you have several iterations and this system converges of course we can code this and code it is even easier look here is the home page from hugging phase 4 Transformer reinforcement learning and if you go there you see they just give you the code for three steps now what are the steps you have here a supervised fine-tuning trainer module a script you have the reward trainer and you have your PPO trainer and now you already know what it is this here our supervised fine-tuning trainer is simply that we fine-tune our model with a data set this is it you take for example a specific model that's available hugging face Facebook here you have a training data set and you define a sequence length and you do the normal fine tuning of a model as I told you you have the base model then you need the reward model now if it's not just a function but it can be also a transformer in your network in general so you need to train this model so you have a specific model and you have training data you have a data set that trains for example let's say it's a bird model and you have a movie review then you have a positive connotation with a sentence or a negative connotation this movie was beautiful this movie was horrible positive negative you train now this reward model and then you combine it and you simply have here a PPL trainer what do you do easy you have here your base model that generates here a response you forward this response here to my reward model I get the reward back this is good this is bad and then you have here the trainer you train now the PPO trainer with the query with the response and the reward and with those three parameters you are now in a position here to further optimize here your supervised fine tuning model using the rewards from the reward model applying the PPO algorithm here you have again in text if you want to see this in action this is easy this is what I showed you before the pre-trained model and the fine tuning and then here our supervised fine tuning trainer script here those lines here create the fine-tuned model this is it what we do every day then as I told you we have here to have here a model that calculates the reward that predicts the reward if the model has been trained on a lot of data then we have here the reward as I told you we have here now the next word the next sentence is generated by our system we forward it here to the reward system we calculate the specific reward it is positive the movie or it's a negative movie and then somehow we want to feed this reward back to optimize here our base model this is all there is so optimize mean or fine-tune model either you have a lot of practical code possibilities to do this but normally you adapt here the weight tensor for a certain behavior that comes here back with your reward function and this is done here as I showed you by step three to your PPO trainer this is all to it so if for example in the past let's say this was our jet GPT our root version that has positive and negative answers to the US and you trained then here with reinforcement learning from Human feedback this is another complication I will talk about later then you have here the PPO trainer PPO is an algorithm was developed by open AI so that now here the adapted weight tensor only provide answers to the user in a nice and friendly way this is all there is to it now time does not stand still so there are new methods new great ideas so we go from PPO to DPO to direct preference optimization so PPO is more or less here I think from the past and even if you go to hugging phase you have here the DPO trainer and integral part of the code that hugging face provides to you online it's a script you just call it here this is the complete code code for our DPO trainer and you can apply now a much more modern algorithm here for the DPO what else yeah if you want to see a complete code example this here is the GitHub Hub this is from funvera and you just go to his examples and you look for the file for the python file DPO and there you have the complete code you can run it you can play with it you get yourself familiar with the code short interrupt let's summarize the reward function the reward function is crucial in reinforcement learning as it quantifies the desirability of different actions a for a given State as a for Action s for state guiding here the learning process of the complete system so we want to find out of different actions that are available to the system and the system has a given State s so what is the state of state is the temperature the state is the velocity of the system the state is the energy of the system so to all the state parameters that we have on a robotic system for example we have different actions an action could be moved your arm forward move the arm backwards right left up down whatever you see for a specific goal you want to train this with a learning process the policy the policy think about policy about the strategy that the agent our base model uses to decide what of the multiple of action robot arm can move what action should it take move straight forward for a certain amount of space so in reinforcement learning a policy is a mapping from the states to the actions the goal is to find the optimal policy that maximizes the cumulative reward function and this often involves learning a function that can accurately predict the expected reward for each possible action in each state of the system so you see you want to find here the best way forward for the system to achieve the given task the beauty is the Transformers can be used to learn this function and Transformers can be used to optimize a policy in reinforcement learning so the Transformer's ability to handle here sequential data words in a sentence and model long-term dependencies makes it well suited for policy optimization tasks the example is you have a little robot to navigate a maze and here I will introduce here the notion of non-markovian Rewards the Transformer architecture he has particular great is particularly good here with non-makovian rewards which are characterized by delays and dependencies on this sequence of States encountered during an episode for example when the reward is only provided at the end let's say you have a mouse in a maze and somewhere in the Maze there's a little bit of a cheese and whenever the mouse finds the way to the cheese the reward Mr Cheese is only provided at the end when the mouse found the solution the right path to the chase as easy as that yeah a Markov decision process interesting it is very simple from the idea if you say that the future states of the environment of a system can be predicted based only on the current state without any knowledge of the past state so you see the current state defines the complete future then we have here Markov decision processes however if this is not the case we are talking about non-markovian rewards so this means that the reward at a given timestamp could depend on the actions taken what it states encountered at previous timescap now think about here again the mouse or the robot navigating a maze to find a piece of cheese if the cheese is located at the end of a specific sequence of ways how to run in the corridors the robot might receive the reward only when it reaches the cheese you see it depends on the sequence of correct actions first left then right turn straight on then two times left so this sequence of action that bring you to your goal and the reward is non-makovian because it depends on the entire sequence of action not just the current action or state and here in the non-markovian reward system we have architectures like the Transformers who are just working great if you want to learn more about Transformers in reinforcement learning there's a very good survey here July 12 2023 this is the archive go have a look at this report this is the principal schema that they explore and they look at all the different options that they are you can see here the different orders you find the link and the original paper where you have an agent in a specific State the agent now starts an action the action has an imprint on the environment you have a reward function that defined if this was a positive or A negative action if the agent wants to continue with this action or wants to take a counter action they run through this and they give you all the different models and papers and theories really interesting paper so as I told you Transformers we don't use Transformers only for a large language model and only for vision Transformers but we have it also in robotic Transformers and this is because they are amazing let's have a look even in the topic of policy learning what they can do and I give you four examples robotics graph based reinforcement learning story generation this is what you do with jet GPT you you write a science fiction story or you have decision Transformers in all of these cases you have a policy learning now let's look at robotics learning the reinforcement learning policy for a trajectory planning is essential in robotics either the arm has to move for the robot itself has to move so you have to have a planning for the trajectory Transformers have been used for processing sequences of high dimensional scene observation for predicting actions for instance uses Vision Transformer to extract spatial representation from a bird's eye view on a vehicle to learn driving policies this is what Tesla is doing with its Dojo chip Tesla is restricted to a two-dimensional sphere you do not have yet you can suddenly fly up in the sky or something but and you have fixed objects that you identify think about you want to drive a Tesla car on Mars you can't because the object on Mars or for the system unidentifiable so the car will have no ability to do this but in a combination with robotics you can learn robotics on unseen objects beautiful beautiful so navigation task path planning navigation this is great you have another way this is just I want to introduce you graph structure I have a lot of videos about graph neural networks and here Transformer are bridging here those two topics because think about this in many robotics applications the environment or the problem itself can be presented as a graph where nodes represent the different states or the different location of a robot and edges represent possible transaction between them to give you an easy example each row in a building could be a node and the doors connecting the different rooms could be the edges so you see you could simply from the map of a skyscraper create here a graph that a Transformer would understand so this topological structure refers now to the arrangement of nodes and edges and we have a graph specific representation that I have a lot of videos about how to do this but the beautiful is we can use this structure for decision making in reinforcement learning task with Transformers because Transformers can be used to process this graph structures and this is something really really beautiful you know that robots for manipulation tasks show you the videos a Transformer used in robotics picking up some chips or assembling some parts or moving object from one corner of the desk to the other corner but of course if you think a little bit more complex if you think about as I told you you have a spacecraft in the atmosphere of Jupiter multi-agent system Transformers can be used also in multi-agent systems where multiple robots are multiple systems agent base models llams lead lighter Rada visual Data Systems needs to coordinate the action to achieve a common goal and here the ability of a transformer is you take in all the current state of all the different agent and you output a distribution of a possible actions for each agents and the Transformer can be trained to adjust this distribution to favor specific actions that lead to the successful completion of a task so for example in Jupiter's atmosphere suddenly our spacecraft has to switch from visual sensors to infrared sensors or to any other sensory array combine the information from different sensors and come to a conclusion what is the object that is right in front of in the path so in all these cases the Transformer can be used to learn an optimal policy that maximizes the expected cumulative reward for the system and this is the beauty now how do we do this training of a transformer in a multi-agent system it is as simple as it could be you have an input representation this is your input you have the model itself with its architecture and as I told you if you have a model and we do RL we need also a reward model a reward function so these are our if you want the recipe our three main ingredients First Step represent the current states of all the agents in a way that can be processed by a Transformer involve encoding each agent State into a vector you know Vector embedding the vector store we do if we read in a PDF file for example to judge GPT and then we concatenate those vectors into a sequence or we perform certain specific mathematical operation with multiple vectors we build different complexity in Vector spaces if the actions of array agents are also considered they can be included in this state representation here really your fantasy is the limit of the performance now the model itself takes the sequence of state representation as an input it processes the sequences we remember I showed you the token we had at first for example the language token then we had the vision token and then we had to do robot actuator tokens we process this sequence of tokens using the self-attention mechanism feed forward which allow the model relations between different age and state and then you define a reward function to evaluate the quality of all the different actions taken by all the different agents and you calculate the the specific set of agents to achieve you go now the training algorithm is again as I showed you trained using a reinforcement learning algorithm like PPO or any suited optimization the goal of the training algorithm is to adjust the parameter of the Transformers to maximize the expected cumulative reward as always interestingly here there is now a balance between exploration and exploitation so during the training you need to have here this sensitive balance between trying out completely new action to discover completely New Path forwards for better strategies or you stick with the currently best known actions and you are only allowing it to deviate a tiny little amount from the known best action thereby limiting yourself from finding complete different ways that might lead to a significant better performance so given the amount of money that you want to invest here in the training this balance between exploration and exploitation is really something you you have to fine-tune yourself you say hey I want to stick to the known actions or you say hey I'm courageous I'm brave I try out completely new action I want to see here if I have an action manifold an action space continuous or discretize manifolds how many path forward there are for my systems and then you have the training Loop and you do again and again and again till you reached here a satisfactory level great now this was the interrupt about reinforcement learning with an eye to our AI Robotics and now the next video will be AI robotics 5 and finally I can show you here the mathematical definitions and the real formal structures to Define robotic policies I hope to see you in my next video