Minecraft AI - NVIDIA uses GPT-4 to create a SELF-IMPROVING 🤯 autonomous agent.

Key Takeaways

so there's a new paper from Nvidia that you might want to see they got AI to play in Minecraft Neato but after reading this paper I'm kind of Blown Away the implications here are a lot bigger than I understood at first so let's start with what the results were at first but stick around because how they did it is I think much more important because we're beginning to see ai's writing code that improves itself if you're curious to know where software development is going and how AI is going to play a part in it you're going to want to see this so they've used gpt4 as sort of a reasoning engine but as you'll see here it's not really attached to Minecraft not really keep in mind that gpt4 can't see vision is coming soon but it's not rolled out yet so it's blind it can't see the screen what's going on so keep that in mind as we look at this you can skip around using the chapters and subscribing to this channel gives you a permanent intelligence buff so don't miss out alright so this paper is called Voyager an open-ended embodied agent with large language models I think I first noticed it because Dr Jim fan one of the researchers on the paper was the one that posted about in Twitter Voyager is the name of the a AI agent and he says Voyager continuously improved Itself by writing refining committing and retrieving code from a skill library and also we open source everything so if you wanted to try this out for yourself you can download it and let it loose in the world of Minecraft or you're choosing here's some of the skills that it can do in action so really fast we introduced Voyager the first llm-powered embodied lifelong learning agent in Minecraft that continuously explores the world acquires diverse skills and makes novel discoveries without human intervention empirically Voyager shows strong in context lifelong learning ability exhibits and Exhibits exceptional Proficiency in playing Minecraft it obtains 3.3 X more unique items travels 2.3 x longer distances and unlocks key Tech Tree Milestones up to 15.3 times faster than previous models Voyager is also able to utilize the learn skill library in a new Minecraft world to solve novel tasks from scratch while other techniques struggle to generalize this is gonna be pretty big as you'll see in a second so here's the Minecraft Tech Tree if you are not familiar with Minecraft basically you run around this generate world all the worlds are random unique it's pretty deep there's a lot of technology and a lot of things to uncover it's a Sandbox game so that you can do a lot of things there's a lot of interaction with animals there's cave systems day and night cycle there's a million different things that kind of happen this world it's very complex it's 3D which makes the damage more difficult for an AI to navigate and you also have things like hunger so you have to eat in order to survive you have things like Health breath in case you're swimming underwater you got to come up for air every once in a while if you're down in a dungeon where there's no light it's going to be completely dark you have to make some sort of light sources in order to light your way it's a complex and deep game and in it you sort of have this progression where you start getting the basic materials to create basic tools and slowly over time that improves to where you can make stone tools iron tools Etc and you kind of go up this Tech Tree and get better and create more things with it so here we see that Voyager consists of three key components an automatic curriculum for open-ended exploration a skill Library for increasingly complex behaviors and an iterative prompting mechanism that use code AS action space so I didn't really get how this whole thing fit together first so let's take a quick step back and let me explain how they sort of put everything together so first of all there's this thing called Mind flare API with a developers can get into a game of Minecraft and do various things like automating movement creating various events that can happen in the game and basically affect the world in there in many different ways so that's a standalone thing that people can use to interact with Minecraft you don't need an AI or gpt4 to do that or anything else it's just a tool now what they're doing is they're getting gpt4 to use a lot of these commands to move the player around in the world so for example if we want them to go and mine something then gbt4 causes a certain function and it uses that if we want them to for example place a workstation somewhere on the floor near the player gpt4 calls a script that does that and your in-game character goes and does that players usually will use a keyboard and mouse to interact with the game gpz4 will use these commands to interact with the game it doesn't get any special powers or anything else like that it's limited by the same sort of rules and physics that the normal player would be so it makes sense so far you have gpt4 you have Minecraft and you have a way for gpt4 to control Minecraft so gpt4 perceives the world of Minecraft not through vision like we said but but rather through prompts it gets updated with some basic information about its surroundings so for example on the left hand side here we see the information given to it so for example here we say here's some things that you have in your inventory what what do you want to do and so the reasoning and then so GPT thinks about it it's reasoning is since you have a wooden pickaxe and some Stones it would be beneficial to upgrade your pickaxe to a stone pickaxe for better efficiency and then it generates a task craft one stone pickaxe so it receives some information about its environment it thinks about what it's going to do and then it creates a task for itself to do it by the way here they use Chain of Thought reasoning in order to have it think through this stuff and then execute on it another thing is another example is let's say we tell it hey you're next to your River and you have a fishing rod what do you to do by the way this is kind of summarized of what it sees in general it gets a lot more information and it gets the same information every single time a little bit later you'll see exactly what it's getting with every single prompt here they're just highlighting sort of like the things that it saw from which to make a decision so here it says well so we have a fishing rod and a river we should fish and then creates a task catch one fish if it's night time there's a zombie nearby it thinks oh we better fight this guy and protect ourselves so task kill one zombie so so far everything makes sense right we have gpt4 we give it some information we tell you here's what you're seeing and then it reasons and decides what to do next based on that information here's where it gets a little bit more interesting it will generate code to do the things that it wants to do so for example here it creates a function called combat zombie so fight the zombie then he uses GPT 3.5 to describe what that function is basically the function is about equipping a stone sword to combat a zombie if a stone sword is not found it will craft one additionally crafts and equips a shield for added protection for added protection excuse me afterwards it proceeds to cook sticks in order to restore hunger once hunger is replenished it actively searches for a zombie and engages in combat and kills it so basically if the goal was to kill one zombie as we saw earlier this function would get the player ready to fight it and get its sword and shield get its food so that it's not hungry so that it has full health Etc and then it would journey to find that zombie and kill it and so then that becomes a skill so killing a zombie fighting a zombie becomes a skill that that then gets saved in the skill Library so gpt4 is creating tasks for itself then it's creating skills that will complete those tasks that's coding them and saving those skills in the skill library for later use and then we have the iterative prompting mechanism so for example if one of the skills that it's trying to create is it found some Akasha plants and it believes that sort of the next thing that it should do is create a an Ax from the Akasha plant it might create a it might create some code like this right saying okay so so get some Akasha plants and then create an Akasha Ax from it and then it goes into iterative prompting mechanism so it creates a function called craft Akasha X and everything else it needs to execute that however an error message comes back and says well there's no such thing as Akash ax gpc4 realizes it should craft a wooden ax instead of Akasha ax since there's no Akasha ax in Minecraft I hope that's pronounced Akasha Acacia Acacia it's pronounced Acacia I apologize Acacia acts whatever so then it changes the code to say craft wooden ax instead of Acacia ax or for example here realizes that it needs two more planks in order to create whatever shovel it needs so I was curious how do they actually prompted to create this code that seems kind of advanced it seems ahead of its time a little bit they actually did not explain it here in the blog post but if you go and look in the paper they do have that in there so once you get past everything in the appendix you have sort of a detailed description of how they built everything so for example when they're telling E4 what it has in its inventory it has this inventory thing that kind of just outputs what they have plus the number of units of it that they have or for example what the nearby blocks are Etc so this kind of an important point to understand is that so instead of vision like a player would be able to see the stuff around it but you'd be limited by some things you couldn't see through walls Etc you can see when it's dark you also would have to recognize objects for what they are Etc and if you had the API and you're using the API you'd kind of have almost not God mode but you'd be able to see a lot more than you normally would be able to see through walls you'd be able to listen for certain events that happened that you might not have been aware of without the API and so what gpt4 has is somewhere in between those because it can't really see but it also doesn't actually have the API Vision it just has a list of things that are fed to it and those things are inventory equipment nearby blocks other blocks that you've seen recently nearby entities so whether that's a pig or something dangerous like a villager or a zombie various containers but biome so it's for example basically like are you in a desert are you by a river Etc and then the time of day that's important because Zombies come out at night of course health and hunger bars and then also sort of your 3D position in the world so in other words this is its Vision this is its vision of the world this is all the information that it has to make all the decisions and all the tasks and write all the software that then it's going to use as skills to get further ahead in the game the reason why this is important is as everybody's talking about increasing the AI models and making more advanced and all that stuff it seems like a lot of the research is showing that the existing models like gpt4 has tons of room for what we can use it for if we just hook it up to the right tools or if we find a better way of extracting certain things from it like Chain of Thought reasoning for example or tree of thoughts reasoning show that it's going to be much better certain tasks like 10 20 times better not because we made the actual model better not because we fine-tuned it or increase number of parameters or better data or anything like that as we were able to ask it better questions which you know how in that book Hitchhiker's Guide to the Galaxy or the world or whatever they built this super computer to answer the greatest life's questions the answer to life the universe and everything and then like a million years later it goes the answer is 42 and they're all kind of angry at it because like what does that mean that kind of responds to them well that's the answer but maybe you need to understand what is the actual question this is kind of similar to it because they're working on creating Vision which is important but as we can see here just a little bit of this information fed into gpt4 kind of almost substitutes Vision in a lot of different ways it's able to reason about all this stuff without necessarily being able to see it so this is the full prompt that they use that they go into gpt4 so basically chat GPT they probably use an API but this is the full prompt that they tell it we'll come back to this in just a second but keep in mind it's sort of like the big goal that we're giving it it's not you know fine x amount of sticks or build a portal they're saying your ultimate goal is to discover as many diverse things as possible accomplish as many diverse tasks as possible and become the best Minecraft player in the world so basically the goal is for it to go out there and do as many things as it can to sort of discover and unlock as many things as it can so we'll come back to that to the full prompt in just a second but let's take a look at how we got it to write code to improve its own abilities to accomplish its own tasks in the world so remember everything gets uploaded to the skill Library so the skill library is basically a collection of all the functions of all the code that it created for itself to be able to interact with the world now it doesn't start with zero it has there's a bunch of stuff in there that's already there's stuff in there that's already created with the Mind flare API so for example there's things like Mind Block so for example this command here would mind collect the specified number of blocks within a 32 block distance there's you know craft item Place items somewhere you can smelt items then there's killing various mobs so that's like the the animals and bad guys in the game Etc but most of these are gonna be pretty simple it's very similar to what a player would do so instead of where a player would click on something or look at something and click a button this just allows them to just execute that thing so that might allow it to do certain things faster than a player could probably I would guess but I don't think it gives it sort of any power that it's greater than a normal player using you know a keyboard and mouse would and also they mentioned here they use Chain of Thought prompting so if some code fails they first ask it to reason why the code from the last round failed so meeting if I tried a certain code and then it came the the Mind flare API threw back an error message then we tell gpt4 about the error message and we say why did that happen so then it thinks about and then it retries to create the code then give step-by-step plans to finish the task and finally generate the code so here's 8.4.2 the full prompt for full system prompt for code generation this was one of the interesting Parts I think so we're telling Chad GPT this or gpt4 you are a hub assistant that writes mind Flair JavaScript code to complete any Minecraft tasks specified by me here are some useful programs written with the Mind flare apis and so here just gives it a bunch of sort of examples and feeds data into it so you know you can say you can explore until you find a pig or you can mine three Cobblestone or place a crafting table near the player check the items inside the chest at a certain coordinates Etc and they also give some helpful comments about when to run the code how to do it better Etc then you have certain goals like moving directly adjacent to a certain block so for example when you're fishing farming filling buckets and beds this is all fed into gpt4 so we're telling hey you know when you're trying to fish for something for example this is the way to do it you want to have your in-game character get next to the block get adjacent to it and then execute XYZ so then it knows okay so I gotta use this function to get them there then another function to start the process of of doing whatever right because gpt4 in itself might not automatically know how to do any of those things it's not trained on Minecraft it hasn't specifically been trained or fine-tuned to run Minecraft so it's reading the stuff and it's learning from it anyways and there's there's pages and pages of this that gets fed into gpt4 so that it knows how to do it there's tons of comments as well as code Etc all right and then we explain to it okay so at each round of conversation I'll give you code from the last round so whatever code you made I'll tell you whether past or failed and then also give you sort of all the information that we talked about all the vision things that we're seeing so we're seeing the biome the time nearby blocks who's next to us Health water Etc Health Food Etc what equipment you have what inventory you have all that stuff and then we're asking you to respond to that so first of all are there any steps missing your plan why does the code not complete the task so that's if something failed why did it fail you know you wrote some code so that you would catch some fish but it failed why did your code to catch fish failed what does the chat log and execution error imply every once in a while I see comments on YouTube videos or Reddit or Twitter or whatever people will say something along the lines of yeah but all these sort of chat Bots that they're putting out there they're not actually artificial intelligence they're sort of just these fancy autocompletes they have no reasoning ability I mean I'd love to know what they think about something like this where we're an AI gbt4 writes code about how to interact with a game that it never saw and if it fails at writing perfect code you can ask it what went wrong what did you get wrong here's here's what the output was what what didn't work and then tell it you know what try it again and then it comes up with a working code I feel like whatever definition you use for reasoning or intelligence is becoming harder and harder to create a definition of it that would include human beings and would leave something like gpt4 out of it like there's obviously an overlap and then so it says plan how do you complete the task step by step you should pay attention to inventory since it tells you what you have the task completeness check is also based on your final inventory code so and then you you write the function taking the bot as the only argument which I assume that means don't use god mode to like just create the items in your inventory actually use your body your embodied agent to run around and do the stuff that you're trying to do it's funny because somewhere in these instructions they actually say don't cheat if they tell it don't cheat just only do what you're allowed to do don't cheat to reuse the above useful programs as much as possible and then specify so use this to collect blocks don't use the Dig function directly so I think it's basically telling you to use certain higher level function instead of like the more crude lower level stuff and three your functions will be reused for building more complex functions therefore you should make it generic and reusable also remember that your inventory changes over time so always check what inventory you have Etc and so a lot of this other stuff is sort of like best practices and also it's trying to make sure that the stuff that it outputs can be used as kind of its own units its own skill set that they can save and put it away and then when it tries to retrieve it everything it needs is there so it's trying to say like complete everything to where like if we save it in a folder later once you take it out everything's there and then last one name your function in a meaningful way so that you can infer the task for the name so meaning name it something meaning name it something that we know what it does that you can guess at what it does and then you should only respond in a format as described below and they give the response format where it explains its plan as well as the code and it has another prompt for generating function descriptions I think they use GPT 3.5 for this one but basically this is what they're going to use to describe what this thing does before they file it away in the skill library and so for example if the function is mine Cobblestone that it describes this function is about mining eight cobblestones using a wooden pickaxe first check if a wooden pickaxe is in the inventory and then goes through all the steps that it does and all the thinking that it does all the checks that it has to do and then through the actual action of performing a task and so if we go back to the full prompt so we already talked about a lot of this so what the ultimate goal is and then what information we give it so that it knows where we are what's around it Etc it's Vision basically and then you must follow the following criteria you should act as a mentor and guide me to the next task based on my current learning progress please be very specific about what resources I need to collect what I need to craft or what mobs I need to kill your next task should not be and then says so simply State the next task like craft five items the next task should not be too hard since I may not have the necessary resources or have learned enough skills to complete it yet the next task should be novel and interesting I should look for rare resources upgrade my equipment and tools using better materials and discover new things I should not be doing the same thing over and over again I may need sometimes to repeat some tasks if I need to collect more resources to complete more difficult tasks only repeat tasks if necessary and so the next few things are basically trying to get it to go and explore and do new stuff so a lot of this is just to keep it going and see how how fast it can advance and since there's some limitations it's saying don't talk about like placing torches or or digging holes or Etc since they require visual confirmation from the screen so avoid that so an example of how it would respond would be so first it spits out the reasoning the inventory is empty now chop down a tree to get some wood and gives you the task obtain a wood log so jumping back to this thing so as you can see here the the Voyager is by far the best fastest more advanced thing as it progresses through the tech tree the Voyager without skill Library as you can see here not only is it slower but as they mentioned in the paper it plateaus somewhere here so the the learning stops or the progress stops it doesn't keep getting better where where this thing certainly seems like it keeps going and keeps going up and up and up as you can see here it's much better than the sort of the competition so here's that diagram again so as you can see now I think it makes a lot more sense so it's doing all these tasks in the game is creating code to create you know new skills for itself right it's testing it it's getting back feedback it's redoing it and then once it is able to make it function then it sort of adds that skill to the skill library and then later it's able to retrieve it and use it in the real world and so then it gets into some of the results so obviously the Voyager is doing a lot better A lot faster and a lot of this stuff I don't think it really matters how awesome the results were I mean it's great for to have a benchmark for future experiments but the fact how they were able to use gpt4 to sort of for its application here is kind of mind-blowing also you can see that Voyager travels a lot further it's not as lazy as the other ones and then the next thing is efficient zero shot generalization of two unseen tasks so zero shot meaning that it's not seeing examples of what it's trying to do it's the question is basically if you figured out how to make a wooden ax and then you figured out how to make a stone ax and then next you you don't know how to make a diamond ax can you figure it out based on your previous learning are you able to generalize that knowledge enough to where you can sort of apply to unseen task but you you haven't done before and interestingly enough and this is where gpt4 comes in that's why I think it's such a powerful thing because it is able to in fact generalize certain knowledge and figure out how to do stuff that it hasn't seen before we clear the agent's memory reset it to a newly instantiate instance in state instantiated world and set it with unseen tasks in the table above so the uh the fewer the iterations the more efficient the method what's interesting to note is that our skill Library constructed from lifelong learning not only enhances voyagers performance so that's this this AI agent but also gives a boost to Auto GPT so that's that's another thing that's outside of what they're doing out of GPT is its own thing so this demonstrates that the skill Library serves as a versatile tool that can be readily employed by other methods effectively acting as a plug-and-plate asset to enhance performance and then the one of the authors of this paper actually talks about they asked them if GPT 3.5 can work as well he's saying no there's a pretty massive performance drop off so here's that team again that was behind it this is Dr Jim fan we're gonna jump back to see his uh post on Twitter in just a second and this is the Mind Dojo building open-ended embodied agents with internet scale knowledge and they were the uh 2022 outstanding paper Award winners it seems so Minecraft has more than 100 million active players who have collectively generated an enormous wealth of data Mendoza features a massive data database collected automatically from the internet yeah agents you can learn from this Treasure Trove of knowledge to harvest actionable insights acquire diverse skills develop complex strategies and discover interesting objectives to pursue all our databases are open access and available to download today talk about how they have 2.2 billion words in English transcripts from the various videos that are online thousands of Wiki Pages tons of Reddit posts comments Etc I think they make a really good case for the fact that Minecraft is an excellent way it's an excellent game to train these AI agents just just because of the vast amounts of data that is already generated and available for use so back to the Twitter post that started this haul so Minecraft is by far the best test bed for it with endless possibilities for agents and he talks about Voyager and the three components that we talk about how to use the mine flare API to create these skills and now as I'm wrapping up this video I just realized I had flux on so the monitor was dimming the entire time so that's not you losing blood flow to your head that's me not setting up the right recording environment I apologize but what do you think about this study do you agree with me that it's pretty insane how far gpt4 just by itself can go just as a base layer how far we can pull it up and what kind of applications it can have do you think this is an example of AI writing its own code and improving itself I know it's not improving gpt4 itself but it is improving its its own ability its own sort of Avatar in the my craft world that's going about doing its business it's improving that character's ability to do all the stuff in the world and also does it is it beginning to feel that code is almost like this intermediary step that will no longer be required pretty soon it seems like we tell gbt4 what to do it writes the code but it also has to like translate it in order to describe it so it's almost like We're translating from one language to another but since we're directly communicating with the computer with just natural language does having to write code almost makes it more difficult can we just go straight to describing what we want to see in natural language and then having gbt4 write the code without us even seeing it how far are we away from that how archaic are code comments gonna seem in five to ten years somebody in response said Matrix which is really kind of a good point because Neo that was his whole thing he was able to see the the code of the Matrix and then do stuff within that world without actually necessarily seeing it of its Vision so gpc4 Works kind of like that it's sort of picking up on these text things and then it's able to manipulate them in the environment all right let me know if I missed anything what do you think is this as mind-blowing as I thought it was or is it underwhelming don't forget you do get a permanent buff to intelligence when you subscribe to this channel so don't miss out my name is Wes Rob thank you for watching foreign foreign