Coding a Vision ChatGPT that plays Poker Autonomously! (GPT-4V Python tutorial)

Key Takeaways

in this video we will see how to create a pokerbot that uses GPD forb the New Vision model from open AI to play automatically to pocker we'll be using poker th in this case this open source desktop game to play poker but this could be used for other poker games or other applications you will find the blog of this video with all the code all the links and all the information in the description so go in the description to find the medium blog and you will be able to copy and paste all the code and it will be easier to follow along so we could also play with gp4 in Pocket sending at every turn which is our card which are the cards in the table and so but it's long to put this by hand with Tex and also we have to put not only our cards and table cards but also it's important to know how many money we have what are the the bets of all the other players their amount of money if it's available like in this case this is a lot of information to put by by hand with text and it's better if we can have this automatically so this new vision models allow us to to send the full screenshot of the game and make faster the all the information of the game to the model with the image not with all the words because it's better one image than a th words as they say so now we are able to do this automatically so much faster so in a real case because this is a simulation in our computer against B but in a real game we have a limited time to to take decisions for our turn so it's better if we can send all the board information in a single image and get a response in a few seconds will be automating this in Python using some libraries but I want to also do a disclaimer here so as I said this is a local game against as you can see other Bots not real players and this money is not real it's a simulation of the money so we are not putting here any payment method or or so and I think that it's legal to use BS to play poker online well maybe it's not always legal but can be legal sometimes but even so I don't recommend doing this in real situation because you can lose a lot of money because these agents are not perfect and they can lose so they can play poker but maybe are not good and can make um huge errors so don't put real money with these systems and uh matter if you don't apply this in real conditions this is only a demonstration and if you can learn how to do this in this poer conditions you can apply the same methodologies for other games or other more useful and better applications okay so with that being said let's see how to pram this pocker Bo that is playing for us we are this Center player and I haven't touched anything so far and it's doing calls bets and it's reasoning why it's the better move at every time so let's see how to use GPT 4B and how to build a pocket Bo with it okay so let's start by doing a basic example of how to use the new GPT 4B model from the API so here we can see the official documentation uh you can find here some description of the model how it works and how to call it in a different ways so we'll be doing a similar example of the ones that we can find here uh you will find all of these links and also the code that we'll be using in the blog that you will find in the description so here you can check more details of how the images are processed like the pixels are transforming into tokens so depending on the resolution we'll be spending more or less tokens and here are some frequently asked questions and in order to use the API you need to have an API key from open AI so you have to come to platform .op a.com and create here an account if you don't have it and then go to API keys and then fulfill the requirements you have to put a way of payment and validate your account so then you'll be able to create here a new secret key and get the the full value of this key and you'll be able to use it to call the API later in the in the code okay so now we'll go to our Visual Studio code editor or your favorite code editor and we'll create a folder for our project this is mine GPT forb pokerbot and now I have to create an images folder I will call it like this and to start I will put here an example image I will use this image as an example you can use this one or any other it's just for demonstration purposes to to check that it works okay so this is my example image I have it here and let's create a notebook to develop our pokerbot let's call it pokerbot Dev because here we'll be developing step by step our logic and the examp examples so we'll start by importing open AI from open Ai and base 64 and in order to use open AI uh we'll have to install it so we can open the the terminal so before installing the python libraries we will create the viral environment this is a recommendation but it's up to you so let's do python virtual environment and now we have here our virtual environment now let's activate the virtual environment like this uh in mac and Linux is different you can search it in Google how to do it and now we can do the PIP install of these libraries of here open AI NPI p g and open CV those are the four libraries that we'll be using the external libraries their ones are already installed in Python so let's install them and now we can already run this so let's start by creating here a helper function this function will be used to encode our image to Bas 64 so we can send it in the in the request so this will encode our image in text format to be passed to the API then we'll be defining the path of our image so this images folder example pi. jpg so this is the path of it and we're converting already this image from here to the basic C4 format now we can already create the client of the openi API so here you will need your API key there are different ways of dealing with the API key with the environment so you can have it for example in your windows or your Linux or Mac environment variables and you won't need to do this but if you haven't done it yet it's the easier way and not much safe if you share this code so make sure that you don't share this code publicly because here you will expose your API key but here you have to replace this string with your real API keys that normally starts with SK this Dash and whatever key you have here so this is one option but as I have my API key in my environment variables I will do another way so I will do it like client and simply calling open AI the class but without any parameter here because it will already find my API key in my environment variables okay now we have the client so we can already create the method to call it so we'll be using the chat method with completions and we'll create this request to the API we'll be using uh the GPT for vision in this case now it's in preview yet so if you come from the future maybe it's not already in preview is in version but for now we will have to put GPT for vision preview we'll be telling first of all that our request of here uh that we are the RO user and the content of our message will be first of all a prom the instructions of what it has to do so our instruction now will be pretty simple like describe what's in the image so this will be only our current instruction and here we'll send another message that will be of Type image URL and we'll send our image en code in B 64 so here in this uh variable of here we are passing the the image that we had here in our folder encoded in B 64 so it can receive it in a web in this Web format let's say okay we'll put 300 tokens as maximum of each output let's put 500 okay so now we can already check the results but first we'll have to print them so here we'll have a response that will be quite complete so let's see what's the full response now we are requesting again to the model and here in the choices we have uh our message set completion message and the content of the message is the image depicts a painting that features Escape sense blah blah blah so it's basically uh quite long response it's describing what it saw in the in the image um basic description of it so let's print only the response of it like this now if we do it again um well we'll have another response so let's let's check what's the new response because it will be slightly different probably so now we have here only the message and not the other details of the response and we have here the response message and probably it will be slightly different why because we have temperature by the fall of 7 or so I think I'm not sure what's the actual value of the Thea temperature value but it's not zero so if we want to have always the same response with the same image we need to put at least temperature at z.0 so then we'll have always the same response but for today demonstration I will put a low value because I want some consistency I don't want M temperature but a little bit maybe it will be good for today experiments so now it explain with more lines with three lines and we could try now to if we remember what was the example image it's a painting so we can ask for the for more specific things like the styling of the of the picture or how to create a painting like this what's the technique or to name similar authors that created paintings in this style so we can imagine many many related questions that will be more useful more specific to this task and we could change this task of here this prom instruction to this more specific and the model would be much much helpful but now let's go to the next step let's see how to install the poker th game that we'll be using today so in order to install Poker th the poker Texas Holdem that it's a open Source game for your local machine so it's for Android Linux uh Windows and Mac OS so it's for almost all platforms to install it you have to come to the official website and here in download you can search the installer for your platform and download it and install it so the thing here with this example is that if you know and you understand how to build this pokerbot using poker th you'll be able to to implement a similar example in other platforms or other games not only pocker but also other kinds of games so whenever you have pocket th installed you can come here to to the main menu and first you can come here to the settings and configure your poer so the initial thing it's mandatory you have to go here and this will be checked by default so make sure that this is unchecked and then optionally we can come here to Nyx and avatars and you can change here the photo the your profile picture and even upload one or leave it blank without any image and you can check here the your avatar name so then this is a little bit important because we will be telling to the vot which player are we so the name of our player and then the GPT 4B will know better which is the the player that he is playing with now we can check okay here and uh in order to get familiarized with poker th you can start a local game this is how we will be playing H it's possible also to create local like in your land uh Network local games and play with colleagues and so without here we we see some cash but it's fictitious so it's not real dollars we haven't put here any payment method or so it's only to make a simulation of a real game and we see here how is the the board maybe your windows will be larger we're playing against n other BS from player one to nine you can see here and we are the one of the middle in the bottom so we are this one of here and we can do now in this in this condition this is our our cards so we are seeing them because we did that uh option and check in the settings and now given these conditions uh we can raise call or fold let's call for now so you can try to play a little bit you know the rules of fer you can see here some information about the possible hands so the next step I recommend you to to make the window as small as possible so this is the smaller size that it allows me to do and also to place it here next to the the top right of the screen because then uh we need we need it to be always consistent at the same place with the same size because we'll be cropping the this uh part of the of our screen and we will be telling the coordinates to crop it so we need it consistently so this is how I do it I do the window as small as possible and I put it in the top right of the screen always like in the same place okay you can finish the game if you want but I will close it for now so now we already have our both ingredients for today that are the GPT 4V working from the API and also pocket th working and we understood how to use it so now we can go back to the vs code and go for the next step the third step that will be doing a manual pokerbot a basic assistant that will be basically taking a screenshots of our game board and giving us some recommendations that will be manually applying so let's see how to create this so we'll come here to the first cell the the demo of the GPD 4B we can copy this one of here paste it in this part of here and we'll be changing two things first one is this path of here the example picture we'll call it poker PNG this will be our image now for the pocker game and the second thing that we'll have to do is to create here a prom instructions so this will be our prom now as it's too large so we'll be removing the previous one and instead of putting here this long text in here we'll be putting this variable PR instructions so now all of this will be going here but here is better managed and better seen so basically now our prom instruction now will be uh more complex so we'll be telling it that it's a this is a game theory research project and you are a mathematician so we are telling this role to to the agent to GPT 4B uh that is expert in poker game theory we have to create this story and so well it's not that fake because we are doing this for research and so we are not playing the real pocker but you have to make this because otherwise we won't like to assist you in playing so because it's not allowed um open AI put so many barriers and firewalls let's say to not allow to do like bad things or gambling or this kind of unethical or dangerous things so we have to create this uh Game Theory research project environment let's say to make it respond us and help us and we'll be telling that the image that it will be receiving is a screenshot of the Texas Holden poker game running locally in a computer so this is true we're not telling any lie and well we will be telling that which options it has it can either BET Race call check or fall so these are the five options that we we can have H so usually we have three of them but this is the total five possibilities that that will be telling it that it has we are not considering the only in option because maybe it's too much dangerous let's say and uh taking to on this information make a short resoning this is important because if we don't tell it to make a short resoning it will respond probably with a long text and we have a limited time to play the poker game maybe in this local game no we can spend as much time as we want because it's local against other Bots but in real conditions we want a fast response and a fast action so we will restrict each output like this and to tell this short reasoning in a single sentence follow with a semicolon and then to to put bet raise call check or fold in capital letters and only that word so in this way it will be easier for us to par later each action let's say and keep it simple follow the instructions consistently and don't explain anything else so with all of this PR engineering it has been working quite well for me let's say so you can try to modify this but most of the times if you don't make it clear for it that it's a research project that we are not gambling for real it won't want to assist you it will tell you that um it's not able to to Gumble or to do this kind of instructions so with this new prom we'll send it here and now what we have to do is go again to poer the edge we can create a new game we have here the window as before and now in in Windows uh when it's our turn that is just now we can do windows shift s and we will be uh doing a cropping of the image I don't know if you can see it but okay I did a crop of the of this window and now we can go to paint and in paint paste the image that we have crop it and now you can go and save it in this folder here so look for your project folder and images and here you have to very important to name the image as poker. PNG that's the path that we to it and this is the path of the new image images SL poer PNG so this one of here so now we have everything ready so let's see what it thinks that we should do in this H condition now it's our turn and it says that with a we hand six and two so okay it identified properly our cards is facing uh arrays and multiple colors theal play is to minimize losses and exit the hand so it's recommending us to fall okay it's um it's a weak hand so probably the best thing to do so in this way we can see here shortly a very short reasoning uh but has been quite good and also the final action to take okay let's try again now in this new H game and I will be again copying the screenshot the part of the screenshot of the of the game I will be pasting here in top so it's a recommendation to leave paint open or the any PR that you're using for this so I will be pasting this in top okay and saving it again uh it will be saved overwriting the previous image so now I have here the updated image this game of here let's try to put this better like this okay let's run this again in these conditions now we have a j and a nine so we see here it response and it's saying that with a J and N basically it's not a bad hand so let's let's call it it says okay let's see what happens now other players are doing its movements okay and now we can see the first three cards and so now in order to play again we have to uh again drop this part of the board go again into paint paste so repeat the process save the image and run again this cell it's not that bad but we'll be automating this in a moment so it says now that with the 776 so it's been identifying pretty well the the card in the table as well with this flop okay so it says that okay it's safe to check so let's let's do it and we can keep with this process again and again and probably it will be do more or less good I think that it's not an expert but I'm neither an expert to you what is GPT for so it works at least let's say and okay let's uh close this for now I will open it again in a moment okay so now the next step will be to create some methods to later be able to automate everything so now we have to calibrate let's say a couple of things uh one is to understand what region we have to crop so we we have to open pocket th again and start the local game as before and place it uh for for me it's here in the top right as you see so I have to place it as I was doing before okay now uh I have to understand what are these pixel values the corners to crop the image and first I will import this these libraries n p GUI and map lip and um I just seen that we have to install map lip I miss it before we'll be using mly basically to display the cropet images to make sure that we are coping what we wanted okay now we can continue so we have these three libraries P gii will be used to crop the board now and later also it will allow us to click automatically and to press keyboard uh Keys automatically by the vot to make the vot able to take actions and nay will be used to crop the images that will be take from the screenshots so this initial method will allow us to see the image the grop image I will have to resize this we have to see the game board here okay so we can run these libraries and now we can reun this and I already optimized these values of here so this is screenshot height so in my entire screen uh you are not seeing now but it's larger than what you see now so H goes from 0 to 630 and the width of this window here is from 1,540 to 2,560 so these are the values of my screen but now for example if this was a wrong value you can see now it distorted so I should adapt this going like value by value until we can see in this image only the the window of the poker game and make sure that we are not losing any important information so you don't crop too much or too less we have to adjust this properly so after some iteration for me those were the proper values where I can see my window properly fitting in the image so now we are able to automatically with these commands up here to take screenshots without using paint the window shift S command and to having to save it manually and the next step will be to find another cop region that is smaller and we want to check now if it's our turn or not and you can see that whenever it's our turn this area here our player area it's lighted up like this we have here a light if we do now a race let's say now it's not lighted up so it's always lighted up for the player that it's it's turn to play so we will be cropping a small area of here and we'll be seeing if that area is is lighted or not so we'll be analyzing with NP the mean of the values of the pixels in that area to understand if it's our turn or not uh every 0.5 seconds we'll be checking if it's our turn or not and whenever again it lights up this area here uh we will know that it's our turn so we'll be able to do the full screenshot this scop of the all the board and send it to GPD 4B and wait for it response but if it's not our turn we have to wait to be our turn because otherwise the image won't have the complete information for us to play okay so to do this other crop we'll do it with a similar method so now we'll be adjusting this PL light height and play light width so we are looking for this height and width of thisal area I already set them uh those are the values for my my case having everything seted like this you will see what I mean so it's a it's a very small region in this corner here that now uh we already did it maybe I have to move my image a little bit or maybe it's better to to adjust the parameter so I want to have this lighted part of here more in the center of this small image so I have to do it maybe with 295 and 35 let's say okay now we see here the corner so now all of this is lighted up because it's our turn and also um whenever you have also the width of it so make sure that you find this small region in the corner so now uh I'm taking the crop of this screenshot this small part and doing the mean of it the the average of the values the pixel values and it's 52 I'm printing it here it's 52 in this case it's our term so now let's try to call for example and now that it's not our turn let's run it again and we see here that it's not lighted and it's 40 now the value of the average of these pixels of here because now it's it's not our term so we have to find the two values for me it's it was 52 when it was our turn and 40 when it's not our turn so we have to find for example 46 as a as a trigger as a threshold to understand if it's above or below this value to understand if it's or not our term so those are the the the two things that we have to find in this part this um this crop values these four crop values and the threshold of when it's our turn and when not let's put here in a comment that um uh 40 not hour turn and 52 hour turn so 46 will be the threshold in in my case if you find a slightly different values just add to your case uh threshold okay now we can already go to the final part so to create the the automatic bot we'll be putting this command here because we'll be developing the code in this cell of here but it will be nicer to run this from the terminal in a normal python file and this command up here will allow us to create uh from the code that we put in this cell to create a separate python file in the same folder with each content but here in this cell we have all the previous methods and it's it's good to to develop and to experiment but uh we'll create the final pob the automatic one in a python file like this so we'll be importing all of these libraries so mainly the the ones that we have been using so far with addition of CV the open CV to to save the images and time now the next step step will be to put here these four values that we have found here so I have to modify this one that I recalibrated so here you have to paste the values that you phone for the pixel coordinates for the crops and now we'll Define a function that will see if it our turn or not so basically it will be taking a screenshot in this image it will converting in nay array it will doing the small cop of The Shadow of the light part I have to rename this I put Shadow but it's a light in reality so uh so basically it will be cropping that small area that corner of our player spot and looking if the mean of that area it's above or below the the threshold value that we found here this 46 so this will be finding the threshold so if it's our turn it will return through in each my turn and otherwise it will return false so that was uh this helper function and now we can already create playing through and the infinite Loop while playing and now if is my turn so we'll be calling this function of here and it will check if it's our turn or not if it's not our turn it will go to the here to the end and it will sleep for let's say half a second otherwise if it's our turn we will start by uh taking the time the time zero so what time is now with this function of here in seconds now we'll capture the screenshot of the game board with the screenshot H and screenshot y now we'll take a screenshot of the entire screen first uh convert it to an M array and then we are able to crop it to have only this part of here the part from the window we will declare again because this is a separate code um the the image path uh so it's here images poker. PNG here we'll be saving every every screenshot and the screenshot that we took writing it here and here we we have the function that we already created before to encode to B 604 this image to be able to send it to the API now it's pretty much as before uh the same steps because now we are putting everything together to automate all the steps so now here we are encoding the image uh again getting the open AI client and remember your API key if you put it before here you can do it again as as we did before but if you have it as me in the environment variables you don't need this you can directly create the client like this now the same Pro instructions as before I haven't changed anything here we know that this works now let's put here a try and accept because it could be normally it will work but U sometimes if there is any error with the API or so this will be better to to handle the the problem so let's try now to do the request to the model as we did before as well okay so this is also the same and now instead of printing the response we will pass the response to this message now as we know that the the message will be the reasoning of the model in a sentence a semicolon the action and a semicolon we already put away the last semicolon and then we know that the action will be in the message is splitted so here the message will be the reasoning a semicolon and the action so we'll be splitting this semicolon and taking the the second part the last time of this list that will be splitting the reasoning and the action and let's strip any blank value that sometimes could be and also putting it in Upper so in in upper case just in case that the model forgot to put the letters but normally it puts all the letters in uppercase but in this way it's more consistent so it it it won't make our logic to Rak that much so now we have our action here and we told the model here in the prom that it can be a BET Race call check or fault but we don't H take this and here in the game always you can see here that F1 F2 F3 so those are the the keys of the keyboard that you can press to take these actions so you can either click in any of these three buttons but you can also go to the keyboard and press any of these three keys or F4 for all in but we are not considering doing all in so only F1 F2 F3 so if each bet or race that are the possible two cases that can go here in this uh top button uh it will be an F3 uh call or check that can be the two options that will be here is F2 and fault is the later case it will be always fault here it will be F1 if it's not any of these four cases so like this we are detecting which key the bot has to press and we will be using a p outo GUI to to do it but first we'll be printing here because we want to know how many time it took to do the API response basically then we will be printing which key we have the here and which action we detected in the message and also to print the full message to to understand better the reasoning and to detect any possible problem with the response of the model then the next thing now that we have the response we will detect again if it's our turn because it can be the case that well not in this local game that we can take as long as we want to take actions but it can be the case that it's no longer our turn for some reason so just to check again if it's our turn otherwise will not take any action but if it's our turn we can print here that we are pressing this key and then P to GUI Justus in case will go to this coordinates that for my case is around here so X is 2,200 so around here and Y is five on the top so I want to go to this H coordinates and click so I want to do this to put this window in in top because if we press F1 F2 or F3 this key here and this window is not on top we won't press it here we won't press it somewhere else so we want to click and then press the the key action the vot has decided to take given these conditions and then we will go to sleep one second just in case to make the game go around and update each state and uh if it's not our turn so in this else case we want to print that instead of pressing this key we are skipping to press this key so just to remember which was the key but we are not pressing it we are keeping to press it uh just to know that something happened and after all the API reasoning and so it was no longer our turn okay and we are this try invation of here and if something happens and this try crashes we want to print the exception that occurred to see if there is any problem in our code or in the API to see what was the problem and also the the response message if there is any because sometimes the problem could be that there has been a response message but it wasn't in this format so we didn't had any good response as we wanted and that will be uh all the code that we need so now if we run this instead of running it we have created this in a file because we had this method here if we remove this this will be running in this uh notebook cell but I prefer to run this from the terminal from this file that has been created here the poker bo. pipe so let's run now um I will have to make this smaller just in case python poker bpip and now it will start you can see here the line and it took 3 seconds uh and this was the action that it wanted to play so it detected that we have a Q and A 10 the cards and it wants to rise so it press F3 automatically and it put the mouse here first to click in this window and it press the key F3 now again it's our turn so we can see here the dash line and now again so it's playing automatically I'm not touching anything uh it's automatically playing and and we can see here in the terminal what actions what is the reasoning what is the time that is taking and sorry but oh okay it failed so normally with this prompt that I found most of the times is wanting to a but you can see here an example of what happens when you don't Achi to trick let's say the prom uh to make it clear that it's is not a real bet that we are doing experiments here without real money but this was a a real case and if something happens if our code doesn't find the action uh we are folding which is the safest action to take so if it doesn't find BET Race color check it's basically doing F1 that is folding so we are skipping to put more money into the table maybe we are losing a good hand but it's better this than betting so we are not losing more money at this we you could leave it here playing against the other BS to see if it's good so if it's better than the other BS or not but at least it's working automatically uh also you can find that uh there is a maximum of request that you can do to the API for now with vision so you could find that after 100 request to the vision API for the same day you cannot do until the next day anymore so this can be also a situation you can request for more but this is Thea behavior for now let's see in the future so this is how it works I hope that this is helpful for you and you can apply this to not only for poker but other conditions if you learn how to automate this full process or to take screenshots from any part of the computer let's say from any other program send it to gp4 Vision make it understand what the task automatically take it respon far sheet you can automate other actions so this can be very helpful not only for this pocket application but for better ones so I hope you enjoyed uh give it a like and sub subcribe and see you in the next video bye-bye