Anthropic Computer Use - Hands On Tutorial

Key Takeaways

okay since I recorded the first video about the computer use on anthropic they've now released code and details of how to do that and in this video I'm going to show you just a quick walk through of how you can get this set up a little bit about how it works stuff like that as we go through it so the docs are up for computer use you can see there's quite a strong warning and I will suggest at the start probably best not to run this like on your main computer unless you're going to run a containerized version which is what I'm going to show you in here but there's some good things in here about how it basically works and stuff like that you can see we're just instantiating the anthropic client just like normal but one of the key things that we're doing is we're passing in these tools for a computer with a display size so that it can scale where to click and stuff like that we've got a text editor and we' also got bash that we can use so these are tools that they've created to be able to work with this and we can see that they've got a nice explanation of how the computer use Works in here that Claude will decide which tool to use and then give a response back and then you'll basically use that tool on your virtual machine to get the results out now the way it works is it's doing screenshots to see what's on the screen and doing things like that if you watch the agent s video one of the things that it does seem that they're lacking is they're not using any of the accessibility features or anything they're just using display features or just using screenshots in this case so like I mentioned before they're heavily on you know emphasizing that you probably should go with a containerized version of this be careful if you're going to run it on your local machine I don't suggest you start out doing like that I could imagine that you will end up customizing containers to have exactly what you want running in them and then be able to run it at as you go through that so they've got a nice repo that they've given here with the the demo code and stuff like that so we can come in we can see the whole idea of using it as a loop an agentic kind of loop is key in that you want it to basically get a screenshot send to the model get an updated action for the tool Etc take that action then get a new screenshot and that requires a loop going on in here if we jump into this we can also see that they've got a very specific system prompt going on in here as well so you could I I guess run this on sort of virtual Windows machines in the cloud and in that case you'd want to change the system prompt and the other thing too is you can change the system prompt a little bit to customize it to the particular tasks that you want to use this for if we come in here and look at the tools we can actually see that it can take actions of key type Mouse move left click left click drag right click middle click double cck click there's a whole bunch of things in there which we would expect to see and they've got some logic in there for dealing with different screen sizes Etc now coming down to show you how I'm going to use it in here again we've got more cautions about using a dedicated machine using something like this we're going to basically run it with this Docker container so the cool thing is they've made a Docker container where you can just take your anthropic key put it in there and then run this Docker container and it will access it for you you can also do this on bedrock and you can also do this on gcp or on vertex AI in here and if you are doing it on those you will basically want to connect in to access this because we're doing it locally I'm just going to use Local Host 880 in here okay so I've copied over you know that code for setting up with Docker in here and you'll be able to see that okay it it I'm going to basically start off by setting the key so I could run this as a file but I'm just going to run it in here um I'm going to kill the key obviously after the video just showing you that okay first off you just want to you know export to set up your key for this and then you just very simply want to run the docker container now I've already run it so it won't need to be pulled down again if you're running it the first time it will will actually pull down all of the docker containers and the files needed Etc to to do this here we can see it's just basically starting up and going if I come in here and look at my Docker desktop you can see that I'm using a laptop at the moment so I didn't have Docker on this I've just installed it one of the easiest ways to do that is just to go to Docker desktop it's shows me that I've basically brought in these images already and I've just started that container so this is the container all up and running for us to use now to use it I basically just launch Local Host 8080 and sure enough you'll see that now we've got the clawed interface set up here and I can come in and type different things okay so you can see I can do a search for something like find me the docs for anthropic computer use let's see if it will do this so it's going to start off by taking screenshot the screenshot that we've got nothing open on here it will then decide to do something and it will do you know a few different things okay it's opened up Firefox in this case now one of the places that it does seem to get stuck occasionally is if you've got popups that it doesn't know about I'm currently traveling in Spain so some of the things come up in Spanish and that seems to perhaps confuse it a little bit okay so in this case it's basically hasn't found what we wanted let's see how it recovers from the 404 I guess I should point out that at this point that I also find it a pain to often find the anthropic docks I think they're not under there's no link on the homepage to find them easily ah now it's gone to to docs. anthropic okay so you can see that it got to the page I found the docs didn't find the computer use docks and then finally we got rate limited so it actually had to stop working because of that just quickly to show you in here you have you your your setup so you can change your anthropic key Etc you can allow it to send number of images or you can have a custom system prompt Etc as you go through this and can reset it to this open up bash and see how big is my hard drive okay so now I'm going to try to do something that's not a browser element and by the way you up here here you can basically toggle things on so you could actually set up some of the things on the computer and stuff okay now I'm asking it open gedit which is the text editor here and write me a sonnet about anthropics goals so let's see how okay so it's opened up the app uh that we can see here it's taking a screenshot and we can see sure enough there it's writing out our little Sonet and we've got a rate limit again there uh stopping us but let's see please fix the indenting so that each line of the Sonet has a new line okay we can see that that didn't work out very well but it was able to open this was able to save it you can see that okay it's decided to call it the anthropic sonnet and save it in there so it's pretty interesting to look at how could we use this for various different tasks both running sort of a local container on our machine but also running a container in the cloud that you could imagine has access to other coded agents and that it just pings the anthropic one every now and then to make some decisions about things as well all right so I'll leave it there for the video I'll put the links below so that you can get you know started with this I may do a custom video on the patreon about how to customize the docker container and look at different things you could do with that as always if you've got any questions or comments please put them in the comments below if you found the video useful please click like And subscribe and I will talk to you in the next video bye for now