AI Models Run Shell Commands | LLM Tools

Key Takeaways

Hey guys, welcome back to the channel. In this video, I'll show you how you can have OpenAI run shell commands on your terminal. So, if you have queries like, "What's my IP address? What version of Python and Node do I have? Show me the running processes on my laptop," OpenAI should be able to answer those even though it does not have direct access to your local machine. As usual, we're going to do a quick demo over here. Then we're going to go over the theory. And finally, we're going to do a deep dive into the code. I'm also going to have both the code and the diagram linked in the description below if you want to check them out. With that said, let's get started with the demo. So, we're going to try some of the queries suggested here. So, let's start with what version of Python do I have. Okay. So now you're going to see we get a prompt from from our back end that the model wants to run one function call which is going to be the command here and oh actually it wants two function call one is python version another is uh py version so let's give it permission and now after it gets the output it tells us that the python version is 3.13. Let's try another one. So, we're going to kill it. Run it up again. And this time, let's say, what's my current directory? Okay, what's my current directory? We're going to give it a second. The model wants us to run uh this command pwd. Uh we're going to give it permission. and it correctly got the answer. So if you do a pwd, you're going to see whatever we have here is exactly what the model told us right here. Okay, let's do one more example and then we can jump into the theory. Okay, so uh let's see maybe we can do the let's do let's do something similar to what we did last time. Uh actually let's do this. Uh show me show me the process that's taking the most amount of RAM slashmemory. All right. So the model wants us to run uh what shall call? It wants us to run uh it wants us to run these two commands. So we're going to give it permission. We're going to give it a second. And then finally you can see that the model thinks that this is the process that's using the most amount of RAM and it also gives me the process ID here and how much RAM it is using. Uh and it actually knows that this refers to the VS code renderer uh which is the top RAM consumer which makes sense. Typically it is docker but I have docker turned off right now so it makes sense that VS code is the highest memory consumer. So this is how it works. Usually without any custom code, if you were to directly type into chat GPT any of these questions, of course you won't get a real answer cuz your model does not have access to uh your local machine. But through uh LLM tools uh you can give access to the model to request running shell commands which you execute and give the response to the model for the model to be able to answer your original question. Let's see how exactly. Uh all right so we're going to go over this. The process starts from our server. So on our server the user has a question or query in this case what version of node do I have. So we pass this to the model along with a default or a built-in tool that openai provides called shell. If we take a look at the code this is what happens. We have the model GBT 5.1 because the shell tool can only be used with models starting 5.1 and up. So we have the model the query if it's a chain uh if it's a conversation chain we need to give it the previous response ID uh just some system instruction that says that uh the the environment is in a MacBook and we pass it the tool. Usually uh you can define your own tools and I think in one or two videos before I showed you how you can pass in your own functions as tools to the LLM but the in this case we uh have a default uh shell tool that openai provides us. So we just tell it to use that tool. Going back to the diagram we give it the query and the tool. Now what openai does is it goes through the query. It knows it has access to the shell tool. So the first thing it does is looking at the query it decides whether it needs to invoke shell command or not. Most of the queries we looked at it needed to invoke shell commands. So when it needs to invoke one of these shell command in its response it sends us something like this. It gives us a type of shell call and then call ID which is a random number. The response can can contain multiple commands within one shell call. If you looked over here, we found one shell call, but within the shell call, we were told to execute two commands. So, it sends that response back to our server. At this point, our server received a response from OpenAI where there's a type property set to shell call and a call ID associated with that. Now in the server, what we do is that first we detect if the model wants us to run initial command and then we aggregate all the commands. Right? In this case, it needed us to run two commands. So we aggregated both the commands so that we can run them either sequentially or in parallel. So we aggregate them. We uh execute the shell command. And then once we execute the shell command, of course we have the output. We have access to the output. And with that output, we build a response where the type needs to be shell call output. and it needs to have a call ID which is the same as the initial call ID that we received from OpenAI. This allows OpenAI to connect this particular shell call to the output that we send back to it later on. Okay. So once we generate the response, we go ahead and call OpenAI server once again. Now in this call, OpenAI now knows what were the original shell commands and what are the associated outputs of the commands. With that information, it can generate the final output. Or if it thinks that, oh no, I don't have all the information. I need to run two more commands. the same process loops all over again until we get that final answer back to our server. Now, with that information, we're going to go back to a demo and then I'm going to dive into the code. So, let's say we have something like what node version am I running? So uh the query once we sent it to the model we received one shell call event from the model and in it it told us to run this command which is just a node uh dash version. We're going to give it permission and uh we executed the command. We sent the output back to the model and the model told us that we're running 23.11. So if we maybe just exit out of it and do a note version, you can see we are indeed running 23.11. Okay, so that's how it works. Uh there you go. So we start with the model or we start with the server. We send the query and the tool to the model. The model determine uh tries to determine if uh it needs to invoke shell command. If it does, it sends us a response with type set to shell call. Our server recognizes that it needs to execute a shell command. It goes ahead and executes the command, builds a response where it makes sure that the type is set to shell call output and it has the exact same call ID. It makes another call to OpenAI with the response of the shell command. OpenAI receives the output and then it generates its final result to the original user query. All right, so now finally let's take a look at the code. I'm also going to have the code and the flowchart linked in the description below if you want to download them or check them out. But let's look at the code now where we have everything starts with the API call here uh which is just using the responses API to make a call where we passed a user query and the tool. Now this is where everything starts. We parse the response and then try to see if there is an item called shell call in the response. whether there's one, whether there's multiple in most of the cases we saw, there was only one shell call event even though there were multiple commands within that one uh shell call. So this is the first thing we do. We start uh we we make a list of all the shell calls that is coming from the model. Now this is where we need to execute the shell call, right? Uh so it says how many we found. we have a list of all the output. As I said, there can be multiple uh commands. So, we want to make sure when we send the response to the API, we're running each and every uh every command and then sending all the responses back to the model. So, for shell call and the calls, we have the call ID. We have the action and command. These are essentially the the commands we have to run. And we finally ask the user for permission. Uh where is it? Yes. So if the user does not have permission, we do some stuff. But normally we where is it here? So command results, right? And then for command results, we go ahead execute the command and we add the output. The output is the uh the output error and the outcome. Okay. And finally we have our dictionary here where we have the shell call output. This is what I showed you before. Uh the call ID and the command results which is the result of the individual calls or individual commands we ran. Finally with everything over here, we send the output back to the model and we again give it access to the uh shell tool just in case it needs us to make followup uh run follow-up commands and finally we return the response back to the user. So that's how it works. I'm going to have both the uh both the code and the diagram linked in the description below if you want to check them out. If you have any questions, leave them in the comments below. Otherwise, I will catch youall in the next one. Take care.