Connecting LLMs to tools

Key Takeaways

if you've been following the AI space you probably know that large language models are helping developers build all sorts of neat and novel applications that's because LMS are excellent at processing and producing nuanced fluid language but llms have some limitations for one out of the box they're not connected to the external world when responding to user queries they don't have access to knowledge outside of their training data this means they can't return real-time information about things like weather or traffic or the news and they also can't take actions like booking flights sending messages or writing information to a database in other words if we want llms to be really useful they need to be able to interact with the outside world when we ask an llm what the weather is at Austin tomorrow we wanted to pull this information from a weather API and if we wanted to do something more complicated like using LM as a helpful assistant that can help us book a trip at some point we're going to have to call some apis to put flights and make hotel reservations so how exactly do we teach llms to use external tools like apis let's take a look let's say we want to build a chatbot that has access to a weather API so that we can ask about the weather and it can give us updates in real time for example what's the weather in Austin how would we architect an app like this well the first step is to write a prompt to get the llm to specify when a call to the weather API is needed and second we need to write the application code to actually go out and execute that API call finally we need to feed the API output back into the llm so that it can produce a response for the user let's break down each of these three steps as usual with LMS the first thing that we want to do is to design a prompt and as a reminder a prompt is the text that we feed into an llm to get it to take on a particular Behavior in this case we want to write a prompt that allows the model to indicate what an API call is necessary here's one way we can structure that prompt we might start with an instruction that tells the llm what our objective is like your objective is to complete the user's tasks then we describe the tools the llm can use to complete the user request in this case the weather API for example you have access to the following tool followed by an example of what using the tool command looks like what we're doing here is simply telling the llm what tools it has access to in plain old English since llms are capable of processing nuanced natural language this is actually often enough to get the llm to take on the behavior that we want however sometimes it's also helpful to add some illustrative examples to our prompt showing the llm that we want Its Behavior to be one example input could be what is the weather in New York tomorrow and the example response we'd want is the command weather and the City New York we passed this prompt into an llm with a new user request like what's the weather in Austin tomorrow the llm should output something like this which is what we want what we've done here is design a prompt that can get an llm to Output tool commands when they're required to complete the user's request however just to be clear the LM isn't actually calling any of these apis it's just producing text that indicates that we the programmer should go out and actually make that API call and that's something we have to design at our application code so let's see what that might look like first we need to detect when the model outputs a tool command in our case we prompting the model to Output the tool command in these curly brackets so we just need to do a string search to identify when a command has been issued then we have to actually go out and execute this command to do that we'll take this string weather Austin and we'll extract the parameters we need to make the API call how we do this depends on the API and how complex it is here we just have to identify the target City Austin we write code to call the API and then we get back some response which is typically in the Json format okay let's recap we got the nlm to Output when an API command is needed to complete the user's query then our application executed the API command let's suppose the response into this Json object which contains the weather in Austin now we could just return the space on blob back to the user technically it answers the question but most people don't want to read Json so instead let's use the llm to take this info and produce a more conversational response we can accomplish this by writing a brand new prompt that shows the llm how to issue commands and how to rephrase the API output into something more conversational we'll rewrite our original prompt do just that here's the prompt we started with remember it included an example of the user asking for the weather in New York let's extend that example by showing the model how to reframe the API output first we add a new field to The Prompt where we append the Json response note that we're not gonna ever actually show the end user this Json but it will be part of the model prompt finally we add an example of how the model should rephrase the Json into something that sounds conversational like setup Now using this prompt when we ask what's the weather in Austin the bot responds starts with a command then we go out and we execute the API commanded code and append the response to the prompt then we call the Alum a second time and this time it takes an API response and voila our end user gets their question answered with real-time accurate weather this is one basic way to design an llm application that can interact with just one tool but how would you extend this to multiple Tools in addition to listing out all the tool commands that we want our analog to have access to we might need a more sophisticated prompt maybe one that even includes logic about when to use each tool one reason prompting technique known as react is a way of writing columns that combines Keen of thought prompting with taking actions as a reminder in a cloud prompting is a technique where we nudge the llm to explain steps of reasoning to arrive and answered the react technique can be especially useful if a single user query requires multiple steps and maybe even multiple tools to complete designing prompts to teach llms how to use tools is an active and developing area of research if you're curious to learn more about different prompting methods we've put some links in the description below that's the basics of connecting large language models to external tools let us know in the comments what tools you're using with llms thank you