AI Agents explained: An Introduction

Key Takeaways

ah welcome Unity today we talk about AI agents and how they interplay with AI models and tools and how we can code agents now you know hn can be ruled based agent autonomous agent llm augmented agent llms agent tool agents and multi-agent system and with our models we have a chronological view we have a large language model our vision language model and our original language acts and Robotics model so how they interplay how we can code agents in the most efficient way at first we have to understand a little bit why we needed agents in the first place so here we go and the good old times of GPT sui no not activity GPT 3 we had an llm that was only living here within gpd3 and had no contacts to the outside world could not search the internet nothing we needed some rule-based agents the very first form this is an agent to connect to the outside world for example to access the weather info API and a human wrote rules for this agent rule-based agents and if we talk about tools this API had access to the data of a thermometer of a hygrometer of a barometer this was the good old times now getting the data accumulating the data what we can do when we have the data a lot of the data a Time series of the data we want to find a pattern in this particular data set this is why we have an AI system in llm so the question became can our llm after some recording of the time periods predict the weather given specific data sets from thermal hardware and barometer so suddenly we said okay what we need an agent needs memory we have to record the past the past action to learn from the data we have to have enough data a huge data set so that our llm can be fine-tuned on this data and of course we mounted that we moved from a rule-based agent system to a little bit more autonomous agent where we have maybe additional external data so first step was our agents get some additional memory now memory is easy we have in context learning ICL and if we have here huge data we can have access to a database or some recording from the past 100 years so we can start our learning to predict here the weather so now we have the memory we have all the history of the data and now our agent needs the ability to learn to analyze a given task may be subdivide the task in subtask and react so memory and the ability to learn a two significant elements of our agent so great the ability to learn well an agent connects back here to the linguistic brain to the large language model because the large language model even in the old times had a minimum ability to do some very basic reasoning so the ability to reason at first you have to do the step one then we do step two and you get it this is done here in our llm system so the agent itself in the context here of our API based or API call based agent it needed access here to the Central Intelligence here the linguistic brain in our LM so we see now how AI models and agent and tools interact so here we have still the case that the human wrote the rules for the agent but if we have or if you want an autonomous agent that responds here we need maybe that an AI writes the rules for this agent give you an example now that we switch up to GPT 3.5 turbo to our chat GPT llm we fine-tune for example a base llm on 16 000 specific apis the way of a directory of apis where database of 16 20 000 apis and this fine-tuned llm we call an API llm or in the literature we'll find it as tool Lama or tool llm great if you want to see the details about this this is the video for you or we say hey we find you on a base llm on all hacking face apis this fine tune model is called gorilla this is an llm to find the best apis to train and to fine-tune artificial intelligence system on hugging phase there are hundreds of thousands of model and data set and possibility and this system this gorilla llm will help you find the correct API in hacking phase given your natural language English prompt for the task that you want the system to perform if you want to find out how to code gorilla and see a demonstration this is the video for you and you may say okay but wait a minute agent I know software agents autonomous agent multi-agent system economic agents intelligent agent environmental interaction with Smart Homes learning agents security agents and and you're right because now we're moving on in our Evolution and the science and the development of agents and now we move from the pure language based models to the vision language model now suddenly we have not only a weather sensor where we get one numerical data back but we have certainly a stream of images from a visual sensor a lidar sensor a radar sensor an infrared sensor an UltraViolet sensor or whatever sensor we have so suddenly we have example the task to identify the object in real time given the images that we have from our Dynamic changing environment and then we have here to find a response by the autonomous agent how to do his job so the agent is given a specific task by the user the user has a query that's why we need an agent to perform here a query I ask gpd4 something but if we have a dynamic environment the agent itself does not know what to do given that we have an evolving Dynamic environment so it has sensors it has a stream of images that it can analyze so the agents needs definitely the intelligence let's say the training of an llm and now if you go to the llm we say here that we have division language model so we have here a space where we construct our Vector space our mathematical space for vision and our space for language where these things happen so we can concatenate this this two mathematical spaces and we can have operations within the space so it is now here the vlm yeah just a second that makes here a decision how to proceed given the task that your autonomous agent got from the user this vlm decides how to go on and the steps for the agent to perform and it says Hey agent listen I analyze the situation I looked at the sensor data I looked at the image I identified all objects I understand the situation and give my training as a vlm system I can tell you I was trained to the best solution to perform the job is to do step one step two step three so now the autonomous agent performs the step the vlm has decided upon given the real-time data and the Sansa provide also some feedback on the success of the failure of whatever the agent did you remember we I have also a video about reinforcement learning might be beneficial to have a look if you're not familiar with the term so isn't this great so give you a very easy example we have a video surveillance of a room in a museum for example and you want to have a task to detect fire and if you have some fire you can send an email or you send an alarm to some human operator or whatever so you have real-time images of the room and now you have different scenarios there is a scenario one where only five people are in the room and you have a very small fire or you have a scenario too where you have 100 people trapped in a small Roman you have a huge fire the response of the system given to training data of the original language model will be different depending on the situation that the vlm system receives and analyzes given all its data so you see here we need an autonomous agent that has learned during its training phase as a vision language model what to do if certain scenarios happen this is all the wisdom of the training data wait and now of course you know it now we go to the full interaction division language action new robotics model a real a model now we only have not only have our sensor data our image data but we also have the possibility to have here actuator robotics robotic actuator that you have the robot moving in the real world so what we have we have our Vision Vector space we have our language Vector space and we have our action vectors pasted the actions the robot can do with the robotic system or whatever we have can do and now now is the interesting time here to understand how you can train a system that has three rather separate Vector spaces and how you can start the training of a vla based Transformer system because we are operating here with robotic Transformer with vision Transformer and language Transformer how you can train the system how you can find you in the system and the answer you find here in this video about AI robotics 3 or T2 this system if it is trained if it is fine-tuned now makes a decision how to proceed and how to move the robot in the real world or perform the task that the user asks here an autonomous agent the robot to perform isn't that fantastic so now we have a little bit more pronounced understanding what are agents we have llm augmented autonomous agent where the core of this agent we have simply a large language model we have a real am augmented autonomous agent if you have camera sensors or if you go full-fledged we have a vla augmented autonomous agent and this is here based on the model that we use and the ability of solving complex tasks and interacting with the environment and now you of course say Okay so the next step if I know about agents what if I have not just one agent but I have multiple agents should I have one superhero agent or should I have specialized expert agents do I need a coordination layer how do I code this agents to perform the task can I use here for example kind of a prompt engineering to tell here a relay system the task that I wanted a system performs and the system itself decides on the step to go through from Step A B C to finally be able to have a successful task completion and you know that I already have a video here on interactive agents so I give you here one example but what I would like to do in my next video is exactly talk about here a complete Theory if we decide we go with one agent how we code it how we Define a prompt system for this specific code or if we use multiple agent how we choose the interaction pattern between the agents and the Central Intelligence that we need to coordinate the agent if something happened or if we need maybe another agent within here this portfolio of agent but this will be the topic of one of my next video I say thank you I hope it was informative would be great to see you again