How to connect CrewAI to different LLMs (GPT4o, Groq, Llama3, Ollama) - Tutorial & LLM comparison

Key Takeaways

hi everyone and welcome back for this video I tested a hierarchical crew with different llm backends locally and in the cloud both free and paid ones I will fully walk you through how to set up the crew set up your environment and change the llm so that really everyone can follow along at the end of the video I will show you how all the llms compare in terms of quality of the results runtime and cost so you don't have to rack up a huge bill testing all of them like me spoiler alert I had some of the best results with a completely free setup the llms I tested were open AI GPT 3.5 turbo GPT 4 Turbo and the new GPT 40 I also tested Gro with meta's new llama 3 model and mixol as well as running llama 3 locally on my own computer using oama I think the results were really insightful and exciting I will first give you a quick review of all the Technologies involved then I will walk you through how to clone the crew AI examples repository and change it for our purposes I will then also do environment setup with you and install all the components needed for all the Technologies we will use and show you how to get the API Keys you need we will then run the same crew setup with different llms and in the end look at a comparison table for the different llm setups in terms of quality runtime and cost llms or large language models are pre-trained language models that allow computers to generate natural language and interact with humans in the form of chatbots like chat GPT as powerful as chatbots like chat GPT are they have severe limitations when it comes to creativity multitasking access to real-time info from the web and non language related tasks AI agents allow for a more more real-time interaction with these models and for the usage of specific tools like access to online search emails and anything else that can be automated via apis AI agent Crews like crew AI take this concept to the next level by allowing several llm powered AI agents to collaborate on a larger project and producing really mind-blowing human level results for a fraction of the time and cost of a human team sometimes completely for free for more detailed description of crew AI take a look at my earlier videos on this channel my most recent video on crew aai covers almost the exact same crew setup we will be using in this video so that might be good to watch if you want to understand the crew setup itself if you want to learn more about what crew AI itself and what it can do check out my first video on cre AI llms are trained on huge data sets of text and are getting better every day since cat gbt first came out we have seen the emergence of first GPT 3.5 then GPT 4 and more recently the mind-blowing reveal of GPT 40 by open AI Facebook's meta has worked on their own open source llm called Lama with the recently released version llama 3 meta's own testing shows that the 70 billion parameter version of llama 3 beats Gemini and Claude in most benchmarks it's still a much smaller model than GPT but as we will see it doesn't perform so poorly with crew Ai and since it is open source it is completely free to use either via the cloud using Gro or on your own computer using AMA olama is a tool that makes it super easy to run all kinds of Open Source llms on your own computer depending on what your computer can handle so basically you can just have a chatbot similar to chat GPT on on your computer for free without needing access to the internet or anything else Gro is an AI company that developed a new type of Chip they call an lpu or language processing unit specifically designed with llms in mind allowing it to be 25 times faster than GPU based systems like cat GPT more importantly Gro allows you to run large open source llms like llama 3 and mixol completely for free using the gro playground grock with a Q is not to be confused with grock with a K which is Twitter's chatbot and was created a long time after Gro with a que here's an article of the company making fun of Elon Musk for apparently stealing their name the last llm we tested was mixol which is an llm by the AI company mistal AI it is also open source and is supposed to outperform llama 2 and GPT 3.5 on most benchmarks okay so this time I really want to show you step by step everything I'm doing so everyone can really follow along but I'll do it quickly so first of all you're going to go to Jean Mor's crei examples directory and you're going to copy the URL for the directory then you're going to your command line and you're going to do get clone and paste the URL of the repo and then you have your ca examples so what I'm going to do I'm going to go in there and we're going to work with the Instagram post repo and change it to do what we wanted to do okay so let's actually open this directory with vs code so we're going to do code Instagram post and in here we're going to change all the files that we want to change to make the crew a little bit more simple to then try out different llm backends with this new crew okay so the first thing we're going to do after opening the code is going to main.py and remove anything that we don't want the crew to do so basically because there's crew two two Crews involved here we're going to remove the second crew because it's going to take uh it's going to take more time if we run two Crews so we're going to just run the add copy crew this is the first part and remove everything after line 46 so we're going to remove all of this not everything we're going to remove the photography crew and everything related to the image if you want a little description on what these Crews do go to my last video on uh using crew AI to automate Instagram posts so this is what we're going to do just without the images so this is the first thing and our default Baseline llm is going to be GPT 3.5 turbo and then we're going to compare all the other L llms to it because that's the one we used in the last video so what we're going to do is to the crew we're going to add these three lines so we're going to define the process as being a hierarchical process we're going to make the manager llm chat openi uh gbd3 turbo so those the model we're going to use we're going to use a temperature of zero to be uh giving very consistent uh results and be very and not be very creative and hopefully this will give us the best comparison across models and we're going to set memory to two for this to work we need to import chat open Ai and process from crew AI so that's what we're going to do now so from crew AI we're going to import agent and crew but also process now to be able to use the hierarchical process and we're going to add the following line from lank chain open import chat open which is going to be used to create the 3.5 turbo llm object which we're using here so now we're going to go to the agents file and in the beginning of the agents file we're going to add a few lines so we're going to add the line from crei tools. tools import the website Search tool and the Ser def tool and then we're going to instantiate the web search tool in the Ser def tool which I preferred over the um scrape and summarize and search internet tools from the browser tools and the search tools that are defined in this uh repo but these come out of the box so there's no need for you to do anything else except import them and then we also have the line so this is now the same line as here so we also have the line from Len chain open I import chat openi which similarly to just before allows us to create the chat open AI um object that will be our llm to do the uh work for the agents so the agents llm will be 3.5 turbo again um as our Baseline model and later on we're going to switch it out with other llms to compare so you see these Squig squiggly lines because currently in this uh environment crei tools is not installed so make sure it's installed we'll get to what you need to install and set up for the environment um a little bit later but for now what we're going to do is we're going to replace these tools provided for the um different agents and we're going to replace the llm for the different agents so here all we're going to do is remove this or comment it out if you want and replace it with agent llm and later we're going to change it to be different llm so but we we we defin it up here to be chat GB gbt 3.5 turbo that's what's going to be used for all the agents but technically you can change it for the different agents so each agent can have their own llm um according to your needs but right now we're using this llm for all these agents as you can see self llm it's just the llm that is defined up here but it could be different llms and we're going to remove all the photography stuff because we don't need it today and now we're gonna replace the tools as I mentioned so we're going to use the web search tool and the Ser Dev tool for the product competitor agent remove these two which I didn't like how they behaved so let's remove them and we're going to also use the web search tool for the other two agents so let's copy this and replace the scrape and summarize tool the search tools search internet and search Instagram work just fine so I'll leave those those are predefined in the tools folder here and same here we're going to get rid of the script and summarize website browser tool um but feel free to play around with the browser tool as well as it comes with the gr examples uh that you just cloned so you can you can figure it out for yourself and the only other thing I want to change is adding two lines to each agent one is Max iteration four this again is to keep the work of the agent short and not go into endless Loops especially for the weaker llms that's going to be important um to be able to compare the different L&M as well and allow delegation this will allow the agents to pass um tasks to each other and we're going to add it for the other two as well and that should be it for agents. py now because because we want to use the latest version of cre crei that also has the hierarchical cruise we need to also change a few things in tasks.py namely we need to provide an expected output which which crew I now expects so you need to add the line expected output equals and then put a description of the expected output that you're going to have for each of the tasks and this is what it looks like so here in in this case I have a product description listing the features benefits and key selling points of the product the report should the report should uh also address Market appeal and suggestions for enhancement or positioning so um I literally just copied it out of the test description but be as specific as you want to be and you know obviously change the text for the test description as well and now do the same for the other tasks and we can again remove the photography related tasks so I went ahead and added the expected outputs for all the um different tasks so you can look at them here um a list of top three competitors of the company in question with a detailed comparison between our company and the competitors but you can be much more clever with the description here um campaign development a step-by-step campaign idea for Instagram and the Instagram ad copy um task has an expected output that is a list of three possible Instagram at copy post um the only other thing that I've included last time was the sentence please perform any online search just once so I added it to all of the tasks the reason being that for some of the llms that I tried it was actually um performing several searches um more than once and it just took forever I don't think that would be a problem for GPT but um I'm adding it anyway to um restrain the the weaker the agents with the weaker llms a little bit not to not to do too many searches when not necessary so the next thing we're going to do is set up the environment for uh our crew and luckily the when you clone the repository it came with a p project. tumml file and a port lock file and if you are going to use poetry this will make it very easy to uh install most of the dependencies we will have to do we will have to make some fixes but it will install most of the dependencies and we can we can fix the things we need to fix manually and it will and poetry will take care of um documenting it in the p project. tomal so because we have a P.L file it will know porry will know to install the exact versions specified in this file and all we have to do is go to the command line and first of all if you don't have poetry you need to install poetry by doing pip install poetry but I already have it installed so nothing new will have been installed and and so we're here with the Poetry lock file in the P project tummel file and all we need to do is type poetry Petry install and in my case I found out that I'm currently using a python version that is not compatible with the requirements in the portlock file so what I'm going to do is activate a python version that is compatible so in my case I'm using pyen for this and as you can see I have these versions available 3.9 3.11 3.12 to and what we need is anything above 3.10 and below 3.12 so so in my case if I'm using p en I'm going to do p en Local 3 11.8 and locally is going to use this version and now I should be able to use P install to install the dependencies and now I'm not getting an error so the only thing we want to change is the crew AI line so we want to add a line that specifies a newer version of crei so version 30.11 is the latest version of CI so that's what we're going to use we're going to remove the original line and we're going to include the extras tools because we're using it for um some of the tools that our agents need and now let's save this file and on the command line run poetry update and as you can see now it installed crei tools and it updated crei from version 112 to version 3011 which is exactly what we wanted so the last thing you need to do is go to your EnV file and add your Ser Dev API key and your open AI API key you can remove these two lines be open AI API key without the underscore otherwise it's fine and save the file as just. EnV and then crei will be able to use it and now we can go to the Shell and activate our Port shell to activate all the to activate the environment with all the dependencies that we need and give it a first try at running our crew by typing python m py and hitting enter so of course I made a couple mistakes um so I fixed some commas and also I had a low delegation in there twice so make sure the keywords are in there just once and AOW delegation was false for one of the agents so I set it to true for all of the agents and hopefully now we can able we can run the crew also in case these um extra packages we don't need give you some problems feel free to remove browser tools and load tools and Os and anything you don't need and save the file and then hopefully you won't run into any errors so let's run the crew and just like last time I I decided to just for comparison purposes use the Galaxy watch 6 again so let's copy the URL of the product page go back to the command line and paste the product website any extra details best travel watch best travel watch in the world again I have zero relationships with h Samsung or this watch I don't own this watch this is just to have a product that people know and the crew gets going okay so the crew has finished its job um this is the result so there are three ad copies um that you are provided with and you can look at the log of everything that the crew has been doing but what's interesting to us is the time it took and then also the price so just a little thing that I added that I didn't tell you guys yet is a snippet that gives me the time it took uh for the crew to finish its job and I didn't want to sit there timing it with my uh phone so all I did is imported the package time to find the beginning time right before kicking off the crew so just time.time um open parenthesis close parenthesis then I run my job so I kick off the crew and at the end I'm going to time it again and at the very end when I after posting the ad copy I'm posting or I'm printing the difference of time of the time of the computer from the end of the job to the beginning of the job so the time in seconds that took the job to complete divided by 60 so I'll get the time in minutes and that's what I get here so it took 1.15 minutes to complete the job for C GPT 3.5 turbo so now we want to run it for GPT 4 which is much more expensive than GPD 3.5 but we want to look at the results and the cost in the end and also the time it takes um for the crew to complete their job so we have two instances where we have to change the model one is the um manager llm here so here we can just change it to gbt 4 Turbo and then also in agents we have our agent llm and here again we want to change it to gbd4 Turbo save both files and run the job again so back to the command line and run the crew so I have to say I'm somewhat surprised by the results of gbd4 Turbo so first of all it took much longer than I thought the crew took around 7 minutes to complete its job so it took much much longer than a GPD 3.5 turbo which actually surprises me because I thought it would be more efficient at do at doing its job however I find the results to be much much more on point so let's look at the marketing strategy for um gbd 35 so here's the step-by-step campaign idea for Instagram by gbd 3.5 post one was an introduction to the Galaxy watch 6 the ultimate travel companions St connected track your health and so on post two was about Health and Fitness Post three about seamless connectivity post four about Sleek design and so on and then I created um the posts however for GPD for Turbo it did a lot more stuff and actually if you look at all the results it goes much more into detail about the different posts and what the point is what the image is and I didn't even specify to create an image because I removed that crew remember but it even so it already thought about the image and it thought about the caption it thought about the call to action and in the final in the final results so it included calls to action interactive elements in some of the post and included polls so it's it's some I would say it's a more advanced team of agents and that does make some sense however it's going to be much more expensive and um will take longer to complete than gbd 3.5 so you have to kind of Judge if you think 3.5 is good enough for your needs and if you know you can always run 3.5 um cheaply and then if it doesn't produce good results you can always upgrade to uh gbd4 and see if the results are better but I I'm actually fairly impressed by gbd 4's improvement over gbd 3.5 so now let's check out the new gbd 40 so just like before we're going to go to our main.py file and change the crew to gbd 40 and we're going to do the same for agents so here we're going to change it to 40 and save both files let make sure you save both files so you're not running a mix of different agent llms save and rerun the crew so with GPD 40 as the manager actually ran into some trouble and I'll explain more at the end of the video but I decided to switch the maner llm to 3.5 turbo again and have just the agents use gbt 40 so we just finished running the crew with the manager being 3.5 and the Agents being 40 so gbt 3.5 turbo and agents being uh gbt 40 and I really like the results I think they're comparable to four I think you know at this level it's difficult to see what's better and what's worse um I think the the post look really good and it was quite a bit faster than uh four than gbd4 turbo which makes sense and um it also cost less only 16 cents so the next thing we wanted to try was using Gro with different llms and I'm going to show you first how to set up Gro and get a grock API key so the first thing you want to do is head over to console. gro.com and log in with your email if you have an account if not you have to make an account they will send you an email and then you have to click on the link in the email you won't be able to create a password so click on the link in the email and and then you are in the grock playground all you have to do now is similarly to open ai go to API Keys here maybe you have to verify that you're human like me and we're going to create a new API key and we're going to call it YouTube tutorial and remember that as on open AI you won't be able to see this API key again so copy it and stored somewhere and now you are able to do grock API calls so the next thing we need to do is install Gro on the command line so if we if we were using poetry to install our dependencies we can now do poetry add Lang chain grock like this with a Q not with a K and now it's done and we can very easily now use Gro instead of jgpt by changing our code so let's go back to our main.py file and the main chain the main thing we need to do is from lank chain grock import chat groc so it's almost the same that's what we had and so still doesn't recognize this because it's this file was opened not in the environment that has this um has this module installed but it will disappear once I uh activate the Poetry shell so we just imported chat Gro from Lang Chang Gro and the only thing we did was change chat open to Chad grock we kept the temperature at zero and the model we're going to use for grock is uh llama 3 with the 8 billion parameter version version in this case as a baseline for uh llama 3 and then maybe we'll try the 72 billion parameter version after this we're going to do the same thing for the agents file so let's just copy this go over to the agents file and paste it here and then we can comment out the rest and we have our agent llm as Chad Croc uh llama 3 with temperature zero we're going to save both files go back to the command line enter our Petry shell and now we're going to run the crew again of course I got an error because we had to import chat Croc also in the agent py file and save it and now we can run it again so I want to show you real quick that uh the crew completely failed when using the Llama 3 model with the 7 billion parameters so this is when I was actually using um GPD 3.5 as the manager llm but the agents had llama 3 7 billion parameters and it just didn't know how to use the tool so it didn't understand what tool it was able to use and so on and then in the previous one I had tried to use the manager um llm also as being um llama 3 with the 7even billion parameters and it it just didn't even know how to delegate the task so this was pretty disappointing so we now have the results for grock using the Mixr language model for the agents but using GPD 3.5 turbo as the manager llm and results are okay um I think they're comparable with gbd 3.5 probably maybe a little bit worse it took around 4 minutes so that was definitely worse than uh gbd3 .5 however it was completely free so that's definitely an upside now we're going to try and see if the 70 billion parameter model by uh llama 3 is going to perform better than the 7 billion parameter model which hopefully it will so all we're doing is replacing the model name now with the appropriate model name um that we can find on Gro so here you can see on the gro in the gro documentation the model ID llama 3 70 billion and then the these four digits and um I did this for the main.py manager llm as well as the agent LM in the agents py file and now let's run the crew again so as you can see I ran into some rate limit issues with um llama 3 80 billion uh 70 billion parameters uh and Gro when using llama 3 both for the manager llm and the agent llms so I decided to um now switch the manager lm2 GPT 3.5 and have the agents use llama 3 and see if that works better and seems like this indeed has worked we have a result after two and some minutes for GPT 3.5 as the manager and llama 3 70 billion as the agent llms and the result looks fine I would say comparable to 3.5 in general um I think I was more impressed with gbd4 but it was overall pretty good and it was very quick here's the um campaign steps I think they're pretty good um better than 3.5 I would say maybe worse than gbd4 but it was free so I think this is a very very good option now to run some of these uh open source models locally on your computer at least you know the ones that your computer can handle which is usually models up to maybe seven or eight billion parameters if you have a regular um private lap laptop like myself um you can download AMA from ama.com so all you need to do is go to ama.com click on download and then depending on your computer you can do any of the following options Mac OS Linux or Windows and follow the instructions to install Lama um I myself am on Linux so my instructions will help most of you very little because most of you will not be using Linux so just follow the instruction for um Mac OS and or Windows or Linux if you have Linux so once you have Ama installed you can go to your command line and type AMA and it will show you the commands that are available for AMA so AMA run will run a model of you know a model you're interested in um AMA pull will pull the model to your computer so basically install download the model to your computer and AMA list will tell you what models you have already available so let's do that I have open Hermes available right now so if I wanted to use llama 3 I would have to do Lama pull llama 3 or alternatively I can just do a Lama run llama 3 in aable before running llama 3 actually do the PO so let's do that I have now um downloaded llama 3 and I did this by doing L run Lama 3 which is the same as pull and then run and now I am actually chatting with llama 3 so we we can try it hey can you tell me me about yourself so you can see it's typing but it's typing very slowly um but it behaves just like cat gbt would behave so uh it's a large language model chat and you can interact with it just like you do with c gbt and um llama 3 is supposed to be almost as powerful as jgb just note while it's responding if you go to theama website you can click on models up here and I will show you the available models so llama 3 is what we're using right now and we'll also try out mixol and but there's a bunch of other models like llama 2 which I don't suggest you use because it's just a weaker older version of llama 3 by meta but there's a bunch of other models you can try out but most of them will not be as powerful as jet gbt or llama 3 so llama 3 is done responding and you know this is what it looks like but it took about a minute to respond which I think is mostly because my computer is overwhelmed um because I have a lot of things open but you know it it will work exactly like CH gbt accept this on your computer and it's completely free which is nice so to now use the local model I have very little hope that the local llm will perform well as a manager llm so so the only thing I'm going to do is Define the agent llms to be the local llm so for that we need to go to the agents py file import from L chains llm AMA so from L chains LM import and then Define the agent LM as such agent LM equals AMA and then specify the model we want to use so the model we want to use is llama 3 same as we just did on grock and then we can comment out this one and save the file go back to our command line and run our crew so the local crew is now done um as a reminder this was a crew with a jet gbt 3.5 turbo manager but the agents were all using local llama 3 and the results are pretty good um I think the results are slightly better than 3.5 or on par but a little bit worse than gp4 however it took 83 minutes to complete so it's much much slower than using C gbt and also slower than using grock however it's completely free so here are the final results in terms of quality of the results I think Chad gbt stays the clear winner with GPT 4 and 40 being at the top although GPT 40 had issues as a manager llm GPT 4 was much more expensive than 3.5 so there you have to decide if the slightly improved quality is worth paying 35 times more money all of this is is of course still cheaper than having a crew of humans do these tasks using a team of 40 agents with a 3.5 manager might be a good trade-off as the quality was great but the cost was only 16 cents in general I have found that mixing llms can be a good solution it might even make sense to have different agents use different llms with the more important task being performed by more powerful agents and the simple tasks by cheaper llms I really liked using Gro it was easy to use and much faster than running an open source llm on your own machine however there are rate limits so you would then have to see what works for you the 8 billion parameter version of metas llama 3 was a failure across the board as a manager it didn't know what to do and the Agents also had no clue how to use the proper tools the 70 billion parameter version of llama 3 performed much better probably on par with GPT 3.5 but you might run into rate limit issues when using a crew that uses llama 3 for both the manager and the Agents using GPT 3.5 as the manager and llama 3 for the agents is probably the best overall approach if you want to run the crew for free the GPT manager should cost you one cent or less while the agents are completely free with Gro mixol also had surprisingly good results when using a GPT 3.5 manager however I think there is very little reason not to use llama 3 instead unless you run into rate limit issues lastly if you don't want to rely on any cloud llm and not be limited by rate limits or if grock isn't free anymore in the future running a local model with llama 3 is an option if your computer is strong enough results were decent but of course given the computational limitations the crew took forever to complete its job so that's it guys if you want to learn more about crew AI in general or the details about the Instagram crew we used here check out my previous videos I also have videos on how to get your open AI API key and how to clone a GitHub repository if you need help with that good luck automating and see you next time