Build LLama 3 Chatbot on Groq Cloud with INSANE 800 TOKENS per second!

Key Takeaways

in this video you'll learn how to build a website featuring a lightning fast chat with memory capabilities using llama 3 grock and streamlit if you've seen the Javis video you already know how incredibly fast Gro can process data let's do a quick recap what is Gro it's similar to GPT but with an incredibly faster processing capability here in this example you see Gro is done with a 500-word poem before GPT even starts on the other hand dreamlit is a python library that lets you build applications in just a few lines of code with our magically simple API together we will follow the gro documentation to build a very basic chat completion then we'll proceed step by step through the process of creating a Gro llama 3 chatbot allowing users to interact via a web page let's dive into the gro Cloud documentation to see how we can get started we follow the quick start tutorial the first step is to set the gro API key let's just copy the statement but before we first create a new file app.py before we can set the API key we need to create a new one for this we have to log into the Grog Cloud the registration to Grog is free and done in seconds just go to API keys and create a new one just enter a name and simply copy the key and paste it in the console once we have set this we can continue to keep things simple we start with a minimalistic example and increase the complexity from there let's now copy the example code so we can try it out the first thing we should do is install grock with Pip install then we can paste the example from the docs what do we have here at the beginning grock is imported then we create a new client using the API key that we just exported in the environment then we have a classic chat completion as we know from open AI in this case with a mix trail 8X 7B model when we run it we get a small essay about the importance of language models and we see how fast that was totally impressive let's make it a bit simpler and say it should now tell a joke let's run it wow our joke came super fast but this video is not about mix trol let's change the model to llama 3 we choose the 70b variant oops Gro doesn't know this model model that means we need to check what the correct name is for that I go to models in the gro documentation and see that it's not just 70b but 8192 must also be added at the end let's try it out and see if it works with the correct name and it looks good we got our joke all right we want to make it a bit more fancy and add a bit more functionality into it let's rename the file app.py to to basic call so that you can access both files later in the GitHub repository we take a look at what exciting examples there are in the documentation I find the idea of the conversational chat bot quite interesting so let's take a closer look at it well we click on the link to the GitHub repository okay we go there to the app py well that looks significantly more complicated at first glance but we will try to build a simplified version of and step by step so we understand everything that is being done here well obviously the chat was created with streamlit streamlit is a wonderful library to create very simple webuis which can interact with python code but sometimes it feels a little bit like magic I suggest that we first get familiar with streamlit and get to know it and for that we first copy the two statements the title and a right statement for this I first create a chatbot py file copying content and importing streamlit we try to run it and get an error message okay and we see that the indentation is not correct line four must of course be moved further to the left well we try it again okay and it tells us that we should start the program with streamlit run chatbot py and now our web application is actually running on Port 8501 on Local Host we'll call it up now and we see we have a website with our chat perfect then let's continue well the next step is apparently to give the chatbot memory the example uses buffer memory similar to a bucket that holds only a few messages before forgetting the latest we keep it simple and we later set the number of messages to 10 that means we will set our memory variable to a fixed value let's paste the memory code we also need the import statements the buffer memory is provided by the Lang chain Library therefore let's install it well and we now set the conversion in memory length to just 10 messages so that it can remember the last 10 messages then we have the use a question and there we have a text input similar to the python input where we can make entries and they are directly saved in a variable it looks like this we can now ask questions like what is the meaning of life but if we press enter not much would happen because we don't have any further logic so let's see how it goes from here the next code checks if the chat history variable exists in the session state if it doesn't it initializes es an empty list for storing chat history if it does exist it Loops through each message in the history and saves the conversation after that we have the initialization of chat Gro which gets the API key and the model in the GitHub example it could be selected via the sidebar we'll make it simple here again and just hardcode it the following code creates a conversation chain object using Gro and the session memory to manage and remember the context of interactions all right let's copy the complete block we see that there are some missing dependencies we need to import chat groke and conversation chain let's copy the import statements from the example let's install the last missing dependency Lang chain Gro and the indentation is off let's move everything to the left very good now we see that it doesn't know the variables Gro API key and model yet let's simply take the code snippet from our base example so that it gets the key from the environment okay but for this we need the OS Library we import OS briefly we get the model name again from our base example that looks good and last but not least we start processing our request and here we have the condition if a user question is there then we actually start the conversation this code processes a user's question through the conversation object stores both the users's question and the ai's response in chat history and then displays the ai's response on the screen let's move the user question of the input field down so that it's clear all right time to try it out oh we get an error message that it doesn't know Lang chain Gro well here we have the problem now that we didn't set up a virtual environment from the beginning so that the dependencies are clearly separated from each other we'll do that now so that we can install the libraries very cleanly within this project we have to install all libraries again but this time in our new virtual environment and not in the global one I'm lucky to still have the commands in the history and we started again and that looks good we can ask again what is the meaning of life and we get a wonderful answer okay now I also want to check if the memory is working well and for that I write once hello I am Sebastian and that he should please use my name in every answer and he confirms that and is pleased to meet and next I ask him whether he thinks it makes sense for developers to develop courses over Ai and perfect the answer already States it directly and he addresses me by my first name that means he hasn't forgotten who I am and that's great now we actually already have a kind of chat GPT which we completely control and which is much faster pretty cool