[Workshop] AI Engineering 101

Key Takeaways

hi um welcome to the first event of a engineer submit you're all here um thanks for coming uh so what is this and why why did we have like a smaller session you know there's uh 500 50 people coming for the the full thing tonight um mostly I wanted to make sure that everyone comes in with some base level of understanding um a lot of um conferences try to show try to pretend uh that everyone knows everything has read every paper has tried every API uh and that it's mathematically impossible and uh I always think that it needs to be a place um for people to get on the same page uh ask the questions they're afraid to ask um I want our conference to be inclusive supportive uh a place where you can learn uh everything that you wanted to learn uh in one spot um and so I was fortunate enough to work of Noah uh on Lon space University um which uh has has taken a little bit to to come together but uh this will be the first time we're running through some of this stuff um and is what we consider to be the basics of what you should know as a brand new AI engineer um and like what is the selection criteria um hello God this I've been warned about this hello hello sorry hi okay it's back I know so um what what the selection criteria um mostly we do we do uh what what is this uh hello hello we might have to switch hi oh okay that might be right I'm not a Navy um so the selection criteria is basically like you should know how to do the things that are known to work uh known to work in sense they're not that speculative um most people will want will expect you to do that in in second job and any um AI based idea that people come to you with you should know the basics of how to build or at least if you don't know how to do it uh nowhere to go to get information so that's the that's the main idea of the starting point um and that's about it so today structured uh as a sort of a two-part session with with some uh fun talks in between first part is um 101 where we go through the SP University stuff uh we have a lunch and learn with great uh prop engineering workshop with Karina from anthropic which is super fun last minute addition and I get to add the anthropic logo to adding page which is nice um and then we have uh um AI engineering 2011 with Charles FR who's who's done some full stting uh bootabs so um you're not going to be an expert today um you will get a sampler of what we think is important and you can go home and uh go deeper on each of these topics so enjoy uh this is not say hi to no hey everyone hey everyone uh first of all thank you for all showing up I hope that y'all going get a ton of value out of this um before I go in there are going to be a couple of setup steps uh if you don't have these two things uh go ahead and do that while I run through these first few slides the first thing is having python making sure that you have the runtime installed on your laptop uh and then the telegram app both of those things will be required for the workshop uh and so if you don't have those I would go ahead and just look them up download them on your laptop and or phone yeah everyone yeah so uh I'll I'll sit here for one one minute uh everybody please make sure that you get the Wi-Fi so that you can go install those programs that I just talked about is a tegram messenger yes uh we'll we'll go through that and then what was the other one python just make sure you have the no you're good you should be good okay cool uh so I'll assume all of you have the Wi-Fi or should also be like it's I'll say it again for everyone with zeros instead of O's um but so what you'll be learning through this course uh is really these five Concepts where we are going to just go through the the basics of what it looks like to use you use programmatically uh the the llms and what it looks like to call the actual API uh we'll go through what it what hello what it hello hello you just SW yeah we we'll try this and I assume if I'm just talking like this y'all can all hear me okay yeah yeah okay uh and for for the zoom it's not recording this so yeah it's suppos to go Voice still yeah okay cool uh so I'm just going to talk like this hopefully y'all can hear me uh if you're in the back and you can't just raise your hand at any point and I will just tone up a little bit so really like I said the first portion that we're going to go through is just what it looks like to actually call uh an llm and get a response back and push that to the user uh this is the same thing that you're getting behind the scenes for programs like chat GPT and apps like that then we're going to go into embeds and Tok tokens which is really kind of uh how these models work under the hood we're going to kind of peel back a few layers of the onion uh and then from there we'll go into generating more text but it's a special kind of text it's our favorite kind is code generation that's going to be a really fun one that has a lot of rabbit holes for you to kind of dig in on your own and really level up I think there's going to be a ton of opportunity in that area specifically so definitely make sure that you're taking notes there um and then as just to kind of round it out it's not all text based llms I do want to get yall some image generation and Voice to Text uh those are both aii models that are very useful right now that you aren't getting a ton uh of coverage on in our little section of the internet um so with that I'll I'll kind of just preface this on like hey why why you're here why you should be learning this I think the fact that y'all are all here you're already kind of sold on the idea but really the the rise of the AI engineer has uh a lot of headwind in it you you have uh this this meme that you know does does the circuits every couple months uh where it's just you're you're able to do exactly this now with with uh the new kind of Dolly 3 that open AI is kind of teasing and is in Early Access right now uh and so really AI Engineers if you kind of cultivate this skill set you're you're going to be uh in high demand for all of these opportunities related to all these different use cases um and this you know take what you will from this this is a AI engineer and we use just AI as a search term you know this is up to 2023 if you just extrapolate that you can imagine that purple line being AI just very much going up and to the right surpassing even uh machine learning Engineers uh as kind of the core thesis for for the whole AI engineering Trend uh is that you as an engineer are going to have a lot more value and there's going to be a lot more people that can do it uh if you are harnessing these and building them into products versus is working on the underlying infrastructure itself um moving forward you have uh some of the things that are in the ecosystem different tools and challenges uh so really uh you have all of these different things uh this is we are not going to be touching all of these different tools today but this is just useful to get in your head these are going to be the products that you're seeing rolling around over the next couple days if you're not using this I would minimize it so that people can see it yep um and so today you'll you'll go through these these five different tools these are all you will touch each one of these today uh through apis in one way or another um so that's kind of our road map uh and to get started we'll get handson with gpt3 um so these two slides I would highly recommend uh now that you have telegram downloaded uh both of these are going to be of utmost importance to you this left one will add you to a broadcast channel uh that I put a bunch of links in so you want to scan that uh and if you have it on your laptop that should send a a link over there uh you will find links to the GitHub repository along with just a bunch of other useful resources and information um and then the right one uh we'll go through that in in a minute but essentially you will scan that and that will uh ask you to uh invite the botfa as a telegram chat uh the botfa is essentially telegram's uh API dispenser uh so you will need to contact the botfa you'll go through a a uh a series of questions with him that look uh a little something I I'll show you what it looks like but I'll I'll just pause here for for two minutes so that all of y'all can scan these QR codes and I will uh check to make sure that everyone is actually joining the channel oh great I'm seeing 27 subscribers y'all are killing it super quick are the slides on the the slides are not on on the GitHub repo no all right I'll leave this up for uh about another 60 seconds make sure that everybody can scan and get these two uh for all of the other things moving forward you will have very easy kind of checkpoint so don't worry if you get a little uh Left Behind as we go through uh we have a lot of information to cover over the next uh two two to two and a half hours uh so really make sure that you're paying attention to the information more so than staying up to dat on the code uh if you fall behind after each step there is a new branch that you can pull down uh to kind of get all the functionality that we're talking about um so with with that I I think all of yall have this so I will move over to Telegram and show yall what I want you to do so we're going to go over to the bot father okay great and so the bot father here you will essentially talk through actually we can just go go through this right now um so let me we can clear the chat history so this is what y'all are looking at we can go ahead and click Start and you can say hey cool he he has all of these commands for us right now that's great so what I want y'all to do is we are going to create a new telegram bot uh all of the functionality that we are building today uh all of these different AI API calls we are going to stitch together into a telegram bot uh this is really cool as a way to share also like mode and yeah uh telegram I I can't does do yeah I I can't I can't blast up telegram I'm sorry uh so with telegram you're going to hit Slash new bot you're going to need a name to call it uh I would recommend just uh maybe maybe your GitHub handle uh so just something cool and now change a user name for your Bot this is going to be its handle on telegram that you can send to other people so for example you could do your GitHub handle so mine is inine gbot you that is your username for the bot and this will give you an HTTP API key right here that starts with a bunch of numbers it looks like uh that at the very bottom I know this is a little bit small for everyone but essentially the flow that you're going to go through is new bot go through the prompts get the name and you should get an API key from that uh and from there we will uh pull down the GitHub repository and add that to our environment variables uh so go ahead and get that API key from the botfa um and then yeah yeah just install tegram yeah and then I just from the app just the main I just scan that code yeah yeah so uh ra raise of hands how how many people were able to get into the telegram chat and into the bota in in their telegram contexts just raise your hand if you did get it okay great uh if uh and raise pants if you don't I I can Circle back afterward so I've got a a smattering of people okay uh don't worry a after this first portion we can go through with the kind of Q&A portion and make sure that you are totally set up there um for those of you that do have it uh this is going to be the chatbot implementation uh The Next Step that you're going to want to do is in that AI 101 telegram channel that most of you joined uh you will go through and you'll see at the very top there is a link uh to that original telegram channel for the bota if you weren't able to get him uh so go ahead and make sure that you invite that guy uh and then there is a GitHub link uh it is GitHub inhg a101 uh it is here I can actually just click click on this so in here you'll see there's there's a bunch of links uh and from here you are going to want to pull down GitHub and this is the branch that you will all be working on again this is a link in that AI 101 telegram channel uh go ahead and clone this down uh the the main branch is what you'll want to start out with go ahead and clone that down uh and run through everything in this read me this little python shell uh go ahead and run through all of this let's make that a little bit bigger so you'll you'll just run through and this will install all of the dependencies that you need and get your environment up and running uh is essentially on once you're here this is a really solid foundation for the rest of the course this is all of the really annoying setup done and out of the way um so again all of that is in this main telegram channel for ai11 make sure that you are in there um and for the actual chatbot implementation so we just got a token from the bota if you don't have that please go through that that workflow and then you're going to need to get an open AI API key uh originally I was going to have all of y'all go through uh so you would in in that link if you want to get your own uh you're going to go to a link that's in that AI 101 Channel which is just platform. open.com and you go through uh and you you would need to register your card and generate an API key through there um just for the sake of uh keeping things moving quickly uh what I will also do here is is um I will actually just send yall the one that I have for this example um so I will put this in that telegram channel here so let me make sure I can do that so everyone for if you don't want to go through and get your own or you don't have one right now you can see in that AI 101 Channel this is going to be the environment variable that you need uh if you pull down the repository you already have a env. example and if you run the script uh it will change that example file to an actual file make sure that that uh token will will allow you to do that um I so again if you're behind all of that information just go to all of the time that telegram Channel throughout the workshop that should have everything that you need and so if you've done all of these steps you clone down the repository I just gave you that open AI key uh you're going to load in your environment variables so what that looks like here you can see that uh bot token here let me make this a little bit bigger for everyone uh uh let's is that pulled down yeah so you should be able to see you've got the tgor botor token and the open aore API uncore key both of these are these are the only two environment variables that you will need uh and once you have that this will be your own bot uh in telegram along with your own API key or the one that I just gave you in that channel and from here what we can do uh is we're going to add an open AI chat endpoint um so what you can see here uh is in our source file we've got this main.py file um and in here this is what you should be working with if you have uh pulled down the repository successfully you'll see we've got a list of imports then we're loading in all of our environment variables and then we are loading up the telegram token uh we've got some messages array this is going to be how we interact with the chat system this is essentially the memory uh that the chat apps use is this back and forth is just an array of objects where the content is is the text of all of the questions uh we have some logging to actually make sure that whenever you're running the program you're getting some amount of feedback uh as as it runs and we have this start command so I'll really quickly in this portion run through uh the telegram bot API kind of architecture so you will Define for each different section you'll have a uh a function that function will take an update and it will take a context uh the update is going to be all of the chat information essentially all the information about the user uh and the context is going to be the bot so you can see here in this very first thing we're going to just call the context dobot send message and the send message command takes a chat ID and it takes some text so the chat ID we get from the update variable and so that's just saying like hey whoever sent me the message send it back to them I am a bot please talk to me so cool we've got that functionality in start but how do we actually make sure that the bot knows that it has this functionality we use that through these handlers so we have this start Handler right here on line 28 and it is a command Handler so command handlers uh if you're familiar with telegram or Discord anytime you have that slash command that is a slash so this first one is going to be anytime the user types slash start uh this command Handler will pick it up and it will run the start uh function that we declared above and then we will add that Handler to our application uh this application is where your actual bot lives you can see we've got the telegram bot token that loads in here and builds up and then it just runs the polling uh so what happens if you have all of your environment variables set up correctly right here uh is if you're going to run so from the root of the directory you can run your python source main.py and cool you can see the application started uh and every couple of seconds it is just going to Ping as it runs through the polling back and forth uh and you'll notice here I have got this is the bot that I started so from the bot father you get uh a link right here so this this would be the new one that that I created uh but I have a previous one that I already made uh so make sure that from the bota uh it has this original link uh and make sure that you invite that so it would look like this and it's it's just another chat make sure that you start it this is the bot cool EXC yeah could you go back to the main yeah and so if you another Branch right because line number six doesn't exist on the latest one uh are you on Main so for yeah you you should be on Main yeah yeah you're saying on line six load do line6 is a space oh then there's another one there was another window though right uh I I would say if if this does not work you should just be able to pull down the GitHub repository put in the API keys in your EnV file and run main.py and you should you should have functionality out of it yeah um you to the the QR code sure oh and I will blow this up yeah this is really important I don't mind taking a while on this guys all of the other ones will be pretty quick uh because you can just checkpoint so if you don't have these uh just take take your time truly we want to get everyone on the same page there's not a rush here you know uh to be honest we are still ahead I was not I did not think everyone would be here bright and early uh so I planned this workshop for starting at 9:30 uh and so we are still 6 minutes early as far as I'm concerned we really want to make sure everyone gets set up and is in the right spot so uh really I know all all these QR codes that can be quite a lot to get through in the initial portion cannot import name name telegram did you run the from the GitHub running running through and installing everything uh so like I did install code like I'm cop if you just copi the code you'll need to install everything so here yeah I did install so unless there like a can you help about this uh should we point out that we have two t who I've got I've got Justin and and Sean okay and Eugene's available to help okay yeah and and really quickly guys uh I I failed to mention this at the beginning I'm kind of like running the workshop through as we go through um we have Justin and Sean and Eugene are all here and and K assist uh all all three all three of y'all uh or Sean and Justin can you both raise raise your hands just hey get get either of their attention they should be able to help you actually get set up uh if you are having questions in in the middle um I don't mind right now cuz we are very much in in the configuration portion this is the most friction that you'll experience uh through here it's pretty much smooth sailing after we get everything configured and set up as as is the wo with software as a whole if trouble yeah yeah yeah you're good yeah the uh through that API key that the bota generates yeah yeah yeah yeah so telegram has an API and we're we're just from that API key it knows where to send the messages from I'm running this IP in here should I see anything uh not yet not yet not not yet so right now it should just be SL start and that's all you get okay so before I move on does anybody any any any one person is okay I will leave this up here cuz like I said we are still 3 minutes early as far as I'm concerned and we're already halfway uh through through the introductory slides does anybody still need this QR code beautiful um yeah that's the Wi-Fi code that is different than this one uh no I I don't yeah no I'm I'm not trying to deal with a printer on top of all of this I I do apologize the QR code and uh everyone the botfa is in this initial one uh so the left one is more important than the right one yes yeah yeah but the botfa is like first first party telegram API and I get that question a lot telegram could do a bit to make The Branding a little bit more official you tell everyone said yeah telegram go to the bot father they're like I don't know that sounds sounds a little sketchy to me um but yeah the botfa is the the official telegram uh doar out of API Keys okay and I will double check okay so I see 62 people in this chat as uh so I'd say we are good on the amount of people that are in here and the botfa is in that one as well um so I appreciate all of yall going through I know the configuration is always the least fun of any software project uh and so what you should get after you have all of that uh is like I said we just run this main.py file that will spit out some logs and the functionality that you get from that is just uh as such let me clear history here uh is you'll just hit start this this is what you've gotten so far is a bot that it doesn't matter if you're typing anything you say hey hello uh we don't have anything we have exactly one Handler that picks up uh the start command so I can hit this over and over and over again but that's it that's not the most exciting functionality that you could get uh so we're going to go ahead and add uh basic chat to to the bot uh and so what that'll look like um to to save y'all from me just typing like code in front of everyone uh and this is a a good segue into what you can do if you fall fall behind uh on each section uh is we have a bunch of branches set up for you so we've got step one 2 3 and four so if you're ever behind you can just Skip To The Next Step uh so what you would do to do that is just get check out step one cool we have now switched to Step One uh and if I reload my file here you can see that I will have a bunch more in my main.py file um and so now that I have done that uh I will walk you through step by step what you need to add if you want to add it on your own which I encourage you to do so to the best of your ability try not to swap branches it's totally fine if you need to but you will get a lot more out of the experience if you actually write each section of code as we go through it so now we we're essentially on step six of the chatbot implementation so I'm going to make that a little bit smaller so that we can blow up this text a little bit more uh and so what you'll want to do is you're going to need to import open AI don't worry about installing it I added all the dependencies for the entire project you aren't going to need to run pip install over and over again you you have you have it all you just need to actually bring the import in so go ahead and import open Ai and and you're going to add this open a. API key uh and you're going to pull in that environment variable that we talked about earlier so this can either be your own open AI API key or the one that I posted in the telegram Channel just now either of those will work um and then from here you'll notice so like I said for each uh piece of functionality we're going to add a new function so we've got this a async chat function that again takes the update and it takes the context and so the very first thing that we do is that messages array that I told you about earlier so we've got this array of messages we're going to app Pinn to that array we're going to say hey there is a role of user and the content is going to be update. message. text like I said update is all of the information in the actual telegram chat so the update. message. text is whatever the user just sent in that line of text to the bot it is going to push that and is going to add it to this array of messages so there are three different roles that uh openai has one of them is system uh so you can see this is kind of us uh setting the initial prompt for the bot saying hey you are a helpful assistant that answers questions and then back and forth you will go through the user and then whenever the AI responds it will be uh the role of assistant so you see it will bounce between user and assistant with just the system prompts at the very beginning so the very first one hey we want to append it to the messages array and then we're going to want to get the check completion So This Is Us calling out to the openai API and so that's open. chat completion. create and that function takes two arguments one of which is the model and that is GP t -3.5 dturbo as a string and then it takes a second argument of messages and that is expecting the array of messages that we just mentioned earlier uh it takes a bunch of other arguments that you can tweak but just for the sake of this this is the only two that you need to get a proper response and so cool what we have essentially just done is we said hey you're a helpful assistant and then the user sent it a question and it's going to take that question and it is going to run through the G GPT 3.5 turbo model and it is going to give you a completion at that variable and so that variable is a rather large object that has a lot of metadata in it and so we really just want the answer if you had some logs maybe you could just send the entire object to the logs but we are only concerned right now with sending a useful response back to the user so we're going to say we're going to call this variable the completion answer and that is going to be the completion object at the choices at the zeroth index and that is a message and content so that's uh rather rather lengthy piece there but essentially that is just yanking the actual llm response that you want from that API response and once we've got the answer back we want to again appin to that messages array so this is uh you'll just think of messages as being the memory for the bot so if it's not in that messages array the the llm has no idea that it happened it is back to its pre-trained model so you'll notice uh once we actually get this running that every time you restart the server it no longer remembers the that previous conversation so if you want to reference previous material this is what allows that to happen is by adding additional context into this messages array in that uh kind of format of the role and content so I know that was a lot for just four lines of code but really this is step by step how you are interacting so it's generally hey uh llm I have this question it's going to say hey cool let me get you a a bunch of information back you're going to yank the useful piece the content out of that and you're going to do something with it in this case we're just going to send it back to the user and so that uses the exact same message that we had in the start command and so again that's the context dobot send message where the chat ID is the update. effective chat. ID and the text is the completion answer so that that gets you right out of the gate uh don't worry about question that'll be in in the next section we'll get to that so really this is what you're you're trying to get through is line 27 to35 here uh is this chat function and and then from there you will follow uh a very similar thing so we had the start Handler and again don't worry about the question Handler we'll get that to that in the next section so you're going to worry about this chat Handler uh which means that you are going to need to import in telegram uh this message Handler so we'll we'll jump to the top here so you see on line four we have the telegram. extension you're going to need to import the filters that's with a lowercase f uh and then you will also want to import over on the left here the message Handler so those are going to be two Imports that you need to add to line four the telegram. extension import and from those two if we go back down you can see the chat Handler uses a message Handler and so the message Handler uh is going to go through this this filters object uh filters is a way for the telegram API to essentially filter through uh various types of media that you could get so in this case we only care to receive messages that have text and only text in them uh and then that they do not have a command in it that's kind of what this this till is is just hey if it's a command I don't want you to listen to it okay uh and then the last one is going to be hey what is chat like what what function do you want me to call whenever I see the criteria of filters. text and the filters or till day filters. command so if those two are met it will invoke the chat function so again that is still the same Handler so we created the function we created the Handler and then we are going to add the Handler of the chat um so again don't worry about the question Handler that a mistake on my end that should be in the next section oh well I I do apologize for that but I I think you get the idea and so so if you have all of that once once you have this and again you run Source main.py permission denied oh that would help if I actually made the command yeah and you'll see this will boot up yours will probably be a little bit faster than mine because of the additional stuff that we added so cool our application is now started and if we go over to our bot now I can say um let's see uh who is Simon cowl we all love uh some American Idol judges and cool we now are getting responses back from our open AI API key we said hey Simon Cal is a British television producer executive blah blah blah blah blah cool um but like I said since we have a ended my message of who is Simon cowl and the bot's response of the actual answer we can now reference that in the conversation so we have uh you can now reference it so I could say um let's see what what is his net worth so we're able to reference like if that Standalone question what is his net worth it has no idea what that is without the a pending of messages going back and forth so you can see that's uh this is essentially what's giving it its memory and allows you to reference the previous conversation uh if I were to spin down the server and then spin it up again it would have reset messages to not have this in the context so we wouldn't be able to reference this anymore uh so with that that is essentially the the chatbot implementation where we're essentially now have uh chat GPT in in your Telegram but um and so that is everything for this section uh there's I'll be posting the slides uh a link to the slides after the talk uh so that you can reference things but there are uh little rabbit holes throughout the talk where you can kind of delve into more um and so I think for this particular section things that are interesting to talk about let me make this a little bit bigger for youall uh is messing with the system R prompt uh and by doing that you can have it perform various activities uh like making it Talk Like a Pirate you can put that in the system prompt and that link will send you to uh essentially uh two GPT Bots having a conversation back and forth with each other one talking like a pirate one talking like a nobleman uh and the other one if you go to that uh link is it's uh step by step it's trying to guard a secret so in the system prompt they have hey the secret is abc123 or whatever and don't give that to the user and it is up to you to kind of trick the AI into giving you the response and each step makes it progressively harder and so you all of that difficulty is entirely encoded into that system roll prompt and making it more robust and giving it more and more information to reason about how the attacker might try and get it to to give up the secret um so none of those are things that we're doing right now uh but I'll move on to Q&A was there any any general questions uh after that chat or after to that section yeah yeah about the memory the way that you are storing the the memory approximately it depends on the model right but uh how can we handle that with the code what I mean is how can we handle when the user is getting to the limits and we need to sort of uh yeah uh so so the question is like hey um for a particular memory how do I manage that in the code where if the user is essentially we're maxed out uh the the llm can only take so much information before it says like hey man I I'm kind of maxed out on capacity here how do you deal with that question uh and that's like a problem in the space currently if you're the term that you'd be looking for is like long-term memory is how do we give these AIS very long long-term memory on like hey I've been talking to you for the last week and I want to be able to reference all of these various conversations um right now the for this specific example it doesn't uh quite equate one to one but one of the answers is what we'll get into in the next section which is uh retrieval augmented generation where you will take the contents of that memory once it gets too long and you will turn it into uh a vector if you don't know what that is right now that's that's fine but essentially you uh store all of that information in a way that the AI it's very information dense and you give the AI the ability to kind of uh look up up like hey for what the user wants let me look at all this previous information uh and maybe I can reference that to answer the question better uh so it kind of condenses all of the memory to give it storage in a in a certain aspect good question yes sir I guess similar theond Max um what would probably happen uh if I had to guess uh how this specific one would break uh is you would probably see here uh that we would fail to respond to the user and there would be some error that's like hey context limit reached uh and so you would see that in the logs and the user wouldn't get any feedback since we don't have a a fail mode implemented any other questions yeah inst the library or something update wasn't part of the [Music] um did you did you run the uh inst the requirements I think I just I'm redoing it so okay don't worry uh there there'll be a break here after the next section we can we can go through make sure that you're up to date um or you can also go visit one of the Tas that can probably get you set up uh yeah so uh one thing one thing I always want to make make sure it's okay um anybody uses jargon that you don't understand please feel free to to ask about it uh I heard words like context whatever this is the place to ask ask about it yeah um the rest of the conference is going to just assume you know it so please um raise your hands because you're not going to be the only one here yeah absolutely and also know like uh there's lots of people that are watching this uh and so for any question that you have you are also kind of representing all the other people that are watching that aren't able to to ask their questions um and and for that this is very just use usage based driven uh we'll get into a lot of the jargon that Sean just talked about in the tokens and embedding section um yes yeah the the Wi-Fi network is prosperity and the password is for everyone with zeros instead of O's and we've got yeah there you go he's done this before yes I'm sorry say that in the Handler yeah uh don't don't worry about the the question Handler anything with the question that that's in the in the next section I accidentally included it in the same Branch don't worry this that's that's what we're going to go over in the section yeah just just the chat Handler yes in the next uh tutorial you're going to take us through is what was behind yeah so uh if if you're behind each uh branch is like a checkpoint so if you go to that branch and you run the install you're you're up to date on on everything yeah yeah so if you're if you're on Step One is this section currently you'll you'll be good uh yeah of course okay uh so getting into tokens and embedding so embedding is actually what uh I just answered with that very first question and how you kind of store all of this like long-term uh information for the chatbot to reference uh and we'll also get into tokens which are related to but slightly different than embedding uh so tokens uh the definition of a token is really just uh you can think of tokens as the atomic unit for these large language models it does not understand uh English it understands tokens everything uh that it deals with is in tokens it generates tokens uh and those are subsequently converted into spoken language such as English um they are hugely hugely important uh as that's what you get charged for this is the the money that you get charged for is based off of the amount of tokens that you are consuming uh with your various API calls or embeddings uh so it's how they interpret words how they understand everything um and what we just talked about uh on the beyond the models limits it's context you can uh memory and context you can think of that as the same thing where con the context limit is like the amount of tokens that it can reason about so if you generated a string let's say its context window was 100 which is like not not the case for any model that'd be like very severely limiting but say it was 100 and the question that you had had 101 tokens uh it wouldn't be able to understand it you have broken its context window uh and chunking is how you handle that to ensure that all of the context is retained through all of this information um generally speaking a token is representative of four characters of English text specifically um there are these things called uh tokenizers which we'll get into in a minute which is essentially the uh implementation of converting words and text into tokens uh there are various different tokenizers some of them are better at uh other languages uh so for example like uh Spanish is very expensive token token wise uh for the open AI tokenizer uh there are other tokenizers that are being you know built by researchers uh like if y'all are familiar with the um project repet uh they built an in-house tokenizer that was specifically meant for for code uh and so this like uh everything all of these variables are always changing and moving quickly so it's important to kind of Reason about everything from first principles um but there are some interesting ones uh that are exceptions like the word raw download clone in ined report print is one token there's uh you you can read this is a a very dense article but this less wrong post uh goes into kind of speculating why that is the case uh but it you are able to break the models with some of these tokens because how we think of that is like that's a weird looking word uh but the representation could be a little bit off uh and this thing on the right you can see is a picture of uh all of the tokens and how it's actually breaking down the text model uh and you can also try this platform. open.com tokenizer uh that is just a playground you don't need a sign up or anything you can just get in there and start typing words and that can get you a bit of an intuition for how it's breaking down all the words into the actual tokens yes sir does each correct yeah uh tokenizers are not you can't exactly just uh swap it's not interoperable what does that do to the system problem requirements nothing yeah so your your system prompt requirements uh you you have this whole English phrase that you've generated on all the instructions and that gets broken down into tokens models model to uh yes so each model uh if you're thinking for General language use so like llama being another example uh it if I'm not sure if it uses the same tokenizer not off the top of my head but even if it had a different one both of the tokenizers are trained and both the models are are you know aligned with their tokenizer to take English text into a way that is useful for the user uh and so getting into embeddings is the next portion so if tokens are kind of the atomic unit uh you can think of embeddings as well the definition is it's a list of floating P Point numbers if you look at tokens they are uh a bunch of numbers and so really uh embeddings is how we are able to store information uh in a really dense way for the llms to be able to reference mathematically uh and kind of get their semantic meaning um and so you know the the purpose of it is that semantics are accurately represented um and so this image on the left is kind of showing you uh for all of these different words how close they are to each other is how close the uh embeddings are to the actual floating Point values are closer to each other uh and so you can see like dogs and cats are close to each other strawberries and blueberries are close to each other um and so all of these words have semantic meaning and how close they are is representative by these embedding models um and so usage and what we are going to go through is how do you take something like semantic search where we have a huge amount of information that we want to reference uh but obviously I can't just put all every single text in Wikipedia in a giant text file and copy paste it and give it to the llm and say Hey I want to give me information about the tailor Swift article uh we have to generate embeddings and query them and contextually contextually relevant content um and so if you're behind from the previous portion uh go ahead and pull down the step one the step one branch uh but this is going to be um oh actually before I get into this uh if if you haven't let's go over to the telegram here I want to make sure that y'all get this uh prior um so pull down the and I get AO AI 101 okay there is uh this link uh to the embedding sled. py file uh make sure that you pull this down go ahead and generate this uh if you are on your own uh create an embedding folder and then copy paste this file uh and just run it uh and what what I mean by that is uh I will show you so if you have that file again reference that telegram channel that you're in for the actual contents of that file uh you will see that there is this embedding folder and in here there's ed. py I want you to just while we go through the rest of the section uh is Python 3 edpy uh and this just going to sit here oh hold on uh okay so whenever you run it run it from the root directory uh make sure that you're on that file so you do Python 3 edding ed. py to make sure that that file runs correctly because file naming and path stuff uh and so this is going to take five fiveish minutes to run your terminal is just going to sit there so make sure that you go ahead and do this step uh and while that is running I will explain what what is happening so I'm going to stop mine because I have already ran it um but essentially I will run through right now uh the entirety of this file uh let's go up here and so like I said embedding ed. py res okay cool okay uh and so this is that ed. py file that again is in that AI 101 telegram Channel yeah yeah so yeah we'll we'll get into this this whole portion here um so like I said copy paste this make sure that it's running don't worry about writing this code yourself it's a little bit tedious so just really make sure that you you go ahead and pull that down copy paste it run it uh so we've got a bunch of imports uh so we've got pandas um the OS and we've got Tik token uh Tik token is a python library that is the tokenizer uh whenever you are running that if you go to that playground link where you type in a bunch of stuff and you get to see the tokens uh it is essentially just doing a visual representation of the tick token Library let's see if I can move my mouse get that out of the way maybe we can go to the bottom yeah okay uh and then we've got this thing so we are pulling in Lang chain for this course and we are using the recursive character text split uh I know that's that's a a quite quite the name there uh but don't worry we will get into what this is used for I know uh you will see Lang chain reference quite frequently as a very popular open source library for doing a lot of different things yes do you have [Music] yes no you're good U while while we sorting it out so um we're actually getting a preview of a lot of the stuff that we have speakers for later so like L speaking and then we also have Linus from notion also talking about visualizing in biddings um and what he showed you is like what most people see like the clusters of Bings but I think uh you could actually you have actually looked at the the numbers um then you really understand at the low level how to manipulate these emits what's possible what's not possible um and I do highly recommend it um so a very classic thing that um I the first time I worked with Sean um or actually I think it was more more Al but um you know like can you embid a whole book should you embid a whole book um and and so like the the maybe unintuitive thing is that um you know if you embed one word versus you book you get the same set of numbers U because a bidding is effectively asking something like what is the average color of a film um and so that that that question makes no sense unless you you break it up into scenes and then ask what's the average color to SC um so I I do I do like to uh can you send it to him it's in the small AI Discord it's just a link if you can send it to him yeah okay so you can see um what's going on under the b n helps a that um you don't need nchain if you if you are comfortable enough but we recommend getting familiar with it because these things are just tools that the community has decided is pretty necessary so this where we starting you off that yeah uh yeah we didn't I didn't think through that one so uh yeah re retry um and if that ends up being a blocker as we go through you will just go go to the open AI platform I I did structure this to generate your own API key uh it's not expensive through if you do this entire Workshop you will generate approximately a nickel in charges uh so watch watch your wallets everyone um uh so definitely if if the rate limit becomes more of an issue we'll we'll take a minute in one of the breaks and everyone will need to yeah yeah yeah yeah no is pretty decent yeah yeah I I generally I I haven't had problems with it share sharing the key for a workshop like this but if you do hit it try again and if you're you're really really hitting it then generate your own um o okay we can also just download the CSV right the rate liit coming from yeah from from the embedding yeah um so there is also a portion um essentially what this file is going to do it is going to take a bunch of text files that you may have noticed whenever you downloaded the initial repository that is a web scrape of the mdn docs it is just a scrape of all of the text uh and what this file is going through is it is grabbing all of that text and it is passing it into the open AI Ada embedding model um but I did foresee that because this takes a while uh you don't get really tight feedback loops on if you did something wrong cuz like I said that file just sits there for like 5 minutes in the terminal with nothing happen happening uh so there's also in that telegram Channel you will see an embedding CSV file if for whatever reason you're not able to generate the embeds that embedding CSV file is the output that you would get from that you can just download that straight from Telegram and it is the same as if you had run this command successfully um so going through that essentially this entire file like I said is just going to do the embedding so we have a bunch of information around essentially cleaning the document so that we are giving it the best uh data and the most uh information dense data possible so we have uh we have this command that will remove a bunch of new lines and just turn them into spaces uh that'll that'll save some tokens um and then essentially what we do is we have this texts array that we're going to store all the text files in and then this is looping through all of that um docs and so with that we read each file and then we are going to replace any underscores with slashes um this is because there is a kind of Easter egg in here for people that want to dive in deeper we won't get into it in this course but this code is set up in such a way that you can ask the AI to site its sources uh because if you look in that text file you'll notice each uh name for the document is actually the path for the actual mdn developer docs uh and so we just replace the underscores or we replaced the dashes in the URL with underscores so that we can store it uh so essentially just undo that so we have the entire link uh and we will embed that in the documents so there is essentially the AI has the information of like hey here is uh the int the CSS web page uh I also have all the information on that web page