Talking to Alpaca with LangChain - Creating an Alpaca Chatbot

Key Takeaways

okay so a lot of people have been asking me how can we hook up alpaca to Lang chain and try it out for a chatbot kind of thing you can do this quite easy I'm gonna show you a way in this video and then later on in another video I'll show you some ways that we could take what we fine-tuned our special version and use that as well so you're going to need to install Transformers from the main branch from GitHub if you just install the normal pip version you won't get the version with the right tokenizer and the right model for doing this so you can basically just bring that in you're also going to need bits and bytes if you're going to do an 8-bit version and of course you're going to need Lang chain so what we're going to be doing is we're going to be using the hugging face llm wrapper in Lang chain to run this so you can see that okay first off we're just going to be bringing in our llama tokenizer and llama for causal language model remember that alpaca is built on llama and is made for a model that's the system similar style as alpaca is so that's where we're using these from Lang chain we're going to be bringing in the hugging face pipeline we're also going to need to just bring in some other things as we go so first off I'm bringing in the template and the llm chain just to show you that okay we can bring this in we can set up our model with bringing it in as eight bits so that we're going to use less memory and it will be faster for inference and here is where the lot of the magic happens is that the way that the Llama model and alpaca model are set up we can actually set up a hugging face pipeline for them which is going to be doing text generation and we just then just passing this model into that here we pass in the tokenizer we pass in things like our max length our temperature our top p and the repetition penalty in here as well and then we can set up a local llm from this hugging face pipeline so this is actually coming from Lang chain and is allowing us to set up a llm with this hugging face pipeline there once we've got that done we can basically try it out with just a normal llm chain so I'm just going to take one of the standard alpaca prompts and style of prompts for the template so we create a prompt template in here we pass this in and then you can see we're basically just setting up our prompt template with the template that we've made here and then the instruction is just going to be the variable that we're going to pass in so this is the stuff we've then basically got this going in here so we can have what is the capital of England and sure enough this is just going to inject this question into where the instruction goes in here and then it's going to give us our answer out our answer out is going to be the capital of England is London and then if I ask it the typical sort of uh alpaca question of okay what are alpacas and how are they different from llamas you can see here we get the standard answer that we got when we just played around with the model by itself without Lane chain all right so setting up the conversation part is one of the key things is we want to make use of the memory here so we're going to llama and alpaca have a good sized token span so we can actually go up to sort of 2000 tokens here what we're doing now unfortunately we fine-tuned it probably for a lot less than that so it will be interesting to see and I would like to hear back from you guys as well how well does it do when you get a really long token Span in there it made you really good you might find at times that it doesn't do as good but okay so we're going to set up our conversation chain so if you remember one of the things that we use in a conversation chain is the whole idea of a memory and the particular memory that I've chosen to use here is this conversation Mission buffer window memory so what this is going to do is give us a window that we pass across the conversation and we're going to represent T number of turns so for example in this case I've decided to set K to 4 which means we're going to have four turns going on and that will be the limit so this should keep us so our token span never gets too wide in the conversation all right setting up the conversation chain we just passing our local llm that we set up earlier with the hugging face pipeline we're going to pass in our memory to be this window memory that we've created here and we're going to just set verbose equals true so we can just see what's going on if we look at the the template here so just to see what the template the standard template is it looks something like this the following is a friendly conversation between a human and an AI the AI is talkative and provides lots of specific details from its context if the AI does not know the answer it truthfully says it does not and so you can see in this prompt we're injecting two things we're injecting the human input and we're also injecting the history or the memory that's coming of the conversation as it's gone through so I decided to modify this a little bit so here's the vision that I modified it to and you obviously play around with this yourself the following is a friendly conversation between a human and an AI called alpaca so we want to just have a little bit of knowledge about who it is so we could also put some things in there that alpaca is three years old alpaca loves to eat apples you could put some other things in there too trying it's always fun to watch people play with this if you've set it up to be quite funny as well so you could try that out to do that we basically just override the conversation.prompt.template so you can see I'm just taking that new conversation prompt template and trying it out so now it's got our name being alpaca stuff in there all right now we get to talking to it the first thing that I try right out was basically just saying hi there I am Sam so it's interesting here we know that the alpaca is fine-tuned not on dialogue right we haven't done a version that's fine-tuned on dialogue yet maybe that will be a future video but here we've done it's financially on tasks so they tend to be tasks where you ask it to give you a fact or you ask it to reproduce a list of something that kind of thing so when I don't actually ask it something it doesn't do a great job it's perhaps a little bit confused in here and what does it do it doesn't generate bad text it just then goes on to generate more text than we asked for so it's taking it in you know that oh it's talking to Sam so hey there Sam it's nice to meet you what can I help you with and then it generates Sam's response back of do you know what time it is and then alpaca sure sure do it's currently and then obviously passing in a token for where you could substitute the tone you could do a Bridge exchange or something they're putting the current time so then I realized that okay this is probably happening because we didn't ask it a question so it's trying to just keep being chatty and make up things itself so if we try it and we ask it okay what is your name so now it's much more on point right it's okay my name is alpaca how may I help you today so we've now got multiple turns in here but we probably shouldn't count the turn where it generated Sam and it's an alpaca it's really should be human AI is one turn in there now I ask you another question can you tell me what an alpaca is sure you give us an answer and it's not too long here you can see that an alpaca is a species of a South American camelid mammal they're typically brown or white in color and have long necks and legs right so they may ask how is it different than alarm so you see at this point now we've got these multiple steps of our memory being passed in of the conversation so we've got all the steps from the start being passed in here because we haven't met the sort of length of the window where it needs to start cutting things off yet okay how is it different than a llama alpacas and llamas are both members of the camera lady family I'm not sure how you pronounce that but they differ in several ways alpacas tend to be smaller than llamas and there and it then it stops there so it should have been able to go on not ideal again remember this hasn't been fine-tuned for conversation so we're just using the fine-tuned version of alpaca can you give me some good names for a pet alarm now if we go back to here look at the memory so this is the next thing I'm going to ask it you can see we've lost the start of the conversation so where I said hi my name is Sam you know that's all been taken out now we're going as far back now as where I asked what is your name so we've got one human two human three human four human and then this being passed in so a memory is a window of four that we've got going on there now you could experiment with the longer memory if you want to but remember all this is going into the language model so you will find that if your memory gets too long then it can actually be quite slow to do things all right so I ask it can you give me some good names for a pet llama sure here are some great options hacha Tikka cushy and wearer a and then I wanted to test it see does it still remember its name so ask it is your name Fred and you can see that we've lost the bit in the conversation where it told us our name so it's only relying on the name being up there yet still it understands from the context no my name is alpaca so that's good we next asked it okay what food should I feed my new lamb so again you see where our window of four is staying fixed so we're losing things that we spoke about early on in the conversation okay and then finally your new llama can eat grass hay even Alfalfa you could also try giving them some vegetables like carrots apples and bananas so this is just setting it up now you could actually try and mess with the prompt more to basically inject the whole personality in there do some things like that but the idea here is that it turns out that llama's doing pretty well I wouldn't say great but it's doing pretty well with the prompt for the chatbot and then passing in this memory that we've got going on the current conversation memory and then generating so this is something you could try and experiment with and this would also work on the Llama model if you want to actually try the Llama model my guess is that probably won't do as well as the alpaca model I don't know maybe I'll try that out and we can look at that in another video anyway as always if you have any questions please put them in the comments and if this was useful to you please click and subscribe I will see you in the next video bye for now foreign