Ollama - Loading Custom Models

Key Takeaways

okay so in the previous AMA videos we've looked at how to set it up and stuff like that and we've also come in here and looked at how to install different models in here the challenge is what if you actually don't see the model that you want in here and for example if it's a fine tune of one of the models that are in here but you don't actually see the model that you want in here so don't despair it's actually quite easy to install custom models into a llama as well so in this video Let's look at how to do that all right so the custom model that I'm going to install here is a model called Jackal Loop so I've been playing with this in collab I might actually make a full video just about this model but for the sake of this video we're going to install this into olama so you can see that this is basically a 7B model it's a fine-tuning of the mistal 7B model in here and what we want to do is we actually want to get the quantized version of this so if we come down the one that we we want is this ggf here so if I click into this you can see that the bloke has converted the weights from this model into this GG UF format so ggf is used to be gml and this is quantized version of models so this is the project called llama CPP it allows us to basically run models it's what is using for models underneath the hood next up we're going to go into and find the files versions of this model and here I've got a bunch of different choices so basically these represent the quality of the model there different quantizations in here I'm going to go for a reasonably big one I'm going to go for this q6k in here but you might want to pick a different one and I'm just going to click download here now if you're using git and you're quite comfortable with Git of course you can just do you can just pull this down via git as well but here we can just come in here we can download it so that's going to take a little bit of time to download but what I'm going to do is download that and I'm going to put that in my models folder so that I can then start processing it okay now the model is actually downloaded we can actually see it here in my models folder here which I've made so you can see I've got some other models that I've downloaded last week and I made custom versions of last week so the next step is that we want to make a model file so if we come in and have a look at the Alama model file we can see you know what the parameters are and the general thing is going to be from and rather than us put in sort of llama 2 or something here we're actually going to point to the file that we downloaded where they're going to set up our template Etc and then we're going to use the model file to actually create our model so let's do that so I'm just going to make a text file for a model file in here okay so you can see here I've just sort of pre-filled the model file in here so we're basically saying from and we're pointing to the checkpoint that we just downloaded the model weights that we just downloaded so remember these are the quantized version of the Jackalope 7B model in here we need to put in a template for a system prompt so this is the template that they're using here and then the system prompts people can actually fill that out when they're actually running the model in here so okay I'm going to save this okay so I saved that out and we can see now I've got my model file there now you can see that I'm going to be making a model called Jackalope using the model file Jackalope that we've got here so if I run that you'll see it passes the model file it's looking for a model it's going to create the various layers it's going to create system layer Etc and then it's going to write the weights and then finally at the end we'll see this success here now if I come in and have a look at my models under model list you'll see that I've got the Hogwarts model which remember is just a different prompt for l to and I've now got this Jackalope model in here so if I want to use this model I can just come in and say okay Alama run Jackalope and it will start up the model you can see it's running and now I can use the model just like I would any other model in here so if I want to ask it a question I can do that I can converse with it just like normal if I want to see the commands of what I can do I can basically come down and see that and you'll see that it's just like normal for these particular things so this is basically how to set up a custom model how to use it there probably are some models that are not going to work but certainly all the fine tunes of like llama 2 the fine tunes of mistel even the Falcon models Etc will work for this so this is something that you can definitely use to basically start trying out different models that you see on the hugging face Hub there anyway as always if You' got any questions or any comments put them in the comment section below please click like And subscribe it will help people see the video I will talk to you in the next video bye for now