Create fine-tuned models with NO-CODE for Ollama & LMStudio!

Key Takeaways

hey there my name is Timothy carat founder of implex labs and creator of anything llm and today I am so excited to show you something that I'm going to show you something that isn't released yet but will be by the time you watch this video and that is a no code way to produce a fine-tuned model now there are a couple caveats to this and I'm going to address them all just so that everybody is on the same page watching this video so the first thing is that at the time of this video or by the time you watch it the feature will be live on our dockerized version which you can find on GitHub not on the desktop app version yet the current desktop app version is 1.5.1 when that number is higher then the feature is live there's no other way to put that if you want to download the desktop client so that you can use these fine tune models that I'm going to show you today you can go to anything lm.com and go todownload and you can download the correct install for whatever your operating system is and have all of this the next detail is that this feature is new and honestly pretty innovating we're going to be able to allow you to just chat with any model you like whether that be GPT or anthropic or even a local llm and from the outputs of your chat create a fine-tuned model that you can again load locally that already has all of your knowledge baked into it and the last thing is that the current fine-tuning I'm going to show you today does not run locally there's a couple reasons for this the majority are really just two reasons one is that gpus for training are expensive so it costs a little bit of money but I think you'll find the prices pretty fair and then second is that fine-tuning is also really hard fine-tuning if you know how to do it this video isn't for you but if you're a person who has been using Rag and local llms and you know that fine tuning is the next step which logical because fine-tuning plus rag is very powerful but you're not technical enough or really want to go through the pain of learning how to set up a fine-tuning pipeline and infrastructure then this video is perfect for you again if you're a super technical person and you know how to fine tune this isn't for you and actually I'll be making a video after this one hopefully that actually shows how to do the fine tuning locally on a consumer grade GPU that you might have access to and it makes everything a little bit easier to do even if you don't know how to code however that video is coming in the future I want to show you this feature today now you may have noticed that I'm in a browser that's because I'm using the dockerized version or the multi-user version of anything llm this feature will exist in the desktop version as I mentioned earlier in the video the UI should look exactly the same so you can follow along in this video even if you're in the future and you're watching this video months after it's been posted now I just want to walk you through the current setup just so that you can get an idea of how powerful streamlined fine-tuning is going to be for you so the first thing I want to do is let's check out what llm I'm using right now I'm actually using gp4 or Omni from open AI yeah I'm going to create content with a what you would call a more powerful model and then use that to create a stronger fine-tuned model and you can see that I already have a workspace started and in this workspace I have a couple documents that are relevant to anything llm uh the readme is the readme or an old version of our readme from GitHub anything lm. text is a very short just text blurb of what anything llm even is and then I even scraped our own website and just added that in those are three different data sources all in the same workspace and then I've gone and just created traditional very simple rag chats I also used some agent capabilities in this chat for web search just to get some better context and answers and so you can see I ask what is anything llm and it says all of this stuff and we have citations for every single answer that comes through and you can see I kind of continued on here for a while um just and then here or I asked the agent to search the web and tell me more because this data wasn't in my workpace but I think in the future its context could be useful and I'd like it in a fine tune so I asked some questions it says that it found some stuff it says you know I was the person who founded it which is fair and accurate um there are lots of contributors and I've very appreciative to every one of you so it's not just a oneman show um asked it about the GitHub repo all of this stuff so we have some chats this is not a lot of chats in fact actually I think it's around 23 chats and what we're hoping to accomplish today in this video is I want to take these chats send them to a cloud GPU create a finetune of llama 38b send that back to me and I'm going to run that model locally and then ask it what is anything llm and what we should hope to see is an accurate answer with just these 23 chats now obviously the more data you have the better your answer can be obviously if your answers are really bad or they're garbage you'll get a garbage model output but you can continuously fine-tune a model over time and that makes it even more powerful and when you take the benefits of a fine tune model and rag you get a really awesome system this brings me to my next point what even is the difference of fine-tuning vers rag this is a very simple chart I really made it right before starting this video because the visual I think helps when you have a fine-tuned model think of educating and how you learn subjects right you may read something in a book and that may become something of background knowledge to answer a question but you couldn't say exactly what page and what book you got a piece of information from that is a little like how fine-tuning works I'm obviously simplifying things dramatically but in general if you need a model to respond in a certain way instead of just relying on a system prompt or you want your answers to be more accurate to some Niche subject that the foundational model isn't trained on perfect if you want perfect citation accurate recall fine-tuning is not the answer that is rag but when you combine the two you've now got a model that behaves in a certain way answers questions in a certain certain way but then also has the ability to cite those answers and has the background knowledge to back it up or fix or correct a piece of information that maybe it was trained on now I understand a little bit of this could have been confusing the process of getting a fine tune is not I really don't want to bore everyone with the technical details because they're frankly not that important here the output is really what people want from a fine tune so let's just make a fine tune and then we'll test the output and see what happens so all I'm going to do is I'm going to open up the settings I'm going to go to my workspace chats and you can see all of my chat history and we're just going to go to order a fine tuned model again let me reiterate that this process will eventually be possible locally although I don't know how widely supported it will be but for the sake of this video and this feature this is a very simple cloud-based service onetime cost we give you the ggf file so that you can run your fine tune locally you get to keep it and take it anywhere you want whether that be LM Studio o Lama or anywhere else and so one of the first things you'll see is we kind of tell you like hey if you want a fine tune be sure that you actually want a fine tune now while we can turn you a fine tune around in an hour that doesn't necessarily mean that the fine tune will be the answer to all your questions right this is a tool much like how rag is a tool or how agents are tools this is just another piece of the puzzle that makes a deskop top based AI system and assistant more powerful we're going to start the fine tune and there are some privacy policy things of course in general if you were wondering how this works we do have to bundle and encrypt the data send it to our cloud service but that data is only used during training it is not used in anywhere else and it is deleted the second that a model is either unfinished or finishes we get rid of your data as fast as possible we understand this is a sensitive point for some people we just ask for your trust I feel like with the content we've put out and the code being open sourced that trust can be earned if again if you're not comfortable with that or you want to do everything locally this particular video is not going to be for you you might want to follow up in the next and so all we have to do here is just provide an email that basically my model is going to be sent to and then we can select a base model right now for the introductory version of this we only have llama 38b and this will hopefully be expanded to other largely supported models like Gemma mistol fi the models that you probably use and then for the model name this is simply just the name of our model it doesn't really impact anything we can call it anything LM or let's do like fine-tune anything out there's actually only 14 chats we're really going to take a Gamble and see if this fine tune even works and if it does that again just shows you the power of how competent fine tuning can really be on top of a good base model obviously we recommend about 50 chats but I'm not going to stop you from creating a fine tune if you believe that your answers are high enough quality and you can filter by positive responses you may realize that in the chat you can leave a thumbs up on every response if you have a model that's giving you answers that are Sometimes good sometimes bad you only want to keep the good ones you can do that if you like and then for the confirmation you can see that we just get kind of a review of everything and the onetime cost you would ever pay is 250 I feel like that cost is very reasonable and effective I would Hazard you to go and find someone that could goer you a fine tune for less than $1,000 and that fine tune even be halfway decent if you again you want to do this all locally and take that under your wing by all means go ahead you can always just go to workspace chats and you can export the data as alpaca and it'll work all the same if you want to go do that it won't be easy but if you want to do it we won't stop you in the interest of time I did start a fine tune earlier than editing this video just so that everything would be in one place and you can see actually that you have we send you an email saying hey your fine tune is in progress and we're going to let you know when your model is ready and we even will provide instructions for AMA and LM studio so you can get these models running locally as fast as possible now if you were to look at the previous email that I just showed you would see that we started this process at 3:30 and we finished it at 3:52 so that's well under an hour and that was only with 14 chats and the time to train actually does not change that much between uh basically data set sizes and so you'll see what we provide you here and of course you can always get new links are expirable download links for you to download not only the full 8bit quantized version of your fine tune but also the instructions to load this fine tune into AMA and LM Studio it is very simple it is basically a drag and a drop and I'm actually going to show you how to load a fine tune model or any custom model for that matter into olama we're going to do that on my Windows machine and I'll show you how to do that once I download this model so let me go download it and I'll start the video back when that's completed for the second part of this video I've downloaded the model I have the olama instructions as well as the LM Studio instructions downloaded so we can follow them and what I first want to showcase is is the default response of our untuned llama 3 Model that I just got from oama directly we can check out see o llama list and you can see I have llama 3 latest installed I downloaded it about 5 days ago um size is about 4.7 gigs so if we said o llama run llama 3 we're going to just open up a quick little session and then just ask what is anything El all am now this is not our fine tune so it should give us a pretty garbage answer and you can see that it does it says anything LM stands for a large language model basically just garbage this isn't really the answer that we wanted let's exit I'm going to open up my downloads folder and you can see we have three files here might be kind of hard to see Windows magnifier not that great okay um so you can see we have a ggf file our 8bit quantized llama 3 mod you can see it's pretty hefty at 8 gigs but if you were to compare this to the 8 bit version that ama gives you they're actually about the same size so fine-tuning didn't impact file size too much now let's get this loaded into olama easiest thing to do here is to just unzip it which I have to do this way so we're going to extract all and we're going to extract it to this folder and so now we should have a folder inside of downloads that is this one oama instructions now I can open the instructions and show you or I can just tell you exactly how this works what we have is we have our 8bit quantized model right I'm just going to copy or actually no I'm going to drag our model into the olama folder just so that that they're in the same place then I'm going to want to edit this make file and then we can load that into olama AMA will copy our model automatically do all the templatized and then we can use it anywhere we use any AMA model it's actually really really cool that they do the auto detection let me show you how to get it set up let's open up this make file like I said and we're going to open with honestly even notepad is okay so you can see that the only contents in this file is basically a path to wherever the model is we can just go up here copy this and give it the full path and I mean I guess the slashes matter so we'll do that and then we're just going to contrl S to save it and now our make file is complete and we're ready to load this into olama easiest way to do that rightclick in the Explorer open in terminal and we're going to be automatically in our folder and all we just have to do is olama create the name of our model will just call anything llm I guess and then we want to designate the make file which is literally right next to us and what's going to happen is this process is take about I don't know five minutes doesn't take that long and it is going to copy the ggf file here put it into the ama's model directory and you could actually delete the downloaded file that we just uh moved into this folder you don't need it anymore you technically have two copies after doing that and you can see actually we're done now so if we are to hop in and go ol llama list we have two models now we have the Llama 3 and anything llm so let's run the anything llm our fine tune and ask that same question what is anything llm it's going to boot up our model and once we're ready to send a prompt we just send it it send it and if you look at the content of this answer this is dead on accurate this is only from 14 chats mind you the data was already baked in all of the documents and rag that just helped tunee this model we now have llama 3 answering questions that are very very out of its training set 100% accurately imagine we use this model inside the anything llm app and have rag on top of it to automatically keep the most upto-date information relevant and get citations and that's all it took to get a fine-tuned model we only again had 14 chats and we're able to produce a model that is a strong foundational model llama 3 but with knowledge that it inherently was not trained on and we can take this model anywhere we want it is yours it was trained on your data and of course if you want to clean up the model so you don't have two copies of your model floating around you can either delete the entire AMA folder uh instructions that we just did here or if you want to delete it from oama you would just do oama RM and then you would just type in the name of the custom model and you can see it was deleted and if we did O Lama list our model is gone so we kept the storage space now again that was a Q8 model that's the largest quantization that there is and you can watch previous videos I've done about what quantization really even is it's basically compression that is it for the AMA section of this video I'm now going to Pivot into loading your custom fine tune into LM studio for the final part of this video I'm going to show you how to load our custom fine-tuned llama 3 8B model into LM Studio the process is frankly more easy if you've never used the terminal because LM studio is just a UI application and again you can use this model wherever you like whether that be in anything llm the app or elsewhere it's your model you can do what you want first thing we're going to do is we're going to download our instructions and make sure that we have our ggf Q8 model that we got from the fine-tuning process we're going to extract this folder and then once it is extracted you'll actually see there's an instructions here and also a folder that already says anything llm we're going to click into that we're going to see this subfolder and you're going to see that there is just an empty placeholder here what all you want to do is just take our model drag it into that subfolder and delete that and that's pretty much it let's boot up LM studio and what you're going to want to do is you're going to want to go to my models and then you're going to want to show in file explorer I believe we're going to keep that open we're going to go to wherever that subfolder was go to the LM Studio instructions and then just right here we're going to take the top level folder just move it into here and that's that should be all we need to do um basically all this does is it just puts the ggf in window of LM Studio where it keeps models and keeps track of all of your downloaded models um you can change this namespace from anything LM to just your name if you wanted to it's totally up to you we just had to call it something and if we go outside into here and we go to select a model we should have this this fine tune anything llm this is the model that we just downloaded and uh actually I should probably start a local locals let's start a little playground actually for this the one thing that you will want to keep in in mind with LM Studio specifically if you're using this in the playground is you want to make sure you use the right prompt template unlike o llama LM Studio does not assume your prompt template we are using llama 3 so just use the Llama 3 template you can add a system prompt if you like you can offload to GPU if you like I'm G to I guess just do Max because I know that my GPU can handle it if you know they have context length they also have like the infinite context stuff whatever you want to do doesn't matter um we should now be good to send a chat let's reload it and then let's ask that same question that we asked to AMA let's ask it what is anything LM and you can see that we get an accurate answer if we were to load in llama 3 it would give us garbage output because it doesn't know what anything LM is it's a cool project but we're just not popular enough to get trained on I guess and so yeah we had to do some custom rag but imagine your use case is standard operating procedures or some kind of uh PDF or textbook or research procedures or research data all of these things are pieces of information that can get baked into a fine tune so that you can just have a Bas level model that is very capable like llama 3 and then add on that extra layer of background knowledge that just needs to be baked into the model and you can do it with as little as apparently 14 chats obviously quality dependent and in summary that is how you can get a a no code fine-tuned model directly on your machine using olama or LM Studio to run it and using anything LM to just make all of that happen without the headache of doing it all if you like this project and you like what we're about anything LM is an open source project we're at 17.7k stars on GitHub which is very awesome I'm extremely thankful for this and hopefully this fine-tuning kind of pipeline just becomes another thing that you can act on if you want to and by no means is required in the app it's just something that can help maybe squeeze another 10% maybe even 20% out of your llm responses and of course if you tune in in the future or subscribe to this channel uh I will actually be walking through how to fine-tune a model on your GPU and with that I just wanted to say thank you so I'll see you in the next video