Chat with your Data using GPT4ALL locally on CPU

Key Takeaways

hello everyone welcome to AI anytime channel in this video we are going to look at newly released gp4 all version 3 by nomic AI so gp4 all is an tool that basically is an AI tool that basically helps you run your llms locally you know in your system or even you can deploy this in your own infrastructure GPT for all is build keeping open source in mind so for example if you want to have a rade tool that can help you interact with large language model or that can help you interact with your document so you can fch all the information that you need right so there are a lot of tools available for example LM Studio o Lama webui and so on and so forth Invidia came up with its own tool that runs on Nvidia gpus locally this is important guys because there are two reasons for it one is that the this is MIT license now what does it mean it means that you can reproduce you can use it commercially you can do whatever you want to do with this you know in very lemon terms of course you have to go through some claes and stuff if you are deploying it at Large Scale in for example in a bank in a healthcare setting Etc right then you have to go through all the legal uh terms and conditions but if you you are if you are looking to install something in your local machine so all of your data remains completely private nothing goes out of your machines or infrastructure then these are the tools to use and not everybody is technical so they can't build a tool like this right you need a lot of developers and skills and trust me there's a huge gap of AI skills right now okay the genuine people you know I don't even call myself an expert in this technology you know the people who have phds and who who does cutting ede research they should call themselves experts because this is a fast moving technology it's not about building a chat bot now let's jump in and see how gp4 all can help us chat with llm chat with our data local dogs and whatnot now if you look at here on my screen it says GPT for all nomic AI chat with local llms on any device I'll be a bit quick on the documentation you can download for Windows if you have a Windows machine you can downlo download for Mac OS if you have an Apple device you can download for UB 2 if you are having an UB 2 or Linux machine you can also do it programmatically if you want to run it in your python file or or a you know notebook or whatever based on L Lama CPP without Lama CPP would it not have been possible guys all credits to Lama CPP and the creators to download it you can see I already have downloaded here and I have double clicked and install in my Windows machine this is how it looks like it's called Welcome to GPT for all the Privacy first llm chat application the same goes with LM Studio you know o Lama wey and cobal Ai and whatnot H2 AI Nvidia platforms everything locally that runs is your privacy first because your data is not living that application or the system that you have it says start chatting local and when I say that I mean that if you are using an open source llm if you use an open a API in a local uh application then your prompt and responses will go to open a data centers you can see it says all new version 3.0 over here it says we are thrilled to introduce our latest release packed with updates and improvements local docs experience completely redesigned local local docs UI and backend new user interface modern design making it easier I haven't tested it yet so when you run it for the first time it will download a lot of models so it will take time so I have to pause the video once we reach that stage now if you look at here in the left hand side we have chats uh very good interface you know it says install a model I don't have any model installed then you have models so I don't have any models so far you have local docs I have no local dog so far if you look at settings now in settings you have light theme you have dark theme theme you have Legacy dark and you have dark and then you have light right you can see it here font size let's keep it a bit medium let's see how does it looks on Med I think medium looks nice guys okay so I I think medium is fine so we keep medium device you can see I have you have Auto CPU you also have you know like if you have GPU it automatically get gets it if you GPU and your default model I don't have any model okay download path where do you want to save the models CPU threads you can again increase decrease the threads API server blah blah blah and whatnot and you can do that this is an application come to model so this is model set setting you can put systems prompt here you can put prom template max length blah blah blah all the inference parameter over here then we have local docs in local docs allowed file extensions you can see it says text PDF mark down rstd use nomic embed API embed documents using the fast in of private local model will not do it because then it has no doesn't make any sense to use this tool if you if you enable this we're not going to use Nomi API they're trying to sell it because they have built something beautiful so they will try to do that embedding devices the computer device used for embeddings auto uses the CPU requires restart it's fine and blah blah blah so you can see a bunch of things over here this is how it looks like this is how the interface looks like now we'll go into uh go into it guys we'll have a first start chatting and then try your local docs and see what kind of results we get here now as you can see if you come to models you can see I have one installed models I have installed 53 mini because I wanted to show you how you can you know uh chat with llms and you know you can build a local rag knowledge based information Discovery or retrieval when you click on ADD model you can find out all the models which are listed here you have Lama 3 instruct and it also shows you guys how much of RAM is required what is the file size most of the model that runs locally on a compute limited device like a CPU or a consumer GPU guys they will be quantized 4bit 5 bit quantized even 3 bit quantized mostly 4bit quantized Q4 km the medium uh quantized model of 4bit and I have shown all of these things if you are a technical person if you want to understand what is quantization how to quantize watch my previous videos you know I can give the links in description here you have Lama 3 instruct and all you can get now HS to Mr DP mral instruct blah blah blah so I already have my 53 over here you can see if you have more models you can select it from here in the chat now I don't have local dogs I'm just going to ask what is 2 + 2 this is my question you can see it says the answer to 2 + 2 is 4 this is a basic arithmetic addition problem where you add the number two to another number two resulting in their sum which equals 4 2 + 2 = to 4 what is this is how it looks like you have feedback you can give feedback you going to thumbs up thumbs down you have history in the left side you can look at the history you can edit you can delete the same way in the chat GPT let's ask one more question and see tell me more about it so now it has memory as well so I was checking if you can ask followup question if it can understand it has memory you can see it says it kind of keeps on going and it gives you something so llm chat is working this is what I wanted to show you I installed 53 Mini model and then I'm showing you chat everything locally in an desktop uh application that's what we are doing it let's jump in and go to local doc if you go back you will see this will be here everything is getting saved in your local machine guys so this data is getting collected in your machine itself okay if you click on new chat you can see the history is there and if you go to new chat in new chat you can load again so you can see the model has been loaded again it says loading 53 mini instruct and it has been loaded now ask write uh write an email to get promotion to my boss or something so I'm just writing it will give you an email request for promotion and professional growth opportunities dear both don't use this email you will not get promotion if you use these kind of emails you have to be really personalized on those things but this is just so this is just to show you that it has streaming responses it has memory built it has history inbuilt everything so do not have to go and chat GPT and ask this kind of question of course it has it hasn't connected with internet it has your everything local it's good for you know interacting with your local file so if you are if you are a marketer if you are someone who likes to uh do a lot of research you can upload your documents and find some information if you are a uh if you are someone who has to go through a lot of financial reports documents annual reports you can upload it over here and chat with it if you are someone who wants to upload the PDFs of any kind you can do that let's go in the local docs I'm going to create it call it demo collection and create collection folder path so I have to check folder then let me create a folder pretty quick and I'm going to create a folder here and I'm going to call it demo do let's select a folder here so in the folder I'm going to go browse go to desktop and select demo docs select folder and I'm going to create collection you can see installing it's using this model guys nomic embed text v1.5 so it's an embedding model nomic embed 1.5b if you look write something like this you can see this is the model that it's going to use it's an embedding model based on sentence Transformers you can see it over here Improvement op anomic amand that utilizes this representation learning which gives developer the flexibility to trade of the emitting size for a negligible reduction in performance you can see it over here the sequence length and dimensions blah blah blah so this is a model it's going to use and it says zero files zero words right now it's installing let it install and then we come back and as as you guys can see here that our collections are ready so I created two Collections and it was pretty slow guys there are some issues with the local docks uh in my experimentation so far it's slow and sometimes it hangs also I think we should also have to keep that in mind it it's hanging sometimes it's slow it's not able to find the right path but as you can see we have that so if you go to local docs I have two collections you can see demo Collections and demo two files these many files these words zero words blah blah blah okay now I go back to my chat so here you can add more collections if you want go to chats and in chat when you click on local docs you have to enable it you can see I have selected on this this collection right now and I ask a question what is AI agent because the document is about that so that is about the document you know that I have this is a docs and if I open demo docs you can find out here AI agent Ops you know talking about all this context you can see it over here this is what we have okay this is one and the other one notes that we have is this is our docs where we have our read me file if I open the read me file you will see that in a vs code it has a crew AI read me file so I'm taking crew a is an AI agent framework python libr that helps you build AI agents and that's what I know I'm uploaded you see it can says an AI agent refers to a software entity that acts autonomously or semi-autonomous in an environment often designed with the ability to perceive its you know its uh through sensors blah blah blah it says in the context provided in your snippet about crew AI crew AI is an example where AI agents are used as crew members blah blah blah and it also tells you the sources so it gives you this readme.md file and this is the problem guys with gp2 for all and not only with gp2 for all for all the retrievers these are engineering problem you have to probably now if you look at this this file has nothing to do with this question the other file that I have that's an story on AI but as they would have some topk document it has to return n number of sources and it has to return this that's where you see the semantic similarity and put a threshold value and then only return the sources which is relevant not all the sources let me ask this you know tell me more about it so if you do that you can see it says searching local docs demo so this is the collection that it's referring to and it basically generates on average it takes around 25 to 30 seconds and we are using 53 mini and I'm using CPU though I have GPU in my machine I'm not utilizing it because I'm showing it to you because not everybody can afford GPU you can see it says crew AI is an Innovative platform blah blah blah and it's fast as well it's streaming it's faster and I think that's sufficient enough for local systems if you have local system you want to retrieve information you do not have to worry about you know giving some $10,000 to a freelancer to build a rag system for you you can utilize these kind of things if you want to do things locally okay now you can see again the sources you can again rate this and whatnot so this is how you can interact and chat with your files locally you know using GPT for all and it's all free you do not have to pay anything it's an MIT license you can bring up all the models you can create local Collections and you can start chatting with it very similar to LM studio if you want to know about LM Studio I can give the link in description have a look Cobalt A2 Ai and viia platform they all they all have similar things and you will see more things in near future customizable options are not that great know with these kind of tool but I think these are great enough to get this in free you can subscribe to their newsletter to get new Js you can find models over here there are different model Lama 3 and mistrals and whatnot okay so that's all for this video guys this is what I wanted to show you I hope you now have an idea you know how to build or how to use not build how to use gp4 all you know and I have also shown in my tens of other videos previously that how you can build a tool like this you know uh you can you can look at my rack playlist and you'll find it over uh there if you have any question thoughts or feedbacks please let me know in the comment box you can also join our Discord server find the link in the video description we are really doing great on Discord guys job opportunities internship solving problems helping each other hackathons freelancing gigs and whatnot everything is there on Discord if you like the video please hit the like icon if you haven't subscribed the channel yet please do subscribe the channel gu that motivates me to create more such videos in near future thank you so much for watching see you in the next one