Local AI (with Docs Query Engine) running just on Laptops!!!

Key Takeaways

is a stepbystep guide of how you can Implement a local document processing and also local document retrieval this is almost like rag retrieval augmented generation but completely local you do not have to have any open a API key or for that matter you don't need any key in this video I'm going to give you the code I'm going to take you through every single step of how we built this particular tool that will help us question PDF or any other document that you have if you were to start the first step the very first step is that you have to install all these libraries llama index Transformers Lang chain sentence Transformers Lang chain could be already part of llama index so you may not have to install it again but these are the libraries that you need so the first Library llama index is what we're going to use for data injection from our local folder and this is what we are also going to use to connect this with o Lama so that you can use ol Lama API as the large language model as opposed to using opena apas now all the libraries are ready the next thing that you need is you need ol Lama if you are not familiar with wama I would strongly encourage you to go check out my video that I'll link it in the YouTube description that covers all the basic to Advanced tutorial about olama so the most important thing is you basically like have to install W Lama and after you install W Lama you can just say o Lama run Lama 2 so this will run llama 2 model on ol Lama and it will also be available a particular end point within your computer so the current at this point AMA should be running and AMA should be running in the background with the endp point available for you to use once that is available then the next thing is you need to define the folder where you have got the data so this particular snippet of code from llama index import simple directory reader so simple directory reader reads everything from this particular folder so it's going to take all the files for example PDF text and all the supported files and it assigns into documents that is what this snippet of code does so first we install all the libraries next we install o Lama and started running the Llama to model and the next thing is we are defining what is the directory folder in your computer from where you want to do data injection for your rag augmented generation or document question and answering the next thing we need to do is we need to download the embeding model so for whatever you are doing with the text or whatever type of document you have ultimately you need to build embeddings that you are going to query using large language model that is powered by W Lama and Lama 2 so for you to create that embedding you need some kind of embedding so typically people use opena embedding but because we have taken a pledge in this video to use completely offline Solutions nothing open a related so here we're going to use llama index. embeddings import hugging face embedding so this lets you download any hugging any embedding that is hosted on hugging faces model Hub so you can see here I've got a couple of examples you can use this model or you can use this model or this is one of the tiniest models or tiniest embedding models that is available in hugging faces model Hub so you can use any of the model so import hugging face embedding so you can use hugging face embedding to create the embed underscore model at this point you have the embedding ready for you to use the next thing that you need to do is like I said uh if your o Lama is not running make sure you run the AMA and the next thing is you need to download something called ol Lama query engine pack this will get downloaded from llama pack so you can go there is a llama hub from there it will get downloaded and once this gets downloaded this gets assigned to O Lama query engine pack but let's say you don't want to download it again and again and that is where you save it inside this directory inside this folder and the next time when you want to use it all you have to do is from _ pack. Bas import AMA query engine pack so you don't have to download it again and again so that is something for you to keep in mind so first time you can use download llama pack and then download this particular llama pack which in this case is O Lama query engine pack and then that gets saved here and next time you can just do this thing that will basically if if this folder AMA pack is in your current working directory then there will be a file called the base and that gets readed here so you're you're you're fine to go ahead with that so now we are set with connecting o Lama to whatever uh the large language model which is like llama to in this particular case and we are also ready to combine the embedding and then start querying so that's where you are going to to load the embeddings so you have got o Lama query engine pack and the model that you going to use is llama 2 model you can use different models as well depending upon what your computer offers and what is a use case and then you're going to put all the documents inside this so that it goes inside AMA pack this is a very critical step because this is where all the information are going to be extracted from this particular folder set of documents and then it's going to create that engine pack which you are to use for querying later on so it's very important for you to um do this thing this process will take more time depending upon how much data that you have got how many files that you have got how many characters or tokens that you have got so if you have got a powerful machine this will take lesser time if you have got like a very old machine this will take more time if you have got a lot of documents this will take more time so the time would change based on what you have got so in my particular case I've got like one only one invoice like this invoice. PDF and for that because this is an old computer not an apple silicon Mac I've it has taken 1.1 minute 42 seconds as you can see after this is done the next thing what we are going to do is we can go ahead just start questioning so you can go here and then say _ pac. run and start questioning so I've got a couple of questions so that you know what it does better what it does not do better here the first question is what is the invoice number and it does a pretty good job in sharing the invoice number let me show you the invoice number you can go to the top it gives you the invoice number what is the invoice number it goes through the embeding and it says based on the context information provided the invoice number is 6135 6291 perfect answer the next thing is what is the seller tax ID so it goes through the document and then it tries to find the seller tax ID because we have got two tax IDs one the seller the other one the client it gives the seller tax ID and you can see seller tax ID here very good job the next one is what is the client tax ID all these responses are taking approximately 2 minutes like I said my mission is not that good so in your case the mission matters a lot The Machine's Ram matters a lot whether you have got GPU or not matters a lot also what kind of model you are using matters a lot and what is the question like how far it is how deep it is all these things matter and the most important thing is um you need to have the right uh set of let's say llms to run this thing you cannot use it in any llm that is not drained for instruct fine tuned or you cannot use it with llms that are not suitable for retrieval so the right set of things are very essential for you to get the most efficient and also the fastest result but still I don't think this is the most optimal um retrieval Time 1 minute or 2 minute is actually quite bad I would as a human being I would I would myself go open the PDF and then start reading it rather than asking AI but the idea here is that using this technique you can probably later on um go improve this so that's that's one of the reason why I decided to put out this video even though the time is not fastest the next one is what is the most expensive item so it it has got the client tax ID correct now we are going to enter into the places where it did not do correct answer what is the most expensive item it took 2 minutes 23 seconds and it said the the based on the information the most expensive item is wine glasses goblet spare clear glass so you can see here it assumes that this is the most expensive item and the reasoning for that is correct so it says the price is this and then that is why it is but unfortunately the object that it picked it thinks this is the unit single unit value while this is the single unit value so that does not make this the most expensive item so that is technically a wrong answer so maybe we can improve that with prompting and all those things but I just wanted to keep it clean and then say what does not work the next one again I'm asking the what is a most expensive item with a different embedding model so first one I use this embedding BGE small EnV 1.5 which is one of the best open- source embeddings the next one is I'm using a smaller embedding to see if it actually makes any difference so even in this particular case it gives a wrong answer very similar long answer like it doesn't like improve or anything it still gives different wrong answer it says home essential gradient stem stemless wine glasses which is this one once again it gives a reason but uh it is a wrong answer I guess it also got messed up with the commas here as you can see here this is 6,000 right and uh if you see here this is like 1 129 so I don't know if the comma the European notation actually messed up here but anyways it messed up the final thing before I close the video I wanted to say that it's not just you can just ask question question and it's not like your typical retrieval engine you can ask things to be created based on what you have got in the document for example went ahead and asked a question write a viral tweet based on the document so here is the Tweet so we have got the Tweet let me print the tweet for you when you print the Tweet response. response attention Enthusiast are you tired of cluttered wine racks broken cork screws and then it creates the Tweet with emot conss and all the other things which is quite nice to be honest like now you have got documents you can ask question you can retrieve information from that but at the same time you can use this and augment this data into your large language model generation that's exactly why it's called retrial augmented generation and you can create new items that are like custom for you completely local running on your laptop not even GPU running on your CPU laptop without any external API or without even internet I hope this tutorial was helpful to you this notebook will be in the YouTube description for you to try it out and if you want to do this same thing first most important thing is you need to install all the libraries you need to install all Lama once again just to remind you I've got a separate AMA tutorial which you might need to see if you were to use this effectively so let me know if you have got any other questions see you in another tutorial