Ollama meets LangChain

Key Takeaways

okay so in this video I want to go through how having olama models running on your computer locally allows you to run Lang chain locally and use those models to do different tasks so I'm just going to look at some simple tasks like how we set it up and stuff like that and then finally I'll finish up with a task where we're going to get it to go and do some scraping for us and extract some information all using the Lama 2 model okay so in here uh we can see if I come down I'm just using vs code here and I've just got a couple of python files open I've made a condor environment just to install Lang chain there's nothing special about that there you can certainly set that up or just use your local python if you really wanted to so if I come into the terminal I can just come and have a look at what we've got going here I can see the models that I've got going in there I've got my Hogwarts model from before I've got the Llama 2 models and I've got some other models which I'll show you in the next video setting those up so first off let's look at just the most simplest thing how do we actually load the model so in this case we're going to use the pre-made llm in Lang chain for Al and we can just set that up with a streaming coreback for this and then we just instantiate our llm pass in the model that we want in this case I'm using llama 2 and then pass in a streaming call back now I can just call the model locally from my python code here okay so now I run the code and you'll see sure enough it's basically triggering the Alama model so it's doing this via an API so actually one of the things I didn't cover in the first video is that alur is running an API that we can actually trigger both with uh L chain but we could also do it with just a standard user interface Etc so that's something that perhaps I'll look at in the future is building a nextjs app with Lang chain and Al llama there all right so the next step up is I'm just going to make a basic change here so I've got the same things from before don't forget this is an LM so I can add in temperature I can add in Max tokens I can add in things like that if I wanted to here what I am going to do is just make a simple prompt template so just give me five interesting facts about and then I'm going to insert that in here so I could say rather than Roman Empire go for the moon so okay you can see that we're going to set up our chain we're just going to pass in the Run CH and print it out as we go in so in here we can also pass in you know ver both equals true or not if we want to see the streaming out of this okay so in this case I'm going to turn off the Callback manager so we're not going to run that and I'm going to turn verose to be false and now if I run it you'll see that it takes a bit of time because it's actually running and getting the data but sure enough once it's finished it will actually just print the data out so this is probably more useful if we wanted to write something to a file file Etc we can see there that it's gone and done it and we've got the five interesting facts about the moon and it's gone around and gotten those quite nicely all right let's look at the last example so this is definitely a bit of a jump from the previous two examples so now we're going to be doing some rag and we're going to do it using a web based loader so we're going to load a web page in we're going to run that and split that up we're going to put that in a chroma DB so you'll need to have chroma in installed for this and we're going to also use an ARG pass to actually pass in a URL that we wanted to get back some data so let's have a look at the Imports here we can see that okay we're bringing in recursive text splitter we're bringing in the web page loader with webbased loader we're bringing in chroma and then we've got some embeddings here so I haven't looked into it too much but apparently Alama seems to according to Lang chain has its own embeddings GPD for all has also made some quantized embeddings that you could use in here for this we can see we're bringing in oama like normal so we're going to just have our main function which is going to just pass through looking for the argument URL we're then going to load that URL we're then going to split that URL we're then going to put it into chroma we're then going to split that data up and we're going to then put it into a chroma DB here then after that we're going to set up our Lang chain prompt so here is the The Prompt that we're using we're pulling it in from Lang chain Hub here and you can simply print this out and have a look at it and play with it yourself and and change it and then we're setting up the retrieval QA chain where we're just going to pass in the llm we're going to pass in our Vector store which is chroma going to pass in our prompt here and then I'm just going to ask her to set up a question question what are the latest headlines on and then I'm going to pass it this in and get the results out here watch them Stream So to set this up when I actually run the file this time I need to pass in an argument called URL I'm going to pass in Tech crunch in here and that's going to kickart everything off so that we can see that okay it's filtered out the tech crunch it's loaded some documents it's loaded the Llama 2 model and now we're going to see that it's just going to basically go through and give us the headlines from there so if we look at Tech crunch we can see that these are the headlines that it's gotten out so I haven't have to write any passing code to do this the language model has done this itself and you could imagine that we could have lots of little tasks like this where we could just send it off with a Cron job or something during the day to bring back information and save it for us on our local drive and you can see here that it's managed to get those top stories and list them out now interestingly sometimes it will list out five sometimes it will list out 10 I didn't put anything in there about that but you could certainly play with that yourself so this gives you an example of just using Lang chain to do simple tasks with a local llm that you could do a variety of different tasks for this if people are interested we could look at setting up a full local rag system for documents and stuff like that using this model certainly the Llama models are pretty decent at being able to get through this and you can run chroma locally etc for this anyway as always if you found the video useful please click like and if you've got any questions please put them in the comments below I will talk to you in the next video bye for now