Let's build a Private StackOverflow App using Open Source LLM

Key Takeaways

hello everyone welcome to AI anytime channel in this video we are going to develop your private stack Overflow yes we are going to develop this application that you can currently see on your screen your private stack Overflow that will help you perform generative search on your code basis now earlier we have seen uh that how we can you know retrieve information from documents csvs images videos Etc different data formats but what if we have to retrieve you know or perform searches on your code basis now why this is important is because you are working for an Enterprise or some companies and you guys have your proprietary code basis you know 20 30 50 years of code basis of different projects how you can you know create embeddings out of those embeddings out of those projects or the code bases and then you can perform some kind of search so when a new developer or when a young developer comes in in the team how he or she can you know perform some search to understand or in you know to increase the efficiency okay or at the workplace and that's the reason we are going to do that this entire application is built on the open source tag we are going to use a large language model by stability AI you know we are going to use that model stable code and for Trans for embeddings as well we are going to use a model from Microsoft uni xcoder base Microsoft has code word uni exporter base graph quota etc etc so we're going to rely completely open source we'll run it on a CPU with a good uh you know with decent enough latency as well I'm going to build a fast API backend so you can also use an API as well as a microservice or something like that if you want to use it go ahead deploy it you know and you can deploy it on some cloud or something just to use that API later on and I'll also show you that how we can use Ninja 2 templates in fast API to build this quickly so you can see currently I'm on your private stack workflow enter your question I'll just I'm just going to paste my question how do we Define HF argument parts and I can have here why did you ask why do uptick why did you select this question now the reason is because I have taken three files from hugging faces Transformers GitHub repository you know a huge credit to Transformers without Transformer you would not have seen this rise in generative AI ecosystem so I have taken this three audio classification run image classification and run question answering dot pi and what I have done I have if I show you here on the home okay let me just go to desktop and index top uh my project coding assets data I have these three files that you see run audio.txt run image.txt and run qa.txt now I have downloaded all these files from GitHub repository you can take any GitHub repository more than 80 programming languages it it performs well and I have taken this run image classification.pi and all the other two files and then I just change the extension from py to txt because I am going to use Link Chain data preprocessor modules to perform some pre-processing steps now if I ask this question how do we Define HF argument paths and I click on submit what it does it takes you know around 30 to 40 seconds depending on what kind of machine you have but it it's it's faster enough okay and how does it work first we create the embeddings the vectors we have to put it in a latent space and we are utilizing this feature extraction model which is very domain specific now you would have this question in your mind that why are we not using sentence transformer for example MP net or mini LM uh version 2 why are we not using those models from Centennial Transformers because those works really good on text and sometimes on images but you have to rely on domain specific model for example if you are dealing with the code basis you know for code basis you might have to look at the models which are really good to extract features from code bases like we have models like codeword code GX Unix coder graph code Etc right so you have to look at these models to extract the embeddings or create the embeddings and then save it in a vector store in memory Vector store we're going to use SAS CPU here in this video but you can use any other Vector store or vector database you want now we're lying on Microsoft uni xcoder base available on hugging face that was that was a great rhyming you know uni exporter base available on hugging face and for as a large language model we are going to use stable code instruct mode it's an instruction based model and I'm going to use this model by stability AI performs really good to be honest uh you know on multiple questions that I have asked or app search for a good enough last language model and 3B and you know they have some licenses we are not going to use the exact same model again we have to you know rely on the blocks uh creativity of effort whatever you say it we're gonna use a ggml variant of this model I'll give you the link in the description how you can download that model when I'm writing the code okay uh if you see it over here okay so we have this def HF argument parser this function Returns the input output and Target as numpy areas I asked the question I basically search for something in my knowledge base the code base how do we Define HF argument parcel and you I got the code and this is what I'm gonna I'm going to develop in this video guys the UI might look little messy but I'm not a web guy your camera web technology expert okay I just I'm an average guy in on those things but you can see uh we have uh your code area and also giving the source document from which document this information has been retrieved or this code has been retrieved therefore we are also doing it over here in the right hand side in the source document you know you can find it out all about your whatever question that you have searched for okay you can find it over here and so this is what we are going to build guys in this video and this is going to be a very exciting video to be honest and you have feel free to use any other large language model you can use llama 2 as well the ggml variant of llama2 that might perform good as well maybe you can use Star Chat star coder or wizard coder as well because those are very code specific models now you can get your code here diff HF argument parser this function Returns the input blah blah blah pass each example in the data sets input output targets and then return the numpy array as well over here fantastic so this is what we are going to build so without any further delay guys now let's I'm very excited so let's build this application okay so uh to build this application now let's first see that what are the things that we need and how the high level architecture will look like so we need uh let me just call it let me just call this private stack overflow okay this is what I want to name this and the first thing that we need is of course data and the data will be nothing but some code files okay this will be some code files that we will take it from GitHub so I already have downloaded it also you can take any other code basis you want okay and this will be available on the GitHub repository as well now you need data for the code files and once you have data we need a framework you know uh I'm gonna use lunch in here which is a framework basically to build llm application faster okay and for that we're gonna use its prompt I'm going to use from template because we have to you know engineer The Prompt a bit and then we need uh it's uh data loader and preprocessor basically so let me just call it preprocessor and then we need some chains so one of the chain that we need is uh in this video we're going to use retrieval QA so this is on framework and then we need an embedding model how to extract feature out of those code bases and embedding model and for this we are relying on uni xcoder so you can see this uh Library over here or the model over here sorry excuse me uni xcoder base the library is hugging face by the way uni xcoder base this is what we are you know using Microsoft it's by Microsoft and Microsoft has a lot of other models as well you know when it comes to code related problems they have cold bird they have graph related models as well that you can check it out now after embedding model we need an llm and in this video we're gonna use you can see the it also has been open here stable code uh the instruction model stable code instruct and Alpha 3B this is the model that I'm going to use in this video okay and this these are the things that we need and we need uh like for backend we're gonna use fast API one of the most underrated library or the framework that has never built you know for python fast API we're gonna use for backend and we'll we'll build a simple kind of an API and also we'll use ginger tool here so I'm not going to write any code in for the front end that basically a simple UI I'll give that uh it's it's built on basically let me just write for front end as well because if you are seeing this and you want to extend this further it has been built using boost trap 5.3 the latest bootstrap version I think it's it's more than 5.3 also they have 5.4 if I'm not wrong but this is based on bootstrap 5.3 okay so this is these are the things that we need okay and uh you have your code now you have your code file you basically chunk this out so first you load this using Lang chain and once it's loaded in the system so for example suppose this is loaded you you create you create chunks of this so basically we chunk this through some you know after splitting it out and then you're passing to an embedding model so that will that will pass through an embedding model so this is pass through an embedding model so let me just write an embedding model here and then we save it in a vector data store here we are saving it in a vector store you know whatever you design it I don't know so fast we are using fast CPU which is by Facebook you can use chroma DB or Q rant or millbrose Etc whatever you want to use you can use fine cone as well but I am a huge advocate of Open Source guys and I don't want to pay money to build applications to be honest okay and then we have fast CPU and then we then okay let me just do this this is done so this is the first step that we are going to do here okay now once user query is so when user asks for a query so there's a query for example okay like HF argument parser or something this is again passed through the same embedding model and we get the relevant chunk out of it so you get the relevant chunk here and this comes from basically this fast the vector store that we have okay here in the relevant chunk and then we use land chains Etc here that that takes care of your prompt Etc you know then give it to an llm to retrieve the information okay or to perform the search so that's what we are going to do here guys very high level so I'm not creating this architecture in detail because there are a lot of videos that I have created recently where I have explained this in most of the videos please watch a couple of other videos where you will see this architecture on very high level but this is now I assume that you would have seen a lot of videos on informational retrieval tasks and this kind of work same way there's a lot of application has been built on GitHub dog GPT document GPT has built an application the other day Dr GPT or something so a lot of development is happening right so anyway okay now jump in guys so uh yeah that's the data that we need so I have downloaded four files all of this python file you can see aoai by dot py which is by Microsoft they have they have open source something I don't know what hap what are the thing that they have open source I was reading it on LinkedIn that they have open source there you know UI or code basis or something okay so that's the code to interact with Azure open AI set it up on their uh private VPC or something I've downloaded their file from GitHub then I have uh carpathies uh meta Lama which is from llama CPP so I have taken that file from his GitHub repository and a huge fan and I have a lot of respect for Andres karapati and then I have a graph operation.java which is uh graph operation which is uh from tensorflow GitHub repository you know for the Java and then we have this by Transformers repository run image classification now what I'm gonna do here I'm going to rename the extension by from py to txt for all of these files because I'm going to lower that through a directory and I'm going to use text loader for that you know but you can also anyway if you don't want to use lunch and you want to write your own pre-processing function and I would recommend that if you are doing something in production to write your own pre-processing function where you will have more control and customization options where that launcher might not provide but just to keep it simple I'm just going to rename this okay you can also in end you can load it in a list or something like that separately as well now this becomes my data source and if I come in the root directory this is the model that you need stable code instruct Alpha 3 billion GG ml version 1 quantized five one dot bin I will give the link of this model in this description the video description you can ignore this file the reason I have kept this file if something goes wrong with the hugging phase embedding module I can utilize this default python file official python file on Unix coder and the reason I'm also you know talking about this nowadays because you know in production you have to be really careful when you're building jni applications you have to put your own garters or something right so it's better to write Uh custom functions for validation pre-processing prompt blocker etc etc okay let me just open this in code here and I'm going to open this in code and true good okay and then I we need to let me activate my link chain environment where I have installed all the required dependencies for this video okay and I'll give that I'll upload that requirements in the GitHub repository okay let's write the code the first thing that I'm going to do is I'm going to use my ingest dot Pi okay that's that's what we're gonna need you know if I show you here uh you can see that uh this is this is what we need embedding model uni export or base and if I just pull it up you will see the first thing that we have to do we have to load the file and then perform some pre-processing steps before creating the embeddings and save it in a vector store in this uh in for this use case okay now let's do that so the first thing that we have to import few libraries and of course we have to start with lunch and thingy here so from langkine dot embeddings or let's let's do step by step so document loader the first thing is document loader from launching dot document loader and I'm gonna import it's Lotus by the way uh import text loader excuse me text loader I'm also going to import directory loader okay so this will help us load the file from a directory and the reason I have text order because it's a text file if you have PDF you have to do PI PDF or PDF or something like that okay depends I think it supports around 14 different type of data formats in line chance you know correct me if I am wrong maybe they have increased by now okay and the next thing is from Lang chain dot we need some splitter guys right so let's do text splitter we have to split that text splitter this is what I need or to make it more efficient let's do recursive character text feature recursive I hope I spell it right recursive character where is tab 9 when we need it okay recursive character text later excuse me recursive characteristic splitter okay this is this is this is fine now text splitter then what we need is uh embedding thingy here so from length chain Dot embeddings and for that I'm going to use hugging face embedding module so hugging face embeddings and in that you can pass most of the embeddings model which is available on hugging face mix because that is supported by Lang chain okay now from Langston dot embeddings import hugging phase embedding now we have created the embeddings now we need a library that will help us another Library the module your class whatever you call it to save that Vector store in a vector store now for that we're going to need from Langston dot Vector stores foreign that's it I'm using fast CPU you can also use fast GPU if you have GPU okay uh that's that's fine uh we have directly loaded Etc this is what you need so let's knock we have a data path so let's call a data path which is and the same terminology is available on my other videos as well so data path is nothing but which is data and let's define a DB fast path the path that we want fast to create it automatically so DB fast path and for that let's call it something like that I always use Vector store Vector store and then DB underscore fast DB underscore fast if you are using chroma DB Maxi or that they have migrated from Duck DB to sqlite so if you're getting any error probably you have to look at that uh nowadays now let's call it create Vector database or something here Vector DV and let's call it create underscore Vector DV excuse me Vector DB and in this the first thing that we're gonna do have a loader as usual so loader and I'm gonna have directory loader and in this directory loader I'm gonna I'm going to do few things and that's why I'm writing the code I otherwise I can also take it from my GitHub repository I'll I do I will take few of the code Snippets but here the first thing is data okay let's have a data path which is data now once you have the data path you have to give a loader class okay and that's why we have to give this loader class here so if you have a PDF you will probably give Pi PDF or something loader or you know something rather PDF loader but I have a text so I want to use text loader okay and I'm going to use few other arguments as well uh within this I I'm going to do okay recursive equals to true and let's keep progress equals to true as well and see if that makes any sense excuse me so progress equals to true and then I also want to use multi-threading because it's really helpful when you want to create embeddings for multiple files a multi multi-threading let me come little closer I cannot see it so use multi threading and of course I want to do it true and then I also need maximum concurrency so let's put some values for Max concurrency uh concurrency I don't know why I'm not getting that suggestion here but I hope it's correct what I'm doing for now let's keep it 8. now Max concurrency use multi-threading true for Progress this makes sense now let's call it raw underscore documents and you know there's raw documents I'm gonna call it Loadout load just going to lower that guys okay with load module here loader.load and take splitter so text splitter equals text splitter dos dot uh recursive yes character text editor and in this recursive character text splitter we're gonna pass our chunk size so chunk underscore size and I'm going to keep let's keep 500 and guys if to be honest see I'm not building a product here to be honest and you have to play around this please keep higher chunk size and also the chunk overlap and you can play around it to see if that makes any performance Improvement I do believe I have seen it when we are building pocs and solutions you know we have to be very careful when we select a chunk size and chunk overlap depending on what kind of data that you are dealing with but just to keep it simple I want to keep chunks either 500 maybe you can also try with thousand or something and then for chunk overlap and let's skip 50 you know keep it small for now to create this faster uh the embeddings ETC chunk size chunk overlap now we have a text later now the next thing is we have to get a text out of here so text so text splitter dot split documents split underscore documents and here we go split under a document that's password document that's it now we are okay with so this contains all of your text related text and and the next thing is I'm gonna have an embeddings variable where I'm going to call hugging face embeddings and in this my first thing is model underscore name which is going to be your Microsoft model so I hope it's Microsoft let me just copy very simple guys what you have to do you have to come over here just copy this entire thing and let me just remove Microsoft and paste it so simple right not rocket science my model name and once model m is done then we can also set some squats so the first thing okay let's have a sister dictionary so you can see it says dict so it's a key value pair okay so let me Define a key which is going to be device and my device is nothing but a CPU machine if you want to use Cuda you can also do that okay Maxi or you have a good enough vram to deal with it device and CPU okay fine and then here we go so DB which is a short form of database fast fast Dot from documents from underscore documents documents hope I have spell it right documents and our Edge text and then pass embeddings that's it now let's save it locally so I am writing this ingest dot Pi because I wanted to show you a couple of things in the loader and the embeddings but when I'm writing the code for app.pi I'm going to copy a lot of code Snippets for my GitHub repository because it's very duplicated right so DB dot save local and in this I'm just going to use DV fast path okay and now let's also run this so if name main main create Vector Dev is a function name that's it okay so we have written this and what we are doing here guys you have to set a data path where you have your data files you can see aoai text to run image classification we have four files and then we have a loader directory loader class we are loading it with some arguments and raw document loader.lore using recursive character text splitter have some chunk size and overlap then splitting it out using an embedding model creating the embeddings and saving it enough Vector store that's it so let's run it okay and you are doing it everything locally we are not using any online service or something like that okay so now let's run this guys and it says got and ah class I don't know where is glass it's CLS sorry that was in a little bit in excitement okay a lot of CLS and now let's do python in just.pi and you can see uh we are 100 with this four of four files uh creating some sizes you can see no sentence is fine okay I don't have it says GPU quantization are unavailable due to bits and bytes Etc I am okay with it okay I don't need those things and what it what it will do you can see the left hand side in this coding assets directory you will see a new folder called or the folder name that will be in Vector store and within that will have something called DB underscore fast and that's what you can see now we have created this automatically and it's so it has been successful and you can see we have two files index dot fast and index.pk are the serialized files okay now let me go inside this Vector store you can see we have two items and you can see the good enough size of index fast and index pkl okay you can put in number of code file thousand of PDF files okay PDFs txt and some Json Etc you can do that and let me just come over here okay so now we have written this guys uh we are okay with our ingest now we're going to write our app.5 we have created embedding so that that part is done now I want to write an app.pi where you use this uh model guys okay so now let me do that and I will also download I'll put my you know index.html static and templates file into it so now let's let's perform that task so let's create an app.pi here so I'm going to create an app.pi and all of our app logic will go inside this okay so for for that we're gonna import few things let's quickly import it the first thing is an llm and we're going to use something called C Transformers so C Transformers is a python binding you know for all of this uh uh and name was the Transformers model I'll say the Transformers model a model which are based on sun Transformers it's a python binding uh for the CC plus plus okay and it's it helps you load this model and uh influence it at least it takes less load and it's really helpful so we have Lama CPP we have kobold CPP we have a lot of other wave you know we use this uh large language models you know on a commodity Hardware uh using gdml or gptq model and that's what that's why we are leveraging something called C Transformers which is by Marella so credits to her you know her or him I don't know sorry so if the gender but credit to whoever has created right so from Langston dot LMS import C Transformers and then we need lunch and Dot chains we're gonna use some chains I'm going to use import retrieval QA uh retrieve work UA I hope this is how we spell it and then I also need llm let's keep llm chain 4 as well if you need something like that from line chain dot up from Langston import from template uh import template this is fine and I need let's keep OS Banner if we need it import OS and then need Json for fast API so let me just uh also use that and is from embeddings also hugging face embedding sweet hugging face embeddings and then fast as a vector store so from Langston dot Vector stores import fast okay then I need some fast API dependencies guys you know fast API we are going to use as a back-end framework here you know to write our python backend code so from Fast API you can also use Django or flask if you understand it as well gradu Etc now from fast fast API input fast API I need form I need request and I need the response and what else I need I need file and it also depends I need HTTP exception for handling error Etc and I need status okay that's what I need and then I need from Fast API import or not fast let's do it from responsive as a redirect response for localhost 8000 so from Fast API responses import redirect responses and then I also need from Fast API dot templating I need Ginger templating that's gonna use index.html templating ginger two templates and I need the static file so from Fast API so whatever files that you're gonna you know create embeddings Etc we are going to show that file right on the UI and that's we're going to save it in a static file excuse me so from Fast API dot static files you're gonna use import static files so let's import and you also need Json label encoder okay so get that Json uh the beautifying or something like that okay so from Fast appear to encoders input how do we import that I think it's in lower case yes just enable encoder what else we need we need EV cone also okay so let's import UV con as a server to run this backend okay and okay let's have Dig Down Low import date that's what we need and how do we Define a fast API guys so we Define like this a fast API app and this is how we Define so app fast API I'm gonna Mount that as well as app dot Mount the first thing that we have to do we have to mount our static folder so let me just call it static and I'm gonna use static files within the directory and my directory is nothing but called Static static and then I'm going to use name as let's static okay so now we have mounted this now let's write a template so templates and templates is nothing but Ginger two templates and directory so just give a directory there and if you have some other names in your directory you have to probably give that name not templates okay now need we need few things okay the first thing that we need we again need that uh DB fast path so let me just copy this here DB fast path and then we will we have to copy few of the you know uh few of the code snippet from our GitHub repository so let's just go and do that so the let me just go on AI anytime excuse me just excuse me okay AI anytime and here what I'm going to do I want to use uh let's use any one of these so I'm going to use llama to Medical chat bot code I'm going to use on model Pi I don't want to write the same thing again and again so I will explain that what the piece of code is doing here so the first thing that we are doing is we are having something called custom I don't know if we don't copy it correctly helpful answer I was just remove this okay now what we are doing here guys that we are writing some you know simple prompt and we say okay look at the context the context that will come from your relevant chunks from that embeddings and then the question that end user is going to search okay so we have that context and question and we are saying use the following pieces of information to answer the easiest question and if you don't know the answer just say that you don't know don't try to make up an answer only return the helpful uh below or something now let's use lunch and prompt template module so I'm going to take this piece of code here you can see set custom prompt and I'm just going to use that come over here paste it and here what we are doing we are creating a function called set custom prompt when we say prompt template for QA retrieval for each Vector store so we use this prompt template that we have also imported here on line number three and within that we are passing our template which is we have defined on line number 24 that you see custom prompt template we are defining it and we are giving an input variables which is context and question that we have defined in line number 27 and line number 28 make sure that you are giving that which is in the curly braces not the labels okay and that's what we are doing and then we are returning the prompt this is one function that I also need this function I can also write that code but you know I'm just going to use that so let's also use this function guys load llm okay so let me just also import this function and what we are doing here we have a function called load underscore llm where we are using C Transformers and we are using a lamba 2 we are not using lamba 2 model but you can use the nama2 model I would recommend trying this with llama2 to get even better accuracy I don't know you have to check it out let me just give this uh stable code it should suggest me in bit okay stable code instruct Alpha and something like that okay 3B I don't know why it's not suggesting me but now it suggests when you give dot it's fantastic Okay so here is the model type you don't have to write a model type of Lama which is this model uh stability AI model that we are using it is not a llama type of model is it GPT Neo X model so what I'm going to do I'm gonna say Okay GPT underscore Neo X that's the model you're going to use okay here model type and then we have Max new tokens which is 512 and let me just set a temperature to you know for example 0.1 or something okay that's what I'm gonna do here so I just said okay temperature is equal to let's call it zero because it's a generative search we are just retrieving the information we don't want it to be more creative and you know generate a lot of a random sentences or something like that okay we don't want it but you can play around this uh parameters inference parameters okay now we're just returning this llm okay now this function is done as well now let's take this function which is your retrieval QA change function so I'm just going to copy this retrieval q a chain and I'm just going to paste this now what we are doing here we have a retrieval qhn function which utilizes your llm your prompt and DB as three input patterns llm equals llm chain type which is stuff retriever return Source document we also need a source document returning the metadata Etc and looking at prompt and DVS that's what we are doing here guys with this function so now let me just do one thing you can see I'm taking it from here you would have probably seen this code as well if you have seen my llama2 medical board video okay so let's now use the QA model function the question answering model function so I'm just going to come here and paste it so let me just do QA let me just come to the closure qm about excuse me let me just bring it okay now the def queue about now this is fine now the last function that we need is we know we can also take this final result where we are calling all this uh functions and getting the response like we're writing a large function which is final result you can see we are calling this keyway bot and we are passing the output we only need the response there key result query okay now what we are doing here guys in this keyword bot function we have an embedding and you can see it's sentence Transformers we don't need the exact same thing so let me just go to ingest dot pi and I'm going to use this embeddings here that we have defined so let me just copy this embedding model and let me just remove this embedding here so I'm just going to remove this and paste it over here so now this holds our embeddings and we are passing it over here and this local DB fast path you can see that at that place we use fast dot save local and here we are doing fast dot load local so we are loading that Vector store in memory Vector store for each of the vector you know from uh above function that we have written and we are looking at in the that folder or the vector store and we are passing that embedding that user is going to search and then DB has been called llm has been called keyway prompt and pretty much DB and the link okay now this looks nice so now what I'm going to do here is I'm going to write my first API function the end points so let's write that API endpoint the first thing that I'm going to do I'm going to say okay app dot get and this is for your uh template response index.html so let's do that so the when we hit localhost 8000 you should see that index.html UI okay let me let me show you so where that I will just copy it so you have to I already have these two folders static and templates I'm just going to cut this coming home next stop my project and coding assets and I'm going to paste it now once I paste it I'll show you what I'm talking so in static I have nothing but in templates I have a file called index HTML and you can see it's a simple index.html file with little bit of CSS I'm using bootstrap 5.3 it has a container and we are using a background you know as a dark background and I have a call sm12 I have some within that row I have some columns and then private stack Overflow your name then you have a form so this is an HTML file I'm not explaining HTML and you can go through it and you can completely modify this connect with database or something right if you want let me just do one thing let me click on open with live server now when I click on open with live server you can see it opens the the UI is this your private stack Overflow but if you do right now anything it will not do anything because we have we are not hitting that end point okay so this is an UI so now let's come back in the code and start writing the code guys now in app.get I'm gonna give any endpoint here because this is going to be my landing page so app.get I'm going to use your asynchronous function that's why fast API is so so powerful and async Dev index I'm going to have a request excuse me that's request or request cologne and then you give request perfect request column request and let's write in a beautify way or something like that okay async Dev index now what we're going to return here is just a template so return templates dot template response I'm gonna use this template response and I'm just going to pass that index.html so let's pass that index.html and it's a key value pair again your request and then becomes request fantastic we are okay with this now guys we have now async def index something like this I think div okay now fine now the next thing that we have to do we have to write a post request so let's do that post request and that will be our final function so let's just write that code for the end point I'm going to make a post request and my post request is nothing but let's call it get underscore response so get underscore response and in that excuse me it's not a function app dot post get response async Dev get response request request and then I'm gonna have a query which is going to be a string so my query is a string so let's do that so query HDR and in that Str I'm gonna have a form so form let me see form and okay so this looks okay I have to close that I am not closing it and then that's it async def get response and then become response equals I'm gonna use that final result function that we have defined at line number 76 final result and I'm going to pass that query that end user is going to search okay so response final result query and I'm gonna write result equals risk and only need the result so let's do that result now let's also print that in terminal to see if we get any error we can track it okay so print result okay now this print result is done so now what we can do for i n response so this we are doing this for Source document guys so let me just think risk and I think it's Source underscore documents so let me just do that and I only need the first document okay and maybe you can you have to look at the work around so right now I think k equals to 2 right so it okay so right now let's only return the first one okay the top one the zero if for I in response then I can look at the metadata so if mirror data not any result in I and if metadata in I then I can do okay Source doc let's create a variable and Source Doc and then I'm gonna say okay I and then source that's it okay this makes sense now let's come out of this for Loop and now let's use just enable encoder to show that okay so encode that so what we can do response underscore data and in that date I am going to use just enable encoder and in this jsonable encoder I'm gonna write Json dot dumps and the first thing is uh json.com I'm gonna say let's print it like this result and Source document equal Source Dot uh [Music] I think there will be one more or something okay looks nice let's see if we get in here guys I think this project is right so this response so now let's see response response and in this response I can pass my response data and I can just return this response okay uh now that's it this looks nice so what I'm gonna do here I'm gonna say okay if name underscore underscore main underscore underscore excuse me okay I don't know why I removed that okay and I can say if you can't avicon dot run and I'm gonna say Okay app app colon app so you can run it in python or something like that okay app clone app then I have host and my host is nothing but localhost and the port is 8 000 and it runs on 8000 ports so Port 8000 not 8080 which is for Jupiter notebook put 8000 and then look reload equals true so you change your code in the back end and it will reflect on the front end we are done with this guy let me quickly explain what what I have done in this we have created this ingest dot Pi where we have created the vector store within this Vector store folder and now we are writing this app.pi imported all the libraries initiated a fast API app mounted the static and Ginger 2 templates for templating giving that local Vector store path then I have custom prompt templates simple prompt for context based retrieval and then I have set that custom prompt using prompt template module in link chain then I'm using C Transformers to load the model giving a right model type and model max new tokens 512 then I'm using a retrieval q a chain if you want to develop a chat with code or chat with GitHub or chat with codebasses please you can also use conversational retrieval chain or conversational you know the chain of Lang chain okay where you can give you know memory and chat history Etc and here we have QA model function QA board we're utilizing that embeddings dblm and then returning the QA and writing a final result you know just to pass the query okay okay and then we have couple of endpoints that this is the main endpoint get response we are just utilizing it so that's it guys and now let's let's run this uh application to see if we are getting any error or if it's running it running fine so let's uh open this excuse so let's open this uh so now let's open the terminal here guys and we're going to run the uh this application so how are we going to run this application is very simple python app dot pi and it will say application startup complete or something on 8000 and then we can open that so let me just come over here and do localhost excuse me localhost 8000 okay and you see this your private stack Overflow uh UI or something and then you can enter your question here okay now let me have a look what question we can ask uh create a graph let me just see if this can create and graph operation definitive operation with the given handle to the I don't know if you know referring to an operation in G let me copy this and see if it's able to uh if this is the right question I don't know create a graph operation instance because this looks even grammatically uh incorrect but it's okay public final class graph of relation extend well let's try it out guys when I click on submit what it's going to do it's going to look at the relevant uh chunks in the director store and then give it to a large language model and that will basically present you the answer or the information in a very concise or summarized way or something like that okay so it will take little time you know to we're running it for the first time it might take a little more time and it takes around 40 to 45 seconds to you know retrieve the answers okay let's see what else we can ask here uh implementation to then we have export meta Lama being text lava to wait uh what we can ask here and he says it didn't give you the code but at least it gave you the uh some information this is a graph operation instance which refers to an operation in a given graph the graph object with this graph operation will be used for all operations of the given graph this makes it easy to save and restore State when doing competition with the graph so it didn't give you the uh code create a graph but it did explain you so let's ask some other question here so let me see what we can ask quickly um [Music] concat with functions okay and Define concat weight function okay or let me ask this uh is export the Llama to wait in that format or something like this okay now if you want to ask this and see what it does this time okay exports the Llama two bits in Lama two c dot bin format okay and most probably sometimes this model does this more this model do hallucinates okay all the most of the open source last language models and so there are ways that you can control the hallucination rate and the entire industry is working you know towards that guys okay so let me just go through the last file which is an aoai text and let me open this run image classification text which says let's see a little training argument fine tuning data let me see meanwhile that is running so let it run main functions sending telemetry okay this looks nice Auto config Auto model for image classification Dot from pretrained okay let me see if I can just copy this next for next question and I can say write a code or something okay so let's say it's still running and you can see it's it says it's happening right the post request that we have made it to the end point and it's creating a new one with main pooling and you know we got something let's see what distribute what did we get here we we did get and we got the source document we got something here but it's not giving you the right like the complete uh code which makes sense uh because you have to look at the max tokens and to increase that and also what kind of machine that you have you know to make to get the complete code out of it okay now let me just ask one more question here and automatic let's make it little coherent so right uh the code uh for auto model for image classification for pre-trained for fine tuning uh and image okay let me know about that okay and and image classification model OKAY image classification model okay I'm gonna do that and let's see this is the last question that we have asked so maybe you can play around the maximum tokens uh the max new tokens and try to include it and keep temperature zero I'll recommend that doing it maybe you can also try using llama too with this and let me know I'll be very curious to know because I haven't tried it to be honest but you can have a code specific embeddings model with a couple of line chains and templates and then you use a large language model for example gamma to 13B at least you know or even you can try it with 7B to see what kind of responses you are getting for this but this entire code will be given on the GitHub repository you can take the same back end and I'll show you you can also directly use the Swagger UI I'll show it after this response so let's let it load okay and we can ask some other questions so aoai which is elude open AI they have something here uh chat models video data prepare body headers with data let's do cognitive search [Music] stream with data so you can take you can see it takes around 45 to 50 seconds and sometimes even more depending on that uh how many chunks has been you know the model is looking at in the director store and all that computation okay so let's see we'll quickly wrap this up now okay okay response create okay conversations and you can see it's running here and then tokenizer unknown token ah okay maybe we can handle that as well let's see if we get the response for it uh otherwise we can also handle that uh unknown token okay and it did it did hallucinated okay so it says original code is based on hugging phase Transformers library and supports a variety of models such as so auto mode right the code I said write the code for auto model for image classification Dot from pre-trained for fine tuning and image classification model but it did hallucinate it okay it didn't give you the right response there okay for this but this is okay so you can uh maybe you can try with some other large language model and see if that improve the performance of this application okay but this is what I wanted to do okay and what I was saying is you can also look at if you if you just hit docs it will take you to the Swagger UI and you can also use uh this get response you know the post method you can click on try it out and here you can pass your query so if you pass your query here you will get that response here in that uh the below section and you can use you can also use this endpoint in a postman API client or something like that okay so you can ask your query here and do that as well so I'll just go on localhost 8000 back to UI and you can see it over here so that's all guys you know that's what I wanted to do in this video but maybe you can extend this further improve the performance of this application the entire stack and let me know if if you are doing if you are able to do something you know uh something great with this uh to be honest so the code will be given you will be given on my GitHub repository let me know if you have any thoughts comments or feedbacks uh on this one that's all guys you know for this video and please subscribe to the channel and please uh share the video and Channel with your friends and to peer thank you so much for watching see you in the next one