GPT-3 powered Q&A App using Langchain, GPT-Index, and Gradio

Key Takeaways

hello everyone welcome to AI anytime channel so in this video we are going to build a question answer application using large language model so we'll use open AI to build a QA app using gradio okay so it will it's going to be a very simple but at the same time very powerful application okay so you you can have your multiple documents right so you have your PDF you have your doc any kind of Text data that you have you know that can be web pages as well now you have abundance of documents and you want to build a QA pipeline or QA application that will take all those documents or the text data and kind of create a large chunk of couples where you can interact with you know the application and you can ask your queries and the application should retrieve the information from those Corpus and it should uh give you the relevant information so you're going to use open AI of course to build this application but at the same time we will also have a look at you know couple of very powerful and very instrumental libraries okay that has been you know launched uh in the last few months okay we will have a look at something called Lang chain and we'll also look at GPT index which is also called as Lama index so we're going to see uh this libraries if you see currently I am on Lang chain GitHub repository uh which says you know uh if you if you see the here the description about building applications with llms which are large language models through composability okay now we are going to have a look what really composability means and why it is so important okay when you are working with large chunk of Text data or large Corpus and we are interacting with uh this alims okay how you can utilize this composibility which is a system design principle I will talk about it and we'll also going to have a look at something called gamma index okay so we're going to use GPT index from lava index and it will help you connect your external data with open AI okay in very limit terms okay so first we'll talk about Lang chain and then we'll go ahead and build this application with you know gradu which provides you a very uh this provides you a framework where you can use this library to build a simple UI you know so we have a lot of Frameworks coming in lately we had earlier we still have extremely we have flask we have we have Django and we have uh radio which is very simple so in end you will the the output of this video will be an application okay so in end you will have an application running on radio you can share that with your friends and you know you can interact with the application for your question answering okay so we'll use our document we'll use gpt4 technical report that was you know uh published by open AI last uh I think this month only so we are going to use that document we'll create our you know embeddings on top of that document with the help of Lang chain and of course open AI directly and and then we are going to ask our queries related to that document okay now why line chain is important here guys okay if you see it says building applications with llms through composibility and it says you know Live Language models are emerging you know as a transformative technology but using llms in isolation is often not enough to create a powerful application now let's understand first what composibility is so compostability is a system design principle you know that deals with the interrelationship of components now you can have multiple components while you are building an application so you can have suppose for example if you have an application where there is a component called you know email sending okay we have you have a set of uh uh code that kind of works when you have to send an email and then you have something called PDF download and then you have something called you know root call analysis so there are different components sitting in silos in isolation in your applications so compostability which is a design principle which is very powerful helps you connect this okay this this is basically for interrelationship of components okay so a highly composable system provides components you know that can be selected and assembled in various combinations you know to satisfy specific user requirements now this is the definition of composability you can also read on internet uh you can Google composibility and you can find it out uh what that really means but why it's important for large language models now imagine you want your llm or to answer uh questions over a large corpus now you can have a large Corpus and you know you your llms to answer questions on those copers so it's very difficult you cannot pass the entire you know Corpus in the prompt you cannot pass your entire large Corpus that you have the Text corpus to The Prompt it's not possible right now so what are some of these tips that you can follow right so if I take this okay so I'm just connecting my okay so now what a few of the steps that you know you you can do to pass this entire Corpus you know in prompt so we talk about prompt engineering uh so I will tell you what is prompt Plumbing okay now why prompt Plumbing is better than prompt engineering so few of the steps that you can do you can first pre-process okay you can pre-process the corpus you can pre-process the Corpus into the uh by iterating over so you can pre-process your Text corpus that you have by iterating over the documents suppose you have a document so you have to first pre-process it the next step is you know to splitting them so split them into chunks into smaller chunks splitting them into chunks and then you summarize them once you summarize them now what you can do you can embed those chunks or you can also call it summaries that summarization that you have we have done in the above step and where are we going to embed those chunks or sentences we are going to embed those in some kind of vector space okay so if you are familiar with natural language processing you might be aware about you know this vectors so in some Vector space now this is the step that is this is the step these are the steps that you have to follow okay when you are dealing with large you know Text corpus okay and you want to work with the large language models with those text purpose now when you get a question so suppose I have a new queue okay this is the question what it does it kind of search for your search for the vector space search for your vector space the vector space that you have created over in this in this step D step right it search for your vector space you know for similar chunks the question that you have asked right for similar chunks this question that you see this question that we have here okay that you have asked okay and of search for the vector space that we have in the point number D and for C and once you get with a similar chunk okay in the prompt along with your question okay so you have it send it send it to the prompt along with your original question that you have asked okay now this process you know is called as prompt Plumbing this process that you have seen okay and it's it's better than prompt engineering okay we are talking about prompt engineering nowadays because uh you know you see I think not days I think you know hours of the day you are you are seeing you might be seeing some kind of models like genetic bi models are coming up you know organizations are working with different models okay so when you're working with large text coppers anyone to use a language model okay to process those and try to you know come up with an actionable insights these are the standards tips okay the procedure okay when you're asking questions so when you get a question you have searched for your vector space for similar chunk and then pass those chunks okay to the llms whatever llm you are using in the prompt along with your question and you get your answer okay so this is the plumbing that we are doing okay this is for a line chain okay I will also explain when I'm writing when I'm going to write the code now will also explain what it really does you know from the code perspective now you're also going to use something called you know in this we are going to use something called GPT in index which is basically also called as Lama index you can have a look here and this GitHub repository basically it helps you with the data ingestion and data indexing now what it does it's for it takes a data structure approach okay here this is a very fascinating okay when you go and read more about GPT index so this this is beyond the scope of this video okay we're not going to talk uh in detail that what kind of approach it follows to work with your data okay so it takes a data structure approach uh GPT index and you know it allows the nlm so suppose you have open AI we have we are using DaVinci 3 for example we are going to use DaVinci 3 in this tutorial so we have DaVinci three so what it does it helps it helps it allows The DaVinci 3 Model to connect with the external data so we have our external data that gpt4 technical report the PDF document that we have so it helps you connect to the external data it is also called knowledge base that can be one document can be multiple document so it it allows nlms to connect with uh uh external data and this is what we are going to use so we are going to use La GPT index and Lang chain no to uh create a question answering application in graduate so what I'm going to do now I'm going uh to my vs code you can see this is my vs code and I have some you know already listed requirements.txt okay so you have to create a requirement.txt if you want to create a virtual environment if you are using Anaconda if you want to install it directly you can also install it so I already have a virtual environment over here you can see and and these are the dependencies that we have to install so open AI because we have to interact with having C3 model and then we have GPT index as I said uh it allows the I mean C3 to connect with the external data and that data is nothing but the gpt4 technical report and then we have Pi pdf2 a very famous python library for PDF operations so it you can do any kind of operations with pi PDF to editing manipulation downloads processing getting informations or anything then we have gradio which is a which is kind of a wave framework okay a library which helps you a python Library basically which helps you create minimalistic UI okay and then we have dot EnV this is basically to handling to to handle this EnV file so what we have to do we have to create an app.pi so I've created an app.pi if you haven't uh still created this V and B how you can create you can create using python hyphen M VNV dot VNV which is your folder name but already I have done it so I'm not going to do that you can create that and then you can activate it okay so you have to activate if you are in Windows you have to go inside the scripts folder and then you have to create a activate this virtual environment if you are on a Linux machine you have to go inside the bin folder and then you have to activate it from there so this is my app.pi okay now let's do one thing okay as I told you that which document that we are going to use so we are going to use this document which is gpt4 technical report okay so I'm going to use this document okay gpt4 technical report which is around 99 Pages now just understand guys so suppose here I only have 99 pages but you have multiple documents uh containing thousands of pages each document contains more than thousand Pages how difficult that is for a human being to go through those document and then try to retrieve some kind of information you know so this fits in in the entire ecosystem of your organization where documents are involved now that can be Insurance document policy documents contracts you know banking banking document that can be your API manual software documentation that can be any kind of documents that you have you just give it to GPT three or four also and you create a question answering bot or question answering application and you can also integrate in your existing application if you have at your current organization so this is what we are going to do so we'll use this gpt4 technical report and we are going to ask some of the queries that we may find relevant okay that is you know that that is somewhere in this document okay so let's start so I'll go back to my vs code I already have my API from open AI if you if you don't have an API you have to go to open AI dashboard and you have to take an API so what we are first going to do I already have installed so first I have to load let's first import the utilities okay so first is import OS okay and then I'll import sys ethical system and then I'll also import from dot EnV from dot EnV import load underscore dot EnV to handle the API keys of open Ai and now let's also import gradio excuse me radio as VR and import gradu as gr and I'm also going to import couple of things from Lang chain and then a few of the things from GPD index so from Lang chain import and how you're going to install Lansing it's all in the requirements.txt so you just have to install GPT index and open open AI That's it so from line chain import open AI okay from line chain import open Ai and now we have to import couple of things from GPT index so from GPT index you can see it's already suggesting me I'm going to import first is that simple directory reader because what this module does simple directory reader it helps uh this uh setup code or the function to load the data so we have a PDF file so it will help you load the data from directory so simple directory reader is one of the things then we also need some of the list index so we have to index those right so GPT list index being one and then we also need Vector index so that's called GPT simple GPT simple Vector index okay then we have GPT simple Vector index then we need some llm predictor which is our core here and then we need prompt helper prompt helper okay so these are these are the Imports that we have you know uh for now okay so now we have imported so now let's get this method here load dot EnV and I will use OS dot in environment or environment and then we'll use open AI dot API EE and I'll just load it from my gate in or that EnV file so that let's call it OS dot get EnV and this will be open AI API key open I think I have just made open so this will be open AI not open a open so this will also be open Ai and we have done this now we will write two function the one function will be for the uh constructing the index and the second function will be the chat uh bot or the QA bot or whatever you call it that function will be there so let's write the first function so the first function that we are going to write we are going to call it Dev construct index so diff construct underscore Index this is going to be our first function and in this we only have our directory path so let's call it directory underscore path that makes sense construct index now first thing is we have to first Define some of the parameters here some of the value that you know that helps you for getting the response so maximum input size what is what what's going to be your maximum input size you know when you are writing the text okay in your prompt okay so I'm just going to call it let's keep it standard which is 4096 and then I'm going to say number of I don't know this is for your output basically so number of output is going to be let's call it 5 12 because we are still using GPT three okay DaVinci 3 which is also been flagged as GPT 3.5 okay which is based on instruct GPT so we are using gpt3 here if you want to use gpt4 if you have got the XS uh from the waitlist please go ahead and use gpt4 because it is uh it is cheaper than gpt3 on a per tokens so you can also use gpt4 if you have the access I haven't got the XS yet that's why I'm still using the main C3 so max input size 4096 number output 512 and then I also have maximum chunk overlap okay so maximum chunk overlap which is 20 so yeah I as I you know shown in this uh when we're showing on the tab okay that how you have to follow the procedure when you're working with large desktop large uh chunk of data you have large Corpus so you need to chunk those right so you're also using Max chunk overlap here 20 and then chunk size limit so I'm just going to call it chunk size and chunk underscore size limit and I'll keep this at 600 okay this is going to be the limit maximum limit of Chunk okay so chunk size limit this is done now we have imported few of the modules being llm predictor and prompt helper are among those who are going to use prompt helpers prompt underscore helper variable let's create this variable and we're going to use this module from GPT index prompt Helper and we'll just use the above defined parameter so maximum input size you know so let's use maximum input size you know and then we'll follow let me just follow the same order so maximum input size then we have num outputs so we'll use num outputs and you can also use so you can have a look at here right say Max in input size and Max chunk over Loop overlap number output you can also say say Max underscore input side equal to your variable name you can also do that so I'll say Max num underscore output then Max chunk Max chunk overlap that we have defined and then we have chunk size limit so what I'm trying to say that you can also do like this okay so you see this here you know in the in the documentation of this function with the chunk size limit so you can also do something like this chunk size limb underscore limit equal to this chunk size limit so you can also Define like this chunk size limit so what I'm going to do now I'm just going to do this let's do alt G so we can have a look so prompt helper is done we have maximum input size num outputs maximum overlap and chunk size limit equal to chunk size limit these are basically the Arts you can also follow the arcs procedure here it looks good it's a it's a good way of writing code okay so prompt helper now we are going to use and here we are going to interact with open AI so I'm going to call it l sorry llm sorry so in your llm you have to write that that we have imported here from Lang chain you know import open AI so in open AI here we'll have our open AI uh set of parameters like temperature to control the randomness and creativity of your response so if you have worked with any of the open AI or gpt3 model you will know what I'm talking about temperature and then penalties and so on right so here we have llm equals open AI the first thing is temperature so let's call it temperature and my temperature equal to be 0.7 which is very standard so temperature equal to 0.7 then model name so what model I am going to use so my model name is going to be DaVinci 3 as I said earlier so text DaVinci 0 0 3 this is going to be my model and then I am going to use match token so max tokens Max underscore tokens and this match tokens are nothing but the number outputs so this is what I'm going to use here number outputs so our prompt Helper and llm predictor is are done okay so in llm crypto what we are doing here we are connecting with the dam in C3 model you see open Ai and we are using temperature as very standard value 0.7 and then maximum token are the number of output is to load the documents so I'm going to call it documents and we'll use this simple directory reader so what we have to do guys before that okay we have to go inside this projects QA app and you have to create a folder called docs where we'll keep our document so I'm calling it docs and in this Docs we have to use this GPT for report so now let me go back to vs code and we'll see this docs folder so whatever documents do you have okay so if you have documents for example 50 documents from different industry or four any other uh knowledge base that you have you have to keep in this docs folder so place your PDFs inside this docs folder or any other format of data but you have to process that before passing it to uh GPD index so inside the stocks folder I have only one file one document for this video which is gpt4.pdf so in this simple directory reader what I'm going to do now I'm just going to do directory path which has been also been our parameter here in the function red and then I'm going to going to just say load underscore data so it will load the data okay from this directory path that we will use in the other functions so documents simple directory reader this is done now we have to index it so index equals GPT simple Vector index and inside this GPT simple Vector index I'm going to pass documents the document that we have you know loaded the data from this directory this document is going to be passed here for the uh this GPT Center and now in nlm predictor I am going to pass llm underscore predictor thank you and then in prompt helper I am going to pass prompt helper prompt helper so index is done so now what next so this will so we have to save the index now it's very important to also save this index so you will get an index I think most probably it is a Json file so it will get index.json and we we should save that in our local directory so what I'm going to do now index dot save to disk function sorry not dict it's safe to disk I'm going to use save to disk and in this path I'll just save it my same directory so I'm just going to call it index.json to save it in the root directory I am in currently and I'll just return this I'll just return the index that's it so this is the first function okay where we are dealing with uh line chain and you know we also looking at uh this data GPT 4 technical report and we are using prompt helper llm predictor and then we are using GPT simple Vector index to get to index it and then save that to uh our root directory as an index.json file so what next now I'm going to do I'm going to write one more function and that's let's call it QA bot I'm calling it QA bot def QA bot and I'll just say input underscore text so in keyboard input underscore text and here what we have to do let's first have index again index GPT simple Vector index dot load will load this uh index.json so again it will load from disk and then we'll let's load this file index.json file that we have saved it okay so you're not able to see right now because we haven't run the code yet okay so index.json and now we'll write our response so response equal index dot query whatever query that we will ask okay whatever query that will have okay so for input text this is going to input text response this is the response mode so I'm just going to say response mode there are different uh response mode you can have a look at response mode in dir in GPT index that when you are importing it you'll be able to see that different type of response now so I'm going to compact response mode equal to Compact and now just return the response dot response okay so it's response dot response so this is the Q a bot okay so what we are doing here we are loading this index.json and then we are doing our response prompt and with whatever query will have it will give you a response in a very compact okay so it will not give you two uh big okay of answer of answer or relevant information that you have it will give you a very compact answer okay so return response dot response now will create the interface for the gradu application so let's call it ah let's call it I phase iphase or interface and I'm going to call it gr dot interface sorry gr dot interface and we will write our code here so gr dot interface excuse me sorry and we are going to call it let's call it so what function that we are using okay so I will call it for example I'll let's call this QA bot okay and then you can see function which is callable function if you read this uh functions documentation here it says class interface this class that we are in from the radio it says the function callable okay so you have to use the callable function here that you are going to call so FN QR QA board get you about and then you're going to write inputs so what are going to be my inputs my input will be gr dot you can see it's suggesting me inputs okay excuse me I'll just have a text box here so gr dot inputs dot text box and in this text box and how many lines of text box you want so I'll say for now let's keep lines as seven and then we'll have label so let's have this label and we can call it enter your query this is going to be my this is going to be my input box so I have a function keyboard and then we have the input now what next we have to do is we have to do the output what output that will get so output will be nothing but the uh text okay so output will be nothing but the text so I'm going to call it text and in last let's have a title for this app okay so title will be title will be custom trained so let's have a iPhone custom trained UA bot or keyway application this is going to be the ifs okay interface variable and now we have to yes let's index and we're going to use that function so construct index and our directory is nothing but the docs so docs where we have the file and then if we are going to launch the interface so how do we launch a radio application which is ifs dot launch and you can see it's showing you everything okay so let's so it also has a feature called share so you can share this applications with your friends you might have worked with engine rock or NG Rock whatever you call it and it also helps you you know to when you are running an application inside a tunnel okay you have some Port tunneling and when you have an uh URL link you save that URL link with your friends if your app is running in your system your friends or your peer or your teammates can also access it you know on internet so this is how you can also use it here so I'm going to use share equals to true because I want to share this right so I need a live link okay so this is the uh application that we have built in the code that we have written right now or so far so what we're going to do now let's run this so let's first have a review what we have done we have imported all the required dependencies then we have created a function called construct underscore index that takes uh this file gpt4.pdf and then first connect with the open AI okay DaVinci 3 Model we'll getting this index we are saving this index.json in our local directory and then we are using one more function called QA board that we are loading the index.json and whatever queries that you ask it will try to look for the similar chunk okay in those index.json and then it will give you a response okay this is what it does and we are returning this dot response and then we are creating uh interface and we have this callable function which is q a bot and our input is a text box where we will input our queries enter your query and then we have outputs as text and then we have title as your app title and then we are passing that file to this function and then we are using the radio application to interact with so let's run it and see if we are able to get the desired desired response or we are getting any kind of error here so let's do that so python app.pi so it might take a little time because it totally depends on the number of pages that you have how big your data is and also on your internet bandwidth because it has to connect with the open AI uh server and you send the data to them and then kind of get the model trained on on your custom data and then you get the response so you can see we have total embedding token uses so around 80 000 of tokens okay and you can see by getting a gradual live application let's copy this application I'll just copy and I'll come back on my browser and I'm running this application so what it will do it will launch a radio application you can see it says custom trained QA application so this is the QA application question and answering it's not quality assessment we haven't built it for testing tool this is the question and answering application now you can see I have a input area where I can input my text which is query and then I have an output where I will get the output now we also have some of the features called flag I will tell you what flag means okay so flag means like when when you want to keep a log okay of your entire activities on Radio application it will save that in a CSV format okay inside of flag folder I'll show you what I'm talking about now let's open that uh file gpt4. that file that we have gpt4 technical report so this is the file that we are going to use guys okay so it released on 16th March 2023 so it was published by open AI okay so we'll ask some of the queries let's read it so I have already read it okay it's a very interesting paper to read so let me ask some of the questions let me first have a look so it has around 100 pages okay so let's ask about the limitations okay so my first question will be can you list some of the limitations some of the limitations of gpt4 I this is my query so I am passing this query now okay in this graduate application what it will do it will have a look for the similar chunk okay in that Vector space when when we created that uh when we're using that index using llm predictor we have saved that index.json file right so it will go through that look for the similar Chunk in those Vector space and then try to come up with a relevant information for this particular query so let's hit the submit button and see if you're able to get any response see here we go so we have got this information in output not fully reliable you know this is the first limitation can suffer from hallucinations has a limited context window does not learn from experience potential for buyers because as always LMS are still buyers because the bias is not in modeling the buyers is still in data the data that they have been trained on so they are still buyers then the potential for buyers you know you can of course eliminate the buyers then some sometimes it also you know gives you this information some inappropriate information that might not be correct then it kind of uh some of the limitations are privacy and you know cyber security and proliferation risk so this working you know it's working I'll say good this is good okay this is this is a very good start I will say because it took us 30 minutes only to build this application so you you have a set of documents you just pass this document you use Lang chain and GPT index and you come up with this kind of application okay so you can also use this uh at your workplace and you can because at workplace in in organizations we have multiple knowledge bases okay and all those data are sitting data are sitting in silos you can bring all those documents together you can create a single platform and use this technique to you know have this QA pipeline or QA application so now let's go back to this folder guys okay so I'll go back to folder and you can see now I have a I have a folder called flag okay so in flagged I have nothing right now but what if I just click on flag okay when I hit this flag it saves the response okay so let me go back to vs course we'll be able to see and let me go inside this flag I have a file called log.csv when you click on log dot CSV it kind of keep the track of your entire activity on that graduate application okay so this is fantastic right so now we have also an index.json if you click on and it's huge right it's 80 000 of token that you saw in the terminal you can read it entirely here it's a huge chunk of text uh data here the data that is the information from that gpt4 technical report let's ask some other question also and see if you're able to get any response can you list some of the capital let's change it to capability so as this can you list some of the capabilities of gpt4 why should I use it now this is my query okay uh to this and you can see a fantastic response a very compact response now you can also you know play around this the compact names and you know you can control these response okay uh using GPT index that the function that we used when you are loading it so it says some of the capabilities of gpt4 include outperforming existing llm models you know uh and then what it's saying 24 26 languages and you should use GPT because it is powerful tool for NLP tasks such as text generation and then can be summarizations and question answering that this is what we are doing see this is the response that we also got from you know uh gpt3 basically lgbt4 okay uh that the the document that we are using it is also capable of learning from making a digital tool in machine translations and dialogue systems which are very instrumental when you want to achieve ADI in future machine translation and dialogue systems are something that will you know open the door for AGI so additionally GPT has been designed with safety in mind okay because you know you you might have seen the CEO of openai Sam Ash morning was talking about you know uh they are you know very focused towards the responsible AI when they creating this kind of Live Language models so this works perfect for me this is working fine you know you can have multiple documents bring it in the folder and then get the index with the help of gpt3 and also use a line chain ngpt index you know to work with it so this is what for this videos guys I hope you liked it you know the code will be available and just for just to see here it also see for whatever question that you have you have asked it says total llm token uses for the first question that we ask are the limitations it uses 625 tokens llm token and embedded token with 13 embedding tokens like the embedding that we have so right so and for this the second question 741 token which is which is which means which makes sense because the response if you see it is uh it is adding two paragraphs here that's why you can see the embedding token uses 19 tokens so this is working perfectly fine guys so what I will do this code will be available on this GitHub repository AI any time of GitHub represent you can also find that in the uh description of this video so if you like this video you know please uh share this video with your friends and peer if you haven't subscribed the channel yet please do subscribe the channel and uh share uh this channel link uh with your friends and to your peer okay I hope you like the video that's all for today's videos guys see you in the next video