A Helping Hand for LLMs (Retrieval Augmented Generation) - Computerphile

Key Takeaways

in a previous video we looked at llms and whether or not they are going to continue to get way better or they're going to sort of plate out we're not going to keep talking about that today you please to know but I was interested in you know some of the things that llms we were talking about that they don't do well so you know maybe they aren't good at talking about very Niche subjects that haven't been well represented in the training set how could you improve their performance on on those areas so maybe what we want to do is is use the ability of a large language model to move text around and convert text and summarize text but in a way that's perhaps a little bit more specific to some text we're interested in so that the accuracy is better that's the idea so what we're going to look at today is something called retrieval augmented generation or rag and very simply we're just going to take our query we're going to combine it with some actual data and look at them at the same time you can actually see this already happening so if you go to Google and you search for something or you go to Bing and you search for something often an AI in some sense will pop up and try and tell you about that thing in my experience that's somewhat helpful but actually I just wanted the link that was the first link down what this is doing is something called rag or retrieval augmented generation so the idea is we know what we've asked about we might not have all the information we need in the weights of the network already it might not know about these things but what it can do is summarize text so maybe we should load in a website to do with that subject and have a look at that at the same time so when Bing or some other search engine does this what they're doing is they bringing in sources of information that retrieval and then they're adding that into that process answering hopefully better text output right and you can see also it has the advantage that it also allows it to site its sources so instead of just saying yeah this is the this is the truth about this thing you can be like well where's the website you actually read that so I can verify this for myself right that's something that's quite useful to do normally we would have something like an llm and something like this so this might be chat GPT GPT 4 it might be llama 2 or some other model there's lots of them now and we're going to put in our query that's a g so I'm going to put in my query now the query will be a question about something so you know what is the capacity of some football stadium right with been the Euros on so we can talk about that um although I don't have very many opinions England not doing well enough as always so we're going to put in our query what is the capacity of this Stadium now the llm might get that right it might know because it's read the Wikipedia articles as part of a training process but a better way of getting accuracy for this would be to bring in some sort of data right so bring in data which we're going to get from the internet live and then we can combine that with our query with a big plus symbol and that's going to go into our llm and then it's going to be our answer now the reason that that's effective is because an llm is quite good at paraphrasing text if you say here's a paragraph that I wrote that's not very good please rewrite it for me that's one of the things they can do quite well um and or at least better than something where you're just asking them to come up with data out of nowhere and this doesn't have to be a Wikipedia page this could be your stock prices it could be information on ticket sales it could be some information on your company so maybe you're trying to run a tech support for your internal staff and you can draw in information with frequently asked questions in it and then it can answer those questions in a in a sort of more natural way and the benefit I suppose would be that they can ask it questions directly as if it was a person when obviously it's not but maybe slightly more um Pleasant experience than just trying to scroll down a website to find the exact thing you need or searching on an internal website and not and getting just lost in a bunch of pages that doesn't don't have the information in you want this is actually a very very straightforward process there are lots of libraries that can help you do this but the idea is that you're actually going to create a bigger query or a bigger prompt for your llm based on your query and any data sources does this stop or combat the hallucination thing because obviously there's a big problem with large language models where they kind of just guess what you want to hear yeah yeah they're trained to produce next words and sometimes those next words are meant just to look nice and not be factually accurate so my I guess my having tried this out a little bit my um my feeling is that it does help a bit it won't of course completely stop an llm from saying what it wants right and it might for example misread in in verted commas the the information right so maybe you you query some data from Wikipedia But the information isn't in the Wikipedia article that you need or maybe it is but it's buried among a bunch of tables neuron store yeah and and you know there are some models that have very very large inputs so Gemini for example the Llama model has an input of 8,000 tokens so if you imagine you give it a 5,000 Word document and say what's that specific bit of information I need it may find it it may not find it right I would say the chances of it completely making something up are at least reduced but that's just my personal feeling on the matter I would say as always in these situations maybe we can have some scientific data on this let me show you a little bit about in a very simple level how it works bearing in mind there are lots of libraries for doing this and lots of different ways to do it so we might use a library like Lang chain to do this Lang chain is a library that allows you to do things like Rag and other you know ways of interfacing with llms Beyond simply giving them a string and reading different tokens back so the idea is we have a query from the user and then that query is going to be augmented by some other data so let's say the user wants you to summarize a paper that they' read so what you would do is you would take the PDF of that paper and you need to put it into the large language model now that's going to need some kind of retrieval process that allows you to scrape the text from that PDF right so maybe we've got some data source and the problem with that data source is maybe it's not in the correct format actually getting information out of Wikipedia is not too difficult because they make that nice and easy but maybe your data is a PDF or it's an image or something more complicated than this so you need some kind of conversion or some kind of data processing so I'm was going to write I don't know processing and this is going to be our sort of processed data or our final data so this is our data that we want to put into our model now we have something called a prompt template now a prompt template is going to include all of the information that the llm is going to have directly into its context so this could include system information like you are a a chatbot trained by open AI right or it could be more specific instructions like I'm going to give you some data I'm going to give you a query please answer that based on that right and don't go off piece and write poems about Pirates um it will still do this by the way if you ask so this prompt template has in it instructions but it also has a place where this data will go and a place where the query will go so maybe it has instructions and the instructions might be really straightforward like you're going to receive some context you're going to receive a query off you go right then it has a place for the data or the context or the information or whatever it might have more than one place so maybe there's sort of numerical results so maybe numbers in here in some way and then you have your query which is the actual prompt that the users put in and then maybe you have final instructions or some other information right and then this is going to be converted into a giant prompt of tokens which goes into the large language model and then it makes its next word prediction and it spits out an answer right so I've actually implemented one of these I say implemented I joined together a couple of libraries and took the lazy way out uh and we can see this working right so it's football time so we're going to load up a Stadium from Berlin where part of the Euros is being held and we're going to grab grab that information from Wikipedia and we're going to stick it into here and hope that we can answer questions about it that's the idea and and this might be slightly more accurate for example than if we just ask questions without any kind of context being added at all so olama is the library I'm using to actually run the large language model so Lama is a nice open source Library where you can run locally hosted versions of large language models I'm running the Llama models here which is where the name comes from those are released by meta and they're open source so they this particular model for example is 70 billion parameters which is pretty vast not as big as chat GPT 4 but big enough to get pretty good texttext translation I'm also using a library called Lang chain which allows me to do the pump templates and things like this Lang chain does a lot of other stuff that I'm not using so for example there's a thing called langra which allows you to have llms to communicating with other llms for example and you can do all kinds of other stuff like have them cool functions and and run maps and generate images and so on and so forth um so this is a very simple piece of code all I'm doing is I've got a couple of functions which allow us to grab content off Wikipedia and I found one of the divs or the uh main parts of the website in HTML it's called MW body content and that thing is essentially where the main text of the article is held so I'm literally just grabbing the Olympia Stadium Berlin Wikipedia page I'm passing it through a couple of very very simple filters to get the right bit of the web page and to strip out all of the HTML so it's just text now you could do this in a much better way than I'm doing because you you know imagine you're going to get bits of image links and and um there's no heading information so there's no kind of structure to the document anymore but it will be good enough I'm going to grab that and then I'm going to stick it in when I make my query so then I have something called a prompt template I have a really simple one you could be much more complex with your prompt templates so mine just says you're an AI assistant who answers questions with the provided cont context and then it has this is the context this is the question and this is where you're going to put your answer right and I have these placeholders of context and question where that's where Lang chain will insert the content that I've given it and the query that I've given it and then that's pretty much it so I'm going to create a Neo Lama model and I'm going to point it at the Llama 370 billion model which is running on our servers I'm going to create a chain where we get the Wikipedia web data and then we pass it through our prompt we pass it through our llama and then convert it to a string you know and I'm going to ask could you tell me a bit about this stadium in a really enthusiastic way aimed at kids right something like that try and make this a bit more exciting I'm someone who has a passing interest only in football so maybe this will be good for getting me uh more excited as well and then we're just going to invoke that chain and all of this will run uh behind the scenes for us so you can see that actually bringing in extra data is really not very complicated and I haven't done much uh in here in terms of code at all so let's run this now so I'm going to bring up my prompt so python rag. py um and it's going to trot off and it's going to run our large language model now this might take a while because we haven't been asking anything of this large language model and it has to spin it up it has to get the GPU ready it has to initialize all the memory some 33 gigabyt of memory if I recall um that's in in just on the graphics card so it takes a little while if you run it multiple times it's a bit faster you can imagine that if you were running this let's say on Bing you need to have a lot of servers primed and ready to serve people with queries I think it takes about 30 seconds the first time than about 5 to 10 seconds for another one I'm sure I could speed it up by being more uh competent with my coding here we go all right so oh my gosh kids let me tell you about the amazing Olympia Stadium in Berlin Germany right vocation though you could be a kids TV yeah I could I'll use it to script all my my future content so what it's done actually is quite good so it's taken yes it's in the star at suitable for kids that's great but it has got some interesting information so 74,000 uh capacity it's hosted concerts um it's got VIP areas and so on and so forth so it has drawn information from the Wikipedia article and put it in here the reason that search engines are doing this is because the content will generally speaking be better of course there'll be counter examples that are sometimes funny sometimes a bit worrying um about how this works but the idea is that if you have the text right there in the context it's at least more likely than it is just hoping that the training process itself has memorized all these facts it's perhaps worth remembering that these models are very very big so this is 70 billion parameters that's a lot of different things you can learn chat GPT is over double this size right and so there's a good chance it'll learn a load about the Olympia Stadium Berlin already right and you'll be and you'll be a to answer questions like you know chat GPT will be able to tell you about these stadiums and things like that the I guess the the thing is that when you have things that are less common right or more difficult that's when this data retrieval might help or things that are very specific to your use case so maybe you have specific content from your company or from something like this that is not going to be in the training set we hope and so in that case they won't be able to answer these questions and so it's those kind of times where bringing in extra data might help hey everyone thanks for watching this video it was brought to you by Jane Street and Jane Street are looking for the next wave of curious and passionate people to join their latest internship program this is an amazing program and I think the sort of people who watch computer file could be the perfect match if you haven't heard of Jane Street there are a COR conative trading firm with officers all around the world they're at The Cutting Edge of things like machine learning distributed systems programmable Hardware statistics and Jan Street are currently taking applications for internships in quantitive trading software engineering research and plenty more areas it's pretty broad the summit internships are in New York London and Hong Kong you'll do amazing work meet Fascinating People and the internship program also includes cool social events guest speakers all hosted in their worldclass high-tech offices I've been to some of them they're very cool places to work in addition to a salary for the summer Jane Street will also cover all your flights and accommodation it's an amazing opportunity last year's interns came from 22 different countries and all sorts of backgrounds you don't need to know about Finance they just want people who are curious collaborative the sorts of people who I imagine might be watching these videos no matter where you are in the world no matter who you are if you're interested check out the link in the video description