Retrieval Augmented Generation with LlamaIndex

Key Takeaways

hello everyone and thank you for joining today's Cod long my name is Ree and I'll be your moderator today we're going to kick off today's session in a couple of minutes we're just waiting so everyone has a chance to join in the meanwhile though we'd love to hear from you so let us know when you're joining from using the chat or the comments depending on what platform you're watching on and yeah tell us something that you'd like to get out of the session today uh if you haven't already then please register for the event you can do that by scanning the QR code on screen uh I'm also going to be posting a link for you to do that in the chat very shortly and you can also head over to dat camp.com webinars where you can also see all of the future sessions that we've got uh in the leadup to the holidays as well um we are going to be using data Camp workspace today so if you don't have a data Camp account yet make sure you register for your free account and I'll be posting a link uh very shortly when we start the session so that you can code along with us well I think that's everything from me uh I will be back to repeat these messages for any new joiners very shortly but until then enjoy the background music hello everyone and thank you for joining today's Cod long my name is Rees and I'll be your moderator today we're going to kick off today's session in about a minute or so we just waiting so everyone has a chance to join let us know where you're joining from using the chat or the comments depending on what platform you're watching on and yeah tell us something that you'd like to get out of the session today um we are going to be using data Camp workspace today so if you don't have an account already then please sign up for a data Camp account and I'll be posting a link so you can go along with us uh and remember if you miss it uh if you have to log off or at any point Jo during the session to that today please make sure you register for the event and then we can send you recording as well as any other resources that we share throughout today so if you haven't registered already scan the QR code on screen head to datac camp.com webinars uh and also uh find the link that I've put in the chat as well um so yeah make sure you get registered and yeah we'll send you the recording as well as um the resources that you need to code along uh to this in your own time um aside from that we are going to be having audience Q&A during the session today so if you have any questions any point let us know in the chat and the comments and we will get those answered for you brilliant I think that's everything for me so now I hand you over to your host for today's session Richie Richie please take it away hi there data scamps and data champs this is Richie now hopefully the fact that you're here means that you've noticed that large language models are kind of a big deal right now they do however have a problem that is sometimes the text they generate isn't true so it's sometimes called hallucinating and one solution to the problem is a technique known as retrieval augmented generation and that's what we're exploring today so the idea is that you take your existing prompt and then you augment it with some extra text that contains relevant information for the response and then the llm has a Fighting Chance of giving you a factually correct answer the other thing you'll be exploring today is llama index which is a framework for building AI applications for search and for retrieval augmented generation and teaching you today is a bit of a Legends uh is Dan Becker he's a longtime data Camp collaborator having created the introduction to deep learning in Python course and beyond that Dan is one of the top thought leaders on machine learning and AI he's the chief generative AI architect at strave where he buil bus AI solutions for Content technology he also runs the build great Ai consultancy and he was previously VP of machine learning development tools at data robot Dan is also a successful kaggle competitor and the author of Automated machine learning for business so without further Ado over to you Dan hey everyone glad you're all here um so like Richie said I'm going to be talking about retrieval augmented generation with llama index that's quite a mouthful so we're going to start calling it rag which many people do and the primary reason that I am focusing on this topic today is that if you go back a year ago there's a lot of excitement about building AI based apps I've done Consulting and what I found is that a disproportionate number of the use cases that we see um in business require rag um that's because they're for companies to have their own private data and they want that to inform the model respond responses and rag is really um the most appropriate way to do that in not all cases and actually I'll talk about um rag versus fine tuning uh quite soon but rag is something that I see popping up so so frequently that I'm excited to share this with you so today uh we're going to go through a case study the data that we're going to do use for this is uh from the news API website with that website can get data about almost any uh topic you want because we are excited about uh large language models we're going to take news from the last month about llama 2 which is uh one of the most popular open source large language models and our goal is just to create this working example of rag or retrieval augmented generation using llama index uh you'll see that there's not an immense amount of code here and it is going to be relatively easy for you to take the rag example that we build and then uh migr that to whatever other purpose you have so um I think this is going to be a really nice resource not only for you to understand but also to Come Away with this code uh and then also we're going to look at how to interrogate and debug these rag systems so that not only can you build it but if it doesn't behave exactly the way you want you can figure out why is that happening and fix it um I think of rag as the most reliable way to use an llm with a proprietary or internal or specific data set that um might where that data set might change over time and in a minute I'll talk more about why that's important but before I do that I just want to give you an overall context about what rag means or how a rag system works so uh I'm going to zoom in this is a and when I share a notebook you'll see you also have access to this image in your not notebook but we are going to think about an app uh this could be a user facing website as an example and the user comes in and they have a question so we are here and we've got this user they're going to ask that question of our system so they might ask how can I serve llama 2 models with AWS that's going to come into our centralized rag system or rag framework there are a number of tools for this so we're going to go through llama index which I think is my favorite but you can also use Lang chain to build uh as the central Hub of this rag index or this rag framework rather and to be honest it's not so wildly complex that you couldn't also build your own framework uh customed to your use case and certainly people do that so the rag framework is going to get the question and let's scroll up it will then say to a vector Vector database here's my question so uh that is how do I serve a llama 2 model on AWS and the vector database we find of all of the documents that we've stored in this set of uh this repository of documents what are the documents that are most relevant to uh this request or this question or this query so the vector database an example there are quite a few of these so pine cone and we V8 uh and chroma is one of my favorites there all these different databases that can store primarily documents but also images and find the document that is most relevant to a given question and it will then pass back a section from that document to the rag framework the rag rag framework will then pass that to a large language model and it will say here's the question I've received so in this case the question was how can I serve llama 2 models with AWS but in case the large language model let's say was trained 12 months ago uh and as a result the answer might have changed or maybe it didn't know in the first place we're also going to take that U the Snippets of text and pass those to the large language model uh as well as the question and so it will receive the text saying here's the question followed by the question and then uh here is information that may help answer this question and we'll embed those paragraphs the large language model at this point has typically a pretty easy job which is we've got a question and then we've also given it the answer and then it's just synthesizing the answer into something that is nicely written uh you can use chat jpt for this we give an example of that you can use llama 2 you can use really any large language model for this it creates the final answer and the answer could either be in this case Amazon Bedrock or depending on the exact large language model it might be a little bit more verbose and it might say the most common tool for deploying a llama 2 model on AWS is Amazon Bedrock um but it it's just creating that answer and then that answer is now passed back to either directly to the user or if we have a web-based app it's passed to the um front end of the app that then shows it to the user so this is an overview of this system um a uh a common decision that you'll need to make before building a rag system is I've got a set of documents or some set of knowledge that I want to have my um AI have access to and you'll need to decide do I want to build a rag based system or do I want to fine-tune that information into my model so first of all say building a rag based system is much more reliable or easier it takes less less fiddling with Hyper parameters um but either of these can use any data one of the things that in my experience and I wouldn't have anticipated this ahead of time um but one of the things in my experience is really really important is the ability to understand why am I getting the answer that I'm getting and can I change that if I'm not happy with the answer so that's this I've called it debug ability that is much easier with um rag if you're getting a new set of documents the time so I can have a persistent database it's stored on an ongoing basis and then I have new documents coming in on a daily or weekly or monthly basis it's easy to update that with rag the training cost for rag is effectively zero there are some advantages to fine tuning so that would be if you want to change the tone or the style of your response finding tuning can allow that um if you want your model to be able to learn to reason in a new domain um find fine tuning can do that to varying degrees and I won't even get into that um and then inference cost is a little bit cheaper with fine tuning because you're not passing in that whole um these whole documents as you're doing inference so these are some of the trade-offs you can use these two together as well um but our experience has been that we use rag far more frequently um and so I'm a big have become a big fan of rag and a big fan of llama index for um for using rag so with all that why don't we jump into the code so uh like I said I think it's useful to have a mental model of um of how rag operates and as a result um I've in this notebook and you'll have ongoing access to the notebook if you register um I've got this diagram at the top and uh as well as a link to where that came from there are a few packages that we need to install so we're going to get data from this um news API uh service um you'll need a token for that uh and for now I'm just going to use my token or this particular notebook has the data pre-loaded so if you don't want to get a token you don't want to go through their website you can just use uh the data that I've already stored to Cod along we'll be focusing on llama index so I've installed that and then later on we're going to use open source models from hugging face and so we'll install that all right so uh I want to get into looking at the data in a moment but before I do that here's just a quick view of the code and it'll be more interactive in a moment but here is a quick view of the code to download the Articles so like I said we're going to have an I've got an API key that I've registered to download articles from uh news API I create the client uh we're going to save the articles in this uh save articles. Json I'll briefly open so you can see there are a few files here so um one is as we look at hugging face or some of the other services if you want to uh set those up we have some guidance on how to set those up to take away if you come back in the future and you are working through this and you can't remember a solution here's uh a notebook with a solution here's the notebook that I'll be working through and then we have um the saved articles so we've got the saved articles and we can uh either retrieve those articles I've got a little flag here so whether we do that in real time uh or if you don't have a key in your you just want to load them then when saved articles is false we'll just load this so I will run this cell now and we have retrieved 100 articles about Lama 2 that were all published in November so let's look at a single item uh you know as data scientists it sounds like something you would do an immense amount of math but one of the things we spend the most time doing is just looking at the raw data and understanding how it is set up so I'm going to inspect a single item if I looked at all of the Articles uh it'd be really a little bit overwhelming so why don't I look at one of them I can look at zero which one you look at doesn't matter uh if you're coding along I would suggest looking at a different one than me so that we can um just you see a different example so here is our raw data we see that this was an article from a web site I'm not familiar with called bis talk uh it is here's the title uh it's quite a long title we have a description and then finally we have the content of the article so the content of the article is further validating how bral brittle the security of generative AI models and their platforms are some security company helped hug and face Dodge a devastating attack by blah blah um because I'm using a free account we are not going to have the full source of the Articles or the full content but you'll see that even the first sentences of an article is enough for this demo and then if you go and get uh a paid account you'll get full articles so we're going to have this for a bunch of different articles I'll look at I'm gonna just pick another one and we can see another uh example so here is uh someone's Q3 earnings called transcript and this is going to be something that is related to the Llama 2 model it's some tech company we can see yet another one operate operationalize llm evaluation at scale using some AWS service so we've got a bunch of different articles and from this we can see the format of them so looking at one is always very valuable uh I'd also like to just get a sense for um the breath of Articles whether there are duplicates um so the general just get a general sense of them and so to do that I'm going to look at all of the uh titles so I will create all titles and you can see the format of this is uh each of these articles has something called has a field called the title and so I'm going to just use a list comprehension uh so I've said we're g to take each of the Articles each of the individual articles and alter articles articles I'm using this word quite a bit we're going to call that an article and then we're going to take the title for that and and why don't we look at the top 20 of those all right so we see um this is nice like one of the things that I do like about the news API is it's got a article from a breath of sources so this is uh something from The Hacker News website so someone wrote a post there it's showing up we've got something else that is an earnings transcript uh here's a service that lets you serve llm models uh with a PHP service so we've got uh an overview of the titles here we could look for whether there are duplicates and we could uh just write a line of code I'll even do that one way of doing that is see if there are any d at titles here uh so we've got 100 articles and from that we've got 96 unique titles so it looks like we do have some duplicates here um if you were building a production system you would probably do a little bit of of D duplication but I think uh for our purposes um we don't need to to worry about that so we've got a sense of the data let's now um start doing the work that we need to build our uh system or our rag our rag system so the rag system is going to look at the text of uh the Articles and it's going to put that into a database so I'm G to do something similar to what I did uh higher up I'm gonna say article text for article in all articles and then again I just want to make sure that I'm going to print that out maybe I'll just look at the top five just to make sure that uh the result we're getting matches our expectations so uh ah let's go back up here and figure out so I called it text but the correct word is content so I'm going to and you can see just looking at the raw data again this is the reason that it's useful to look at uh your raw data so here I've got the text of all the Articles and here's the uh here some results um Yan laon is meta's Chief scientist it's got an article from him makes sense since uh meta is also the creator of um llama 2 and now we're getting into the code to build a rag system so we're going to go through this uh a few different times we're going to build a rag system the first one is just using open AI but then later on we'll do it with hugging face um the there's four lines of code here these four lines of code it's a little bit different from like a list comprehension where if you've used python for a while you might just know those lines of code these are somewhat bespoke or specific to L index and so now you've got a resource and you can see uh see these lines of code but it's really very few lines of code to get started llama index does a really nice job of um what developers call Progressive disclosure of complexity which is to say that you can get started and it doesn't seem very complex at all so this is uh four lines and then if you want to really fine-tune it and change a lot of different knobs to change the behavior of it uh you can get very very deep and you could set up a system that has hundreds of lines of code um but this actually will work pretty well so let me walk you through these four lines of code so the first is we're going to need to um do two things with large language models one is take the document and get a vector representation or an embedding of the meaning of each of the documents so that we can put that in the vector database so the vector database when we say what is a relevant document to a given question it has a way of measuring relevance and so we're going to use a large language model for that and then we're going to use a large language model to create the final answer and again you can see that in this um image at the top of the notebook and if you're following along and have registered and have this notebook open you can always scroll up there to see that context excuse me the next is we're going to create a set of documents to insert in the um Vector database as the next thing we're going to do is insert those uh documents into a vector database llama index by default has its own inmemory database that's getting created with this particular line here um but if you wanted to have any of the mainstream Vector databases whether that's chroma or we8 or pine cone or or that's probably 10 mainstream Vector databases it is really just one extra line of code to to set up your vector database and then use that instead of this default but this is creating the um Vector store that has pairs of documents and um Vector representations or embeddings for those and then the last is we're going to create the query engine so this is really the query engine is really the piece that um we are excited about it is the endtoend pipeline that uses all of this so we've got the index we just say create a a query from it um there's one argument that is particularly useful which is when we ask a question we're going to get back a set of documents and pass those to the large language model that gives the final answer how many do we want uh in this case I said two um but you could change that to be five or one and you'll get different quality results depending on that number um if you were just uh I think it's always nice when you are um just getting started in a new area to find one or two parameters to mess around with and see how it changes your results this is a nice one that's easy to change and see how it changes your results I'm going to run this line of code okay and now we are going to use this query engine so the way that you do that a query engine has a method called query and we're just going to give it a question so uh I'll just start I used this question earlier in the slides um what AWS call it a or Amazon web services um because it's a large language model uh it really knows that those are the same thing it has in some sense a bit of common sense um but what AWS service would I use to deploy a llama 2 model llama three model I need to uh have something that can see into the future so I will run this and you'll see that I got back a response it um the very first item in it is the answer and it's uh the correct answer so that's a good sign the other real takeaway I'd have is this is like horribly complex this is like there's so much in here um we're going to spend a moment looking at it but then we're g to try and write a function that's simplifies this for us so that we can run this on different queries and and see results but not have to parse through all of this mess to do it okay so what is uh the content of this response so we've got the response itself the next item is uh called Source nodes Source nodes is a reference back to the documents that the model used to formulate this response so a source nodes is going to be a list of nodes because I said I want two documents there going to be two items in this uh python list each item is going to be a node with score score is just a measure of how similar this document is or how relevant it is to the query and then there all this mess in here so there's uh an ID which we won't go into um metad data so this is actually a another topic which if you really get into this metadata lets you have a way of filtering and saying I want only data that is um only documents from the last week or from a certain publication and metadata filtering let you do that um so if you get really into this uh field metad metadata and metadata filtering is an interesting um area but the piece of this which we likely would would most want to look at is the text so uh we've got two Source nodes each of them has some text and if I were just trying to generally debug I would typically when I run a query want to see the actual response as well as the text of each of the nodes that we rely on so I'm going to create a convenience function called search which um which runs this query so the first thing we're going to do is uh let me run this query and then we're just going to pull out the relevant information so that in the future I can run this search and get a simplified answer rather than something that is quite complex so we're going to say I'll call it output equals and we saw that that was this response item so query result. response and then someone's getting crate engine is not defined uh I think they probably there's this was in the chat um they probably didn't run this uh item up here that build the query engine so if you get an error about query engine not being toine just uh rerun the cell above it yeah just one thing to add to that um there are some instructions if you go to the file tool on the left and go to the account setup notebook if you need to create an open AI account then uh that'll show you how to do that ah thanks Rich okay so the data sources that we rely on I'll actually call it data source texts and we're going to pull out the node withth score so we're going to take output dot let me uh Source nodes I want to make sure that I get the capitalization rate and then for each of these we want to pull out the text so and we're get starting to get a bunch of um questions I'll tell you I'm gonna just um batch those I'm going to save those for a bit and then come back to those after we've gone through uh this notebook or uh I think we've got we'll probably spend about 15 more minutes going through this and then uh I'll answer these questions because we're getting good questions in the chat so let me return output and data sources with texts there's a chance that uh I didn't get this exactly right but we'll test it so we'll test it with the same question we used earlier same problem that one had it earlier uh let me I didn't close this bracket and all right so we've let's debug this for a moment we've got a string has no Source nodes so that means that uh this so this right here is a string and I said output. Source noes but it really should be query results and we'll rerun this node with score it should be text so uh I got it there's always a little bit of debugging involved but here we got something that is a utility function and in the future if you want you can just copy this function uh what is the best AWS service to deploy a llama 2 model and instead of this whole big mess that we got higher up we've got a response which has two parts and which is I think very interpretable one is the raw answer that's this first piece and then the second piece that we got back is the text that the model relied on when forming that answer um and then when you're debugging these systems you're going to use this quite a bit to figure out I like to break up uh if you're getting an answer you're unhappy with I'd like to break that into two pieces one is is the llm getting uh data or documents that have the writing answer and if not then we're going to debug our um our query engine or our our query retrieval and then if we are getting the right answer and that's feeding into the llm and it's still getting the wrong answer then we would debug um what's happening in that second stage um so this this is a quite useful function I think you'll use it on a on a regular basis um let's now switch to building this a version of this ourselves um to change so so far we've done this with open AI apis now let's switch and use it with open source uh llms those are going to be coming from um hugging face which is I think the most popular place to find and serve large anguish models um in order to not be using the word llama 2 in too many different places with too many different meanings I've used uh I'm G to use this Zephyr model which also is I think a newer model and probably my favorite model right now so we're g to use uh The Zephyr model which is stored on hugging face we're going to um need to do the equivalent if we go back to our lines of code that created um our previous rag system the first one just created the uh object that can access the model so we're going to need to do that um this is the sort of thing that you wouldn't off the top of your head have any chance of knowing what to do and so I'm GNA do a Google search just to show you that it's not so so hard to find this um so we are using llama index uh we want to use the hugg and face inference API um because going to be serving our models and if you do that search um some of the top examples really a few of these would have um what you what you're looking for would have examples I quite like um to just work from examples and this is a notebook that shows explicitly how to set up um large language models with llama index um I'm not going to go through this particular notebook just in the interest of interest of time but we could find that um there are a few examples here of how to do it um so we've got if I wanted to run the model locally here's a line of with an example of how to do it I'm gonna copy these two lines of code over so one of these is if you don't have a hugging phas token which is you just haven't registered through hugging phase don't have an account um and there's a notebook showing how to how to do this in our workspace um how to get that hug and face token um then you would if you don't have that token then you would use this if you do have a token then you will use this bottom line the advantages of having a token is that they are um less likely to rate limit you or um think you're abusing their service and as a result block your access so let's um Let's uh I'm going to copy that all down here I do have a token so I'm going to uncomment this if you are running this on your own you probably don't have a token or don't have it stored here and so you might not um use this HF uncore token instead you would comment this out you would comment this bottom one out that uses huging face token and instead just use this um remotely run but I'm going to use my token and um let's see go back and they've got the model name um let's see if we can pass in the model name as well oops later on I'm going to call this HF model so we'll do and again I've just created model name right here I want to make sure that I'm specifying that and that's run okay the one change that as you move away from defaults a lot of the information that you um use is stored in what's called a service context so as you do more customization you're going to um add more arguments to the service context for now we're just going to add that the llm we're using is this hugging face model and now we're going to um recreate some of these lines that we have above so the documents are the same but we're going to recreate the index and query engine so you'll see we just needed to add index equals question mark and query engine equals question mark query engine is again not changed at all uh the only difference is that we need to pass in our service context so here I've saved something called service context which just says um what are the additional arguments or the customizations we're going to use I'm going to pass that in and the rest is really just the same so let's test it uh let me I'll say query engine uh we just need to pass it a string I'll be consistent uh we'll use the same thing that we have used so far so we'll do what AWS service is a good choice to deploy llama 2 models uh this is the same we this is an object uh we need to actually call the query method on it and here is our result um you see like this is one of those idiosyncrasies of different models when we used the when they first ran this up top we were using open AI it gave us a tur answer um here we're just using a different llm still getting the same documents it gives llama 2 is um more longwinded so it says Amazon Bedrock is a good choice to deploy LL 2 models and so on um yeah it has a little bit more information and we could um yeah we could run search again uh to get a um a simpler version that doesn't have all of this extended um information around it be easier to process uh but I think that you've come away here with enough that you don't in 45 minutes know everything about building a rag system but between this code and a general understanding you could run some experiments and uh and then have something to build off of uh this is something that I've done an immense amount of that's all been driven by the needs of projects where people have their own data they want a large langage model that works off that data and this is uh the framework that I use so I know we've got a bunch of questions um I don't know if um reer Richie have have queued those up but happy to to answer accumulated questions or new ones brilliant uh thank you so much Dan I have to say I thought it was very cool way it's almost exactly the same code whether you're using like the opena model or whether you're using llama 2 so it justes make it easy if you say okay we want to change the model see if it performs better just yeah pretty simple swapping code there all right we're going to go to audience questions in a moment uh before we get to that may me just stop sharing your screen I just have a question for you can you uh you mentioned that you've been building quite a lot of these things so can you just talk through some of the most common use cases for this sort of system yeah you know um there's one that I'm working on most actively at the moment uh and that I've heard so there's I've done this twice and I've heard of it happening a lot so you've got a uh customer service function today that's done with a mix of to a lesser extent chatbots that are just not very good or historically chat Bots that were just not very good and many of us have had an experience of typing something into a chat like H I just want to talk to a human or humans and the reason for that is that humans actually know the company's policies so uh something that we're doing is building a chatbot where the someone goes in they ask a question and now the chatbot can go and read and basically have access to the whole um database of company policies it looks up the relevant policy and then it crafts an answer that is um you know and that both says what the company policy is and is responsive to the users's um question one of the really nice things about this use case is that you can imagine hypothetically a few years ago we could have done a search but with search if you didn't use the exact words that showed up in the company policy it didn't know synonyms and one of the great things about large language models is they convert everything into an embedding and people can um can ask a question that doesn't have any overlap in words with the exact policy and maybe the policy also was like written with a bunch of jargon in the large language model can simplify it and give really um human answers so that's um that's something that uh uh we've done this something which also is to give another example of a use case um we've built an internal tool for a Fortune 500 company uh their HR professionals just want to be able to identify so HR contracts um with employees tend to be very long and for startups they tend to be very consistent but for Enterprise companies that have accumulated a bunch of companies that they've purchased over time there's a lot of idiosyncrasies between them and and so uh we built a project for internal HR employees and they want to know um do we ever have any Clauses in any of our contracts with our employees that say exactly x uh and we built a rag system for them that can answer that and that saves them an immense amount of time through versus trying to just browse through on their own I should really like those I mean I know there's a few people going oh yeah artificial general intelligence is coming it's going to save the world but actually you know I'll settle for like a chat bot that I don't want to strangle through the computer screen uh so these simple things are are just a little quality of life improvements that definitely appreciate um okay so uh we've got a few questions from the audience uh so the first one just coming from Mark so Mark just asking about um how are embeddings created and how are these vectors created so maybe can you just give a quick overview of like how embeddings work in terms of converting text to numbers and how the the search works to like how do you find relevant text okay so I'm going to answer this in three parts the first which is not responsive to Mark's question but is just I want to say what a cool thing that's coming out in the um this sort of like the frontier in the last few weeks is some of the vector databases are starting to and and llama index is now starting to to support images so that instead of just text documents if you have a bunch of images uh that um could be let's say diagrams about how to how to assemble something um those can now also be stored in these databases in Vector databases and those can also be fed into a large language model and because now a lot of the large language models are multimodal um both documents and images can be stored that way but that was actually an a side I got excited about let's talk about how um documents can be converted into vectors that would be I said there three parts one is a cool thing then the second two parts are answering Mark's question how are they converted into to vectors and then the third is how does the vector database identify what documents are relevant so um when you uh some of the deep learning this is the first part of that which is how do you create the vector or the so-called embedding this is pretty techn and so I'm going to give the best answer in one minute that I can give but you could actually take a um uh a deep deep learning courses on data camp or or anywhere else um and get a deeper dive into this so large language models operate in a few modes or when they're trained they have a few different goals they can pursue the one that people talk about the most and some people mistakenly think is the only one is take a bunch of Tex and try and predict the next word so uh the way that we would let's say go through Wikipedia we say here's the first seven words of an article predict the eth then could take words two through eight and predict the ninth and so on and the way that they do that is with um a bunch of operations the first is the first is to take uh embeddings of individual words and then aggregate those up into embeddings of sentences when you first start training the model you just literally randomly assign random numbers to the vectors for each of the words so that might be a poor way to do it that might lead to poor results but I think conceptually like the idea of random number generation we can probably wrap our head head around and then in the training of the model you continually figure out is there a way that to predict the next word which is at this point a bunch of operations on vectors is there a way that I can change the embedding for a given word in order to increase the likelihood that the model would predict the correct next word and then there's um a layer beyond that for how do we aggregate those words um this is something called Transformers um and these recurrent these sort of like the next level of how the model works but um a large language model even like the ones that you're familiar with you know if you go into chat GPT it was pre-trained on exactly this task and um then when you run it let's say you're using chat GPT the last thing it does is um there's a a little bit of a caveat to how chat GPT Works where it's using this thing called rhf but it is approximately speaking predicting the next word in what would be a good answer and um the step right before it converts that into a word it actually has a vector representation of the meaning of everything that has been written so far so when we um have a document we run it through a large language model and instead of predicting the next word or predicting the word that would be a good response um we stop just short of that and we say we're going to take the vector representation or the vector representing the document we have so far we're just going to store that in the vector database so this is like I said super super complex I apologize if I couldn't quite get to the idea of it in so briefly um but the third part is less complicated so that last part of how do we determine what's relevant this part I can say briefly so if you have two vectors there are a lot of ways to measure the distance between them and this is covered like in you know if you do it in uh if you go along in math um for long enough you probably at some point learned compar distances between two vectors so the most popular is ukian distance that's like if I have two points how do I just measure uh the distance between them um and then so there are other ways of measuring distances but we just for all of the documents in our database calculate the distance between the vector for that document and the vector that represents the query and then we find the two that have the shortest distances and that's our measure of similarity long-winded answer but um yeah that's I thought you you done very well for just like explaining sort of many years of uh deep learning research in a few minutes so I guess uh my sort of very simple like interpretation of that is words are hard to deal with but we know how to do linear algebra so we convert words into numbers and then that's right just uh simp simple not quite well University level math all happens in the background and the magic transition bit that's your embedding model yep I want to actually quickly I'm looking at some of the questions mark just popped in a really good one I want to uh okay let me have a look at Mark uh this one uh sorry there's a newer one let me oh hold on the newest one there okay another one from Mark yeah so this actually I want to address this because I've seen this done wrong actually llama index makes it hard to do wrong but um can there be a mismatch between the chosen embedding and the model because it seems that we start with opening and then use Zephyr so large language models get used in two ways one is to create the embeddings and then the other is to create the answers so as long as we are consistent in how we create the embeddings and that the consistency there is when we create the embed create the embeddings for the documents that go into Vector database we need to use the same way of creating the the embedding of the query so that when we look at there similarity there they're sort of comparable but you can take any model you want for the embedding uh that leads to the measure of which documents are similar and then when we synthesize an answer all that's getting is text that's getting the the question and the text of the document so you can use any um model you want for um for answering the final question okay so just G to use the same one in both cas and you should be fine you can use the same one uh or you could use different ones in either way as long as you're consistent with one for embeddings and then um whatever yeah as long as you're consistent with embeddings you have a lot of flexibility all right super uh okay so uh next question uh so related to this this one comes from Jeremy so I think one of the new cases you talked about was like retrieving news articles and so quite often you want more recent documents to be uh considered more relevant than older articles ah so perect yeah how how do you deal with that time component when you're retrieving so um first of all training models is a not so messy like we've gotten better we as world have gotten better at it but it takes a lot of compute takes from hours to it it takes a a lot of time and a lot of compute and so fine-tuning a model is a meaningful ende on the other hand inserting things into databases is relatively straightforward and so I can take my Vector database and on a daily basis just have a little job that runs that takes any new documents inserts them into the database it'll typically take seconds or if you have a lot of documents it might take a couple minutes but even that would be like a lot of documents and then the retrieval augmented generation system whenever it gets a new new query it's going to go consult the database and the database in real time figures out what are the most relevant documents and so you can just set up a little um recurring job to keep the database up to date and that means as a result your system will always have up to be based on upto-date data um so when you're um doing that find the the most relevant documents normally and you said before it's using like dot product or some kind of linear algebra sort of thing to say these are Rel documents is there a way to include a Time component for that say this document is newer therefore ret well so there is um in a sense there's three ways to think about this and two of them work so one is I'm not gonna one of those three ways is I'm going to ignore timing the the second way uh is to say I only want articles from the last X number of days this would use something called called metadata filtering and metadata filtering is a way for you to when you make the request say I want to run this filter before I even look at document similarity um and so I only want articles from the last 30 days or from only want articles from a certain time period or only want articles from a certain publication and so metadata filtering lets you do that the Third Way which can't be done is to say I want some sort of like um measure of similarity the conventional way of just document similarity but then like weekly favor newer over older in some sort of fuzzy way I suppose that could be programmed but um there's no simple way to do that really it's um if you care about recency or some other metadata you would filter on that and then find the new the nearest or most similar documents that meet that criteria okay all right that's useful to know um all right this sounds like aight technical question from Ben saying uh what's your point about using cross encoder reranking we get a lot of hallucinations using plain uh context generation based on the embeddings uh so yeah basically uh can you tell us about cross encoder reranking you know I'm uh Ben's got a an angle on this that I'm not familiar with so I don't I've not I'm not even familiar with Crossing coder reranking okay uh Ben you're teaching the next one of these sounds like yeah yeah you're pretty advanced in what you're asking okay uh right next question comes from uh Sak saying uh so you talked about like one of the use cases being a policy bot so maybe like looking at Contract something like that uh do you have any recommendations uh around how you might go about doing that yeah I mean one of the things that for all for many of us before we worked in large language models we worked in just conventional predictive Ai and predictive machine learning um one of the recurring themes in predictive machine learning is to First create a way of quickly and reliably measuring the quality of your results um for me that can be uh I've got a bank of 10 or 20 different queries that I want to make sure I can answer well ideally eventually it gets up to 100 um I can I will then build something very very simple and that could be as simple as the code that you've seen today not fiddling with any changes and think of that as my Baseline running um run that against uh the set of documents figure out like are those good enough for me to trust if not then I am going to um have to iterate so that I have something that is sufficiently high quality um and then uh once I've got and then once I've got that the next is just sort of the infrastructure piece of it so um we typically use uh fast API as to as the entrance into the back end fast API has some code that um well so typically we've got the vector D Vector database that lives as a long long running service so that you don't need to reload that database every time we get a new request in um but then we've got a fast API backend it will get a query uh and then basically run code this um query engine. query type code uh hitting the um the vector database getting the response uh and then sending that back to the front end and then that um that fast API back end uh that can live either on a service like ec2 or on AWS Lambda or you know if you're on some other Cloud their um serverless these things tend to be lightweight enough and they have sufficient state that you can run it either way and uh I would say more than half the time we're just running it on some serverless service like awsm to or the equivalent on other clouds okay so it sounds like um Beyond this sort of Ming about with just the the AI side of things is actually quite a lot of sort of backend infrastructure and messing about with cloud services that you need to deal with yeah I mean I think that's probably typical for productionizing almost anything yeah yeah there there is some infrastructure that we always always needs to set up um but uh yeah just in some ways a typical amount for deploying ml systems into production okay super uh we're coming to time there are one or two more great questions from the audience I want to get to but before we go to that we've got uh four more webinars before the end of the year so uh for those of you who are interested in coming back uh tomorrow we've got a session session on uh the secret to successful data literacy program so if you're interested in learning or having your colleagues learn more about AI then please do come along to that on Friday we've got another code along uh on AI so this time it's on fine-tuning GPT using the open AI API and then next week' got two sessions so there's a beginner session on exploratory data analysis in Python and then Wednesday uh we a session on using uh snowflake the data warehouse for sales and marketing analytics so uh please do go to datac camp.com webinars and uh register for those I will see you uh for those later this week and next week um so uh let's do two more questions all right so this one comes from Marcel saying um if you got uh very different amounts of text uh in each um entry in your vec