Early days of RAG and LlamaIndex - Jerry Liu

Key Takeaways

All right, so about that history of rag, modern rag was actually kind of born in 2022 then, right? Um, yeah, I mean I think like I forgot the exact date of the like the original retrieval augmented generation paper, which is not by me. Obviously it was by others. And so this is either in 2021 or 2022 or even 2020. Um, and the like it basically proposed this overall idea of oh, you know, you take in some set of like documents and then you watch it embed them, like put them through an embedding model and then put it into some some storage system that's able to like serve like the relevant documents through retrieval, right? And so that's why it's called like retrieval augmented generation. Like you want to do like a retrieval pass over some storage system before you actually put it into the LLM prompt. So that kind of that resurfaced basically as more and more people start building with LLM apps. People like kind of start discovering that oh, hey, this thing is like a cool idea. And my initial version was not doing that. It wasn't using embeddings because at the time, I don't know why. I think it might have just been like due to like a design like my my goal at the time when I first started was not necessarily to make this useful. It was to just like do something cool. And so to me like doing something cool was like oh, what if we just didn't have embeddings or I thought about it briefly, but I was like what I really wanted to do is just have the LLM figure it out completely on its own, right? And I I I still think that would be a quite an interesting concept. Like instead of just like, you know, relying on a separate model, just have a language model completely similar to like a human, just completely figure out how to reason, organize things and then also traverse them via text. Um, yeah. And that kind of reflects in the current state of Llama Index, right? Because I mean, you it's kind of the central, I mean, as far as I can see, it's kind of the central part of Llama Index that you use language models as well in during the ingestion process, not only in the in the generation process. Yeah, so the default rag paradigm really only uses the LLM at the very end. So you have like ingestion, like ingestion doesn't need LLMs. You just take in you know, some data, parse it and then you just like chunk it using an algorithm. And of course you use an embedding model to put it into some vector store. And then the retrieval process doesn't use an LLM cuz at its simplest it's just like top K embedding look up. So like, you know, you look up stuff by embedding similarity. And so in a standard rag pipeline, the general the the way the place where LLMs actually come in is at the very end. And it's only responsible for um kind of just like synthesizing an answer from a piece of unstructured text. And to be totally honest, like, you know, it like even like at the start when we were just like implementing this, I thought it was a little basic and it didn't really like make use LLMs to its full potential, right? Cuz like LLMs are not just for generation and and simple reasoning. They can actually help you make decisions. They can actually help you like like a greater layer of like just like understanding and decision making. And so if you really wanted to make these systems more interesting, you could use LLMs kind of like at the beginning. So for instance, during the data ingestion phase or you know, during query time. Instead of just using it at the very end for generation, use it for like query understanding, use it for like processing, like evaluating like the quality of your retrieved context. And then for instance, like not only just retrieving from vector store, actually using a variety of different tools. And so on the ingestion side, the places that you can use LLMs. And so this this this overall concept is pretty interesting, which is um the whole point of like ingestion is to process data for your LLM app. And so that's kind of like ETL for LLMs, right? But you can also use LLMs for ETL because, you know, LLMs have an inherent capability of understanding unstructured data and transforming it. And that part I think is interesting. Like so for instance, let's say, you know, for each unstructured document, you wanted to extract like a summary, the table of contents, like, you know, extract like a set of like topics or tags for each Basically, you can figure out a clever way to prompt the LLM by feeding it in a bunch of data from the document to basically first extract out a set of like structured annotations or tags. And this represents like a data transformation basically cuz you're trying to like feed in some input unstructured data and transform that into structured data. And then you can basically attach those tags on top of the unstructured data as well. And so these like this is just an example of like metadata extraction that's also powered by LLMs. And this is something that, you know, uses LLMs, but is also like, you know, useful for just like the any sort of downstream application you want to build. Because if you're trying to build a rag system over this, having metadata tags is often times very useful. It gives you like better retrieval results, better generation quality and and all this types of types of things. And so I I think that interplay between LLMs and kind of like data transformation is very interesting cuz you can use it for like in the middle, but also you it helps for any sort of like applications you want to build later on. Yeah, that's pretty some advanced rag techniques over there. And it kind of brings goes back to a little bit to your I mean, it it's a little bit adjacent to your original idea of creating this kind of systems, right? Yeah, I think I think the, you know, I I thought about this in the at the beginning of the project, but the project was not really um like close to kind of like realizing that vision at the time. But if you think about like the overall picture of like where I think LLM power software will evolve, it's basically like um there's a new type of like data like a data stack that's emerging, a new set of operations within that data stack to basically power like like AI software. And to really like kind of like we want to basically provide the right tooling to help developers build that data stack. And so this is helping them figure out how do you like, you know, move data from one place to another specifically for LLMs to use. And that could include like LLMs in the middle as well. And this also includes the orchestration piece on top of that data. How do you figure out how to get LLMs to interact with the data through these different types of interfaces? Awesome. Hello everyone. Thank you for watching the clip. If you enjoyed the clip, you can click the link in the description or somewhere right here on the screen to watch the full conversation for free. If the podcast is not up yet, you can always subscribe to my Patreon to get early access and support the channel and this podcast. Or alternatively, you can subscribe here on YouTube, click the bell icon and you will be notified when the full episode comes out next week. Thank you very much for watching and I will see you next time.