Discover LlamaIndex: Ask Complex Queries over Multiple Documents

Key Takeaways

hey everyone welcome to discover llama Index this is a new tutorial series that we're starting up to tell you guys all about the latest features of llama index and how you can really use it to supercharge your large language model applications especially when dealing with large amounts of data I'm Simon the co-founder and CTO of llama index and today I'm super excited to tell you guys just a little bit more about our latest feature the sub question query engine before we dive down into the details let's recap at a high level what a query engine is in Lama index a query engine really is a high level natural language interface that we build over your data source so that you can very easily ask a question over your data and extract very nuanced insights out of it now the key challenge in building such a unified query interface over your data is that there's often a wide range of complicated questions that you might want to ask over multiple data sources now for example you might have multiple financial documents and you want to compare and contrast the financial statements of multiple companies over multiple years now just naively combining all those data sources and doing topk retrieval of Chunk documents for a final synthesis is simply not good enough to get a very good and nuanced answers for complex questions now how do we do this the sub question query engine is one step in the direction of answering these complex questions the key intuition is that when you have a complex multi-part question the best way to deal with it is to First decompose it into multiple sub questions that are relevant to specific data sources now for example here we have three data sources and we might want to decompose the question that targets a subset of them first and then combines the response over multiple data sources to get the final answer now this approach really leverages the composability aspect of our framework by first defining sub query engines over each of these data sources and then Define a top level sub question query engine on top now how does this work exactly to start when given an initial complex questions we will use the large language model to generate sub questions given the description of our data sources that are available then we will execute the sub questions on the select data sources and then gather all the sub responses and then lastly we will take all the relevant contacts from these sub sources and synthesize the final answer now let's take a look at this in action to help us do some complex analysis over multiple financial documents which are called 10ks okay now that we have our notebook open we can finally go ahead and play around with some code before I dive down into the notebook let me just show you the data that we'll be working with so we have these two PDFs which are the lift 10K document for 2021 and the Uber document for 2021 so these are fairly large financial documents that I think have oh yeah 307 Pages for this one and 238 Pages for this one so quite a lot of data here and it's very heterogeneous there are texts bullet points and I think a whole bunch of tables where a lot of financial terms are found okay going back to the notebook let's just uh go straight into it by doing some imports we're going to first configure our large language model service um here we're just setting the temperature to zero so the output is more deterministic we're using gpt3 and then basically setting the max token to negative one so that we don't limit the amount of outputs for these models we're going to wrap the Ln predictor into our service context which we'll be using to pass into our indexing query engine now for the first step we're going to use this simple directory reader to read these two files separately so I think this might take a little bit of time since these are 300 Page documents okay now we have the lift 10K loaded 238 documents I think that's correct yep and see how long this takes okay there we go now the next step is to build our indices here we're just going to use the vector store index what is happening under the hood here when we call the index style from document is that each of these documents will be chunked up into text chunks we'll be calling the embedding API to get a dense Vector embedding for each of these chunks and save it to R in-memory Vector store so this takes a little bit of time just because there's a lot of data so in total 500 pages of text that needs to kind of pass through the API calls and get back the responses now okay so we finished building the left 10K index was 379 notes you can see that is a little bit higher than the 238 pages I think most of the pages gets broken to one or two nodes given our default chunk size right now okay many time now just waiting for the Uber index to be built okay there we go now we're going to build individual query engines for each of these indexes here when we do s-query engine what we're really what's really happening under the hood is uh we're setting up the configuration so the query engine knows what to do at great time now the next step is to add sort of the metadata to identify these query engines to the top level sub question query engine here we're just specifying the name of this query Engine 2 and also giving a description so the top level large language model can know how to make use of this data source okay now that we're we have that built we're ready to run some queries the first one that we're going to ask is to compare and contrast the customer segments and geographies that grew the fastest over the lift-in Uber documents so as we run these you should be able to see various sub questions being generated and executed on the sub query engines before synthesizing the final answer okay so we generated four sub questions and the first one is what customer segments grew the fastest for Uber being asked on the Uber 10K documents um you can see a sub response generated another question being asked for the Uber 10K about what geography screw the fastest in 2021 the corresponding responses the customer segments for Lyft and then the geography that grew the fastest foreclosed so one thing to know is that I think given the context here the model actually decided that not possible to answer this question was to give in the context um okay now that all the sub questions have been generated all of these question and answer pairs will be used to create a final response and you can see yeah the large language model is pretty good at kind of stitching together all these information for a final synthesis and that really talks about um the various aspects comparing between lift now just to Showcase another example we can compare the revenue growth of uber and Lyft from 2020 to 2021 this should be a pretty straightforward answer that really just depends on querying the right index using the right sub question so in this case we only needed two sub questions one is the revenue girls of uber being as an Uber document and the revenue growth of lift being asked on lift document and as you can see we have a clear comparison between the two as the final answer where Uber had a higher rate of Revenue growth than left cool that's all hopefully you enjoyed this tutorial