Insights using Google Gemini for Summarization and Retrieval

Key Takeaways

I'd like to present two use cases that highlight the value of Weights Andes weave combined with the powerful large language model Gemini Pro the first use case is summarization of long documents many of our customers face time constraints but need to process a lot of information this is where Gemini's large context window becomes particularly beneficial the Second Use case is the Morgan Stanley research rack model many customers use retrieval augmented generation to access and respond to questions related to proprietary or internal documents let's start with the summarization use case we can see the model is defined in weave where we can see all the parameters including the model name which refers to Gemini 1.5 Pro latest checkpoint the promt template uh in the summary model refers to a Json schema we can also see Json schema stored as one of the input parameters into the model we can see uh a title and a summary that we want to get uh as an output of the model we also see that um the summary should be a plain short text without markdown and we can also see that the prompt refers to the length of the document which which should be less than 200 words defining these requirements in the prompt is good but evaluating your llm application against these requirements builds confidence that the model actually meets them by defining your model in weave you can easily reference and deploy it for example on gcp these setups provide a valuable observability by storing all your model call we can see all the calls to our predict method and inspect the inputs and the outputs and see how the model is doing now for the evaluation we want to check that our summary um is generated according to the defined schema and is less than 200 words we have a data set of 18 long papers and evaluate our model on two metrics uh the formatting and the conciseness we check if the formatting meets our requirements and if the word count stays within our set limit we can see that we achieve around 89% accuracy uh on both metrics but we need to dig deeper to understand where the model fails in most cases uh the model meets our requirement but it fails in two instances when we inspect um these instances where the model is failing and we can see that the problem is with uh the predict method and the error that we're facing is 504 deadline exceeded error and that tells me that uh whenever we actually reach uh the model API it responds reliably but in some cases We are failing to reach the API properly and that means that's something that related to our setup for example our um request quota might require an increase and by uh looking at our evaluation in this way we can see where our application might um require an improvement and we can get better results over time the Second Use case uh involves the Morgan Stanley research rack model here we use the predict method um and highlight how easy it is to instrument with in your code um you can see that um instrumenting we is as simple as adding at weave. op decorator to the functions you want to trace in this case we are dealing with a simple uh predict function but in many cases you will have nested functions and as you decorate each of these nested functions in weave you will get a detailed Trace that will allow you to debug where your application is failing if we go back to the call and we can see that a question that a user asked uh is actually not responded by the model and that's something that we might want to check in detail and understand what's Happening Here we can see one of the input parameters at context is set to false and if we look at the code again we can see by setting this parameter to false we're actually not benefiting from any uh context document that uh might uh might need to be added to the to the prompt so in this case the model does not have any insight into how to answer the user question and it properly responds I didn't no if you look at one of the other instances where uh the same uh method is called uh with the same question but now uh our context argument is set to true we can now see that um the model was able to properly respond to this question in summary adding weave to your llm application is super forward by using the at we. op decorator you get powerful observability you can save uh data sets you can save models you can save evaluations and that allows you to build llm applications with a better productivity and deploy them with confidence that you're actually meeting your user requirements and solving uh the right business problems