Testing LLMs Smarter: Multi-Model Experiments & Insights

Key Takeaways

Okay, I have this bedrock knowledge base I have indexed right with large amount of data. Okay, and that I have also uploaded a ground truth file of 50 questions and answers. I will I want every different combination with different prompt options, right? I can have a zeros prompt, one shot, twoot, threeshot prompt and I attach my prompt file as well. And then I have this system prompt, right? And I have selected four different models. Right? Now remember these could be caching models, routing models, it could be with guardrails, without guardrails. Different options I have selected here from my flow to registry. I have selected two different options for uh nearest neighbor. So when the vector is retrieves, I want top three chunks or top five chunks. I'm using Ragus for my evaluation and I would like model one which is not being used here to test. Right? So I you don't want to double dip you don't want to use evaluation model same as the retrieval model right that is doing inference. So retrieval models are this model 5678 but my evaluation is being done with a larger model in this case it's I'm just naming it model one and my queries would be embedded with Titan embedding uh from bedrock I could use something else as well these are choices I have right so you can go through these right you can select multiple prompts options for uh you can select your knowledge base that you want to select from or kind of just test it out of the box without knowledge is you can use multiple KN&N options like I said and you can choose your model to evaluate with Ragus. If you're not familiar with Ragus and DPL are two standard libraries for LLM evaluation. There are multiple ones actually but these two are pretty commonly used and they are focused on context precision, context recall, right? And then there is maliciousness and all that. Depending on your use case, you can use specific ones but context precision and recall are very commonly used, right? So we are going to do the same thing. We are going to use ragas for context precision and recall. So you'll get results of multiple experiments. So the if you see the number of combinations could be very well 30 40 different combinations in flowch. If you set it up like this and say create project, it will actually run these 30 40 experiments for you using the models you registered with the combinations and we'll do the evaluation with the ground truth file that you provided and report everything on the dashboard like this. Right? And it will tell you what the expected cost is. That's called directional cost. It will tell you how much time it took and the corresponding precision and recall. Answer relevancy is how relevant the answer is. So that's the most important metric in this particular case. So I sorted this table by uh answer 11 C. You'll notice one thing right that inferencing model is same model 7 right but it has different context precision and answer 11 C as well as duration