Simulating & Evaluating Multi turn Conversations

Most LLM applications today are chat-based. How would you evaluate the conversations? One way to evaluate is to create a simulation with another AI — where one model plays the role of the user to interact with your chat-based application, and you observe how the conversation unfolds over multiple turns to test for metrics such as helpfulness, consistency, and goal completion. We’re excited to launch OpenEvals — a set of utilities to simulate full conversations and evaluate your LLM application’s performance. OpenEvals: https://github.com/langchain-ai/openevals Notebook: https://github.com/catherine-langchain/agentevals/blob/main/multi-turn-eval.ipynb 0:00 Introduction 0:40 Overview of Multi-Turn Simulation in OpenEvals 2:34 Example of Evaluating a Deployed Agent over Multi-Turn Conversation