Accelerate agent testing with Evals for Agent Interoperability

Key Takeaways

Hi, I'm Darini Jois, principal product manager at Microsoft. We heard from customers that they struggle to evaluate agents effectively. They often don't have a standard way to measure performance. >> And I'm Adashkan, principal applied scientist. That's why we built Eval for agent interrop, an open-source tool that brings structure and transparency to agent evaluation so you can benchmark across realistic scenarios with confidence. >> We empathize with agent developers who often spend weeks building scenario specific data sets for testing, a process that's both time consuming and diverts focus from actual agent innovation. Evals for agent interrop solves this by providing curated evaluation suites with synthetic data so you can start testing immediately. Plus, you can configure your own rubrics. Whether you care about tone, compliance or accuracy. >> Customers have expressed that even when tests exist, results are often opaque. They can't isolate why an agent passed or failed. With evs for agent interrop evaluations are systematic and transparent. You'll see granular metrics like tool use accuracy, latency, and groundedness. So you know exactly where your agent shines and where it needs work. >> In a world where countless agents claim the same capability, how do you confidently choose the one that truly meets your standards? >> Our leaderboard aggregates results across runs, letting you sort by what matters most. speed, compliance, accuracy, so decisions move from guesswork to datadriven. >> This is just the beginning. Today, we offer a small set of curated scenarios. Soon, eval will be standardized and scalable across a multitude of scenarios. >> Now, let's see eval for agent interop in action. Over to Alistair for a live demo. >> Thanks, Adar. Hi, my name is Alistair. I'm a principal architect at Microsoft. In this demo, I'll walk through how eval for agent interop can help you evaluate, review, and compare agents by benchmarking them against real world scenarios, making your agent development process more efficient. Now, let's jump into evals for agent interop and start by exploring the evaluation data sets. Each data set is curated to test a specific scenario. For example, in this list, we have a data set for testing email collaboration and meeting scheduling. Data sets consist of a set of tests and assertions. Let's drill into the meeting scheduling data set. Within the data set, you can see that we have seven tests. Each test case has the input that will be passed to your agent, the tools that are expected to be called, and details about what is expected from each tool call. These details give you insight into exactly what is being measured, and will be used later to evaluate the agents responses. Now that we've explored the data sets, let's get our agent ready for testing. First, we will navigate to the agents page where the list of registered agents is displayed. Here, we can add a new agent. In our case, I have one running locally that can help with meeting scheduling. Let's register it by clicking register and providing the name, the endpoint where the agent can be called, the model, and a description. Now that we've registered, we can start running an evaluation by clicking run evals. This presents a list of available data sets. Let's pick the meeting scheduling data set. Now the evaluation is running against our agent. In the background, agent evaluator is invoking the agent once for each test case in the data set, passing in appropriate inputs and logging the result. The evaluator is also providing mock tool responses to the agent to ensure that each test run is consistent. To support this, we've modified our sample agent to use agent eval's built-in MCP server. As the results come in, the view will update in real time. And depending on the number of tests, this could take several minutes to complete. After running the evaluation, it's time to review the results. For every test, you'll get a full breakdown, starting with the test description, whether it passed or failed. You can then drill into a specific test case to see the agents actual response, the expected outcome, and why the agent passed or failed the test. In the details view, you can see which tools were called by your agent, the expectation for each tool call, and the evaluator's reasoning for deciding why an assertion passed or failed. This feedback loop is crucial for iterating on your agent and quickly identifying areas for improvement. Now, let's talk about the leaderboard. As you test your agents, you can use the leaderboard to see how they stack up and if they are improving over time. Each evaluation is aggregated and ranked so that you can quickly see how each of your agents are performing against the evaluation data sets. Each row in the leaderboard shows the agents AI model and an overall score summarizing its pass rate across the selected data sets. You can filter data sets to compare agents for specific tasks. If you want the details, clicking on an agent takes you back to the scenario level results, giving you deeper insights into its strengths and weaknesses. That's a quick overview of Evals for agent interop. It's designed to bring clarity and actionable insights to agent development. If you're ready to test your agent or if you want to contribute, try out evals for agent interop by visiting our open source repository on GitHub. Here you can find the application code, the evaluation data sets we are open sourcing, and the details needed to get up and running. We're excited to see what you can build and get your feedback.