On Randomness in Agentic Evals

1. 1Collect a large number of agentic trajectories to estimate performance variance
2. 2Analyze the variance in single-run pass@1 estimates to determine reliability
3. 3Consider using multiple runs or alternative evaluation metrics to improve reliability

AI researchers and engineers working on agentic systems can benefit from understanding the limitations of current evaluation methods, as it can impact the development of more robust and reliable models

💡 Single-run performance estimates may not be reliable for agentic systems due to substantial variance

🤖 Agentic system evaluations may be flawed due to high variance in single-run performance estimates