On Randomness in Agentic Evals
📰 ArXiv cs.AI
Agentic system evaluations may not be reliable due to substantial variance in single-run performance estimates
Action Steps
- Collect a large number of agentic trajectories to estimate performance variance
- Analyze the variance in single-run pass@1 estimates to determine reliability
- Consider using multiple runs or alternative evaluation metrics to improve reliability
Who Needs to Know This
AI researchers and engineers working on agentic systems can benefit from understanding the limitations of current evaluation methods, as it can impact the development of more robust and reliable models
Key Insight
💡 Single-run performance estimates may not be reliable for agentic systems due to substantial variance
Share This
🤖 Agentic system evaluations may be flawed due to high variance in single-run performance estimates
Key Takeaways
Agentic system evaluations may not be reliable due to substantial variance in single-run performance estimates
Full Article
Title: On Randomness in Agentic Evals
Abstract:
arXiv:2602.07150v3 Announce Type: replace-cross Abstract: Agentic systems are evaluated on benchmarks where agents interact with environments to solve tasks. Most papers report a pass@1 score computed from a single run per task, assuming this gives a reliable performance estimate. We test this assumption by collecting 60,000 agentic trajectories on SWE-Bench-Verified, spanning three models and two scaffolds. We find substantial variance: single-run pass@1 estimates vary by 2.2 to 6.0 percentage
Abstract:
arXiv:2602.07150v3 Announce Type: replace-cross Abstract: Agentic systems are evaluated on benchmarks where agents interact with environments to solve tasks. Most papers report a pass@1 score computed from a single run per task, assuming this gives a reliable performance estimate. We test this assumption by collecting 60,000 agentic trajectories on SWE-Bench-Verified, spanning three models and two scaffolds. We find substantial variance: single-run pass@1 estimates vary by 2.2 to 6.0 percentage
DeepCamp AI