The open-source LLM eval frameworks I actually compared, and the question that sorts them

Title: The open-source LLM eval frameworks I actually compared, and the question that sorts them

URL Source: https://medium.com/@ethan-writes-AI/the-open-source-llm-eval-frameworks-i-actually-compared-and-the-question-that-sorts-them-b19e978d391d?source=rss------llm-5

Published Time: 2026-06-21T02:11:08Z

Markdown Content:
# The open-source LLM eval frameworks I actually compared, and the question that sorts them | by Ethan Walker | Jun, 2026 | Medium

[Sitemap](https://medium.com/sitemap/sitemap.xml)

[Open in app](https://play.google.com/store/apps/details?id=com.medium.reader&referrer=utm_source%3DmobileNavBar&source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40ethan-writes-AI%2Fthe-open-source-llm-eval-frameworks-i-actually-compared-and-the-question-that-sorts-them-b19e978d391d&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)

[](https://medium.com/?source=post_page---top_nav_layout_nav-----------------------------------------)

Get app

[Write](https://medium.com/m/signin?operation=register&redirect=https%3A%2F%2Fmedium.com%2Fnew-story&source=---top_nav_layout_nav-----------------------new_post_topnav------------------)

[Search](https://medium.com/search?source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Fmedium.com%2F%40ethan-writes-AI%2Fthe-open-source-llm-eval-frameworks-i-actually-compared-and-the-question-that-sorts-them-b19e978d391d&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)


# The open-source LLM eval frameworks I actually compared, and the question that sorts them

[](https://medium.com/@ethan-writes-AI?source=post_page---byline--b19e978d391d---------------------------------------)

[Ethan Walker](https://medium.com/@ethan-writes-AI?source=post_page---byline--b19e978d391d---------------------------------------)

Follow

3 min read

·

1 hour ago

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fvote%2Fp%2Fb19e978d391d&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ethan-writes-AI%2Fthe-open-source-llm-eval-frameworks-i-actually-compared-and-the-question-that-sorts-them-b19e978d391d&user=Ethan+Walker&userId=37952ea64570&source=---header_actions--b19e978d391d---------------------clap_footer------------------)

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Frepost%2Fp%2Fb19e978d391d&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ethan-writes-AI%2Fthe-open-source-llm-eval-frameworks-i-actually-compared-and-the-question-that-sorts-them-b19e978d391d&user=Ethan+Walker&userId=37952ea64570&source=---header_actions--b19e978d391d---------------------repost_header------------------)

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fbookmark%2Fp%2Fb19e978d391d&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ethan-writes-AI%2Fthe-open-source-llm-eval-frameworks-i-actually-compared-and-the-question-that-sorts-them-b19e978d391d&source=---header_actions--b19e978d391d---------------------bookmark_footer------------------)

[Listen](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2Fplans%3Fdimension%3Dpost_audio_button%26postId%3Db19e978d391d&operation=register&redirect=https%3A%2F%2Fmedium.com%2F%40ethan-writes-AI%2Fthe-open-source-llm-eval-frameworks-i-actually-compared-and-the-question-that-sorts-them-b19e978d391d&source=---header_actions--b19e978d391d---------------------post_audio_button------------------)

Share

“Eval framework” covers app-output graders, RAG-specific scorers, and ac