▶ Videos →

📰 Dev.to · Maya Andersson

5 articles · Updated every 3 hours · View all reads

All Articles 133,316 Blog Posts 137,787 Tech Tutorials 34,567 Research Papers 25,941 News 18,856 ⚡ AI Lessons

Your LLM-as-judge has a position bias you are not measuring

Dev.to · Maya Andersson 🧠 Large Language Models ⚡ AI Lesson 2w ago

Your LLM-as-judge has a position bias you are not measuring

If your pairwise judge sees answer A before answer B, it tends to prefer A. If you never swap the...

We fixed the worst prompt variant. It got better. That doesn't mean the fix worked.

Dev.to · Maya Andersson 🧠 Large Language Models ⚡ AI Lesson 3w ago

We fixed the worst prompt variant. It got better. That doesn't mean the fix worked.

A pattern I've seen on more than one team: weekly eval run finishes, someone sorts the leaderboard,...

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

Dev.to · Maya Andersson 🧠 Large Language Models ⚡ AI Lesson 4w ago

I checked six LLM-as-judge tools against human labels. The scoreboard was the wrong thing to read.

Most LLM-as-judge comparisons rank tools by which one gives you a number fastest. That is the wrong...

We put confidence intervals on our LLM-judge scores. The error bars ate three weeks of "trend"

Dev.to · Maya Andersson 🧠 Large Language Models ⚡ AI Lesson 1mo ago

We put confidence intervals on our LLM-judge scores. The error bars ate three weeks of "trend"

We track weekly agreement between an LLM judge and human labels (Cohen's kappa) on a sample of...

More eval traces will not stabilize your kappa. Stratify the ones you have

Dev.to · Maya Andersson 🧠 Large Language Models ⚡ AI Lesson 1mo ago

More eval traces will not stabilize your kappa. Stratify the ones you have

TL;DR: Our LLM-as-judge agreement (Cohen's kappa against human labels) swung between 0.41 and 0.63...