Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors

1. 1Identify construct-irrelevant factors that may influence LLM-based scoring systems
2. 2Analyze the robustness of LLM-based scoring systems to adversarial conditions
3. 3Develop strategies to mitigate the impact of construct-irrelevant factors on scoring systems
4. 4Evaluate the performance of LLM-based scoring systems in comparison to human raters

AI engineers and ML researchers can benefit from understanding the limitations of LLM-based scoring systems, while educators and test developers need to consider the potential biases in automated assessment tools

💡 LLM-based scoring systems are not immune to biases and require careful evaluation and mitigation strategies

🚨 LLM-based scoring systems can be biased by construct-irrelevant factors 🚨